Repository navigation
Description
Split out from the profiling discussion in #3662 (the MixEncodeTransmitData refactor thread) at @ann0see's request. This is the UDP-send finding from that profiling — out of scope for the mixer refactor, but a larger lever.
Context
While profiling the headless server for the mixer vectorization work (#3662 / #3739), the bigger surprise was where the rest of the CPU goes. Once the mix loop is vectorized, a large share of server CPU is the UDP socket path, not the audio code — and the send side scales with packet count × client count.
The server issues one sendto per client per frame: about N sends per frame for N connected clients. The payloads are tiny (tens to a few hundred bytes), so the fixed per-syscall cost (user/kernel mode switch, socket lookup, UDP framing) dominates the actual data movement. At N=50 that is ~50 send syscalls per frame, tens of thousands per second, most of it fixed overhead that grows linearly with the number of clients.
Proposal
Batch the per-frame outgoing packets with the Linux sendmmsg(2) call. The server already produces one outgoing packet per client per frame in CServer::MixEncodeTransmitData; instead of sending each one immediately, queue them and emit the whole frame with a single sendmmsg at the end of OnTimer. Each mmsghdr carries its own destination msg_name, so the different client addresses are handled in one call. This turns N send syscalls per frame into 1.
Measured result (working prototype)
I built a prototype and measured it. Batch storage + one sendmmsg flush per frame at the CSocket layer; CChannel::PrepAndSendPacket routes server sends into the batch, the client stays per-packet, and non-Linux platforms fall back to the existing per-packet path (compile-guarded).
Load = 20 synthetic tone clients, server pinned to 1 core (AMD EPYC 7R13, Zen3), Linux, GCC 13.3, on top of the vectorized mixer (#3739). Metric = server-process task-clock (all threads) over a fixed 40 s window of steady load; 6 interleaved runs per build; syscall counts from perf tracepoints.
| build | server CPU / 40 s | as CPUs | send syscalls / 40 s |
|---|---|---|---|
| baseline (#3739) | 12.50 s ± 0.20 | 0.313 | 152,986 sendto, 0 sendmmsg |
+ sendmmsg batch |
11.12 s ± 0.11 | 0.278 | 2,980 sendto, 15,000 sendmmsg |
- Server CPU −11% at 20 clients (1.38 s less CPU per 40 s of load). The two arms do not overlap: min baseline > max batch, gap ≈ 7× the baseline sd.
- Audio send syscalls 150k → 15k (10× fewer) — exactly one
sendmmsgper 2.67 ms frame. The ~3k residualsendtoare non-audio protocol/ping sends, left per-packet on purpose. - Load identical (20 connected every run); audio is byte-identical (the same packet bytes go to the same destinations, only the syscall boundary moves to end-of-frame).
A follow-up profile of the batched build confirms the mechanism: the send path drops to ~6% of server CPU (FlushSendBatch → a single __sys_sendmmsg per frame), and Opus encode/decode becomes the dominant remaining cost.
The saving grows with client count — the collapse factor is ~N, so a 50-client server sees a bigger cut than the 10× here. This is a larger lever than the mixer work because it scales with the thing that hurts big servers.
Scope / caveats
sendmmsgis Linux-specific. macOS and Windows keep the current per-packet path via a compile guard — no behaviour change there. The Linux server (the common self-hosted / public-server case) is what benefits.- Touches
CSocket(batch buffer + flush), the one send call site inCChannel::PrepAndSendPacket, and one flush call inCServer::OnTimer. No audio math changes, so bit-exactness is not a concern; the risk surface is socket handling (partial sends,EINTR, per-message errors in themmsghdrarray — the prototype handles the first two and drops the tail of a frame on a hard per-message error, which UDP / the jitter buffer already tolerate). - Receive batching with
recvmmsgis a possible follow-up, but the current receive loop already drains the socket toEAGAINper wakeup, so the send side is the clear first win. Left out of this change to keep it small.
I have the patch and the perf harness; happy to open a PR against this, or iterate on the design here first.
Activity
ann0see commentedon Jul 21, 2026
Ok. The question is if batching adds latency. I'd assume that this is very likely but would of course need some more in depth measurements then. Maybe a toggle like multithreding. But please do NOT open any PRs. We need to get the design right first and then decide on possible approaches.
ann0see commentedon Jul 21, 2026
Also mixing is O(n*n) while sending seems to be O(n) so asymptotically mixing is worse - I suppose. That syscals are slow isn't too surprising though as they need a context switch
ann0see commentedon Jul 21, 2026
Anyway this means that a performance review of the new socket code is useful also.
mcfnord commentedon Jul 21, 2026
Scaling measurement, since "grows with client count" deserved data rather than assertion. Same harness (interleaved A/B, server pinned to 1 core,
perf task-clockover a fixed 40 s window of steady load), sweeping the client count:Both axes grow with N. The batched build issues a constant ~15,000 send syscalls — one
sendmmsgper 2.67 ms frame — no matter how many clients, while the per-packet baseline grows linearly (76k → 306k). So the CPU saving nearly doubles from 12% at 10 clients to 21% at 40, and it's still climbing: the server core is only ~63% utilized at N=40, so this isn't a saturation artifact. A 64-client server would fold ~32sendtoper frame into one call.So the win is largest exactly where it matters most — the busy public servers. Load was identical per pair (connected count matched every run) and audio is byte-identical; only the send syscall boundary moves to end-of-frame.
mcfnord commentedon Jul 22, 2026
Three answers to the points above — latency, receive batching, and the O(n²) question.
Latency (the main concern). You're right it's non-zero, so I measured it. The batch holds each frame's packets from when they're produced until one flush at the end of
OnTimer, so I instrumented exactly that window (start of the per-client mix/encode loop → flush), steady state, same 1-core harness:That window is the worst case, and only for the first-encoded client — its packet waits the whole window, while the last-encoded client waits ~0, so the average added latency across clients is about half the window (~0.2 ms at 20, ~0.34 ms at 40). And it's bounded by the frame period by construction:
OnTimerhas to finish inside 2.67 ms or the server falls behind, so a packet can never be held past its own frame — no reordering, no skipped frame. Even the 40-client tail (1.8 ms) stays under 2.67 ms. Against real client RTT (tens of ms) and the jitter buffer (20–40 ms+), it's in the noise.A runtime toggle like
--multithreadingis easy to add if you'd still prefer opt-in — happy to gate it that way for a conservative default.Receive batching — I tried
recvmmsgtoo, and it's not worth it. Interleaved A/B, both builds already batching sends, so the delta is purely the receive change:Receive syscalls dropped ~3× (213k → 71k at N=20), but datagrams arrive spread out in time, so each
recvmmsgonly pulls ~3 at a time and the per-call overhead (rebuilding the message array + a copy into the shared receive buffer) eats the saving. The existing drain-to-EAGAINloop is already cheap. So I'd keep the change tosendmmsgonly — smaller surface, and that's where the win actually is.O(n²) mixing vs O(n) sending. Agreed, asymptotically mixing is worse. But in the range real servers run — profile at 20 clients, 1 core — mixing self-time is ~3.4%, while the send path was the single largest bucket before batching. So the crossover is well above today's loads, which argues for doing both (the mixer refactor in #3662 and the send batching), not either/or. The send batch is just the bigger lever at current client counts and it's a small, contained change.
Holding on the PR per your note — happy to keep this at the design level. Anything you'd want measured next?
ann0see commentedon Jul 22, 2026
From a strict latency standpoint every increase in latency would be unacceptable. So not sure if batching is a way to go then.
pljones commentedon Jul 22, 2026
Yeah - we have to deal with 150 client servers that might not be multi-threaded. If this would push them over the frame period, it's broken.
By the look of it, it's never more responsive and the CPU saving - in absolute terms rather than percentage - isn't particularly significant. 60 clients goes from 0.634 of a core to 0.499 of a core. Doubling to 120 and you're maxing out that core either way. (I guess the 150 client single threaded server's a write off, then...)