From 1f6fde2a3e3ed19af761cd65e4b3abdf70655d83 Mon Sep 17 00:00:00 2001 From: Russell Ballestrini Date: Thu, 4 Jun 2026 16:59:37 -0400 Subject: [PATCH] =?UTF-8?q?CLAUDE.md:=20audio=20jitter=20buffering=20secti?= =?UTF-8?q?on=20=E2=80=94=20userland=20AudioWorklet,=20not=20browser=20hin?= =?UTF-8?q?ts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the 2026-06-04 lesson stack so future readers don't repeat the "playoutDelayHint=4 should cushion the listener" mistake. Key points documented: - jitterBufferTarget ignored for high-bitrate stereo Opus on FF Android (verified side-by-side: voice 1.8s avg, video 4s, music 0.21s, same target) - userland AudioWorklet is the reliable cushion - sticky-started re-arm — emit silence on brief drains, only re-fill after ~267ms sustained silence; otherwise every 2.67ms hiccup tears down playback - role-aware buffer depth: listener 4s, others 0.5s, mesh always 0.5s - worklet retarget on role-change (postMessage, not rebuild) - UI gating: "connecting — buffering 4s" until first started message - audio priority='high' at sender keeps mic ahead of video keyframe bursts - HTTP /stream pull is the fallback path (recently un-deadlocked) --- CLAUDE.md | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index d711adf..b7176c7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -245,3 +245,73 @@ Full reload is only required for changes to the page shell itself (DOM structure, button wiring, CSS, the entry/orientation flow before joining a space). For everything that lives inside an existing PC, prefer `leave` + `enter`. + +### Audio jitter buffering — userland AudioWorklet, not browser hints + +**The browser-native jitter-buffer controls cannot be trusted for music.** +Verified 2026-06-04 with side-by-side telemetry on a Firefox Android phone, +same PeerConnection, three receivers, identical 4s target: + +- `RTCRtpReceiver.playoutDelayHint` is spec'd as a hint — "the user agent MAY + use this." Browsers do whatever they want. +- `RTCRtpReceiver.jitterBufferTarget` is spec'd as a hard target. Honored for + voice-rate Opus and for video. **Ignored for high-bitrate stereo Opus + (256 kbps music) on Firefox Android.** The native music-stream code path + inside libwebrtc isn't wired to the new API there. + +Result: a phone listener with the spec'd 4s target had a 0.21s buffer on the +music stream. Any host-side stall (X11 window wiggle, GC pause, encoder +spike) was instantly audible. + +**The reliable cushion is a userland `AudioWorklet`.** See +`JitterBufferProcessor` (inline Blob URL) in `web/zebra-spaces.html`. The +worklet sits between `MediaStreamAudioSourceNode` and `GainNode`, queues +incoming 128-sample blocks, holds emission until `targetSamples` are +buffered, then emits with constant delay. The buffer cushions the music +regardless of what the native receiver does. + +Rules that took blood to find: + +1. **Sticky-started is non-negotiable.** Once the buffer fills and `started` + becomes true, do NOT set `started=false` on a single empty-queue tick. + That tears down playback and forces a full re-buffer (~4s of silence) on + every 2.67ms upstream micro-stall — sounds like constant chopping. + Tolerate ~267ms of consecutive empty blocks (`emptyStreak >= + rearmThresholdBlocks=100`) before re-arming; emit silence in the interim. + +2. **Role-aware buffer depth.** Listener gets 4s (lean-back, latency doesn't + matter, ride out wiggle-stalls). Speaker / cohost / host get 0.5s (small + enough for conversation, big enough to smooth ordinary jitter). Mesh + peers always 0.5s. `playoutDelayForRole(role)` and `SPEAKER_PLAYOUT_DELAY_SEC` + in `web/zebra-spaces.html`. + +3. **Worklet retarget on role change**, don't rebuild. Post + `{cmd:'retarget', targetSeconds}` to the worklet's port — recomputes + targetSamples/maxSamples and shrinks the queue if smaller. Audio path + stays continuous; only the buffer depth adjusts. `retargetAllReceivers(role)` + walks every live receiver and applies the new target. + +4. **UI gating on buffer-ready.** Worklet posts `{cmd:'started'}` on first + fill. Listener status text sits in "connecting — buffering 4s audio…" + until the first started message lands, then flips to "connected as + listener." Without this, users see "connected" but hear nothing for ~4s + and assume the app is broken. + +5. **Bound the queue at 1.5× target** to absorb clock drift without growing + unbounded. Drop oldest on overflow. + +6. **Set hints AND target anyway** at `ev.receiver.playoutDelayHint = ...` + and `ev.receiver.jitterBufferTarget = ... * 1000` — they're free, they + work where the browser implements them (video, voice mic), and they + layer cleanly with the userland worklet downstream. + +7. **Audio gets `priority='high'` at the sender.** Video gets `'low'`. Stops + a screen-share keyframe burst (e.g. X11 wiggle dirty regions) from + queuing audio packets behind it. `setSenderBitrate` / `setSenderMaxBitrate` + already apply this. + +**HTTP DJ pull (`/stream` on the SFU)** exists as a fallback for listeners +who don't have AudioWorklet support — natural deep buffer on the +`