diff --git a/CLAUDE.md b/CLAUDE.md index d711adf..b7176c7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -245,3 +245,73 @@ Full reload is only required for changes to the page shell itself (DOM structure, button wiring, CSS, the entry/orientation flow before joining a space). For everything that lives inside an existing PC, prefer `leave` + `enter`. + +### Audio jitter buffering — userland AudioWorklet, not browser hints + +**The browser-native jitter-buffer controls cannot be trusted for music.** +Verified 2026-06-04 with side-by-side telemetry on a Firefox Android phone, +same PeerConnection, three receivers, identical 4s target: + +- `RTCRtpReceiver.playoutDelayHint` is spec'd as a hint — "the user agent MAY + use this." Browsers do whatever they want. +- `RTCRtpReceiver.jitterBufferTarget` is spec'd as a hard target. Honored for + voice-rate Opus and for video. **Ignored for high-bitrate stereo Opus + (256 kbps music) on Firefox Android.** The native music-stream code path + inside libwebrtc isn't wired to the new API there. + +Result: a phone listener with the spec'd 4s target had a 0.21s buffer on the +music stream. Any host-side stall (X11 window wiggle, GC pause, encoder +spike) was instantly audible. + +**The reliable cushion is a userland `AudioWorklet`.** See +`JitterBufferProcessor` (inline Blob URL) in `web/zebra-spaces.html`. The +worklet sits between `MediaStreamAudioSourceNode` and `GainNode`, queues +incoming 128-sample blocks, holds emission until `targetSamples` are +buffered, then emits with constant delay. The buffer cushions the music +regardless of what the native receiver does. + +Rules that took blood to find: + +1. **Sticky-started is non-negotiable.** Once the buffer fills and `started` + becomes true, do NOT set `started=false` on a single empty-queue tick. + That tears down playback and forces a full re-buffer (~4s of silence) on + every 2.67ms upstream micro-stall — sounds like constant chopping. + Tolerate ~267ms of consecutive empty blocks (`emptyStreak >= + rearmThresholdBlocks=100`) before re-arming; emit silence in the interim. + +2. **Role-aware buffer depth.** Listener gets 4s (lean-back, latency doesn't + matter, ride out wiggle-stalls). Speaker / cohost / host get 0.5s (small + enough for conversation, big enough to smooth ordinary jitter). Mesh + peers always 0.5s. `playoutDelayForRole(role)` and `SPEAKER_PLAYOUT_DELAY_SEC` + in `web/zebra-spaces.html`. + +3. **Worklet retarget on role change**, don't rebuild. Post + `{cmd:'retarget', targetSeconds}` to the worklet's port — recomputes + targetSamples/maxSamples and shrinks the queue if smaller. Audio path + stays continuous; only the buffer depth adjusts. `retargetAllReceivers(role)` + walks every live receiver and applies the new target. + +4. **UI gating on buffer-ready.** Worklet posts `{cmd:'started'}` on first + fill. Listener status text sits in "connecting — buffering 4s audio…" + until the first started message lands, then flips to "connected as + listener." Without this, users see "connected" but hear nothing for ~4s + and assume the app is broken. + +5. **Bound the queue at 1.5× target** to absorb clock drift without growing + unbounded. Drop oldest on overflow. + +6. **Set hints AND target anyway** at `ev.receiver.playoutDelayHint = ...` + and `ev.receiver.jitterBufferTarget = ... * 1000` — they're free, they + work where the browser implements them (video, voice mic), and they + layer cleanly with the userland worklet downstream. + +7. **Audio gets `priority='high'` at the sender.** Video gets `'low'`. Stops + a screen-share keyframe burst (e.g. X11 wiggle dirty regions) from + queuing audio packets behind it. `setSenderBitrate` / `setSenderMaxBitrate` + already apply this. + +**HTTP DJ pull (`/stream` on the SFU)** exists as a fallback for listeners +who don't have AudioWorklet support — natural deep buffer on the +`