Fox 2026-06-04: "lips on the video share are not synced with the audio."
Source of the desync:
- Audio total delay = native_jbuf + worklet_buffered (because the
AudioWorklet adds its 4s cushion ON TOP of whatever the native
receiver does)
- Video total delay = native_jbuf only (no worklet downstream)
- Net: video leads audio by up to ~4s on music listeners; by less on
voice (where native honors the hint, ~3s); audible mismatch in either.
Algorithm: per-publisher dynamic match. Every ~683ms the worklet
posts {cmd:'buffered', seconds: this.buffered/sampleRate}. JS-side
handler refreshes lip-sync for that uuid:
1. Resolve uuid → publisher pubHex
2. Read audio receiver's getStats jbuf (native part)
3. audioTotal = native_jbuf + worklet_buffered
4. Push to per-publisher history (size 5)
5. Compute median (FEC-style: 3-of-5 must agree before lock-in)
6. If |median - lastApplied| > 0.05s, set every video receiver for
that publisher: playoutDelayHint = median, jitterBufferTarget = median*1000
Hamming-spirit on a control signal per fox's request: median-of-5
rejects single-sample outliers from network jitter or getStats noise.
50ms hysteresis below perception threshold so the video target
doesn't whip on tiny shifts.
Registers:
- registerLipSyncAudio(pubHex, uuid, receiver) — at mic-kind ontrack
- registerLipSyncVideo(pubHex, kind, receiver) — at screen/camera/game ontrack
- Map: pubHex → {audioUuid, audioReceiver, nativeJbufSec, videoReceivers, history, lastApplied}
The role-aware base delay (playoutDelayForRole) is still applied at
attach as a sane initial value; lip-sync then refines per-publisher
within a few seconds.
Russell Ballestrini
8d18216854