Fox 2026-06-04: "the cohost on fedora chrome has the wiggle issue when host
is messing with tabs is there any way to sync the other stream and recover
the glitches assuming it is still buffering where the low-latency version
is glitched, this means we need to delay more than what we are or some
other trick mixed in halp..." (plus: "the video would need to be slid
depending on the mode to keep it in sync. complicated but possible.")
Before: mesh audio bypassed the worklet — went straight to an <audio>
element with the native receiver's ~50ms buffer. The SFU worklet path
had 0.5s of cushion that absorbed 200ms host stalls; mesh did not, so
the same wiggle that was inaudible on SFU was clicky on mesh.
After: ONE worklet per remote uuid, source swappable in-place via the
new setWorkletStream(uuid, newStream). Both the SFU stream (cached in
sfuStreamsByPubHex) and the mesh stream contain the same publisher's
content at slightly different network delays — so disconnecting the
old source and connecting a new one to the same worklet is seamless
(the queue's 0.5s of already-buffered samples covers the transition
while the new source fills it).
Wiring:
- mesh ontrack: setWorkletStream(uuid, meshStream). Fallback to a
fresh attachAudioStreamViaWorklet if no worklet existed (rare —
only when AudioContext failed at SFU attach).
- mesh connectionState='failed': setWorkletStream(uuid, sfuCachedStream)
+ registerLipSyncAudio(pubHex, uuid, sfuReceiver). attachCachedSfuStreamFor
remains as the no-worklet fallback.
- HTTP /stream toggle still uses rampWorkletGain (separate <audio>
element path).
Lip-sync receiver rebinding ("video slid depending on the mode"):
- New sfuAudioReceivers map caches the SFU receiver per publisher.
- handleRemoteSfuTrack mic path: registerLipSyncAudio with SFU receiver
AND store it in sfuAudioReceivers cache.
- mesh ontrack: registerLipSyncAudio with the MESH receiver — the
next worklet 'buffered' message will retarget video to mesh's jbuf
+ worklet (≈ 0.55s) instead of SFU's (≈ 1s).
- mesh fail: registerLipSyncAudio back to the cached SFU receiver
→ video re-targets again.
Removes the previous gain-ramp hack for SFU↔mesh transitions —
single worklet means no parallel paths, no need to crossfade. The
gain-ramp is still used for HTTP toggle (where there genuinely are
two paths: worklet + HTTP <audio>).
CPU cost: same as before — the SFU sub PC still decodes audio for
every speaker (we just route the decoded stream to the worklet or
not). Net change is "the mesh <audio> element is gone" — small
saving.
Russell Ballestrini
c3ff58ca39