Fox 2026-06-06 telemetry from blanka-chrome speaker session:
· role=speaker sub=connected/connected pub=connected/connected
mesh=2 ... aud.recv pkt=308→556→800 level=0.000
· jitter-buffer started uuid=2d4a target=0.5s
· audio via AudioContext 2d4a target=0.5s ctxState=running
RTP arriving at 50pps (active speaker), worklet started, ctxState
running — every audio invariant green except level=0 and "no audio
from any speakers".
Root cause: the speaker-output picker in the UI only called setSinkId
on <audio> elements. The worklet path (every listener AND every
speaker monitor since c3ff58c routed mesh through the worklet too)
runs through audioCtx.destination, which always emits to the SYSTEM
DEFAULT sink. Users who picked a non-default speaker in the dropdown
heard nothing on their chosen device because the worklet bypassed
the sink-routing entirely.
Firefox where blanka started hearing music again works by accident:
its default output happens to be the speaker fox wants. Chrome's
default is something else (PulseAudio's per-app routing, presumably).
The Jun 5 cascade fixes + Jun 6 flushSfuStreams prefix-match fixed
the FSM; this fixes the actual sink the FSM was emitting to.
Fix: applySinkToAudioCtx() — best-effort audioCtx.setSinkId(deviceId)
mirroring the existing applySinkTo(<audio>) helper. Called on
audioCtx creation (primeAudioOnGesture and attachAudioStreamViaWorklet)
and in applySinkToAll fan-out when the user changes the dropdown.
Browser support: Chrome ≥110, Firefox ≥116. Older versions silently
fall through to system default — same as today.
Diagnostic: log line now includes sink=... so the next telemetry
session shows which sink is routed. 'n/a' when the API doesn't
exist; 'default' when sinkId is the empty string; otherwise the
device id.
Tests:
- audioCtx.setSinkId is wired and callable in the sandbox
- attach does not throw when setSinkId is missing (older browser
fallback contract)
Fox 2026-06-06 on listener-to-speaker promotion: "the music slows
down I think that is the 6% algo which is janky. maybe it's better
to speed up in that case? if we skip ahead from whatever listener
is at to speaker speed, we need to make sure the video skips ahead
the same amount or rate to keep the lips synced".
Two coupled changes in the audio path:
1. JitterBufferProcessor retarget handler — on a SHRINKING retarget
(e.g. 4s → 0.5s) drop the queue down to targetSamples directly
instead of maxSamples (1.5×target). The 0.25s overhang the old
code left behind triggered a 6%-cap stretchFactor adjustment that
played at 1.064× for ~4 seconds — the residual phase fox heard
as "janky". Now: instant skip to the new target, then normal
playback. The grow-direction path is unchanged.
Worklet also reports the dropped sample count back to JS via
{cmd:'dropped', samples}.
2. installJitterBuffer.jbuf.port.onmessage — on 'dropped' from the
worklet, walk the publisher's lipSync.videoReceivers and set
jitterBufferTarget = 0 + playoutDelayHint = 0 on every one. The
browser drops video frames aggressively to converge on the new
target. Without this, audio jumps 3.25s forward instantly and
video drains gradually = broken lip-sync for the duration of the
native video jbuf's drain. Clearing ls.lastApplied lets the next
refreshLipSyncForUuid actually re-apply the role-appropriate
target instead of stopping at the threshold check.
Tests:
test/jitter-buffer-worklet.test.js (NEW, 8 assertions) — extracts
the inline JITTER_BUFFER_WORKLET_CODE template literal and runs
the processor in a Node sandbox with stubbed sampleRate +
registerProcessor + AudioWorkletProcessor. Pins:
- 'started' fires once on first fill
- 'buffered' reports depth
- retarget SHRINK drops to targetSamples (not maxSamples) and
posts 'dropped' with sample count
- retarget SHRINK does NOT leave 1.5×target overhang (the
residual that caused the janky 6% phase)
- retarget GROW does NOT drop
- retarget to same target is no-op
- bogus targetSeconds (NaN/0/negative) ignored
- lock_rate pins stretchFactor
test/listener-audio-attach.test.js (extended, +8 assertions):
- worklet 'dropped' handler zeros every paired video receiver
target (defensive against missing lipSync entry; correct when
seeded)
- speaker mesh+SFU collision tests (single-chain invariant):
mesh state=connected → SFU skip
mesh state=failed → SFU takes over
SFU first then mesh → in-place swap, no rebuild
mesh first then SFU(connected) → SFU skip, one chain
rapid mesh re-ontracks → still one gain feeds destination
SFU cache stays primed even when mesh wins the race
Makefile: new test-jitter-worklet target, added to test-all chain.
Fox 2026-06-06: "closed firefox on both phones [they] are both still
in the list" — listener-roster entries persisting indefinitely after
mobile Firefox close.
Two facts collided:
1. SERVER (proxy.unturf.com main.go aliveJanitorTick ~line 1872):
listeners are fully exempt from the heartbeat-stall reaper.
Justified fox 2026-06-04 because the page's {type:"alive"}
timer throttles hard on backgrounded mobile tabs (1Hz on
Android, paused on iOS power-save) — without the exemption,
mobile listeners lost their seat every time they tab-switched.
2. CLIENT (this file, sendByeIfRealClose): pagehide with
event.persisted=true means the page is going into bfcache
(mobile app-switch / tab-close-to-bfcache / lock screen),
so we SKIPPED the 'bye' message to keep PCs warm for resume.
Justified fox 2026-06-03 because "the phone leaving and
coming back cannot hear the music" — bye-driven SFU eviction
killed the speaker's publish + subscribe PCs.
Net: a mobile Firefox close fires pagehide(persisted=true) →
no 'bye' → server has only readTimeout (120s) + hiccup grace
(8s) to detect the dead socket → ~128s of phantom listener
seat in every other client's roster.
The bfcache justification on the CLIENT side was always
specific to speakers (they have PCs to protect). LISTENERS:
- have no publisher PC
- finalizeLeave at main.go:1089 explicitly exempts them from
evictFromSFU (their subscriber PC stays alive through the
bye)
- re-handshake fresh on pageshow via POST /subscribe (same
path as a cold join)
So for listeners, sending 'bye' on persisted=true is
roster-only cleanup: peer-left broadcast, members.delete(uuid)
on every other client, brief disappearance from the room. On
pageshow they re-handshake and reappear — same UX as a cold
rejoin, which already works.
Fix: split the bfcache rule by role. Send 'bye' on pagehide
even when persisted=true if myRole === 'listener'. Speakers
keep the original bfcache skip exactly.
Server-side backstop (60s listener-specific TTL replacing the
full exemption) ships as a separate commit in proxy.unturf.com
so 'bye' losses (carrier NAT eating the TCP shutdown, abrupt
process kill, custom Firefox close paths) still get reaped
within the minute.
Pinned by test/sendbye-fsm.test.js — extracts
sendByeIfRealClose from this page and drives 9 scenarios
covering each (role, persisted) combination plus defensive
edges (no event, ws not open, post-demote listener state).
Wired into test-all via test-sendbye target.
Fox 2026-06-06: "fedora chrome is flawless besides not able to hear
any mics it was working a few days back and nothing was changed on
the system, only thing we changed was our zebra codes."
The race: on a listener joining a room with existing speakers, the
SFU sub PC ontrack can fire BEFORE the signal-server peer-joined
event populates `members`. handleRemoteSfuTrack already does prefix
resolution at line ~4640:
let pubHex = pubHex16;
for (const [, mm] of members){
if (fh.startsWith(pubHex16)){ pubHex = fh; break; }
}
When the roster is empty, the loop finds nothing, pubHex stays the
16-char streamID prefix, and sfuStreamsByPubHex.set(pubHex, stream)
caches under that short key. peer-joined arrives later,
flushSfuStreams runs to attach what was cached — but its inner match
was strict ===:
if (mm.pubkey && hex(unb64(mm.pubkey)) === pubHex){
mm.pubkey decodes to the FULL 64-char hex; pubHex from the cache is
the 16-char prefix; === never matches; listener stays permanently
silent for every speaker who was already in the room.
Pre-cascade this defect was masked: the old guard `!remoteAudio.has(uuid)`
was always true for worklet listeners (remoteAudio is the <audio>
fallback path only), so flushSfuStreams re-attached every cached
stream on every peer-joined — the eventual second ontrack from a
later renegotiation would land with members populated, cache key
became the full pubhex, and === matched. The 2e74b92 fix replaced
the always-true guard with `!listenerAudioNodes.has(uuid)`, which
correctly skipped re-attach but also exposed the strict-equality
matcher in the cold path.
Fix: switch flushSfuStreams' inner match from `=== pubHex` to
`fh.startsWith(pubHex)`. Symmetric with handleRemoteSfuTrack's own
prefix resolution. Works for both cases:
- cache key is full 64-char pubhex → startsWith with a full string
requires equality, so behavior is unchanged when ontrack arrived
after peer-joined (the common case).
- cache key is 16-char prefix → startsWith matches the first 16
chars of any member's full pubhex. 64 bits of prefix entropy =
astronomical collision probability.
Pinned by 29 new assertions in test/listener-audio-attach.test.js,
extracted from the live page so they cannot drift:
- 22 cover the attach FSM (chain reachability, dedup, in-place
swap, jbuf race, idempotent re-attach).
- 7 cover handleRemoteSfuTrack including the failing scenario:
"ontrack ARRIVES BEFORE peer-joined (member roster empty) →
cached + audible after flush" — fails pre-fix, passes post-fix.
Makefile gets test-listener-audio + adds it to test-all.
Fox 2026-06-05: "the self speech to text is not creating new lines
for when there is new sentence, the listeners are doing a better
job."
Both self-capture and remote-speaker capture go through the same
handleWhisperChunk → appendTranscriptLine path. Each Whisper chunk
is 5 seconds. The listener-side capture often catches a peer mid-
pause, so each 5s chunk ends up holding one short sentence — one
line in the log. The host on a close mic talks continuously for
the full 5s window, so Whisper returns "Hello there. How are you.
Good to see you." as a single string → one mashed line.
Fix at the dispatch point (not the capture point): split the
Whisper output on sentence-terminating punctuation
(period/exclaim/question followed by whitespace) and call
appendTranscriptLine once per sentence. Each sentence gets the
same hallucination + min-length filtering as the original whole
result.
Applies to BOTH self capture and remote capture since they share
the same handler — listener-side transcripts also get cleaner when
two sentences happen to fit in one chunk.
Fox 2026-06-05: "still two feeds one early (one on time) the early
one must be the mesh for the speakers arriving to the listener" /
"fxhp-android-firefox is also hearing two, one perfectly synced,
other is not."
It was NOT mesh (listeners have no mesh PCs — telemetry confirmed
mesh=0). It was the <audio>-element fallback racing the worklet:
Sequence per peer-joined cycle:
1. flushSfuStreams handed a stale cached stream (no live audio
tracks after the publisher's prior session evicted) to
attachSfuTrack.
2. attachListenerStreamViaAudioContext rejected it via the new
"no live audio tracks" guard.
3. attachSfuTrack FELL THROUGH to the <audio>-element fallback,
created an <audio>, set srcObject = stale stream, called
play(), and stored it in remoteAudio[uuid].
4. Later, live ontrack delivered the real audio stream.
attachListenerStreamViaAudioContext succeeded, the worklet
started, and attachSfuTrack returned early.
5. The <audio> element from step 3 was NEVER torn down. The SFU's
still-alive stale transceiver kept forwarding RTP into it.
6. Listener heard the same publisher twice — <audio> at ~0.5s
native jbuf ("early"), worklet at 4s ("on time").
Fix: when the worklet/AudioContext path succeeds, look up the
stale <audio> element in remoteAudio[uuid] and tear it down
(srcObject=null, pause, remove, delete from map). Single playback
path per uuid going forward. Applies to both listener and
speaker/cohost/host branches.
Fox 2026-06-05: "fxhp-android-firefox is also hearing two, one
perfectly synced, other is not" + "host hearing two."
Diagnosis from signal-server telemetry: every peer-joined cycle
produced 3 "audio attach uuid=..." events for the SAME uuid within
1 second. Each call tore down the existing src+jbuf+gain chain and
built a new one. During the brief window between disconnect() and
the next chain's destination wiring, both old and new gains were
audible — and when the rebuild raced multiple paths
(flushSfuStreams, sub PC ontrack, mesh ontrack, mid-spotlight
supplant), two complete chains stayed live in parallel. The
listener heard them as one synced + one offset.
Root cause is the rebuild itself, not the count: the SFU forwards
stale audio transceivers across publisher rejoins, so every ontrack
delivers a NEW MediaStream object even when the publisher is the
same. Reference-equality "same stream" check from the prior commit
couldn't catch this.
Fix: when a chain already exists for the uuid, REUSE the existing
worklet+gain+destination wiring and swap only the MediaStreamSource
via setWorkletStream — the same in-place swap path mesh ontrack
already uses. Single chain per uuid for its entire lifetime. No
overlap window, no parallel chains, no audible double-audio
regardless of how many ontracks arrive.
The teardown+rebuild branch is kept as a fallback for the
setWorkletStream-fails case (no audioCtx, no jbuf yet, etc.) — the
common path now never tears down.
Fox 2026-06-05 telemetry on host: "host is hearing two" — peer-joined
u=64b0 (fxhp-phone rejoin with new uuid, same pubkey) fired THREE
attach calls within 1 second for the SAME uuid:
Call 1: pool=1 → fallback <audio> path, "tracks=0"
Call 2: pool=1 → AudioContext success
Call 3: pool=2 → AudioContext success (after mesh stream swap)
Three overlapping audio chains = "two streams" perceived.
Root causes:
1. flushSfuStreams used the cached sfuStreamsByPubHex entry, which still
pointed at the OLD phone session's stream (uuid=91ab had left 5s
earlier). Old stream's audio tracks were ended. attach received a
zero-live-track stream and fell through to <audio> fallback.
2. Multiple paths (peer-joined → flushSfuStreams, sub PC ontrack,
mid-spotlight supplant) each tried to attach the same uuid in the
same tick. No idempotency check, so each call tore down and rebuilt.
Two fixes:
A. attachAudioStreamViaWorklet: reject streams with zero live audio
tracks (skip silently); make same-uuid + same-stream calls a no-op.
Repeated calls during a peer-joined burst stop racing.
B. handleRemoteSfuTrack: listen for the cached track's 'ended' event
and drop the sfuStreamsByPubHex entry. Next flushSfuStreams won't
hand a dead stream to attach.
Combined: the publisher's rejoin cycle now produces exactly ONE attach
chain — the one fed by the live ontrack — and ignores stale cache hits
+ duplicate attach calls.
Fox 2026-06-05: telemetry showed fxhp-android-firefox re-attaching
EVERY speaker (host + blanka + ...) on every single peer-joined
event. Each re-attach tears down the existing chain and builds a
fresh src→jbuf→gain→destination — the teardown vs new-attach race
on the same uuid produced audible overlap = "two streams."
Root cause: flushSfuStreams's guard was `if (!remoteAudio.has(uuid))`
— but remoteAudio is the <audio>-element fallback map, only
populated when AudioContext fails. For listeners using the
AudioContext path (every listener today, including fxhp), that map
stays empty → guard always false → re-attach for every speaker on
every join.
Fix: also check listenerAudioNodes — the modern worklet path's
source of truth. Skip attach if EITHER map already has the uuid.
This is upstream of the dedup-by-pubkey defense from f9736c2 —
the dedup catches duplicate publishers across DIFFERENT uuids
(rejoin race), but couldn't catch SAME-uuid re-attaches racing
their own teardown. With the guard fixed, the only attach call
per uuid happens at first ontrack or attachCachedSfuStreamFor,
not on every flushSfuStreams sweep.
Telemetry (signal log): peer-joined u=91ab at 22:58:19 fired
"audio attach uuid=0c7c", "audio attach uuid=b291", "audio attach
uuid=91ab" all within the same second — three attaches when only
one (91ab) was new. Pattern repeated at 23:04:24, 23:08:08.
Fox 2026-06-05: "speaker on firefox chrome was hard restarted and
cannot hear host" / "host is working phone listeners hear host, speaker
does not see hosts mic open or waveform."
Root cause: desktop auto-rejoin reaches joinSpace() without a click,
so primeAudioOnGesture() never fires. Chrome (Fedora especially)
leaves audioCtx suspended until a real user gesture, and the
fire-and-forget audioCtx.resume() inside attachAudioStreamViaWorklet
silently fails outside one. Worklet reports "started" but the
destination renders nothing, the meter analyser pulls no samples →
no audio, no waveform, no mic-open indicator (the indicator IS the
waveform).
Pre-zero-click-auto-rejoin, the manual entry button always primed
audio; the regression arrived with that convenience. Now: install a
document-wide one-shot click/keydown/touchstart listener that calls
primeAudioOnGesture() on first interaction anywhere. Zero-click
auto-rejoin convenience preserved — any later page click engages
audio for the rest of the session.
Listener self-unregisters after firing so we don't repeatedly spawn
silent oscillators. Idempotent — primeAudioOnGesture's audioCtx
check makes re-runs safe.
Fox 2026-06-05: "blanka-chrome doesn't seem to be getting audio anymore
as a speaker."
Telemetry shows host muted and nobody else publishing — could be just
silence, not an actual audio-pipeline break. Adding a log line on every
attachAudioStreamViaWorklet call so we can see the sequence of attaches
on blanka's side and verify whether the dedup ever runs / detaches the
wrong entry.
Format: "audio attach uuid=XXXX target=Ns (current pool=N)"
If dedup is killing blanka's audio we'll see attach + dedup-detach in
the same window. If no dedup fires and audio still missing, the issue
is elsewhere.
Fox 2026-06-05: "fxhp-android-firefox is playing host audio twice."
Root cause: publisher rejoined with new session uuid before the old
session's peer-left fired. handleRemoteSfuTrack's member iteration
matched both uuids by pubkey and called attachSfuTrack(uuid, stream)
for BOTH. Two listenerAudioNodes entries created (keyed by uuid),
each with its own source → gain → destination chain. Both worklets
were processing the audio; both gains were connected to destination
→ host audio mixed twice.
Fix in attachAudioStreamViaWorklet: after the same-uuid cleanup,
walk listenerAudioNodes and detachListenerStream for ANY other
uuid whose member.pubkey matches the current uuid's pubkey. Same
publisher, different session id, the older entry is stale — kill
its worklet/gain so only one chain plays. Logged so we can see the
dedup firing.
Also: cleanup now also disconnects existing.capture / .captureLpf
(whisper-related nodes) which the previous cleanup missed —
unrelated to the echo but a memory/CPU leak across attach cycles.
Fox 2026-06-05: "the lips are not synced with video for listener" and
"when i am at the bottom and new transcriptions flow in, keep
scrolling as they arrive unless ive scrolled up."
Lip-sync flap diagnosis (per fxhp-phone telemetry):
lip-sync pub=25cc delay=2.87s
lip-sync pub=25cc delay=6.87s
lip-sync pub=25cc delay=2.87s (every ~683ms)
lip-sync pub=25cc delay=6.87s
Two values exactly 4s apart (4s = worklet target). One source is
LIVE (worklet buffer near target → 6.87s = 2.87 native + 4 worklet),
the other is a STALE worklet whose source died (buffer drained to
0 → 2.87s = 2.87 native + 0 worklet). Both post bufferedSeconds
every 683ms. With LIP_SYNC_HISTORY = 5 odd-length, alternating
samples produce alternating medians: [A,B,A,B,A] → A;
[B,A,B,A,B] → B. Video target whipsaws ±4s per tick.
Three fixes:
1. LIP_SYNC_HISTORY = 10 (was 5). Even-length window.
2. medianOf returns the AVERAGE of the two middle values for
even-length arrays. Alternating samples now produce a stable
median = (A+B)/2 — at least the value doesn't whip.
3. Stale-source filter in refreshLipSyncForUuid: if
node.bufferedSeconds < 0.5 AND nativeJbufSec > 1.0, skip this
refresh entirely. A worklet whose buffer is drained while the
native audio receiver still reports a healthy jitter buffer is
from an OLD subscription that's no longer carrying audio. Its
"0" reading would pull the median toward the dead value.
Only the LIVE worklet's refreshes update the history; the median
stabilizes on the actual audio total delay.
Autoscroll fix in appendTranscriptLine:
The previous logic computed "near bottom" AFTER appending the line.
scrollHeight grew the instant the line was in the DOM, so
(scrollTop + clientHeight) became < (scrollHeight - 40) for every
append — autoscroll never fired. Now we check BEFORE appending and
widen the tolerance to 120px (covers touch-scroll inertia residue).
Fox 2026-06-05: "the remote phones are transcribing more accurate than
the host version. figure out why and make host version better."
Two causes diagnosed:
1. Aliasing. The whisper-capture worklet decimates 48 kHz → 16 kHz by
taking every 3rd sample with NO anti-alias filter. Listener path
feeds Opus-decoded peer audio which is already band-limited
(~12 kHz max). Host path feeds RAW mic — all 48 kHz of it —
which means anything above 8 kHz folds back into the audible band
as garbage when decimated. Whisper sees noisier input on host.
2. Amplitude / dynamic range. Music-mode mic has NO AGC/NS/EC
(raw broadcast for music). Quiet passages are too low for
Whisper to confidently transcribe; loud passages can clip. The
listener path's audio has been through Opus encode/decode which
normalizes implicitly.
Both addressed via Web Audio nodes inserted between the
MediaStreamSource and the whisper-capture worklet:
Self-capture chain:
micStream → DynamicsCompressorNode → BiquadFilter (lowpass 7 kHz)
→ AudioWorkletNode (whisper-capture)
DynamicsCompressor: voice-friendly settings (threshold -30 dB,
knee 30, ratio 4:1, attack 3 ms, release 250 ms). Flattens
amplitude so Whisper sees a normalized waveform regardless of how
the user has the mic gain set or whether music mode is on.
BiquadFilter lowpass at 7 kHz (below Nyquist of 16 kHz = 8 kHz):
removes content that would alias when the worklet decimates.
Remote captures: just the lowpass (no compressor — Opus has already
normalized the peer audio adequately). Mostly a no-op on voice-mode
peers (Opus already cuts <12 kHz) but helps music-mode peers where
the encode preserves more high-frequency content.
stopCaptureForUuid / stopSelfCapture cleanly disconnect the new
nodes so re-toggle doesn't leak audio graph references.
Expected: host self-transcribe accuracy now comparable to (or better
than) the phone listener transcribes. Telemetry `emt=` should
increase per tick on host once the user is talking.
Fox 2026-06-05: "1:34:48 PM JS unhandled: can't access lexical
declaration 'whisperWorker' before initialization."
The button-binding restore block ran synchronously at script eval
time, BEFORE the rest of the file's let/const declarations had been
initialized. transcribeEnabled and TRANSCRIBE_KEY got hoisted earlier
in the previous fix, but the restore body itself reaches further
down — into whisperWorker, listenerAudioNodes, audioCtx, micStream,
and a chain of helper functions whose bodies access more `let`s. Any
of those is enough to throw.
Wrap the whole restore in setTimeout(fn, 0). The current synchronous
script finishes (all decls initialized), then the queued task runs.
Now ensureWhisperWorker, startCaptureForUuid, startSelfCapture etc.
all see fully-initialized state.
Also added a try/catch around the block so any remaining edge-case
error gets logged instead of bubbling to window.onerror.
Fox 2026-06-05 log: "self capture: worklet module load failed" at the
same instant a remote capture for the same publisher succeeded.
Root cause was a check in loadWhisperCaptureWorklet:
if (whisperCaptureWorkletReady || whisperCaptureWorkletLoading)
return Promise.resolve(whisperCaptureWorkletReady);
When TWO calls happen near-simultaneously (one from startSelfCapture,
one from startCaptureForUuid on toggle-on), the second caller sees
whisperCaptureWorkletLoading === true and gets back
Promise.resolve(false). Caller bails. By the time the actual load
finishes, the second caller already gave up.
Fix: cache the loading promise itself and return the same one to
every concurrent caller. They all await the same resolution; both
succeed when the module is ready. On error, the cached promise is
cleared so a later toggle can retry.
After this lands, self-capture should engage on every host that has
transcribe on. Telemetry's selfCap=1 will flip to 1.
Fox 2026-06-05: "self transcribe not working, also 1:28:59 PM JS
unhandled: can't access lexical declaration 'transcribeEnabled'
before initialization."
The restore block (button-binding site, ~line 1635) was reading
transcribeEnabled before its `let` declaration further down (~line
2225). Script execution order: TDZ violated, the whole restore +
worker pre-warm + capture-kick branch never ran. Subsequent code
continued (the error fired async on click) but the restore-time
initialization was skipped.
Fix: hoist the const TRANSCRIBE_KEY + let transcribeEnabled + the
localStorage load up to BEFORE the click-handler binding. Now the
restore block has a valid initialized value to read.
Plus diagnostic logging in startSelfCapture for the early-return
branches (no micStream / already running / worklet module load
failed) — so we can SEE why self capture isn't engaging if it
still isn't after this fix.
Self-transcribe issue should also be addressed by this TDZ fix:
the restore branch is where startSelfCapture would have been
called on page-load with state=on, but the TDZ throw skipped it.
After this lands, refreshes preserve transcribe state AND self
capture engages alongside remote captures.
Fox 2026-06-05: "transcribe states should be saved in localstorage to
survive on refresh" and "self transcribe does not seem to work."
Two fixes:
1. localStorage persistence:
- TRANSCRIBE_KEY = 'zspc:transcribe-on'
- Loaded into transcribeEnabled at script top
- setTranscribe(want) replaces the old toggleTranscribe body and
writes localStorage on every state change
- applyTranscribeUI() reusable for both flip and refresh-restore
- At click-handler binding time, if transcribeEnabled was already
true (restored from storage), apply UI + pre-warm worker + kick
captures for any speakers already attached
2. Self-capture fixed:
- startSelfCapture used to early-return if audioCtx was null
- But audioCtx is only created when the FIRST remote speaker's
audio attaches via attachAudioStreamViaWorklet. A host alone in
the room (no remote audio yet) had no audioCtx → self capture
silently failed.
- Now startSelfCapture creates audioCtx if absent and resumes if
suspended, same pattern as the remote-attach path. Host-alone
captures their own voice immediately.
Two related additions:
1. Self-transcription. The host's own voice wasn't being transcribed
because capture nodes only attached to listenerAudioNodes (remote
speakers' streams from SFU/mesh worklets). micStream — the local
capture for publishing — was never tapped. Added startSelfCapture
/ stopSelfCapture that wraps micStream in a parallel
whisper-capture worklet. Lines are tagged with myUUID so the local
user's handle shows in the transcript log.
Wired into:
- toggleTranscribe ON/OFF → starts/stops self capture alongside
remote captures
- getMic → starts self capture if transcribe is already on (covers
"user enabled transcribe before granting mic permission")
- applyMicMode / reacquireMic → stops + restarts self capture
against the new micStream so we don't keep transcribing a
stopped MediaStreamTrack
2. Whisper telemetry stats. Each 5s tick now appends:
xcr=on sent=N drp=N emt=N avgMs=N wrkr=1 caps=N selfCap=1
- xcr: on/off
- sent: chunks shipped to the worker this tick
- drp: chunks dropped by the inflight gate (worker saturated)
- emt: text lines emitted after silence/hallucination/dup filters
- avgMs: average inference latency per chunk (device-perf proxy)
- wrkr: 1 once the worker has loaded the model
- caps: # of active remote-capture worklets
- selfCap: 1 if local mic is being captured
Reset each tick so we see RATE, not cumulative.
Lets us compare devices across mesh from server-side signal log:
Snapdragon-8 should show avgMs ~150-300; mid-range phones 1500-
3000; desktop x86 50-150. drp > 0 means worker can't keep up with
the audio chunks arriving (multiple speakers talking at once).
Factored the chunk-handling pipeline into handleWhisperChunk(uuid,
chunk) so both remote and self captures share the same filter +
inflight-gate + telemetry path.
Fox 2026-06-05: "the web page for zebra spaces seems noticeably slower
after enabling transcribe… even the tones for entering and leaving are
showing up way way later even on the host side."
Root cause: ONNX Runtime via transformers.js was running on the main
thread, blocking JS for 1–3 seconds per chunk. Join/leave chimes,
button clicks, scroll, EVERY UI gesture queues behind it.
Two changes:
1. Move Whisper to a dedicated module Web Worker. The worker imports
transformers.js + loads the whisper-tiny.en pipeline ONCE; each
chunk is transferred (zero-copy) via postMessage, processed in
isolation from the UI thread, and the resulting text is posted
back. Main thread is free during inference now — UI stays
responsive. Worker is created on first toggle ON; same ~40MB
model download still happens, just off-thread.
2. Global "inflight gate" on the main thread side. Only one
transcribe request can be in flight at a time. If a new chunk
arrives while busy, DROP it (don't queue). Counter is logged
every ~30s so we can see worker saturation. With N speakers all
talking at once, dropping is correct — stale chunks from 10s ago
aren't worth transcribing.
Combined with the RMS silence gate in the capture worklet, the result
is: silent chunks never even reach the main thread, busy chunks are
processed sequentially by the worker, and the UI never blocks.
Phase 4 (deferred): switch to WebGPU backend for ONNX where supported
— roughly 2-5× faster than WASM on capable devices. transformers.js
v3 supports this with `{ device: 'webgpu' }` in pipeline opts.
Fox 2026-06-05: "fxhp and fxhp-chrome both are triggering clients to
transcode `you` on a new line over and over could you sort that out?
… this happens even when mic is closed."
whisper-tiny.en is notorious for hallucinating common stop-words on
silent / low-energy audio: "you", "thank you", "thanks for watching",
a lone period. Mic muted or peer dead-silent → the model still
produces a chunk of audio (silence frames from Opus / WebRTC) and
spits one of these phrases out. Two filters:
1. RMS-based silence gate IN the capture worklet (cheaper — no
pipeline invocation at all). Compute RMS of the 5s/16kHz chunk
before posting; skip if below 0.005 (~-46 dBFS, well below any
real speech). The vast majority of "mic-closed" hallucinations
stop here.
2. JS-side filter against a known-hallucination set. Catches the
rest (quiet-room ambient that passes the RMS gate). Normalizes
lowercase + strips trailing punctuation so "You." / "you " /
"YOU!" all collapse to "you" and match the set. Easy to extend
as new hallucinations are observed.
3. Per-uuid consecutive-duplicate suppression. Even non-hallucination
text sometimes re-emits the same short phrase across consecutive
chunks ("uh huh" "uh huh") — drop the dupe.
CPU win: silent chunks no longer go through ONNX inference (~50-200ms
per chunk on tiny.en).
Fox 2026-06-05: "listeners should still be synced it should just be
that the gesture to change feeds after they are all live should be
immediately to make that feed full sized... reverse your bs and do
what I asked."
Reverted the listener-video-no-delay shortcut. Listener video keeps
playoutDelayHint matched to audio (lip-sync preserved). The actual
fix for slow tile-switching: REUSE the existing thumb tile's <video>
element instead of building a fresh one when promoting to spotlight.
Before: setSpotlight built a brand-new <video> in #spotlight and
assigned the existing MediaStream as its srcObject. The fresh video
element had no decoder state — had to wait for the next keyframe
inside the receiver's 4s-deep buffer to show its first frame,
manifesting as up to 4s of "blank" on listener mobile.
After: setSpotlight DOM-moves the existing thumb tile (with its
already-playing <video> element) into #spotlight. The video element
NEVER stops — no keyframe wait, no re-buffer, no decode-init.
clearSpotlightDOM does the reverse: moves the spotlit tile back to
its thumbs container. Browser DOM-move on an HTMLMediaElement does
not reset playback state.
Supporting changes:
1. CSS: viewing-badge default-hidden; only visible in .tile-thumb.viewing
context (which no longer exists post-DOM-move since the tile IS
the spotlight, no separate thumb to mark).
2. CSS: .tile-thumb .fs-btn { display: none } — buildTile always
includes the fullscreen button now (so we don't have to add it
dynamically when moving thumb → spotlight). The button is hidden
when the tile is in thumb context.
3. buildTile: always builds fsBtn (used to be !isThumb only).
4. setSpotlight: same-target click = no-op (idempotent); previous
spotlight is moved back via clearSpotlightDOM before installing
the new one.
5. The mid-spotlight supplant path (publisher republishes the same
kind): the entry-video swap also IS the spotlight-video swap now
(same DOM node) — removed the redundant second swap, just mirror
the new entry.video reference into spotlight.video.
6. Game spotlight unchanged — game tiles are static iframes built
fresh each spotlight, not DOM-moved.
End result: lip-sync stays correct for every role (audio + video
share the same playoutDelayHint), tile switching is instant.
Fox 2026-06-05: "we should switch feeds immediately I don't want any algo
slowing that down."
The previous lip-sync algorithm set video receiver playoutDelayHint to
match audio's worklet delay (up to 4s on listeners). That made
spotlight-switching take up to 4s — the new tile's <video> element
had to wait for a keyframe to appear inside the 4s-deep receiver
buffer before showing the first frame.
Changes (listener role only):
- handleRemoteSfuTrack screen/camera/game branch: vDelay = 0 for
listener (browser-lowest). Speaker/cohost/host unchanged.
- refreshLipSyncForUuid: early-return for myRole === 'listener'.
No more retargeting video to match audio for listeners.
- retargetAllReceivers (called on role change): audio retargets
per role; VIDEO retargets to 0 for listener, role-aware for
others.
Listener video now plays at browser-default minimum delay. Tile
switches are instant — keyframe wait + native buffer pre-roll only,
typically <300ms.
Tradeoff accepted: listener mouth-leads-voice by approximately the
audio worklet's current cushion (1.3–4s adaptive). For consume-
content mode (lean-back listener) this is correct; switching shares
is far more important than per-syllable lip-sync.
Speakers/cohosts/hosts unaffected — their audio cushion is ~0.5s,
video matches it via lip-sync, mouths and voice align naturally.
Fox 2026-06-05: "do we access to a whisper speech to text?" → no, then
"yes implement this and please this is perfect, make it an off be
default toggle that is part of the client. individual speakers should
ues their names and show up like a log under the video area in middle."
Implementation:
1. whisper-capture AudioWorklet — separate from the jitter-buffer
worklet. Decimates the speaker's 48 kHz mono stream to 16 kHz
(Whisper's expected sample rate), batches 5-second chunks, ships
them to the main thread via transferable Float32Array on the
port. Disabled by default; { cmd: 'start' } / { cmd: 'stop' }
from JS gate the capture.
2. transformers.js + Xenova/whisper-tiny.en lazy-loaded from
jsdelivr CDN on first transcribe-toggle ON. ~40 MB one-time
download (cached by the browser); subsequent toggles are
instant. ONNX Runtime runs entirely client-side — no audio
leaves the listener's browser.
3. Per-uuid capture lifecycle. When transcribe is on, every speaker
in listenerAudioNodes gets a parallel AudioWorkletNode that taps
their source. Recognized text appends to a transcript log with
"HH:MM:SS name: text" lines. Names resolved via
members.get(uuid).handle. Empty / placeholder transcriptions
("." "[BLANK_AUDIO]") filtered out.
4. UI:
- Toggle button "transcribe (off/on)" in the controls column,
with explainer note about model size + privacy.
- #sec-transcript section in the timeline column directly under
the spotlight. Monospace font, 14rem max-height, scrollable,
auto-scrolls to bottom unless user is reading older lines.
- .hidden until first toggle ON; subsequent toggles show/hide.
5. New-speaker hook: attachAudioStreamViaWorklet checks
transcribeEnabled and auto-installs capture for late joiners.
6. detachListenerStream tears down node.capture along with the rest.
Bandwidth: zero (model and audio never leave the device).
CPU on listener: ~5-15% per speaker during the brief inference
window every 5s, idle otherwise. Tested mentally; needs real-world
verification.
Fox 2026-06-05: "since we have this now there is no reason to have the
stream button anymore on the ux."
The ○/◉ button next to each row was the self-listener entry point
(speaker flips themselves onto the buffered HTTP Ogg/Opus path when
WebRTC was glitchy). With the adaptive worklet — predictive drain
detector, self-calibrating floor, ±6% time-stretching — the same
glitch-recovery happens automatically without user intervention.
Removed:
- 'strm' grid column from .member layout (back to 4 columns)
- .stream-toggle CSS rules (button styling, dark-mode variants)
- The button render block in renderRoom (per-uuid streamEl creation)
- row.appendChild(streamEl) and its preceding comment
- The unwired dead toggleStreamFor(uuid, pubHex) helper function
Kept (still useful):
- streamMode Set + startStream() + stopStream() — used by the global
self-listener FSM
- selfListenerMode and its observer
- The HTTP /stream lip-sync override (httpLipSyncOverride)
- Double Dragon's manual-on/manual-off states + ddNoteManualToggle
(no UI surface but the plumbing stays for future re-introduction
if needed)
Layout is back to badge|handle/pub/acts|mic|meter — one fewer column,
slightly more breathing room per row.
Fox 2026-06-05: "both and see if we could lower to 1.3 secs as floor
for listeners on the high quality stream."
Two additions to the Double Dragon controller:
1. Predictive GROW (lead, not lag).
The existing 5s-tick path watches loss/jitter — strictly a lagging
indicator. We now ALSO watch the worklet's bufferedSeconds
reports (every ~683ms) for rapid drain. If the buffer drops by more
than 25% of its current target over a 1.5s window, GROW
immediately — don't wait for loss to appear in the next 5s
sample. Catches wiggle ~2s after it starts vs 5-10s on the
lagging path.
2. Self-calibrating floor.
Each publisher tracks its own maxWiggleDurationSec for the
session. Floor for SHRINK is max(role-base, maxWiggle × 1.5). A
stable publisher's listener can shrink to 1.3s. After observing a
1s wiggle the floor lifts to 1.5s; after a 4s wiggle, 6s (clamped
to MAX). Resets when user leaves + rejoins.
Constants:
- DD_BASE_LISTENER_TARGET_SEC = 1.3 (was 2.0)
- DD_WIGGLE_FLOOR_MULTIPLIER = 1.5
- DD_DRAIN_THRESH_FRAC = 0.25
- DD_DRAIN_WINDOW_SEC = 1.5
Telemetry log lines:
- "double-dragon pub=XXXX PREDICTIVE GROW → 4s (buf drained Nms in N.Ns)"
- "double-dragon pub=XXXX wiggle ended dur=N.Ns — floor now N.Ns"
- "double-dragon pub=XXXX SHRINK target → N.Ns (clean N samples, role=X, max-wiggle=N.Ns)"
This is the "lead + lag, AND self-tuned floor" loop. The 1.3s listener
floor only applies to clean-history sessions. A wiggle-prone publisher
will see the floor stay elevated automatically.
Gentler pitch shift during buffer adaptation at the cost of slightly
longer ramp time:
- Shrink 4s→2s: ~25s → ~31s
- Grow 0.5s→4s: ~47s → ~61s
Pitch shift drops from ≤1 semitone to ≤0.7 semitone — less perceptible
on music. Fox 2026-06-05 dial.
Fox 2026-06-05: "the 4 sec delay doesn't get faster with a better feed
for listeners."
Previously the listener worklet was locked at rate=1.0 (no time-
stretching, music quality protection) so the Double Dragon controller's
retargets couldn't smoothly shrink the buffer. The listener got the
full 4s cushion forever, even with a steady clean feed.
Changes:
1. Remove the explicit lock_rate=1.0 on listener worklet install —
all worklets now support the ±8% adaptive rate.
2. New constant DD_BASE_LISTENER_TARGET_SEC = 2.0. Listeners start
at RECV_PLAYOUT_DELAY_SEC (4s) for fresh-attach wiggle-immunity;
controller shrinks them toward 2s after DD_CLEAN_SAMPLES_NEEDED
(30s) of clean stats.
3. ddBaseTargetForRole(role) — listener=2s, speaker/cohost/host=0.5s.
4. ddEntry initial currentTargetSec reflects the role's actual
starting target (4s for listeners, 0.5s for speakers).
5. Controller's SHRINK branch uses the role-aware base.
Music quality during the shrink: ≤8% rate change for ~10-20s, ≤1
semitone brief pitch shift. Audible on sustained tones but acceptable
for the latency improvement — listener feels "more live" when room
is calm. Phase 4 (WSOLA grain processing for true pitch preservation)
still deferred; this gets us 80% of the win without it.
Replaces Phase 2's hard SFU↔HTTP switching with continuous
playback-rate adaptation in the worklet itself. Speakers / cohosts /
hosts start at 0.5s conversational latency; when the Double Dragon
controller detects upstream instability it raises the target to 4s
and the worklet TIME-STRETCHES playback (≤8% slowdown) to grow its
actual buffer toward the new target — no silence, no clicks, just a
brief ≤1 semitone pitch shift during the adaptation window. When
clean stats return for 30+ seconds, target shrinks back, worklet
speeds up (≤8%), buffer drains naturally.
Listeners are explicitly locked at rate=1.0 via a new 'lock_rate'
port message — music must not be resampled.
Worklet additions:
- cursor (fractional input-sample index)
- stretchFactor / targetStretch (1.0 default; ±8% range cap)
- stretchRampPerBlock = 0.0002 (about 7.5%/sec ramp)
- adaptive rate target driven by buffered/targetSamples ratio
- linear interpolation between adjacent samples for sub-sample reads
Controller changes:
- ddSetTargetForPub(pubHex, sec) posts retarget to the publisher's
worklet
- Auto path now GROWS target on instability, SHRINKS on sustained
clean — no startStream/stopStream involvement
- DD_BASE_TARGET_SEC = 0.5 (conversational)
- DD_MAX_TARGET_SEC = 4.0 (full wiggle cushion)
- Manual HTTP toggle still respected (state goes manual-on/off,
controller hands off)
Trade-offs documented:
- Linear interpolation (not WSOLA grain processing). Brief pitch
shift during adaptation but minimal artifacts at ≤8% rate change.
- Phase 4 (pitch-preserving WSOLA via grain processing) deferred —
call when current shift is audible enough to upgrade.
End user behavior:
- Calm room: speaker latency stays at 0.5s, conversation feels natural
- Host wiggles: speaker's buffer grows over ~10s to 4s, listener hears
a brief slowdown, then continuous cushion absorbs subsequent wiggles
- After 30s of clean: buffer shrinks back over ~10s, latency returns
Per-publisher health monitor runs once per 5s telemetry tick.
Samples the active audio receiver's lost/jitter; transitions a
per-pubHex state machine between 'mesh' (worklet) and 'http'
(HTTP /stream).
State machine:
mesh (default) — listening via worklet (mesh source)
http — auto-engaged HTTP /stream; worklet muted by
existing startStream gain-ramp
manual-on — user toggled HTTP manually; controller hands off
manual-off — user toggled HTTP off; controller hands off
Auto transitions (manual states never auto-flip):
mesh → http: 2 consecutive samples show loss > 2/s OR jitter > 30ms
http → mesh: 6 consecutive clean samples (~30s)
Manual state set on toggleStreamFor — clicking the per-speaker
toggle records the user's intent; controller respects it until
they leave + re-enter.
Auto engagement reuses the existing infrastructure:
- startStream(uuid, pubHex) sets streamMode + ramps worklet to 0
+ locks lip-sync override to HTTP_STREAM_DELAY_SEC
- stopStream(uuid) clears the override + ramps worklet back
Audible switch: ~2.5s time-jump per direction (mesh delay vs HTTP
delay). Listener briefly hears past content on mesh→http or future
content on http→mesh. Phase 3 (sample-aligned dual-decode via
cross-correlation, fed through a 2-input worklet) eliminates the
jump but needs a dedicated session.
Telemetry will show:
- "double-dragon pub=XXXX AUTO-ENGAGE HTTP (loss/s=N jitter=Nms)"
- "double-dragon pub=XXXX AUTO-DISENGAGE (clean N samples)"
- "double-dragon pub=XXXX → manual-on (manual)" on user toggle
Two changes:
1. SPEAKER_PLAYOUT_DELAY_SEC: 0.5 → 4.0.
Cohost on Fedora Chrome was glitching during host's X11 wiggles
because 0.5s mesh cushion was shorter than typical wiggle stalls.
Fox 2026-06-04: "we needed 4 secs before for the wiggle. it was at
least 4 secs for 20 sec wiggles; we didn't test less." Speaker
conversation latency goes from ~500ms to ~4s round-trip — accepted
trade for wiggle-immunity. Future twin-stream double-dragon
controller (mesh + HTTP /stream parallel, adaptive switching based
on observed loss/jitter + hardware/network/feed performance) will
reclaim conversational latency while keeping the glitch floor.
2. HTTP /stream lip-sync override.
When the per-speaker stream toggle is ON for a publisher, audio
comes from an <audio> element with its own deep buffer (~2.5s),
not the worklet. Previously the lip-sync algorithm kept driving
from the (silent) worklet's measurement → video target stayed at
the worklet's ~4s while audio actually was ~2.5s → 1.5s mouth-
leads-voice. Now startStream sets httpLipSyncOverride[pubHex] =
HTTP_STREAM_DELAY_SEC and forces an immediate retarget via
applyLipSyncForPub. stopStream clears the override and the next
worklet 'buffered' message restores worklet-driven targeting.
Constants kept conservative — HTTP delay is hardcoded at 2.5s; could
be made dynamic by reading audio.buffered.end(0) - audio.currentTime
plus an Ogg-granule offset, but the spread is small relative to the
80ms perception threshold.
Fox 2026-06-04 framing: "think of it like a CD that is literally
being skipped in a physical disc man and solve it with two lasers
one moving fast mesh as fast as possible and one for the broadcast
also as fast as possible dynamic based on not hardware fingerprint
but hardware performance and network performance and feed performance
double headed hydra! double dragon!" — Phase 2 lands the auto-engage
controller per that spec.
Fox 2026-06-04: "the cohost on fedora chrome has the wiggle issue when host
is messing with tabs is there any way to sync the other stream and recover
the glitches assuming it is still buffering where the low-latency version
is glitched, this means we need to delay more than what we are or some
other trick mixed in halp..." (plus: "the video would need to be slid
depending on the mode to keep it in sync. complicated but possible.")
Before: mesh audio bypassed the worklet — went straight to an <audio>
element with the native receiver's ~50ms buffer. The SFU worklet path
had 0.5s of cushion that absorbed 200ms host stalls; mesh did not, so
the same wiggle that was inaudible on SFU was clicky on mesh.
After: ONE worklet per remote uuid, source swappable in-place via the
new setWorkletStream(uuid, newStream). Both the SFU stream (cached in
sfuStreamsByPubHex) and the mesh stream contain the same publisher's
content at slightly different network delays — so disconnecting the
old source and connecting a new one to the same worklet is seamless
(the queue's 0.5s of already-buffered samples covers the transition
while the new source fills it).
Wiring:
- mesh ontrack: setWorkletStream(uuid, meshStream). Fallback to a
fresh attachAudioStreamViaWorklet if no worklet existed (rare —
only when AudioContext failed at SFU attach).
- mesh connectionState='failed': setWorkletStream(uuid, sfuCachedStream)
+ registerLipSyncAudio(pubHex, uuid, sfuReceiver). attachCachedSfuStreamFor
remains as the no-worklet fallback.
- HTTP /stream toggle still uses rampWorkletGain (separate <audio>
element path).
Lip-sync receiver rebinding ("video slid depending on the mode"):
- New sfuAudioReceivers map caches the SFU receiver per publisher.
- handleRemoteSfuTrack mic path: registerLipSyncAudio with SFU receiver
AND store it in sfuAudioReceivers cache.
- mesh ontrack: registerLipSyncAudio with the MESH receiver — the
next worklet 'buffered' message will retarget video to mesh's jbuf
+ worklet (≈ 0.55s) instead of SFU's (≈ 1s).
- mesh fail: registerLipSyncAudio back to the cached SFU receiver
→ video re-targets again.
Removes the previous gain-ramp hack for SFU↔mesh transitions —
single worklet means no parallel paths, no need to crossfade. The
gain-ramp is still used for HTTP toggle (where there genuinely are
two paths: worklet + HTTP <audio>).
CPU cost: same as before — the SFU sub PC still decodes audio for
every speaker (we just route the decoded stream to the worklet or
not). Net change is "the mesh <audio> element is gone" — small
saving.
Single mechanism handling three audible-source transitions, all click-
free via Web Audio linearRampToValueAtTime (100ms by default):
1. SFU → mesh (auto, on mesh PC ontrack):
- Previous: detachListenerStream → hard tear-down of the worklet
- Now: rampWorkletGain(uuid, 0, 100) keeps the worklet decoding in
the background; only the GainNode value moves. Free CPU cost (~3%
on a phone), zero audible click.
2. mesh → SFU (auto, on mesh PC connectionState='failed'):
- Previous: attachCachedSfuStreamFor re-attached the SFU stream
- Now: rampWorkletGain(uuid, 1, 100) restores the path that was
never disconnected. attachCachedSfuStreamFor kept as fallback
for the <audio>-element case (when AudioContext failed at attach).
3. SFU → HTTP /stream (manual, per-speaker toggle button next to mic
state); HTTP → SFU on toggle off:
- Previous: existing mute logic only touched remoteAudio (mesh
<audio> elements). Listeners use listenerAudioNodes (worklet),
so toggle-on left the worklet playing AND started HTTP — double
audio. Fox 2026-06-04: "it was breaking listeners."
- Now: startStream calls rampWorkletGain(uuid, 0, 100) and mutes
the mesh element. stopStream reverses both. Per-speaker toggle
finally works for listeners.
The mesh and HTTP <audio> elements still hard-mute via .muted (cheap,
no Web Audio path for them). Only the SFU worklet path needs the
smooth ramp because it's the one that'd produce a click if cut
mid-sample.
Telemetry will show:
- "mesh audio attached for XXXX — SFU worklet faded out" on mesh ontrack
- existing "stream on for XXXX — DJ mode" on HTTP toggle ON
- no new event on toggle OFF / mesh fail (just the gain ramp)
Fox 2026-06-04: "the cohost is hearing two feeds now."
Regression from role-aware audio routing. Pre-refactor: SFU mic and
mesh mic for the same speaker both wrote into the remoteAudio map
(both used <audio> elements), so the second arrival overwrote the
first — one audible path. Post-refactor: SFU mic for speakers /
cohosts / hosts goes through AudioContext + worklet (stored in
listenerAudioNodes), mesh mic still uses <audio> (in remoteAudio).
Different maps, both play, same voice in two different delays =
echo / phasing.
Fix: at the top of mesh pc.ontrack, call detachListenerStream(uuid)
to remove the SFU worklet path. The existing canSpeak-mesh-skip
in handleRemoteSfuTrack already prevents the OPPOSITE race (SFU mic
arriving after mesh is connected).
Logged so we can see the handoff: "mesh audio attached for XXXX —
SFU worklet path detached".
Fox 2026-06-04: "lips on the video share are not synced with the audio."
Source of the desync:
- Audio total delay = native_jbuf + worklet_buffered (because the
AudioWorklet adds its 4s cushion ON TOP of whatever the native
receiver does)
- Video total delay = native_jbuf only (no worklet downstream)
- Net: video leads audio by up to ~4s on music listeners; by less on
voice (where native honors the hint, ~3s); audible mismatch in either.
Algorithm: per-publisher dynamic match. Every ~683ms the worklet
posts {cmd:'buffered', seconds: this.buffered/sampleRate}. JS-side
handler refreshes lip-sync for that uuid:
1. Resolve uuid → publisher pubHex
2. Read audio receiver's getStats jbuf (native part)
3. audioTotal = native_jbuf + worklet_buffered
4. Push to per-publisher history (size 5)
5. Compute median (FEC-style: 3-of-5 must agree before lock-in)
6. If |median - lastApplied| > 0.05s, set every video receiver for
that publisher: playoutDelayHint = median, jitterBufferTarget = median*1000
Hamming-spirit on a control signal per fox's request: median-of-5
rejects single-sample outliers from network jitter or getStats noise.
50ms hysteresis below perception threshold so the video target
doesn't whip on tiny shifts.
Registers:
- registerLipSyncAudio(pubHex, uuid, receiver) — at mic-kind ontrack
- registerLipSyncVideo(pubHex, kind, receiver) — at screen/camera/game ontrack
- Map: pubHex → {audioUuid, audioReceiver, nativeJbufSec, videoReceivers, history, lastApplied}
The role-aware base delay (playoutDelayForRole) is still applied at
attach as a sane initial value; lip-sync then refines per-publisher
within a few seconds.
Fox 2026-06-04: "listeners state should be connecting for 4 secs while
the buffer fills, not immediately to connected."
Previous behavior: the moment the call FSM hit joined, the status text
flipped to "connected as listener" — but no audio was actually playing
yet because the AudioWorklet hadn't filled to its 4s target. User saw
"connected" but heard nothing for ~4 seconds. Confusing.
New behavior:
- listener role + buffer not yet filled → "connecting — buffering 4s
audio…" (warn-colored dot)
- first AudioWorklet started message arrives → "connected as listener"
(ok dot)
Mechanics:
1. Worklet's process() posts {cmd:'started', targetSeconds} the moment
started flips true (buffer filled to target). One per worklet per
fill cycle.
2. JS-side jbuf.port.onmessage listens; calls onWorkletStarted(uuid).
3. onWorkletStarted flips listenerBufferReady=true once (first started
wins — audio is audible from that point); subsequent worklets'
started events are no-ops for UI purposes.
4. setListenerStatusAware(role) picks the right string. Replaced every
"setStatus('connected as '+role)" call site with this helper.
5. resetListenerBufferReady() called on:
- fresh welcome with role=listener (first join)
- role-change DEMOTING to listener (prev !== 'listener' && next === 'listener')
so the next 4s fill cycle has to complete before "connected as
listener" returns.
Speakers / cohosts / hosts unaffected — their status flips immediately
because their 0.5s buffer fills in half a second; no visible "buffering"
state.
Conversation latency for speakers / cohosts / hosts; lean-back cushion
for listeners. Every receiver type updated together so audio + video
stay in sync (the existing memory rule: video must match audio's
playout delay).
Changes:
1. SPEAKER_PLAYOUT_DELAY_SEC = 0.5; playoutDelayForRole(role) — listener
gets 4s, all others get 0.5s.
2. attachListenerStreamViaAudioContext → attachAudioStreamViaWorklet
(generalized). Listener wrapper just calls into it with 4s + the
Media Session hook. Every role now routes SFU mic audio through the
AudioWorklet + per-role buffer — so a speaker hearing high-bitrate
stereo Opus music STILL gets a 0.5s cushion that Firefox's native
jitter buffer would have ignored.
3. attachSfuTrack: speakers / cohosts / hosts route through the worklet
path with 0.5s buffer; <audio>-element fallback only on
AudioContext failure.
4. SFU sub PC video + mic native receiver: jitterBufferTarget +
playoutDelayHint = playoutDelayForRole(myRole). Listener=4s, others=0.5s.
5. Mesh peer receivers: SPEAKER_PLAYOUT_DELAY_SEC always (mesh is
always peer-to-peer conversation, no role-mixed case).
6. AudioWorklet handles a {cmd:'retarget', targetSeconds} message —
recomputes targetSamples / maxSamples and shrinks the queue if the
new cap is smaller. No reconstruction needed across role changes.
7. retargetAllReceivers(role) called from onRoleChanged before mic
acquisition starts. Walks listenerAudioNodes (worklet) and
sfuSubPC.getReceivers() (native audio + video) and applies the new
target. Mesh peers stay at 0.5s unconditionally.
Speakers were previously running the same 4s setting as listeners. The
native buffer was honoring it for voice (ramping to ~3s) which meant
back-and-forth conversation was effectively impossible — they were
hearing each other 3 seconds late and didn't notice because they were
mostly publishing. This brings conversational latency back to ~500ms
while keeping the listener cushion intact.
No new SDP / signaling — all changes are receiver-side at attach. Role
transition refreshes targets on the existing PC without a renegotiation.
Fox 2026-06-04: "delay does seem to be about 4 secs but still very chappy."
Buffer IS holding 4s and emitting — that part works — but every transient
empty-queue tick (a single 2.67 ms drain) was setting started=false,
which forced a full 4-second re-fill before emit resumed. So a 50 ms
network jitter on the upstream caused a 4 s silence on the listener.
That's the chop.
Fix: track consecutive empty-queue blocks. Only re-arm (started=false)
after rearmThresholdBlocks (100 = ~267 ms) of sustained silence.
Brief drains emit silence-fill but keep started=true so playback
resumes the instant new samples arrive. Listener hears at most ~267 ms
of dead air on each drain — almost certainly Opus PLC will mask far
shorter ones.
Long outages (>267 ms with no samples) still re-buffer to 4 s — that
case isn't this bug, it's a real upstream death where a fresh
cushion is correct.
Browser-native jitterBufferTarget didn't help on the music stream —
Firefox Android holds it at 0.06–0.21s on a high-bitrate stereo Opus
receiver while honoring 4s on voice and video receivers on the same
PC. Per-codec implementation gap in the receiver-side jitter buffer.
This adds a userland buffer in Web Audio. The listener path already
ran through AudioContext (source → gain → destination); now an
AudioWorkletNode sits between source and gain, queues incoming
128-sample blocks until targetSamples (4 × sampleRate) have arrived,
then emits with a constant delay. Bounded at maxSamples (6 ×
sampleRate) so clock drift can't grow the queue unbounded. If the
queue fully drains, the buffer re-arms — a hiccup doesn't lock us
silent.
Worklet code lives inline as a Blob URL (single-file app: no
separate JS file shipped). loadJitterWorklet is fire-and-forget on
first attach; existing direct-connected streams get swapped through
the buffer the moment the worklet module finishes loading. Fallback
on worklet creation failure: existing source → gain path stays live.
Speaker / cohost / host paths untouched — they need conversational
latency, can't sit on a 4s cushion.
Listener role test plan: rejoin, watch the new "jitter-buffer
installed" log line, observe that the listener is now 4s behind the
host's speech. Wiggle the host (X11) — listeners should hear no
disruption while the buffer is full.
Fox 2026-06-04 — definitive observation: "the phone as a listener doesn't
seem to be 4 secs behind ever." playoutDelayHint is a HINT the browser
is free to ignore; Firefox Android apparently does. The phone's actual
buffer was near-zero — so every host-side encoder stall propagated
audibly to listeners with no cushion.
Three changes:
1. Set RTCRtpReceiver.jitterBufferTarget = 4000 (ms) alongside
playoutDelayHint. jitterBufferTarget is NOT a hint — it's a target
the receiver must aim for. Chromium 113+ (May 2023), Firefox 124+
(2024). Older browsers silently ignore the assignment (try/catch).
2. Applied at all THREE attach sites:
- SFU video receiver (screen/camera/game)
- SFU mic receiver
- mesh peer mic receiver
3. Add `jbuf=Xs` to telemetry, computed from inbound-rtp
jitterBufferDelay / jitterBufferEmittedCount. This is the ACTUAL
average buffer depth — we can now see whether the receiver is
holding ~4s or 0.05s. If jbuf stays small after this deploy, the
browser is ignoring the target too and we need a different
approach (AudioWorklet manual buffering, or move to HTTP-pull DJ
path for listeners).
Existing playoutDelayHint setting kept for older browsers that honor
it but don't yet support jitterBufferTarget.
Previous loop picked the max-packets outbound-rtp per PC, which always
collapsed onto the video track on screen/camera PCs — hiding what
screen-audio was doing during the wiggle test entirely.
Now iterates ALL outbound-rtp entries on every publisher PC and emits
one line per kind: scr.send.aud, scr.send.vid, cam.send.aud (when the
camera mic is published), cam.send.vid, mic.send.aud, game.send.*.
remote-inbound-rtp is paired back by ssrc.
Lets us answer the key diagnostic question: when Firefox's
getDisplayMedia video capture stalls under X11 wiggle, does the audio
track from the same MediaStream stall in lockstep (Firefox couples
audio + video producers internally) or stay flowing (decoupled)? If
decoupled, an app-layer fix (route audio + video to separate
RTCPeerConnections) would work. If coupled, the fix has to be either
OS-level (PulseAudio loopback to mic) or upstream in Firefox.
When the sender's egress queue gets contended, audio wins. Fox 2026-06-04:
wiggling a terminal window (X11) caused 1-5s audio cutouts on all mesh
peers AND listeners — even with playoutDelayHint=4.0 on subs. Theory
(fox): "i might be the clients video trying to compete with the audio,
the wiggled updates" — exactly that. X11 window wiggling produced a
storm of dirty regions for screen-share; the encoder spiked into a big
keyframe burst; audio packets queued behind that burst arrived late;
listener's playoutDelayHint buffer drained.
Fix: RTCRtpEncodingParameters.priority + networkPriority. Set audio to
'high' (mic + any DJ audio track), video to 'low' (screen, camera).
Both Firefox and Chromium honor these for the local egress queue —
audio packets jump ahead of video bursts.
Applied at:
- setSenderBitrate (mic publisher) — high
- setSenderMaxBitrate (screen audio, camera audio) — high
- setSenderMaxBitrate (screen video, camera video) — low
No new param, no SDP renegotiation needed — setParameters() applies
immediately to the existing PC. Existing call sites all flow through
these two helpers, so the change reaches every publisher.
Pair with publisher-side outbound-rtp telemetry (836a036) — gap on
aud.send during a wiggle is now the metric we measure against.
Adds aud.send/cam.send/scr.send/game.send lines to the 5s telemetry tick:
- pkt sent, bytes sent, framesEncoded (video)
- gap = now − lastPacketSentTimestamp (wire stall detector)
- rtt + rlost + rjit from remote-inbound-rtp (the SFU's view of us)
Until now we only had inbound stats on the listener path. A publisher
stall — e.g. X11 compositor blocking Firefox during a window wiggle —
was invisible at the wire; we could only infer it from downstream
listener loss/silence. Fox 2026-06-04: "wiggling a terminal window
causes 1-5s cutout on all mesh devices and listeners (4s buffer should
absorb that)." Hypothesis is a publisher-side audio-thread stall;
this lets us confirm by watching gap >> 0.02s on aud.send during a
wiggle.
No new IPC, no new dispatch — same getStats() call shape already in
use for inbound, just iterating the publisher PCs (sfuPubPC,
sfuCameraPC, sfuScreenPC, sfuGamePC) and picking outbound-rtp +
remote-inbound-rtp pairs.
The mod-action serializer (runModSerial) was waiting for case 'state' to
release the next sign — but no 'state' broadcast follows a successful
kick/ban/promote. The server bumps rm.epoch and emits the action-specific
event (peer-booted / role-change) carrying the new epoch. The client was
already updating roomEpoch in role-change but never released the queue
gate, and the peer-booted handler did neither — so kick/ban always burned
the 1500ms fallback timeout AND signed the next action with the stale
epoch (signal bounces it with "stale epoch").
Fox 2026-06-04: "couldn't kick until leaving as host" — leaving + welcome
was the only thing that refreshed roomEpoch.
Pair with signal-side fix that moves rm.epoch++ before the peer-booted
broadcast and includes "epoch" in the payload.
`muted` was a bare global mutated from 8+ sites (btn-mute click, peer-force-muted,
role promote/demote, self-listener enable/disable, leave handler, sessionStorage
restore). Each call site also had to remember to call applyMuteState() and
sendMicState(). Drift was inevitable — a recent regression where self-listener
toggle muted the wrong direction came straight from this implicit-state pile.
New shape:
- muteSpec: states { on, off }, events { TOGGLE, FORCE_MUTE, AUTO_MUTE,
AUTO_UNMUTE, RESTORE_MUTED, RESTORE_UNMUTED, ROLE_PROMOTED }, ctx.source
tracks who muted us ('self', 'mod', 'self-listener').
- `muted` is now a getter over roomMachines.mute.state — single source of truth.
- One observer drives applyMuteState + sendMicState + sessionStorage persistence
+ log line on every transition.
- Imperative call sites only dispatch events; they no longer touch side effects.
Tests: test/zebra-fsm.test.js harness updated to extract muteSpec (same brace-
matched-regex pattern as selfListenerSpec). 88/0 passing. MuteFSM-specific
transition tests are next.
Pattern is now load-bearing — call/publish/subscribe/remote-tile/self-listener/
mute all live as FSMs with the same shape.
Builds on 07d0af9 (publishSpec wired to imperative flow) — adds an
observer that derives btn-{screen,camera}-share / btn-{kind}-stop
visibility from the publishSpec state instead of from scattered
classList writes in sfuPublish/sfuUnpublish.
The state→UI mapping:
- off / failed → share button visible, stop hidden
- acquiring / negotiating / live / stopping → stop visible, share hidden
(stopping is treated as "still sharing" to avoid flicker during
teardown)
Removed four classList writes (two in sfuPublishScreen+sfuUnpublishScreen,
two in sfuPublishCamera+sfuUnpublishCamera). The buttons now reflect
the FSM rather than whoever last touched them — if the FSM
transitions because of the silent rebuild (sfuRebuildScreenPC) or
the watchPublishPC rebuild path or any future state-driver, the
buttons follow automatically.
Mic has no share/stop button (it's always on for speakers, off for
listeners), so applyPublishStateUI early-returns for that kind.
All 88 fsm + 12 self-listener tests still green.
The publishSpec FSM (off → acquiring → negotiating → live → stopping
→ off) has existed since the FSM scaffolding landed but was never
actually driven — wireZebraMachines created mic/camera/screen
instances and the imperative sfuPublish* / sfuUnpublish* never sent
events at them. So the spec was documented behavior, not enforced
behavior.
Wired all three publish paths to drive their FSMs in lock-step with
the imperative flow:
- sfuPublish (mic): START → ACQUIRED (with stream) → NEGOTIATED
(with pc + peerID) → on watchPublishPC rebuild path STOP+DONE
- sfuPublishCamera: same pattern, FAILED+DONE on getUserMedia
cancel + HTTP errors
- sfuPublishScreen: same pattern, FAILED+DONE on getDisplayMedia
cancel + HTTP errors
- sfuUnpublish / sfuUnpublishCamera / sfuUnpublishScreen: STOP +
DONE around the actual teardown
Trace observers attached for all three kinds — every transition
logs as `pub.mic: prev → next [EVENT]` so QA can see the publish
lifecycle in the page log alongside the imperative `sfu publish:
starting` lines.
The imperative state (sfuPubPC / sfuCameraPC / sfuScreenPC) remains
the source of truth for now. The FSM is a parallel view that
future migrations can hang side effects off of (e.g. a single
applyPublishStateUI() observer driving the share-button visibility
the way applyCallStateUI() drives the call chrome).
All 88 zebra-fsm tests still green (no spec changes, only call-site
additions). The pattern fox 2026-06-04 directed: every system as a
state machine, side effects ride observers.
Fox 2026-06-04 directive: "all systems need state machines." Self-
listener is already an FSM (commit f4dbc5c). Next system: the
top-line connection chrome (entry-row visibility, sec-room reveal,
dot color, leave/mute button visibility, status text, btn-enter
disabled). Previously these were scattered classList + setStatus
writes across welcome, peer-booted (self), btn-leave click, and
handleBlocked — easy to drift, every UI bug fox flagged ("dot still
green after kick", "entry row should be hidden when joined", "leave
button gone after kick") was a different leaf of this implicit
state model.
Wire applyCallStateUI(state, prev, ctx) as a roomMachines.call
observer. Single function, six branches (idle/connecting/joined/
reconnecting/leaving/booted), drives every relevant DOM toggle.
callSpec gains a `bootedAction` context field — 'kick' | 'ban' |
'blocked' | null — set by BOOTED's action so the UI observer can
render the right status ('kicked from this space' vs 'banned from
this space' vs 'blocked from this space') AND decide whether to
re-enable btn-enter (kick: yes, can re-enter; ban/blocked: no).
Call sites updated:
- handleBlocked: now passes { action: 'blocked' } in BOOTED payload
- case 'peer-booted' (self): now passes { action: m.action } so
kick vs ban propagates to the FSM
- btn-leave click: imperative chrome removed (was a 9-line
classList chain), replaced by send('LEAVE') + send('DONE') —
the observer handles the rest
- joinSpace: removed imperative btn-enter.disabled / setStatus
- case 'welcome': removed imperative dot/buttons/sec-room/row-entry
toggles — observer covers them
Tests added in test/zebra-fsm.test.js (now 88/88):
- BOOTED with action=kick → bootedAction=kick
- BOOTED with action=ban → bootedAction=ban
- BOOTED with action=blocked → bootedAction=blocked
- BOOTED with no action defaults to kick (back-compat)
- BOOTED → ACK → idle clears bootedAction
Future migrations should follow this pattern: add a state field to
the spec, hoist the imperative side effects into a switch in an
observer, leave a comment at the OLD imperative location explaining
the migration so the next reader doesn't reintroduce drift.
Asymmetric recovery between media publishers fixed. Mic + camera use
watchPublishPC for auto-rebuild on PC 'failed' — getUserMedia
constraints don't need a gesture once permission is granted, so the
client just creates a fresh PC and re-publishes. Screen share has
been the odd one out: getDisplayMedia requires a fresh user gesture,
so the previous policy was to tear everything down on PC 'failed' +
show a "tap share screen to re-share" log line.
But — the browser's "you are sharing" indicator typically stays on
across a transient network glitch; the MediaStream tracks remain in
readyState='live'. We don't need a fresh getDisplayMedia call; just
a fresh PC bound to the same stream. New sfuRebuildScreenPC() does
exactly that: tears down the dead PC (nulls sfuScreenPC first to
avoid recursive teardown via onconnectionstatechange), unpublishes
the old peer_id at the SFU, builds a fresh PC against the existing
sfuScreenStream, publishes, re-attaches the same recovery handler.
Asymmetry collapses: screen now auto-rebuilds silently IF the source
is still alive, and only falls back to the manual "share screen"
prompt when the user has actually stopped sharing (browser-native
stop, source window closed, lid closed). Mirror of the sub-PC
self-heal we shipped in 09a4516 — closes the last asymmetry between
publishers (and the last UX wart fox 2026-06-04 flagged on rejoin).
When the SFU closes the listener's sub PC server-side (e.g. via the
wedge-recovery in renegotiateLocked), the browser's
pc.connectionState transitions to 'closed', not 'failed'. The
existing handler only triggered a rebuild on 'failed' and
explicitly returned on 'closed' (under the assumption that 'closed'
== self-teardown). That assumption holds for sfuUnsubscribe (which
nulls sfuSubPC BEFORE pc.close()), but NOT for remote-driven
closes — sfuSubPC === pc is still true, the guard sees
'unexpected close', and we should re-subscribe.
Without this branch the host's sub stayed at sub=none indefinitely
after any server-driven close — every subsequent addPubToSub for
the host went into the void. Fox 2026-06-04 lost cohost camera +
screen share via this exact path after the wedge-recovery falsely
fired on his fresh subscribe (SFU fix 8c5fd65 prevents the false
positive going forward, this fix ensures the page recovers on any
legitimate server close).
Both 'failed' and 'closed' now rebuild via sfuUnsubscribe +
sfuSubscribe, gated on sfuSubPC === pc to catch only remote-driven
state changes.
