Fox 2026-06-05: "transcribe states should be saved in localstorage to
survive on refresh" and "self transcribe does not seem to work."
Two fixes:
1. localStorage persistence:
- TRANSCRIBE_KEY = 'zspc:transcribe-on'
- Loaded into transcribeEnabled at script top
- setTranscribe(want) replaces the old toggleTranscribe body and
writes localStorage on every state change
- applyTranscribeUI() reusable for both flip and refresh-restore
- At click-handler binding time, if transcribeEnabled was already
true (restored from storage), apply UI + pre-warm worker + kick
captures for any speakers already attached
2. Self-capture fixed:
- startSelfCapture used to early-return if audioCtx was null
- But audioCtx is only created when the FIRST remote speaker's
audio attaches via attachAudioStreamViaWorklet. A host alone in
the room (no remote audio yet) had no audioCtx → self capture
silently failed.
- Now startSelfCapture creates audioCtx if absent and resumes if
suspended, same pattern as the remote-attach path. Host-alone
captures their own voice immediately.
Two related additions:
1. Self-transcription. The host's own voice wasn't being transcribed
because capture nodes only attached to listenerAudioNodes (remote
speakers' streams from SFU/mesh worklets). micStream — the local
capture for publishing — was never tapped. Added startSelfCapture
/ stopSelfCapture that wraps micStream in a parallel
whisper-capture worklet. Lines are tagged with myUUID so the local
user's handle shows in the transcript log.
Wired into:
- toggleTranscribe ON/OFF → starts/stops self capture alongside
remote captures
- getMic → starts self capture if transcribe is already on (covers
"user enabled transcribe before granting mic permission")
- applyMicMode / reacquireMic → stops + restarts self capture
against the new micStream so we don't keep transcribing a
stopped MediaStreamTrack
2. Whisper telemetry stats. Each 5s tick now appends:
xcr=on sent=N drp=N emt=N avgMs=N wrkr=1 caps=N selfCap=1
- xcr: on/off
- sent: chunks shipped to the worker this tick
- drp: chunks dropped by the inflight gate (worker saturated)
- emt: text lines emitted after silence/hallucination/dup filters
- avgMs: average inference latency per chunk (device-perf proxy)
- wrkr: 1 once the worker has loaded the model
- caps: # of active remote-capture worklets
- selfCap: 1 if local mic is being captured
Reset each tick so we see RATE, not cumulative.
Lets us compare devices across mesh from server-side signal log:
Snapdragon-8 should show avgMs ~150-300; mid-range phones 1500-
3000; desktop x86 50-150. drp > 0 means worker can't keep up with
the audio chunks arriving (multiple speakers talking at once).
Factored the chunk-handling pipeline into handleWhisperChunk(uuid,
chunk) so both remote and self captures share the same filter +
inflight-gate + telemetry path.
Fox 2026-06-05: "the web page for zebra spaces seems noticeably slower
after enabling transcribe… even the tones for entering and leaving are
showing up way way later even on the host side."
Root cause: ONNX Runtime via transformers.js was running on the main
thread, blocking JS for 1–3 seconds per chunk. Join/leave chimes,
button clicks, scroll, EVERY UI gesture queues behind it.
Two changes:
1. Move Whisper to a dedicated module Web Worker. The worker imports
transformers.js + loads the whisper-tiny.en pipeline ONCE; each
chunk is transferred (zero-copy) via postMessage, processed in
isolation from the UI thread, and the resulting text is posted
back. Main thread is free during inference now — UI stays
responsive. Worker is created on first toggle ON; same ~40MB
model download still happens, just off-thread.
2. Global "inflight gate" on the main thread side. Only one
transcribe request can be in flight at a time. If a new chunk
arrives while busy, DROP it (don't queue). Counter is logged
every ~30s so we can see worker saturation. With N speakers all
talking at once, dropping is correct — stale chunks from 10s ago
aren't worth transcribing.
Combined with the RMS silence gate in the capture worklet, the result
is: silent chunks never even reach the main thread, busy chunks are
processed sequentially by the worker, and the UI never blocks.
Phase 4 (deferred): switch to WebGPU backend for ONNX where supported
— roughly 2-5× faster than WASM on capable devices. transformers.js
v3 supports this with `{ device: 'webgpu' }` in pipeline opts.
Fox 2026-06-05: "fxhp and fxhp-chrome both are triggering clients to
transcode `you` on a new line over and over could you sort that out?
… this happens even when mic is closed."
whisper-tiny.en is notorious for hallucinating common stop-words on
silent / low-energy audio: "you", "thank you", "thanks for watching",
a lone period. Mic muted or peer dead-silent → the model still
produces a chunk of audio (silence frames from Opus / WebRTC) and
spits one of these phrases out. Two filters:
1. RMS-based silence gate IN the capture worklet (cheaper — no
pipeline invocation at all). Compute RMS of the 5s/16kHz chunk
before posting; skip if below 0.005 (~-46 dBFS, well below any
real speech). The vast majority of "mic-closed" hallucinations
stop here.
2. JS-side filter against a known-hallucination set. Catches the
rest (quiet-room ambient that passes the RMS gate). Normalizes
lowercase + strips trailing punctuation so "You." / "you " /
"YOU!" all collapse to "you" and match the set. Easy to extend
as new hallucinations are observed.
3. Per-uuid consecutive-duplicate suppression. Even non-hallucination
text sometimes re-emits the same short phrase across consecutive
chunks ("uh huh" "uh huh") — drop the dupe.
CPU win: silent chunks no longer go through ONNX inference (~50-200ms
per chunk on tiny.en).
Fox 2026-06-05: "listeners should still be synced it should just be
that the gesture to change feeds after they are all live should be
immediately to make that feed full sized... reverse your bs and do
what I asked."
Reverted the listener-video-no-delay shortcut. Listener video keeps
playoutDelayHint matched to audio (lip-sync preserved). The actual
fix for slow tile-switching: REUSE the existing thumb tile's <video>
element instead of building a fresh one when promoting to spotlight.
Before: setSpotlight built a brand-new <video> in #spotlight and
assigned the existing MediaStream as its srcObject. The fresh video
element had no decoder state — had to wait for the next keyframe
inside the receiver's 4s-deep buffer to show its first frame,
manifesting as up to 4s of "blank" on listener mobile.
After: setSpotlight DOM-moves the existing thumb tile (with its
already-playing <video> element) into #spotlight. The video element
NEVER stops — no keyframe wait, no re-buffer, no decode-init.
clearSpotlightDOM does the reverse: moves the spotlit tile back to
its thumbs container. Browser DOM-move on an HTMLMediaElement does
not reset playback state.
Supporting changes:
1. CSS: viewing-badge default-hidden; only visible in .tile-thumb.viewing
context (which no longer exists post-DOM-move since the tile IS
the spotlight, no separate thumb to mark).
2. CSS: .tile-thumb .fs-btn { display: none } — buildTile always
includes the fullscreen button now (so we don't have to add it
dynamically when moving thumb → spotlight). The button is hidden
when the tile is in thumb context.
3. buildTile: always builds fsBtn (used to be !isThumb only).
4. setSpotlight: same-target click = no-op (idempotent); previous
spotlight is moved back via clearSpotlightDOM before installing
the new one.
5. The mid-spotlight supplant path (publisher republishes the same
kind): the entry-video swap also IS the spotlight-video swap now
(same DOM node) — removed the redundant second swap, just mirror
the new entry.video reference into spotlight.video.
6. Game spotlight unchanged — game tiles are static iframes built
fresh each spotlight, not DOM-moved.
End result: lip-sync stays correct for every role (audio + video
share the same playoutDelayHint), tile switching is instant.
Fox 2026-06-05: "we should switch feeds immediately I don't want any algo
slowing that down."
The previous lip-sync algorithm set video receiver playoutDelayHint to
match audio's worklet delay (up to 4s on listeners). That made
spotlight-switching take up to 4s — the new tile's <video> element
had to wait for a keyframe to appear inside the 4s-deep receiver
buffer before showing the first frame.
Changes (listener role only):
- handleRemoteSfuTrack screen/camera/game branch: vDelay = 0 for
listener (browser-lowest). Speaker/cohost/host unchanged.
- refreshLipSyncForUuid: early-return for myRole === 'listener'.
No more retargeting video to match audio for listeners.
- retargetAllReceivers (called on role change): audio retargets
per role; VIDEO retargets to 0 for listener, role-aware for
others.
Listener video now plays at browser-default minimum delay. Tile
switches are instant — keyframe wait + native buffer pre-roll only,
typically <300ms.
Tradeoff accepted: listener mouth-leads-voice by approximately the
audio worklet's current cushion (1.3–4s adaptive). For consume-
content mode (lean-back listener) this is correct; switching shares
is far more important than per-syllable lip-sync.
Speakers/cohosts/hosts unaffected — their audio cushion is ~0.5s,
video matches it via lip-sync, mouths and voice align naturally.
Fox 2026-06-05: "do we access to a whisper speech to text?" → no, then
"yes implement this and please this is perfect, make it an off be
default toggle that is part of the client. individual speakers should
ues their names and show up like a log under the video area in middle."
Implementation:
1. whisper-capture AudioWorklet — separate from the jitter-buffer
worklet. Decimates the speaker's 48 kHz mono stream to 16 kHz
(Whisper's expected sample rate), batches 5-second chunks, ships
them to the main thread via transferable Float32Array on the
port. Disabled by default; { cmd: 'start' } / { cmd: 'stop' }
from JS gate the capture.
2. transformers.js + Xenova/whisper-tiny.en lazy-loaded from
jsdelivr CDN on first transcribe-toggle ON. ~40 MB one-time
download (cached by the browser); subsequent toggles are
instant. ONNX Runtime runs entirely client-side — no audio
leaves the listener's browser.
3. Per-uuid capture lifecycle. When transcribe is on, every speaker
in listenerAudioNodes gets a parallel AudioWorkletNode that taps
their source. Recognized text appends to a transcript log with
"HH:MM:SS name: text" lines. Names resolved via
members.get(uuid).handle. Empty / placeholder transcriptions
("." "[BLANK_AUDIO]") filtered out.
4. UI:
- Toggle button "transcribe (off/on)" in the controls column,
with explainer note about model size + privacy.
- #sec-transcript section in the timeline column directly under
the spotlight. Monospace font, 14rem max-height, scrollable,
auto-scrolls to bottom unless user is reading older lines.
- .hidden until first toggle ON; subsequent toggles show/hide.
5. New-speaker hook: attachAudioStreamViaWorklet checks
transcribeEnabled and auto-installs capture for late joiners.
6. detachListenerStream tears down node.capture along with the rest.
Bandwidth: zero (model and audio never leave the device).
CPU on listener: ~5-15% per speaker during the brief inference
window every 5s, idle otherwise. Tested mentally; needs real-world
verification.
Fox 2026-06-05: "since we have this now there is no reason to have the
stream button anymore on the ux."
The ○/◉ button next to each row was the self-listener entry point
(speaker flips themselves onto the buffered HTTP Ogg/Opus path when
WebRTC was glitchy). With the adaptive worklet — predictive drain
detector, self-calibrating floor, ±6% time-stretching — the same
glitch-recovery happens automatically without user intervention.
Removed:
- 'strm' grid column from .member layout (back to 4 columns)
- .stream-toggle CSS rules (button styling, dark-mode variants)
- The button render block in renderRoom (per-uuid streamEl creation)
- row.appendChild(streamEl) and its preceding comment
- The unwired dead toggleStreamFor(uuid, pubHex) helper function
Kept (still useful):
- streamMode Set + startStream() + stopStream() — used by the global
self-listener FSM
- selfListenerMode and its observer
- The HTTP /stream lip-sync override (httpLipSyncOverride)
- Double Dragon's manual-on/manual-off states + ddNoteManualToggle
(no UI surface but the plumbing stays for future re-introduction
if needed)
Layout is back to badge|handle/pub/acts|mic|meter — one fewer column,
slightly more breathing room per row.
Fox 2026-06-05: "both and see if we could lower to 1.3 secs as floor
for listeners on the high quality stream."
Two additions to the Double Dragon controller:
1. Predictive GROW (lead, not lag).
The existing 5s-tick path watches loss/jitter — strictly a lagging
indicator. We now ALSO watch the worklet's bufferedSeconds
reports (every ~683ms) for rapid drain. If the buffer drops by more
than 25% of its current target over a 1.5s window, GROW
immediately — don't wait for loss to appear in the next 5s
sample. Catches wiggle ~2s after it starts vs 5-10s on the
lagging path.
2. Self-calibrating floor.
Each publisher tracks its own maxWiggleDurationSec for the
session. Floor for SHRINK is max(role-base, maxWiggle × 1.5). A
stable publisher's listener can shrink to 1.3s. After observing a
1s wiggle the floor lifts to 1.5s; after a 4s wiggle, 6s (clamped
to MAX). Resets when user leaves + rejoins.
Constants:
- DD_BASE_LISTENER_TARGET_SEC = 1.3 (was 2.0)
- DD_WIGGLE_FLOOR_MULTIPLIER = 1.5
- DD_DRAIN_THRESH_FRAC = 0.25
- DD_DRAIN_WINDOW_SEC = 1.5
Telemetry log lines:
- "double-dragon pub=XXXX PREDICTIVE GROW → 4s (buf drained Nms in N.Ns)"
- "double-dragon pub=XXXX wiggle ended dur=N.Ns — floor now N.Ns"
- "double-dragon pub=XXXX SHRINK target → N.Ns (clean N samples, role=X, max-wiggle=N.Ns)"
This is the "lead + lag, AND self-tuned floor" loop. The 1.3s listener
floor only applies to clean-history sessions. A wiggle-prone publisher
will see the floor stay elevated automatically.
Gentler pitch shift during buffer adaptation at the cost of slightly
longer ramp time:
- Shrink 4s→2s: ~25s → ~31s
- Grow 0.5s→4s: ~47s → ~61s
Pitch shift drops from ≤1 semitone to ≤0.7 semitone — less perceptible
on music. Fox 2026-06-05 dial.
Fox 2026-06-05: "the 4 sec delay doesn't get faster with a better feed
for listeners."
Previously the listener worklet was locked at rate=1.0 (no time-
stretching, music quality protection) so the Double Dragon controller's
retargets couldn't smoothly shrink the buffer. The listener got the
full 4s cushion forever, even with a steady clean feed.
Changes:
1. Remove the explicit lock_rate=1.0 on listener worklet install —
all worklets now support the ±8% adaptive rate.
2. New constant DD_BASE_LISTENER_TARGET_SEC = 2.0. Listeners start
at RECV_PLAYOUT_DELAY_SEC (4s) for fresh-attach wiggle-immunity;
controller shrinks them toward 2s after DD_CLEAN_SAMPLES_NEEDED
(30s) of clean stats.
3. ddBaseTargetForRole(role) — listener=2s, speaker/cohost/host=0.5s.
4. ddEntry initial currentTargetSec reflects the role's actual
starting target (4s for listeners, 0.5s for speakers).
5. Controller's SHRINK branch uses the role-aware base.
Music quality during the shrink: ≤8% rate change for ~10-20s, ≤1
semitone brief pitch shift. Audible on sustained tones but acceptable
for the latency improvement — listener feels "more live" when room
is calm. Phase 4 (WSOLA grain processing for true pitch preservation)
still deferred; this gets us 80% of the win without it.
Replaces Phase 2's hard SFU↔HTTP switching with continuous
playback-rate adaptation in the worklet itself. Speakers / cohosts /
hosts start at 0.5s conversational latency; when the Double Dragon
controller detects upstream instability it raises the target to 4s
and the worklet TIME-STRETCHES playback (≤8% slowdown) to grow its
actual buffer toward the new target — no silence, no clicks, just a
brief ≤1 semitone pitch shift during the adaptation window. When
clean stats return for 30+ seconds, target shrinks back, worklet
speeds up (≤8%), buffer drains naturally.
Listeners are explicitly locked at rate=1.0 via a new 'lock_rate'
port message — music must not be resampled.
Worklet additions:
- cursor (fractional input-sample index)
- stretchFactor / targetStretch (1.0 default; ±8% range cap)
- stretchRampPerBlock = 0.0002 (about 7.5%/sec ramp)
- adaptive rate target driven by buffered/targetSamples ratio
- linear interpolation between adjacent samples for sub-sample reads
Controller changes:
- ddSetTargetForPub(pubHex, sec) posts retarget to the publisher's
worklet
- Auto path now GROWS target on instability, SHRINKS on sustained
clean — no startStream/stopStream involvement
- DD_BASE_TARGET_SEC = 0.5 (conversational)
- DD_MAX_TARGET_SEC = 4.0 (full wiggle cushion)
- Manual HTTP toggle still respected (state goes manual-on/off,
controller hands off)
Trade-offs documented:
- Linear interpolation (not WSOLA grain processing). Brief pitch
shift during adaptation but minimal artifacts at ≤8% rate change.
- Phase 4 (pitch-preserving WSOLA via grain processing) deferred —
call when current shift is audible enough to upgrade.
End user behavior:
- Calm room: speaker latency stays at 0.5s, conversation feels natural
- Host wiggles: speaker's buffer grows over ~10s to 4s, listener hears
a brief slowdown, then continuous cushion absorbs subsequent wiggles
- After 30s of clean: buffer shrinks back over ~10s, latency returns
Per-publisher health monitor runs once per 5s telemetry tick.
Samples the active audio receiver's lost/jitter; transitions a
per-pubHex state machine between 'mesh' (worklet) and 'http'
(HTTP /stream).
State machine:
mesh (default) — listening via worklet (mesh source)
http — auto-engaged HTTP /stream; worklet muted by
existing startStream gain-ramp
manual-on — user toggled HTTP manually; controller hands off
manual-off — user toggled HTTP off; controller hands off
Auto transitions (manual states never auto-flip):
mesh → http: 2 consecutive samples show loss > 2/s OR jitter > 30ms
http → mesh: 6 consecutive clean samples (~30s)
Manual state set on toggleStreamFor — clicking the per-speaker
toggle records the user's intent; controller respects it until
they leave + re-enter.
Auto engagement reuses the existing infrastructure:
- startStream(uuid, pubHex) sets streamMode + ramps worklet to 0
+ locks lip-sync override to HTTP_STREAM_DELAY_SEC
- stopStream(uuid) clears the override + ramps worklet back
Audible switch: ~2.5s time-jump per direction (mesh delay vs HTTP
delay). Listener briefly hears past content on mesh→http or future
content on http→mesh. Phase 3 (sample-aligned dual-decode via
cross-correlation, fed through a 2-input worklet) eliminates the
jump but needs a dedicated session.
Telemetry will show:
- "double-dragon pub=XXXX AUTO-ENGAGE HTTP (loss/s=N jitter=Nms)"
- "double-dragon pub=XXXX AUTO-DISENGAGE (clean N samples)"
- "double-dragon pub=XXXX → manual-on (manual)" on user toggle
Two changes:
1. SPEAKER_PLAYOUT_DELAY_SEC: 0.5 → 4.0.
Cohost on Fedora Chrome was glitching during host's X11 wiggles
because 0.5s mesh cushion was shorter than typical wiggle stalls.
Fox 2026-06-04: "we needed 4 secs before for the wiggle. it was at
least 4 secs for 20 sec wiggles; we didn't test less." Speaker
conversation latency goes from ~500ms to ~4s round-trip — accepted
trade for wiggle-immunity. Future twin-stream double-dragon
controller (mesh + HTTP /stream parallel, adaptive switching based
on observed loss/jitter + hardware/network/feed performance) will
reclaim conversational latency while keeping the glitch floor.
2. HTTP /stream lip-sync override.
When the per-speaker stream toggle is ON for a publisher, audio
comes from an <audio> element with its own deep buffer (~2.5s),
not the worklet. Previously the lip-sync algorithm kept driving
from the (silent) worklet's measurement → video target stayed at
the worklet's ~4s while audio actually was ~2.5s → 1.5s mouth-
leads-voice. Now startStream sets httpLipSyncOverride[pubHex] =
HTTP_STREAM_DELAY_SEC and forces an immediate retarget via
applyLipSyncForPub. stopStream clears the override and the next
worklet 'buffered' message restores worklet-driven targeting.
Constants kept conservative — HTTP delay is hardcoded at 2.5s; could
be made dynamic by reading audio.buffered.end(0) - audio.currentTime
plus an Ogg-granule offset, but the spread is small relative to the
80ms perception threshold.
Fox 2026-06-04 framing: "think of it like a CD that is literally
being skipped in a physical disc man and solve it with two lasers
one moving fast mesh as fast as possible and one for the broadcast
also as fast as possible dynamic based on not hardware fingerprint
but hardware performance and network performance and feed performance
double headed hydra! double dragon!" — Phase 2 lands the auto-engage
controller per that spec.
Fox 2026-06-04: "the cohost on fedora chrome has the wiggle issue when host
is messing with tabs is there any way to sync the other stream and recover
the glitches assuming it is still buffering where the low-latency version
is glitched, this means we need to delay more than what we are or some
other trick mixed in halp..." (plus: "the video would need to be slid
depending on the mode to keep it in sync. complicated but possible.")
Before: mesh audio bypassed the worklet — went straight to an <audio>
element with the native receiver's ~50ms buffer. The SFU worklet path
had 0.5s of cushion that absorbed 200ms host stalls; mesh did not, so
the same wiggle that was inaudible on SFU was clicky on mesh.
After: ONE worklet per remote uuid, source swappable in-place via the
new setWorkletStream(uuid, newStream). Both the SFU stream (cached in
sfuStreamsByPubHex) and the mesh stream contain the same publisher's
content at slightly different network delays — so disconnecting the
old source and connecting a new one to the same worklet is seamless
(the queue's 0.5s of already-buffered samples covers the transition
while the new source fills it).
Wiring:
- mesh ontrack: setWorkletStream(uuid, meshStream). Fallback to a
fresh attachAudioStreamViaWorklet if no worklet existed (rare —
only when AudioContext failed at SFU attach).
- mesh connectionState='failed': setWorkletStream(uuid, sfuCachedStream)
+ registerLipSyncAudio(pubHex, uuid, sfuReceiver). attachCachedSfuStreamFor
remains as the no-worklet fallback.
- HTTP /stream toggle still uses rampWorkletGain (separate <audio>
element path).
Lip-sync receiver rebinding ("video slid depending on the mode"):
- New sfuAudioReceivers map caches the SFU receiver per publisher.
- handleRemoteSfuTrack mic path: registerLipSyncAudio with SFU receiver
AND store it in sfuAudioReceivers cache.
- mesh ontrack: registerLipSyncAudio with the MESH receiver — the
next worklet 'buffered' message will retarget video to mesh's jbuf
+ worklet (≈ 0.55s) instead of SFU's (≈ 1s).
- mesh fail: registerLipSyncAudio back to the cached SFU receiver
→ video re-targets again.
Removes the previous gain-ramp hack for SFU↔mesh transitions —
single worklet means no parallel paths, no need to crossfade. The
gain-ramp is still used for HTTP toggle (where there genuinely are
two paths: worklet + HTTP <audio>).
CPU cost: same as before — the SFU sub PC still decodes audio for
every speaker (we just route the decoded stream to the worklet or
not). Net change is "the mesh <audio> element is gone" — small
saving.
Single mechanism handling three audible-source transitions, all click-
free via Web Audio linearRampToValueAtTime (100ms by default):
1. SFU → mesh (auto, on mesh PC ontrack):
- Previous: detachListenerStream → hard tear-down of the worklet
- Now: rampWorkletGain(uuid, 0, 100) keeps the worklet decoding in
the background; only the GainNode value moves. Free CPU cost (~3%
on a phone), zero audible click.
2. mesh → SFU (auto, on mesh PC connectionState='failed'):
- Previous: attachCachedSfuStreamFor re-attached the SFU stream
- Now: rampWorkletGain(uuid, 1, 100) restores the path that was
never disconnected. attachCachedSfuStreamFor kept as fallback
for the <audio>-element case (when AudioContext failed at attach).
3. SFU → HTTP /stream (manual, per-speaker toggle button next to mic
state); HTTP → SFU on toggle off:
- Previous: existing mute logic only touched remoteAudio (mesh
<audio> elements). Listeners use listenerAudioNodes (worklet),
so toggle-on left the worklet playing AND started HTTP — double
audio. Fox 2026-06-04: "it was breaking listeners."
- Now: startStream calls rampWorkletGain(uuid, 0, 100) and mutes
the mesh element. stopStream reverses both. Per-speaker toggle
finally works for listeners.
The mesh and HTTP <audio> elements still hard-mute via .muted (cheap,
no Web Audio path for them). Only the SFU worklet path needs the
smooth ramp because it's the one that'd produce a click if cut
mid-sample.
Telemetry will show:
- "mesh audio attached for XXXX — SFU worklet faded out" on mesh ontrack
- existing "stream on for XXXX — DJ mode" on HTTP toggle ON
- no new event on toggle OFF / mesh fail (just the gain ramp)
Fox 2026-06-04: "the cohost is hearing two feeds now."
Regression from role-aware audio routing. Pre-refactor: SFU mic and
mesh mic for the same speaker both wrote into the remoteAudio map
(both used <audio> elements), so the second arrival overwrote the
first — one audible path. Post-refactor: SFU mic for speakers /
cohosts / hosts goes through AudioContext + worklet (stored in
listenerAudioNodes), mesh mic still uses <audio> (in remoteAudio).
Different maps, both play, same voice in two different delays =
echo / phasing.
Fix: at the top of mesh pc.ontrack, call detachListenerStream(uuid)
to remove the SFU worklet path. The existing canSpeak-mesh-skip
in handleRemoteSfuTrack already prevents the OPPOSITE race (SFU mic
arriving after mesh is connected).
Logged so we can see the handoff: "mesh audio attached for XXXX —
SFU worklet path detached".
Fox 2026-06-04: "lips on the video share are not synced with the audio."
Source of the desync:
- Audio total delay = native_jbuf + worklet_buffered (because the
AudioWorklet adds its 4s cushion ON TOP of whatever the native
receiver does)
- Video total delay = native_jbuf only (no worklet downstream)
- Net: video leads audio by up to ~4s on music listeners; by less on
voice (where native honors the hint, ~3s); audible mismatch in either.
Algorithm: per-publisher dynamic match. Every ~683ms the worklet
posts {cmd:'buffered', seconds: this.buffered/sampleRate}. JS-side
handler refreshes lip-sync for that uuid:
1. Resolve uuid → publisher pubHex
2. Read audio receiver's getStats jbuf (native part)
3. audioTotal = native_jbuf + worklet_buffered
4. Push to per-publisher history (size 5)
5. Compute median (FEC-style: 3-of-5 must agree before lock-in)
6. If |median - lastApplied| > 0.05s, set every video receiver for
that publisher: playoutDelayHint = median, jitterBufferTarget = median*1000
Hamming-spirit on a control signal per fox's request: median-of-5
rejects single-sample outliers from network jitter or getStats noise.
50ms hysteresis below perception threshold so the video target
doesn't whip on tiny shifts.
Registers:
- registerLipSyncAudio(pubHex, uuid, receiver) — at mic-kind ontrack
- registerLipSyncVideo(pubHex, kind, receiver) — at screen/camera/game ontrack
- Map: pubHex → {audioUuid, audioReceiver, nativeJbufSec, videoReceivers, history, lastApplied}
The role-aware base delay (playoutDelayForRole) is still applied at
attach as a sane initial value; lip-sync then refines per-publisher
within a few seconds.
Fox 2026-06-04: "listeners state should be connecting for 4 secs while
the buffer fills, not immediately to connected."
Previous behavior: the moment the call FSM hit joined, the status text
flipped to "connected as listener" — but no audio was actually playing
yet because the AudioWorklet hadn't filled to its 4s target. User saw
"connected" but heard nothing for ~4 seconds. Confusing.
New behavior:
- listener role + buffer not yet filled → "connecting — buffering 4s
audio…" (warn-colored dot)
- first AudioWorklet started message arrives → "connected as listener"
(ok dot)
Mechanics:
1. Worklet's process() posts {cmd:'started', targetSeconds} the moment
started flips true (buffer filled to target). One per worklet per
fill cycle.
2. JS-side jbuf.port.onmessage listens; calls onWorkletStarted(uuid).
3. onWorkletStarted flips listenerBufferReady=true once (first started
wins — audio is audible from that point); subsequent worklets'
started events are no-ops for UI purposes.
4. setListenerStatusAware(role) picks the right string. Replaced every
"setStatus('connected as '+role)" call site with this helper.
5. resetListenerBufferReady() called on:
- fresh welcome with role=listener (first join)
- role-change DEMOTING to listener (prev !== 'listener' && next === 'listener')
so the next 4s fill cycle has to complete before "connected as
listener" returns.
Speakers / cohosts / hosts unaffected — their status flips immediately
because their 0.5s buffer fills in half a second; no visible "buffering"
state.
Conversation latency for speakers / cohosts / hosts; lean-back cushion
for listeners. Every receiver type updated together so audio + video
stay in sync (the existing memory rule: video must match audio's
playout delay).
Changes:
1. SPEAKER_PLAYOUT_DELAY_SEC = 0.5; playoutDelayForRole(role) — listener
gets 4s, all others get 0.5s.
2. attachListenerStreamViaAudioContext → attachAudioStreamViaWorklet
(generalized). Listener wrapper just calls into it with 4s + the
Media Session hook. Every role now routes SFU mic audio through the
AudioWorklet + per-role buffer — so a speaker hearing high-bitrate
stereo Opus music STILL gets a 0.5s cushion that Firefox's native
jitter buffer would have ignored.
3. attachSfuTrack: speakers / cohosts / hosts route through the worklet
path with 0.5s buffer; <audio>-element fallback only on
AudioContext failure.
4. SFU sub PC video + mic native receiver: jitterBufferTarget +
playoutDelayHint = playoutDelayForRole(myRole). Listener=4s, others=0.5s.
5. Mesh peer receivers: SPEAKER_PLAYOUT_DELAY_SEC always (mesh is
always peer-to-peer conversation, no role-mixed case).
6. AudioWorklet handles a {cmd:'retarget', targetSeconds} message —
recomputes targetSamples / maxSamples and shrinks the queue if the
new cap is smaller. No reconstruction needed across role changes.
7. retargetAllReceivers(role) called from onRoleChanged before mic
acquisition starts. Walks listenerAudioNodes (worklet) and
sfuSubPC.getReceivers() (native audio + video) and applies the new
target. Mesh peers stay at 0.5s unconditionally.
Speakers were previously running the same 4s setting as listeners. The
native buffer was honoring it for voice (ramping to ~3s) which meant
back-and-forth conversation was effectively impossible — they were
hearing each other 3 seconds late and didn't notice because they were
mostly publishing. This brings conversational latency back to ~500ms
while keeping the listener cushion intact.
No new SDP / signaling — all changes are receiver-side at attach. Role
transition refreshes targets on the existing PC without a renegotiation.
Fox 2026-06-04: "delay does seem to be about 4 secs but still very chappy."
Buffer IS holding 4s and emitting — that part works — but every transient
empty-queue tick (a single 2.67 ms drain) was setting started=false,
which forced a full 4-second re-fill before emit resumed. So a 50 ms
network jitter on the upstream caused a 4 s silence on the listener.
That's the chop.
Fix: track consecutive empty-queue blocks. Only re-arm (started=false)
after rearmThresholdBlocks (100 = ~267 ms) of sustained silence.
Brief drains emit silence-fill but keep started=true so playback
resumes the instant new samples arrive. Listener hears at most ~267 ms
of dead air on each drain — almost certainly Opus PLC will mask far
shorter ones.
Long outages (>267 ms with no samples) still re-buffer to 4 s — that
case isn't this bug, it's a real upstream death where a fresh
cushion is correct.
Browser-native jitterBufferTarget didn't help on the music stream —
Firefox Android holds it at 0.06–0.21s on a high-bitrate stereo Opus
receiver while honoring 4s on voice and video receivers on the same
PC. Per-codec implementation gap in the receiver-side jitter buffer.
This adds a userland buffer in Web Audio. The listener path already
ran through AudioContext (source → gain → destination); now an
AudioWorkletNode sits between source and gain, queues incoming
128-sample blocks until targetSamples (4 × sampleRate) have arrived,
then emits with a constant delay. Bounded at maxSamples (6 ×
sampleRate) so clock drift can't grow the queue unbounded. If the
queue fully drains, the buffer re-arms — a hiccup doesn't lock us
silent.
Worklet code lives inline as a Blob URL (single-file app: no
separate JS file shipped). loadJitterWorklet is fire-and-forget on
first attach; existing direct-connected streams get swapped through
the buffer the moment the worklet module finishes loading. Fallback
on worklet creation failure: existing source → gain path stays live.
Speaker / cohost / host paths untouched — they need conversational
latency, can't sit on a 4s cushion.
Listener role test plan: rejoin, watch the new "jitter-buffer
installed" log line, observe that the listener is now 4s behind the
host's speech. Wiggle the host (X11) — listeners should hear no
disruption while the buffer is full.
Fox 2026-06-04 — definitive observation: "the phone as a listener doesn't
seem to be 4 secs behind ever." playoutDelayHint is a HINT the browser
is free to ignore; Firefox Android apparently does. The phone's actual
buffer was near-zero — so every host-side encoder stall propagated
audibly to listeners with no cushion.
Three changes:
1. Set RTCRtpReceiver.jitterBufferTarget = 4000 (ms) alongside
playoutDelayHint. jitterBufferTarget is NOT a hint — it's a target
the receiver must aim for. Chromium 113+ (May 2023), Firefox 124+
(2024). Older browsers silently ignore the assignment (try/catch).
2. Applied at all THREE attach sites:
- SFU video receiver (screen/camera/game)
- SFU mic receiver
- mesh peer mic receiver
3. Add `jbuf=Xs` to telemetry, computed from inbound-rtp
jitterBufferDelay / jitterBufferEmittedCount. This is the ACTUAL
average buffer depth — we can now see whether the receiver is
holding ~4s or 0.05s. If jbuf stays small after this deploy, the
browser is ignoring the target too and we need a different
approach (AudioWorklet manual buffering, or move to HTTP-pull DJ
path for listeners).
Existing playoutDelayHint setting kept for older browsers that honor
it but don't yet support jitterBufferTarget.
Previous loop picked the max-packets outbound-rtp per PC, which always
collapsed onto the video track on screen/camera PCs — hiding what
screen-audio was doing during the wiggle test entirely.
Now iterates ALL outbound-rtp entries on every publisher PC and emits
one line per kind: scr.send.aud, scr.send.vid, cam.send.aud (when the
camera mic is published), cam.send.vid, mic.send.aud, game.send.*.
remote-inbound-rtp is paired back by ssrc.
Lets us answer the key diagnostic question: when Firefox's
getDisplayMedia video capture stalls under X11 wiggle, does the audio
track from the same MediaStream stall in lockstep (Firefox couples
audio + video producers internally) or stay flowing (decoupled)? If
decoupled, an app-layer fix (route audio + video to separate
RTCPeerConnections) would work. If coupled, the fix has to be either
OS-level (PulseAudio loopback to mic) or upstream in Firefox.
When the sender's egress queue gets contended, audio wins. Fox 2026-06-04:
wiggling a terminal window (X11) caused 1-5s audio cutouts on all mesh
peers AND listeners — even with playoutDelayHint=4.0 on subs. Theory
(fox): "i might be the clients video trying to compete with the audio,
the wiggled updates" — exactly that. X11 window wiggling produced a
storm of dirty regions for screen-share; the encoder spiked into a big
keyframe burst; audio packets queued behind that burst arrived late;
listener's playoutDelayHint buffer drained.
Fix: RTCRtpEncodingParameters.priority + networkPriority. Set audio to
'high' (mic + any DJ audio track), video to 'low' (screen, camera).
Both Firefox and Chromium honor these for the local egress queue —
audio packets jump ahead of video bursts.
Applied at:
- setSenderBitrate (mic publisher) — high
- setSenderMaxBitrate (screen audio, camera audio) — high
- setSenderMaxBitrate (screen video, camera video) — low
No new param, no SDP renegotiation needed — setParameters() applies
immediately to the existing PC. Existing call sites all flow through
these two helpers, so the change reaches every publisher.
Pair with publisher-side outbound-rtp telemetry (836a036) — gap on
aud.send during a wiggle is now the metric we measure against.
Adds aud.send/cam.send/scr.send/game.send lines to the 5s telemetry tick:
- pkt sent, bytes sent, framesEncoded (video)
- gap = now − lastPacketSentTimestamp (wire stall detector)
- rtt + rlost + rjit from remote-inbound-rtp (the SFU's view of us)
Until now we only had inbound stats on the listener path. A publisher
stall — e.g. X11 compositor blocking Firefox during a window wiggle —
was invisible at the wire; we could only infer it from downstream
listener loss/silence. Fox 2026-06-04: "wiggling a terminal window
causes 1-5s cutout on all mesh devices and listeners (4s buffer should
absorb that)." Hypothesis is a publisher-side audio-thread stall;
this lets us confirm by watching gap >> 0.02s on aud.send during a
wiggle.
No new IPC, no new dispatch — same getStats() call shape already in
use for inbound, just iterating the publisher PCs (sfuPubPC,
sfuCameraPC, sfuScreenPC, sfuGamePC) and picking outbound-rtp +
remote-inbound-rtp pairs.
The mod-action serializer (runModSerial) was waiting for case 'state' to
release the next sign — but no 'state' broadcast follows a successful
kick/ban/promote. The server bumps rm.epoch and emits the action-specific
event (peer-booted / role-change) carrying the new epoch. The client was
already updating roomEpoch in role-change but never released the queue
gate, and the peer-booted handler did neither — so kick/ban always burned
the 1500ms fallback timeout AND signed the next action with the stale
epoch (signal bounces it with "stale epoch").
Fox 2026-06-04: "couldn't kick until leaving as host" — leaving + welcome
was the only thing that refreshed roomEpoch.
Pair with signal-side fix that moves rm.epoch++ before the peer-booted
broadcast and includes "epoch" in the payload.
`muted` was a bare global mutated from 8+ sites (btn-mute click, peer-force-muted,
role promote/demote, self-listener enable/disable, leave handler, sessionStorage
restore). Each call site also had to remember to call applyMuteState() and
sendMicState(). Drift was inevitable — a recent regression where self-listener
toggle muted the wrong direction came straight from this implicit-state pile.
New shape:
- muteSpec: states { on, off }, events { TOGGLE, FORCE_MUTE, AUTO_MUTE,
AUTO_UNMUTE, RESTORE_MUTED, RESTORE_UNMUTED, ROLE_PROMOTED }, ctx.source
tracks who muted us ('self', 'mod', 'self-listener').
- `muted` is now a getter over roomMachines.mute.state — single source of truth.
- One observer drives applyMuteState + sendMicState + sessionStorage persistence
+ log line on every transition.
- Imperative call sites only dispatch events; they no longer touch side effects.
Tests: test/zebra-fsm.test.js harness updated to extract muteSpec (same brace-
matched-regex pattern as selfListenerSpec). 88/0 passing. MuteFSM-specific
transition tests are next.
Pattern is now load-bearing — call/publish/subscribe/remote-tile/self-listener/
mute all live as FSMs with the same shape.
Builds on 07d0af9 (publishSpec wired to imperative flow) — adds an
observer that derives btn-{screen,camera}-share / btn-{kind}-stop
visibility from the publishSpec state instead of from scattered
classList writes in sfuPublish/sfuUnpublish.
The state→UI mapping:
- off / failed → share button visible, stop hidden
- acquiring / negotiating / live / stopping → stop visible, share hidden
(stopping is treated as "still sharing" to avoid flicker during
teardown)
Removed four classList writes (two in sfuPublishScreen+sfuUnpublishScreen,
two in sfuPublishCamera+sfuUnpublishCamera). The buttons now reflect
the FSM rather than whoever last touched them — if the FSM
transitions because of the silent rebuild (sfuRebuildScreenPC) or
the watchPublishPC rebuild path or any future state-driver, the
buttons follow automatically.
Mic has no share/stop button (it's always on for speakers, off for
listeners), so applyPublishStateUI early-returns for that kind.
All 88 fsm + 12 self-listener tests still green.
The publishSpec FSM (off → acquiring → negotiating → live → stopping
→ off) has existed since the FSM scaffolding landed but was never
actually driven — wireZebraMachines created mic/camera/screen
instances and the imperative sfuPublish* / sfuUnpublish* never sent
events at them. So the spec was documented behavior, not enforced
behavior.
Wired all three publish paths to drive their FSMs in lock-step with
the imperative flow:
- sfuPublish (mic): START → ACQUIRED (with stream) → NEGOTIATED
(with pc + peerID) → on watchPublishPC rebuild path STOP+DONE
- sfuPublishCamera: same pattern, FAILED+DONE on getUserMedia
cancel + HTTP errors
- sfuPublishScreen: same pattern, FAILED+DONE on getDisplayMedia
cancel + HTTP errors
- sfuUnpublish / sfuUnpublishCamera / sfuUnpublishScreen: STOP +
DONE around the actual teardown
Trace observers attached for all three kinds — every transition
logs as `pub.mic: prev → next [EVENT]` so QA can see the publish
lifecycle in the page log alongside the imperative `sfu publish:
starting` lines.
The imperative state (sfuPubPC / sfuCameraPC / sfuScreenPC) remains
the source of truth for now. The FSM is a parallel view that
future migrations can hang side effects off of (e.g. a single
applyPublishStateUI() observer driving the share-button visibility
the way applyCallStateUI() drives the call chrome).
All 88 zebra-fsm tests still green (no spec changes, only call-site
additions). The pattern fox 2026-06-04 directed: every system as a
state machine, side effects ride observers.
Fox 2026-06-04 directive: "all systems need state machines." Self-
listener is already an FSM (commit f4dbc5c). Next system: the
top-line connection chrome (entry-row visibility, sec-room reveal,
dot color, leave/mute button visibility, status text, btn-enter
disabled). Previously these were scattered classList + setStatus
writes across welcome, peer-booted (self), btn-leave click, and
handleBlocked — easy to drift, every UI bug fox flagged ("dot still
green after kick", "entry row should be hidden when joined", "leave
button gone after kick") was a different leaf of this implicit
state model.
Wire applyCallStateUI(state, prev, ctx) as a roomMachines.call
observer. Single function, six branches (idle/connecting/joined/
reconnecting/leaving/booted), drives every relevant DOM toggle.
callSpec gains a `bootedAction` context field — 'kick' | 'ban' |
'blocked' | null — set by BOOTED's action so the UI observer can
render the right status ('kicked from this space' vs 'banned from
this space' vs 'blocked from this space') AND decide whether to
re-enable btn-enter (kick: yes, can re-enter; ban/blocked: no).
Call sites updated:
- handleBlocked: now passes { action: 'blocked' } in BOOTED payload
- case 'peer-booted' (self): now passes { action: m.action } so
kick vs ban propagates to the FSM
- btn-leave click: imperative chrome removed (was a 9-line
classList chain), replaced by send('LEAVE') + send('DONE') —
the observer handles the rest
- joinSpace: removed imperative btn-enter.disabled / setStatus
- case 'welcome': removed imperative dot/buttons/sec-room/row-entry
toggles — observer covers them
Tests added in test/zebra-fsm.test.js (now 88/88):
- BOOTED with action=kick → bootedAction=kick
- BOOTED with action=ban → bootedAction=ban
- BOOTED with action=blocked → bootedAction=blocked
- BOOTED with no action defaults to kick (back-compat)
- BOOTED → ACK → idle clears bootedAction
Future migrations should follow this pattern: add a state field to
the spec, hoist the imperative side effects into a switch in an
observer, leave a comment at the OLD imperative location explaining
the migration so the next reader doesn't reintroduce drift.
Asymmetric recovery between media publishers fixed. Mic + camera use
watchPublishPC for auto-rebuild on PC 'failed' — getUserMedia
constraints don't need a gesture once permission is granted, so the
client just creates a fresh PC and re-publishes. Screen share has
been the odd one out: getDisplayMedia requires a fresh user gesture,
so the previous policy was to tear everything down on PC 'failed' +
show a "tap share screen to re-share" log line.
But — the browser's "you are sharing" indicator typically stays on
across a transient network glitch; the MediaStream tracks remain in
readyState='live'. We don't need a fresh getDisplayMedia call; just
a fresh PC bound to the same stream. New sfuRebuildScreenPC() does
exactly that: tears down the dead PC (nulls sfuScreenPC first to
avoid recursive teardown via onconnectionstatechange), unpublishes
the old peer_id at the SFU, builds a fresh PC against the existing
sfuScreenStream, publishes, re-attaches the same recovery handler.
Asymmetry collapses: screen now auto-rebuilds silently IF the source
is still alive, and only falls back to the manual "share screen"
prompt when the user has actually stopped sharing (browser-native
stop, source window closed, lid closed). Mirror of the sub-PC
self-heal we shipped in 09a4516 — closes the last asymmetry between
publishers (and the last UX wart fox 2026-06-04 flagged on rejoin).
When the SFU closes the listener's sub PC server-side (e.g. via the
wedge-recovery in renegotiateLocked), the browser's
pc.connectionState transitions to 'closed', not 'failed'. The
existing handler only triggered a rebuild on 'failed' and
explicitly returned on 'closed' (under the assumption that 'closed'
== self-teardown). That assumption holds for sfuUnsubscribe (which
nulls sfuSubPC BEFORE pc.close()), but NOT for remote-driven
closes — sfuSubPC === pc is still true, the guard sees
'unexpected close', and we should re-subscribe.
Without this branch the host's sub stayed at sub=none indefinitely
after any server-driven close — every subsequent addPubToSub for
the host went into the void. Fox 2026-06-04 lost cohost camera +
screen share via this exact path after the wedge-recovery falsely
fired on his fresh subscribe (SFU fix 8c5fd65 prevents the false
positive going forward, this fix ensures the page recovers on any
legitimate server close).
Both 'failed' and 'closed' now rebuild via sfuUnsubscribe +
sfuSubscribe, gated on sfuSubPC === pc to catch only remote-driven
state changes.
Fox 2026-06-04: 'don't truncate uuids in the latency page'. Mesh
peer rows previously showed `peer abcd…wxyz` (shortHex 4+4 chars)
when no handle was known, or just `peer HANDLE` (uuid hidden) when
it was. Neither is useful for QA — same shortHex prefix can collide
across sessions, and handle alone makes it hard to correlate with
SFU/signal logs.
New format: `peer HANDLE FULL_UUID` when both known, else just
`peer FULL_UUID`. Full 32-char hex always present.
Fox 2026-06-04: 'on join the space name and enter button should be
hidden it should only show when disconnected. this prevents people
without the password or link to be able to see it on the screen.'
The rendezvous code is the join secret — anyone reading it off the
host's screen can join the space (or rejoin under a fresh identity
to bypass moderation).
Added id="row-entry" to the existing entry row, then hide it in the
'welcome' case (alongside the existing reveal of sec-room), and
restore it on:
- leave (btn-leave click → cleanup block)
- handleBlocked (banned / signal block)
- peer-booted (self) — so a kicked listener can read the code to
manually re-enter
Same hidden-utility pattern as the mute/leave buttons. No FSM yet
for the entry-row visibility — its only state is "joined? yes/no",
which is already captured by the call FSM. Future cleanup could
hang the toggle off a callState observer rather than scattered
classList writes.
Fox 2026-06-04 directive: every system should be a state machine
with unit + integration + functional test coverage. Implicit-state
defects keep biting (kicked-listener-UI-still-green, two-kick race,
cohost-toggle-kills-phone, audio-wedge-no-recovery). Starting the
formalization with the most-broken-today system: self-listener mode.
Spec (selfListenerSpec):
off ──ENABLE / TOGGLE──▶ on
on ──DISABLE / TOGGLE / UNMUTE / DEMOTED / CLEAR──▶ off
Sits next to publishSpec, subscribeSpec, callSpec, remoteTileSpec
in zebra-spaces.html. Composed by wireZebraMachines() into
roomMachines.selfListener.
UNMUTE edge encodes fox's invariant: "unmuting should seamlessly
switch them back to the now of the conversation webrtc mesh" — if
the user clicks unmute while on, they implicitly drop back to off.
Side effects (mic mute, streamMode enrolment, remoteAudio muting)
move out of enableSelfListenerMode/disableSelfListenerMode (deleted)
into runSelfListenerEnable / runSelfListenerDisable, called by an
observer attached to the FSM. Pure spec stays Node-testable; the
runtime drives the actual audio plumbing from observed transitions.
Boolean selfListenerMode flag deleted. window.selfListenerMode is
now a getter against the FSM state — single source of truth, no
drift possible. All callers (toggle-button click, mute-unmute,
peer-joined, role-demote, leave) now dispatch FSM events instead
of calling helpers directly.
Tests in test/self-listener-fsm.test.js:
- starts in off
- TOGGLE / ENABLE / DISABLE transitions
- UNMUTE drops to off (the fox-invariant)
- UNMUTE / CLEAR while off is no-op
- DEMOTED drops to off
- CLEAR drops to off
- unknown event refuses
- observer fires on real transitions with prev/state
- runtime observer skips prev===state edges
Existing test/zebra-fsm.test.js updated to extract+expose
selfListenerSpec alongside the other specs (the wireZebraMachines
extract is the integration test).
Makefile gets test-self-listener target + slot in test-all.
All test suites green:
- self-listener: 12 / 12
- zebra-fsm: 83 / 83
- mod-actions: 6 / 6
- web-protocol: 3348 / 3348
- multi-peer-mesh: 8 / 8
- video-track-removal: 18 / 18
Pairs with the signal-server listener alive-ttl exemption (c16f6c1).
When the AudioContext listener path binds a remote stream, declare
a MediaSession to the OS: metadata + playbackState=playing + no-op
play/pause action handlers.
On Android Firefox + iOS Safari this:
- keeps the tab in media-priority mode (less aggressive JS
throttling, AudioContext stays running)
- surfaces lock-screen / notification-area transport controls
- signals to the OS scheduler not to freeze this tab
Together with the server exemption, mobile listeners can tab away
to email / browser / chat and zebra-spaces audio keeps playing.
Fox 2026-06-04: 'when a listener is kicked the get the green left
states and still have a leave button even though they are out of the
room — should be gone'. The peer-booted (self) branch tore down PCs
+ mic + cleared ACTIVE_CALL_KEY but left the UI looking like a
connected listener — green dot, leave button visible, status still
saying 'connected as listener'.
Fix in the same case 'peer-booted' branch where m.uuid === myUUID:
- dot flips to 'dot warn' (hollow / amber), not 'dot ok' (green)
- status flips to 'kicked from this space' (or 'banned' if action=ban)
- btn-leave hidden + disabled (no room to leave from anymore)
- btn-mute hidden + disabled (no mic to toggle)
- btn-enter re-enabled so the user can manually rejoin if they want
Two diagnostic adds for chasing the cohost-toggle-breaks-phone bug:
1. inbound-rtp lastPacketReceivedTimestamp on every audio + video
receiver, expressed as 'lp=N.Ns ago'. Pkt-delta only shows audio
stopped on the NEXT tick (5s later); lp pins the EXACT real-time
moment RTP went silent. 'lp=0.1s' = healthy. 'lp=12s' = receiver
has been dead for 12s. Lets us tell at a glance whether the
receiver is starved or just idle.
2. Signal-event firehose. Every received signal message gets a one-
line log with type + epoch + relevant uuid/role/action. Filters
out high-rate noise (sdp-from, mic-state, spotlight, state) so the
page log stays readable. Now when phone audio breaks at 16:29:07,
we can scroll the phone's page log and see exactly which signal
events arrived in the seconds leading up.
Format: '« role-change e=12 u=ab12 role=speaker action=…'
The « marker keeps signal events visually distinct from the ·
telemetry ticks and free-form logLines.
Two regressions in the existing localStorage-restore path that forced
the host to re-pick the monitor input after every hard refresh:
1. Pre-permission Firefox returns deviceId='' for every device in
enumerateDevices(). refreshMicList ran at page load BEFORE the
entry-click gesture granted gUM, every match against the saved
micDeviceId failed, and the `else: micDeviceId = sel.value` clobber
silently reset the saved selection to ''. After that, getMic()
picked the default mic instead of the monitor.
Fix: detect the all-empty case (allEmpty) and bail out — preserve
the saved selection until a real post-permission enumerate runs.
2. Chrome (and other browsers in some configs) rotates deviceIds
across browser sessions for privacy. Saved deviceId stops matching
anything. Old code fell through to the clobber.
Fix: also save the human-readable label (e.g. "Monitor of WH-
1000XM5") and fall back to label-match when deviceId doesn't
resolve. When the label matches, refresh micDeviceId to the
current session's value + persist the new deviceId.
Two new localStorage keys: MIC_LABEL_KEY, SPK_LABEL_KEY. Change
handlers strip the 'input '/'output ' prefix from the option's
textContent before saving. Same pattern applied to both refreshMic-
List and refreshSpeakerList.
Doesn't help when the saved label also doesn't match any current
device (e.g. headphones unplugged) — sel.value defaults to first
device, same as before. But the common case fox 2026-06-04 hit
("monitor selected, hard refresh, monitor not restored, manual re-
pick needed") is now zero-tap.
Desktop keeps the zero-click auto-rejoin convenience. Mobile (Android
/ iOS) pre-fills the rendezvous code and surfaces 'click enter to
resume — CODE' as a status, requiring one tap to land in the space.
Why mobile-only block: Firefox Android (and likely iOS Safari) needs
primeAudioOnGesture's silent-oscillator wake to actually start the
AudioContext render thread, and primeAudioOnGesture only runs inside
the btn-enter click handler. An auto-rejoin that bypasses the click
leaves audioCtx suspended → listener MediaStreamSource attaches to a
dead destination → silence. Confirmed today by fox via the
ctxState=suspended telemetry on the phone.
This is the same trade fox originally took in 12df78a but scoped
better: desktop got the auto-rejoin restored in f388d64 because they
don't need the audio gesture, and now mobile pays one tap to keep
listener audio working.
Phone telemetry post-deploy showed attachListenerStreamViaAudioContext
logging ctxState=suspended despite resume() being called at entry-
click time. Firefox Android requires more than just resume() to keep
the audio render thread alive — the context goes back to suspended
the moment the gesture window closes if nothing is actively playing
through destination.
Fix: inside primeAudioOnGesture, after resume(), play a brief silent
oscillator (50ms, gain=0) through audioCtx.destination. That forces
the render thread to ACTUALLY START rather than just queueing-
pending. After this the context stays running for the session and
every MediaStreamAudioSource attached to destination plays through.
Added 'audioCtx primed state=...' log so we can confirm the state
flipped to 'running' in the entry click.
Telemetry on the Firefox Android phone listener exposed the real
mechanism: every <audio>.play() rejects with "play method is not
allowed by the user agent", the audio pool exhausts as the wedge-
recovery cycles, and every fresh element produced fails identically.
Pre-blessing the pool via SILENCE_WAV at entry click does NOT carry
over when srcObject is later swapped to a WebRTC MediaStream —
Firefox Android grants <audio>.play() engagement PER ELEMENT PER
SOURCE, and the gesture window expires before the SFU subscribe
round-trip finishes.
AudioContext has a different model: one resume() inside the user
gesture covers every MediaStreamSource subsequently connected to its
destination. No per-source re-engagement needed.
Implementation:
- attachListenerStreamViaAudioContext(uuid, stream): creates a
MediaStreamSource + GainNode, connects through to
audioCtx.destination
- detachListenerStream(uuid): disconnects on peer-left
- attachSfuTrack: if myRole === 'listener', try the AudioContext
path first; on failure (older browsers) falls through to the
existing <audio> path
- primeAudioOnGesture now CREATES audioCtx if absent (listeners
never grab a mic so it wouldn't otherwise exist) + resumes it
inside the gesture
- peer-left handler also calls detachListenerStream
Listener-only because speakers/cohosts/hosts have an active mic +
setSinkId speaker-picker requirements that still want <audio>
elements. Listeners don't pick speaker output devices and don't
talk — Web Audio is the simpler path that survives the mobile
autoplay regime.
Meter unaffected — startMeter creates its own MediaStreamSource for
analysis (multiple sources per stream is allowed in Web Audio).
Replaces the user-visible "Audio paused — tap anywhere to resume"
notice (which I added in 305c60f without remembering fox already
tried and rejected it in 6c9d1b8 → 24bfc81: "tap-anywhere added
user-visible noise").
Real mechanism fox identified: hard refresh on the HOST fixed the
phone — because the republish triggers a fresh ontrack on the phone,
which leases a NEW pool element from the pre-blessed pool with
intact autoplay engagement. Mirror that automatically without user
interaction: when the 5s telemetry tick sees a remoteAudio element
wedged (rs>=2 paused at ct=0 with a live srcObject), tear it out,
lease a fresh pool element, rebind the same MediaStream, play().
Silent recovery — no notice, no tap, no user-visible noise.
Doesn't replace the original pool blessing (still primed at entry
click), just provides a continuous self-heal whenever the
srcObject-swap kills engagement on an existing element.
Telemetry from fxhp-phone proved the bug isn't play()-promise-rejects.
Firefox Android: receiver decodes audio (aud.recv level=0.449 in the
stats) but the <audio> element stays pa=1 ct=0.00 forever. play()
returns a resolved promise on the pre-blessed pool element, but
assigning the WebRTC srcObject silently breaks the autoplay grant
without firing an error — flagAudioBlocked's promise-reject hook
never gets called.
Add a state-based trigger inside the existing 5s telemetry tick: if
any audio element has rs>=2 (HAVE_CURRENT_DATA) but is paused at
ct=0, call flagAudioBlocked() to surface the tap-anywhere notice.
Listener role only — speakers have other audio paths that shouldn't
be disturbed by a global click handler.
Re-uses the existing tap-to-resume scaffolding from b8fd50c (single
document-level click listener that fires activateListenerAudio()
inside the gesture and self-removes). One tap on the phone screen
should now recover all paused audio.
Two fixes pulled from today's QA:
1. activateListenerAudio() existed in the page but was orphaned —
never called from anywhere after a prior refactor. Wire it back
via flagAudioBlocked(): when attachSfuTrack's play() promise
rejects (Firefox Android autoplay block past the entry gesture
window), set audioPlaybackBlocked=true, show a warn notice "Audio
paused by browser. Tap anywhere to resume." and arm a single
document-level click/touchstart listener. First tap fires
activateListenerAudio() inside the gesture, retries play() on
every paused element (rtc + stream), clears the notice.
Telemetry that exposed this: phone showed
`rtc[3110] rs=4 ns=1 pa=1 ct=0.00` — element had data, network
idle, paused, never started playing. No recovery path until the
user reloaded.
Scoped — the document-click handler is armed only while the flag
is true, removed on first tap. Doesn't add noise to working
sessions (which was why the prior tap-anywhere was reverted).
2. Kick clears sessionStorage[ACTIVE_CALL_KEY]. The confirm() text
says "they can rejoin" — but that means MANUALLY (type the code,
click enter), not automatically on a hard-refresh / bfcache
restore via auto-rejoin. Fox 2026-06-04: "we have auto-join on
the phone that is kicked just rejoins". Closes that loophole;
victim can still re-enter manually.
Server increments roomEpoch on every successful mod action and the new
value rides back on the next 'state' broadcast. Two kicks fired in
rapid succession both signed with the same epoch N — first succeeds
(server now at N+1), second rejected with "stale epoch" because the
client hasn't received the state-update yet. Repro 2026-06-04: host
kicked two phones, only one was actually evicted; signal log showed
1 AUDIT + 1 /internal/evict + "signal: stale epoch" client-side.
Fix: serialize mod-action sends with a promise that resolves on the
next 'state' broadcast or 1.5s timeout. signBytes() runs AFTER the
wait so the signature uses the freshest known roomEpoch. Applied to
every mod action (invite/grant/promote/demote/mute/kick/ban) for
defense in depth — any pair of mod actions had the same race.
Adds 'mod[label] epoch=N (queue ready)' and 'mod[label] settled
epoch=N+1' breadcrumbs so the page log shows the queue draining in
real time.
While in a room every 5 seconds the page log gets a one-liner with:
- sub/pub PC connectionState + iceConnectionState
- mesh peer count, selfListenerMode, streamMode size, muted flag
- inbound-rtp audio: packetsReceived, packetsLost, bytesReceived,
jitter, audioLevel (from RTCPeerConnection.getStats)
- inbound-rtp video: packetsReceived, packetsLost, framesDecoded
- per remoteAudio + streamAudio <audio> element: readyState,
networkState, paused, muted, currentTime, MediaError code (and
src tail for streamAudio)
This lets us tell at a glance whether a "silent" listener is:
- starved (pkt count stuck → RTP not arriving)
- decoded but muted/paused (ct stuck, mu=1 / pa=1)
- network-stalled (ns=3)
- waiting on autoplay (paused=1 but rs=4)
attachSfuTrack now also logs the track's enabled/muted/readyState at
attach time and one-shot listeners on playing/pause/ended/stalled/
error per element. Pair these timelines across devices to find
exactly which leg of the chain broke when audio cuts out.
fox: "way more telemetry NOW". This is that.
The previous version skipped uuid === myUUID in the enrolment loop,
so a host alone with listeners (the common solo-DJ case) flipped the
toggle and got silence because there was nobody else to stream from.
Include self in the loop — listeners hear every speaker including
us, so "what listeners hear" must pull our own /stream too. Self has
no remoteAudio entry to mute (self-echo skip in handleRemoteSfuTrack),
so that part is a no-op for the self row.
Speakers/cohosts/hosts get a per-row stream toggle ONLY on their own
row that flips their playback for the entire room from the live
WebRTC mesh to the buffered HTTP Ogg/Opus broadcast tap. Auto-mutes
the user's mic when ON (they'd be 2-4s behind the conversation, can't
talk into the delay). Unmuting flips it OFF, restoring the live mesh.
This gives a non-WebRTC audio path that survives cellular ICE/DTLS
churn — when the mesh dies the HTTP <audio> jitter buffer keeps
serving until the user opts back into live conversation.
Side change: dropped the host-controls-other-rows variant of the
toggle (fox: each role only switches themselves). selfListenerMode
populates streamMode with every audible peer, mutes their WebRTC
remoteAudio, and pulls each via /stream?pub=PUBHEX.
Also adds breadcrumb logging to startStream (loadstart/canplay/stalled/
error code) so the next failed click leaves a trail — earlier sessions
clicked the toggle and zero /stream GETs reached the SFU; we can now
tell on which leg the fetch breaks.
