docs: compress CLAUDE.md + ticket 0002 diagnosis + cross-refs

CLAUDE.md "Audio jitter buffering" section was the longest in the file
(~70 lines of restated rules). Collapse to a 5-rule list with pointers
to memory + tickets for the deep dives. The old structure restated
content that already lived in feedback_audio_jitter_buffer_strategy
and the per-defect tickets.

Telemetry + tickets sections compressed similarly. Same content,
fewer words, plus explicit pointers so a future session knows where
to look for the canonical version of each rule.

Ticket 0001 grows a Cross-references section pointing back to
CLAUDE.md + memory + the pinning test.

Ticket 0002 (firefox camera) updated with the signal-log diagnosis:
every attempt shows "The request is not allowed by the user agent or
the platform in the current context." = NotAllowedError fired
instantly, no permission prompt. Conclusion: firefox has a remembered
"Block" for the origin. picked.cam= is empty so it isn't the saved-
deviceId case. Lays out two prongs: user-side permission reset
(immediate) and code-side NotAllowedError handling (next pass) so the
page surfaces a user-friendly notice instead of just logging the error
text. Cross-refs to CLAUDE.md telemetry section + the catch site.

docs/tickets/README.md gets a "See also: CLAUDE.md" pointer up top so
the loop closes from the ticket index back to the shared context.

Memory side wired bidirectionally: chromium-decoder-anchor and
audio-jitter-buffer-strategy now point at CLAUDE.md + ticket 0001.
This commit is contained in:
Russell Ballestrini 2026-06-07 20:19:33 -04:00
parent 40de2aa9c0
commit c361a0304f
No known key found for this signature in database
4 changed files with 156 additions and 191 deletions

194
CLAUDE.md
View file

@ -246,167 +246,75 @@ structure, button wiring, CSS, the entry/orientation flow before joining a
space). For everything that lives inside an existing PC, prefer
`leave` + `enter`.
### Audio jitter buffering — userland AudioWorklet, not browser hints
### Audio pipeline — load-bearing invariants
**The browser-native jitter-buffer controls cannot be trusted for music.**
Verified 2026-06-04 with side-by-side telemetry on a Firefox Android phone,
same PeerConnection, three receivers, identical 4s target:
Five rules. Break any and audio dies in a non-obvious way. Each links
to its deep dive.
- `RTCRtpReceiver.playoutDelayHint` is spec'd as a hint — "the user agent MAY
use this." Browsers do whatever they want.
- `RTCRtpReceiver.jitterBufferTarget` is spec'd as a hard target. Honored for
voice-rate Opus and for video. **Ignored for high-bitrate stereo Opus
(256 kbps music) on Firefox Android.** The native music-stream code path
inside libwebrtc isn't wired to the new API there.
1. **Userland AudioWorklet is the real jitter cushion.** Browser
`RTCRtpReceiver.playoutDelayHint` / `jitterBufferTarget` are
honored inconsistently (ignored for high-bitrate stereo Opus on
Firefox Android). `JitterBufferProcessor` in
`web/zebra-spaces.html` queues 128-sample blocks between
`MediaStreamAudioSourceNode` and `GainNode`. Sticky-started:
tolerate ≥267 ms of empty blocks before re-arming or you re-buffer
on every 2.67 ms micro-stall.
`memory: feedback-audio-jitter-buffer-strategy`
Result: a phone listener with the spec'd 4s target had a 0.21s buffer on the
music stream. Any host-side stall (X11 window wiggle, GC pause, encoder
spike) was instantly audible.
2. **Role-aware target seconds.** Listener 4s
(`RECV_PLAYOUT_DELAY_SEC`), speaker / cohost / host 0.7s
(`SPEAKER_PLAYOUT_DELAY_SEC`). Retarget by posting
`{cmd:'retarget', targetSeconds}` to the worklet port — don't
rebuild. `retargetAllReceivers(role)`. UI gates on the worklet's
`{cmd:'started'}` message so the user isn't shown "connected"
while still buffering.
`memory: feedback-4s-playout-validated-cellular-music`
**The reliable cushion is a userland `AudioWorklet`.** See
`JitterBufferProcessor` (inline Blob URL) in `web/zebra-spaces.html`. The
worklet sits between `MediaStreamAudioSourceNode` and `GainNode`, queues
incoming 128-sample blocks, holds emission until `targetSamples` are
buffered, then emits with constant delay. The buffer cushions the music
regardless of what the native receiver does.
3. **Chromium decoder anchor is non-negotiable.** Every remote
`MediaStreamTrack` consumed by a `MediaStreamAudioSourceNode`
ALSO needs a hidden muted `<audio>` element with
`srcObject = stream`, or chromium produces silence (firefox
unaffected). Lives on the per-uuid node as `node.anchor`;
`setWorkletStream` swaps it alongside the source.
`docs/tickets/0001`, `memory: feedback-chromium-decoder-anchor`
Rules that took blood to find:
4. **SFU is the only receive-audio path.** Mesh PCs (`peers`) carry
outbound mic only; inbound mesh audio is ignored. The previous
mesh→worklet swap stalled chromium decoders at the source-swap
moment. If you reintroduce mesh receive, gate the swap on the
mesh receiver's `jitterBufferEmittedCount > 0` BEFORE
disconnecting the SFU source.
`docs/tickets/0001` (root cause section)
1. **Sticky-started is non-negotiable.** Once the buffer fills and `started`
becomes true, do NOT set `started=false` on a single empty-queue tick.
That tears down playback and forces a full re-buffer (~4s of silence) on
every 2.67ms upstream micro-stall — sounds like constant chopping.
Tolerate ~267ms of consecutive empty blocks (`emptyStreak >=
rearmThresholdBlocks=100`) before re-arming; emit silence in the interim.
5. **HTTP DJ pull (`/stream` on the SFU)** is the AudioWorklet-less
fallback. Worklet is primary; HTTP-pull is the safety net.
2. **Role-aware buffer depth.** Listener gets 4s (lean-back, latency doesn't
matter, ride out wiggle-stalls). Speaker / cohost / host get **0.7s**
(`SPEAKER_PLAYOUT_DELAY_SEC` in `web/zebra-spaces.html` — bumped from
0.5s when we removed the mesh→worklet swap; the worklet cushion now
absorbs the 200ms host-wiggle unaided on the SFU receive path).
`playoutDelayForRole(role)` returns the active target.
3. **Worklet retarget on role change**, don't rebuild. Post
`{cmd:'retarget', targetSeconds}` to the worklet's port — recomputes
targetSamples/maxSamples and shrinks the queue if smaller. Audio path
stays continuous; only the buffer depth adjusts. `retargetAllReceivers(role)`
walks every live receiver and applies the new target.
4. **UI gating on buffer-ready.** Worklet posts `{cmd:'started'}` on first
fill. Listener status text sits in "connecting — buffering 4s audio…"
until the first started message lands, then flips to "connected as
listener." Without this, users see "connected" but hear nothing for ~4s
and assume the app is broken.
5. **Bound the queue at 1.5× target** to absorb clock drift without growing
unbounded. Drop oldest on overflow.
6. **Set hints AND target anyway** at `ev.receiver.playoutDelayHint = ...`
and `ev.receiver.jitterBufferTarget = ... * 1000` — they're free, they
work where the browser implements them (video, voice mic), and they
layer cleanly with the userland worklet downstream.
7. **Audio gets `priority='high'` at the sender.** Video gets `'low'`. Stops
a screen-share keyframe burst (e.g. X11 wiggle dirty regions) from
queuing audio packets behind it. `setSenderBitrate` / `setSenderMaxBitrate`
already apply this.
**HTTP DJ pull (`/stream` on the SFU)** exists as a fallback for listeners
who don't have AudioWorklet support — natural deep buffer on the
`<audio>` preload side. Was deadlocking on `bcastMu` until 2026-06-04
(commit fixed `bcastInitCapture.Write` self-recursion). Worklet is the
primary path; HTTP-pull is the safety net.
### Chromium decoder anchor — required for remote audio tracks
**Every remote `MediaStreamTrack` consumed by a
`MediaStreamAudioSourceNode` must ALSO be attached to a hidden muted
`<audio>` element in the same page.** Without that anchor, chromium
does NOT run its WebRTC audio decoder for the track, and the source
node produces silence — even though packets arrive at full rate,
`audioCtx.state === 'running'`, and the worklet's `started` event
fires (queue filled with zeros).
Firefox does NOT have this restriction. Defects of this shape present
as **chrome-only silence with every JS-side invariant green**.
The anchor lives on the `listenerAudioNodes` entry as `node.anchor`
(see `attachAudioStreamViaWorklet`). It is muted, hidden, and
autoplays. `setWorkletStream` swaps `anchor.srcObject` alongside the
source node. `detachListenerStream` tears the anchor down.
Diagnostic signal: pavucontrol Playback shows the chrome stream
present, level meter pinned at 0 while RTP telemetry shows
`aud.recv pkt=N → N+50/sec bytes=growing level=0.000 jbuf=?` and
worklet `started` event has fired. → anchor is the load-bearing
fix. Do NOT remove it during a refactor.
Full root-cause writeup: `docs/tickets/0001-fedora-chrome-cannot-hear-speakers.md`.
Memory: `feedback-chromium-decoder-anchor`.
### Audio receive path — SFU only
**SFU is the only receive-audio path for every role**, including
speakers. Mesh PCs (`peers` map) carry our OUTBOUND mic only. Their
inbound audio is ignored (`pc.ontrack` in `connectToPeer` is a
breadcrumb log only).
This is a 2026-06-07 change. The previous mesh→worklet swap caused
chromium decoders to silently stall at the source-swap moment. The
0.5s → 0.7s `SPEAKER_PLAYOUT_DELAY_SEC` bump compensates for the lost
mesh latency benefit by giving the worklet enough cushion to absorb
the 200ms host-wiggle on the SFU path.
If you reintroduce a mesh receive path, the right shape is
verify-before-swap (poll mesh receiver stats for
`jitterBufferEmittedCount > 0` BEFORE disconnecting the SFU source).
A/V QoS: audio sender `priority='high'`, video `'low'` — keeps a
screen-share keyframe burst from queuing audio packets behind it.
Set hints + jitterBufferTarget anyway at the receiver; they're free
and layer cleanly with the worklet downstream.
`memory: feedback-av-qos-and-sync`
### Telemetry — signal-server CLIENT_LOG
Every `logLine` call in `web/zebra-spaces.html` ships to the
zebra-spaces-signal server as a `client-log` WebSocket message. The
server logs to `/var/log/zebra-spaces-signal.log` on
proxy.uncloseai.com with the publisher's pubkey + handle attached.
Diagnostic SSH (read-only):
Every `logLine` ships to `zebra-spaces-signal` as a `client-log` WS
message. Logs land at `/var/log/zebra-spaces-signal.log` on
`proxy.uncloseai.com`.
```bash
ssh -i ~/.ssh/digitalocean -p 22222 root@proxy.uncloseai.com \
'tail -2000 /var/log/zebra-spaces-signal.log | grep "actor_handle=\"HANDLE\""'
```
Per-tick (~5s) telemetry line shape:
Per-tick (~5 s) shape: `· role=… sub=… pub=… mesh=N … ctx.sink=… aud.recv … vid.recv … mesh.recv u=XXXX … mic.send.aud … cam.send.vid …`
```
· role=X sub=X/X pub=X/X mesh=N sListen=N streamMode=N muted=X xcr=X ctx.sink=…
aud.recv pkt=N lost=N bytes=N jitter=N level=N jbuf=Xs lp=Xs (per SFU sub PC audio receiver)
vid.recv pkt=N lost=N frames=N jbuf=Xs lp=Xs (per SFU sub PC video receiver)
mesh.recv u=XXXX pkt=N … level=N jbuf=Xs (per mesh PC audio receiver)
mic.send.aud / cam.send.vid / scr.send.vid / etc. (outbound stats per publish PC)
```
One-shot fingerprint on entry: `session: ua=… ctx={sr/baseLat/sinkSupp/sink} devs={out/in/cam} picked={mic/cam/spk}`
One-shot session fingerprint on entry (after `joined as X — uuid …`):
```
session: ua=chrome/desktop ctx={sr=48000 baseLat=0.0107 sinkSupp=1 sink=…}
devs={out=N in=N cam=N} picked={mic=… cam=… spk=…}
```
The signal-server endpoint backs `cors-proxy.uncloseai.com`. Per-service
`/version` is exposed at `/zebra-spaces-signal/version` /
`/zebra-spaces-sfu/version` / `/zebra-signal/version` / `/version` (each
returns the upstream Go binary's commit + build_time).
Live `/version` per service: `cors-proxy.uncloseai.com/zebra-spaces-signal/version` (and `/zebra-spaces-sfu/version`, `/zebra-signal/version`, `/version`). Each returns `{git_commit, build_time, service}` of the upstream Go binary.
### Defect tracking — docs/tickets/
Anything that takes more than 15 minutes or needs cross-browser /
cross-session repro lives as a numbered markdown file under
`docs/tickets/`. Format spec + index in `docs/tickets/README.md`. New
tickets get the next number. The resolution loop is:
1. Reproduce or grab signal-server telemetry.
2. Write a failing test in `test/` that pins the contract.
3. Fix until test passes.
4. `make stamp`, push both repos.
5. Update ticket Status to `fixed`, link commit, note what to
re-test in production.
Anything > 15 min or needing cross-browser repro → numbered ticket.
Format + index in `docs/tickets/README.md`. Loop:
telemetry → failing test → fix → `make stamp` → push both repos →
mark `fixed` with commit link.

View file

@ -40,6 +40,15 @@ Pinned by tests in `test/listener-audio-attach.test.js`:
- anchor follows the stream on setWorkletStream
- anchor torn down on detach
## Cross-references
- `../../CLAUDE.md` "Audio pipeline — load-bearing invariants" rules 3
+ 4 (anchor + always-SFU) are sourced from this ticket.
- `feedback-chromium-decoder-anchor` memory carries the
apply-everywhere version of rule 3 so a future session can't
silently undo it.
- `test/listener-audio-attach.test.js` pins the anchor contract.
## Symptom
On fedora chrome, no audio from any speaker in the room is audible. Affects

View file

@ -1,71 +1,115 @@
# 0002: fedora firefox cannot share camera
**Status:** open
**Priority:** High (feature unavailable — speakers expect to share camera)
**Surface:** fedora firefox — speaker or host attempting to publish camera
**Status:** open (diagnosis pinned; awaiting user-side action + retest with new telemetry)
**Priority:** High (feature unavailable)
**Surface:** fedora firefox — speaker / host attempting to publish camera
**First seen:** 2026-06-06
**Reporter:** fox
**Reporter:** fox (blanka identity)
## Symptom
On fedora firefox, the "share camera" button does not result in a camera
publish. Same firefox can share screen and publish mic — only camera
publish fails. Same machine + chrome publishes camera fine.
Click "share camera" on fedora firefox → instant `pub.camera: off →
acquiring → off [FAILED]` cycle. Chrome on the same machine shares
camera fine. Mic publish works (so general WebRTC + permission grants
are not all broken).
## Telemetry / Repro
## Telemetry
Not yet captured.
Every attempt in `/var/log/zebra-spaces-signal.log` produces the same
verbatim error:
Repro candidates to try:
1. Open fedora firefox → enter a room as speaker
2. Click "share camera"
3. Note whether the browser permission prompt appears
4. If granted, note whether the camera tile renders locally
5. Grab the in-page signal log + browser console output
```
click: btn-camera-share
pub.camera: off → acquiring [START]
camera open cancelled: The request is not allowed by the user agent or the platform in the current context.
pub.camera: acquiring → off [FAILED]
```
Things to specifically look for:
- `getUserMedia` errors (`NotFoundError`, `NotReadableError`,
`OverconstrainedError`, `NotAllowedError`)
- Whether `navigator.mediaDevices.enumerateDevices()` lists the camera
pre-permission (label is empty, deviceId present)
- Whether the camera deviceId saved in `localStorage` (`CAM_DEV_KEY`)
matches a device firefox still recognizes (deviceIds rotate across
sessions on chrome but should be stable on firefox)
Captured pre-deploy of the Jun 6 telemetry improvement (no `e.name`,
no `constraints:` suffix). Next attempt on the live page will surface
both, but the error text is already definitive.
## Hypothesis
Session fingerprint (latest):
```
session: ua=firefox/desktop ctx={sr=48000 baseLat=0.0000 sinkSupp=0 sink=n/a}
devs={out=1 in=2 cam=1} picked={mic=dmV7HOOY cam= spk=+tR1ptfV}
```
Candidates, none yet confirmed:
- `picked.cam=` is **empty** → no saved cameraDeviceId. Not the
OverconstrainedError + stale-id case my Jun 6 auto-recovery
handles.
- `devs.cam=1` → firefox enumerates one camera. Hardware is there.
- Error fires within the SAME SECOND as the click → **no permission
prompt is being shown**.
1. **Saved cameraDeviceId pointing at a stale / removed device.** Firefox
would `OverconstrainedError` and the page's catch logs `err`. Fix
would be: on `OverconstrainedError`, retry with `deviceId` unset and
re-prompt the picker — same shape as the mic-label-fallback at
line ~5041.
## Diagnosis
2. **Firefox v4l2 backend conflicting with chrome's hold on the camera.**
If chrome was using the camera, firefox can't open it (single-consumer
on most fedora kernels). User-visible workaround: close chrome's
camera-using tab.
Firefox's error text "The request is not allowed by the user agent or
the platform in the current context" maps to **`NotAllowedError`**.
On a fresh request, Firefox shows a permission prompt. The
instant-fail signature means Firefox has a **remembered "Block"** for
the origin (`www.unturf.com`) on camera — most likely the user (or a
previous version of the page) dismissed the prompt with "Block" +
"Remember this decision," and Firefox auto-denies forever after.
3. **Firefox MediaTrackConstraints quirk** — possibly the page's
`videoConstraints` (line ~4493) demanding a resolution / framerate
the camera doesn't support → `OverconstrainedError`. Fix: relax the
constraints to `{ ideal }` for non-essential fields.
Mic permission is independent and was granted separately, which is
why audio publish works.
4. **Permission state stuck.** Firefox's permission prompt is gated per
origin + device; a previous "block" decision can persist.
`about:permissions` would show this. Not a code issue if so.
This is NOT a code defect — the page's `getUserMedia` call is correct
and chrome accepts the identical call. But the page can still help
the user recover.
5. **PipeWire / v4l2loopback fakery on the system** — if the camera
device is actually a virtual capture (OBS Virtual Cam etc), firefox
may reject it where chrome accepts.
## Path forward
Two prongs:
### Prong 1 — user-side recovery (immediate)
1. Visit `www.unturf.com/zebra-report/` (or wherever the page is
loaded from).
2. Click the **lock icon** in the URL bar → **Connection secure**
**More information****Permissions** tab.
3. Find **Use the camera** → uncheck "Use Default" → select **Allow**
(or **Always Ask**).
4. Reload, click share camera.
Alternative: `about:preferences#privacy` → scroll to **Permissions**
**Camera → Settings…** → find the origin → change from Block /
remove the entry.
### Prong 2 — code-side help (next code pass)
When `e.name === 'NotAllowedError'`, the page should surface a
**non-technical user notice** with the recovery steps above, AND
suggest clicking the camera-icon in Firefox's URL bar (Firefox
shows that icon after a blocked permission, click → "Always Allow"
without leaving the page).
Don't auto-retry — Firefox won't change its decision until the user
acts. Show the notice, log the breadcrumb, leave the FSM in `off`.
## Fix
(none yet — needs telemetry)
Pending. After fox's user-side recovery + retest with the Jun 6
telemetry live (showing `e.name`), I'll add prong 2 to
`sfuPublishCamera`'s catch.
## Cross-references
- CLAUDE.md "Telemetry" section — how to grep the signal-server log
for these events.
- The catch site in `web/zebra-spaces.html` is `sfuPublishCamera`
(search `camera open cancelled`).
- Auto-recovery for `OverconstrainedError + saved deviceId` already
landed Jun 6; this prong covers the orthogonal `NotAllowedError`
case.
## Status notes
- **2026-06-07 (open):** ticket created. Next step is to capture the
firefox console / signal log when the share-camera button is clicked
and see which of the hypotheses fits.
- **2026-06-07:** diagnosis pinned via signal-server log grep. Five
identical instant-fail attempts across 4 sessions over 2 days. No
prompt shown → Firefox saved-block on origin. Awaiting fox
permission reset + retest.
- **2026-06-06:** Jun 6 telemetry improvement (`e.name` + constraints
suffix) deployed. Auto-recovery for the saved-deviceId case shipped
but doesn't apply here (picked.cam is empty).

View file

@ -4,6 +4,10 @@ Defect tracker for zebra-report. Anything that takes more than 15 minutes
or needs to be reproduced across browsers, devices, or sessions lives here
as a numbered markdown file.
See also: `../../CLAUDE.md` "Audio pipeline — load-bearing invariants"
and "Telemetry — signal-server CLIENT_LOG" for the shared context
every ticket here references.
## File format
Each ticket is `docs/tickets/NNNN-slug.md` where NNNN is monotonically