Russell Ballestrini
c361a0304f
CLAUDE.md "Audio jitter buffering" section was the longest in the file (~70 lines of restated rules). Collapse to a 5-rule list with pointers to memory + tickets for the deep dives. The old structure restated content that already lived in feedback_audio_jitter_buffer_strategy and the per-defect tickets. Telemetry + tickets sections compressed similarly. Same content, fewer words, plus explicit pointers so a future session knows where to look for the canonical version of each rule. Ticket 0001 grows a Cross-references section pointing back to CLAUDE.md + memory + the pinning test. Ticket 0002 (firefox camera) updated with the signal-log diagnosis: every attempt shows "The request is not allowed by the user agent or the platform in the current context." = NotAllowedError fired instantly, no permission prompt. Conclusion: firefox has a remembered "Block" for the origin. picked.cam= is empty so it isn't the saved- deviceId case. Lays out two prongs: user-side permission reset (immediate) and code-side NotAllowedError handling (next pass) so the page surfaces a user-friendly notice instead of just logging the error text. Cross-refs to CLAUDE.md telemetry section + the catch site. docs/tickets/README.md gets a "See also: CLAUDE.md" pointer up top so the loop closes from the ticket index back to the shared context. Memory side wired bidirectionally: chromium-decoder-anchor and audio-jitter-buffer-strategy now point at CLAUDE.md + ticket 0001.
320 lines
15 KiB
Markdown
320 lines
15 KiB
Markdown
# Agent Blackops
|
||
|
||
This repo is operated by **agent blackops** — ml agent for fox/timehexon on the unsandbox/unturf/permacomputer platform.
|
||
|
||
## Identity
|
||
|
||
Full shard: `~/git/unsandbox.com/blackops/BLACKOPS.md`
|
||
|
||
## Rules
|
||
|
||
- I propose, fox decides. Unsure = ask. Can't ask = stop.
|
||
- No autonomous ops decisions. No destructive commands without explicit instruction.
|
||
- Fail-closed. Cleanup crew, not demolition.
|
||
- Check the time every session. Gaps are information.
|
||
- DRY in context — single source of truth, no sprawl.
|
||
- Never say "AI" — always say "machine learning."
|
||
- Prefer "defect" over "bug."
|
||
|
||
## Orientation
|
||
|
||
```bash
|
||
date -u
|
||
pwd
|
||
git log --oneline -5
|
||
git status
|
||
```
|
||
|
||
Then ask fox what the mission is.
|
||
|
||
## Zebra Report System
|
||
|
||
**Concept**: covert bidirectional communication channel using browser tab volume as the modulation medium — dial-up modem principles, userland only, no kernel involvement, no network stack.
|
||
|
||
### Collaborators & Stakeholders
|
||
|
||
| Handle | Role |
|
||
|--------|------|
|
||
| **foxhop** | fox — handler, operator, TimeHexOn |
|
||
| **brackishbert** | collaborator |
|
||
| **SEW** | collaborator |
|
||
| **russell@unturf** | Russell Ballestrini — unturf founder, permacomputer manifesto, `ago` library |
|
||
| **TimeHexOn** | oracle platform — primary deployment target |
|
||
| **groupr** | related project |
|
||
|
||
### How it works
|
||
|
||
PulseAudio exposes each browser tab as a separate sink input, visible and controllable in `pavucontrol`. Volume is settable per-tab in userland with no kernel involvement. Each tab has a range of **0–100** (101 discrete levels — 101 dalmatians).
|
||
|
||
By modulating volume at a consistent rate (bauds), two sides can exchange data:
|
||
|
||
- **transmitter**: steps volume through values at a fixed clock rate
|
||
- **receiver**: reads volume at the same clock rate, decodes the steps back to data
|
||
- **bidirectional**: two tabs (or two processes watching different tabs) run opposite directions simultaneously
|
||
|
||
### Signal space
|
||
|
||
- 101 levels = ~6.66 bits per symbol
|
||
- practical: use power-of-2 subsets — 2 levels (1 bit), 4 levels (2 bits), 64 levels (6 bits)
|
||
- higher symbol depth trades noise margin for throughput
|
||
- low baud rate = high reliability, low throughput (like 300 baud dialup)
|
||
- high baud rate = races PulseAudio update latency
|
||
- measured ceiling on neoblanka: ~1000–1200 baud (PA IPC ~350–400µs avg)
|
||
|
||
### Binaries
|
||
|
||
| Binary | Description |
|
||
|--------|-------------|
|
||
| `tx` | transmitter — reads stdin, modulates tab volume |
|
||
| `rx` | receiver — reads tab volume, writes decoded bytes to stdout |
|
||
| `chat` | bidirectional chat — two tabs, two threads |
|
||
| `bt` | **Battle Toads** — stereo dual-channel, 2x bandwidth |
|
||
|
||
### Project Battle Toads
|
||
|
||
One stereo browser tab carries **two independent UART streams** simultaneously — L channel and R channel. PulseAudio's `pa_cvolume` is per-channel; a single `get_sink_input_info` call returns both L and R volumes.
|
||
|
||
- TX sets L and R to independent bit values each symbol
|
||
- RX decodes L and R from a single PA poll — no extra IPC cost
|
||
- Net: 2x throughput at same baud rate, same PA polling budget
|
||
- Web carrier upgraded to stereo: two oscillators (440Hz L, 441Hz R) merged into a stereo stream → PA sees `channels=2`
|
||
|
||
```bash
|
||
# After opening web/index.html and clicking 'start audio' (stereo tab):
|
||
./bt -T MY_SINK -R THEIR_SINK -b 500
|
||
```
|
||
|
||
### Auto-negotiate (handshake protocol)
|
||
|
||
RX benchmarks its own PA polling speed and signals the max safe baud to TX. No manual baud matching needed.
|
||
|
||
```bash
|
||
./rx -s RX_SINK -t TX_SINK # RX benchmarks, sends offer at 50 baud
|
||
./tx -s TX_SINK -r RX_SINK # TX listens for offer, locks to RX's rate
|
||
```
|
||
|
||
Handshake frame: `[0x5A 0x42 0x01 baud_lo baud_hi xor_cksum]` — 6 bytes at 50 baud (~1.2s).
|
||
|
||
**Known defect**: 3-way handshake not yet implemented. TX can fire before RX enters receive loop at high baud rates. Fix: RX-ready signal back to TX before data phase.
|
||
|
||
### Tools
|
||
|
||
- `pactl set-sink-input-volume` — set volume by sink-input index
|
||
- `pactl list sink-inputs` — enumerate tabs, read current volume
|
||
- `pavucontrol` — visual verification of modulation
|
||
- `./tx -l` — list all PA sink inputs with index, volume, channels
|
||
- sink-input index maps to tab; stable within a session
|
||
|
||
### Use cases
|
||
|
||
- agent-to-agent signaling without touching the filesystem or network stack
|
||
- side-channel between sandboxed browser tab and host process
|
||
- low-bandwidth status heartbeat (alive/dead/mode) at ~1–10 baud
|
||
- covert channel for oracle↔host communication on TimeHexOn
|
||
|
||
### Constraints
|
||
|
||
- sink-input index resets when tab navigates or crashes — handshake needed on reconnect
|
||
- PA polling latency sets the baud ceiling — benchmark with `./rx -s SINK -t SINK2` before sending
|
||
- stereo (channels=2) required for Battle Toads — open web/index.html, click 'start audio'
|
||
- userland only — survives without root
|
||
- **Operation Voyeur**: all terminal output is public — never pass secrets through these channels unencrypted. The web page does ECDH key exchange + AES-256-GCM before TX.
|
||
|
||
### Web UI privacy — never display peer IPs
|
||
|
||
`chat.html` and `zebra-audio.html` must never print peer IP addresses or ports
|
||
in the page UI or in any visible log. Our users do not run Wireshark — if it
|
||
is not on the screen, peers cannot dox each other. Candidate types
|
||
(`host`/`srflx`/`relay`) from `pc.getStats()` are abstract and fine to show
|
||
(they tell you direct vs relayed); `loc.address` / `loc.port` /
|
||
`rem.address` / `rem.port` are not. The WebRTC stack already obfuscates host
|
||
candidates via mDNS by default — do not undo that work in the UI.
|
||
|
||
### CSS layout — grid only, no flex
|
||
|
||
All page layout on every zebra page is **CSS grid**. No `display: flex` for
|
||
layout. Reasons we settled on this:
|
||
|
||
- Grid lets us pin children to explicit columns (`grid-column: 1/2/3`) so
|
||
a `display:none` on one child never causes siblings to slide into its
|
||
slot. Flex auto-reorders, grid does not.
|
||
- One mental model for both axes. Flex needs rules per row + per item,
|
||
grid expresses the same intent in one `grid-template-*` block.
|
||
- A global `.hidden { display: none !important }` utility lives in the
|
||
page CSS — combined with explicit grid placement it produces a layout
|
||
that survives any child being toggled in or out.
|
||
|
||
When refactoring or adding UI, use:
|
||
|
||
- `display: grid` + `grid-template-columns` for column layout
|
||
- `grid-column: N` on every child of a grid so its position is explicit
|
||
- `grid-template-rows` + `grid-row` for vertical placement when needed
|
||
- `grid-template-areas` for small named-region layouts
|
||
- `gap` for spacing (instead of margins)
|
||
|
||
Avoid:
|
||
|
||
- `display: flex` on any container that arranges multiple elements
|
||
horizontally or vertically as part of the page layout
|
||
- Implicit positioning that relies on DOM order — always set
|
||
`grid-column` (and `grid-row` if relevant) on every grid child
|
||
- `flex-basis` / `flex-grow` mathematics — `1fr` is the equivalent and
|
||
reads cleaner
|
||
|
||
If you need a one-off horizontal alignment of two short inline things
|
||
(e.g. a label + a value), grid still works fine
|
||
(`grid-template-columns: auto 1fr`). Don't reach for flex.
|
||
|
||
### Web style guide — form-row patterns
|
||
|
||
`.row` is the single primitive for every horizontal form strip in every
|
||
zebra page. Don't invent new wrappers. CSS auto-detects the row shape via
|
||
`:has()` and picks the right grid template.
|
||
|
||
Supported row shapes (DOM order matters):
|
||
|
||
| Shape | Template applied | Use when |
|
||
|---|---|---|
|
||
| `<button> <button> ...` (no input) | `grid-auto-columns: max-content` (default — pack) | "stop sharing" + "stop camera" + select-camera; vault buttons |
|
||
| `<label> <input>` or `<label> <select>` | `auto 1fr` (input cell stretches) | handle row, mic-input row |
|
||
| `<label> <input> <button>` | `auto 1fr` + extra cells packed | rare; same template as above + a trailing packed cell |
|
||
| `<input> <button>` (input first) | `1fr` (input fills, button packs after) | rendezvous code + enter; password + export; share URL + copy |
|
||
| anything + `<span class="note">` | the `.note` is auto-placed on its own row beneath via `grid-column: 1 / -1` | descriptive subtext after buttons |
|
||
|
||
Rules:
|
||
|
||
- Never put inline `style="flex:1"` on a row child. Use `.row` and trust
|
||
the template selector.
|
||
- Don't add a stretchy `<div>` to fake spacing. If you need a button
|
||
group on the right, append the buttons as siblings — they'll pack
|
||
right of the stretchy cell.
|
||
- A `<span class="note">` inside `.row` *always* drops to its own line.
|
||
If you want note text on the same line as a button, use a different
|
||
class (e.g. inline `<span>`).
|
||
- Long checkbox labels get `white-space: normal` automatically, so a
|
||
music-mode-style "raw mic, no echo/noise cancellation (for playing
|
||
audio through it)" wraps cleanly inside its column.
|
||
|
||
If a new row shape doesn't fit the patterns above, add the case to this
|
||
table and add the matching `:has()` selector — don't reach for inline
|
||
styles or flex.
|
||
|
||
### Web page integrity stamping
|
||
|
||
Each deployed page (`web/chat.html`, `web/zebra-audio.html`, `web/how-it-works.html`,
|
||
`web/host-your-own.html`) carries a footer with the build date + its own MD5 + SHA-256.
|
||
**Run `make stamp` before deploying any page change** — it sets today's date and
|
||
recomputes the hashes (`web/stamp.js`).
|
||
|
||
- A file can't hold its own hash, so the hashes are computed with the two hash
|
||
fields **zeroed**, then written back (same length). Self-consistent and
|
||
idempotent: re-running `make stamp` gives identical hashes unless the content
|
||
changed.
|
||
- The stamp is **static HTML written at build time** — no JavaScript computes or
|
||
injects it in the browser. `stamp.js` is build tooling, never loaded by a page.
|
||
- **Verify a served page**: blank the md5 field to 32 zeros and the sha256 field
|
||
to 64 zeros, then re-hash with `sha256sum`/`md5sum` — must match the footer.
|
||
- **Deploy = push to BOTH repos.** A page change is not live until it lands in
|
||
both. Pushing only the source changes nothing served; pushing only the deploy
|
||
repo orphans the source of truth. Both, every time:
|
||
1. **source** — this repo (`zebra-report`): edit `web/*.html` → `make stamp` →
|
||
commit → **push to origin**.
|
||
2. **served** — `www.unturf.com`: copy the page(s) into
|
||
`~/git/www.unturf.com/zebra-report/` (`chat.html` → `index.html`;
|
||
`zebra-audio.html`, `how-it-works.html`, `host-your-own.html` keep their
|
||
names) → commit → **push to origin**.
|
||
|
||
### Picking up a new SDP/codec deploy — `leave` then `enter`, no full reload
|
||
|
||
When a deploy changes SDP munging (`preferStereoOpus`, codec fmtp params,
|
||
RTCP feedback negotiation, or the SFU's codec registration), the page-level
|
||
JS update is not enough on its own. Every existing `RTCPeerConnection`
|
||
(`sfuPubPC`, `sfuSubPC`, `sfuScreenPC`, `sfuCameraPC`, every mesh peer) is
|
||
locked to whatever was negotiated when it was created — its codec params,
|
||
its rtcp-fb, its stereo flag, its NACK behaviour. The PC won't re-negotiate
|
||
those on its own, and the new client code can't retroactively rewrite the
|
||
old SDP.
|
||
|
||
So after the new build is live, the user does **not** need a full tab reload:
|
||
|
||
- **`leave` then `enter` the space.** That tears down every PC and rebuilds
|
||
them, so the next offer/answer round trip is the new client talking to
|
||
the new SFU with the new params. Fresh negotiation, all changes active.
|
||
|
||
Full reload is only required for changes to the page shell itself (DOM
|
||
structure, button wiring, CSS, the entry/orientation flow before joining a
|
||
space). For everything that lives inside an existing PC, prefer
|
||
`leave` + `enter`.
|
||
|
||
### Audio pipeline — load-bearing invariants
|
||
|
||
Five rules. Break any and audio dies in a non-obvious way. Each links
|
||
to its deep dive.
|
||
|
||
1. **Userland AudioWorklet is the real jitter cushion.** Browser
|
||
`RTCRtpReceiver.playoutDelayHint` / `jitterBufferTarget` are
|
||
honored inconsistently (ignored for high-bitrate stereo Opus on
|
||
Firefox Android). `JitterBufferProcessor` in
|
||
`web/zebra-spaces.html` queues 128-sample blocks between
|
||
`MediaStreamAudioSourceNode` and `GainNode`. Sticky-started:
|
||
tolerate ≥267 ms of empty blocks before re-arming or you re-buffer
|
||
on every 2.67 ms micro-stall.
|
||
→ `memory: feedback-audio-jitter-buffer-strategy`
|
||
|
||
2. **Role-aware target seconds.** Listener 4s
|
||
(`RECV_PLAYOUT_DELAY_SEC`), speaker / cohost / host 0.7s
|
||
(`SPEAKER_PLAYOUT_DELAY_SEC`). Retarget by posting
|
||
`{cmd:'retarget', targetSeconds}` to the worklet port — don't
|
||
rebuild. `retargetAllReceivers(role)`. UI gates on the worklet's
|
||
`{cmd:'started'}` message so the user isn't shown "connected"
|
||
while still buffering.
|
||
→ `memory: feedback-4s-playout-validated-cellular-music`
|
||
|
||
3. **Chromium decoder anchor is non-negotiable.** Every remote
|
||
`MediaStreamTrack` consumed by a `MediaStreamAudioSourceNode`
|
||
ALSO needs a hidden muted `<audio>` element with
|
||
`srcObject = stream`, or chromium produces silence (firefox
|
||
unaffected). Lives on the per-uuid node as `node.anchor`;
|
||
`setWorkletStream` swaps it alongside the source.
|
||
→ `docs/tickets/0001`, `memory: feedback-chromium-decoder-anchor`
|
||
|
||
4. **SFU is the only receive-audio path.** Mesh PCs (`peers`) carry
|
||
outbound mic only; inbound mesh audio is ignored. The previous
|
||
mesh→worklet swap stalled chromium decoders at the source-swap
|
||
moment. If you reintroduce mesh receive, gate the swap on the
|
||
mesh receiver's `jitterBufferEmittedCount > 0` BEFORE
|
||
disconnecting the SFU source.
|
||
→ `docs/tickets/0001` (root cause section)
|
||
|
||
5. **HTTP DJ pull (`/stream` on the SFU)** is the AudioWorklet-less
|
||
fallback. Worklet is primary; HTTP-pull is the safety net.
|
||
|
||
A/V QoS: audio sender `priority='high'`, video `'low'` — keeps a
|
||
screen-share keyframe burst from queuing audio packets behind it.
|
||
Set hints + jitterBufferTarget anyway at the receiver; they're free
|
||
and layer cleanly with the worklet downstream.
|
||
→ `memory: feedback-av-qos-and-sync`
|
||
|
||
### Telemetry — signal-server CLIENT_LOG
|
||
|
||
Every `logLine` ships to `zebra-spaces-signal` as a `client-log` WS
|
||
message. Logs land at `/var/log/zebra-spaces-signal.log` on
|
||
`proxy.uncloseai.com`.
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/digitalocean -p 22222 root@proxy.uncloseai.com \
|
||
'tail -2000 /var/log/zebra-spaces-signal.log | grep "actor_handle=\"HANDLE\""'
|
||
```
|
||
|
||
Per-tick (~5 s) shape: `· role=… sub=… pub=… mesh=N … ctx.sink=… aud.recv … vid.recv … mesh.recv u=XXXX … mic.send.aud … cam.send.vid …`
|
||
|
||
One-shot fingerprint on entry: `session: ua=… ctx={sr/baseLat/sinkSupp/sink} devs={out/in/cam} picked={mic/cam/spk}`
|
||
|
||
Live `/version` per service: `cors-proxy.uncloseai.com/zebra-spaces-signal/version` (and `/zebra-spaces-sfu/version`, `/zebra-signal/version`, `/version`). Each returns `{git_commit, build_time, service}` of the upstream Go binary.
|
||
|
||
### Defect tracking — docs/tickets/
|
||
|
||
Anything > 15 min or needing cross-browser repro → numbered ticket.
|
||
Format + index in `docs/tickets/README.md`. Loop:
|
||
telemetry → failing test → fix → `make stamp` → push both repos →
|
||
mark `fixed` with commit link.
|