Russell Ballestrini
86df9622d1
Fox 2026-06-05: "do we access to a whisper speech to text?" → no, then
"yes implement this and please this is perfect, make it an off be
default toggle that is part of the client. individual speakers should
ues their names and show up like a log under the video area in middle."
Implementation:
1. whisper-capture AudioWorklet — separate from the jitter-buffer
worklet. Decimates the speaker's 48 kHz mono stream to 16 kHz
(Whisper's expected sample rate), batches 5-second chunks, ships
them to the main thread via transferable Float32Array on the
port. Disabled by default; { cmd: 'start' } / { cmd: 'stop' }
from JS gate the capture.
2. transformers.js + Xenova/whisper-tiny.en lazy-loaded from
jsdelivr CDN on first transcribe-toggle ON. ~40 MB one-time
download (cached by the browser); subsequent toggles are
instant. ONNX Runtime runs entirely client-side — no audio
leaves the listener's browser.
3. Per-uuid capture lifecycle. When transcribe is on, every speaker
in listenerAudioNodes gets a parallel AudioWorkletNode that taps
their source. Recognized text appends to a transcript log with
"HH:MM:SS name: text" lines. Names resolved via
members.get(uuid).handle. Empty / placeholder transcriptions
("." "[BLANK_AUDIO]") filtered out.
4. UI:
- Toggle button "transcribe (off/on)" in the controls column,
with explainer note about model size + privacy.
- #sec-transcript section in the timeline column directly under
the spotlight. Monospace font, 14rem max-height, scrollable,
auto-scrolls to bottom unless user is reading older lines.
- .hidden until first toggle ON; subsequent toggles show/hide.
5. New-speaker hook: attachAudioStreamViaWorklet checks
transcribeEnabled and auto-installs capture for late joiners.
6. detachListenerStream tears down node.capture along with the rest.
Bandwidth: zero (model and audio never leave the device).
CPU on listener: ~5-15% per speaker during the brief inference
window every 5s, idle otherwise. Tested mentally; needs real-world
verification.
|
||
|---|---|---|
| blog | ||
| include | ||
| src | ||
| test | ||
| web | ||
| .gitignore | ||
| CLAUDE.md | ||
| Makefile | ||
| README.md | ||
zebra-report
Covert peer-to-peer chat over PulseAudio sink-input volume. Userland only. No kernel module. No network packets carry the chat content. Public domain.
Build & run
Single entry point: make. Every workflow goes through a target — never
invoke gcc or python3 by hand.
make # builds all five binaries (default target = `all`)
make test # builds and runs unit tests (pure logic, no PA needed)
make test-all # unit + integration + functional (PA + sink-inputs required)
make blog # rebuilds the static blog under web/blog/
make serve # builds blog, serves web/ on http://127.0.0.1:8765
make clean # removes binaries and generated blog output
make is the contract. If a workflow isn't a target, add the target before
adding the workflow.
Dependencies
gcc, make, pkg-config # build chain
libpulse-dev # PulseAudio client lib (pkg-config: libpulse)
python3 # blog builder + dev server
Ubuntu / Debian:
sudo apt install build-essential pkg-config libpulse-dev python3
make checks libpulse via pkg-config --cflags libpulse and
pkg-config --libs libpulse; if pkg-config can't find it, the build fails
early with a clear message.
Targets
make all — six binaries
| Binary | Source | Role |
|---|---|---|
tx |
src/tx.c |
reads stdin, transmits via volume modulation |
rx |
src/rx.c |
reads target sink-input volumes, decodes to stdout |
chat |
src/chat.c |
bidirectional tx+rx, line-based chat UI |
bt |
src/bt.c |
"battle toads" dual-channel stereo UART (2× throughput) |
carrier |
src/carrier.c |
publishes a silent PA sink so volume reads have something to read |
zebrad |
src/zebrad.c |
PA→WS introspector for web/chat.html |
All link against libpulse, librt, and libpthread. Headers come from
include/zebra.h (protocol constants) and include/modem.h (inline encode/
decode + benchmark).
make test — unit tests
test/unit pure logic, no PA, no audio
Covers: signal encoding (bit_to_vol / vol_to_bit), timing arithmetic
(ts_add_ns with overflow), baud math (baud_from_avg_ns), handshake frame
build/parse for both OFFER and READY (magic, checksum, corruption).
Currently: 84 cases pass. Required to be green before any commit.
make test-all — integration + functional
Integration and functional tests need a running PulseAudio daemon and
real sink-inputs to operate on. They are not run by make test by default.
# integration tests need ONE sink-input
ZEBRA_TEST_SINK=<sink_input_index> ./test/integration
# functional tests need TWO sink-inputs (data + ctrl channels)
ZEBRA_DATA_SINK=<idx> ZEBRA_CTRL_SINK=<idx> ./test/functional
Find sink-input indices with pactl list sink-inputs short — the first
column is the index. The test/integration benchmark sub-test reports
measured baud for the current host.
To avoid disrupting real audio applications, spawn dedicated silent sink-inputs as test targets:
paplay --raw --format=s16le --rate=44100 --channels=1 \
--stream-name=zebra-test /dev/zero &
# then use `pactl list short sink-inputs` to find this stream's index
Currently: 17/17 integration pass; 8/10 functional pass. The two functional failures are timing-bound on non-realtime kernels at the auto-negotiated baud and are not code defects (the same data path passes at fixed 50 baud).
make blog & make serve
make blog runs python3 blog/build.py, which reads Markdown sources from
blog/posts/ and writes static HTML into web/blog/. make serve rebuilds
the blog and serves web/ on port 8765 for local preview.
The chat UI (web/chat.html) is plain static HTML and works directly under
make serve — open http://127.0.0.1:8765/chat.html.
make zebrad — PulseAudio → WebSocket introspector
web/chat.html modulates audio output via Web Audio GainNode (mic stays off).
A browser tab cannot read another tab's PA state, so to close the receive
loop each peer runs zebrad. Decoded frames are forwarded over a local
WebSocket; chat.html auto-connects to ws://127.0.0.1:7777.
make zebrad
./zebrad --verbose # default: @DEFAULT_MONITOR@, port 7777
./zebrad --source <name> --port 7777
Single C file, no third-party deps beyond libpulse. Embedded WebSocket
server: SHA-1 + base64 inline, server→client binary frames only (RFC 6455
opcode 0x82); any inbound data closes the socket (browser auto-reconnects).
Bound to 127.0.0.1 only — never accessible from the LAN.
Adaptive baud: starts at ZEBRA_BAUD_HANDSHAKE (50), watches for a READY
frame, locks in the negotiated rate from its payload. Peak tracker has
2-second half-life decay so threshold adapts to room volume.
make clean
Removes:
tx rx chat bt carrier
test/unit test/integration test/functional
web/blog/index.html
web/blog/001-volume-modem/
web/blog/002-sse-chatroom/
Source files, headers, fonts, recovered binaries on disk, and committed
artifacts under web/ (other than the generated blog) are not touched.
Layout
include/ protocol constants + inline encode/decode/benchmark
src/ five binaries: tx, rx, chat, bt (battle toads), carrier
test/ unit (pure), integration (PA needed), functional (full chain)
blog/ markdown sources + python build
web/ static site: index, kernel, chat, fonts, generated blog
Makefile every workflow lives here
CLAUDE.md agent operating rules for this repo
License
Public domain. Patches gratefully accepted via merge request at
git.unturf.com/engineering/unturf/zebra-report.