Fox 2026-06-05: "do we access to a whisper speech to text?" → no, then
"yes implement this and please this is perfect, make it an off be
default toggle that is part of the client. individual speakers should
ues their names and show up like a log under the video area in middle."
Implementation:
1. whisper-capture AudioWorklet — separate from the jitter-buffer
worklet. Decimates the speaker's 48 kHz mono stream to 16 kHz
(Whisper's expected sample rate), batches 5-second chunks, ships
them to the main thread via transferable Float32Array on the
port. Disabled by default; { cmd: 'start' } / { cmd: 'stop' }
from JS gate the capture.
2. transformers.js + Xenova/whisper-tiny.en lazy-loaded from
jsdelivr CDN on first transcribe-toggle ON. ~40 MB one-time
download (cached by the browser); subsequent toggles are
instant. ONNX Runtime runs entirely client-side — no audio
leaves the listener's browser.
3. Per-uuid capture lifecycle. When transcribe is on, every speaker
in listenerAudioNodes gets a parallel AudioWorkletNode that taps
their source. Recognized text appends to a transcript log with
"HH:MM:SS name: text" lines. Names resolved via
members.get(uuid).handle. Empty / placeholder transcriptions
("." "[BLANK_AUDIO]") filtered out.
4. UI:
- Toggle button "transcribe (off/on)" in the controls column,
with explainer note about model size + privacy.
- #sec-transcript section in the timeline column directly under
the spotlight. Monospace font, 14rem max-height, scrollable,
auto-scrolls to bottom unless user is reading older lines.
- .hidden until first toggle ON; subsequent toggles show/hide.
5. New-speaker hook: attachAudioStreamViaWorklet checks
transcribeEnabled and auto-installs capture for late joiners.
6. detachListenerStream tears down node.capture along with the rest.
Bandwidth: zero (model and audio never leave the device).
CPU on listener: ~5-15% per speaker during the brief inference
window every 5s, idle otherwise. Tested mentally; needs real-world
verification.
Russell Ballestrini
86df9622d1