engineering/zebra-report
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

zebra-spaces: clean shrink-retarget (drop to target, skip-couple video)

Browse source 
Fox 2026-06-06 on listener-to-speaker promotion: "the music slows
down I think that is the 6% algo which is janky. maybe it's better
to speed up in that case? if we skip ahead from whatever listener
is at to speaker speed, we need to make sure the video skips ahead
the same amount or rate to keep the lips synced".

Two coupled changes in the audio path:

1. JitterBufferProcessor retarget handler — on a SHRINKING retarget
   (e.g. 4s → 0.5s) drop the queue down to targetSamples directly
   instead of maxSamples (1.5×target). The 0.25s overhang the old
   code left behind triggered a 6%-cap stretchFactor adjustment that
   played at 1.064× for ~4 seconds — the residual phase fox heard
   as "janky". Now: instant skip to the new target, then normal
   playback. The grow-direction path is unchanged.

   Worklet also reports the dropped sample count back to JS via
   {cmd:'dropped', samples}.

2. installJitterBuffer.jbuf.port.onmessage — on 'dropped' from the
   worklet, walk the publisher's lipSync.videoReceivers and set
   jitterBufferTarget = 0 + playoutDelayHint = 0 on every one. The
   browser drops video frames aggressively to converge on the new
   target. Without this, audio jumps 3.25s forward instantly and
   video drains gradually = broken lip-sync for the duration of the
   native video jbuf's drain. Clearing ls.lastApplied lets the next
   refreshLipSyncForUuid actually re-apply the role-appropriate
   target instead of stopping at the threshold check.

Tests:

  test/jitter-buffer-worklet.test.js (NEW, 8 assertions) — extracts
  the inline JITTER_BUFFER_WORKLET_CODE template literal and runs
  the processor in a Node sandbox with stubbed sampleRate +
  registerProcessor + AudioWorkletProcessor. Pins:
    - 'started' fires once on first fill
    - 'buffered' reports depth
    - retarget SHRINK drops to targetSamples (not maxSamples) and
      posts 'dropped' with sample count
    - retarget SHRINK does NOT leave 1.5×target overhang (the
      residual that caused the janky 6% phase)
    - retarget GROW does NOT drop
    - retarget to same target is no-op
    - bogus targetSeconds (NaN/0/negative) ignored
    - lock_rate pins stretchFactor

  test/listener-audio-attach.test.js (extended, +8 assertions):
    - worklet 'dropped' handler zeros every paired video receiver
      target (defensive against missing lipSync entry; correct when
      seeded)
    - speaker mesh+SFU collision tests (single-chain invariant):
      mesh state=connected → SFU skip
      mesh state=failed → SFU takes over
      SFU first then mesh → in-place swap, no rebuild
      mesh first then SFU(connected) → SFU skip, one chain
      rapid mesh re-ontracks → still one gain feeds destination
      SFU cache stays primed even when mesh wins the race

Makefile: new test-jitter-worklet target, added to test-all chain.
This commit is contained in:
Russell Ballestrini 2026-06-06 14:55:03 -04:00
parent 5e8cdadb29
commit 9215eaa3f8
No known key found for this signature in database
4 changed files with 518 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

66
web/zebra-spaces.html
View file

@ -1878,14 +1878,37 @@ class JitterBufferProcessor extends AudioWorkletProcessor {
if (e.data.cmd === 'retarget'){
const t = +e.data.targetSeconds;
if (!isFinite(t) || t <= 0) return;
const prevTargetSamples = this.targetSamples;
this.targetSeconds = t;
this.maxSeconds = t * 1.5;
this.targetSamples = Math.round(this.targetSeconds * sampleRate);
this.maxSamples = Math.round(this.maxSeconds * sampleRate);
while (this.buffered > this.maxSamples && this.queue.length > 0){
/* On a SHRINKING retarget (listener-to-speaker promotion: 4s
* → 0.5s), drop down to targetSamples directly instead of
* maxSamples. The maxSamples threshold leaves a 1.5× target
* overhang that triggers an audible 6%-cap speed-up phase
* lasting ~4 seconds — what fox 2026-06-06 called "janky"
* during promotion. One hard skip-forward, then normal
* playback — no residual adjustment.
*
* Report the skip back to JS so paired video receivers can
* advance the same amount: an audio jump without a matching
* video jump = broken lip-sync for the duration of the
* native video jbuf's gradual drain. fox 2026-06-06: "if we
* skip ahead from whatever listener is at to speaker speed,
* we need to make sure the video skips ahead the same amount
* or rate to keep the lips synced". */
const shrinking = this.targetSamples < prevTargetSamples;
const dropTo = shrinking ? this.targetSamples : this.maxSamples;
let skipped = 0;
while (this.buffered > dropTo && this.queue.length > 0){
const drop = this.queue.shift();
this.buffered -= drop[0].length;
this.dropped += drop[0].length;
skipped += drop[0].length;
}
if (skipped > 0){
try { this.port.postMessage({ cmd: 'dropped', samples: skipped }); } catch(_){}
}
} else if (e.data.cmd === 'lock_rate'){
/* listeners get this — explicitly forbid time-stretching so
@ -2542,6 +2565,43 @@ function installJitterBuffer(uuid, node){
* enough to catch a wiggle within ~2s of it starting,
* vs the 5s tick path which can be 5-10s late. */
ddNoteWorkletBuffered(uuid, e.data.seconds);
} else if (e.data.cmd === 'dropped'){
/* The worklet just skipped audio forward (shrinking retarget
* — listener→speaker promotion drops 4s→0.5s buffer). Drag
* the paired video receivers forward by the same amount so
* lip-sync survives the jump. fox 2026-06-06: "if we skip
* ahead from whatever listener is at to speaker speed, we
* need to make sure the video skips ahead the same amount
* or rate to keep the lips synced".
*
* No native "skip ahead" API exists for RTCRtpReceiver, but
* jitterBufferTarget is a hard target the browser converges
* to. Setting it to 0 forces aggressive frame-drop until the
* native video jbuf drains; the next refreshLipSyncForUuid
* 'buffered' tick (~2s) restores the proper role-appropriate
* target. Net: audio jumps instantly + video jumps almost
* instantly (browser frame-drop is fast) = lip-sync stays
* within ~tens of ms. */
try {
const mm = members.get(uuid);
const pubHex = mm && mm.pubkey ? hex(unb64(mm.pubkey)) : null;
if (pubHex){
const ls = lipSync.get(pubHex);
if (ls && ls.videoReceivers){
for (const [, rx] of ls.videoReceivers){
try { rx.jitterBufferTarget = 0; } catch(_){}
try { rx.playoutDelayHint = 0; } catch(_){}
}
/* clear lastApplied so the next refreshLipSyncForUuid
* actually re-applies the role's target — without this
* the threshold check could see "no significant change"
* and leave video stuck at 0. */
ls.lastApplied = 0;
}
}
const skippedSec = e.data.samples / (audioCtx ? audioCtx.sampleRate : 48000);
logLine('', 'jitter-buffer skipped '+skippedSec.toFixed(2)+'s uuid='+uuid.slice(0,4)+' — video re-targeting');
} catch(_){}
}
};
/* All roles now allow time-stretching — listeners benefit too
@ -7904,8 +7964,8 @@ logLine('', 'ready — pick a handle, type a rendezvous code, enter the space');
<footer style="margin:2.2rem auto 0;font-size:0.65rem;color:#999;line-height:1.7;word-break:break-all;font-family:monospace">
<span id="pi-seal" style="color:#777;cursor:default;user-select:none" title="">page integrity</span> &nbsp;·&nbsp; built <span class="stamp-date">2026-06-06</span><br>
md5 <span class="stamp-md5">973996244966a40f7370e23bec5dcbb5</span><br>
sha256 <span class="stamp-sha">66f3d4eebd028f7ea16ea58df624ed0d2223505ff5c99715730379757e613630</span><br>
md5 <span class="stamp-md5">bd769bd46730955722b172d51f93e673</span><br>
sha256 <span class="stamp-sha">a8f6964a1f1e0c87131b1df57a635659fdbbb215a29c7b0032997eb8f6bc5cfc</span><br>
<span style="color:#bbb">hashes are of this page with these two fields zeroed — to verify, blank them and re-hash</span><br>
<span style="color:#bbb">one self-contained file — <strong>save a copy</strong> and verify against these hashes; point at your own servers with ?signal= and ?turncred=, or <a href="host-your-own.html" style="color:#999">host your own community</a></span>
</footer>