engineering/lumbda
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

bend.html: new Worker health heartbeat section (between Fleet & Catalog)

Browse source 
Documents the (health) op + *worker-health* cache + VRAM-ranked
pick shipped in lumbda 58fd787. Three subsections:

  - Worker side: (health) op shape with measured numbers
  - Client side: cache TTL (5 s ok / 30 s down) + ranking logic
  - Failure handling: dispatch errors flip workers down;
    with-exception-handler wraps probes so a dead peer never
    aborts a multi-worker iteration

Integrated into bend.html's fleet narrative — backward-compat
note so older workers without (health) still register as
available with vram=0.

Bullets stay technical & terse; no commit hashes, no internal
implementation quirks (eq? 0 #f / Python exception propagation
stay in our commit log).
This commit is contained in:
russell@unturf.com 2026-06-06 09:36:53 -04:00
parent 816ca9c33e
commit 4ed8d0ca04
No known key found for this signature in database
1 changed files with 30 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

30
www/bend.html
View file

@ -81,6 +81,36 @@ make gpu-worker LUMBDA=asm # smallest footprint</code></pre>
<p>Multi-host fan-out was validated at 1.6&times; aggregate throughput on small workloads, but routine round-robin against the 4090 would steal VRAM from qwen. Caller opts in explicitly when a workload justifies fan-out: <code>(bend-set-workers! '((&quot;3090-ai.foxhop.net&quot; . 9091) (&quot;ai.foxhop.net&quot; . 9092)))</code> brings the 4090 online for that call.</p>
</section>
<section id="health">
<h2>Worker health heartbeat</h2>
<p>Every dispatch goes through a lazy-refresh health probe so bend never wastes a payload on a worker that is down, overloaded, or VRAM-starved. The worker exposes a <code>(health)</code> op returning measured numbers; the client caches results & ranks healthy peers by free VRAM descending.</p>
<h3>Worker side &mdash; <code>(health)</code></h3>
<pre><code>; over the same wire as any other op:
(health)
&rarr; (ok (load-avg 0.42)
(vram-free-mb 22777)
(uptime-ms 1780752288371))</code></pre>
<ul>
<li><code>load-avg</code> &mdash; 1-minute load average from <code>/proc/loadavg</code></li>
<li><code>vram-free-mb</code> &mdash; free VRAM from <code>nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits</code>; <code>0</code> on a host without an NVIDIA GPU</li>
<li><code>uptime-ms</code> &mdash; current wallclock (millisecond), so a worker that hangs &amp; restarts between probes shows a uptime jump</li>
</ul>
<p>Backward compatible: older workers without a <code>(health)</code> handler return <code>(error (unknown-op health))</code> &mdash; the client treats that as ok with <code>vram-free-mb=0</code> so a pre-heartbeat build still ranks as available.</p>
<h3>Client side &mdash; <code>*worker-health*</code> cache</h3>
<p>An alist keyed by <code>"host:port"</code> maps each known peer to a record <code>(last-checked-ms status vram-mb)</code> where <code>status &isin; {ok, down}</code>.</p>
<ul>
<li><strong>Cache TTL on <code>ok</code>: 5 s.</strong> A fresh probe runs only when a cache entry ages past this; idle bursts never re-probe.</li>
<li><strong>Cooldown on <code>down</code>: 30 s.</strong> A worker that fails an <code>(health)</code> call or a regular dispatch gets skipped for half a minute before a re-probe, so a transient failure costs at most one call.</li>
<li><strong>VRAM-ranked pick.</strong> When more than one peer caches as <code>ok</code>, <code>bend-pick-worker</code> sorts by <code>vram-free-mb</code> descending &amp; returns the top entry. Long-running jobs route to whichever box carries the most spare GPU memory.</li>
<li><strong>Round-robin fallback.</strong> When every worker is in down-cooldown, the client falls back to plain round-robin so a recovering box still gets a real attempt rather than refusing dispatch outright.</li>
</ul>
<h3>Failure handling</h3>
<p>Mid-flight failures in <code>bend-dispatch-to-gpu</code> &mdash; <code>tcp-connect</code> raising on a connect-refused, the worker dropping a connection mid-reply, a malformed reply &mdash; flip the worker to <code>down</code>. The next pick skips it for the cooldown window. A single failed call never aborts a multi-worker iteration: probes wrap in <code>with-exception-handler</code> so a dead peer surfaces as <code>'transport-fail</code> rather than propagating up.</p>
</section>
<section id="catalog">
<h2>Form catalog</h2>
<p>A form earns a slot here only after we have published a benchmark or measured one on our hardware. "I think this would be fast" does not earn a slot &mdash; the form-status column says <code>planned</code> until numbers exist.</p>