Multi-host fan-out was validated at 1.6× aggregate throughput on small workloads, but routine round-robin against the 4090 would steal VRAM from qwen. Caller opts in explicitly when a workload justifies fan-out: (bend-set-workers! '(("3090-ai.foxhop.net" . 9091) ("ai.foxhop.net" . 9092))) brings the 4090 online for that call.
Every dispatch goes through a lazy-refresh health probe so bend never wastes a payload on a worker that is down, overloaded, or VRAM-starved. The worker exposes a (health) op returning measured numbers; the client caches results & ranks healthy peers by free VRAM descending.
(health); over the same wire as any other op:
+(health)
+→ (ok (load-avg 0.42)
+ (vram-free-mb 22777)
+ (uptime-ms 1780752288371))load-avg — 1-minute load average from /proc/loadavgvram-free-mb — free VRAM from nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits; 0 on a host without an NVIDIA GPUuptime-ms — current wallclock (millisecond), so a worker that hangs & restarts between probes shows a uptime jumpBackward compatible: older workers without a (health) handler return (error (unknown-op health)) — the client treats that as ok with vram-free-mb=0 so a pre-heartbeat build still ranks as available.
*worker-health* cacheAn alist keyed by "host:port" maps each known peer to a record (last-checked-ms status vram-mb) where status ∈ {ok, down}.
ok: 5 s. A fresh probe runs only when a cache entry ages past this; idle bursts never re-probe.down: 30 s. A worker that fails an (health) call or a regular dispatch gets skipped for half a minute before a re-probe, so a transient failure costs at most one call.ok, bend-pick-worker sorts by vram-free-mb descending & returns the top entry. Long-running jobs route to whichever box carries the most spare GPU memory.Mid-flight failures in bend-dispatch-to-gpu — tcp-connect raising on a connect-refused, the worker dropping a connection mid-reply, a malformed reply — flip the worker to down. The next pick skips it for the cooldown window. A single failed call never aborts a multi-worker iteration: probes wrap in with-exception-handler so a dead peer surfaces as 'transport-fail rather than propagating up.
A form earns a slot here only after we have published a benchmark or measured one on our hardware. "I think this would be fast" does not earn a slot — the form-status column says planned until numbers exist.