Adds a (health) op handler on the worker side & a lazy-refresh
health cache + VRAM-aware selection on the bend client side.
WORKER (gpu-worker.lsp)
(health) returns (ok (load-avg L) (vram-free-mb V) (uptime-ms U))
- L from /proc/loadavg first field
- V from `nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits`
(returns 0 when nvidia-smi missing — host w/o NVIDIA GPU)
- U from current-time-ms; client detects a worker that hung
& restarted between probes via uptime jump
Backward-compat: workers without (health) return
(error (unknown-op health)); client treats that as ok+vram=0.
CLIENT (bend.lsp)
*worker-health* alist keyed "host:port" → (last-checked-ms status vram-mb)
Cache TTL on ok = 5 s; cooldown on down = 30 s.
bend-pick-worker now:
- filters out workers in down-cooldown
- sorts healthy peers by free VRAM descending
- falls back to round-robin if every worker is in cooldown
bend-dispatch-to-gpu flips workers to down on tcp-connect-fail
or empty-reply so a transient failure costs at most one call.
Two lumbda quirks caught while building:
- (eq? 0 #f) → #t in lumbda. worker-probe-health returns 0
(a number) for the unknown-op fallback, but if we'd checked
(eq? vram #f) we would have mis-marked the worker down.
Now uses (number? vram) instead.
- tcp-connect raises a Python ConnectionRefusedError (NOT a
LispErr) on dead-host probes. lumbda's `guard` only catches
LispErr; only with-exception-handler catches Python
exceptions. Probe now wraps via with-exception-handler so a
single dead worker never aborts a fleet iteration.
Smoke on Python tier:
mixed (127.0.0.1:1 dead + 3090-ai live) → cache shows down for
the dead one (30s cooldown), ok for live (22777 MB free VRAM,
measured by the worker's nvidia-smi probe).
russell@unturf.com
58fd787ebf