lumbda/factory/README.md

factory/ — bend at scale

Bounded-parallel emit + dispatch + autoscaling for lumbda research loops driving GPU bend backends. Originally built for foxhop ecdsa secp256k1 attack-surface work (~/git/www.foxhop.net/ecdsa/), shared back upstream under AGPLv3.

What it does

Drives a queue of .lsp cells → emitted .bin payloads → dispatched to a backend (default bend-cuda) → results in a portal file. Survives crashes, OOM, transient CUDA errors via a two-tier DLQ. Adapts concurrency to live VRAM headroom + observed per-fork peak memory.

Components

• bend-supervisor.sh — process orchestrator. Launches pool + dispatcher + autoscaler + DLQ runner.
• bend-emit-pool.sh — bounded parallel emit loop. Auto-pulls LUMBDA_REPO_DIR per iter.
• bend-dispatcher.sh — singleton dispatcher. Reads supervisor.config tier caps live, dispatches .bin to backend.
• bend-autoscaler.sh — V2 live-state controller. Samples VRAM free + running forks + EWMA peak → plans per-tier caps.
• bend-supervisor-dlq-runner.sh — DLQ classifier. Auto-retries transient classes, escalates persistent to rDLQ.
• lib-tier.sh — bin-size classifier. Tier bands + HUGE solo-dispatch.
• lib-heartbeat.sh — supervisor liveness signaling.

V2 controller — live VRAM model

Rather than reserving PEAK × allocated_slots for every tier (V1's approach, prone to over-reservation under skew), V2 samples actual state:

• nvidia-smi --query-gpu=memory.free → live headroom (not fixed budget × fraction)
• pgrep -f "$LUMBDA_BACKEND_PROC_PATTERN" + nvidia-smi --query-compute-apps=used_memory → fork count + observed peak
• EWMA over observed peaks → per-worker VRAM estimate (file autoscaler.ewma)
• Zero-floor on empty tiers (no MIN=1 reservation)
• Fall-through admission (single-tier demand → grant whole budget to that tier)
• ±25 % per-poll rate-of-change damping
• DLQ growth-rate damping (halve next plan if DLQ growth > threshold)

See bend-autoscaler.sh header for signal format + operator overrides.

HUGE solo-dispatch

Cells that classify as HUGE (4-12 GB bins) get whole-card solo dispatch: when a HUGE bin is queued or in flight, all other tiers plan to 0 until HUGE completes. Damped at 25 %/poll so already-running cells finish naturally (~2 min drain).

Trigger: r_huge > 0 OR inflight_huge > 0. Release: inflight_huge = 0 AND no queued HUGE.

Two-tier DLQ + rDLQ

bend-supervisor-dlq-runner.sh polls $LUMBDA_QUEUE_DIR/dlq/ every 30s. Per entry, classifies + acts:

• Auto-retry classes (missing-bin, bisect-pool-race, cuda-error-transient, no-portal) → move .bin back to queue, .ready touched, .retries++. Cap AUTO_RETRY_MAX=3 then escalates.
• Escalate classes (cuda-oom, cuda-illegal-addr, memory-cap-refused, bin-load-fail, tier-classify, emit-broken, unknown) → move to $LUMBDA_QUEUE_DIR/rdlq/ with full state preservation. Owner inspects + patches + retries.

rDLQ state per cell:

rdlq/<tag>.bin         (preserved iff stage != emit AND salvage=yes)
rdlq/<tag>.lsp         (cell source — required for retry-reemit path)
rdlq/<tag>.reason      (original DLQ reason + classifier verdict)
rdlq/<tag>.class       (cuda-oom | bin-load-fail | ...)
rdlq/<tag>.stage       (emit | dispatch | sim)
rdlq/<tag>.retries     (cumulative)
rdlq/<tag>.first-seen  (UTC timestamp)

Cold-recovery baseline

V2 writes $LUMBDA_QUEUE_DIR/supervisor.config.baseline every poll iff total >= CPU_CEILING/4. Last known-healthy cap vector. On autoscaler crash with stale supervisor.config:

cp "$LUMBDA_QUEUE_DIR/supervisor.config.baseline" "$LUMBDA_QUEUE_DIR/supervisor.config"

then restart bend-autoscaler.sh. Baseline overwritten each healthy poll, tracks workload evolution.

Configuration

All knobs in CONTRACT.md. Defaults assume a generic lumbda dev setup; domain consumers (foxhop ecdsa, etc.) set env vars then exec.

Example consumer wrapper:

#!/bin/bash
export LUMBDA_REPO_DIR="$HOME/git/www.foxhop.net"
export LUMBDA_DOMAIN_DIR="$HOME/git/www.foxhop.net/ecdsa"
export LUMBDA_QUEUE_DIR="/tmp/ecdsa-queue"
export LUMBDA_BACKEND_CMD="$LUMBDA_DOMAIN_DIR/cuda/bend-cuda"
export LUMBDA_EMITTER_CMD="lumbda --fast $LUMBDA_DOMAIN_DIR/lumbda/emit-stream.lsp"
exec "$HOME/git/lumbda/factory/bend-supervisor.sh" "$@"

Testing

Integration tests at ~/git/lumbda/tests/integration/:

• test-bash-script-syntax.sh — bash -n lint gate. Catches the apostrophe-in-bash-c class that took the foxhop factory down 24 min on 2026-06-12.
• test-autoscaler-v2.sh — V2 reducer covering 7 defect classes (skewed-demand starve, zero-floor reservation, multi-tier greedy, ±25 % damping, cold-start ramp, DLQ surge halve, post-damp CPU ceiling).
• test-dlq-runner-classify.sh — DLQ verdict classifier reducer.

Run: make -C ~/git/lumbda test-integration.

License

AGPLv3. Originally developed at foxhop for secp256k1 attack-surface research; shared upstream as obligated by AGPLv3 share-back when the work runs as a service or against shared infrastructure.