# factory/ — bend at scale

Bounded-parallel emit + dispatch + autoscaling for lumbda research loops driving GPU bend backends. Originally built for foxhop ecdsa secp256k1 attack-surface work (`~/git/www.foxhop.net/ecdsa/`), shared back upstream under AGPLv3.

## What it does

Drives a queue of `.lsp` cells → emitted `.bin` payloads → dispatched to a backend (default `bend-cuda`) → results in a portal file. Survives crashes, OOM, transient CUDA errors via a two-tier DLQ. Adapts concurrency to live VRAM headroom + observed per-fork peak memory.

## Components

- `bend-supervisor.sh` — process orchestrator. Launches pool + dispatcher + autoscaler + DLQ runner.
- `bend-emit-pool.sh` — bounded parallel emit loop. Auto-pulls `LUMBDA_REPO_DIR` per iter.
- `bend-dispatcher.sh` — singleton dispatcher. Reads supervisor.config tier caps live, dispatches `.bin` to backend.
- `bend-autoscaler.sh` — V2 live-state controller. Samples VRAM free + running forks + EWMA peak → plans per-tier caps.
- `bend-supervisor-dlq-runner.sh` — DLQ classifier. Auto-retries transient classes, escalates persistent to rDLQ.
- `lib-tier.sh` — bin-size classifier. Tier bands + HUGE solo-dispatch.
- `lib-heartbeat.sh` — supervisor liveness signaling.

## V2 controller — live VRAM model

Rather than reserving `PEAK × allocated_slots` for every tier (V1's approach, prone to over-reservation under skew), V2 samples actual state:

- `nvidia-smi --query-gpu=memory.free` → live headroom (not fixed budget × fraction)
- `pgrep -f "$LUMBDA_BACKEND_PROC_PATTERN"` + `nvidia-smi --query-compute-apps=used_memory` → fork count + observed peak
- EWMA over observed peaks → per-worker VRAM estimate (file `autoscaler.ewma`)
- Zero-floor on empty tiers (no `MIN=1` reservation)
- Fall-through admission (single-tier demand → grant whole budget to that tier)
- ±25 % per-poll rate-of-change damping
- DLQ growth-rate damping (halve next plan if DLQ growth > threshold)

See `bend-autoscaler.sh` header for signal format + operator overrides.

## HUGE solo-dispatch

Cells that classify as HUGE (4-12 GB bins) get whole-card solo dispatch: when a HUGE bin is queued or in flight, all other tiers plan to 0 until HUGE completes. Damped at 25 %/poll so already-running cells finish naturally (~2 min drain).

Trigger: `r_huge > 0` OR `inflight_huge > 0`. Release: `inflight_huge = 0` AND no queued HUGE.

## Two-tier DLQ + rDLQ

`bend-supervisor-dlq-runner.sh` polls `$LUMBDA_QUEUE_DIR/dlq/` every 30s. Per entry, classifies + acts:

- **Auto-retry classes** (`missing-bin`, `bisect-pool-race`, `cuda-error-transient`, `no-portal`) → move `.bin` back to queue, `.ready` touched, `.retries++`. Cap `AUTO_RETRY_MAX=3` then escalates.
- **Escalate classes** (`cuda-oom`, `cuda-illegal-addr`, `memory-cap-refused`, `bin-load-fail`, `tier-classify`, `emit-broken`, `unknown`) → move to `$LUMBDA_QUEUE_DIR/rdlq/` with full state preservation. Owner inspects + patches + retries.

rDLQ state per cell:
```
rdlq/<tag>.bin         (preserved iff stage != emit AND salvage=yes)
rdlq/<tag>.lsp         (cell source — required for retry-reemit path)
rdlq/<tag>.reason      (original DLQ reason + classifier verdict)
rdlq/<tag>.class       (cuda-oom | bin-load-fail | ...)
rdlq/<tag>.stage       (emit | dispatch | sim)
rdlq/<tag>.retries     (cumulative)
rdlq/<tag>.first-seen  (UTC timestamp)
```

## Cold-recovery baseline

V2 writes `$LUMBDA_QUEUE_DIR/supervisor.config.baseline` every poll iff `total >= CPU_CEILING/4`. Last known-healthy cap vector. On autoscaler crash with stale `supervisor.config`:

```bash
cp "$LUMBDA_QUEUE_DIR/supervisor.config.baseline" "$LUMBDA_QUEUE_DIR/supervisor.config"
```

then restart `bend-autoscaler.sh`. Baseline overwritten each healthy poll, tracks workload evolution.

## Configuration

All knobs in `CONTRACT.md`. Defaults assume a generic lumbda dev setup; domain consumers (foxhop ecdsa, etc.) set env vars then exec.

Example consumer wrapper:
```bash
#!/bin/bash
export LUMBDA_REPO_DIR="$HOME/git/www.foxhop.net"
export LUMBDA_DOMAIN_DIR="$HOME/git/www.foxhop.net/ecdsa"
export LUMBDA_QUEUE_DIR="/tmp/ecdsa-queue"
export LUMBDA_BACKEND_CMD="$LUMBDA_DOMAIN_DIR/cuda/bend-cuda"
export LUMBDA_EMITTER_CMD="lumbda --fast $LUMBDA_DOMAIN_DIR/lumbda/emit-stream.lsp"
exec "$HOME/git/lumbda/factory/bend-supervisor.sh" "$@"
```

## Testing

Integration tests at `~/git/lumbda/tests/integration/`:
- `test-bash-script-syntax.sh` — `bash -n` lint gate. Catches the apostrophe-in-bash-c class that took the foxhop factory down 24 min on 2026-06-12.
- `test-autoscaler-v2.sh` — V2 reducer covering 7 defect classes (skewed-demand starve, zero-floor reservation, multi-tier greedy, ±25 % damping, cold-start ramp, DLQ surge halve, post-damp CPU ceiling).
- `test-dlq-runner-classify.sh` — DLQ verdict classifier reducer.

Run: `make -C ~/git/lumbda test-integration`.

## License

AGPLv3. Originally developed at foxhop for secp256k1 attack-surface research; shared upstream as obligated by AGPLv3 share-back when the work runs as a service or against shared infrastructure.