Two follow-up defects from 2026-06-14 factory triage:
(1) DLQ runner classifier did not recognize "vram-oversized" reason text
introduced by foxhop dispatcher pre-flight (commit 8d45e0d on the foxhop
side). 65 of 87 rDLQ cells got escalated as class=unknown instead of a
properly named bucket. Adds pattern + escalate-class entry + reducer
test case mapped to (vram-oversized sim no — bin is dead weight on this
card, salvage skipped).
(2) sweep-doctrine reducers had no per-test timeout. K=5 doctrine tests
(test-k5-apply-forward-ipmul + 4 siblings) ran lumbda at 99% CPU for
2h43m on a remote node without ever emitting their DOCTRINE verdict
line — accumulating 30+ runaway lumbda procs under two stuck `make
sweep-doctrine` invocations. run.sh + run-parallel.sh now wrap our
lumbda invocation in `timeout ${SWEEP_DOCTRINE_TEST_TIMEOUT_S:-300}`;
hit exits 124, our existing "no DOCTRINE line" branch logs HARNESS-FAIL.
K=5 substrate has a documented non-terminating compute defect AND a
load-time buffer overflow (commit 6d18c59 on foxhop). Bisect deferred
per ticket 0007 in foxhop tree; needs qemu apparatus we currently lack.
shopt -s nullglob (set so outer for-loop tolerates empty *.bin) made
an unmatched .inflight-* glob expand to nothing. Bare "ls >/dev/null"
then succeeded by listing CWD, and our if-branch incorrectly skipped
every orphan whose .inflight-* did not match. Net effect: medium-sized
orphan bins (100 MiB+, valid QECCOPS1 magic, no markers) accumulated
in our queue indefinitely across pool restarts.
Replace with compgen -G which returns success only when our pattern
matches, immune to nullglob.
Adds tests/integration/test-heal-orphan-bins.sh as TCRAUDT reducer
covering our contract: medium-size orphan promotes to .ready,
sub-threshold truncated to .done, bad-magic DLQs to dlq/.
Incident 2026-06-14: 8 vecC-tri-*-w8c.bin orphans (493 MiB each, Jun 12
emit) survived multiple foxhop pool restarts. Live factory queue
visible to operator as a persistent ~8-bin floor that never drained.
29 new files publish factory infra (V2 autoscaler with live VRAM
sampling + EWMA peak tracking, HUGE solo-dispatch, two-tier DLQ/rDLQ
classifier + retry), general quantum circuit primitives (Cuccaro
ripple-carry adder, Clifford gate library, Clifford tableau simulator,
mod-arith family, dialog GCD reversible inverse, Karatsuba multiplier,
Solinas fast reduction), and a TCRAUDT reducer harness. Originally
developed in ~/git/www.foxhop.net/ecdsa/ for secp256k1 attack-surface
research; published upstream as obligated by AGPLv3.
Parametrization contract at factory/CONTRACT.md. Consumers export
LUMBDA_REPO_DIR + LUMBDA_QUEUE_DIR + LUMBDA_BACKEND_CMD + LUMBDA_EMITTER_CMD
then exec factory scripts. No fork-and-modify; single source of truth
upstream.
Integration tests gate 7 V2 defect classes that wedged a live factory
on 2026-06-12 (skewed-demand starve, zero-floor reservation,
multi-tier greedy, +-25%% damping, cold-start ramp, DLQ surge halve,
post-damp CPU ceiling) + 28 DLQ classifier cases (auto-retry vs
escalate partition) + bash -n syntax lint across every script.
GPU backend stays in consumer trees; rationale in
factory/GPU-BACKEND-NOTE.md. Bend wire protocol + gpu-worker.lsp
already upstream at examples/cuda-fanout/.
make factory-lint bash -n on every factory/*.sh
make test-integration V2 reducer + DLQ classifier + syntax gate
make sweep-doctrine TCRAUDT reducer gate (serial)
make sweep-doctrine-parallel xargs -P fan-out
Verified on neoblanka: factory-lint 12 scripts PASS; test-integration
14 V2 cases + 28 DLQ classifier cases + 12 syntax cases all PASS.
