bend!-call from the WAT tier
- New WAT import: (import "env" "bend_call"). The host loader supplies
a sync XMLHttpRequest that POSTs the payload to a configured URL
(workers only — sync XHR isn't allowed on main thread).
- New primitive (bend!-call "<payload>") returns the response as a
lumbda string. Works in playground and REPL once the bend URL is
saved in the new top bar.
- bendUrl persists in plain localStorage (not encrypted — it's a
server address, not a secret).
- Tests stub the import with a no-op so unit / integration / functional
suites keep instantiating cleanly.
Playground + REPL zoom 133% by default
- html { zoom: 1.33 } so the styleguide sizes read comfortably without
requiring browser-level zoom.
Free-form default = cross-tier assertion runner in Lisp
- The default editor content for "free form" is now a small assertion
framework matching tests/functional.lsp's PASS/FAIL convention. A
starter the user can extend, runs identically on the three tiers.
C tier + Python tier (bend) are wired through the runner stub; full
bend integration in those tiers comes next once their loaders learn
about setBendUrl.
Adds (write x) and (write-string s) primitives. write quotes strings
and #\-prefixes chars — what tests/functional.lsp's assert-equal
uses to print failures.
New target: make wasm-test-functional-cross (also rolled into wasm-test)
runs the 205-assertion tests/functional.lsp against each WASM tier and
reports pass counts:
c-wasm: 205 / 205 (full parity with native c)
asm-wasm: 93 / 111 reaches mid-suite before stack overflow on
a deeply recursive test; the 84% it reaches
passes. Documented progress toward full parity
with asm/lumbda.s.
The runner exit-soft on the asm tier — it's a measurement, not a gate.
Free-form radio adds a 5th demo slot. When selected, a vault bar appears
under the controls: enter a password, "unlock" derives a per-device
vault and decrypts (or creates fresh). Edits in the editor auto-save
350ms after typing stops. Reload + same password restores the code.
Same Web Crypto stack as /repl/ (PBKDF2 + AES-GCM, vault id =
SHA-256(password || device-salt)).
Layout: one shared vertical scroller — code pane and output pane both
grow with content, the body scrolls. No more independent in-pane
scrollers fighting the page.
Home page split into "Demo" and "REPL" sections with their own CTAs.
Interactive REPL at lumbda.com/repl with:
- multi-tab sessions (click + to add, × to close, double-click to rename)
- per-tab tier selector (python/c/asm/all-three race)
- persistent transcripts encrypted in localStorage via Web Crypto
(PBKDF2 + AES-GCM, vault id = SHA-256(password || device-salt) —
same pattern as unsandbox's vault-encryption-design.md, native
crypto.subtle API instead of CryptoJS)
- ephemeral mode (skip vault, transcripts vanish on reload)
- one worker per (tab × tier) — state persists across evals in a tab
- reboot tier button (terminate this tab's worker, fresh state next eval)
- cancel button (kills the running worker in active tab)
Home page now links to both /playground/ and /repl/.
Tier state itself does NOT persist across reloads — the transcript does,
but defines/set!/hash-tables vanish with the worker. Portal save/resume
in WAT (deferred) will let a tier session survive close+reopen.
WAT prelude (evaluated after primitive binding at init) adds:
map, filter, fold-left, fold-right, for-each, any, every,
count, find, sort (quicksort), vector-map, vector-for-each,
vector-fill!, string-split, string-trim, string->list,
random-state, assert-equal/true/false.
Higher-order ops are now Lisp-defined, not primitive bloat. Eval-time
parse + bind happens once per WASM instance startup.
Playground output: per fox, single append-only column instead of
3-up grid. Tiers still race in parallel workers; whichever finishes
first appears first in the output. Live ms counters move to the status
bar (python 312ms · c 47ms · asm 89ms).
Two new heap tags:
7 = vector [tag, len, elem_0, elem_1, ...] 8 + 4*len bytes
8 = hashtable [tag, count, alist_ptr] 12 bytes
Vector primitives (88-95):
vector, vector?, make-vector, vector-length,
vector-ref, vector-set!, vector->list, list->vector
Hash table primitives (96-106):
make-hash-table, hash-table?, hash-table-set!,
hash-table-ref, hash-table-ref/default,
hash-table-delete!, hash-table-exists?, hash-table-size,
hash-table-keys, hash-table-values, hash-table->alist
Hash table lookup is linear (equal? on each key) — fine for browser-scale
demos. Same linear-scan caveat as the symbol intern; would warrant a real
hash function at scale.
print_value now renders vectors as #(a b c) and hashtables as #<hashtable>.
One Worker per tier (python/c/asm). "All three" mode dispatches
Promise.all so the tiers race on independent threads — a slow Pyodide
no longer blocks C and asm. Each tier-block ticks its own ms counter
until its worker resolves.
Output grid: 3 columns when 3 tier-blocks render, else stacked.
Status bar announces the winner: "ok — c won in 47 ms".
Cancel terminates every active worker.
User-visible changes
- Cancel button — terminates the running worker. Pyodide's slow mandelbrot
no longer freezes the UI; click cancel and the elapsed counter freezes
at "(cancelled @ NNNN ms)".
- Live ms counter ticks per animation frame while a tier is busy, so the
Pyodide tier's ~5-15 s wait is visible instead of looking hung.
- Restyled to match lumbda.com: chunkfive wordmark, --green: #227842
(light) / #5ec07a (dark), lumbda-logo-green.png, lowercase "lumbda"
everywhere. Pulls fonts/chunkfive locally so the playground stays
self-contained.
Architecture
- All tier evaluations now run inside a Web Worker (wasm/app/worker.mjs)
so the main thread stays responsive. Cancel = worker.terminate(); next
eval respawns a fresh worker.
- Loaders use new URL("./...", import.meta.url) so paths resolve against
the loader file's own location — works identically in window and
worker contexts, no baseURL argument needed.
- C tier Emscripten build flipped to EXPORT_ES6=1; loader uses dynamic
`import()` of the factory module. Integration test updated accordingly.
- Python loader uses `import("pyodide.mjs")` (ES module) instead of
document.createElement, which doesn't exist in workers.
Bug fixes
- Asm tier state leak: running the same demo twice on a cached WASM
instance produced corrupted output (every other cell on row 2+ rendered
as " " instead of the expected shade char). Root cause: top-level eval
passed `global_env` as the env, so closures captured stale globals;
fixed by passing NIL — env_lookup falls back to the CURRENT global_env
via its existing two-pass walk. Multi-run regression added to the
functional test suite.
- fib-ack demo: (ack 3 4) was too heavy for Pyodide (minutes). Cut to
(ack 3 3) + (fib 20) max so every tier finishes in seconds.
Test discipline
- Root `make test-all` now includes `wasm-test`. Adding a language
feature without exercising it on all six implementations is no longer
possible by accident.
- Functional suite: 11 assertions (was 10) — adds asm multi-run stability.
- Integration + unit: still 20 + 8.
Adds a "Try it in your browser" section at the bottom of www/index.html
pointing to /playground/, plus a footer link for redundancy. Lands the
three-tier WASM SPA committed in 346b873 as a discoverable surface on
lumbda.com.
shopt -s nullglob (set so outer for-loop tolerates empty *.bin) made
an unmatched .inflight-* glob expand to nothing. Bare "ls >/dev/null"
then succeeded by listing CWD, and our if-branch incorrectly skipped
every orphan whose .inflight-* did not match. Net effect: medium-sized
orphan bins (100 MiB+, valid QECCOPS1 magic, no markers) accumulated
in our queue indefinitely across pool restarts.
Replace with compgen -G which returns success only when our pattern
matches, immune to nullglob.
Adds tests/integration/test-heal-orphan-bins.sh as TCRAUDT reducer
covering our contract: medium-size orphan promotes to .ready,
sub-threshold truncated to .done, bad-magic DLQs to dlq/.
Incident 2026-06-14: 8 vecC-tri-*-w8c.bin orphans (493 MiB each, Jun 12
emit) survived multiple foxhop pool restarts. Live factory queue
visible to operator as a persistent ~8-bin floor that never drained.
Adds a parallel build of all three Lumbda implementations to WASM, a
single-page playground at www/playground/, and a verified test suite.
Tiers
- Python: Pyodide (CPython-in-WASM) hosting lumbda.py
- C: Emscripten build of c/ (tree-walker + bytecode VM; jit.c
stubbed, gc.c uses its existing no-Boehm fallback)
- Asm: hand-written asm/lumbda.wat — parallel impl to asm/lumbda.s.
Reader, eval (lambda/define/if/cond/let/and/or/quote/set!),
recursion across mutated top-level env, bump allocator with
memory.grow, 24 primitives. ~1200 lines of raw WAT.
SPA (wasm/app/, deployed to www/playground/)
- CodeMirror 6 editor (Scheme highlighting) on left, output on right
- Radios: 4 demos (Mandelbrot, Fib+Ack, Sieve, self-interp meta-eval)
x 4 tiers (Python | C | Asm | All three)
- All-three mode renders the three tier outputs side by side with
per-tier elapsed timing
Tests (38 verified assertions)
- 20 unit (Node): per-tier module loads, eval smoke
- 8 integration (Node): each demo on c+asm WASM byte-matches the
canonical native Python run
- 10 functional (Playwright headless Chromium): page mounts, every
demo runs on every tier, all-three renders
Makefile
- Root targets: wasm-build, wasm-test, wasm-test-fn, wasm-serve,
wasm-deploy, wasm-clean
- wasm/Makefile orchestrates the three tier builds; deploy copies
dist/ into www/playground/
Asm tier notes
- WAT linear symbol intern + linear env lookup is MOAD-0001 at scale;
documented in the asm/lumbda.wat header and in the SPA footer. The
demos hit ~30 globals so the linear walks are cheap enough.
- Bump allocator never frees (matches asm/lumbda.s heap discipline);
memory.grow expands by 1 MB chunks. Browser tab tears down at unload.
Toolchain (developer prerequisites)
- Emscripten 6.0.0 via emsdk at ~/git/emsdk
- wabt 1.0.36 at ~/git/wabt
- Playwright for functional tests (symlinked from ~/git/agnt)
29 new files publish factory infra (V2 autoscaler with live VRAM
sampling + EWMA peak tracking, HUGE solo-dispatch, two-tier DLQ/rDLQ
classifier + retry), general quantum circuit primitives (Cuccaro
ripple-carry adder, Clifford gate library, Clifford tableau simulator,
mod-arith family, dialog GCD reversible inverse, Karatsuba multiplier,
Solinas fast reduction), and a TCRAUDT reducer harness. Originally
developed in ~/git/www.foxhop.net/ecdsa/ for secp256k1 attack-surface
research; published upstream as obligated by AGPLv3.
Parametrization contract at factory/CONTRACT.md. Consumers export
LUMBDA_REPO_DIR + LUMBDA_QUEUE_DIR + LUMBDA_BACKEND_CMD + LUMBDA_EMITTER_CMD
then exec factory scripts. No fork-and-modify; single source of truth
upstream.
Integration tests gate 7 V2 defect classes that wedged a live factory
on 2026-06-12 (skewed-demand starve, zero-floor reservation,
multi-tier greedy, +-25%% damping, cold-start ramp, DLQ surge halve,
post-damp CPU ceiling) + 28 DLQ classifier cases (auto-retry vs
escalate partition) + bash -n syntax lint across every script.
GPU backend stays in consumer trees; rationale in
factory/GPU-BACKEND-NOTE.md. Bend wire protocol + gpu-worker.lsp
already upstream at examples/cuda-fanout/.
make factory-lint bash -n on every factory/*.sh
make test-integration V2 reducer + DLQ classifier + syntax gate
make sweep-doctrine TCRAUDT reducer gate (serial)
make sweep-doctrine-parallel xargs -P fan-out
Verified on neoblanka: factory-lint 12 scripts PASS; test-integration
14 V2 cases + 28 DLQ classifier cases + 12 syntax cases all PASS.
Drop the static *vram-budget-mib*=22000 cap that fox flagged as wrong:
'we shouldn't limit with a max — the algo should determine how many
children based on the bend forms usage in vram.'
New algorithm:
- *gpu-total-mib* (24576 default, RTX 3090) + *gpu-headroom-mib* (1024 pad)
- *vram-per-cell-max-mib* (4096 seed) tracks largest cell observed.
- admit-fork? returns true iff
(current_vram + projected_cell + headroom) < gpu_total
- wait-admit blocks at run-loop top using projected = current per-cell
max. Self-tunes: tiny cells → many concurrent, huge cells → few.
Helper file-size-mib (stat -c %s) reads bin file size as cheap proxy
for per-cell VRAM (bin file on disk ≈ peak VRAM bend-cuda loads).
Open: cross-fork learning. record-cell-vram! runs IN THE CHILD so
parent's *vram-per-cell-max-mib* doesn't see updates without a fork-
shared signal (TODO: parent peek bin path before forking, or child
writes per-cell-size to small file the parent reads). For now the
seed value + max-tracking-in-future-runs handle the common case
where all cells are similar size.
Cross-tier API parity with c-tier (81ac49e) + python-tier — all three
lumbda runtimes now share the substrate for fork-per-accept patterns.
Implementation: direct syscalls (no libc):
- SYS_FORK=57 → bi_forkself, returns 0/pid via make_int
- SYS_WAIT4=61 + WNOHANG=1 → bi_waitpid_nonblock, returns pid or 0
- SYS_EXIT=60 → bi_exit_immediate (same as bi_exit on asm — no atexit
to bypass; present for cross-tier API parity)
- SYS_NANOSLEEP=35 → bi_sleep, stack-allocated timespec (tv_sec=N,
tv_nsec=0), returns VAL_VOID
Built + tested in vm-runner.sh VM (Ubuntu 2G/2vCPU): both lumbda
and lumbda-gc + fork-cycle test = 3/3 children reap clean, exit-
immediate returns to parent waitpid correctly. Same behavioral
contract as c-tier (commit 81ac49e) and python-tier.
Single-PID parent persistent listener; each accept forks a short-lived
child handler that owns one bend-cuda subprocess + responds + exits.
Linux COW handles memory; OS scheduler distributes across cores.
N concurrent requests = N children + parent — naturally VRAM-isolated.
VRAM admission: wait-vram-clear queries nvidia-smi before each fork,
blocks accept when used > *vram-budget-mib* (default 22000, override
via LUMBDA_VRAM_BUDGET_MIB env). 24G card with avg 1-2GB per bin
supports 4-12 concurrent comfortably.
Requires lumbda c-tier fork-self / waitpid-nonblock / exit-immediate /
sleep primitives (commit 81ac49e). Child uses exit-immediate not exit
to avoid dual-cleanup hang on shared parent state.
Substrate for fork-per-accept pattern in gpu-worker.lsp — enables
async bend dispatch with internal load balancing.
c-tier (builtins.c):
- bi_fork_self: fork() wrapper, returns 0 in child / pid in parent / #f on fail
- bi_waitpid_nonblock: waitpid(-1, WNOHANG), returns reaped pid or 0
- bi_exit_immediate: _exit() wrapper — REQUIRED in fork-self children,
regular exit() runs atexit handlers against shared parent state and
hangs the child (observed empirically 2026-06-11 via vm-runner.sh).
- bi_sleep: real wall-clock sleep(3) — yields CPU. Replaces busy-loop
patterns that would (a) burn CPU and (b) SIGKILL in cgroup-limited
VMs (observed: 100M iter let-loop SIGKILL'd after 5s in qemu vm).
python-tier (lumbda.py): _fork_self / _waitpid_nonblock / _exit_immediate
/ _sleep mirrors via os.fork / os.waitpid / os._exit / time.sleep.
Tested in vm-runner.sh VM (Ubuntu 2G/2vCPU): 3-child fork-cycle test
spawns + reaps cleanly 3/3 in both c-tier + python-tier. The exact
test pattern that crashed neoblanka host pre-fix now works fine.
Asm tier: deferred. Lock retained at chmod a-x ~/git/lumbda/asm/lumbda*
per CLAUDE.md threat model.
Earlier rename kept DEMO_OPS env fallback + *binary-demo-ops* var name
as transitional back-compat. With both hosts redeployed on bend-cuda
that's no longer needed.
Renamed:
*binary-demo-ops* -> *binary-bend-cuda*
DEMO_OPS env -> removed (only BEND_CUDA recognized now)
Also bulk-updated cuda-fanout sibling docs (DESIGN, CATALOG, plans/)
that still spelled the old name.
Slot reserved for future bend-rocm / bend-cpu via parallel env vars.
bend's (health) now reports whether $BEND_QUEUE_DIR/FEEDER_PAUSE
marker is set. Consumers (feeder, factory-status, ops scripts) get
pause state in the same single RPC as bend liveness + supervisor
proc counts + queue depths.
Returns:
1 marker present (operator wants this host out of rotation)
0 no marker (host in active rotation)
-1 BEND_QUEUE_DIR env unset (host has no associated queue)
Lets a future feeder version drop separate SSH pause-marker probes
in favor of the bend health RPC. Today's feeder still does the SSH
check; this just opens the door for the simpler model.
Mirror of pool-procs added earlier — bend health now reports both
pool-procs and dispatcher-procs counts so a single RPC tells the
feeder/factory consumer whether either supervisor is dead while
queue has work.
Failure mode this fixes: 4090's bend-dispatcher hit MAX_IDLE_LOOPS
drain-exit; balance moved 5 cells into its queue but no dispatcher
to consume them. Bend health response previously didn't surface the
gap; feeder's separate SSH probe (now added in www.foxhop.net commit)
caught it but a single RPC is cheaper than per-host SSH.
Follows foxhop ecdsa repo rename of infrastructure scripts. The pool
process is now named bend-emit-pool (job-agnostic), not ecdsa-emit-pool.
health-pool-procs pgrep updated to match.
bend co-lives with ecdsa-emit-pool on each foxhop production host.
When pool dies but bend stays up, .lsp cells pile un-emitted; bend
sits idle waiting for .ready bins that never arrive. Today's incident
took 30+ min to surface because feeder couldn't tell from bend health
alone — needed a separate SSH+pgrep per host.
Add pool/queue counts to (health) so one RPC returns the full picture:
(ok (load-avg L) (vram-free-mb V) (uptime-ms U)
(pool-procs P) (queue-ready R) (queue-emitting E) (queue-done D))
Helpers:
health-pool-procs pgrep -cf ecdsa-emit-pool
health-queue-count EXT ls $BEND_QUEUE_DIR/*.EXT | wc -l
BEND_QUEUE_DIR env var — set when bend is launched on a host with an
associated pool. Absent → queue counts return -1 (caller treats as
'unknown / not applicable').
Caller now has single-RPC view of bend + pool + queue health; feeder
can drop its separate SSH pool-watchdog probe in favor of the bend
(health) RPC field.
Lets bend (examples/cuda-fanout/gpu-worker.lsp) run multiple worker
processes behind a single listening port. Each worker calls
tcp-listen on the same port; the kernel distributes incoming
connections across the bound sockets.
Foxhop production use case: 2 bend workers per GPU host (3090 + 4090)
to consume the .ready queue at 2x throughput without an external
load balancer.
The CUDA binary that bend's gpu-worker.lsp spawns to run QECCOPS1
ops.bin sims is renamed from demo_ops to bend-cuda (production naming;
AMD ROCm equivalent will land as bend-rocm).
*binary-demo-ops* now reads BEND_CUDA env first, falls back to legacy
DEMO_OPS env, then default path /home/fox/git/www.foxhop.net/ecdsa/cuda/
bend-cuda. The legacy demo_ops symlink is in place at the install side
so back-compat holds during the rename transition.
See www.foxhop.net commit (sibling repo) for the foxhop ecdsa side
of the rename.
Root cause for repeated bend gpu-worker crashes mid-pipeline:
;;; bend RECV cuda-sim-ops-bin ops=monitor-r00007-c006.bin
error: not a pair: #f
When demo_ops crashes mid-write (GPU OOM, segfault, etc) the portal
file may exist but be empty or malformed. read-from-string returns
#f. Both portal-timing + portal-find-number then call (cdr #f) =
runtime crash, taking the port-8320 listener down.
Fix: gate both walkers on (pair? form) BEFORE descending. When form
is not a pair (= #f from empty/malformed portal), return #f cleanly.
Caller's (or cpu-ms 'NA) display already handles #f.
Watchdog still exists for true OOM / kill -9, but routine empty-portal
events no longer take bend down.
Reproduces by sending a malformed S-expr to portal-find-number or
portal-timing.
ecdsa's Phase B emit at full secp256k1 width writes a 4–6 GB QECCOPS1
ops binary. The existing path — accumulate in a string-output port,
materialize via get-output-string, write — peaks RAM at 3× body
size (port internal buffer + Scheme string copy + write-binary-file
concat). A 6 GB body needs ~18 GB transient; OOMs a 16 GB QEMU guest.
This commit shifts emit-stream onto a constant-RAM file-port path
and fixes binary-correctness defects in the supporting primitives.
New primitives (mirrored across c/builtins.c + lumbda.py):
- open-binary-output-file path
Opens in "w+b" so the caller can seek back to rewrite a header.
- port-set-position! port offset
fseek absolute offset on a file port. emit-stream reserves a
16-byte placeholder header, streams the body, then seeks back to
byte 0 to rewrite the QECCOPS1 + n_ops u64 LE once n_ops is known.
- append-binary-file path data
Opens in "ab" and fwrite's the bytes through. Pairs with
write-binary-file so callers can land header + body in two writes
instead of (string-append header body).
- append-port-to-binary-file path port
Streams a string-output port's buffer to disk via fwrite without
materializing (get-output-string port). Lets callers keep their
existing string-output sink and avoid the body-size string copy
if they stay on string-port emit.
Binary-correctness fixes:
- bi_write_string to a file port used fputs, which calls strlen.
Binary payloads containing 0x00 truncated at the first null byte.
Switched the file-port branch to fwrite with the string's known
->len (same fix family as the earlier bi_get_output_string
strlen defect).
- bi_write_char per-byte fflush guarded to stdout only. With
millions of gate-bytes per second, flushing after every fputc to
a file port was a 100× slowdown. File ports buffer until close
or explicit flush-port — keep stdout's per-byte feedback path,
drop fflush on every file-port byte.
- port_write_str grows 1.5× past 256 MB instead of 2× throughout.
At realloc time the transient peak is old + new; 2× at 8 GB →
16 GB transient needs 24 GB. 1.5× bounds peak at 2.5× and keeps
multi-GB string-port workloads inside a 16 GB VM.
Tests: 88/88 c-test, 4/4 regression-named-let-leak, 205/205
functional, zoe-favorites all tiers. Binary roundtrip with embedded
nulls at 10/1000/100000 bytes passes byte-for-byte.
End-to-end: foxhop ecdsa DIALOG_GCD secp256k1 emit lands a 4.7 GB
binary at 322 MB peak RSS in 7:41 wall on a 16 GB QEMU guest.
Two C-tier defects surfaced when running foxhop's ecdsa Phase B emit
through the lumbda C interpreter. Both produced wrong bytes in the
generated QECCOPS1 binary; both Python tier handled correctly.
1. string-ref on a byte >= 0x80 returned a char with codepoint -1.
The store was a `char` array (signed on x86_64); `s->data[idx]`
sign-extends 0xFF into a negative int before VAL_CHAR wraps it.
Round-tripping (char->integer (string-ref s 0)) for a 0xFF byte
gave -1 instead of 255. The Scheme-level (u64-le n) packer uses
(make-string 1 (integer->char (modulo v 256))) for each byte and
reads them back; sign-extension corrupted the high-bit bytes.
Cast through `unsigned char` in bi_string_ref.
2. pack_u64_slot treated bignum slots as raw fixnums.
ecdsa's emit-stream binds (no-slot *no-slot*) where *no-slot* is
18446744073709551615 (u64 max). On Python tier that's a regular
big-int. C tier carries it as a bignum NaN-box slot. The packer
knew about VAL_FALSE → 0xFFFF... but called as_int(v) on bignums,
reading the NaN-box payload bits (pointer-to-Bignum) and writing
that pointer as the field's 64-bit LE value.
Add an IS_BIGNUM branch that extracts the low 64 magnitude bits
directly. Cross-tier emit-stream code stays unchanged.
Tests: 88/88 c-test, 4/4 regression-named-let-leak, 205/205
functional. DIALOG_GCD smoke at p=11 n+1=5 now byte-identical to
Python tier (1,941,816 bytes match exactly).
Two pre-existing defects in c/builtins.c made (open-output-string)
unusable for binary emit:
- bi_get_output_string ran the buffer through make_string_from_cstr,
which calls strlen. Any 0x00 in the payload truncated body at
that byte. Use port's known str_len directly via make_string.
- bi_write_char ignored string-port destinations entirely — it
pulled AS_PORT(p)->fp (NULL for string ports), fell back to
stdout, and silently routed gate bytes to terminal output
instead of the port buffer. Route to port_write_str when target
port kind is PORT_STRING, matching bi_write_string's behavior.
Surfaced while validating the precise-GC fix against ecdsa's
Phase B emit (writes 56-byte op records full of embedded nulls
through a string-output port, get-output-string at the end).
With these fixes ecdsa's n+1=64 emit drops a 333 MB binary in
42.6 seconds — pre-fix it timed out at 600 s with a 17-byte
header-only file (strlen truncated body at byte 1; the visible
gate bytes had been escaping to stdout the whole time).
Binary roundtrip test (write n bytes alternating x / 0x00 to
output-string port, read back via get-output-string):
n=10 len=10 ok
n=1000 len=1000 ok
n=100000 len=100000 ok
All upstream tests still pass (88/88 c-test, 4/4 regression,
205/205 functional, zoe across tiers).
Boehm's conservative pointer scan cannot recognize lumbda's Value
layout — heap pointers live in the low 48 bits with QNAN + tag bits
in the upper mantissa, so a raw word never looks like a heap address.
Until now main.c neutralized this with GC_disable(): every allocation
leaked, OOMing any long-running workload.
Add precise tracing via a custom Boehm kind:
- New c/gc.c: mark proc walks 8-byte words in mixed mode — when the
QNAN bits are set with a pointer-bearing tag (0/2/4/5/6) extract
the low-48 pointer; otherwise fall through to raw-pointer
validation. GC_set_push_other_roots callback decodes NaN-boxed
Values on the C stack via setjmp anchor + scan up to the stack
base captured at process start.
- Allocations holding Values (Pair, Env bindings, ValueStack data,
ULVector data, HTEntry, Proc params + body, FullCont stack,
CodeObj instrs, SymbolEntry) route through lumbda_value_malloc.
Pure-byte sites (bignum limbs, char buffers, source files) stay
on regular GC_MALLOC.
- main.c / test.c / bench.c capture stack-base then drop GC_disable.
types.c also zeros popped slots on the value stack so stale pointers
do not survive a vs_pop and pin freed objects — independent
correctness fix that pays off once GC actually runs.
Build: USE_GC=1 (default when /usr/include/gc.h exists).
Tests with GC enabled:
- 88/88 c-test
- 4/4 regression-named-let-leak (test that motivated GC_disable)
- 205/205 functional (Python + C)
- zoe-favorites all tiers (Python + C + asm + asm-full)
alloc-test 1M cons drop-loop:
- Before: 0.60s wall, 156 MB RSS, leaks every cell
- After: 0.37s wall, 4 MB RSS, ~1500 GC cycles each freeing ~370 KB
Ports lumbda.py _emit_circuit_to_ops_bin_stream to C tier. Walks a
Scheme registers + ops list, writes each 56-byte QECCOPS1 op record
straight to disk via fopen/fwrite, then seeks back to patch the n_ops
header at the tail. O(1) host memory regardless of n_ops.
Cross-tier byte-identity verified:
* 500-op mixed-tag synthetic circuit on host: Python ↔ C identical
* real-point-add n+1=5 (p=11, 94,214 ops, 5.0 MB) inside VM
* real-point-add n+1=9 (p=251, 674,872 ops, 36 MB) inside VM
sha256 matches every width.
Speedup on equal Scheme source (build + emit combined):
* n+1=5: Python 56.8 s → C 1.1 s (52×)
* n+1=9: Python 365 s → C 8.4 s (43×)
Combined ratio exceeds the prior 11-16× C-tier envelope because the
build phase (Phase B mod-arith construction) also accelerates on C;
emit-only ratio is ~3-9× and grows with op count.
C test suite: 85 → 88 passing (emit-stream-basic, alloc-free, empty).
Shared functional suite: 205/205 still passing on both tiers.
Implementation notes:
* pack_op_record packs u32 kind + u32 pad + 6× u64 LE, matching
op-specs->bytes byte layout exactly.
* Layout hashtable Symbol → fixnum base, mirroring walk-circuit-ops.
* NO_SLOT sentinel: 0xFFFFFFFFFFFFFFFF written directly to u64 slots
that the Scheme side did not populate.
* libc's default fwrite buffer (~4 KB) handles batching at ~73 ops
per write — same throughput class as Python tier's 8 KiB list batch.
Folds 76 commits of substance since 2026-04-24 into the whitepaper
without losing any of the novel cross-domain glue the language earns
its keep on.
Additions
- §4.3 extended from four scopes to five, with bend as feedback across
heterogeneous compute (host ↔ GPU). New diagram
diagrams/five-scopes-feedback.png stacks the boundaries.
- §11.8 Bend: cross-tier GPU dispatch. Wire modes (S-exp text, BSHK
binary), worker hosting per tier, spawn-process-stdio + flush-port
cross-tier IPC, worker health heartbeat with VRAM-ranked pick,
6-row catalog of live forms (cuda-shake-fanout, cuda-secp256k1-mul,
cuda-bignum-cgbn, cuda-radix-sort, cuda-blake3-tree,
cuda-sim-ops-bin). New diagram diagrams/bend-dispatch.png.
- §2.1 C-tier bignum: arbitrary-precision integers (Boehm-GC managed),
which unblocked the secp256k1 widths the GPU forms need on the host
side.
- §12.1 post-cycle audit: recv-exact O(n²)→O(n), asm scheme_read
overflow, asm gc_sweep page-fault, C-tier JIT cur_code restore
across CALL/RETURN.
- §13 GPU Phase 1 moved from future to shipped; Phase 2 trampolining
and Phase 3 interaction combinators sharpened.
- Source repo link (git.unturf.com/engineering/unturf/lumbda) added
to the cover page band and the Citation block.
Compressions (no novel glue dropped — audited per fox's constraint)
- §8.2-§8.5 EML derivation prose collapsed into a single §8.2
derivation chain code block; §8.6 renumbered to §8.3.
- §6.6.3 adaptive-meta-GC narrative compressed; result table kept.
- §7.4.1 GC-build S-expression portal tightened to two paragraphs;
"language is its own wire format" insight kept.
Preserved in full per audit
- §7.5 portable RNG state across tiers
- §7.5.1 one-side kernel entropy + portable bit-identical continuation
- §11.7 sendfile + adaptive preload (27 KB asm-gc within 2% of Caddy)
- §6.6.1 collaborative arena + mark verifier
- §6.6.5 precise block typing
- §8.1 + §8.3 Lumbda hosts its own EML proof checker, ~16× faster
than Lean cold
Net RST: 1622 → 1655 lines.
inject-whitepaper-css.py adds CSS for a fixed 280px left sidebar plus a
DOMContentLoaded script that walks `section[id] > h2, > h3` and builds a
chapter/section nav. IntersectionObserver tracks active section as you
scroll; the rail scrolls itself to keep the current entry in view. Below
1100px the rail hides behind a hamburger toggle with backdrop dismiss.
Cancels docutils responsive.css `body > *` blanket padding on the rail
via the same `all: revert` pattern already used for the uncloseai
floating button. 59/59 source sections present in rendered DOM.
Two link fixes:
- Drop dead /gpu-mesh link. That wiki RST exists in our private
repo but our remarkbox host has not landed it under a public URL
yet. Describe coordinator pattern inline so a reader sees the
shape without following an unreachable href.
- /ecdsa link was also wrong. Real published page lives at our
remarkbox UUID slug. Updated to actual URL.
Closed-source reminder for future drift: foxhop research code is
NOT open source. Never link to a script path under foxhop-book; if a
concept needs explaining, inline a description instead of pointing
at our private repo.
Earlier text linked to 'foxhop ecdsa scripts/coordinator-mesh.py' as
if reachable at a public URL. That script lives only in the private
foxhop-book repo — no public path exists. Point readers at our
foxhop.net/gpu-mesh wiki page instead (lives in remarkbox, publicly
reachable, describes the cost-routed coordinator pattern at concept
level).
Previous override example dropped to PORT=9001 — drifted off our BEND
mnemonic. Updated so LUMBDA=python & LUMBDA=asm overrides keep our
default port. Off-BEND port override stays available for a 2nd worker
on a single host, called out as such.
Adds tagged bignum support alongside the existing 48-bit fixnum on the C
tier. Tag 6 = bignum, heap struct sign-magnitude with u64 little-endian
limbs. Reader emits bignums for any literal past the fixnum range; +, -,
*, quotient, remainder, modulo, expt, =, <, >, abs, odd?, even?,
integer?, exact?, number->string, string->number all promote fixnum →
bignum on overflow & demote back when results fit. Boehm GC owns every
allocation. Schoolbook O(n²) mul + shift-subtract divmod is sufficient
at our 4-limb / 256-bit scale.
Before: (expt 2 48) = 0, (expt 2 256) = 0, secp256k1-p = -4294968273.
After: all three return their exact arbitrary-precision values, matching
Python tier byte-for-byte.
Validated:
- c/test.c — 85/85 pass (+2 new bignum unit tests).
- tests/functional.lsp — 205/205 pass on both C & Python tiers.
- tests/bignum-cross-tier.lsp — 33/33 pass byte-identical on both tiers
(diff produces no output).
- ecdsa/runs/lumbda-sweep-003/c-tier-bignum-probe.lsp — all four
assertions now match the Python oracle.
- ecdsa Phase B byte-identity sweep inside QEMU guest:
n+1=9 p=251 sha256 c668bbe3... — matches Python oracle.
n+1=18 p=131071 sha256 8a031f96... — matches Python oracle.
n+1=33 p=2³²-5 sha256 0bc56905... — matches Python oracle.
Previously the n+1=33 C tier emitted sha256 b024d6d9... (26,078 fewer
Toffolis due to silent fixnum wrap). Bignums close that gate.
secp256k1 production-width emit (n+1=257) is now structurally unblocked
on C tier; downstream agent (#55) drives that next-step on the ecdsa
side. Asm tier inherits in a follow-up port.
Foxhop ecdsa point-add emit at n+1=257 secp256k1 (~15M ops) overruns
host RAM via the accumulator path — walk-circuit-ops builds a 4 GB
Scheme list, op-specs->bytes builds 840 MB body string before
write-binary-file ships it. sweep-secp256k1.lsp OOMs at 8 GB.
Streaming primitive walks the circuit & writes each 56-byte op record
directly into an open binary file handle in a single pass:
(emit-circuit-to-ops-bin-stream out-path registers ops)
Per-op: pack 56 bytes via the same struct.pack format as
op-specs->bytes, append to an 8 KiB batch buffer, flush to disk when
threshold hits. After the walk closes, seek(8) patches the n_ops
header field. No intermediate list, no intermediate body bytes.
Memory profile at n+1=32 / 146 ops: ~64 KB grew during emit (host
file buffer + interpreter overhead). Holds flat at n+1=256 / 1154 ops
& n+1=1024 / 4610 ops. Will hold flat at n+1=257 secp256k1 / ~15M
ops — the byte-level operation count grows linearly but RSS does not.
Dispatch table mirrors _walk_circuit_ops exactly so bytes are
bit-identical to (op-specs->bytes (walk-circuit-ops ...)) by
construction. Verified byte-for-byte at p=11 (48 763 ops), p=251
(159 263 ops), & a hand-rolled n+1=32 hand circuit.
Foxhop wrapper lands as (emit-ops-bin-stream out-path c) in
ecdsa/lumbda/emit-ops-bin.lsp. Existing (emit-ops-bin ops out-path)
accumulator API stays untouched for backward compatibility — all
288 ecdsa/tests/unit/test-emit-ops-bin.lsp assertions still pass.
Earlier port-flip commit (8d66bc0) accidentally added a submodule
pointer at .claude/worktrees/agent-a7b0eac1 via a stale entry in
lumbda.py's git add -A run. No .gitmodules file ever existed for it,
so clones would carry a dangling gitlink. Remove from index & add
.claude/ to .gitignore so agent state stays untracked.
Extends the cuda-sim-ops-bin row in our form catalog with a measured
property: kernel wall scales linearly with Σ Toffoli across circuit
variants.
foxhop ecdsa lever sweep at p=251 (n+1=9 secp256k1-toy) dispatched 6
lumbda-emitted ops.bin variants through demo_ops on a 3090. Wall vs
Toffoli per shot:
Fermat-schoolbook 167984 Tof/shot 261.4 ms/batch
refined-Solinas 24176 Tof/shot 26.6 ms/batch
9.83x GPU wall reduction matches 6.95x Toffoli ratio + parallel
Clifford drop. No scheduler surprise — pick any QECCOPS1 circuit,
predict GPU wall from a cheap CPU op-counter.
Links to https://www.foxhop.net/ecdsa for the upstream lever-attack
context.
demo_ops in foxhop ecdsa repo now defaults --rng-mode shake (Fiat-Shamir
over op stream, ports eval_circuit::fiat_shamir_seed verbatim). Σ Toffoli
& avg Toffoli on stock ops.bin match upstream eval_circuit bit-for-bit:
15,999,651,264 / 1,773,011.000 across 141 batches (9024 shots) on a
3090. Our cuda-sim-ops-bin numbers now compare directly to upstream's
public Pareto scoreboard with no calibration constant.
Catalog row & bend.html row updated to document the shake/lfsr toggle
& the upstream-match guarantee. Throughput figure also refreshed to
1.27x at 141 batches (from prior 1.07x at 128 batches).
Documents the (health) op + *worker-health* cache + VRAM-ranked
pick shipped in lumbda 58fd787. Three subsections:
- Worker side: (health) op shape with measured numbers
- Client side: cache TTL (5 s ok / 30 s down) + ranking logic
- Failure handling: dispatch errors flip workers down;
with-exception-handler wraps probes so a dead peer never
aborts a multi-worker iteration
Integrated into bend.html's fleet narrative — backward-compat
note so older workers without (health) still register as
available with vram=0.
Bullets stay technical & terse; no commit hashes, no internal
implementation quirks (eq? 0 #f / Python exception propagation
stay in our commit log).
Direct C translations of our Python-tier primitives at lumbda.py
_walk_circuit_ops / _op_specs_to_bytes / _count_lumbda_ops (commit
99701c8). Same algorithms; native dispatch via interned-Value identity
on cached symbol globals.
NO_SLOT representation differs from Python tier. Python stores the
literal 18446744073709551615 fixnum (arbitrary-precision int). C-tier
fixnums cap at 48 bits via NaN-boxing PAYLOAD_MASK, so we substitute
VAL_FALSE as our slot sentinel inside op-spec vectors; pack_u64_slot
writes 0xFFFFFFFFFFFFFFFF whenever it sees VAL_FALSE. Both tiers
produce byte-identical QECCOPS1 output.
API:
- (walk-circuit-ops registers ops) -> list of 7-element op-spec vectors
- (op-specs->bytes specs) -> latin-1 string of 56*N bytes
- (count-lumbda-ops ops) -> 3-element vector (toffoli clifford total)
Symbol cache (g_sym_ccx / g_sym_x / ...) initializes lazily on first
call; intern() is idempotent so repeat init costs nothing. Layout uses
make_hashtable + ht_set/ht_ref/ht_delete for O(1) qubit-base lookup
matching our Python dict-based implementation.
Measured wall on foxhop ecdsa's canonical p=11 textbook+refined emit
pair inside our QEMU guest:
Python tier: 6.9 sec (baseline after primitive lift)
C tier: 0.71 sec (9.7x over Python)
Output verified byte-identical against Python-tier reference files via
cmp on both textbook & refined paths.
Asm tier port deferred. Asm tier's documented role serves bulk 9024-
shot validation (simulator runs against an emitted ops.bin, 160x Python
on portal round-trip per whitepaper s6.6.4) — emit pipeline targets
Python / C tier. ~700-900 lines of hand-written x86_64 asm + QEMU
debug cycles, 2-4 day effort, no current asm-tier emit consumer.
Adds a (health) op handler on the worker side & a lazy-refresh
health cache + VRAM-aware selection on the bend client side.
WORKER (gpu-worker.lsp)
(health) returns (ok (load-avg L) (vram-free-mb V) (uptime-ms U))
- L from /proc/loadavg first field
- V from `nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits`
(returns 0 when nvidia-smi missing — host w/o NVIDIA GPU)
- U from current-time-ms; client detects a worker that hung
& restarted between probes via uptime jump
Backward-compat: workers without (health) return
(error (unknown-op health)); client treats that as ok+vram=0.
CLIENT (bend.lsp)
*worker-health* alist keyed "host:port" → (last-checked-ms status vram-mb)
Cache TTL on ok = 5 s; cooldown on down = 30 s.
bend-pick-worker now:
- filters out workers in down-cooldown
- sorts healthy peers by free VRAM descending
- falls back to round-robin if every worker is in cooldown
bend-dispatch-to-gpu flips workers to down on tcp-connect-fail
or empty-reply so a transient failure costs at most one call.
Two lumbda quirks caught while building:
- (eq? 0 #f) → #t in lumbda. worker-probe-health returns 0
(a number) for the unknown-op fallback, but if we'd checked
(eq? vram #f) we would have mis-marked the worker down.
Now uses (number? vram) instead.
- tcp-connect raises a Python ConnectionRefusedError (NOT a
LispErr) on dead-host probes. lumbda's `guard` only catches
LispErr; only with-exception-handler catches Python
exceptions. Probe now wraps via with-exception-handler so a
single dead worker never aborts a fleet iteration.
Smoke on Python tier:
mixed (127.0.0.1:1 dead + 3090-ai live) → cache shows down for
the dead one (30s cooldown), ok for live (22777 MB free VRAM,
measured by the worker's nvidia-smi probe).
