Tab switching during a long-running eval used to silently abandon
the calc — output stopped streaming, no snapshot, nothing to come
back to. Now setActiveTab pauses the outgoing tab's eval (and
optionally portal-saves the env), terminates the worker, and on
re-entry hydrates + re-fires the original input.
Pieces:
* serve-coop.py + make serve-repl — dev server that emits
Cross-Origin-Opener-Policy: same-origin and
Cross-Origin-Embedder-Policy: require-corp so SharedArrayBuffer
is constructable in the browser. Same headers production needs.
* C tier eval-loop pause poll — c/eval.c grows lumbda_check_pause(),
guarded by #ifdef LUMBDA_WASM. Called at the top of leval()'s
while(1); masked to every 1024th iteration so the polling cost
stays under noise floor. When the JS-library import
js_lumbda_pause_requested returns 1, lisp_error("paused")
longjmps out so module-global env survives intact for the
portal-snapshot that follows.
* SAB plumbing — main thread allocates new SharedArrayBuffer(4),
hands it through worker config → runner.setPauseFlag →
lumbda-c.loader.setPauseFlag → globalThis._lumbdaCPauseFlag.
Atomics.store / Atomics.load on index 0 is the signalling
channel. Falls back to null when COOP/COEP isn't isolated, in
which case pause degrades to a hard worker.terminate().
* autoPauseTab() — on setActiveTab away, snapshots the tier
(C tier with SAB) or hard-cancels (other tiers / no SAB),
stashes tab.autoPause = {tier, blob, inputSrc, savedAt},
terminates the workers so the heap is reclaimed.
* autoResumeTab() — on setActiveTab into a tab with autoPause,
reboots the tier, hydrates MEMFS, runs (portal-load! ...), then
re-fires the original input via sendInput so the eval restarts
from the saved state. Asm + Python paths re-run from scratch
until their poll sites land.
Also closes two UX papercuts from fox: chip ⇣ export icon bumped
from 0.85em muted to 1em green so it's actually discoverable; the
scope toggle now reads "scope: this tab" / "scope: all tabs" so the
button label describes the state rather than a target.
When inner (let loop ((i ...))) shadows an outer (let loop ((a ...) (b ...))),
the inner named-let block was overwriting j->loop_slots[] without saving
the outer's slot positions. After the inner block restored loop_sym /
loop_nparams / loop_params, the outer's recursive call (loop new-a new-b)
would write the new args into the inner's stale slot positions instead
of the outer's slots, causing the outer body to see stale binding values
or trigger 'set! undefined' on tail-call args.
loop_slots is a member ARRAY (not pointer) — memcpy'd at line 876 when
setting up loop context — so the existing pointer save/restore for
loop_params didn't cover it.
Verified:
- jit_named_let_factorial still PASS (3628800)
- new nested-shadowing test: outer 3-param + 4 inner 1-param now PASS
- exact replica of squaring.lsp round84-fold structure now PASS
Bug surfaced when investigating test-round84-keep-quotient-product
failure in www.foxhop.net/ecdsa tests. Test still has a deeper substrate
bug beyond this JIT fix (width 0 ancilla in non-fast mode), but this
fix is independently correct + closes the loop_slots scope leak.
Both bytecode VMs had a latent O(n^2) defect on self-recursive tail
calls invoked from inside let/let*/letrec/letrec*/do bodies. The
self-tail-call op assumed reusing "current env" was safe, but current
env was the innermost let* frame, not the lambda body env. Each iter
pushed a fresh let* frame on top (PUSH_ENV at compile site), the
self-tail-call rebound params into that frame & jumped to ip=0 without
unwinding. Env chain grew linearly with iters; every var lookup walked
O(n) chain; effective O(n^2) behaviour.
Symptom observed 2026-06-14: 156k circ-ops walk hung > 5min instead of
1.4s. K=5 doctrine reducers ran 30+ runaway lumbda procs at 99% CPU
across multiple `make sweep-doctrine` invocations before we tracked
it back to language layer (initially misdiagnosed as K=5 substrate).
Fix: track scope depth at compile time on CodeObj (scope_depth bumped
on PUSH_ENV emit, decremented on POP_ENV emit). Record self_base at
lambda body entry (0 unless internal defines pushed a frame). At
self-tail-call emit, encode pops_needed = scope_depth - self_base in
the op arg. Runtime handler unwinds that many env frames before
rebinding params + jumping to ip=0.
Tree-walker (c/lumbda without --fast) already worked - it walks the
ast & lets recursion clean up frames naturally. Asm tier also fine -
no self-tail-call op, uses different lambda-call convention.
Verification:
python tier: 571 tests PASS, our 100k let* repro 1.04s wall (was infinite)
c tier: 205 tests PASS, same repro 0.05s wall (was infinite)
asm tier: 158 tests PASS (no fix needed, never had the bug)
Portal-resume backwards-compat: pre-fix portals stored OP_SELF_TAIL_CALL
arg as 2-tuple. Deserializer fills pops=0 when 'pops' key is absent,
so an old portal resumes at correct behaviour at the cost of slow walk
on its very next self-tail-call body (no worse than pre-fix).
Memory note saved at reference_lumbda_let_star_in_tail_loop in our
foxhop blackops memory for future agents.
num_div for two integers used to fall back to double when the
quotient wasn't exact. R7RS / python lumbda require exact-in →
exact-out for /. Fixed: the rational_normalize path was already
wired for the is_exact branch; the int/int branch now calls it
too instead of make_double.
(/ 67 7) → 67/7 (was 9.5714285714285712)
(/ 1 3) → 1/3 (was 0.33333…)
(/ 6 2) → 3 (exact stays integer)
(+ 1/3 1/6) → 1/2 (rational arithmetic propagates)
C native + C-WASM tier now match python lumbda on / between integers.
asm tier rationals remain pending — that needs bignums in asm first.
native c-test: 205/205 still passes.
Substrate for fork-per-accept pattern in gpu-worker.lsp — enables
async bend dispatch with internal load balancing.
c-tier (builtins.c):
- bi_fork_self: fork() wrapper, returns 0 in child / pid in parent / #f on fail
- bi_waitpid_nonblock: waitpid(-1, WNOHANG), returns reaped pid or 0
- bi_exit_immediate: _exit() wrapper — REQUIRED in fork-self children,
regular exit() runs atexit handlers against shared parent state and
hangs the child (observed empirically 2026-06-11 via vm-runner.sh).
- bi_sleep: real wall-clock sleep(3) — yields CPU. Replaces busy-loop
patterns that would (a) burn CPU and (b) SIGKILL in cgroup-limited
VMs (observed: 100M iter let-loop SIGKILL'd after 5s in qemu vm).
python-tier (lumbda.py): _fork_self / _waitpid_nonblock / _exit_immediate
/ _sleep mirrors via os.fork / os.waitpid / os._exit / time.sleep.
Tested in vm-runner.sh VM (Ubuntu 2G/2vCPU): 3-child fork-cycle test
spawns + reaps cleanly 3/3 in both c-tier + python-tier. The exact
test pattern that crashed neoblanka host pre-fix now works fine.
Asm tier: deferred. Lock retained at chmod a-x ~/git/lumbda/asm/lumbda*
per CLAUDE.md threat model.
Lets bend (examples/cuda-fanout/gpu-worker.lsp) run multiple worker
processes behind a single listening port. Each worker calls
tcp-listen on the same port; the kernel distributes incoming
connections across the bound sockets.
Foxhop production use case: 2 bend workers per GPU host (3090 + 4090)
to consume the .ready queue at 2x throughput without an external
load balancer.
ecdsa's Phase B emit at full secp256k1 width writes a 4–6 GB QECCOPS1
ops binary. The existing path — accumulate in a string-output port,
materialize via get-output-string, write — peaks RAM at 3× body
size (port internal buffer + Scheme string copy + write-binary-file
concat). A 6 GB body needs ~18 GB transient; OOMs a 16 GB QEMU guest.
This commit shifts emit-stream onto a constant-RAM file-port path
and fixes binary-correctness defects in the supporting primitives.
New primitives (mirrored across c/builtins.c + lumbda.py):
- open-binary-output-file path
Opens in "w+b" so the caller can seek back to rewrite a header.
- port-set-position! port offset
fseek absolute offset on a file port. emit-stream reserves a
16-byte placeholder header, streams the body, then seeks back to
byte 0 to rewrite the QECCOPS1 + n_ops u64 LE once n_ops is known.
- append-binary-file path data
Opens in "ab" and fwrite's the bytes through. Pairs with
write-binary-file so callers can land header + body in two writes
instead of (string-append header body).
- append-port-to-binary-file path port
Streams a string-output port's buffer to disk via fwrite without
materializing (get-output-string port). Lets callers keep their
existing string-output sink and avoid the body-size string copy
if they stay on string-port emit.
Binary-correctness fixes:
- bi_write_string to a file port used fputs, which calls strlen.
Binary payloads containing 0x00 truncated at the first null byte.
Switched the file-port branch to fwrite with the string's known
->len (same fix family as the earlier bi_get_output_string
strlen defect).
- bi_write_char per-byte fflush guarded to stdout only. With
millions of gate-bytes per second, flushing after every fputc to
a file port was a 100× slowdown. File ports buffer until close
or explicit flush-port — keep stdout's per-byte feedback path,
drop fflush on every file-port byte.
- port_write_str grows 1.5× past 256 MB instead of 2× throughout.
At realloc time the transient peak is old + new; 2× at 8 GB →
16 GB transient needs 24 GB. 1.5× bounds peak at 2.5× and keeps
multi-GB string-port workloads inside a 16 GB VM.
Tests: 88/88 c-test, 4/4 regression-named-let-leak, 205/205
functional, zoe-favorites all tiers. Binary roundtrip with embedded
nulls at 10/1000/100000 bytes passes byte-for-byte.
End-to-end: foxhop ecdsa DIALOG_GCD secp256k1 emit lands a 4.7 GB
binary at 322 MB peak RSS in 7:41 wall on a 16 GB QEMU guest.
Two C-tier defects surfaced when running foxhop's ecdsa Phase B emit
through the lumbda C interpreter. Both produced wrong bytes in the
generated QECCOPS1 binary; both Python tier handled correctly.
1. string-ref on a byte >= 0x80 returned a char with codepoint -1.
The store was a `char` array (signed on x86_64); `s->data[idx]`
sign-extends 0xFF into a negative int before VAL_CHAR wraps it.
Round-tripping (char->integer (string-ref s 0)) for a 0xFF byte
gave -1 instead of 255. The Scheme-level (u64-le n) packer uses
(make-string 1 (integer->char (modulo v 256))) for each byte and
reads them back; sign-extension corrupted the high-bit bytes.
Cast through `unsigned char` in bi_string_ref.
2. pack_u64_slot treated bignum slots as raw fixnums.
ecdsa's emit-stream binds (no-slot *no-slot*) where *no-slot* is
18446744073709551615 (u64 max). On Python tier that's a regular
big-int. C tier carries it as a bignum NaN-box slot. The packer
knew about VAL_FALSE → 0xFFFF... but called as_int(v) on bignums,
reading the NaN-box payload bits (pointer-to-Bignum) and writing
that pointer as the field's 64-bit LE value.
Add an IS_BIGNUM branch that extracts the low 64 magnitude bits
directly. Cross-tier emit-stream code stays unchanged.
Tests: 88/88 c-test, 4/4 regression-named-let-leak, 205/205
functional. DIALOG_GCD smoke at p=11 n+1=5 now byte-identical to
Python tier (1,941,816 bytes match exactly).
Two pre-existing defects in c/builtins.c made (open-output-string)
unusable for binary emit:
- bi_get_output_string ran the buffer through make_string_from_cstr,
which calls strlen. Any 0x00 in the payload truncated body at
that byte. Use port's known str_len directly via make_string.
- bi_write_char ignored string-port destinations entirely — it
pulled AS_PORT(p)->fp (NULL for string ports), fell back to
stdout, and silently routed gate bytes to terminal output
instead of the port buffer. Route to port_write_str when target
port kind is PORT_STRING, matching bi_write_string's behavior.
Surfaced while validating the precise-GC fix against ecdsa's
Phase B emit (writes 56-byte op records full of embedded nulls
through a string-output port, get-output-string at the end).
With these fixes ecdsa's n+1=64 emit drops a 333 MB binary in
42.6 seconds — pre-fix it timed out at 600 s with a 17-byte
header-only file (strlen truncated body at byte 1; the visible
gate bytes had been escaping to stdout the whole time).
Binary roundtrip test (write n bytes alternating x / 0x00 to
output-string port, read back via get-output-string):
n=10 len=10 ok
n=1000 len=1000 ok
n=100000 len=100000 ok
All upstream tests still pass (88/88 c-test, 4/4 regression,
205/205 functional, zoe across tiers).
Boehm's conservative pointer scan cannot recognize lumbda's Value
layout — heap pointers live in the low 48 bits with QNAN + tag bits
in the upper mantissa, so a raw word never looks like a heap address.
Until now main.c neutralized this with GC_disable(): every allocation
leaked, OOMing any long-running workload.
Add precise tracing via a custom Boehm kind:
- New c/gc.c: mark proc walks 8-byte words in mixed mode — when the
QNAN bits are set with a pointer-bearing tag (0/2/4/5/6) extract
the low-48 pointer; otherwise fall through to raw-pointer
validation. GC_set_push_other_roots callback decodes NaN-boxed
Values on the C stack via setjmp anchor + scan up to the stack
base captured at process start.
- Allocations holding Values (Pair, Env bindings, ValueStack data,
ULVector data, HTEntry, Proc params + body, FullCont stack,
CodeObj instrs, SymbolEntry) route through lumbda_value_malloc.
Pure-byte sites (bignum limbs, char buffers, source files) stay
on regular GC_MALLOC.
- main.c / test.c / bench.c capture stack-base then drop GC_disable.
types.c also zeros popped slots on the value stack so stale pointers
do not survive a vs_pop and pin freed objects — independent
correctness fix that pays off once GC actually runs.
Build: USE_GC=1 (default when /usr/include/gc.h exists).
Tests with GC enabled:
- 88/88 c-test
- 4/4 regression-named-let-leak (test that motivated GC_disable)
- 205/205 functional (Python + C)
- zoe-favorites all tiers (Python + C + asm + asm-full)
alloc-test 1M cons drop-loop:
- Before: 0.60s wall, 156 MB RSS, leaks every cell
- After: 0.37s wall, 4 MB RSS, ~1500 GC cycles each freeing ~370 KB
Ports lumbda.py _emit_circuit_to_ops_bin_stream to C tier. Walks a
Scheme registers + ops list, writes each 56-byte QECCOPS1 op record
straight to disk via fopen/fwrite, then seeks back to patch the n_ops
header at the tail. O(1) host memory regardless of n_ops.
Cross-tier byte-identity verified:
* 500-op mixed-tag synthetic circuit on host: Python ↔ C identical
* real-point-add n+1=5 (p=11, 94,214 ops, 5.0 MB) inside VM
* real-point-add n+1=9 (p=251, 674,872 ops, 36 MB) inside VM
sha256 matches every width.
Speedup on equal Scheme source (build + emit combined):
* n+1=5: Python 56.8 s → C 1.1 s (52×)
* n+1=9: Python 365 s → C 8.4 s (43×)
Combined ratio exceeds the prior 11-16× C-tier envelope because the
build phase (Phase B mod-arith construction) also accelerates on C;
emit-only ratio is ~3-9× and grows with op count.
C test suite: 85 → 88 passing (emit-stream-basic, alloc-free, empty).
Shared functional suite: 205/205 still passing on both tiers.
Implementation notes:
* pack_op_record packs u32 kind + u32 pad + 6× u64 LE, matching
op-specs->bytes byte layout exactly.
* Layout hashtable Symbol → fixnum base, mirroring walk-circuit-ops.
* NO_SLOT sentinel: 0xFFFFFFFFFFFFFFFF written directly to u64 slots
that the Scheme side did not populate.
* libc's default fwrite buffer (~4 KB) handles batching at ~73 ops
per write — same throughput class as Python tier's 8 KiB list batch.
Adds tagged bignum support alongside the existing 48-bit fixnum on the C
tier. Tag 6 = bignum, heap struct sign-magnitude with u64 little-endian
limbs. Reader emits bignums for any literal past the fixnum range; +, -,
*, quotient, remainder, modulo, expt, =, <, >, abs, odd?, even?,
integer?, exact?, number->string, string->number all promote fixnum →
bignum on overflow & demote back when results fit. Boehm GC owns every
allocation. Schoolbook O(n²) mul + shift-subtract divmod is sufficient
at our 4-limb / 256-bit scale.
Before: (expt 2 48) = 0, (expt 2 256) = 0, secp256k1-p = -4294968273.
After: all three return their exact arbitrary-precision values, matching
Python tier byte-for-byte.
Validated:
- c/test.c — 85/85 pass (+2 new bignum unit tests).
- tests/functional.lsp — 205/205 pass on both C & Python tiers.
- tests/bignum-cross-tier.lsp — 33/33 pass byte-identical on both tiers
(diff produces no output).
- ecdsa/runs/lumbda-sweep-003/c-tier-bignum-probe.lsp — all four
assertions now match the Python oracle.
- ecdsa Phase B byte-identity sweep inside QEMU guest:
n+1=9 p=251 sha256 c668bbe3... — matches Python oracle.
n+1=18 p=131071 sha256 8a031f96... — matches Python oracle.
n+1=33 p=2³²-5 sha256 0bc56905... — matches Python oracle.
Previously the n+1=33 C tier emitted sha256 b024d6d9... (26,078 fewer
Toffolis due to silent fixnum wrap). Bignums close that gate.
secp256k1 production-width emit (n+1=257) is now structurally unblocked
on C tier; downstream agent (#55) drives that next-step on the ecdsa
side. Asm tier inherits in a follow-up port.
Direct C translations of our Python-tier primitives at lumbda.py
_walk_circuit_ops / _op_specs_to_bytes / _count_lumbda_ops (commit
99701c8). Same algorithms; native dispatch via interned-Value identity
on cached symbol globals.
NO_SLOT representation differs from Python tier. Python stores the
literal 18446744073709551615 fixnum (arbitrary-precision int). C-tier
fixnums cap at 48 bits via NaN-boxing PAYLOAD_MASK, so we substitute
VAL_FALSE as our slot sentinel inside op-spec vectors; pack_u64_slot
writes 0xFFFFFFFFFFFFFFFF whenever it sees VAL_FALSE. Both tiers
produce byte-identical QECCOPS1 output.
API:
- (walk-circuit-ops registers ops) -> list of 7-element op-spec vectors
- (op-specs->bytes specs) -> latin-1 string of 56*N bytes
- (count-lumbda-ops ops) -> 3-element vector (toffoli clifford total)
Symbol cache (g_sym_ccx / g_sym_x / ...) initializes lazily on first
call; intern() is idempotent so repeat init costs nothing. Layout uses
make_hashtable + ht_set/ht_ref/ht_delete for O(1) qubit-base lookup
matching our Python dict-based implementation.
Measured wall on foxhop ecdsa's canonical p=11 textbook+refined emit
pair inside our QEMU guest:
Python tier: 6.9 sec (baseline after primitive lift)
C tier: 0.71 sec (9.7x over Python)
Output verified byte-identical against Python-tier reference files via
cmp on both textbook & refined paths.
Asm tier port deferred. Asm tier's documented role serves bulk 9024-
shot validation (simulator runs against an emitted ops.bin, 160x Python
on portal round-trip per whitepaper s6.6.4) — emit pipeline targets
Python / C tier. ~700-900 lines of hand-written x86_64 asm + QEMU
debug cycles, 2-4 day effort, no current asm-tier emit consumer.
The S-expression wire format was the bottleneck at huge payload sizes
-- 23.8 s end-to-end for 1M x 16 B inputs on the Python tier, while
the actual CUDA kernel finishes the same workload in ~47 ms. The
hex-S-exp parser ate everything between.
New binary wire mode (magic 'BSHK' prefix; payload is the daemon's
binary portal format verbatim) bypasses S-expression parsing entirely.
Worker writes the blob to disk, calls daemon process-bin, reads result,
prepends 'BSHR' magic, replies.
Measured 3090-ai, daemon warm, localhost:
workload Py S-exp Py binary C S-exp C binary
100 x 16 B 3.43 ms 0.74 ms 0.40 ms 0.15 ms
1k x 16 B 23.24 ms 0.76 ms 2.77 ms 0.22 ms
10k x 16 B 218.82 ms 1.27 ms CLIFF 0.88 ms
100k x 16 B 2,219 ms 10.18 ms CLIFF 10.35 ms
1M x 16 B 23,811 ms 159 ms CLIFF 157 ms
150x speedup at 1M inputs on Python tier. C tier S-exp CLIFFs
between 1k and 10k inputs (reader payload limit); binary mode
bypasses the CLIFF entirely. At 100k+ inputs both tiers converge
since file I/O + CUDA kernel dominates over wire framing.
Host comparison: hashlib.shake_256 over 1M tiny inputs takes ~2 s
on a single Python core. Bend via binary worker = 157 ms = 12x
faster than host. Bend now wins at huge workloads, not just heavy
ones.
Implementation:
lumbda.py
* tcp-send/tcp-recv switched to latin-1 (1:1 byte mapping)
so binary payloads pass through cleanly. UTF-8 was mangling
bytes with replacement chars.
* write-binary-file / read-binary-file primitives.
c/builtins.c
* write-binary-file / read-binary-file matching Python tier.
examples/cuda-fanout/wire.lsp
* wire-send-raw / wire-recv-raw helpers that frame a raw
payload string without S-expression serialization.
examples/cuda-fanout/gpu-worker.lsp
* handle-binary-shake: write portal blob, daemon process-bin,
read result, wire-send 'BSHR' + bytes.
* handle-one dispatches on first 4 bytes of payload: 'BSHK'
goes to binary path, anything else stays S-exp.
examples/cuda-fanout/bench_tiers.py
* make_payload_binary builds the BSHK protocol payload.
* --binary flag in CLI.
www/index.html
* full S-exp + binary comparison table.
* 'bend now beats host hashlib at huge workloads' headline finding.
Two new primitives in builtins.c, paralleling the Python tier shipped in
the previous commit. gpu-worker.lsp now runs on the C tier byte-identically
to the Python tier.
(spawn-process-stdio path args) → (stdin-port . stdout-port)
fork + pipe + execvp; child's stdin & stdout wired back to parent
as line-buffered FILE* ports. Accepts both strings and symbols in
the args list (matches Python tier's permissive conversion).
(flush-port port)
fflush() on the port's FILE*. No-op when fp is null.
End-to-end on 3090-ai with C-tier lumbda everywhere:
shell A: ./lumbda /tmp/launch-c.lsp
→ gpu-worker: ready cuda-shake-fanout ← ./shake256-fanout
→ gpu-worker listening on port 9091
shell B: ./lumbda smoke-bend.lsp # run 3×
=== smoke-bend ===
1. cost estimator picks local for 3 inputs: OK
2. worker available? #t
3. bend! (cuda-shake-fanout '("00" "01" "deadbeef") 32):
(b8d01df855… 94da6280b2… fa094fa86e…)
Three runs identical bytes. Same hashes as Python tier. Same hashes as
hashlib.shake_256 host reference.
Cross-tier matrix (proves wire protocol is tier-agnostic):
client tier worker tier status
─────────────────────────────────────
C tier C tier PASS — 2 sequential runs, byte-identical
Python tier C tier PASS — same hashes
C tier Python tier implicit by symmetry (same wire bytes
both directions; Python-server tested
against Python-client in prior commit)
Per-tier status after this commit:
Python tier ✓ end-to-end
C tier ✓ end-to-end + cross-tier byte-identical to Python tier
asm tier → still needs spawn-process-stdio via raw fork+pipe+
execve syscalls. Scheme files unchanged.
Bash process substitution <(...) passes /proc/self/fd/N — a pipe, not
a regular file. load_file used fseek(SEEK_END)+ftell to size a single-
read buffer; on a pipe ftell returns -1, which casts to SIZE_MAX as
fread's nbyte argument and blows the heap. Glibc fortify caught it
as '*** buffer overflow detected ***'.
Detect non-seekable input via the fseek return code and fall back to
a doubling growable buffer instead. Seekable path unchanged.
Repro: ~/git/lumbda/c/lumbda <(echo '(display 1)(newline)')
Resolved by 45a90b8 (vm restores cur_code across CALL/RETURN).
The reproducer at lines 5-11 of the TODO file runs cleanly now
under both `lumbda --fast` and `lumbda -j --fast`, and the
proof/eml_proof_in_lumbda.lsp `normalize` function no longer
needs the `(define (iter ...))` workaround that the TODO
documented. Future regressions are caught by
tests/regression-named-let-leak.{lsp,sh}.
Five-line fix that ends the F1 hang in foxhop.net
ecdsa/tests/unit/probe-c-confirm.lsp.
Symptom — under --fast, a defined function whose body is a
tail-recursive named-let that calls another user-defined function
per iteration loops forever at 100 percent CPU. Trace pins the
bytecode dispatch:
walk1.body: PUSHE MKCLO DUP BIND LOOKUP TCALL ->loop
loop: LOOKUP NULL? JIF LOOKUP CALL ->always-true
always-true: CONST RET (returns #t)
always-true (!): JIF LOOKUP CDR STAIL -> ip=0 of always-true (!)
loop forever
cur_code was the call-frame-local register holding the currently
executing CodeObj. OP_CALL updated it on entry but neither OP_RETURN
nor the builtin-fallback restore path in OP_TAIL_CALL put it back
on return. Subsequent OP_SELF_TAIL_CALL read cur_code->self_params
from the still-stale callee proc (NULL for always-true since it has
no named-let), guard skipped the env rebind, then set ip=0 — without
ever updating the loop variable. Loop variable stayed pinned at the
initial list and our walk never reached its base case.
Fix — VMFrame gains a cur_code field. Three frame-push sites save
it on entry (OP_CALL, OP_TAIL_CALL builtin fallback frame-restore,
OP_CALL_CC compiled-proc entry); two frame-pop sites restore it on
return (OP_RETURN, OP_TAIL_CALL builtin fallback).
Verified inside foxhop.net's ecdsa QEMU guest:
- foxhop.net/ecdsa/tests/unit/probe-c-confirm.lsp F1..F4 — all pass
- foxhop.net/ecdsa/tests/unit/test-sim.lsp under --fast — 21/21 pass
- foxhop.net/ecdsa/lumbda/main.lsp under --fast — 6 shots, score 18,
byte-identical with our Python tier
- make functional-test — 205/205 on Python and C tiers
- make regression-named-let-leak — 4/4 across Python, tree-walker,
and --fast JIT
c/TODO-named-let-bytecode.md can stop applying its `(define (iter ...))`
workaround once this lands.
Three coupled changes that unblock the ecdsa quantum-circuit
simulator's run on the C tier from neoblanka.
1. c/Makefile autodetects libgc-dev — if /usr/include/gc.h is
present, the build links Boehm and defines USE_BOEHM_GC. Without
GC, ul_free is a no-op (lumbda.h:35) and every allocation leaks;
small REPL snippets work but workloads with thousands of envs
OOM the process. Override with USE_GC=0 to force the malloc-only
path for diagnostics.
2. c/main.c calls GC_INIT before init_symbols, then GC_disable.
GC_INIT registers the stack base for conservative scan — without
it some Linux configs miss roots. GC_disable is a deliberate
stopgap: lumbda Values are NaN-boxed pointers that conservative
Boehm cannot recognize as pointers, so live targets get reclaimed
(env binding symbol payloads, SymbolEntry strings) and lookups
fail with "undefined: <sym>". Reproducing this without GC_disable
on the GC build: any sim.lsp call chain triggers the corruption
after ~100 named-let iterations. Until tracing is precise,
growing the heap is safer than wrong results. Long-running
workloads run under ulimit -v.
3. c/builtins.c gains rename-file and delete-file matching the
Python tier (lumbda.py:3468). sim.lsp's write-portal! pattern
(write to .tmp, rename) needs rename-file to land cross-tier
identical results.
4. tests/regression-named-let-leak.lsp + .sh pin four shapes that
blew up ecdsa: the c/TODO-named-let-bytecode.md repro, the F1
shape from foxhop.net's lumbda-c-tier-leak-SP.md (12-line
minimum), a 200-iter scaled variant, and a sim.lsp run-ops!
mirror. Wired into root Makefile as regression-named-let-leak;
added to test-all. Wrapper caps memory at 256 MB virt and 15s
per tier so a leak regression fails the run instead of consuming
host RAM.
Known limits:
- --fast JIT still has the named-let + inner user-fn call hang
(separate TODO; tree-walker handles this fine).
- GC_disable means the heap grows; workloads must bound their work
budget. ecdsa's sim runs comfortably in 5 MB.
Verified inside a 2G/2vCPU QEMU guest (foxhop.net ecdsa/vm-runner.sh):
- test-c (tree-walker) — 35/35 PASS
- bench-c (tree-walker) — score 18 matches Python tier byte-identical
- F1 probe (tree-walker) — all four steps PASS
Zoë Trout's favorites at wedgewack.org/ursa.lisp.txt are Common Lisp:
iterative LOOP macros, setf cascades, defun with &optional, image-
based stone-lisp culture. Her first contribution to lumbda was a
question — "do we care for our programs, and how long are they alive
for?" — and the answer now extends beyond the RNG portal (§7.5) to
iteration style itself.
Four-phase delivery, all under ticket 0004:
Phase A — idiomatic Scheme ports at examples/ursa-scheme.lsp.
Every Zoë defun rewritten as named-let + tail recursion + list-
backed work queue + type-predicate dispatch.
Phase B — CL compat shim at cl-compat.lsp.
defun (with &optional), setf (simple vars, multi-pair), flet,
multiple-value-bind, t / nil (nil=#f so cond/if compose),
evenp/oddp/plusp/minusp/zerop, mod/ash/logbitp/nreverse,
cl-when/cl-unless (plain when is a void-returning lumbda special
form), declare (no-op), cddddr (missing accessor).
Phase C — cl-loop macro covering 14 patterns.
while/until/repeat, for VAR from A to/below/downto B, for VAR =
INIT [then STEP], for VAR across VEC, of-type T, do, when/unless
return, finally (return VAL). Sequential do*-style stepping via
gensym + cl-subst. Look-ahead termination so `repeat 4 for s = 4
then (- (* s s) 2) finally (return s)` returns 37634 (pre-step)
rather than 1416317954 (post-step). Every expansion ends in a
named-let tail call — TCO holds for loops of any length.
Phase D — load examples/ursa.lisp.txt with minimal annotation.
Preserves Zoë's CL. Minimal edits documented in file header:
load cl-compat.lsp, loop→cl-loop, when→cl-when, random→random-int,
&key→&optional. rho/digits omitted (need make-array/CLOS — see
ticket 0004 for scope boundary).
Defect uncovered along the way (c/types.c env_lookup): a "global
shortcut" checked global env immediately after missing the local
frame, SKIPPING intermediate parent scopes. Broke lexical scoping
whenever a parent scope shadowed a global. Reproduced with
(define s 4)
(let ((s 100)) (let ((m 0)) s)) ; returned 4, should return 100
Any nested let whose body referenced a shadowed name silently read
the global. Fix: remove the shortcut, walk the parent chain end-to-
end. 1255 assertions across five suites pass unchanged after fix —
surfaced only because cl-loop iterator names routinely collide with
globals accumulated in a stone-lisp image.
Whitepaper §9.2 documents the CL-in-Scheme design and the guarantees
that survive (TCO, portal determinism, cross-impl reproducibility).
Zoë added to authors + acknowledgments; reacknowledgment reframes
her first contribution as the deeper program-lifetime question, with
RNG portal as a derivative (§7.5) and cl-loop as the follow-up.
Tests: tests/cl-compat.lsp (44 assertions) and tests/ursa.lsp (28
assertions) exercise both paths under Python + C via tests/zoe-
favorites-test.sh, wired into make test-all.
MOAD notes: unmoad flags memq/assq in cl-compat.lsp over cl-loop-
keywords (~30 elements, constant) and var->new (≤4 state vars per
loop). Both are macro-expansion-time, bounded-small-N — not runtime
hot paths. Pre-existing c/types.c findings (strcmp-in-loop for
record-type lookup) are not from this change.
Ticket 0002 — reads 8 bytes from /dev/urandom (little-endian u64) and
seeds xoshiro256**. Opt-in kernel entropy for stochastic runs; the
default stays deterministic (k=0 at startup), so ticket 0001's
portal-reproducibility contract is unchanged.
Real-world flow now one call away:
Machine A: (random-seed-from-os!) + run simulation + portal-save
Machine B: portal-resume — same stream, bit-for-bit
All three impls fail loud on /dev/urandom trouble (LispErr in Python
and C, stderr + exit(1) in asm) — no silent fallback to a weak seed.
Tests:
- tests/functional.lsp: 2 new shared asserts (entropic + replay)
- asm/test.sh: 2 new asm-local checks (149 total, was 147)
- make test-all green across Python (205), C (205), asm (149)
Whitepaper §7.5 gains one sentence noting the OS-seed path.
unmoad: zero new findings in added code.
Completes ticket 0001 started in 27f468c. All three impls now carry
bit-identical xoshiro256**; portal state round-trips across process
boundaries in every producer x consumer cell (Python <-> C <-> asm).
asm impl:
- 4 new builtins: random-seed!, random-int, random-state, random-state!
- g_rng_state in BSS (4 x u64); rng_splitmix64_step, rng_seed, rng_next
- Binary portal header bumped LUMBDAB1/48 -> LUMBDAB2/80; carries
32 bytes of rng state at offsets 40..64, reserved moved to 72
- No float support in asm, so (random) intentionally omitted there
- _start seeds with 0 so the stream is deterministic from startup
Python + C (supplements 27f468c):
- rng_seed(0) auto-invoked at module load / register_portal_builtins
so (random) without explicit (random-seed!) returns a real value
instead of the all-zero xoshiro fixed point
Tests:
- tests/functional.lsp: 7 new shared assertions (Python + C)
- asm/test.sh: 5 new asm-local assertions (142 -> 147)
- tests/portal-rng-save.lsp / portal-rng-load.lsp: portable S-expression
portal that captures both state AND next-5 baseline so loader self-
verifies without a separate harness
- tests/portal-cross-test.sh: 9 new producer x consumer RNG cells; all
18 cells pass end-to-end
Verified: seed=42, (random-int 1000000) draws 1..10 =
558742 543102 559009 124193 317476 750584 200754 814407 344958 929085
identical in Python, C, and asm.
unmoad scan: zero new findings in added code.
Zoe's contribution question: does our portal preserve RNG state so a
simulation can continue in another process with the same random stream?
Answer today: no — no RNG existed. Answer now (Python + C): yes, bit-identical.
- New builtins: random-seed!, random, random-int, random-state, random-state!
- xoshiro256** (Blackman & Vigna 2018) — deterministic, portable, no libc rand
- State = 4 x u64; portal-v1 JSON gains 'rng' field with 8 x u32 halves
- Python and C produce bit-identical streams (verified: seed=42, 10 draws)
- Asm impl + cross-impl tests + whitepaper note: next commits
Ticket: docs/tickets/0001-portal-rng.md
Historical internal name "uncommonlisp" retired in favor of the
public name "lumbda" ahead of lumbda.com going live. Scope of
this commit:
Source files renamed:
uncommonlisp.py -> lumbda.py
asm/uncommonlisp.s -> asm/lumbda.s
c/uncommonlisp.h -> c/lumbda.h
whitepaper/uncommonlisp-whitepaper -> whitepaper/lumbda-whitepaper (.rst + .pdf)
Binaries renamed (tracked ones; c/ was always gitignored):
asm/uncommonlisp, asm/uncommonlisp-gc, asm/uncommonlisp.o,
asm/uncommonlisp-gc.o -> asm/lumbda(-gc)(.o)
c/.gitignore -> ignores lumbda
Internal string updates (sed pass ordered longest-first):
asm/uncommonlisp -> asm/lumbda
c/uncommonlisp -> c/lumbda
uncommonlisp.py -> lumbda.py
UNCOMMONLISP_BIN -> LUMBDA_BIN (asm/test.sh env var)
"uncommonlisp> " -> "lumbda> " (asm REPL prompt baked into binary)
UNCOMMONLISP -> LUMBDA (macros, comments)
uncommonlisp -> lumbda (prose)
Binary portal magic updated:
"ULPORTAL" -> "LUMBDAB1" # "Lumbda Binary v1"
Old portal files are not backward-compatible — this is a deliberate
break since it's the rename moment. S-expression portals already
carry their own ";; lumbda-portal v1" header and remain cleanly
versioned.
WHITEPAPER.pdf / WHITEPAPER.rst symlinks repointed to the renamed
files. Makefile's whitepaper target targets lumbda-whitepaper.pdf.
Not changed (intentional, separate phases):
- Filesystem directory /home/fox/git/uncommonlisp itself
(fox renames locally and the gitlab repo URL in a follow-up)
- tests.py hardcoded cwd=/home/fox/git/uncommonlisp
(matches the current on-disk location; will flip when the
directory rename ships)
- Git history (immutable; old commits still say uncommonlisp,
which is correct — that's what they were)
Verified:
137 asm no-GC + 137 asm GC + 571 Python + 83 C + 189 shared
functional tests all pass under the new names.
bench-gc-http (2000 req): all 4 cells behave as expected
(cells 1/2 flat, 3 leaks, 4 bounded at 1 chunk).
Python REPL, C REPL, asm REPL all start cleanly.
Hunted the C --fast compiler bug that was hanging on the EML proof.
Narrowed to a specific pattern:
(let loop ((t start))
(let ((next (fn t)))
(if next (loop next) t)))
A named-let whose body is (let ((x (...))) (if x (recurse x) base)).
The recursive call inside the inner let+if branch never reaches the
loop closure — hangs or segfaults.
Reproducible with a 4-line test case; filed as
c/TODO-named-let-bytecode.md with minimal repro, suspected cause
(env-chain mismatch between PUSH_ENV and TAIL_CALL), and a known-
good workaround.
Workaround landed in proof/eml_proof_in_lumbda.lsp's `normalize`:
replaced the named-let with an internal recursive `define`, which
compiles correctly under --fast. Same logic, different surface
syntax. All four Lumbda tiers now verify the proof.
Benchmark refreshed (make bench-proof):
cold cached
Lumbda asm 46 ms 7 ms
Lumbda C --fast 65 ms 9 ms
Lumbda C (tree-walker) 87 ms 12 ms
Lumbda Python --fast 651 ms 232 ms
Lean 4 722 ms 5 ms
All four tiers now green. Asm still fastest (46 ms cold vs Lean's
722 ms — ~16× faster). Cached Lumbda asm 7 ms vs Lean 5 ms (within
1.5×). The C --fast tier went from "hangs" to 65 ms cold — competitive
with asm once the compiler bug is dodged.
Whitepaper §8.6 table updated; prior "(hangs)" row is gone;
footnote on the named-let workaround links the TODO file.
ack(3,8) was reported as "segfault" for the C impl in the previous
whitepaper revision. That was a stale observation — C has --fast
(bytecode VM with explicit frame stack) that handles deep recursion
cleanly. The benchmark table compared the wrong modes.
Corrected apples-to-apples:
- Python --fast (bytecode VM) — 17,004 ms on ack(3,8)
- C --fast (bytecode VM) — 1,433 ms **fastest of the three**
- asm native (tree-walker) — 2,322 ms
C's --fast wins every workload. asm still beats Python --fast by
~7x despite being a tree-walker, because it skips Python's per-op
overhead entirely.
c/main.c: --help text updated to clarify that --fast is required
(or `ulimit -s unlimited`) for deep recursion in the default
tree-walker mode. Attempted flipping --fast to default; reverted
because that surfaced a cumulative-state buffer overflow in the
bytecode compiler that only triggers after the full 189-test
functional suite but not on isolated scripts. Left as a TODO in
the code comment. 189 C tests + full test-all still pass.
Whitepaper §6.4 table now shows all three impls in their
high-performance configuration. Also noted that a pthread-with-
larger-stack wrapper would let the C tree-walker handle deep
recursion without --fast — tracked as low-priority future work
since --fast is strictly faster regardless.
Closes the last loop promised in the whitepaper's Future Work: a
node serves its state as an S-expression portal over HTTP, another
node pulls it down with tcp-connect + tcp-recv and materializes the
bindings locally via (eval (read-from-string line)).
examples/portal-http-server.lsp (90 lines):
- Holds some state (counter, my-int, my-list, my-fib, my-str)
- GET /portal → S-expression body: a sequence of (define ...) forms
- GET / → HTML index
- Uses heap-snapshot / heap-restore for O(1) memory on asm
examples/portal-http-client.lsp (90 lines):
- tcp-connect, send HTTP/1.0 GET, receive full response
- Strip headers (walk to first \r\n\r\n)
- Split body by \n, eval each non-empty, non-comment line
- The remote bindings are now live locally
3×3 server/client matrix: all 9 combinations green. Every runtime
hosts, every runtime consumes. The wire format is Scheme source;
no schema, no JSON, no Protobuf.
Prerequisite fix: `eval` semantics aligned across all three impls.
Python and C's `eval` special form previously evaluated its result
in the CALLER's env, so a nested (eval (read-from-string
"(define x 42)")) would install x in the local function scope —
invisible to later top-level code. asm's bi_eval always used the
global env (r14). With this commit, all three impls evaluate the
eval'd result in the global env, matching asm's existing behavior.
Python: uncommonlisp.py leval eval-handler now does `env = env.g`
before continuing the trampoline.
C: c/eval.c SYM_EVAL branch now does `env = env->global`.
asm: no change (already correct).
One pre-existing Python defect surfaced by the client:
`count` is a SRFI-1-style builtin (`d(S('count'), ...)`), so a
local let-loop variable named `count` collides with it in the
inline-cache lookup path and OP_LOOK_ADD1 fires on the builtin
instead of the local. Worked around by renaming the loop
accumulator to `cnt`. Underlying Env.lookup shortcut-to-global
issue is out of scope for this commit.
Regression: 975 tests still green.
Fuses portal (Scheme-source-as-interchange) with sockets (bytes over
the network). Wire protocol: one S-expression per connection. Same
server + client .lsp runs byte-identically in Python, C, and asm.
New primitives in all three impls:
- read-from-string — parse one sexp from a string
Asm gets two more:
- symbol->string — standard R7RS, was missing
- eval — evaluate a Scheme value in the global env (Python + C had
it as a special form; asm exposes it as a builtin)
examples/rpc-server.lsp (port 9080):
- Whitelisted dispatch: ping / add / mul / fib / echo
- Never calls eval on client input; safe by construction
- Uses heap-snapshot/restore for O(1) memory on asm
- ~90 lines, portable
examples/rpc-client.lsp:
- Sends one request, reads one response, displays both
- 45 lines, portable
examples/repl-server.lsp (port 9081):
- DANGER: full remote eval. Any Scheme form accepted and evaluated
in the server's global env. Persistent across connections.
- Deliberately does NOT use heap-snapshot — remote (define x ...)
lives in the global env above any snapshot point; rewinding would
invalidate the new binding. The ulimit -v 512 MB safety net
(documented in CLAUDE.md) ensures an escaped process can't crash
the machine.
- ~70 lines, portable. Demonstrates what "the language IS the
interchange format" gets you at the limit: a single socket and
a single primitive (eval) carry a full-powered REPL.
Verified 3×3 server×client matrix: all 9 combinations green.
All 132 asm + 571 py + 189 shared + 83 c tests still pass.
One quirk discovered and worked around: in asm, a closure captures
its env chain by pointer at define time. Forward-referenced names
in mutually-recursive toplevel defines can fail under specific
heap-restore patterns — see the leaf-first ordering note in
rpc-server.lsp.
Three wins in one commit.
1) heap-snapshot / heap-restore (asm arena primitive)
asm has no GC. Long-running servers leaked ~64 MB per heap growth.
Two new builtins let a programmer capture r15 and later rewind to
it, recycling intermediate allocations in O(1) memory.
Python + C get no-op versions so portable .lsp code can call them
unconditionally.
examples/http-server.lsp now takes a snapshot at top level and
rewinds after every request. Measured asm RSS: 88 KB initial,
100 KB after 100 requests, 100 KB after 1100 requests — flat.
Prior behavior was +64 MB per few thousand requests.
2) examples/http-client-bench.lsp — native HTTP load generator
Uses only the six tcp-* primitives + current-time-ms. Runs
identically in all three impls. Eliminates curl's ~2 ms/req
fork+exec overhead, so real server throughput shows up:
Python server ← Python client 2403 rps
C server ← C client 2439 rps
asm server ← asm client 2994 rps
asm server ← C client 2500 rps
The earlier curl-based bench was clamped near 400 rps by the
client; the actual servers handle 6–7× that.
3) MOAD-0001 cleanup
- c/builtins.c bi_string_replace: strncmp-at-every-position
(hand-rolled, sedimentary) → strstr (libc-tuned, typically
Boyer-Moore-Horspool). O(N*k) → O(N + matches*k).
- uncommonlisp.py _tokenize_lines: per-token src.count('\n', 0, pos)
→ precompute line_starts once, bisect_right per token.
O(N*M) → O(M + N log M).
Also adds current-time-ms to all three impls so benchmarks can
time themselves without relying on the Python/C float `current-time`
(asm has no floats). Seconds-since-epoch tagged as a 61-bit int.
Test counts unchanged: 571 py + 132 asm + 189 shared + 83 c = 975.
All green via make test-all.
Added tcp-listen/accept/connect/recv/send/close to Python, C, and asm.
One examples/http-server.lsp runs identically in all three impls and
serves HTTP/1.0 with routing, content-type, and content-length headers.
asm additions:
- SYS_SOCKET/BIND/LISTEN/ACCEPT/CONNECT/SETSOCKOPT syscalls
- 6 tcp-* builtins using the existing port encoding (SPECIAL ≥ 1000)
- bi_tcp_connect: dotted-quad IPv4 parser, no DNS dependency
Defects fixed along the way (surfaced by the HTTP server):
- string-append: was 2-arg only; now variadic (walks arg list twice)
- number->string: was stubbed to VAL_VOID; now correctly writes digits
into a heap-allocated string (incl. negative handling)
- String-literal reader: \r and \0 escape sequences now handled (was
silently dropping backslash, treating them as literal 'r' / '0')
- tcp_accept: sockaddr buffer was 8 bytes, now 16 (was corrupting
caller's stack when accept wrote full struct sockaddr_in)
Pinocchio benchmark (tests/web-benchmark.sh):
At concurrency=20, 1000 requests, serving a 1KB body:
uncommonlisp Python 373 req/s
uncommonlisp C 370 req/s
uncommonlisp asm 370 req/s
python3 http.server 381 req/s (stdlib reference)
busybox httpd 382 req/s (production reference)
All five converge within 3% — the client (curl fork/exec) is the
bottleneck, not the server. Our single-threaded blocking servers
are indistinguishable from battle-tested ones at this load.
Binary sizes:
uncommonlisp asm 45 KB (HTTP + everything else)
busybox httpd 2.1 MB (multi-call binary)
python3 8 MB (interpreter)
The asm HTTP server is 46× smaller than busybox and 176× smaller
than Python, serves from 7 Linux syscalls, and the entire protocol
handler is 70 lines of portable Scheme.
Test counts: 132 asm (up 1), rest unchanged. All green.
Full continuations: FullCont captures frames/stack/env with deep copy.
VM trampoline via setjmp/longjmp. Multi-shot safe via deep_copy_env.
Portal (new c/portal.c): serialize env + continuation to JSON,
resume on another machine. portal-checkpoint! triggers mid-VM save.
83/83 C unit tests + 181/181 functional tests pass.
Real native machine code via mmap(PROT_EXEC). No exec(). No strings.
Raw x86_64 bytes: mov, add, sub, imul, cmp, je, jne, call, ret, jmp.
ack(3,4): 0.12ms JIT vs 1.5ms CPython vs 28ms interpreter
fib-rec(20): 0.16ms JIT vs 3.4ms CPython vs 40ms interpreter
Added cond support to JIT (cascaded comparisons → conditional jumps).
Fixed JIT cache: sentinel value prevents retry on unjittable functions.
System V AMD64 ABI: args in rdi/rsi/rdx, callee-saved r12-r15.
Tail calls use jmp (true TCO at machine code level).
691 lines of jit.c. 114 functional tests pass. All C tests pass.
tests/functional.lsp — single .lsp file, runs identically in Python and C.
Covers: arithmetic, comparison, booleans, pairs, lists, strings, characters,
vectors, hash tables, control flow, let/lambda/closures, do loops, define,
recursion, TCO (100k depth), quasiquote, macros, type predicates, call/cc,
error handling, mergesort, higher-order programs.
Fixed C call/cc: proper escape continuations via setjmp/longjmp.
make test-all runs: Python unit (571) + C unit (58) + shared functional (114).
Complete C port of the Scheme interpreter. Same .lsp files run in
both Python and C with identical output.
Architecture:
- NaN-boxed 64-bit values (zero-alloc numbers)
- Hash-map environments with parent chain + global shortcut
- Interned symbols
- TCO via explicit loop (eval) and TAIL_CALL/SELF_TAIL_CALL (VM)
- Bytecode compiler with all opcodes including superinstructions
- 58 unit + integration tests
Makefile targets:
make test-all run Python (571) + C (58) tests
make examples run examples in both, compare output
make friction benchmark same .lsp in Python vs C
make c-build build C interpreter
make c-test run C tests
make c-repl C REPL
