Lets bend (examples/cuda-fanout/gpu-worker.lsp) run multiple worker
processes behind a single listening port. Each worker calls
tcp-listen on the same port; the kernel distributes incoming
connections across the bound sockets.
Foxhop production use case: 2 bend workers per GPU host (3090 + 4090)
to consume the .ready queue at 2x throughput without an external
load balancer.
ecdsa's Phase B emit at full secp256k1 width writes a 4–6 GB QECCOPS1
ops binary. The existing path — accumulate in a string-output port,
materialize via get-output-string, write — peaks RAM at 3× body
size (port internal buffer + Scheme string copy + write-binary-file
concat). A 6 GB body needs ~18 GB transient; OOMs a 16 GB QEMU guest.
This commit shifts emit-stream onto a constant-RAM file-port path
and fixes binary-correctness defects in the supporting primitives.
New primitives (mirrored across c/builtins.c + lumbda.py):
- open-binary-output-file path
Opens in "w+b" so the caller can seek back to rewrite a header.
- port-set-position! port offset
fseek absolute offset on a file port. emit-stream reserves a
16-byte placeholder header, streams the body, then seeks back to
byte 0 to rewrite the QECCOPS1 + n_ops u64 LE once n_ops is known.
- append-binary-file path data
Opens in "ab" and fwrite's the bytes through. Pairs with
write-binary-file so callers can land header + body in two writes
instead of (string-append header body).
- append-port-to-binary-file path port
Streams a string-output port's buffer to disk via fwrite without
materializing (get-output-string port). Lets callers keep their
existing string-output sink and avoid the body-size string copy
if they stay on string-port emit.
Binary-correctness fixes:
- bi_write_string to a file port used fputs, which calls strlen.
Binary payloads containing 0x00 truncated at the first null byte.
Switched the file-port branch to fwrite with the string's known
->len (same fix family as the earlier bi_get_output_string
strlen defect).
- bi_write_char per-byte fflush guarded to stdout only. With
millions of gate-bytes per second, flushing after every fputc to
a file port was a 100× slowdown. File ports buffer until close
or explicit flush-port — keep stdout's per-byte feedback path,
drop fflush on every file-port byte.
- port_write_str grows 1.5× past 256 MB instead of 2× throughout.
At realloc time the transient peak is old + new; 2× at 8 GB →
16 GB transient needs 24 GB. 1.5× bounds peak at 2.5× and keeps
multi-GB string-port workloads inside a 16 GB VM.
Tests: 88/88 c-test, 4/4 regression-named-let-leak, 205/205
functional, zoe-favorites all tiers. Binary roundtrip with embedded
nulls at 10/1000/100000 bytes passes byte-for-byte.
End-to-end: foxhop ecdsa DIALOG_GCD secp256k1 emit lands a 4.7 GB
binary at 322 MB peak RSS in 7:41 wall on a 16 GB QEMU guest.
Two C-tier defects surfaced when running foxhop's ecdsa Phase B emit
through the lumbda C interpreter. Both produced wrong bytes in the
generated QECCOPS1 binary; both Python tier handled correctly.
1. string-ref on a byte >= 0x80 returned a char with codepoint -1.
The store was a `char` array (signed on x86_64); `s->data[idx]`
sign-extends 0xFF into a negative int before VAL_CHAR wraps it.
Round-tripping (char->integer (string-ref s 0)) for a 0xFF byte
gave -1 instead of 255. The Scheme-level (u64-le n) packer uses
(make-string 1 (integer->char (modulo v 256))) for each byte and
reads them back; sign-extension corrupted the high-bit bytes.
Cast through `unsigned char` in bi_string_ref.
2. pack_u64_slot treated bignum slots as raw fixnums.
ecdsa's emit-stream binds (no-slot *no-slot*) where *no-slot* is
18446744073709551615 (u64 max). On Python tier that's a regular
big-int. C tier carries it as a bignum NaN-box slot. The packer
knew about VAL_FALSE → 0xFFFF... but called as_int(v) on bignums,
reading the NaN-box payload bits (pointer-to-Bignum) and writing
that pointer as the field's 64-bit LE value.
Add an IS_BIGNUM branch that extracts the low 64 magnitude bits
directly. Cross-tier emit-stream code stays unchanged.
Tests: 88/88 c-test, 4/4 regression-named-let-leak, 205/205
functional. DIALOG_GCD smoke at p=11 n+1=5 now byte-identical to
Python tier (1,941,816 bytes match exactly).
Two pre-existing defects in c/builtins.c made (open-output-string)
unusable for binary emit:
- bi_get_output_string ran the buffer through make_string_from_cstr,
which calls strlen. Any 0x00 in the payload truncated body at
that byte. Use port's known str_len directly via make_string.
- bi_write_char ignored string-port destinations entirely — it
pulled AS_PORT(p)->fp (NULL for string ports), fell back to
stdout, and silently routed gate bytes to terminal output
instead of the port buffer. Route to port_write_str when target
port kind is PORT_STRING, matching bi_write_string's behavior.
Surfaced while validating the precise-GC fix against ecdsa's
Phase B emit (writes 56-byte op records full of embedded nulls
through a string-output port, get-output-string at the end).
With these fixes ecdsa's n+1=64 emit drops a 333 MB binary in
42.6 seconds — pre-fix it timed out at 600 s with a 17-byte
header-only file (strlen truncated body at byte 1; the visible
gate bytes had been escaping to stdout the whole time).
Binary roundtrip test (write n bytes alternating x / 0x00 to
output-string port, read back via get-output-string):
n=10 len=10 ok
n=1000 len=1000 ok
n=100000 len=100000 ok
All upstream tests still pass (88/88 c-test, 4/4 regression,
205/205 functional, zoe across tiers).
Boehm's conservative pointer scan cannot recognize lumbda's Value
layout — heap pointers live in the low 48 bits with QNAN + tag bits
in the upper mantissa, so a raw word never looks like a heap address.
Until now main.c neutralized this with GC_disable(): every allocation
leaked, OOMing any long-running workload.
Add precise tracing via a custom Boehm kind:
- New c/gc.c: mark proc walks 8-byte words in mixed mode — when the
QNAN bits are set with a pointer-bearing tag (0/2/4/5/6) extract
the low-48 pointer; otherwise fall through to raw-pointer
validation. GC_set_push_other_roots callback decodes NaN-boxed
Values on the C stack via setjmp anchor + scan up to the stack
base captured at process start.
- Allocations holding Values (Pair, Env bindings, ValueStack data,
ULVector data, HTEntry, Proc params + body, FullCont stack,
CodeObj instrs, SymbolEntry) route through lumbda_value_malloc.
Pure-byte sites (bignum limbs, char buffers, source files) stay
on regular GC_MALLOC.
- main.c / test.c / bench.c capture stack-base then drop GC_disable.
types.c also zeros popped slots on the value stack so stale pointers
do not survive a vs_pop and pin freed objects — independent
correctness fix that pays off once GC actually runs.
Build: USE_GC=1 (default when /usr/include/gc.h exists).
Tests with GC enabled:
- 88/88 c-test
- 4/4 regression-named-let-leak (test that motivated GC_disable)
- 205/205 functional (Python + C)
- zoe-favorites all tiers (Python + C + asm + asm-full)
alloc-test 1M cons drop-loop:
- Before: 0.60s wall, 156 MB RSS, leaks every cell
- After: 0.37s wall, 4 MB RSS, ~1500 GC cycles each freeing ~370 KB
Ports lumbda.py _emit_circuit_to_ops_bin_stream to C tier. Walks a
Scheme registers + ops list, writes each 56-byte QECCOPS1 op record
straight to disk via fopen/fwrite, then seeks back to patch the n_ops
header at the tail. O(1) host memory regardless of n_ops.
Cross-tier byte-identity verified:
* 500-op mixed-tag synthetic circuit on host: Python ↔ C identical
* real-point-add n+1=5 (p=11, 94,214 ops, 5.0 MB) inside VM
* real-point-add n+1=9 (p=251, 674,872 ops, 36 MB) inside VM
sha256 matches every width.
Speedup on equal Scheme source (build + emit combined):
* n+1=5: Python 56.8 s → C 1.1 s (52×)
* n+1=9: Python 365 s → C 8.4 s (43×)
Combined ratio exceeds the prior 11-16× C-tier envelope because the
build phase (Phase B mod-arith construction) also accelerates on C;
emit-only ratio is ~3-9× and grows with op count.
C test suite: 85 → 88 passing (emit-stream-basic, alloc-free, empty).
Shared functional suite: 205/205 still passing on both tiers.
Implementation notes:
* pack_op_record packs u32 kind + u32 pad + 6× u64 LE, matching
op-specs->bytes byte layout exactly.
* Layout hashtable Symbol → fixnum base, mirroring walk-circuit-ops.
* NO_SLOT sentinel: 0xFFFFFFFFFFFFFFFF written directly to u64 slots
that the Scheme side did not populate.
* libc's default fwrite buffer (~4 KB) handles batching at ~73 ops
per write — same throughput class as Python tier's 8 KiB list batch.
Adds tagged bignum support alongside the existing 48-bit fixnum on the C
tier. Tag 6 = bignum, heap struct sign-magnitude with u64 little-endian
limbs. Reader emits bignums for any literal past the fixnum range; +, -,
*, quotient, remainder, modulo, expt, =, <, >, abs, odd?, even?,
integer?, exact?, number->string, string->number all promote fixnum →
bignum on overflow & demote back when results fit. Boehm GC owns every
allocation. Schoolbook O(n²) mul + shift-subtract divmod is sufficient
at our 4-limb / 256-bit scale.
Before: (expt 2 48) = 0, (expt 2 256) = 0, secp256k1-p = -4294968273.
After: all three return their exact arbitrary-precision values, matching
Python tier byte-for-byte.
Validated:
- c/test.c — 85/85 pass (+2 new bignum unit tests).
- tests/functional.lsp — 205/205 pass on both C & Python tiers.
- tests/bignum-cross-tier.lsp — 33/33 pass byte-identical on both tiers
(diff produces no output).
- ecdsa/runs/lumbda-sweep-003/c-tier-bignum-probe.lsp — all four
assertions now match the Python oracle.
- ecdsa Phase B byte-identity sweep inside QEMU guest:
n+1=9 p=251 sha256 c668bbe3... — matches Python oracle.
n+1=18 p=131071 sha256 8a031f96... — matches Python oracle.
n+1=33 p=2³²-5 sha256 0bc56905... — matches Python oracle.
Previously the n+1=33 C tier emitted sha256 b024d6d9... (26,078 fewer
Toffolis due to silent fixnum wrap). Bignums close that gate.
secp256k1 production-width emit (n+1=257) is now structurally unblocked
on C tier; downstream agent (#55) drives that next-step on the ecdsa
side. Asm tier inherits in a follow-up port.
Direct C translations of our Python-tier primitives at lumbda.py
_walk_circuit_ops / _op_specs_to_bytes / _count_lumbda_ops (commit
99701c8). Same algorithms; native dispatch via interned-Value identity
on cached symbol globals.
NO_SLOT representation differs from Python tier. Python stores the
literal 18446744073709551615 fixnum (arbitrary-precision int). C-tier
fixnums cap at 48 bits via NaN-boxing PAYLOAD_MASK, so we substitute
VAL_FALSE as our slot sentinel inside op-spec vectors; pack_u64_slot
writes 0xFFFFFFFFFFFFFFFF whenever it sees VAL_FALSE. Both tiers
produce byte-identical QECCOPS1 output.
API:
- (walk-circuit-ops registers ops) -> list of 7-element op-spec vectors
- (op-specs->bytes specs) -> latin-1 string of 56*N bytes
- (count-lumbda-ops ops) -> 3-element vector (toffoli clifford total)
Symbol cache (g_sym_ccx / g_sym_x / ...) initializes lazily on first
call; intern() is idempotent so repeat init costs nothing. Layout uses
make_hashtable + ht_set/ht_ref/ht_delete for O(1) qubit-base lookup
matching our Python dict-based implementation.
Measured wall on foxhop ecdsa's canonical p=11 textbook+refined emit
pair inside our QEMU guest:
Python tier: 6.9 sec (baseline after primitive lift)
C tier: 0.71 sec (9.7x over Python)
Output verified byte-identical against Python-tier reference files via
cmp on both textbook & refined paths.
Asm tier port deferred. Asm tier's documented role serves bulk 9024-
shot validation (simulator runs against an emitted ops.bin, 160x Python
on portal round-trip per whitepaper s6.6.4) — emit pipeline targets
Python / C tier. ~700-900 lines of hand-written x86_64 asm + QEMU
debug cycles, 2-4 day effort, no current asm-tier emit consumer.
The S-expression wire format was the bottleneck at huge payload sizes
-- 23.8 s end-to-end for 1M x 16 B inputs on the Python tier, while
the actual CUDA kernel finishes the same workload in ~47 ms. The
hex-S-exp parser ate everything between.
New binary wire mode (magic 'BSHK' prefix; payload is the daemon's
binary portal format verbatim) bypasses S-expression parsing entirely.
Worker writes the blob to disk, calls daemon process-bin, reads result,
prepends 'BSHR' magic, replies.
Measured 3090-ai, daemon warm, localhost:
workload Py S-exp Py binary C S-exp C binary
100 x 16 B 3.43 ms 0.74 ms 0.40 ms 0.15 ms
1k x 16 B 23.24 ms 0.76 ms 2.77 ms 0.22 ms
10k x 16 B 218.82 ms 1.27 ms CLIFF 0.88 ms
100k x 16 B 2,219 ms 10.18 ms CLIFF 10.35 ms
1M x 16 B 23,811 ms 159 ms CLIFF 157 ms
150x speedup at 1M inputs on Python tier. C tier S-exp CLIFFs
between 1k and 10k inputs (reader payload limit); binary mode
bypasses the CLIFF entirely. At 100k+ inputs both tiers converge
since file I/O + CUDA kernel dominates over wire framing.
Host comparison: hashlib.shake_256 over 1M tiny inputs takes ~2 s
on a single Python core. Bend via binary worker = 157 ms = 12x
faster than host. Bend now wins at huge workloads, not just heavy
ones.
Implementation:
lumbda.py
* tcp-send/tcp-recv switched to latin-1 (1:1 byte mapping)
so binary payloads pass through cleanly. UTF-8 was mangling
bytes with replacement chars.
* write-binary-file / read-binary-file primitives.
c/builtins.c
* write-binary-file / read-binary-file matching Python tier.
examples/cuda-fanout/wire.lsp
* wire-send-raw / wire-recv-raw helpers that frame a raw
payload string without S-expression serialization.
examples/cuda-fanout/gpu-worker.lsp
* handle-binary-shake: write portal blob, daemon process-bin,
read result, wire-send 'BSHR' + bytes.
* handle-one dispatches on first 4 bytes of payload: 'BSHK'
goes to binary path, anything else stays S-exp.
examples/cuda-fanout/bench_tiers.py
* make_payload_binary builds the BSHK protocol payload.
* --binary flag in CLI.
www/index.html
* full S-exp + binary comparison table.
* 'bend now beats host hashlib at huge workloads' headline finding.
Two new primitives in builtins.c, paralleling the Python tier shipped in
the previous commit. gpu-worker.lsp now runs on the C tier byte-identically
to the Python tier.
(spawn-process-stdio path args) → (stdin-port . stdout-port)
fork + pipe + execvp; child's stdin & stdout wired back to parent
as line-buffered FILE* ports. Accepts both strings and symbols in
the args list (matches Python tier's permissive conversion).
(flush-port port)
fflush() on the port's FILE*. No-op when fp is null.
End-to-end on 3090-ai with C-tier lumbda everywhere:
shell A: ./lumbda /tmp/launch-c.lsp
→ gpu-worker: ready cuda-shake-fanout ← ./shake256-fanout
→ gpu-worker listening on port 9091
shell B: ./lumbda smoke-bend.lsp # run 3×
=== smoke-bend ===
1. cost estimator picks local for 3 inputs: OK
2. worker available? #t
3. bend! (cuda-shake-fanout '("00" "01" "deadbeef") 32):
(b8d01df855… 94da6280b2… fa094fa86e…)
Three runs identical bytes. Same hashes as Python tier. Same hashes as
hashlib.shake_256 host reference.
Cross-tier matrix (proves wire protocol is tier-agnostic):
client tier worker tier status
─────────────────────────────────────
C tier C tier PASS — 2 sequential runs, byte-identical
Python tier C tier PASS — same hashes
C tier Python tier implicit by symmetry (same wire bytes
both directions; Python-server tested
against Python-client in prior commit)
Per-tier status after this commit:
Python tier ✓ end-to-end
C tier ✓ end-to-end + cross-tier byte-identical to Python tier
asm tier → still needs spawn-process-stdio via raw fork+pipe+
execve syscalls. Scheme files unchanged.
Bash process substitution <(...) passes /proc/self/fd/N — a pipe, not
a regular file. load_file used fseek(SEEK_END)+ftell to size a single-
read buffer; on a pipe ftell returns -1, which casts to SIZE_MAX as
fread's nbyte argument and blows the heap. Glibc fortify caught it
as '*** buffer overflow detected ***'.
Detect non-seekable input via the fseek return code and fall back to
a doubling growable buffer instead. Seekable path unchanged.
Repro: ~/git/lumbda/c/lumbda <(echo '(display 1)(newline)')
Resolved by 45a90b8 (vm restores cur_code across CALL/RETURN).
The reproducer at lines 5-11 of the TODO file runs cleanly now
under both `lumbda --fast` and `lumbda -j --fast`, and the
proof/eml_proof_in_lumbda.lsp `normalize` function no longer
needs the `(define (iter ...))` workaround that the TODO
documented. Future regressions are caught by
tests/regression-named-let-leak.{lsp,sh}.
Five-line fix that ends the F1 hang in foxhop.net
ecdsa/tests/unit/probe-c-confirm.lsp.
Symptom — under --fast, a defined function whose body is a
tail-recursive named-let that calls another user-defined function
per iteration loops forever at 100 percent CPU. Trace pins the
bytecode dispatch:
walk1.body: PUSHE MKCLO DUP BIND LOOKUP TCALL ->loop
loop: LOOKUP NULL? JIF LOOKUP CALL ->always-true
always-true: CONST RET (returns #t)
always-true (!): JIF LOOKUP CDR STAIL -> ip=0 of always-true (!)
loop forever
cur_code was the call-frame-local register holding the currently
executing CodeObj. OP_CALL updated it on entry but neither OP_RETURN
nor the builtin-fallback restore path in OP_TAIL_CALL put it back
on return. Subsequent OP_SELF_TAIL_CALL read cur_code->self_params
from the still-stale callee proc (NULL for always-true since it has
no named-let), guard skipped the env rebind, then set ip=0 — without
ever updating the loop variable. Loop variable stayed pinned at the
initial list and our walk never reached its base case.
Fix — VMFrame gains a cur_code field. Three frame-push sites save
it on entry (OP_CALL, OP_TAIL_CALL builtin fallback frame-restore,
OP_CALL_CC compiled-proc entry); two frame-pop sites restore it on
return (OP_RETURN, OP_TAIL_CALL builtin fallback).
Verified inside foxhop.net's ecdsa QEMU guest:
- foxhop.net/ecdsa/tests/unit/probe-c-confirm.lsp F1..F4 — all pass
- foxhop.net/ecdsa/tests/unit/test-sim.lsp under --fast — 21/21 pass
- foxhop.net/ecdsa/lumbda/main.lsp under --fast — 6 shots, score 18,
byte-identical with our Python tier
- make functional-test — 205/205 on Python and C tiers
- make regression-named-let-leak — 4/4 across Python, tree-walker,
and --fast JIT
c/TODO-named-let-bytecode.md can stop applying its `(define (iter ...))`
workaround once this lands.
Three coupled changes that unblock the ecdsa quantum-circuit
simulator's run on the C tier from neoblanka.
1. c/Makefile autodetects libgc-dev — if /usr/include/gc.h is
present, the build links Boehm and defines USE_BOEHM_GC. Without
GC, ul_free is a no-op (lumbda.h:35) and every allocation leaks;
small REPL snippets work but workloads with thousands of envs
OOM the process. Override with USE_GC=0 to force the malloc-only
path for diagnostics.
2. c/main.c calls GC_INIT before init_symbols, then GC_disable.
GC_INIT registers the stack base for conservative scan — without
it some Linux configs miss roots. GC_disable is a deliberate
stopgap: lumbda Values are NaN-boxed pointers that conservative
Boehm cannot recognize as pointers, so live targets get reclaimed
(env binding symbol payloads, SymbolEntry strings) and lookups
fail with "undefined: <sym>". Reproducing this without GC_disable
on the GC build: any sim.lsp call chain triggers the corruption
after ~100 named-let iterations. Until tracing is precise,
growing the heap is safer than wrong results. Long-running
workloads run under ulimit -v.
3. c/builtins.c gains rename-file and delete-file matching the
Python tier (lumbda.py:3468). sim.lsp's write-portal! pattern
(write to .tmp, rename) needs rename-file to land cross-tier
identical results.
4. tests/regression-named-let-leak.lsp + .sh pin four shapes that
blew up ecdsa: the c/TODO-named-let-bytecode.md repro, the F1
shape from foxhop.net's lumbda-c-tier-leak-SP.md (12-line
minimum), a 200-iter scaled variant, and a sim.lsp run-ops!
mirror. Wired into root Makefile as regression-named-let-leak;
added to test-all. Wrapper caps memory at 256 MB virt and 15s
per tier so a leak regression fails the run instead of consuming
host RAM.
Known limits:
- --fast JIT still has the named-let + inner user-fn call hang
(separate TODO; tree-walker handles this fine).
- GC_disable means the heap grows; workloads must bound their work
budget. ecdsa's sim runs comfortably in 5 MB.
Verified inside a 2G/2vCPU QEMU guest (foxhop.net ecdsa/vm-runner.sh):
- test-c (tree-walker) — 35/35 PASS
- bench-c (tree-walker) — score 18 matches Python tier byte-identical
- F1 probe (tree-walker) — all four steps PASS
Zoë Trout's favorites at wedgewack.org/ursa.lisp.txt are Common Lisp:
iterative LOOP macros, setf cascades, defun with &optional, image-
based stone-lisp culture. Her first contribution to lumbda was a
question — "do we care for our programs, and how long are they alive
for?" — and the answer now extends beyond the RNG portal (§7.5) to
iteration style itself.
Four-phase delivery, all under ticket 0004:
Phase A — idiomatic Scheme ports at examples/ursa-scheme.lsp.
Every Zoë defun rewritten as named-let + tail recursion + list-
backed work queue + type-predicate dispatch.
Phase B — CL compat shim at cl-compat.lsp.
defun (with &optional), setf (simple vars, multi-pair), flet,
multiple-value-bind, t / nil (nil=#f so cond/if compose),
evenp/oddp/plusp/minusp/zerop, mod/ash/logbitp/nreverse,
cl-when/cl-unless (plain when is a void-returning lumbda special
form), declare (no-op), cddddr (missing accessor).
Phase C — cl-loop macro covering 14 patterns.
while/until/repeat, for VAR from A to/below/downto B, for VAR =
INIT [then STEP], for VAR across VEC, of-type T, do, when/unless
return, finally (return VAL). Sequential do*-style stepping via
gensym + cl-subst. Look-ahead termination so `repeat 4 for s = 4
then (- (* s s) 2) finally (return s)` returns 37634 (pre-step)
rather than 1416317954 (post-step). Every expansion ends in a
named-let tail call — TCO holds for loops of any length.
Phase D — load examples/ursa.lisp.txt with minimal annotation.
Preserves Zoë's CL. Minimal edits documented in file header:
load cl-compat.lsp, loop→cl-loop, when→cl-when, random→random-int,
&key→&optional. rho/digits omitted (need make-array/CLOS — see
ticket 0004 for scope boundary).
Defect uncovered along the way (c/types.c env_lookup): a "global
shortcut" checked global env immediately after missing the local
frame, SKIPPING intermediate parent scopes. Broke lexical scoping
whenever a parent scope shadowed a global. Reproduced with
(define s 4)
(let ((s 100)) (let ((m 0)) s)) ; returned 4, should return 100
Any nested let whose body referenced a shadowed name silently read
the global. Fix: remove the shortcut, walk the parent chain end-to-
end. 1255 assertions across five suites pass unchanged after fix —
surfaced only because cl-loop iterator names routinely collide with
globals accumulated in a stone-lisp image.
Whitepaper §9.2 documents the CL-in-Scheme design and the guarantees
that survive (TCO, portal determinism, cross-impl reproducibility).
Zoë added to authors + acknowledgments; reacknowledgment reframes
her first contribution as the deeper program-lifetime question, with
RNG portal as a derivative (§7.5) and cl-loop as the follow-up.
Tests: tests/cl-compat.lsp (44 assertions) and tests/ursa.lsp (28
assertions) exercise both paths under Python + C via tests/zoe-
favorites-test.sh, wired into make test-all.
MOAD notes: unmoad flags memq/assq in cl-compat.lsp over cl-loop-
keywords (~30 elements, constant) and var->new (≤4 state vars per
loop). Both are macro-expansion-time, bounded-small-N — not runtime
hot paths. Pre-existing c/types.c findings (strcmp-in-loop for
record-type lookup) are not from this change.
Ticket 0002 — reads 8 bytes from /dev/urandom (little-endian u64) and
seeds xoshiro256**. Opt-in kernel entropy for stochastic runs; the
default stays deterministic (k=0 at startup), so ticket 0001's
portal-reproducibility contract is unchanged.
Real-world flow now one call away:
Machine A: (random-seed-from-os!) + run simulation + portal-save
Machine B: portal-resume — same stream, bit-for-bit
All three impls fail loud on /dev/urandom trouble (LispErr in Python
and C, stderr + exit(1) in asm) — no silent fallback to a weak seed.
Tests:
- tests/functional.lsp: 2 new shared asserts (entropic + replay)
- asm/test.sh: 2 new asm-local checks (149 total, was 147)
- make test-all green across Python (205), C (205), asm (149)
Whitepaper §7.5 gains one sentence noting the OS-seed path.
unmoad: zero new findings in added code.
Completes ticket 0001 started in 27f468c. All three impls now carry
bit-identical xoshiro256**; portal state round-trips across process
boundaries in every producer x consumer cell (Python <-> C <-> asm).
asm impl:
- 4 new builtins: random-seed!, random-int, random-state, random-state!
- g_rng_state in BSS (4 x u64); rng_splitmix64_step, rng_seed, rng_next
- Binary portal header bumped LUMBDAB1/48 -> LUMBDAB2/80; carries
32 bytes of rng state at offsets 40..64, reserved moved to 72
- No float support in asm, so (random) intentionally omitted there
- _start seeds with 0 so the stream is deterministic from startup
Python + C (supplements 27f468c):
- rng_seed(0) auto-invoked at module load / register_portal_builtins
so (random) without explicit (random-seed!) returns a real value
instead of the all-zero xoshiro fixed point
Tests:
- tests/functional.lsp: 7 new shared assertions (Python + C)
- asm/test.sh: 5 new asm-local assertions (142 -> 147)
- tests/portal-rng-save.lsp / portal-rng-load.lsp: portable S-expression
portal that captures both state AND next-5 baseline so loader self-
verifies without a separate harness
- tests/portal-cross-test.sh: 9 new producer x consumer RNG cells; all
18 cells pass end-to-end
Verified: seed=42, (random-int 1000000) draws 1..10 =
558742 543102 559009 124193 317476 750584 200754 814407 344958 929085
identical in Python, C, and asm.
unmoad scan: zero new findings in added code.
Zoe's contribution question: does our portal preserve RNG state so a
simulation can continue in another process with the same random stream?
Answer today: no — no RNG existed. Answer now (Python + C): yes, bit-identical.
- New builtins: random-seed!, random, random-int, random-state, random-state!
- xoshiro256** (Blackman & Vigna 2018) — deterministic, portable, no libc rand
- State = 4 x u64; portal-v1 JSON gains 'rng' field with 8 x u32 halves
- Python and C produce bit-identical streams (verified: seed=42, 10 draws)
- Asm impl + cross-impl tests + whitepaper note: next commits
Ticket: docs/tickets/0001-portal-rng.md
Historical internal name "uncommonlisp" retired in favor of the
public name "lumbda" ahead of lumbda.com going live. Scope of
this commit:
Source files renamed:
uncommonlisp.py -> lumbda.py
asm/uncommonlisp.s -> asm/lumbda.s
c/uncommonlisp.h -> c/lumbda.h
whitepaper/uncommonlisp-whitepaper -> whitepaper/lumbda-whitepaper (.rst + .pdf)
Binaries renamed (tracked ones; c/ was always gitignored):
asm/uncommonlisp, asm/uncommonlisp-gc, asm/uncommonlisp.o,
asm/uncommonlisp-gc.o -> asm/lumbda(-gc)(.o)
c/.gitignore -> ignores lumbda
Internal string updates (sed pass ordered longest-first):
asm/uncommonlisp -> asm/lumbda
c/uncommonlisp -> c/lumbda
uncommonlisp.py -> lumbda.py
UNCOMMONLISP_BIN -> LUMBDA_BIN (asm/test.sh env var)
"uncommonlisp> " -> "lumbda> " (asm REPL prompt baked into binary)
UNCOMMONLISP -> LUMBDA (macros, comments)
uncommonlisp -> lumbda (prose)
Binary portal magic updated:
"ULPORTAL" -> "LUMBDAB1" # "Lumbda Binary v1"
Old portal files are not backward-compatible — this is a deliberate
break since it's the rename moment. S-expression portals already
carry their own ";; lumbda-portal v1" header and remain cleanly
versioned.
WHITEPAPER.pdf / WHITEPAPER.rst symlinks repointed to the renamed
files. Makefile's whitepaper target targets lumbda-whitepaper.pdf.
Not changed (intentional, separate phases):
- Filesystem directory /home/fox/git/uncommonlisp itself
(fox renames locally and the gitlab repo URL in a follow-up)
- tests.py hardcoded cwd=/home/fox/git/uncommonlisp
(matches the current on-disk location; will flip when the
directory rename ships)
- Git history (immutable; old commits still say uncommonlisp,
which is correct — that's what they were)
Verified:
137 asm no-GC + 137 asm GC + 571 Python + 83 C + 189 shared
functional tests all pass under the new names.
bench-gc-http (2000 req): all 4 cells behave as expected
(cells 1/2 flat, 3 leaks, 4 bounded at 1 chunk).
Python REPL, C REPL, asm REPL all start cleanly.
Hunted the C --fast compiler bug that was hanging on the EML proof.
Narrowed to a specific pattern:
(let loop ((t start))
(let ((next (fn t)))
(if next (loop next) t)))
A named-let whose body is (let ((x (...))) (if x (recurse x) base)).
The recursive call inside the inner let+if branch never reaches the
loop closure — hangs or segfaults.
Reproducible with a 4-line test case; filed as
c/TODO-named-let-bytecode.md with minimal repro, suspected cause
(env-chain mismatch between PUSH_ENV and TAIL_CALL), and a known-
good workaround.
Workaround landed in proof/eml_proof_in_lumbda.lsp's `normalize`:
replaced the named-let with an internal recursive `define`, which
compiles correctly under --fast. Same logic, different surface
syntax. All four Lumbda tiers now verify the proof.
Benchmark refreshed (make bench-proof):
cold cached
Lumbda asm 46 ms 7 ms
Lumbda C --fast 65 ms 9 ms
Lumbda C (tree-walker) 87 ms 12 ms
Lumbda Python --fast 651 ms 232 ms
Lean 4 722 ms 5 ms
All four tiers now green. Asm still fastest (46 ms cold vs Lean's
722 ms — ~16× faster). Cached Lumbda asm 7 ms vs Lean 5 ms (within
1.5×). The C --fast tier went from "hangs" to 65 ms cold — competitive
with asm once the compiler bug is dodged.
Whitepaper §8.6 table updated; prior "(hangs)" row is gone;
footnote on the named-let workaround links the TODO file.
ack(3,8) was reported as "segfault" for the C impl in the previous
whitepaper revision. That was a stale observation — C has --fast
(bytecode VM with explicit frame stack) that handles deep recursion
cleanly. The benchmark table compared the wrong modes.
Corrected apples-to-apples:
- Python --fast (bytecode VM) — 17,004 ms on ack(3,8)
- C --fast (bytecode VM) — 1,433 ms **fastest of the three**
- asm native (tree-walker) — 2,322 ms
C's --fast wins every workload. asm still beats Python --fast by
~7x despite being a tree-walker, because it skips Python's per-op
overhead entirely.
c/main.c: --help text updated to clarify that --fast is required
(or `ulimit -s unlimited`) for deep recursion in the default
tree-walker mode. Attempted flipping --fast to default; reverted
because that surfaced a cumulative-state buffer overflow in the
bytecode compiler that only triggers after the full 189-test
functional suite but not on isolated scripts. Left as a TODO in
the code comment. 189 C tests + full test-all still pass.
Whitepaper §6.4 table now shows all three impls in their
high-performance configuration. Also noted that a pthread-with-
larger-stack wrapper would let the C tree-walker handle deep
recursion without --fast — tracked as low-priority future work
since --fast is strictly faster regardless.
Closes the last loop promised in the whitepaper's Future Work: a
node serves its state as an S-expression portal over HTTP, another
node pulls it down with tcp-connect + tcp-recv and materializes the
bindings locally via (eval (read-from-string line)).
examples/portal-http-server.lsp (90 lines):
- Holds some state (counter, my-int, my-list, my-fib, my-str)
- GET /portal → S-expression body: a sequence of (define ...) forms
- GET / → HTML index
- Uses heap-snapshot / heap-restore for O(1) memory on asm
examples/portal-http-client.lsp (90 lines):
- tcp-connect, send HTTP/1.0 GET, receive full response
- Strip headers (walk to first \r\n\r\n)
- Split body by \n, eval each non-empty, non-comment line
- The remote bindings are now live locally
3×3 server/client matrix: all 9 combinations green. Every runtime
hosts, every runtime consumes. The wire format is Scheme source;
no schema, no JSON, no Protobuf.
Prerequisite fix: `eval` semantics aligned across all three impls.
Python and C's `eval` special form previously evaluated its result
in the CALLER's env, so a nested (eval (read-from-string
"(define x 42)")) would install x in the local function scope —
invisible to later top-level code. asm's bi_eval always used the
global env (r14). With this commit, all three impls evaluate the
eval'd result in the global env, matching asm's existing behavior.
Python: uncommonlisp.py leval eval-handler now does `env = env.g`
before continuing the trampoline.
C: c/eval.c SYM_EVAL branch now does `env = env->global`.
asm: no change (already correct).
One pre-existing Python defect surfaced by the client:
`count` is a SRFI-1-style builtin (`d(S('count'), ...)`), so a
local let-loop variable named `count` collides with it in the
inline-cache lookup path and OP_LOOK_ADD1 fires on the builtin
instead of the local. Worked around by renaming the loop
accumulator to `cnt`. Underlying Env.lookup shortcut-to-global
issue is out of scope for this commit.
Regression: 975 tests still green.
Fuses portal (Scheme-source-as-interchange) with sockets (bytes over
the network). Wire protocol: one S-expression per connection. Same
server + client .lsp runs byte-identically in Python, C, and asm.
New primitives in all three impls:
- read-from-string — parse one sexp from a string
Asm gets two more:
- symbol->string — standard R7RS, was missing
- eval — evaluate a Scheme value in the global env (Python + C had
it as a special form; asm exposes it as a builtin)
examples/rpc-server.lsp (port 9080):
- Whitelisted dispatch: ping / add / mul / fib / echo
- Never calls eval on client input; safe by construction
- Uses heap-snapshot/restore for O(1) memory on asm
- ~90 lines, portable
examples/rpc-client.lsp:
- Sends one request, reads one response, displays both
- 45 lines, portable
examples/repl-server.lsp (port 9081):
- DANGER: full remote eval. Any Scheme form accepted and evaluated
in the server's global env. Persistent across connections.
- Deliberately does NOT use heap-snapshot — remote (define x ...)
lives in the global env above any snapshot point; rewinding would
invalidate the new binding. The ulimit -v 512 MB safety net
(documented in CLAUDE.md) ensures an escaped process can't crash
the machine.
- ~70 lines, portable. Demonstrates what "the language IS the
interchange format" gets you at the limit: a single socket and
a single primitive (eval) carry a full-powered REPL.
Verified 3×3 server×client matrix: all 9 combinations green.
All 132 asm + 571 py + 189 shared + 83 c tests still pass.
One quirk discovered and worked around: in asm, a closure captures
its env chain by pointer at define time. Forward-referenced names
in mutually-recursive toplevel defines can fail under specific
heap-restore patterns — see the leaf-first ordering note in
rpc-server.lsp.
Three wins in one commit.
1) heap-snapshot / heap-restore (asm arena primitive)
asm has no GC. Long-running servers leaked ~64 MB per heap growth.
Two new builtins let a programmer capture r15 and later rewind to
it, recycling intermediate allocations in O(1) memory.
Python + C get no-op versions so portable .lsp code can call them
unconditionally.
examples/http-server.lsp now takes a snapshot at top level and
rewinds after every request. Measured asm RSS: 88 KB initial,
100 KB after 100 requests, 100 KB after 1100 requests — flat.
Prior behavior was +64 MB per few thousand requests.
2) examples/http-client-bench.lsp — native HTTP load generator
Uses only the six tcp-* primitives + current-time-ms. Runs
identically in all three impls. Eliminates curl's ~2 ms/req
fork+exec overhead, so real server throughput shows up:
Python server ← Python client 2403 rps
C server ← C client 2439 rps
asm server ← asm client 2994 rps
asm server ← C client 2500 rps
The earlier curl-based bench was clamped near 400 rps by the
client; the actual servers handle 6–7× that.
3) MOAD-0001 cleanup
- c/builtins.c bi_string_replace: strncmp-at-every-position
(hand-rolled, sedimentary) → strstr (libc-tuned, typically
Boyer-Moore-Horspool). O(N*k) → O(N + matches*k).
- uncommonlisp.py _tokenize_lines: per-token src.count('\n', 0, pos)
→ precompute line_starts once, bisect_right per token.
O(N*M) → O(M + N log M).
Also adds current-time-ms to all three impls so benchmarks can
time themselves without relying on the Python/C float `current-time`
(asm has no floats). Seconds-since-epoch tagged as a 61-bit int.
Test counts unchanged: 571 py + 132 asm + 189 shared + 83 c = 975.
All green via make test-all.
Added tcp-listen/accept/connect/recv/send/close to Python, C, and asm.
One examples/http-server.lsp runs identically in all three impls and
serves HTTP/1.0 with routing, content-type, and content-length headers.
asm additions:
- SYS_SOCKET/BIND/LISTEN/ACCEPT/CONNECT/SETSOCKOPT syscalls
- 6 tcp-* builtins using the existing port encoding (SPECIAL ≥ 1000)
- bi_tcp_connect: dotted-quad IPv4 parser, no DNS dependency
Defects fixed along the way (surfaced by the HTTP server):
- string-append: was 2-arg only; now variadic (walks arg list twice)
- number->string: was stubbed to VAL_VOID; now correctly writes digits
into a heap-allocated string (incl. negative handling)
- String-literal reader: \r and \0 escape sequences now handled (was
silently dropping backslash, treating them as literal 'r' / '0')
- tcp_accept: sockaddr buffer was 8 bytes, now 16 (was corrupting
caller's stack when accept wrote full struct sockaddr_in)
Pinocchio benchmark (tests/web-benchmark.sh):
At concurrency=20, 1000 requests, serving a 1KB body:
uncommonlisp Python 373 req/s
uncommonlisp C 370 req/s
uncommonlisp asm 370 req/s
python3 http.server 381 req/s (stdlib reference)
busybox httpd 382 req/s (production reference)
All five converge within 3% — the client (curl fork/exec) is the
bottleneck, not the server. Our single-threaded blocking servers
are indistinguishable from battle-tested ones at this load.
Binary sizes:
uncommonlisp asm 45 KB (HTTP + everything else)
busybox httpd 2.1 MB (multi-call binary)
python3 8 MB (interpreter)
The asm HTTP server is 46× smaller than busybox and 176× smaller
than Python, serves from 7 Linux syscalls, and the entire protocol
handler is 70 lines of portable Scheme.
Test counts: 132 asm (up 1), rest unchanged. All green.
Full continuations: FullCont captures frames/stack/env with deep copy.
VM trampoline via setjmp/longjmp. Multi-shot safe via deep_copy_env.
Portal (new c/portal.c): serialize env + continuation to JSON,
resume on another machine. portal-checkpoint! triggers mid-VM save.
83/83 C unit tests + 181/181 functional tests pass.
Real native machine code via mmap(PROT_EXEC). No exec(). No strings.
Raw x86_64 bytes: mov, add, sub, imul, cmp, je, jne, call, ret, jmp.
ack(3,4): 0.12ms JIT vs 1.5ms CPython vs 28ms interpreter
fib-rec(20): 0.16ms JIT vs 3.4ms CPython vs 40ms interpreter
Added cond support to JIT (cascaded comparisons → conditional jumps).
Fixed JIT cache: sentinel value prevents retry on unjittable functions.
System V AMD64 ABI: args in rdi/rsi/rdx, callee-saved r12-r15.
Tail calls use jmp (true TCO at machine code level).
691 lines of jit.c. 114 functional tests pass. All C tests pass.
tests/functional.lsp — single .lsp file, runs identically in Python and C.
Covers: arithmetic, comparison, booleans, pairs, lists, strings, characters,
vectors, hash tables, control flow, let/lambda/closures, do loops, define,
recursion, TCO (100k depth), quasiquote, macros, type predicates, call/cc,
error handling, mergesort, higher-order programs.
Fixed C call/cc: proper escape continuations via setjmp/longjmp.
make test-all runs: Python unit (571) + C unit (58) + shared functional (114).
Complete C port of the Scheme interpreter. Same .lsp files run in
both Python and C with identical output.
Architecture:
- NaN-boxed 64-bit values (zero-alloc numbers)
- Hash-map environments with parent chain + global shortcut
- Interned symbols
- TCO via explicit loop (eval) and TAIL_CALL/SELF_TAIL_CALL (VM)
- Bytecode compiler with all opcodes including superinstructions
- 58 unit + integration tests
Makefile targets:
make test-all run Python (571) + C (58) tests
make examples run examples in both, compare output
make friction benchmark same .lsp in Python vs C
make c-build build C interpreter
make c-test run C tests
make c-repl C REPL
