New: wasm/tests/parity-cross-tier.mjs runs the parity-corpus.mjs (216
test cases tagged by whitepaper section / R7RS concept) against three
tiers — native python (reference), c-wasm, asm-wasm — and fails on any
unknown divergence. Known gaps live in KNOWN_DIVERGE so the table stays
green while the bignum / call/cc / etc. work proceeds.
Wired into `make wasm-test` so a regression against any spec claim gets
caught before merge.
Bugs caught and fixed:
- python remainder: was `signed_a % signed_b * sign(a)`, which double-
applied the sign of a (python's % floors) — gave -3 for (-17, 5)
instead of the R7RS-correct -2. Now uses abs() on both sides.
- asm-wasm modulo: was i32.rem_s (truncated, remainder semantics)
where R7RS modulo wants sign of divisor. Added the "if rem and
divisor disagree on sign, add divisor" branch.
Cross-tier numbers after fix:
216 passing
3 known diverge: expt-2-100, expt-3-50, big-arith — all asm-wasm
(no bignums on the asm tier yet; whitepaper §2.1 claim still open)
0 fail
REPL layout: body is now the scroll container, prompt-bar is
position:fixed at the viewport bottom so it doesn't get pushed off
screen by a long transcript. Empty space above the prompt on a fresh
session reads like a terminal.
All other tests still pass: 20 unit, 8 integration, 11 functional.
Both bytecode VMs had a latent O(n^2) defect on self-recursive tail
calls invoked from inside let/let*/letrec/letrec*/do bodies. The
self-tail-call op assumed reusing "current env" was safe, but current
env was the innermost let* frame, not the lambda body env. Each iter
pushed a fresh let* frame on top (PUSH_ENV at compile site), the
self-tail-call rebound params into that frame & jumped to ip=0 without
unwinding. Env chain grew linearly with iters; every var lookup walked
O(n) chain; effective O(n^2) behaviour.
Symptom observed 2026-06-14: 156k circ-ops walk hung > 5min instead of
1.4s. K=5 doctrine reducers ran 30+ runaway lumbda procs at 99% CPU
across multiple `make sweep-doctrine` invocations before we tracked
it back to language layer (initially misdiagnosed as K=5 substrate).
Fix: track scope depth at compile time on CodeObj (scope_depth bumped
on PUSH_ENV emit, decremented on POP_ENV emit). Record self_base at
lambda body entry (0 unless internal defines pushed a frame). At
self-tail-call emit, encode pops_needed = scope_depth - self_base in
the op arg. Runtime handler unwinds that many env frames before
rebinding params + jumping to ip=0.
Tree-walker (c/lumbda without --fast) already worked - it walks the
ast & lets recursion clean up frames naturally. Asm tier also fine -
no self-tail-call op, uses different lambda-call convention.
Verification:
python tier: 571 tests PASS, our 100k let* repro 1.04s wall (was infinite)
c tier: 205 tests PASS, same repro 0.05s wall (was infinite)
asm tier: 158 tests PASS (no fix needed, never had the bug)
Portal-resume backwards-compat: pre-fix portals stored OP_SELF_TAIL_CALL
arg as 2-tuple. Deserializer fills pops=0 when 'pops' key is absent,
so an old portal resumes at correct behaviour at the cost of slow walk
on its very next self-tail-call body (no worse than pre-fix).
Memory note saved at reference_lumbda_let_star_in_tail_loop in our
foxhop blackops memory for future agents.
Substrate for fork-per-accept pattern in gpu-worker.lsp — enables
async bend dispatch with internal load balancing.
c-tier (builtins.c):
- bi_fork_self: fork() wrapper, returns 0 in child / pid in parent / #f on fail
- bi_waitpid_nonblock: waitpid(-1, WNOHANG), returns reaped pid or 0
- bi_exit_immediate: _exit() wrapper — REQUIRED in fork-self children,
regular exit() runs atexit handlers against shared parent state and
hangs the child (observed empirically 2026-06-11 via vm-runner.sh).
- bi_sleep: real wall-clock sleep(3) — yields CPU. Replaces busy-loop
patterns that would (a) burn CPU and (b) SIGKILL in cgroup-limited
VMs (observed: 100M iter let-loop SIGKILL'd after 5s in qemu vm).
python-tier (lumbda.py): _fork_self / _waitpid_nonblock / _exit_immediate
/ _sleep mirrors via os.fork / os.waitpid / os._exit / time.sleep.
Tested in vm-runner.sh VM (Ubuntu 2G/2vCPU): 3-child fork-cycle test
spawns + reaps cleanly 3/3 in both c-tier + python-tier. The exact
test pattern that crashed neoblanka host pre-fix now works fine.
Asm tier: deferred. Lock retained at chmod a-x ~/git/lumbda/asm/lumbda*
per CLAUDE.md threat model.
ecdsa's Phase B emit at full secp256k1 width writes a 4–6 GB QECCOPS1
ops binary. The existing path — accumulate in a string-output port,
materialize via get-output-string, write — peaks RAM at 3× body
size (port internal buffer + Scheme string copy + write-binary-file
concat). A 6 GB body needs ~18 GB transient; OOMs a 16 GB QEMU guest.
This commit shifts emit-stream onto a constant-RAM file-port path
and fixes binary-correctness defects in the supporting primitives.
New primitives (mirrored across c/builtins.c + lumbda.py):
- open-binary-output-file path
Opens in "w+b" so the caller can seek back to rewrite a header.
- port-set-position! port offset
fseek absolute offset on a file port. emit-stream reserves a
16-byte placeholder header, streams the body, then seeks back to
byte 0 to rewrite the QECCOPS1 + n_ops u64 LE once n_ops is known.
- append-binary-file path data
Opens in "ab" and fwrite's the bytes through. Pairs with
write-binary-file so callers can land header + body in two writes
instead of (string-append header body).
- append-port-to-binary-file path port
Streams a string-output port's buffer to disk via fwrite without
materializing (get-output-string port). Lets callers keep their
existing string-output sink and avoid the body-size string copy
if they stay on string-port emit.
Binary-correctness fixes:
- bi_write_string to a file port used fputs, which calls strlen.
Binary payloads containing 0x00 truncated at the first null byte.
Switched the file-port branch to fwrite with the string's known
->len (same fix family as the earlier bi_get_output_string
strlen defect).
- bi_write_char per-byte fflush guarded to stdout only. With
millions of gate-bytes per second, flushing after every fputc to
a file port was a 100× slowdown. File ports buffer until close
or explicit flush-port — keep stdout's per-byte feedback path,
drop fflush on every file-port byte.
- port_write_str grows 1.5× past 256 MB instead of 2× throughout.
At realloc time the transient peak is old + new; 2× at 8 GB →
16 GB transient needs 24 GB. 1.5× bounds peak at 2.5× and keeps
multi-GB string-port workloads inside a 16 GB VM.
Tests: 88/88 c-test, 4/4 regression-named-let-leak, 205/205
functional, zoe-favorites all tiers. Binary roundtrip with embedded
nulls at 10/1000/100000 bytes passes byte-for-byte.
End-to-end: foxhop ecdsa DIALOG_GCD secp256k1 emit lands a 4.7 GB
binary at 322 MB peak RSS in 7:41 wall on a 16 GB QEMU guest.
Foxhop ecdsa point-add emit at n+1=257 secp256k1 (~15M ops) overruns
host RAM via the accumulator path — walk-circuit-ops builds a 4 GB
Scheme list, op-specs->bytes builds 840 MB body string before
write-binary-file ships it. sweep-secp256k1.lsp OOMs at 8 GB.
Streaming primitive walks the circuit & writes each 56-byte op record
directly into an open binary file handle in a single pass:
(emit-circuit-to-ops-bin-stream out-path registers ops)
Per-op: pack 56 bytes via the same struct.pack format as
op-specs->bytes, append to an 8 KiB batch buffer, flush to disk when
threshold hits. After the walk closes, seek(8) patches the n_ops
header field. No intermediate list, no intermediate body bytes.
Memory profile at n+1=32 / 146 ops: ~64 KB grew during emit (host
file buffer + interpreter overhead). Holds flat at n+1=256 / 1154 ops
& n+1=1024 / 4610 ops. Will hold flat at n+1=257 secp256k1 / ~15M
ops — the byte-level operation count grows linearly but RSS does not.
Dispatch table mirrors _walk_circuit_ops exactly so bytes are
bit-identical to (op-specs->bytes (walk-circuit-ops ...)) by
construction. Verified byte-for-byte at p=11 (48 763 ops), p=251
(159 263 ops), & a hand-rolled n+1=32 hand circuit.
Foxhop wrapper lands as (emit-ops-bin-stream out-path c) in
ecdsa/lumbda/emit-ops-bin.lsp. Existing (emit-ops-bin ops out-path)
accumulator API stays untouched for backward compatibility — all
288 ecdsa/tests/unit/test-emit-ops-bin.lsp assertions still pass.
Three Python-tier primitives that lift foxhop ecdsa's emit-ops-bin
pipeline out of the Scheme interpreter:
- walk-circuit-ops registers ops → list of op-spec vectors. Mirrors
ecdsa/lumbda/emit-ops-bin.lsp's walk-op dispatch table (alloc, free,
x, z, cx, cz, swap, ccx, ccz) in pure Python with interned-symbol
identity dispatch & a Python dict for the qubit-layout. Replaces a
Scheme named-let walk that paid ~5-9 ms per op via per-iteration
closure / let* / cons / append overhead — wall dropped 252 sec to
237 ms on a 32 k-op p=11 case.
- op-specs->bytes vector-list → 56*N latin-1 string. Each op-spec is a
7-element vector packed via struct.Struct('<IIQQQQQQ').pack; results
joined once with b''.join + .decode('latin-1') so the existing
write-binary-file primitive ships the body byte-for-byte. Replaces
per-op (string-append (u32-le ...) (u32-le 0) (u64-le ...) ...) in
Scheme that paid ~2-3 ms per op; wall dropped 49 sec to 84 ms on the
same case.
- count-lumbda-ops ops → vector(toffoli clifford total). Tags ccx →
toffoli, x|cx → clifford, anything else → total only. Replaces a
pure-Scheme named-let count that hit ~200 sec per variant on a 32 k-op
list.
Net foxhop ecdsa wall on the canonical p=11 textbook+refined emit pair
dropped from ~10 min to ~7 sec — ~85x end-to-end on Python tier. Output
byte-identical against a pre-rewrite reference file (cmp clean on both
textbook & refined paths).
C-tier port: TBD (same algorithms, separate translation unit).
The S-expression wire format was the bottleneck at huge payload sizes
-- 23.8 s end-to-end for 1M x 16 B inputs on the Python tier, while
the actual CUDA kernel finishes the same workload in ~47 ms. The
hex-S-exp parser ate everything between.
New binary wire mode (magic 'BSHK' prefix; payload is the daemon's
binary portal format verbatim) bypasses S-expression parsing entirely.
Worker writes the blob to disk, calls daemon process-bin, reads result,
prepends 'BSHR' magic, replies.
Measured 3090-ai, daemon warm, localhost:
workload Py S-exp Py binary C S-exp C binary
100 x 16 B 3.43 ms 0.74 ms 0.40 ms 0.15 ms
1k x 16 B 23.24 ms 0.76 ms 2.77 ms 0.22 ms
10k x 16 B 218.82 ms 1.27 ms CLIFF 0.88 ms
100k x 16 B 2,219 ms 10.18 ms CLIFF 10.35 ms
1M x 16 B 23,811 ms 159 ms CLIFF 157 ms
150x speedup at 1M inputs on Python tier. C tier S-exp CLIFFs
between 1k and 10k inputs (reader payload limit); binary mode
bypasses the CLIFF entirely. At 100k+ inputs both tiers converge
since file I/O + CUDA kernel dominates over wire framing.
Host comparison: hashlib.shake_256 over 1M tiny inputs takes ~2 s
on a single Python core. Bend via binary worker = 157 ms = 12x
faster than host. Bend now wins at huge workloads, not just heavy
ones.
Implementation:
lumbda.py
* tcp-send/tcp-recv switched to latin-1 (1:1 byte mapping)
so binary payloads pass through cleanly. UTF-8 was mangling
bytes with replacement chars.
* write-binary-file / read-binary-file primitives.
c/builtins.c
* write-binary-file / read-binary-file matching Python tier.
examples/cuda-fanout/wire.lsp
* wire-send-raw / wire-recv-raw helpers that frame a raw
payload string without S-expression serialization.
examples/cuda-fanout/gpu-worker.lsp
* handle-binary-shake: write portal blob, daemon process-bin,
read result, wire-send 'BSHR' + bytes.
* handle-one dispatches on first 4 bytes of payload: 'BSHK'
goes to binary path, anything else stays S-exp.
examples/cuda-fanout/bench_tiers.py
* make_payload_binary builds the BSHK protocol payload.
* --binary flag in CLI.
www/index.html
* full S-exp + binary comparison table.
* 'bend now beats host hashlib at huge workloads' headline finding.
Two new primitives in the Python tier dispatch table make
gpu-worker.lsp (pure Scheme) replace mock-worker.py:
(spawn-process-stdio path args) → (stdin-port . stdout-port)
spawns a long-running subprocess via subprocess.Popen with
stdin/stdout piped + line-buffered. Returns a Pair the
caller holds across many request cycles.
(flush-port port)
flushes a write port. No-op when port has no flush method.
read-line also extended to accept file-like ports (subprocess pipes)
not just StringInputPort / sys.stdin.
gpu-worker.lsp fixes:
- run-loop split out as its own tail-recursive function (named-let
inside cond was harder to debug than non-named explicit recursion)
- handle-cuda-shake-fanout unwraps (quote ...) wrapping that
bend.lsp adds when it serializes `'expr` through write-to-string
End-to-end on 3090-ai (lumbda Python tier as both client + worker):
shell A: python3 -u lumbda.py /tmp/launch-worker.lsp
→ gpu-worker: ready cuda-shake-fanout ← ./shake256-fanout
→ gpu-worker listening on port 9091
shell B: python3 lumbda.py smoke-bend.lsp # run 3×
=== smoke-bend ===
1. cost estimator picks local for 3 inputs (cost too small): OK
2. worker available? #t
3. bend! (cuda-shake-fanout '("00" "01" "deadbeef") 32):
("b8d01df855…" "94da6280b2…" "fa094fa86e…")
All three runs identical bytes. All three hashes byte-identical to
hashlib.shake_256 — verified across the full chain:
lumbda Python → bend macro → wire-send (length-prefixed S-exp)
→ gpu-worker.lsp (pure Scheme) → spawn-process-stdio
→ shake256-fanout --daemon (warm CUDA context on 3090)
→ kernel → output portal → wire-send response → bend returns
No Python mock anywhere — except the leaf CUDA binary, which is the
point of the contribution.
Documented in README.md including the python3 -u footnote for
buffering. Once a Scheme-level (flush-port (current-output-port))
is wired into the worker loop, even -u becomes optional.
Per-tier status after this commit:
Python tier ✓ end-to-end working
C tier → still needs spawn-process-stdio + flush-port in
its primitive dispatch (Scheme files unchanged)
asm tier → same, plus raw fork+pipe+execve syscalls for the
spawn primitive
Two defects uncovered during secp256k1 oracle port; both forced workarounds
downstream in ecdsa/lumbda/secp256k1.lsp that can now retire.
(1) `quotient` used `int(a / b)` — Python float division. Past 2^53 the
float lost precision, so mod-pow on secp256k1's 2^256 prime corrupted
every modular inverse with off-by-one errors in the square-and-multiply
loop. Replaced with R7RS-spec truncate-toward-zero integer division.
Verified: (quotient 7 2)=3, (quotient -7 2)=-3, (quotient 7 -2)=-3,
(quotient -7 -2)=3, (quotient 0 5)=0, (quotient 2305843009213693950 2)
now returns 1152921504606846975 (was 1152921504606846976, off by 1).
(2) Reader `_atom` looped `for conv in (int, float)` and Python's float()
accepts bare 'inf', 'infinity', 'nan' as IEEE specials. So `'(infinity)`
silently parsed as `(+inf.0)` and `(symbol? 'infinity)` returned #f.
R7RS spells these +inf.0 / -inf.0 / +nan.0 explicitly. Restricted the
float-parse path to a strict decimal/exponent regex; the named IEEE
specials still match their proper spellings.
Verified: (symbol? 'infinity)=#t (was #f), (symbol? 'inf)=#t,
'(infinity) reads as the symbol list, (positive? +inf.0)=#t still works.
Tests: 571 Python unit + 205 functional (Python + C) all pass.
Ticket 0002 — reads 8 bytes from /dev/urandom (little-endian u64) and
seeds xoshiro256**. Opt-in kernel entropy for stochastic runs; the
default stays deterministic (k=0 at startup), so ticket 0001's
portal-reproducibility contract is unchanged.
Real-world flow now one call away:
Machine A: (random-seed-from-os!) + run simulation + portal-save
Machine B: portal-resume — same stream, bit-for-bit
All three impls fail loud on /dev/urandom trouble (LispErr in Python
and C, stderr + exit(1) in asm) — no silent fallback to a weak seed.
Tests:
- tests/functional.lsp: 2 new shared asserts (entropic + replay)
- asm/test.sh: 2 new asm-local checks (149 total, was 147)
- make test-all green across Python (205), C (205), asm (149)
Whitepaper §7.5 gains one sentence noting the OS-seed path.
unmoad: zero new findings in added code.
Completes ticket 0001 started in 27f468c. All three impls now carry
bit-identical xoshiro256**; portal state round-trips across process
boundaries in every producer x consumer cell (Python <-> C <-> asm).
asm impl:
- 4 new builtins: random-seed!, random-int, random-state, random-state!
- g_rng_state in BSS (4 x u64); rng_splitmix64_step, rng_seed, rng_next
- Binary portal header bumped LUMBDAB1/48 -> LUMBDAB2/80; carries
32 bytes of rng state at offsets 40..64, reserved moved to 72
- No float support in asm, so (random) intentionally omitted there
- _start seeds with 0 so the stream is deterministic from startup
Python + C (supplements 27f468c):
- rng_seed(0) auto-invoked at module load / register_portal_builtins
so (random) without explicit (random-seed!) returns a real value
instead of the all-zero xoshiro fixed point
Tests:
- tests/functional.lsp: 7 new shared assertions (Python + C)
- asm/test.sh: 5 new asm-local assertions (142 -> 147)
- tests/portal-rng-save.lsp / portal-rng-load.lsp: portable S-expression
portal that captures both state AND next-5 baseline so loader self-
verifies without a separate harness
- tests/portal-cross-test.sh: 9 new producer x consumer RNG cells; all
18 cells pass end-to-end
Verified: seed=42, (random-int 1000000) draws 1..10 =
558742 543102 559009 124193 317476 750584 200754 814407 344958 929085
identical in Python, C, and asm.
unmoad scan: zero new findings in added code.
Zoe's contribution question: does our portal preserve RNG state so a
simulation can continue in another process with the same random stream?
Answer today: no — no RNG existed. Answer now (Python + C): yes, bit-identical.
- New builtins: random-seed!, random, random-int, random-state, random-state!
- xoshiro256** (Blackman & Vigna 2018) — deterministic, portable, no libc rand
- State = 4 x u64; portal-v1 JSON gains 'rng' field with 8 x u32 halves
- Python and C produce bit-identical streams (verified: seed=42, 10 draws)
- Asm impl + cross-impl tests + whitepaper note: next commits
Ticket: docs/tickets/0001-portal-rng.md
Historical internal name "uncommonlisp" retired in favor of the
public name "lumbda" ahead of lumbda.com going live. Scope of
this commit:
Source files renamed:
uncommonlisp.py -> lumbda.py
asm/uncommonlisp.s -> asm/lumbda.s
c/uncommonlisp.h -> c/lumbda.h
whitepaper/uncommonlisp-whitepaper -> whitepaper/lumbda-whitepaper (.rst + .pdf)
Binaries renamed (tracked ones; c/ was always gitignored):
asm/uncommonlisp, asm/uncommonlisp-gc, asm/uncommonlisp.o,
asm/uncommonlisp-gc.o -> asm/lumbda(-gc)(.o)
c/.gitignore -> ignores lumbda
Internal string updates (sed pass ordered longest-first):
asm/uncommonlisp -> asm/lumbda
c/uncommonlisp -> c/lumbda
uncommonlisp.py -> lumbda.py
UNCOMMONLISP_BIN -> LUMBDA_BIN (asm/test.sh env var)
"uncommonlisp> " -> "lumbda> " (asm REPL prompt baked into binary)
UNCOMMONLISP -> LUMBDA (macros, comments)
uncommonlisp -> lumbda (prose)
Binary portal magic updated:
"ULPORTAL" -> "LUMBDAB1" # "Lumbda Binary v1"
Old portal files are not backward-compatible — this is a deliberate
break since it's the rename moment. S-expression portals already
carry their own ";; lumbda-portal v1" header and remain cleanly
versioned.
WHITEPAPER.pdf / WHITEPAPER.rst symlinks repointed to the renamed
files. Makefile's whitepaper target targets lumbda-whitepaper.pdf.
Not changed (intentional, separate phases):
- Filesystem directory /home/fox/git/uncommonlisp itself
(fox renames locally and the gitlab repo URL in a follow-up)
- tests.py hardcoded cwd=/home/fox/git/uncommonlisp
(matches the current on-disk location; will flip when the
directory rename ships)
- Git history (immutable; old commits still say uncommonlisp,
which is correct — that's what they were)
Verified:
137 asm no-GC + 137 asm GC + 571 Python + 83 C + 189 shared
functional tests all pass under the new names.
bench-gc-http (2000 req): all 4 cells behave as expected
(cells 1/2 flat, 3 leaks, 4 bounded at 1 chunk).
Python REPL, C REPL, asm REPL all start cleanly.
