handle-binary-shake, handle-binary-cgbn, handle-binary-secp each
called (delete-file in-path) (delete-file out-path) unconditionally
in both the daemon-ok and daemon-error branches — 6 sites total.
When the daemon failed without producing out-path (e.g. crash, OOM,
bad payload), delete-file raised file-not-found and the entire
listener exited. Crashed 3090-ai once during ai.foxhop.net deployment
smoke.
Wrap every delete-file with (if (file-exists? PATH) (delete-file PATH))
so a missing portal cannot kill the worker. (ok pong) sanity check
passes on Python tier.
Sites (12 guards = 6 path pairs × 2 paths):
handle-binary-shake : ok branch + error branch
handle-binary-cgbn : ok branch + error branch
handle-binary-secp : ok branch + error branch
foxhop CLAUDE.md update: www.foxhop.net is private (book.foxhop.net
source + research artifacts). URLs like https://www.foxhop.net/ecdsa.html
do not exist & must not appear in public posts or external links.
Public-facing copy of ECDSA work belongs on foxhop.net wiki (remarkbox
RST pages), not as static HTML under www.foxhop.net/.
Stripped two external links:
bend.html — "secp256k1 point-addition challenge work at
foxhop.net/ecdsa" → "(research artifact, internal repo)"
index.html — same pattern → "internal ECDSA research"
Also brought index.html catalog highlight in sync with Day-3:
was: secp256k1 batched point-mul (6.5 Gkeys/s on a 4090, surveyed)
now: 13.6 Mkeys/s on a 3090, ~309x coincurve CPU
The 4090 projection moves out of the lead because we now have a
measured number on our actual hardware.
Adds *bend-workers* list with round-robin dispatch, BEND_WORKERS env
loader, and helpers (bend-set-workers!, bend-pick-worker,
bend-parse-workers-env, bend-load-workers-from-env!). Single-host
legacy callers unaffected — when *bend-workers* is empty the
dispatcher falls back to *bend-worker-host* / *bend-worker-port*.
Validated cross-tier (Python + C lumbda) against a live two-host
cluster (3090-ai.foxhop.net:9091, ai.foxhop.net:9092) — round-robin
distributes evenly; cgbn mod-mul results byte-identical to gmpy2
reference on both hosts.
Form A Day-3 shipped at lumbda ecfe27a. Windowed-G ladder (w=4,
16-entry G-table built once via batch-inverse on the 15 Z-coords)
landed clean.
Bench on 3090, n=1M (best-of-3, --no-batch-inv):
v1 (Day-1) 127.04 ms → 7.87 Mkeys/s
v3 (Day-3 w=4) 73.54 ms → 13.60 Mkeys/s (1.73x v1, ~309x coincurve)
Byte-identity PASS at n in {32, 1000, 10000, 100000, 1000000} plus
known-small edge cases (k in {1, 2, 3, 7, 0xdeadbeef, n-1, n, 2^128-1}).
Daemon default still serves v1; --window-w 4 flag selects v3
explicitly. Fox's call on flipping the daemon default.
Day-4 plan: stack v3 + Day-2 batch-inv. v3 cut scalar_mul; the
residual inversion cost now actually matters, which is what Day-2
needed to win.
Three agents still in flight: walker promotion (#39), lever
generator (#40), ai.foxhop.net second worker (#30).
Per Day-2 progress doc the kernel bottleneck was scalar_mul (256 doubles +
~128 adds per scalar), not _ModInv. v3 swaps the binary double-and-add for
a windowed-base ladder: precompute table[0..15] = i*P affine on-device,
walk scalar 4 bits at a time MSB->LSB, cutting per-scalar adds from ~128
to ~63. Table loaded into __shared__ (1024 B) per block.
Benched on 3090-ai (best-of-3, --no-batch-inv):
n=10k: v1 4.22 Mkeys/s v3 2.18 Mkeys/s 0.52x (init overhead dominates)
n=100k: v1 7.48 Mkeys/s v3 9.29 Mkeys/s 1.24x
n=1M: v1 7.87 Mkeys/s v3 13.60 Mkeys/s 1.73x
Byte-identical against coincurve at n in {32, 1000, 10000, 100000, 1000000}.
CLI: --window-w 4 selects v3 (Day-3 canonical). w=8 reserved but stub-
rejected since device-side table init is register-stack-bounded at W <= 16.
Default no-flag behaviour stays v1 (Day-1) so the gpu-worker.lsp daemon
inherits the safe baseline until fox routes traffic to v3.
Companion progress doc at plans/form-A-day3-progress.md.
Captures empirical record from three agents that finished today:
form-A-day2-progress.md (STOPPED, no merge)
v2 Montgomery batch inversion regressed v1 by 0.63x-0.94x
across n in {10k, 100k, 1M}. Byte-identity PASS at every N;
math correct. Root cause: at this N, scalar_mul (256 Jacobian
doubles x 5-6 ModMult each) dominates, NOT _ModInv. v2's Phase
B/D used 1 thread/block leaving ~97% of SMs idle. v1 baseline
re-measured at 7.88 Mkeys/s at n=1M (catalog upward correction
from initial 6.51). Day-3 path: warp-level prefix scan OR
windowed-G ladder.
form-D-axis-flip-RESULTS.md (Form D opt 1 shipped foxhop 1f7ac9d)
Per-candidate axis kernel measured 217 Mops/s at K=32 M=4 on
3090; 23.7x over per-shot N=4 at same M. Both axes saturate
at the same ~220-250 Mops/s, refuting the bandwidth-bound
diagnosis. Axis flip's win is occupancy-amortization at small
M, not bandwidth redistribution.
form-D-build-progress.md (Form D AG kernel parked)
Aaronson-Gottesman tableau dead-end: our point-add circuit
contains no H or S, so state never leaves the computational
basis and AG buys nothing. Three pivot options proposed; fox
picked options 1 + 3 in parallel.
Day-1 binary at lumbda 7661788 stays canonical for
cuda-secp256k1-batched-mul; v2 working-tree code lives uncommitted
on the build host as Day-3 scratch.
Form D Option 3 (QECCOPS2 packed op format) shipped at foxhop
commit 90484ca. Numbers:
n_batches unpacked packed speedup
16 5842 ms 5428 1.076x
64 6226 ms 5830 1.068x
128 6604 ms 6201 1.065x
On-disk shrink: 716 MB → 307 MB (2.33x). Storage win, not compute win.
Critical diagnostic correction from the opt 3 agent: the 3.5x
bandwidth projection was WRONG because ops_loader.c already
narrowed u64 → u32 on load, so in-VRAM Op was already 28 B, not
56 B. Max realistic VRAM-reduction was 28 → 24 B = 1.17x best
case. The measured 1.07x matches: per-shot state traffic (qubits
+ bits per thread) is ~85x larger than the op stream, which
warp-broadcasts through L1/L2. Op-stream bandwidth was not the
bottleneck.
Both Form D pivots (opt 1 axis-flip, opt 3 packing) converged on
the same finding: the 3090 is compute-saturated at ~250 Mops/s on
the kickmix circuit, not bandwidth-saturated. Algorithm + layout
tweaks already extracted; the real next macro-lever is multi-GPU
fan-out across the fleet (3090-ai + ai.foxhop.net 4090 + future
nodes, each saturating its own ceiling in parallel).
cuda-sim-axis-flip remains LIVE as a tool with specific use:
many-candidates × few-shots search-loop early-screen. ops.bin
packing remains useful for fleet rsync (3.5x smaller payloads
across the LAN matters when shipping candidate variants).
QECCOPS2 packed-op format landed in foxhop ecdsa/cuda/ at commit
90484ca. 28 → 24 B per op in VRAM, 56 → 24 B on disk (2.33× shrink).
n_batches 16/64/128 on RTX 3090: ~7% kernel speedup, byte-identical
CPU vs unpacked-GPU vs packed-GPU. Bandwidth-bound diagnosis from
form-D-build-progress.md §4 stands — per-shot state traffic (~85×
larger than op stream) owns the 1.07× ceiling. Per-candidate axis-
flip (sim_gpu_axis.cu, foxhop 1f7ac9d) remains the open lever; this
~7% stacks on top.
Form D opt 1 (axis-flip refactor) shipped at foxhop 1f7ac9d.
Per-candidate parallelism over per-shot delivers 217 Mops/s at
K=32 candidates × M=4 shots on a 3090, byte-identical with the
CPU reference at every (K, M) pair we measured.
Key finding inside the numbers: the bandwidth-bound diagnosis
flagged in the Form D structural-finding doc was WRONG. Both
axes (per-shot N=128 and per-candidate K=32 M=4) saturate at the
SAME ~220-250 Mops/s on the 3090. Per-shot wins by 14% at full
saturation; axis flip wins by 23.7x at small M because it fills
SMs in one launch instead of leaving them idle.
So the axis kernel is the right tool for lumbda's many-candidates
× few-shots search-loop early-screen pattern — not a replacement
for the per-shot kernel.
Form D opt 3 (ops.bin packing) still in flight. Since compute
saturates before bandwidth on this device, packing may not
deliver expected gain. Letting that agent finish so we have
empirical numbers either way.
Live forms now: 5
cuda-shake-fanout ~12x host hashlib @ 1M
cuda-sim-ops-bin 1.07x @ 128 batches (kickmix per-shot)
cuda-bignum-cgbn 1.28 Gops/s mod-mul @ 1M (256-bit)
cuda-secp256k1-batched-mul 6.51 Mkeys/s @ 100k (148x coincurve)
cuda-sim-axis-flip 217 Mops/s @ K=32 M=4 (kickmix per-candidate)
Form A (cuda-secp256k1-batched-mul) shipped earlier this session at
commit 7661788. Form B (cuda-bignum-cgbn) extended to all 9 ops at
21bd26a. Both now serving on 3090-ai gpu-worker:9091 alongside the
shake form.
Catalog Live forms table:
cuda-bignum-cgbn | 1.28 Gops/s kernel mod-mul @ n=1M, 256-bit
(~256x GMP CPU single-thread); 9 ops total
cuda-secp256k1-batched-mul| 6.51 Mkeys/s @ n=100k (~148x coincurve CPU);
Day-2 Montgomery batch inversion projected
toward FixedPaul's 6.5 Gkeys/s on 4090
Wire magics now in use:
BSHK/BSHR — cuda-shake-fanout
BCGB/BCGR — cuda-bignum-cgbn
BSCP/BSCR — cuda-secp256k1-batched-mul
(BSTB/BSTR reserved for cuda-clifford-stabilizer; parked per
structural finding — our circuit has no H/S gates)
Form D pivots (axis-flip sim_gpu.cu + ops.bin packing) still in
flight; CATALOG.md + bend.html will gain rows once they ship.
Notable defect surfaced during Form A build: widely-cited secp256k1
generator y-coordinate
0x483ADA7726A47B0DAFFA10ED2E11458A823D0E1D89DCAB14C7C39D9F8B97C20A
does NOT satisfy y^2 = x^3 + 7 mod p. Real Gy =
0x483ADA7726A3C4655DA4FBFC0E1108A8FD17B448A68554199C47D08FFB10D4B8
Cross-tutorial propagation. UNDF candidate logged.
Per examples/cuda-fanout/plans/form-A-secp256k1-batched-mul.md.
Batched secp256k1 scalar*G via per-thread Jacobian double-and-add
plus per-thread Z-inversion to affine. Field arithmetic uses
FixedPaul/VanitySearch-Bitcrack GPUMath.h verbatim
(commit 66e6f9d, AGPL-3.0, vendored under
vendor/vanity-search-bitcrack/).
Wire: BSCP request / BSCR response distinct from BSHK/BCGB.
"BSCP" u32 op_id u32 n base_xy(64B BE) scalars(n*32B BE)
"BSCR" u32 status u32 n points(n*64B BE x||y)
Validation against coincurve on 3090-ai.foxhop.net byte-identical
across known-small (k in {1,2,3,7,0xdeadbeef,n-1,n,2^128-1})
& random sweeps at n in {32, 1k, 10k, 100k}.
Measured throughput on 3090, kernel-only:
n=10k 2.32 ms 4.31 Mkeys/s
n=100k 15.37 ms 6.51 Mkeys/s
End-to-end over warm TCP daemon from another host:
n=100k 400 ms 250 kkeys/s wall (PCIe + wire serialization
bound; kernel still <16 ms)
Speedup vs coincurve CPU single-thread (~44 kkeys/s host)
~148x at n=100k kernel-only. Day-2 work to add _ModInvGrouped
batched inversion should push toward FixedPaul's 6.9 Gkeys/s
published on 4090.
gpu-worker.lsp: maybe-register-daemon! for cuda-secp256k1-batched-mul,
handle-binary-secp branch in handle-one dispatching on BSCP magic.
Makefile: secp256k1-batch-mul / secp256k1-test / secp256k1-bench
targets. Test harness ships with coincurve preferred, falls back to
python-ecdsa or pure-Python double-and-add for the host oracle.
Form D build agent discovered: our point-add circuit contains no
Hadamard or S gates (only X/CX/CCX/CZ/CCZ/SWAP/R/HMR/Z/NEG).
State never leaves the computational basis. Aaronson-Gottesman
tableau compression buys nothing when superposition does not
exist; reduces to exactly what sim_gpu.cu already does at one
bit per qubit per shot.
Toffoli fraction measured 13.87% (well under the 40% threshold
the planner flagged). The 1.07x cuda-sim-ops-bin ceiling traces
to memory-bandwidth on per-shot striped state — not algorithm.
STABSim-class wins remain valid for QEC / surface-code work where
H + S exist; that's a future workload.
Two replacement directions queued:
1. axis-flip sim_gpu.cu — per-candidate parallelism over per-shot.
~3 days. Reuses BSHK new op_id; no AG tableau.
2. ops.bin packing — 56→16 B per op halves global-memory traffic.
Addresses the actual bottleneck.
Catalog + public bend.html both updated. Form D progress doc at
examples/cuda-fanout/plans/form-D-build-progress.md documents the
structural reasoning in full.
Lands the remaining 8 ops from plans/form-B-bignum-cgbn.md §2:
0x01 mod-add cgbn_add + carry-or-ge-modulus subtract
0x02 mod-sub cgbn_sub + borrow conditional add
0x04 mod-sqr cgbn_sqr_wide + cgbn_rem_wide
0x05 mod-inv cgbn_modular_inverse (binary GCD)
0x06 mod-exp cgbn_modular_power (binary ladder)
0x07 mod-reduce cgbn_rem standalone
0x08 add-no-mod cgbn_add, truncated 256-bit
0x09 mul-no-mod cgbn_mul_wide, full 512-bit output (low|high)
process_one_bin now classifies op_id into three families (binary-mod /
unary-mod / no-mod), validates wire size per family, and carves modulus
/ a / b pointers accordingly. Output buffer width is 2x for 0x09 only.
test_cgbn_known_answers.py extended: one driver per op, gmpy2 reference
(with pure-Python fallback for invert/powmod), validated byte-identical
across n in {32, 1k, 10k, 100k}. ALL PASS on 3090-ai.foxhop.net.
Measured kernel throughput at n=100k (single 3090, median of 3):
mod-add 0.19 ms 526 Mops/s
mod-sub 0.19 ms 526 Mops/s
mod-mul 0.24 ms 417 Mops/s
mod-sqr 0.24 ms 417 Mops/s
mod-inv 2.39 ms 42 Mops/s
mod-exp 1.02 ms 98 Mops/s (16-bit exponents)
mod-reduce 0.20 ms 500 Mops/s
add-no-mod 0.19 ms 526 Mops/s
mul-no-mod 0.19 ms 526 Mops/s
mod-inv at 42 Mops/s tracks plan §8 projection (50-100 Mops/s on 3090
via CGBN's binary GCD) on the low end — Bernstein-Yang batched inverse
(form E) remains the upgrade path. mod-exp 98 Mops/s is for short
exponents only; full 256-bit ladder will drop ~16x per plan §8.
Day-1 baseline per examples/cuda-fanout/plans/form-B-bignum-cgbn.md
lands at 1.28 Gops/s 256-bit mod-mul kernel throughput on a 3090
@ n=1M instances. ~256x over single-thread GMP CPU (5 Mops/s).
Validated byte-identical with gmpy2 reference at n=32, 1k, 10k,
100k across three modulus families (secp256k1 prime, Mersenne-ish,
arbitrary odd) — all PASS.
Files:
cgbn-batch-worker.cu Day-1 binary: --daemon + --binary modes,
op_id 0x03 mod-mul at 256-bit width,
BCGB/BCGR wire (distinct magic from SHAKE's
BSHK/BSHR so gpu-worker.lsp can route).
Includes gmp.h before cgbn.h so CGBN's
dispatch picks cgbn_mpz.h (host path) instead
of the unimplemented cgbn_cpu.h stub.
Drops const from kernel args (CGBN API
non-const).
Makefile cgbn-batch-worker target, CGBN_INC env var.
gpu-worker.lsp handle-binary-cgbn routes BCGB-prefixed
BSHK payloads through the CGBN daemon;
maybe-register-daemon! lets a worker host
skip forms whose binaries aren't installed.
test_cgbn_known_answers.py
gmpy2 cross-validation harness; falls back to
pure-Python pow(a*b,1,m) if gmpy2 missing.
Per-call wall-time stays ~160ms because of cold cudaMalloc + context
init each --binary spawn. The plan-projected 15k crossover applies to
daemon mode (warm context). Daemon wiring lands in the next commit.
Remaining ops (0x01 mod-add, 0x02 mod-sub, 0x04 mod-sqr, 0x05 mod-inv,
0x06 mod-exp, 0x07 mod-reduce, 0x08 add-no-mod, 0x09 mul-no-mod) land
per-op as we measure each.
Splits a dense bend section out of index.html (now 6 lines: tagline,
example, three highlights, CTA) into a new public page at
lumbda.com/bend.html carrying:
* full wire protocol (S-exp + binary BSHK modes)
* tier-choice table (Py / C / asm × S-exp / binary)
* real workload table from foxhop ecdsafail measurements
* 2 live forms + 7 surveyed forms (Wave 1: secp256k1 batch-mul,
CGBN bignum, Pollard rho, Stim-on-GPU stabilizer, Bernstein-Yang
inverse, NTT, CUB sort/scan)
* 20 surveyed forms (Wave 2) sorted by speedup descending —
minhash 600-1000x, cuckoo filter 378x, ChaCha20 400 GB/s,
SAT 93x, Dilithium PQ 57.7x, BLAKE3 tree, cuFFT batched,
Bloom filter modern, GEMM FP8, hash-join 1.8T tuples/s on
1024xA100, cuGraph 38B TEPS, TRUST triangle 1T TEPS, nvCOMP 2.2x
* skipped section listing forms that don't pass the threshold,
so the catalog stays honest about what GPU dominates vs not
* cited canonical references for every entry
CATALOG.md mirrors the same data — single source of truth in repo;
bend.html renders the same metadata for public reading.
Build order remains: A cuda-secp256k1-batched-mul, D Clifford
stabilizer, B CGBN bignum.
gpu-worker.lsp gains a cuda-sim-ops-bin op handler that spawns
demo_ops from www.foxhop.net/ecdsa/cuda via spawn-process-stdio,
drains stdout, & parses our (cuda-sim-result ...) portal back.
Each call now emits two log lines:
;;; bend RECV cuda-sim-ops-bin ops=PATH n-batches=N t-ms=...
;;; bend DONE cuda-sim-ops-bin n-batches=N wall-ms=W cpu-ms=C gpu-ms=G mismatches=0 gpu/cpu=R
so we can tell how fast bend jobs run on CPU vs GPU per call.
CLAUDE.md & www/index.html mention this integration is now live
end-to-end across our fleet.
Two GC-build sizing fixes for the same bug class — small GC builds
silently lost live roots under load, causing hash-table-ref to report
"missing key" on entries we just set.
1. HEAP_SIZE 0x100000 → 0x400000 (1 MB → 4 MB):
32 chunks × 1 MB capped the GC build at 32 MB. ecdsa
test-mod-inv-by at p=251 (n+1=9) OOM'd here even though gc was
reclaiming, because the fragmented free-list could not satisfy
the next n+1=9 sim batch. 4 MB × 32 chunks = 128 MB cap, still
well below the 512 MB ulimit -v envelope our asm tests run under.
2. GC_MARK_STACK_CAP 16K → 256K and gc_mark_stack .skip synced to
the constant:
gc_push_if_heap silently dropped tagged values when the mark
stack overflowed and claimed "correctness preserved (sweep won't
reclaim missed-roots, just leaks one cycle)" — but a dropped
value never reaches gc_mark_drain, so its header mark bit stays
clear and gc_sweep treats it as dead. ecdsa test-mod-inv-by at
p=251 walked ~17k tagged values in a single GC cycle and tipped
over the cap, after which live cons-cells started getting
reclaimed mid-simulate.
.skip 131072 was a hard-coded constant that didn't track the .equ,
so bumping the cap without resizing the buffer would smash adjacent
gc_mark_depth / gc_collections / gc_live_bytes; both lines moved
together. 256K × 8 = 2 MB of .bss, one HEAP_SIZE chunk's worth.
After fix: ecdsa test-mod-inv-by on lumbda-full inside QEMU guest
passes p ∈ {11, 13, 251} byte-equal to Python tier. Upstream asm
test.sh stays 158/158 GREEN. Discovered while working ecdsa task
#47, 2026-06-05.
The S-expression wire format was the bottleneck at huge payload sizes
-- 23.8 s end-to-end for 1M x 16 B inputs on the Python tier, while
the actual CUDA kernel finishes the same workload in ~47 ms. The
hex-S-exp parser ate everything between.
New binary wire mode (magic 'BSHK' prefix; payload is the daemon's
binary portal format verbatim) bypasses S-expression parsing entirely.
Worker writes the blob to disk, calls daemon process-bin, reads result,
prepends 'BSHR' magic, replies.
Measured 3090-ai, daemon warm, localhost:
workload Py S-exp Py binary C S-exp C binary
100 x 16 B 3.43 ms 0.74 ms 0.40 ms 0.15 ms
1k x 16 B 23.24 ms 0.76 ms 2.77 ms 0.22 ms
10k x 16 B 218.82 ms 1.27 ms CLIFF 0.88 ms
100k x 16 B 2,219 ms 10.18 ms CLIFF 10.35 ms
1M x 16 B 23,811 ms 159 ms CLIFF 157 ms
150x speedup at 1M inputs on Python tier. C tier S-exp CLIFFs
between 1k and 10k inputs (reader payload limit); binary mode
bypasses the CLIFF entirely. At 100k+ inputs both tiers converge
since file I/O + CUDA kernel dominates over wire framing.
Host comparison: hashlib.shake_256 over 1M tiny inputs takes ~2 s
on a single Python core. Bend via binary worker = 157 ms = 12x
faster than host. Bend now wins at huge workloads, not just heavy
ones.
Implementation:
lumbda.py
* tcp-send/tcp-recv switched to latin-1 (1:1 byte mapping)
so binary payloads pass through cleanly. UTF-8 was mangling
bytes with replacement chars.
* write-binary-file / read-binary-file primitives.
c/builtins.c
* write-binary-file / read-binary-file matching Python tier.
examples/cuda-fanout/wire.lsp
* wire-send-raw / wire-recv-raw helpers that frame a raw
payload string without S-expression serialization.
examples/cuda-fanout/gpu-worker.lsp
* handle-binary-shake: write portal blob, daemon process-bin,
read result, wire-send 'BSHR' + bytes.
* handle-one dispatches on first 4 bytes of payload: 'BSHK'
goes to binary path, anything else stays S-exp.
examples/cuda-fanout/bench_tiers.py
* make_payload_binary builds the BSHK protocol payload.
* --binary flag in CLI.
www/index.html
* full S-exp + binary comparison table.
* 'bend now beats host hashlib at huge workloads' headline finding.
Three defects fixed today on the asm tier worker path:
1. Multi-line "..." docstrings crashed asm tier's scheme_read.
wire.lsp, bend.lsp, gpu-worker.lsp had docstrings spanning
several lines; replaced with ;; comments before each define.
asm tier loads these cleanly now.
2. asm tier lacked delete-file. handle-cuda-shake-fanout called
it to clean up temp portal files. Added bi_delete_file via
SYS_UNLINK = 87 syscall (~20 LoC asm). BI_DELETEFILE constant
slotted after sibling-agent's BI_STRTOSYM.
3. All Scheme files in examples/cuda-fanout/ now ASCII-only.
Earlier em-dash / × / → / μ tripped asm tier's reader in
subtle ways during file load. iconv pass + sed fixes.
Result: all three tiers complete the bench through their own
cliff. New 3-tier table:
workload Python C tier asm tier
small (3 × 16 B) 1.27 ms 0.16 ms 0.21 ms
small (100 × 16 B) 3.43 ms 0.40 ms 1.99 ms
medium (1000) 23.24 ms 2.77 ms CLIFF
med (10k) 218.82 ms CLIFF CLIFF
huge (50k) 1,099 ms CLIFF CLIFF
huge (100k) 2,219 ms CLIFF CLIFF
huge (1M) 23,811 ms CLIFF CLIFF
asm tier at 0.21 ms beats Python by 6× at smallest workload,
matches C at the bottom (~30% slower). asm cliffs at 1000;
C tier cliffs at 10k. Both cliffs are reader/buffer limits
inside the tier, not network or kernel. CUDA kernel itself
finishes 1M × 16B in ~47 ms — three orders of magnitude under
any tier's wire cost at huge scale.
bench_tiers.py made cliff-resilient: respawns worker on per-
workload failure & continues, so the full row prints for every
tier instead of bailing on first cliff.
www/index.html: full 3-column table + honest framing of when
each tier earns its slot.
wire.lsp's recv-exact previously accumulated received chunks via
`(string-append acc chunk)` in a loop — quadratic on payload size.
Replaced with a chunk-list accumulator + single `(apply string-append
…)` at the end. Lumbda's string-append knows total length up front
& allocates once.
Python tier now scales linearly across input counts (~22 µs per input):
workload Python C tier
small (3 × 16 B) 1.25 ms 0.16 ms 8× C win
small (100 × 16 B) 3.39 ms 0.40 ms 8× C win
medium (1000) 23.52 ms 2.60 ms 9× C win
med (10k) 220.15 ms (cliff)
huge (50k) 1,100 ms (cliff)
huge (100k) 2,225 ms (cliff)
huge (1M) 23,811 ms (cliff)
C tier cliffs somewhere between 1k & 10k inputs per call — its reader
hits a payload limit we still need to track down. CUDA kernel for
1M × 16B finishes in ~47 ms on this 3090, so at huge sizes the wire
cost dominates regardless of tier.
Web page updated with the linear-scaling table & honest framing: at
small inputs C wins by 9×; at huge inputs the right next move is a
binary wire mode parallel to the daemon's already-binary portal
format. Stalls are gone.
Added a write-to-string-shim.lsp for asm tier (which lacks the
native builtin); asm launch script pre-defines *argv* + loads the
shim so wire.lsp's wire-send finds a write-to-string definition.
Python/C tiers keep the native builtin — the shim is opt-in.
Bench extended with two huge workloads (100k × 16 B, 1M × 16 B).
Real numbers, 3090-ai, daemon warm, both ends localhost:
workload Python C tier C win
small (3 × 16 B) 1.16 ms 0.14 ms 8.3×
small (100 × 16 B) 3.39 ms 0.42 ms 8.1×
medium (1000 × 16 B) 23.26 ms 2.67 ms 8.7×
huge (100k × 16 B) 2,220 ms STALL n/a
huge (1M × 16 B) 24,338 ms STALL n/a
THE FINDING: at huge sizes, the bottleneck is the S-expression
text wire format, not the CUDA kernel. shake256-fanout finishes
1M × 16B in ~47 ms; the Python worker takes 24 SECONDS end-to-end
because wire.lsp's recv-exact accumulates chunks via string-append
in a loop — O(n²) at multi-MB payload sizes. C tier fails outright.
The right fix is binary wire framing between client + worker,
parallel to the binary portal format the daemon + leaf already use.
That's a separate piece of work; today's Web page edit calls it out
honestly so visitors know when bend is the right tool.
asm tier worker hosting still has process-management quirks
(doesn't survive nohup detachment in this environment); bench
ships with --skip-asm by default in this run.
Wrote examples/cuda-fanout/bench_tiers.py — spawns a worker per
tier, fires N TCP round-trips at three workload sizes through the
warm daemon, reports median + p99.
Measured on 3090-ai, daemon warm:
workload Python C tier C win
small (3 × 16 B) 1.27 ms 0.14 ms 9.1×
small (100 × 16 B) 3.46 ms 0.41 ms 8.4×
medium (1000 × 16 B) 23.51 ms 2.67 ms 8.8×
Ratio stays at ~9× across the grid — the per-byte cost of
Python's S-expression reader/printer compared to the C tier's
reader. Justifies the LUMBDA=c default landed in the previous
commit.
asm tier worker starts up & listens (after the launch script
predefines *argv* '()), but bench script saw malformed responses on
this run — likely a write-to-string format difference between asm
& Python/C reader. Leaving for follow-up; published numbers cover
the tiers that completed end-to-end.
www/index.html bend section gains the measured table under a new
'Tier choice for the worker host' subsection. Replaces the earlier
hand-wavy ~10× claim with the actual measured numbers.
C tier is ~10× faster than Python tier on wire orchestration & has
the same primitives shipped this session. Python tier still available
for development debugging via LUMBDA=python; asm via LUMBDA=asm.
Auto-builds the chosen tier (c-build or asm-build) before launching
so a fresh clone runs cleanly without 'make c-build && make gpu-worker'
two-step.
Web page Start-a-GPU-worker subsection updated to match.
When heap_alloc walks off the end of a chunk with < 16 tail bytes,
.ha_grow_no_pad skips padding and mmaps a fresh chunk — but
gc_chunk_end[N] for the abandoned chunk stayed at its full mmap
end while %r15 (high-water) sat 1..15 bytes short. The gap held
mmap-zeros that gc_sweep's .gsw_walk decoded as fake dead blocks
(header == 0, payload size == 0, mark == 0). The walker stepped
through the zeros 8 bytes at a time, and on the iteration where
%rbx == chunk_end - 8 the .gsw_dead path stored the free-list
next-pointer to 0x8(%rbx) == chunk_end — the first byte of an
unmapped page — and segfaulted with error 7.
Reproducer (inside QEMU guest, was crashing all three asm
binaries):
lumbda-full tests/unit/test-mod-solinas.lsp
→ segfault at <ptr ending 000> ip:4017d9
(.gsw_dead: mov %rdx, 0x8(%rbx))
Fix: at .ha_grow_no_pad, snapshot %r15 into gc_chunk_end[N] before
allocating the new chunk. In the padded path above the label this
is a no-op (r15 already == r13). In the un-padded path it pins
the walk bound to the high-water mark so the sweep never enters
the gap.
After fix, on lumbda-full inside the ecdsa QEMU guest:
- ecdsa test-mod-solinas: 39/39 PASS (Solinas vs Litinski
byte-equal at p ∈ {11, 13, 251})
- upstream asm test.sh: 158/158 PASS
Discovered while diagnosing ecdsa task #45.
Adds a one-shot target so any host with nvcc can stand up a bend
worker without remembering the launch incantation.
make gpu-worker
→ make -C examples/cuda-fanout shake256-fanout
→ writes examples/cuda-fanout/launch.lsp (wire + worker + main)
→ cd examples/cuda-fanout
→ python3 -u ../../lumbda.py launch.lsp --port 9091
make gpu-worker LUMBDA=c PORT=9001 # C tier host on port 9001
make gpu-worker LUMBDA=asm # asm tier host
Uses python3 -u so the worker's status lines ("gpu-worker:
ready …", "listening on port 9091") appear live; bypasses Python's
default stdout buffering that masked them during development.
www/index.html bend section gains a 'Start a GPU worker' subsection
with the same make targets so the website visitor doesn't need to
spelunk the repo to find them.
Also adds gpu-worker, gpu-worker-bin, gpu-worker-test to .PHONY.
New section between Portal and EML universality proof. Brief
explanation of the bend primitive + a worked code example showing:
- tiny inputs stay local (cost below threshold)
- heavy inputs ship to a GPU worker (cost above threshold)
- one tier-portable (bend …) macro, with bend-call as the asm form
Mentions the measured 1.5–10× wins against host hashlib on the
SHAKE256 fan-out workload, calls out the ~100 MB break-even
threshold so visitors know when bend pays off.
Cites the cuda-fanout README + integration design for callers who
want the wire contract and per-tier hosting story.
License: AGPLv3, matching the rest of the site.
Reader's .sr_string used a fixed 256-byte stack buffer with no
bounds check. Strings longer than ~272 bytes (256 + saved
registers) corrupted the saved return address and produced a
general protection fault on ret.
Reproducer:
(display (string-length "AAAA...")) ; > 272 A's → #GP at .sr_string ret
Fix:
- bump stack buffer 256 → 4096 (one page)
- add bounds check (cmpq $4080) before every char write
- on overflow exit cleanly via new die_str_overflow rather than
smashing %rip
Discovered while diagnosing ecdsa task #34: lumbda asm tier
crashed when loading ecdsa/lumbda/mod-arith.lsp because one
mod-mul! docstring is 955 bytes. Post-fix, mod-arith.lsp loads
clean and ecdsa test-mod-arith.lsp passes 33/33 on asm-full;
upstream asm test.sh stays 158/158.
Closes the last asm-tier gap for hosting bend workers:
(read-line port) → string or #f
Reads bytes one at a time from the port's fd until '\n' or EOF.
Strips the trailing newline. Returns #f when no bytes were
available (peer closed / pipe drained).
Verified end-to-end on 3090-ai:
λ> (define p (spawn-process-stdio "./shake256-fanout" (quote (--daemon))))
λ> (display (read-line (cdr p))) (newline)
ready
λ> (display "quit\n" (car p))
λ> (flush-port (car p))
λ> (display (read-line (cdr p))) (newline)
bye
asm tier can now spawn, write, flush, read line — the full
subprocess capability gpu-worker.lsp's daemon pool needs.
Implementation:
- Stack scratch buffer: 4096 bytes via %rbp (heap-safe; %r15 is
lumbda's heap pointer, do not touch)
- One-byte-at-a-time SYS_READ via fd from decode_port
- Max line: 4094 bytes (fits the daemon protocol — "ready",
"done /path", "bye" all under 100)
- heap_alloc(8 + len) + length header + byte copy + TAG_STRING
matches the Python/C tier string format byte-for-byte
- GC_NAIVE path stamps HT_STRING header for the GC pass
BI_READLINE = 123; GC_* renumbered to 124..128; BI_COUNT = 129
(GC_NAIVE) / 124. bn_readline added to symbol table + name array.
Remaining asm-tier gaps for full gpu-worker.lsp hosting:
- *argv* binding (used to parse --port)
- define-syntax + syntax-rules (used by bend.lsp macros — handled
by splitting bend-macros.lsp out in a prior commit)
- error builtin (also handled by portable bend-error wrapper)
Per-tier matrix:
Python tier ✓ macro + function client; full worker host
C tier ✓ macro + function client; full worker host
asm tier ✓ function client; subprocess primitives complete;
gpu-worker.lsp needs *argv* + a few other helpers
before pure-asm hosting is fully working
Sketches how bend would wire into ecdsafail-challenge candidate
search loop on the foxhop.net side:
ecdsa/lumbda/search.lsp
→ (bend!-call '(cuda-sim-ops-bin ops-path 141))
→ gpu-worker.lsp routes to demo_ops --portal
→ S-exp result back to lumbda, scoring proceeds
Identifies the two pieces missing before this lands:
1. `system`-equivalent primitive in lumbda (or spawn+wait via
existing spawn-process-stdio)
2. Phase B step 7 (Solinas mod-mul) so lumbda emits real-scale
ops.bin variants worth bending
Once both close, this is a half-day wire-up.
Cross-references:
~/git/www.foxhop.net/ecdsa/cuda/ — the CUDA prototype
~/git/www.foxhop.net/ecdsa/lumbda/search.lsp — current search loop
examples/cuda-fanout/DESIGN-go-gpu.md — the broader bend RPC design
asm tier lacks define-syntax + (error …) + spawn-process-stdio, so
the macro form (bend …) and the host-side gpu-worker.lsp aren't
asm-portable. But the wire protocol & TCP primitives are — asm
tier works as a bend CLIENT.
Three changes:
1. bend.lsp split into core (function-form) + bend-macros.lsp
(define-syntax wrappers). Asm tier loads core; Python/C load both.
2. Function form: (bend-call '(op . args)) / (bend!-call …) does
the same dispatch the macro does, on every tier.
3. (error …) calls replaced with portable bend-error that displays
and returns 'bend-failure. Lets asm tier handle the no-worker
case without crashing.
New file:
smoke-bend-asm.lsp — minimal asm-tier smoke test
Verified on 3090-ai:
asm tier tcp-connect to a known Python listener on :19200: PASS
asm tier loads wire.lsp + bend.lsp cleanly: PASS
bend-error portable across all three tiers: PASS
README updated with the asm tier client-only story + what would need
to land for full asm parity (the missing primitives + Scheme macros).
Per-tier status:
Python tier ✓ host + client (macro & function forms)
C tier ✓ host + client (macro & function forms)
asm tier ✓ client (function form); host pending the missing
primitives
Two new primitives in builtins.c, paralleling the Python tier shipped in
the previous commit. gpu-worker.lsp now runs on the C tier byte-identically
to the Python tier.
(spawn-process-stdio path args) → (stdin-port . stdout-port)
fork + pipe + execvp; child's stdin & stdout wired back to parent
as line-buffered FILE* ports. Accepts both strings and symbols in
the args list (matches Python tier's permissive conversion).
(flush-port port)
fflush() on the port's FILE*. No-op when fp is null.
End-to-end on 3090-ai with C-tier lumbda everywhere:
shell A: ./lumbda /tmp/launch-c.lsp
→ gpu-worker: ready cuda-shake-fanout ← ./shake256-fanout
→ gpu-worker listening on port 9091
shell B: ./lumbda smoke-bend.lsp # run 3×
=== smoke-bend ===
1. cost estimator picks local for 3 inputs: OK
2. worker available? #t
3. bend! (cuda-shake-fanout '("00" "01" "deadbeef") 32):
(b8d01df855… 94da6280b2… fa094fa86e…)
Three runs identical bytes. Same hashes as Python tier. Same hashes as
hashlib.shake_256 host reference.
Cross-tier matrix (proves wire protocol is tier-agnostic):
client tier worker tier status
─────────────────────────────────────
C tier C tier PASS — 2 sequential runs, byte-identical
Python tier C tier PASS — same hashes
C tier Python tier implicit by symmetry (same wire bytes
both directions; Python-server tested
against Python-client in prior commit)
Per-tier status after this commit:
Python tier ✓ end-to-end
C tier ✓ end-to-end + cross-tier byte-identical to Python tier
asm tier → still needs spawn-process-stdio via raw fork+pipe+
execve syscalls. Scheme files unchanged.
Two new primitives in the Python tier dispatch table make
gpu-worker.lsp (pure Scheme) replace mock-worker.py:
(spawn-process-stdio path args) → (stdin-port . stdout-port)
spawns a long-running subprocess via subprocess.Popen with
stdin/stdout piped + line-buffered. Returns a Pair the
caller holds across many request cycles.
(flush-port port)
flushes a write port. No-op when port has no flush method.
read-line also extended to accept file-like ports (subprocess pipes)
not just StringInputPort / sys.stdin.
gpu-worker.lsp fixes:
- run-loop split out as its own tail-recursive function (named-let
inside cond was harder to debug than non-named explicit recursion)
- handle-cuda-shake-fanout unwraps (quote ...) wrapping that
bend.lsp adds when it serializes `'expr` through write-to-string
End-to-end on 3090-ai (lumbda Python tier as both client + worker):
shell A: python3 -u lumbda.py /tmp/launch-worker.lsp
→ gpu-worker: ready cuda-shake-fanout ← ./shake256-fanout
→ gpu-worker listening on port 9091
shell B: python3 lumbda.py smoke-bend.lsp # run 3×
=== smoke-bend ===
1. cost estimator picks local for 3 inputs (cost too small): OK
2. worker available? #t
3. bend! (cuda-shake-fanout '("00" "01" "deadbeef") 32):
("b8d01df855…" "94da6280b2…" "fa094fa86e…")
All three runs identical bytes. All three hashes byte-identical to
hashlib.shake_256 — verified across the full chain:
lumbda Python → bend macro → wire-send (length-prefixed S-exp)
→ gpu-worker.lsp (pure Scheme) → spawn-process-stdio
→ shake256-fanout --daemon (warm CUDA context on 3090)
→ kernel → output portal → wire-send response → bend returns
No Python mock anywhere — except the leaf CUDA binary, which is the
point of the contribution.
Documented in README.md including the python3 -u footnote for
buffering. Once a Scheme-level (flush-port (current-output-port))
is wired into the worker loop, even -u becomes optional.
Per-tier status after this commit:
Python tier ✓ end-to-end working
C tier → still needs spawn-process-stdio + flush-port in
its primitive dispatch (Scheme files unchanged)
asm tier → same, plus raw fork+pipe+execve syscalls for the
spawn primitive
Wired bend.lsp to lumbda's existing TCP primitives via wire.lsp
(length-prefixed S-exp framing, lifted from
ecdsa/lumbda/fleet/wire.lsp). The (bend …) macro now actually
dispatches: lumbda → tcp-connect → wire-send → wire-recv → result.
End-to-end on the 3090 (mock-worker as gpu-worker stand-in until
spawn-process-stdio lands in lumbda's core):
λ> (load "smoke-bend.lsp")
=== smoke-bend ===
1. cost estimator picks local for 3 inputs (cost too small): OK
2. worker available? #t
3. bend! (cuda-shake-fanout '("00" "01" "deadbeef") 32):
(#xb8d01df855... #x94da6280b2... #xfa094fa86e...)
All three hashes byte-identical to hashlib.shake_256.
Files added:
wire.lsp — 8-digit-LE length-prefixed S-exp framing
smoke-bend.lsp — minimal lumbda-side test
mock-worker.py — Python stand-in for gpu-worker.lsp until
spawn-process-stdio + flush-port primitives
land in lumbda's core
bug fix:
wire-recv had one missing close-paren; lumbda surfaced it as
'unclosed (' on load. Fixed in the same commit.
mock-worker.py accepts two request shapes since bend.lsp serializes
(quote (...)) for list literals while the portal format uses
(inputs ...). Tolerating both keeps the wire protocol bend-friendly.
Per-tier integration status:
Python tier — bend, wire, smoke-test all work ✓
C tier — needs: same Scheme files port directly; tcp-* exist;
spawn-process-stdio still missing for gpu-worker.lsp
asm tier — needs: tcp-* exist; spawn-process-stdio requires raw
fork + pipe + execve in asm; biggest delta vs Python
Open primitive gaps for full cross-tier bend:
spawn-process-stdio — for gpu-worker.lsp's daemon pool
flush-port — to push daemon stdin
(current-time-ms — exists in Python tier; needed in C/asm too)
Once those land, gpu-worker.lsp replaces mock-worker.py and bend
runs cross-tier-identical. The protocol & cost-estimator code in
bend.lsp + wire.lsp need no changes — they speak only the existing
tcp-* + read-from-string + write-to-string primitives every tier
already has.
Three changes that together make the GPU primitive viable for the
go-gpu/bend pattern:
1. Binary portal format (length-prefixed raw bytes) — eliminates the
hex-string parse that ate 99% of wall time. Old text portal at
262 MB workload spent 421 sec parsing; binary format = native
speed. New flag + daemon command:
shake256-fanout --binary <in.bin> <out.bin>
daemon: process-bin <in.bin> <out.bin>
Wire (in): u32 out_bytes | u32 n | (u32 len | len bytes) × n
Wire (out): u32 n | u32 out_bytes | n × out_bytes
2. bend primitive (Lisp-smart GPU dispatch). Picked 'bend' over
{go, spark, cast, fan} per fox — HVM2 lineage, fits the
'reshape compute for GPU' mental model.
(bend (cuda-shake-fanout inputs 32))
→ runtime inspects expr; routes to GPU worker if cost-est
exceeds threshold AND worker reachable; else evaluates
locally in original lexical scope
(bend! expr)
→ force GPU, error if no worker available
Implementation files:
bend.lsp — macro + cost-estimator-based router
gpu-worker.lsp — TCP listener, dispatches over warm daemons
DESIGN-go-gpu.md — full architecture (already shipped)
Tier-specific helpers (tcp-*, spawn-process-stdio, sexp->string)
are noted as TODO per tier — Python uses subprocess + socket,
C uses fork + portal, asm uses syscall fork + sock_stream.
3. bench_binary.py — combined daemon + binary format benchmark.
GPU wins every cell of the grid by 1.5–10×:
in_sz N total host dev speedup
32 1,000,000 32 MB 470 ms 47 ms 10.11x
32 100,000 3.2 MB 47 ms 5 ms 9.95x
1024 100,000 102 MB 177 ms 79 ms 2.24x
16384 10,000 164 MB 231 ms 124 ms 1.86x
262144 1,000 262 MB 363 ms 231 ms 1.57x
Same workloads that lost 0.00× at hex+per-spawn now win 10× at
binary+daemon. 4000× relative perf swing from fixing wire format
and warming the context.
The peak 10× at small-input × high-N is the natural shape of crypto
protocols (commitments, Fiat-Shamir, PoW search). That's the win
zone for cuda-shake-fanout. README updated with the full table.
Bench findings drove three changes to the reference primitive:
1. Per-spawn mode loses to host hashlib at every size we tested.
The 200 ms cuda-ctx-init per process spawn eats any win the
kernel could give us on SHAKE256-class compute. Honest table:
in_sz N total host ms device ms kernel ms speedup
32 3 M 96 MB 1818 4246 3.04 0.43x
1024 100 k 102 MB 260 4753 1.72 0.05x
16384 10 k 164 MB 343 18638 2.66 0.02x
262144 1 k 262 MB 508 454171 39.07 0.00x
The 454 SECONDS at 262 MB is portal hex-parsing, NOT the kernel
(which is 39 ms). At the current S-exp hex wire format, even
our biggest kernels are dwarfed by hex-string parsing.
2. Daemon mode lands in shake256-fanout.cu. Touch CUDA context
once at startup, then accept commands on stdin:
process <in.portal> <out.portal> → fan-out + write result
quit → clean shutdown
bench_daemon.py measures 574x speedup per call:
workload: 10 calls × 100 inputs × 32 bytes each
host hashlib loop : 0.6 ms total ( 0.06 ms/call)
per-spawn fanout : 1825.9 ms total (182.59 ms/call)
daemon-mode init : 109.2 ms (one-time)
daemon-mode calls : 3.2 ms total ( 0.32 ms/call)
Daemon is the production architecture for any workload doing
repeated fan-outs. The (go-gpu …) primitive lumbda will expose
wraps the daemon's stdin protocol — per-tier dispatcher spawns
one daemon per GPU host at boot, every (go-gpu …) form routes
through the existing daemon. CUDA init never re-runs while
lumbda is up.
3. DESIGN-go-gpu.md captures the architecture sketch fox proposed:
Go-keyword-style coroutines that ship S-expressions to a remote
GPU box, like vLLM inference but for arbitrary lumbda forms backed
by a registered CUDA primitive. Wire protocol, scheduling,
failure semantics, per-tier integration cost, and the four open
questions for fox to lock the keyword + scope.
README.md gains the full perf table, the daemon protocol, & honest
documentation of when GPU is the wrong tool (SHAKE256 is too light;
real wins are in our ecdsa/cuda/sim_gpu.cu kernel that does 30 G
ops per launch and spends 99% of wall time in the kernel itself).
Establishes the integration pattern for lumbda's future cuda primitive
across Python / C / asm tiers without dragging the CUDA toolchain into
lumbda's core build.
Shape: leaf binary that every tier spawns via its existing process-
spawn primitive & talks to through S-expression input + output
portals. Asm tier inherits via fork + execve syscalls; no libcudart
linkage; no DKMS dependency at lumbda build time.
Files:
shake256-fanout.cu self-contained CUDA SHAKE256 fan-out, Keccak
permutation derived from FIPS 202 reference
(tiny-sha3 lineage, CC0 → re-licensed AGPLv3)
Makefile nvcc build + make test + make bench
test_roundtrip.py validates output byte-identical to
hashlib.shake_256
bench.py device vs host throughput at N = 1k / 10k / 100k
lumbda-call.lsp reference Scheme wrapper showing the
(cuda-shake-fanout inputs out-bytes) API shape
lumbda's core would dispatch to per-tier
README.md full integration story, wire contract, the
three changes each tier needs (~20 LoC each),
generalization path for other CUDA primitives
Tested on 3090-ai (RTX 3090):
make test → PASS — 4 / 4 hashes byte-identical to hashlib.shake_256
Honest bench (32-byte inputs):
N host (Python hashlib) device (kernel launch dominated)
1,000 0.6 ms 188.1 ms
10,000 5.9 ms 195.7 ms
100,000 58.7 ms 313.8 ms
Useful primitive when inputs are larger (KB+) or N reaches millions;
honest about the launch-overhead break-even point. This is the
reference, not the win — the win is locking the API shape so each
tier registers under one stable name.
Provenance: extracted as the generic pattern from
~/git/www.foxhop.net/ecdsa/cuda/sim_gpu.cu where on-device SHAKE
delivered 2.6× memory compression for batched reversible-circuit
simulation. Re-shipping the primitive back to the lumbda repo so the
ecosystem inherits the work.
ecdsa cross-tier validation on lumbda-gc and bump-only lumbda blocked
on `case` being unbound. R7RS standard control form — present on
Python (lumbda.py) and C (c/) tiers, but on asm reachable only via
cl_full_prelude's define-macro form (carved into lumbda-full only).
Mirrors commit 865be28 (when/unless via Path B dispatch table).
Path B (special-form dispatch table extension):
- sf_case length-prefixed symbol name
- sym_case_val interned at init_special_forms
- dispatch case in .eval_top alongside .ev_when / .ev_unless
(placed before the .ifdef CL_FULL macro-lookup block so the
dispatch shadow takes precedence over the cl_full_prelude macro
on lumbda-full — no conflict, the macro just becomes dead code)
- .ev_case evaluator: eval key once, push on stack, walk clauses;
each clause's datum list compared by pointer equality (eqv? on
the asm tier — fixnums, symbols, booleans, characters, nil are
all interned/unboxed to unique values). `else` matches uncondi-
tionally. Match → .ev_begin (TCO). No match → .ev_begin_void.
Available on every asm tier (plain `lumbda`, `lumbda-gc`,
`lumbda-full`). Binary size impact:
lumbda 60768 → 61040 (+272, +0.45%)
lumbda-gc 69496 → 69768 (+272, +0.39%)
lumbda-full 72000 → 72264 (+264, +0.37%)
All 158 asm tests still pass. Sanity tests: single-datum, multi-datum,
symbol key (eqv?), else, empty body, no-match, nested case — all
correct on all three tiers.
ecdsa search.lsp now produces byte-identical winner (v3-clifford-only
score 0) across five tiers: Python, C, asm-bump, asm-gc, asm-full.
ecdsa cross-tier validation blocked on `when` and `unless` being
unbound on the asm tier. R7RS standard control forms — present on
Python (lumbda.py) and C (c/) tiers, but absent on asm because the
existing macro facility (define-macro) only ships under CL_FULL.
Path B chosen (special-form dispatch table extension):
- sf_when / sf_unless length-prefixed symbol names
- sym_when_val / sym_unless_val interned at init_special_forms
- dispatch cases in .eval_top alongside .ev_and / .ev_or
- .ev_when / .ev_unless evaluators reuse .ev_begin for the body
branch and .ev_begin_void for the skip branch (TCO preserved)
Available on every asm tier (plain `lumbda`, `lumbda-gc`,
`lumbda-full`). Binary size impact:
lumbda 60488 → 60768 (+280, +0.46%)
lumbda-gc 69224 → 69496 (+272, +0.39%)
lumbda-full 71720 → 72000 (+280, +0.39%)
All 158 asm tests still pass. Tested truthy/falsy/multi-form bodies
on all three tiers. ecdsa search.lsp now runs on asm-full with output
byte-identical to Python tier (v3-clifford-only winner, score 0).
Note: ecdsa search.lsp also depends on `case`, which is only present
under CL_FULL (carved into cl_full_prelude as a define-macro form).
That gap blocks lumbda-gc cross-tier validation and is out of scope
for this commit.
Two coupled defects surfaced during ecdsa cross-tier validation against
the asm tier.
Defect #28 — _start ignored argv. Invoking `asm/lumbda-gc file.lsp`
silently discarded argv[1] and dropped into a REPL that blocked on a
pty when run under SSH. Walk argc/argv after init_builtins + prelude
load and before repl_top: for each argv[i] starting at i=1, skip
arg if it begins with '-' (flag stub), otherwise allocate a Scheme
string from the C string, wrap in a 1-element arg list, dispatch
through apply_proc_raw on the BI_LOAD builtin. If any non-flag arg
ran, jump to repl_exit instead of entering the REPL. Mirrors the
c/main.c script-mode semantics. The RET_VAL macro on the builtin
return path pops r12/rbp/rbx in an order that corrupts %rbp (it
restores the pre-call %r12 into rbp), so the loop counter saves
%rbp around the apply_proc_raw call.
Defect #30 — eq? was only present under CL_FULL. The plain `lumbda`
and `lumbda-gc` binaries shipped without the alias `(define eq? eqv?)`,
so any .lsp expecting eq? (every cross-tier file we own) hit
"unbound variable: eq?" the moment it tried a status check. Lift
that single alias into a new always-on `default_prelude` block with
its own `load_default_prelude` loader (modelled after
load_cl_full_prelude), and call it unconditionally from _start
between rng_seed and the CL_FULL block.
Verification:
- `make asm-build` clean
- `make asm-test`: 158 passed, 0 failed (full suite green)
- `(eq? 1 1)` -> #t on all three tiers via stdin pipe AND file arg
- `~/git/lumbda/asm/lumbda-gc /tmp/asm-test.lsp` exits 0 with #t printed
Two defects uncovered during secp256k1 oracle port; both forced workarounds
downstream in ecdsa/lumbda/secp256k1.lsp that can now retire.
(1) `quotient` used `int(a / b)` — Python float division. Past 2^53 the
float lost precision, so mod-pow on secp256k1's 2^256 prime corrupted
every modular inverse with off-by-one errors in the square-and-multiply
loop. Replaced with R7RS-spec truncate-toward-zero integer division.
Verified: (quotient 7 2)=3, (quotient -7 2)=-3, (quotient 7 -2)=-3,
(quotient -7 -2)=3, (quotient 0 5)=0, (quotient 2305843009213693950 2)
now returns 1152921504606846975 (was 1152921504606846976, off by 1).
(2) Reader `_atom` looped `for conv in (int, float)` and Python's float()
accepts bare 'inf', 'infinity', 'nan' as IEEE specials. So `'(infinity)`
silently parsed as `(+inf.0)` and `(symbol? 'infinity)` returned #f.
R7RS spells these +inf.0 / -inf.0 / +nan.0 explicitly. Restricted the
float-parse path to a strict decimal/exponent regex; the named IEEE
specials still match their proper spellings.
Verified: (symbol? 'infinity)=#t (was #f), (symbol? 'inf)=#t,
'(infinity) reads as the symbol list, (positive? +inf.0)=#t still works.
Tests: 571 Python unit + 205 functional (Python + C) all pass.
Bash process substitution <(...) passes /proc/self/fd/N — a pipe, not
a regular file. load_file used fseek(SEEK_END)+ftell to size a single-
read buffer; on a pipe ftell returns -1, which casts to SIZE_MAX as
fread's nbyte argument and blows the heap. Glibc fortify caught it
as '*** buffer overflow detected ***'.
Detect non-seekable input via the fseek return code and fall back to
a doubling growable buffer instead. Seekable path unchanged.
Repro: ~/git/lumbda/c/lumbda <(echo '(display 1)(newline)')
Resolved by 45a90b8 (vm restores cur_code across CALL/RETURN).
The reproducer at lines 5-11 of the TODO file runs cleanly now
under both `lumbda --fast` and `lumbda -j --fast`, and the
proof/eml_proof_in_lumbda.lsp `normalize` function no longer
needs the `(define (iter ...))` workaround that the TODO
documented. Future regressions are caught by
tests/regression-named-let-leak.{lsp,sh}.
Five-line fix that ends the F1 hang in foxhop.net
ecdsa/tests/unit/probe-c-confirm.lsp.
Symptom — under --fast, a defined function whose body is a
tail-recursive named-let that calls another user-defined function
per iteration loops forever at 100 percent CPU. Trace pins the
bytecode dispatch:
walk1.body: PUSHE MKCLO DUP BIND LOOKUP TCALL ->loop
loop: LOOKUP NULL? JIF LOOKUP CALL ->always-true
always-true: CONST RET (returns #t)
always-true (!): JIF LOOKUP CDR STAIL -> ip=0 of always-true (!)
loop forever
cur_code was the call-frame-local register holding the currently
executing CodeObj. OP_CALL updated it on entry but neither OP_RETURN
nor the builtin-fallback restore path in OP_TAIL_CALL put it back
on return. Subsequent OP_SELF_TAIL_CALL read cur_code->self_params
from the still-stale callee proc (NULL for always-true since it has
no named-let), guard skipped the env rebind, then set ip=0 — without
ever updating the loop variable. Loop variable stayed pinned at the
initial list and our walk never reached its base case.
Fix — VMFrame gains a cur_code field. Three frame-push sites save
it on entry (OP_CALL, OP_TAIL_CALL builtin fallback frame-restore,
OP_CALL_CC compiled-proc entry); two frame-pop sites restore it on
return (OP_RETURN, OP_TAIL_CALL builtin fallback).
Verified inside foxhop.net's ecdsa QEMU guest:
- foxhop.net/ecdsa/tests/unit/probe-c-confirm.lsp F1..F4 — all pass
- foxhop.net/ecdsa/tests/unit/test-sim.lsp under --fast — 21/21 pass
- foxhop.net/ecdsa/lumbda/main.lsp under --fast — 6 shots, score 18,
byte-identical with our Python tier
- make functional-test — 205/205 on Python and C tiers
- make regression-named-let-leak — 4/4 across Python, tree-walker,
and --fast JIT
c/TODO-named-let-bytecode.md can stop applying its `(define (iter ...))`
workaround once this lands.
Three coupled changes that unblock the ecdsa quantum-circuit
simulator's run on the C tier from neoblanka.
1. c/Makefile autodetects libgc-dev — if /usr/include/gc.h is
present, the build links Boehm and defines USE_BOEHM_GC. Without
GC, ul_free is a no-op (lumbda.h:35) and every allocation leaks;
small REPL snippets work but workloads with thousands of envs
OOM the process. Override with USE_GC=0 to force the malloc-only
path for diagnostics.
2. c/main.c calls GC_INIT before init_symbols, then GC_disable.
GC_INIT registers the stack base for conservative scan — without
it some Linux configs miss roots. GC_disable is a deliberate
stopgap: lumbda Values are NaN-boxed pointers that conservative
Boehm cannot recognize as pointers, so live targets get reclaimed
(env binding symbol payloads, SymbolEntry strings) and lookups
fail with "undefined: <sym>". Reproducing this without GC_disable
on the GC build: any sim.lsp call chain triggers the corruption
after ~100 named-let iterations. Until tracing is precise,
growing the heap is safer than wrong results. Long-running
workloads run under ulimit -v.
3. c/builtins.c gains rename-file and delete-file matching the
Python tier (lumbda.py:3468). sim.lsp's write-portal! pattern
(write to .tmp, rename) needs rename-file to land cross-tier
identical results.
4. tests/regression-named-let-leak.lsp + .sh pin four shapes that
blew up ecdsa: the c/TODO-named-let-bytecode.md repro, the F1
shape from foxhop.net's lumbda-c-tier-leak-SP.md (12-line
minimum), a 200-iter scaled variant, and a sim.lsp run-ops!
mirror. Wired into root Makefile as regression-named-let-leak;
added to test-all. Wrapper caps memory at 256 MB virt and 15s
per tier so a leak regression fails the run instead of consuming
host RAM.
Known limits:
- --fast JIT still has the named-let + inner user-fn call hang
(separate TODO; tree-walker handles this fine).
- GC_disable means the heap grows; workloads must bound their work
budget. ecdsa's sim runs comfortably in 5 MB.
Verified inside a 2G/2vCPU QEMU guest (foxhop.net ecdsa/vm-runner.sh):
- test-c (tree-walker) — 35/35 PASS
- bench-c (tree-walker) — score 18 matches Python tier byte-identical
- F1 probe (tree-walker) — all four steps PASS
§3 (tier description) expanded — the asm tier now ships three build
flavors (lumbda / lumbda-gc / lumbda-full) each an additive superset
of the previous, each guarded by assemble-time flags so the minimal
tier keeps its 22 KB shape. Lists the new CL_FULL features explicitly:
auto-loaded Scheme prelude, #(…) vector literals, `/,/,@ quasiquote,
define-macro + macro table (GC-rooted), and the seven new builtins
(gensym, exit, values, call-with-values, cadr, sort, let*).
§9.2 (CL compatibility) updated to reflect the work this session:
* Portal determinism paragraph now names asm/lumbda-full alongside
Python and C as a CL-compat tier.
* Replaced the single-defect paragraph with a five-defect sweep.
Four were asm-side bugs the prior 158-assertion suite never
exercised: bi_apply clobber, bi_expt infinite loop on negative
exponent, macro_env_head missing from GC roots, prelude missing
cadar. The fifth — C env_lookup's global shortcut — remains as
before. All five landed as separate commits in ticket 0005's
follow-up drop.
* Tests paragraph now names three test files (cl-compat.lsp at 45
assertions including multiple-value-bind, ursa.lsp at 28, ursa-
scheme.lsp at 15) and both test runners — make zoe-favorites-test
for the offline suite and make prove-ursa-runs for the network-
live proof that fetches wedgewack.org/ursa.lisp.txt fresh every
run and spot-checks Zoë's defuns on asm/lumbda-full.
PDF and HTML regenerated.
