Adds a second asm build (asm/uncommonlisp-gc) behind the GC_NAIVE
assembler flag, providing the benchmark baseline we previously had
no data for. Same binary, same surface, different allocator:
- 8-byte header per heap block (size << 1 | mark), placed at -8
from the tagged pointer so existing untag + offset accesses
stay unchanged.
- Chunk list tracked in a side array, letting sweep walk every
mmap'd region by header-chained blocks instead of guessing.
- Free list rebuilt each sweep, first-fit alloc with split on
large-leftover (>= 24 bytes).
- Mark phase enumerates five root classes: %r14 (global env,
untagged chain), sym_else_val, sym_table entries, every
sym_hash_bucket chain, and a conservative scan from current
%rsp to the initial stack_top captured at _start. The stack
scan runs twice per word — once as a tagged value, once as a
potential untagged env-node pointer (size-guarded to 24 bytes
so it can't walk off a wrong-size block).
- Transitive marking via an explicit 16K-entry mark stack;
gc_mark_env walks untagged env chains from %r14 and from every
closure's env field.
- heap_alloc preserves the non-GC ABI (only %rax clobbered) so
existing callers like bi_append, which holds state in %rcx
across make_pair, keep working.
- Overflow path uses check-then-write bumps and pads the old
chunk's tail with a single dead block before growing, so sweep
never walks into uninitialized mmap'd memory.
- HEAP_SIZE shrinks to 1 MB under GC_NAIVE so the collector
actually runs on ordinary workloads.
- Two diagnostic builtins in the GC build: (gc-collect) to force
a collection, (gc-stats) -> (collections . live-bytes).
Control-group bench (examples/bench-gc-memory.lsp, 2000 iterations
of build-sum-discard over 200-element lists, i5-8350U):
tier time_ms peak_rss final_rss
asm no-GC 1097 133.9 MB 133.9 MB (grows, never shrinks)
asm naive GC 1431 1.1 MB 1.1 MB (steady state)
124x less memory at a ~30% throughput cost. That is the number we
were guessing at before. Reproduce: make bench-gc.
Tests: 137 asm (no-GC) + 137 asm (GC) + 189 shared functional pass.
The two asm builds are tested independently via UNCOMMONLISP_BIN in
asm/test.sh; asm/Makefile now builds both and exposes a test-gc
target.
Adds 6 hash-set builtins (make-hash-set, hash-set?, hash-set-add!,
hash-set-contains?, hash-set-size, hash-set->list). Same sentinel
scheme as hash-table but tag word = -2 (hash-table is -1, vector
is >= 0). One cons cell per entry (vs two for hash-table) since
a set stores keys only — that's where the speedup over the Scheme-
level vector-based ht-* lib comes from.
Benchmark (tests/bench-hashset.sh, via make bench-hashset),
N=5000, i5-8350U asm tier:
portable native speedup
insert ~130 ms ~7 ms ~20x
hit-lookup ~125 ms ~8 ms ~15x
miss-lookup ~240 ms ~12 ms ~20x
Portable is the ht-* lib from proof-netspace-server-lib.lsp
(vectors + cons chains + modulo, pure Scheme). Native replaces
the Scheme-level bucket walk with an asm loop that dereferences
pairs directly — no env lookups, no frame building per iteration.
All 137 asm + 189 functional (Python + C) tests still green.
Extracts the 300-line server body into proof-netspace-server-lib.lsp
so multi-node demos can share it without duplication. The existing
proof-netspace-server.lsp entry point stays stable — now a 25-line
config wrapper that sets defaults and loads the lib.
New 2-node scaffolding:
proof-netspace-node-a.lsp — port 9086, cache /tmp/lumbda-A-*
proof-netspace-node-b.lsp — port 9087, cache /tmp/lumbda-B-*
spiral-client.lsp — drives both nodes, seeds them with
partially-overlapping theorem sets,
runs one A→B and one B→A envelope
round-trip, reports sizes
spiral-demo.sh — orchestrator: starts both nodes,
runs client, tears down cleanly.
Accepts python|c|asm — all three
converge identically (A=3 B=3 → A=5 B=5).
Proves the envelope primitive at use-case scale: N independent caches
mesh-converge in O(N) spiral passes. Foundation for the "looping and
spiraling across time and space of manifolds" runtime topology.
Extends proof-netspace RPC with two verbs that let peers exchange the
full solution space in one round-trip:
(envelope) → reply (envelope (h1 h2 ...))
(merge (h1 h2 ...)) → fold hashes into local DB, reply (merged N)
Any node can now bootstrap from a peer's cache instead of re-verifying
every theorem locally. Two nodes that swap envelopes both become
supersets of what either knew — the primitive for mesh-wide spiral.
*proof-db* swapped from linear alist to a hash-set. O(N·M) merge drops
to O(M). The hash-table is a ~20-line pure-Lumbda library over
make-vector / vector-ref / vector-set! — runs unmodified in all three
tiers. No asm hash-table primitive needed.
Also fixes a pre-existing asm defect: bi_makevec clobbered %rax via
the GETARG macro's internal scratch use, causing SIGSEGV on every
(make-vector N fill) call. The bug shipped because asm/test.sh only
covered the variadic (vector ...) constructor; tests/functional.lsp
had one make-vector assert but was never wired into asm's harness.
Added five make-vector assertions to asm/test.sh (132 → 137).
Portal snapshot rewritten to emit (set! *proof-db* ...) so the
top-level binding is actually mutated on restart — previous
(define ...) form bound locally on some code paths, leaving the
in-memory DB empty after load.
Verified: make test-all green (137 asm + 189 functional + Python/C
tests), 3-tier matrix cold+warm+restart all clean.
Closes the last loop promised in the whitepaper's Future Work: a
node serves its state as an S-expression portal over HTTP, another
node pulls it down with tcp-connect + tcp-recv and materializes the
bindings locally via (eval (read-from-string line)).
examples/portal-http-server.lsp (90 lines):
- Holds some state (counter, my-int, my-list, my-fib, my-str)
- GET /portal → S-expression body: a sequence of (define ...) forms
- GET / → HTML index
- Uses heap-snapshot / heap-restore for O(1) memory on asm
examples/portal-http-client.lsp (90 lines):
- tcp-connect, send HTTP/1.0 GET, receive full response
- Strip headers (walk to first \r\n\r\n)
- Split body by \n, eval each non-empty, non-comment line
- The remote bindings are now live locally
3×3 server/client matrix: all 9 combinations green. Every runtime
hosts, every runtime consumes. The wire format is Scheme source;
no schema, no JSON, no Protobuf.
Prerequisite fix: `eval` semantics aligned across all three impls.
Python and C's `eval` special form previously evaluated its result
in the CALLER's env, so a nested (eval (read-from-string
"(define x 42)")) would install x in the local function scope —
invisible to later top-level code. asm's bi_eval always used the
global env (r14). With this commit, all three impls evaluate the
eval'd result in the global env, matching asm's existing behavior.
Python: uncommonlisp.py leval eval-handler now does `env = env.g`
before continuing the trampoline.
C: c/eval.c SYM_EVAL branch now does `env = env->global`.
asm: no change (already correct).
One pre-existing Python defect surfaced by the client:
`count` is a SRFI-1-style builtin (`d(S('count'), ...)`), so a
local let-loop variable named `count` collides with it in the
inline-cache lookup path and OP_LOOK_ADD1 fires on the builtin
instead of the local. Worked around by renaming the loop
accumulator to `cnt`. Underlying Env.lookup shortcut-to-global
issue is out of scope for this commit.
Regression: 975 tests still green.
Adds a transparent S-expression relay (examples/rpc-relay.lsp) plus a
sequential load generator (examples/rpc-chain-bench.lsp) and a bench
script (tests/rpc-chain-bench.sh) that wires them into multi-hop
chains across runtimes.
The relay is pure byte-forwarding: tcp-accept, tcp-recv, tcp-connect
to backend, tcp-send, tcp-recv reply, tcp-send back. Never parses.
Which is the point — S-expressions are the envelope.
Same rpc-relay.lsp runs as relay in any impl; chains are arbitrary
combinations of {Py, C, asm} nodes.
Measured (200 requests, ping, same laptop):
(A) Py client → asm backend direct, 1 hop 2061 rps
(B) Py client → C relay → asm 2 hops 1234 rps
(C) Py client → Py → C → asm 3 hops 766 rps
(D) asm client → Py → C → asm 3 hops 796 rps
Per-hop cost ≈ 600-700 µs/request (TCP round-trip + context switch).
Safety: every server spawn used the six-layer pattern from CLAUDE.md
(ulimit -v 512MB + timeout 30 + trap + explicit kill + pgrep verify).
Four benchmark cells × up to 3 servers each = 10+ server spawns.
Zero strays, zero safety-net activations.
Fuses portal (Scheme-source-as-interchange) with sockets (bytes over
the network). Wire protocol: one S-expression per connection. Same
server + client .lsp runs byte-identically in Python, C, and asm.
New primitives in all three impls:
- read-from-string — parse one sexp from a string
Asm gets two more:
- symbol->string — standard R7RS, was missing
- eval — evaluate a Scheme value in the global env (Python + C had
it as a special form; asm exposes it as a builtin)
examples/rpc-server.lsp (port 9080):
- Whitelisted dispatch: ping / add / mul / fib / echo
- Never calls eval on client input; safe by construction
- Uses heap-snapshot/restore for O(1) memory on asm
- ~90 lines, portable
examples/rpc-client.lsp:
- Sends one request, reads one response, displays both
- 45 lines, portable
examples/repl-server.lsp (port 9081):
- DANGER: full remote eval. Any Scheme form accepted and evaluated
in the server's global env. Persistent across connections.
- Deliberately does NOT use heap-snapshot — remote (define x ...)
lives in the global env above any snapshot point; rewinding would
invalidate the new binding. The ulimit -v 512 MB safety net
(documented in CLAUDE.md) ensures an escaped process can't crash
the machine.
- ~70 lines, portable. Demonstrates what "the language IS the
interchange format" gets you at the limit: a single socket and
a single primitive (eval) carry a full-powered REPL.
Verified 3×3 server×client matrix: all 9 combinations green.
All 132 asm + 571 py + 189 shared + 83 c tests still pass.
One quirk discovered and worked around: in asm, a closure captures
its env chain by pointer at define time. Forward-referenced names
in mutually-recursive toplevel defines can fail under specific
heap-restore patterns — see the leaf-first ordering note in
rpc-server.lsp.
Three wins in one commit.
1) heap-snapshot / heap-restore (asm arena primitive)
asm has no GC. Long-running servers leaked ~64 MB per heap growth.
Two new builtins let a programmer capture r15 and later rewind to
it, recycling intermediate allocations in O(1) memory.
Python + C get no-op versions so portable .lsp code can call them
unconditionally.
examples/http-server.lsp now takes a snapshot at top level and
rewinds after every request. Measured asm RSS: 88 KB initial,
100 KB after 100 requests, 100 KB after 1100 requests — flat.
Prior behavior was +64 MB per few thousand requests.
2) examples/http-client-bench.lsp — native HTTP load generator
Uses only the six tcp-* primitives + current-time-ms. Runs
identically in all three impls. Eliminates curl's ~2 ms/req
fork+exec overhead, so real server throughput shows up:
Python server ← Python client 2403 rps
C server ← C client 2439 rps
asm server ← asm client 2994 rps
asm server ← C client 2500 rps
The earlier curl-based bench was clamped near 400 rps by the
client; the actual servers handle 6–7× that.
3) MOAD-0001 cleanup
- c/builtins.c bi_string_replace: strncmp-at-every-position
(hand-rolled, sedimentary) → strstr (libc-tuned, typically
Boyer-Moore-Horspool). O(N*k) → O(N + matches*k).
- uncommonlisp.py _tokenize_lines: per-token src.count('\n', 0, pos)
→ precompute line_starts once, bisect_right per token.
O(N*M) → O(M + N log M).
Also adds current-time-ms to all three impls so benchmarks can
time themselves without relying on the Python/C float `current-time`
(asm has no floats). Seconds-since-epoch tagged as a 61-bit int.
Test counts unchanged: 571 py + 132 asm + 189 shared + 83 c = 975.
All green via make test-all.
Prevent recurrence of 2026-04-16 incident where a leaked asm HTTP
server grew to 19.3 GB RSS and crashed the machine.
examples/http-server.lsp:
- Adds *max-requests* = 50000 hard ceiling. Server self-terminates
before unbounded heap growth reaches dangerous levels.
- Loop tracks request count, exits cleanly + closes server socket.
tests/web-benchmark.sh:
- SPAWNED_PIDS array tracks every background process.
- EXIT/INT/TERM trap kills them all (SIGTERM then SIGKILL).
- stop_server does SIGTERM with 500ms grace period then SIGKILL.
- Final straggler check via pgrep narrows to actual HTTP server
processes (not shell/tmux with "uncommonlisp" in the name).
- pkill -9 fallback as belt-and-suspenders.
CLAUDE.md:
- New "Asm memory discipline" section documents the bump allocator
leak behavior and the required operational discipline.
- Test counts updated (571 py + 83 c + 132 asm + 189 shared = 975).
Added tcp-listen/accept/connect/recv/send/close to Python, C, and asm.
One examples/http-server.lsp runs identically in all three impls and
serves HTTP/1.0 with routing, content-type, and content-length headers.
asm additions:
- SYS_SOCKET/BIND/LISTEN/ACCEPT/CONNECT/SETSOCKOPT syscalls
- 6 tcp-* builtins using the existing port encoding (SPECIAL ≥ 1000)
- bi_tcp_connect: dotted-quad IPv4 parser, no DNS dependency
Defects fixed along the way (surfaced by the HTTP server):
- string-append: was 2-arg only; now variadic (walks arg list twice)
- number->string: was stubbed to VAL_VOID; now correctly writes digits
into a heap-allocated string (incl. negative handling)
- String-literal reader: \r and \0 escape sequences now handled (was
silently dropping backslash, treating them as literal 'r' / '0')
- tcp_accept: sockaddr buffer was 8 bytes, now 16 (was corrupting
caller's stack when accept wrote full struct sockaddr_in)
Pinocchio benchmark (tests/web-benchmark.sh):
At concurrency=20, 1000 requests, serving a 1KB body:
uncommonlisp Python 373 req/s
uncommonlisp C 370 req/s
uncommonlisp asm 370 req/s
python3 http.server 381 req/s (stdlib reference)
busybox httpd 382 req/s (production reference)
All five converge within 3% — the client (curl fork/exec) is the
bottleneck, not the server. Our single-threaded blocking servers
are indistinguishable from battle-tested ones at this load.
Binary sizes:
uncommonlisp asm 45 KB (HTTP + everything else)
busybox httpd 2.1 MB (multi-call binary)
python3 8 MB (interpreter)
The asm HTTP server is 46× smaller than busybox and 176× smaller
than Python, serves from 7 Linux syscalls, and the entire protocol
handler is 70 lines of portable Scheme.
Test counts: 132 asm (up 1), rest unchanged. All green.
Portal saves the full machine state — env chain, compiled procedures,
continuations, frame stack — to a JSON file. Another interpreter
instance loads it and resumes execution from the exact instruction.
Demo: start a primality test on machine A, checkpoint mid-computation,
resume on machine B. 1000000007 prime check: machine B picks up from
i=30000 and finishes in 6% of the original time.
Implementation:
- PortalSerializer: graph-aware with identity tracking for shared env
references. Handles cycles (closures referencing their own env).
- portal-checkpoint!: triggers mid-execution save from within VM loop.
Hooks into TAIL_CALL (loop back-edge) for compiled code.
- --portal-resume CLI flag: load .portal file and resume continuation.
- portal-save / portal-resume Scheme builtins.
571 tests green (7 new portal tests: unit + integration + functional).
