The previous Env.lookup had a shortcut that checked self.g (global)
right after self.b (local). This skipped any intermediate parent
frame that shadowed a global name. The concrete bug was a let-loop
parameter named `count` (also a SRFI-1 builtin): when an inner
(let ((next ...))) pushed a new frame between the loop body and
the loop binding, `count` was not in self.b, the shortcut found
the global builtin, and returned it — instead of walking up one
more parent to the loop's parameter frame.
Fix: remove the shortcut, walk self → self.p → ... → global in
order. O(chain depth) instead of O(1) for the common case, but
correct. Chain depths are small in practice.
The inline cache (bytecode VM's OP_LOOKUP) had the mirror issue —
it verified only `arg not in env.b`, missing parent shadows.
Updated to walk the chain from env up to the cached env (always
global) and check each intermediate frame before returning the
cached value. The cache still reads the value fresh from the
cached env's bindings dict so `set!` on a global is observed
immediately (previously a cached value would go stale on set!
even though the test suite's test_compile_mutual_recursion
depended on this behavior).
Cache is now populated only when the lookup resolved identity-
equal to the global's current binding — i.e. no intermediate
shadow — using `val is g.b[arg]` as the guard.
Regression: all 975 tests still green (571 py + 132 asm + 189
shared + 83 c), including test_compile_mutual_recursion that
exercises set!-after-compile.
Discovered while debugging portal-http-client.lsp, where the
portal body's (define counter ...) form landed correctly but a
nearby let-loop accumulator named `count` resolved to the Python
builtin `count` (SRFI-1 count procedure). The server itself, and
asm and C clients, were unaffected — asm's env lookup walks the
chain, and C's env lookup has no equivalent shortcut.
Closes the last loop promised in the whitepaper's Future Work: a
node serves its state as an S-expression portal over HTTP, another
node pulls it down with tcp-connect + tcp-recv and materializes the
bindings locally via (eval (read-from-string line)).
examples/portal-http-server.lsp (90 lines):
- Holds some state (counter, my-int, my-list, my-fib, my-str)
- GET /portal → S-expression body: a sequence of (define ...) forms
- GET / → HTML index
- Uses heap-snapshot / heap-restore for O(1) memory on asm
examples/portal-http-client.lsp (90 lines):
- tcp-connect, send HTTP/1.0 GET, receive full response
- Strip headers (walk to first \r\n\r\n)
- Split body by \n, eval each non-empty, non-comment line
- The remote bindings are now live locally
3×3 server/client matrix: all 9 combinations green. Every runtime
hosts, every runtime consumes. The wire format is Scheme source;
no schema, no JSON, no Protobuf.
Prerequisite fix: `eval` semantics aligned across all three impls.
Python and C's `eval` special form previously evaluated its result
in the CALLER's env, so a nested (eval (read-from-string
"(define x 42)")) would install x in the local function scope —
invisible to later top-level code. asm's bi_eval always used the
global env (r14). With this commit, all three impls evaluate the
eval'd result in the global env, matching asm's existing behavior.
Python: uncommonlisp.py leval eval-handler now does `env = env.g`
before continuing the trampoline.
C: c/eval.c SYM_EVAL branch now does `env = env->global`.
asm: no change (already correct).
One pre-existing Python defect surfaced by the client:
`count` is a SRFI-1-style builtin (`d(S('count'), ...)`), so a
local let-loop variable named `count` collides with it in the
inline-cache lookup path and OP_LOOK_ADD1 fires on the builtin
instead of the local. Worked around by renaming the loop
accumulator to `cnt`. Underlying Env.lookup shortcut-to-global
issue is out of scope for this commit.
Regression: 975 tests still green.
Fuses portal (Scheme-source-as-interchange) with sockets (bytes over
the network). Wire protocol: one S-expression per connection. Same
server + client .lsp runs byte-identically in Python, C, and asm.
New primitives in all three impls:
- read-from-string — parse one sexp from a string
Asm gets two more:
- symbol->string — standard R7RS, was missing
- eval — evaluate a Scheme value in the global env (Python + C had
it as a special form; asm exposes it as a builtin)
examples/rpc-server.lsp (port 9080):
- Whitelisted dispatch: ping / add / mul / fib / echo
- Never calls eval on client input; safe by construction
- Uses heap-snapshot/restore for O(1) memory on asm
- ~90 lines, portable
examples/rpc-client.lsp:
- Sends one request, reads one response, displays both
- 45 lines, portable
examples/repl-server.lsp (port 9081):
- DANGER: full remote eval. Any Scheme form accepted and evaluated
in the server's global env. Persistent across connections.
- Deliberately does NOT use heap-snapshot — remote (define x ...)
lives in the global env above any snapshot point; rewinding would
invalidate the new binding. The ulimit -v 512 MB safety net
(documented in CLAUDE.md) ensures an escaped process can't crash
the machine.
- ~70 lines, portable. Demonstrates what "the language IS the
interchange format" gets you at the limit: a single socket and
a single primitive (eval) carry a full-powered REPL.
Verified 3×3 server×client matrix: all 9 combinations green.
All 132 asm + 571 py + 189 shared + 83 c tests still pass.
One quirk discovered and worked around: in asm, a closure captures
its env chain by pointer at define time. Forward-referenced names
in mutually-recursive toplevel defines can fail under specific
heap-restore patterns — see the leaf-first ordering note in
rpc-server.lsp.
Three wins in one commit.
1) heap-snapshot / heap-restore (asm arena primitive)
asm has no GC. Long-running servers leaked ~64 MB per heap growth.
Two new builtins let a programmer capture r15 and later rewind to
it, recycling intermediate allocations in O(1) memory.
Python + C get no-op versions so portable .lsp code can call them
unconditionally.
examples/http-server.lsp now takes a snapshot at top level and
rewinds after every request. Measured asm RSS: 88 KB initial,
100 KB after 100 requests, 100 KB after 1100 requests — flat.
Prior behavior was +64 MB per few thousand requests.
2) examples/http-client-bench.lsp — native HTTP load generator
Uses only the six tcp-* primitives + current-time-ms. Runs
identically in all three impls. Eliminates curl's ~2 ms/req
fork+exec overhead, so real server throughput shows up:
Python server ← Python client 2403 rps
C server ← C client 2439 rps
asm server ← asm client 2994 rps
asm server ← C client 2500 rps
The earlier curl-based bench was clamped near 400 rps by the
client; the actual servers handle 6–7× that.
3) MOAD-0001 cleanup
- c/builtins.c bi_string_replace: strncmp-at-every-position
(hand-rolled, sedimentary) → strstr (libc-tuned, typically
Boyer-Moore-Horspool). O(N*k) → O(N + matches*k).
- uncommonlisp.py _tokenize_lines: per-token src.count('\n', 0, pos)
→ precompute line_starts once, bisect_right per token.
O(N*M) → O(M + N log M).
Also adds current-time-ms to all three impls so benchmarks can
time themselves without relying on the Python/C float `current-time`
(asm has no floats). Seconds-since-epoch tagged as a 61-bit int.
Test counts unchanged: 571 py + 132 asm + 189 shared + 83 c = 975.
All green via make test-all.
Added tcp-listen/accept/connect/recv/send/close to Python, C, and asm.
One examples/http-server.lsp runs identically in all three impls and
serves HTTP/1.0 with routing, content-type, and content-length headers.
asm additions:
- SYS_SOCKET/BIND/LISTEN/ACCEPT/CONNECT/SETSOCKOPT syscalls
- 6 tcp-* builtins using the existing port encoding (SPECIAL ≥ 1000)
- bi_tcp_connect: dotted-quad IPv4 parser, no DNS dependency
Defects fixed along the way (surfaced by the HTTP server):
- string-append: was 2-arg only; now variadic (walks arg list twice)
- number->string: was stubbed to VAL_VOID; now correctly writes digits
into a heap-allocated string (incl. negative handling)
- String-literal reader: \r and \0 escape sequences now handled (was
silently dropping backslash, treating them as literal 'r' / '0')
- tcp_accept: sockaddr buffer was 8 bytes, now 16 (was corrupting
caller's stack when accept wrote full struct sockaddr_in)
Pinocchio benchmark (tests/web-benchmark.sh):
At concurrency=20, 1000 requests, serving a 1KB body:
uncommonlisp Python 373 req/s
uncommonlisp C 370 req/s
uncommonlisp asm 370 req/s
python3 http.server 381 req/s (stdlib reference)
busybox httpd 382 req/s (production reference)
All five converge within 3% — the client (curl fork/exec) is the
bottleneck, not the server. Our single-threaded blocking servers
are indistinguishable from battle-tested ones at this load.
Binary sizes:
uncommonlisp asm 45 KB (HTTP + everything else)
busybox httpd 2.1 MB (multi-call binary)
python3 8 MB (interpreter)
The asm HTTP server is 46× smaller than busybox and 176× smaller
than Python, serves from 7 Linux syscalls, and the entire protocol
handler is 70 lines of portable Scheme.
Test counts: 132 asm (up 1), rest unchanged. All green.
Benchmark exercises full save×load matrix across Python/C/asm plus the
mismatch cases (wrong format, truncated input, missing file, corrupt
header). Cases that used to segfault or report wrong paths now degrade
cleanly.
asm (uncommonlisp.s):
- (define var) with no value now binds to VOID instead of segfault
- portal-resume checks sys_read returned full 48-byte header; sanity-
checks heap_size and heap_base before committing r14/r15 restore.
Corrupt/truncated portals now return #f cleanly.
py (uncommonlisp.py):
- file-not-found error inside a nested (load) now reports the actual
missing path (via FileNotFoundError.filename) rather than the outer
script path.
- _load wraps UnicodeDecodeError (binary file loaded as text) into a
LispErr with the path; no more raw Python traceback.
tests/portal-benchmark.sh: 50-iter benchmark, 4 parts
(save / load / cross-process / mismatch-classification).
Representative numbers (this laptop, 2026-04-16):
setup+save: Python 126ms, C 3ms, asm 0.9ms
cross-proc: Py→Py 260ms, C→C 6ms, asm→asm 1.5ms
All three test suites still pass: 571 py unit, 131 asm, 189 shared.
JIT now matches CPython speed:
fib-rec(20): JIT 14ms vs VM 1245ms (87x faster)
ack(3,4): JIT 13ms vs VM 1053ms (82x faster)
Both at parity with hand-written CPython.
Fused opcodes: LOOK_ADD1 (lookup + increment) and LOOK_SUB1
(lookup + decrement) emitted directly by compiler for (+ sym 1)
and (- sym 1) patterns. Eliminates one dispatch per loop iteration.
sum-to(50000) ratio improved from 59x to 45x vs Python.
ackermann(3,4) steady at 83x. 571 tests green.
Also defines LOOK_LOOK, CONST_EQ_JF, LOOK_CONST_CALL2
superinstruction opcodes (VM handlers ready, compiler emission
for remaining patterns deferred to next pass).
Portal saves the full machine state — env chain, compiled procedures,
continuations, frame stack — to a JSON file. Another interpreter
instance loads it and resumes execution from the exact instruction.
Demo: start a primality test on machine A, checkpoint mid-computation,
resume on machine B. 1000000007 prime check: machine B picks up from
i=30000 and finishes in 6% of the original time.
Implementation:
- PortalSerializer: graph-aware with identity tracking for shared env
references. Handles cycles (closures referencing their own env).
- portal-checkpoint!: triggers mid-execution save from within VM loop.
Hooks into TAIL_CALL (loop back-edge) for compiled code.
- --portal-resume CLI flag: load .portal file and resume continuation.
- portal-save / portal-resume Scheme builtins.
571 tests green (7 new portal tests: unit + integration + functional).
Source maps: tokenizer tracks line numbers, parser attaches to Pair nodes,
compiler records in CodeObj.source_map, errors display source line.
Multi-shot continuations: FullCont snapshots env at capture time via
deep copy. Each invocation gets a fresh env copy. Generators and
multi-shot patterns both work correctly.
Inline caching: VM caches global env lookups per instruction site.
Local shadow check (arg not in env.b) prevents stale cache hits.
Compiler tracks compile-time scopes to suppress specialization/folding
when builtins are locally shadowed.
Bytecode serialization: JSON-based .lspc format. save-compiled/load-compiled
builtins. --compile CLI flag precompiles .lsp files. Round-trip tested for
all operand types including nested closures and quoted data.
564 tests green (35 new: unit + integration + functional for each feature).
- Full call/cc: upward continuations via explicit VM frame stack + trampoline.
Generators now work: (make-gen (lambda (yield) (yield 1) (yield 2)))
- Explicit frame stack: compiled→compiled CALL no longer grows Python stack.
Non-tail calls use frames list instead of recursive vm_exec.
- Env global shortcut: lookup checks local then global before walking chain,
O(1) for builtins instead of O(depth).
- Constant folding: arithmetic on literals folded at compile time.
- Auto-compile covers inline lambdas (not just define).
- 522 tests green, 7-19x speedup over interpreter.
- auto-compile! parameter: compile defines on the fly for zero-effort speedup
- VM optimization: local aliases, inlined truthiness checks (~15% faster)
- disassemble builtin: human-readable bytecode listing
- call/cc compiled natively in VM (escape-only, no longer falls back to eval)
- Makefile: add lint, bench-verbose, clean targets
- 518 tests green
Stack-based VM with 17 opcodes, TCO via in-place frame reset,
compile-time macro expansion, and seamless interop between
compiled and interpreted code. 514 tests green.
Core additions:
- StringInputPort / StringOutputPort — open-input-string, open-output-string,
get-output-string, read/read-char/peek-char/read-line on ports
- R7RS ErrorObject class — error-object?, error-object-message,
error-object-irritants; guard and with-exception-handler now receive
ErrorObject instances instead of raw strings
- Exact rational arithmetic via Fraction — (/ 1 3) => 1/3, literal 1/3
syntax, numerator/denominator, exact/inexact conversions
- Module system — (module name (export ...) body...) and (import name) /
(import (name sym ...)) for selective import
- define-record-type now a Python special form supporting (inherit parent)
for single-inheritance with subtype predicates
- Pretty-printer — pp/pretty-print with configurable line width
Correctness:
- R7RS letrec* body semantics for internal defines — names pre-declared
before any initializer runs, enabling mutual recursion in let bodies
- string->number detects #x/#b/#o/#d and 0x/0b/0o prefixes
- show() guarantees decimal point in float output
- number->string rounds through show() for consistent float representation
Performance:
- _body_env fast path: skip scan when first body form is not define/begin
- Proc.has_defs flag: skip _body_env entirely for procs without internal defines
- Single-form body optimization: skip begin wrapper, set expr directly
Tests: 396 → 455 (+59 new tests covering all new features)
