asm/lumbda-full now runs Zoë's CL source end-to-end per commit
c6658e0. Rewrite the Known Issues section into a Resolved section
explaining what each of the four underlying asm bugs was (bi_apply
clobber, bi_expt infinite loop on negative exponent, macro_env_head
missing from GC roots, cadar missing from the prelude) and why the
158-test asm suite did not catch them before.
Third asm variant — built with CL_FULL=1 GC_NAIVE=1 via new Makefile
target. Adds the macro machinery needed for cl-compat.lsp on the asm
tier, keeping every addition behind .ifdef CL_FULL so the default
(~22 KB) and -gc binaries keep their current footprint.
Landed in this drop:
* Reader: backtrack on digit-prefixed symbols. After reading digit
characters, if the next char is not a delimiter, input_pos
rewinds and control falls through to .sr_symbol. Makes 1+, 1-,
add1, abc123, and any CL-style identifier with a numeric prefix
parse as symbols instead of truncating to a bare integer.
* Reader: `` ` `` / `,` / `,@` produce (quasiquote X) / (unquote X)
/ (unquote-splicing X) forms. Same build shape as the existing
`'` quote branch.
* Evaluator: .ev_quasiquote + quasiquote_expand walk the template.
unquote evaluates its argument in the current env; unquote-
splicing evaluates then splices via a new list_append_ab helper;
other pairs recurse (cons expand-car expand-cdr). Atoms pass
through. No nested quasiquote depth (deliberate; ticket 0005
scope).
* Evaluator: .ev_define_macro + macro_env_head linked list. Each
(define-macro (name p...) body) prepends a 24-byte
(sym, closure, next) node. Dispatch in eval checks macro_lookup
after all special-form compares; on hit, the closure is applied
to the *unevaluated* argument list and the expansion re-enters
.eval_top under TCO.
* Binding: rest-arg support extended to .apr_bind inside
apply_proc_raw. Previously only .ac_bind (direct .app_closure
path) handled `(lambda (a . b) ...)` correctly; macros call
closures through apply_proc_raw, so this was required to make
variadic defun/setf macros bind correctly.
* Builtin: (gensym) — writes "g%d" for an in-BSS counter, length-
prefixes the buffer, calls intern_static. Available in every
variant (not CL_FULL-gated — useful outside macros too).
* Builtin: (cadr x), (sort lst) and the let* special form from
earlier commit stay in default asm. These are Scheme staples.
* Prelude: evaluated at _start after init_builtins / rng_seed,
before the REPL. Embedded string, input state saved + restored
around the load. Defines caar, cdar, caddr, cadddr, cddr,
cdddr, cddddr, 1+, 1-, add1, sub1, square, eq? (= eqv? for
interned symbols), memq, list-ref, assq, and `case` as a macro.
cl-compat.lsp: two small changes to work under asm's single-list
`map`:
* Added cl-zip helper. Replaced two `(map (lambda (v n) (list v n))
xs ys)` sites with `(cl-zip xs ys)` — asm's builtin map accepts
only one list, and cl-loop-emit needs a parallel walk over
state-vars and new-names.
* Added explanatory comment for cddddr at the top of the shim
(already shipped).
Tests:
* make asm-test (lumbda) — 158/158 pass.
* make asm-test-gc — 158/158 pass.
* make asm-test-full — 158/158 pass on synchronous run.
* Zoë's `examples/ursa.lisp.txt` LOADS on asm/lumbda-full.
`(expt-mod 3 7 100)` = 87.
Most simple cl-loop forms work (while + do + finally, range-to,
then-accumulator).
Known open issues documented in docs/tickets/0005-asm-cl-full.md:
* cl-loop-emit produces wrong output for inputs with `simple` iters
(`(simple a 5)` → state binding dropped). Python/C return the
correct form; asm version is missing the binding. Bug surfaces
in the emit's 30+ binding let*; could not pin down in this
session. Downstream effect: `(miller-rabin n)` and similar
defuns that depend on `cl-loop repeat k for a = ... unless ...
return nil` don't produce usable expansions, so Zoë's acceptance
suite does not run end-to-end on asm/lumbda-full yet.
* examples/ursa-scheme.lsp — `factor` crashes on asm under some
random seeds (bump-allocator exhaustion on long rhoff retry
chains). Out of CL_FULL scope; tracked in same ticket.
Next steps live in ticket 0005. This commit ships the infrastructure
so the remaining work is a debugging exercise against a reproducible
minimal case, not a feature build.
Zoë Trout's favorites at wedgewack.org/ursa.lisp.txt are Common Lisp:
iterative LOOP macros, setf cascades, defun with &optional, image-
based stone-lisp culture. Her first contribution to lumbda was a
question — "do we care for our programs, and how long are they alive
for?" — and the answer now extends beyond the RNG portal (§7.5) to
iteration style itself.
Four-phase delivery, all under ticket 0004:
Phase A — idiomatic Scheme ports at examples/ursa-scheme.lsp.
Every Zoë defun rewritten as named-let + tail recursion + list-
backed work queue + type-predicate dispatch.
Phase B — CL compat shim at cl-compat.lsp.
defun (with &optional), setf (simple vars, multi-pair), flet,
multiple-value-bind, t / nil (nil=#f so cond/if compose),
evenp/oddp/plusp/minusp/zerop, mod/ash/logbitp/nreverse,
cl-when/cl-unless (plain when is a void-returning lumbda special
form), declare (no-op), cddddr (missing accessor).
Phase C — cl-loop macro covering 14 patterns.
while/until/repeat, for VAR from A to/below/downto B, for VAR =
INIT [then STEP], for VAR across VEC, of-type T, do, when/unless
return, finally (return VAL). Sequential do*-style stepping via
gensym + cl-subst. Look-ahead termination so `repeat 4 for s = 4
then (- (* s s) 2) finally (return s)` returns 37634 (pre-step)
rather than 1416317954 (post-step). Every expansion ends in a
named-let tail call — TCO holds for loops of any length.
Phase D — load examples/ursa.lisp.txt with minimal annotation.
Preserves Zoë's CL. Minimal edits documented in file header:
load cl-compat.lsp, loop→cl-loop, when→cl-when, random→random-int,
&key→&optional. rho/digits omitted (need make-array/CLOS — see
ticket 0004 for scope boundary).
Defect uncovered along the way (c/types.c env_lookup): a "global
shortcut" checked global env immediately after missing the local
frame, SKIPPING intermediate parent scopes. Broke lexical scoping
whenever a parent scope shadowed a global. Reproduced with
(define s 4)
(let ((s 100)) (let ((m 0)) s)) ; returned 4, should return 100
Any nested let whose body referenced a shadowed name silently read
the global. Fix: remove the shortcut, walk the parent chain end-to-
end. 1255 assertions across five suites pass unchanged after fix —
surfaced only because cl-loop iterator names routinely collide with
globals accumulated in a stone-lisp image.
Whitepaper §9.2 documents the CL-in-Scheme design and the guarantees
that survive (TCO, portal determinism, cross-impl reproducibility).
Zoë added to authors + acknowledgments; reacknowledgment reframes
her first contribution as the deeper program-lifetime question, with
RNG portal as a derivative (§7.5) and cl-loop as the follow-up.
Tests: tests/cl-compat.lsp (44 assertions) and tests/ursa.lsp (28
assertions) exercise both paths under Python + C via tests/zoe-
favorites-test.sh, wired into make test-all.
MOAD notes: unmoad flags memq/assq in cl-compat.lsp over cl-loop-
keywords (~30 elements, constant) and var->new (≤4 state vars per
loop). Both are macro-expansion-time, bounded-small-N — not runtime
hot paths. Pre-existing c/types.c findings (strcmp-in-loop for
record-type lookup) are not from this change.
Closes tickets 0001 (portal-rng) and 0002 (os-entropy-seed).
Ticket 0001 goal 4 called for proof that seeding with k, drawing N,
saving, clearing, resuming in any impl, and drawing M more produces a
full stream matching a single-process Python baseline bit-for-bit.
Prior tests/portal-cross-test.sh exercised producer-consumer agreement
but used a producer-side self-computed baseline; it did not compare
against an independent Python run that never saves or resumes.
tests/portal-rng-cross-test.sh computes a single-process Python baseline
once (seed=42, N+M=10 draws, no portal), then runs all 9 producer x
consumer cells (Python, C, asm each side) and checks that producer's
first N plus consumer's M equals the independent baseline. All 12
assertions pass.
Wired into make test-all. Ticket status updated to resolved on both
0001 and 0002 with dated one-line resolution notes.
Design-only ticket. C's 48-bit NaN-boxed TAG_INT silently truncates
any result > 2^47 (e.g. 10^16 becomes -133099161583616), while Python
(bignum) and asm (61-bit) compute correctly. Recommends heap-allocated
bigint via new TAG_BIGINT, preserving NaN-boxing and JIT fast path for
the inline 48-bit common case. Phased migration with Phase 0 fail-loud
stopgap before the full bignum lands.
Ticket 0002 — reads 8 bytes from /dev/urandom (little-endian u64) and
seeds xoshiro256**. Opt-in kernel entropy for stochastic runs; the
default stays deterministic (k=0 at startup), so ticket 0001's
portal-reproducibility contract is unchanged.
Real-world flow now one call away:
Machine A: (random-seed-from-os!) + run simulation + portal-save
Machine B: portal-resume — same stream, bit-for-bit
All three impls fail loud on /dev/urandom trouble (LispErr in Python
and C, stderr + exit(1) in asm) — no silent fallback to a weak seed.
Tests:
- tests/functional.lsp: 2 new shared asserts (entropic + replay)
- asm/test.sh: 2 new asm-local checks (149 total, was 147)
- make test-all green across Python (205), C (205), asm (149)
Whitepaper §7.5 gains one sentence noting the OS-seed path.
unmoad: zero new findings in added code.
Zoe's contribution question: does our portal preserve RNG state so a
simulation can continue in another process with the same random stream?
Answer today: no — no RNG existed. Answer now (Python + C): yes, bit-identical.
- New builtins: random-seed!, random, random-int, random-state, random-state!
- xoshiro256** (Blackman & Vigna 2018) — deterministic, portable, no libc rand
- State = 4 x u64; portal-v1 JSON gains 'rng' field with 8 x u32 halves
- Python and C produce bit-identical streams (verified: seed=42, 10 draws)
- Asm impl + cross-impl tests + whitepaper note: next commits
Ticket: docs/tickets/0001-portal-rng.md
Historical internal name "uncommonlisp" retired in favor of the
public name "lumbda" ahead of lumbda.com going live. Scope of
this commit:
Source files renamed:
uncommonlisp.py -> lumbda.py
asm/uncommonlisp.s -> asm/lumbda.s
c/uncommonlisp.h -> c/lumbda.h
whitepaper/uncommonlisp-whitepaper -> whitepaper/lumbda-whitepaper (.rst + .pdf)
Binaries renamed (tracked ones; c/ was always gitignored):
asm/uncommonlisp, asm/uncommonlisp-gc, asm/uncommonlisp.o,
asm/uncommonlisp-gc.o -> asm/lumbda(-gc)(.o)
c/.gitignore -> ignores lumbda
Internal string updates (sed pass ordered longest-first):
asm/uncommonlisp -> asm/lumbda
c/uncommonlisp -> c/lumbda
uncommonlisp.py -> lumbda.py
UNCOMMONLISP_BIN -> LUMBDA_BIN (asm/test.sh env var)
"uncommonlisp> " -> "lumbda> " (asm REPL prompt baked into binary)
UNCOMMONLISP -> LUMBDA (macros, comments)
uncommonlisp -> lumbda (prose)
Binary portal magic updated:
"ULPORTAL" -> "LUMBDAB1" # "Lumbda Binary v1"
Old portal files are not backward-compatible — this is a deliberate
break since it's the rename moment. S-expression portals already
carry their own ";; lumbda-portal v1" header and remain cleanly
versioned.
WHITEPAPER.pdf / WHITEPAPER.rst symlinks repointed to the renamed
files. Makefile's whitepaper target targets lumbda-whitepaper.pdf.
Not changed (intentional, separate phases):
- Filesystem directory /home/fox/git/uncommonlisp itself
(fox renames locally and the gitlab repo URL in a follow-up)
- tests.py hardcoded cwd=/home/fox/git/uncommonlisp
(matches the current on-disk location; will flip when the
directory rename ships)
- Git history (immutable; old commits still say uncommonlisp,
which is correct — that's what they were)
Verified:
137 asm no-GC + 137 asm GC + 571 Python + 83 C + 189 shared
functional tests all pass under the new names.
bench-gc-http (2000 req): all 4 cells behave as expected
(cells 1/2 flat, 3 leaks, 4 bounded at 1 chunk).
Python REPL, C REPL, asm REPL all start cleanly.
Root-cause fix for the residual crashes I had documented as known
issues in §6.6.4. Every heap_alloc call site was setting its type
byte with `orq $(HT_X << 8), -8(%rax)` — but OR merges with the
stale type byte from a free-list-reused block. A pair previously
used as a vector (type 5 = 0b101) re-allocated as pair (type 1 =
0b001) ends up with merged type 0b101 = still vector. Walker then
treats the pair as a vector, reads the pair's car as a "length",
and walks off the block end — hence the hash-set bench's "unbound
variable: t", memory bench's "unbound variable: lst", arena bench's
"unbound variable: k".
Fix: overwrite the byte instead of OR-ing. 18 sites converted from
`orq $(HT_X << 8), -8(%rax)` to `movb $HT_X, -7(%rax)`. Every
previously-residual crash gone on first rerun.
Refreshed benchmark numbers throughout §6.6:
§6.6 Memory table: 122× less memory at 26% slowdown (was 124×,
30%). Range shifted because the fix also accelerated the common
paths; ratio stable.
§6.6.4 HTTP soak at 50,000 requests × 16 concurrent × 4 cells:
no-GC + snapshot 630 req/s peak 100 KB growth 4 KB
GC + snapshot 633 req/s peak 120 KB growth 4 KB
no-GC + no snapshot 625 req/s peak 458 MB OOM at cap
GC + no snapshot 610 req/s peak 1,092 KB growth 852 KB
Cell 4 now sustains 50K requests with steady-state 1-chunk memory.
Previous residual edge at 50K (cell 4 failing to start) was a
manifestation of the same type-byte bug, now gone.
§6.6.3 Adaptive numbers collapsed to within ~1% across all three
workloads (was 6% / 7% / 17% deltas). Paper updated to honestly
report adaptive as a null experiment on these shapes — neutral
cost, same stats surface, default on.
§6.6 diagram: bench-gc.png refreshed to match new numbers.
137 asm no-GC + 137 asm GC + 189 shared functional all pass.
Hash-set / memory / arena / adaptive / HTTP benches all clean.
Adds §6.6.3 "Collaborative Meta-GC: From Greedy to Adaptive" with
the three-workload benchmark (friendly / hostile / mixed × greedy
/ adaptive). Honest read of the numbers:
friendly greedy 732 ms 1000 resets, 0 escapes
friendly adaptive 691 ms 1000 resets, 0 escapes (-6%)
hostile greedy 568 ms 0 resets, 1000 escapes
hostile adaptive 607 ms 0 resets, 1000 escapes, 11 skipped (+7%)
mixed greedy 1981 ms 17 resets, 1983 escapes
mixed adaptive 1694 ms 14 resets, 1986 escapes, 2 skipped (-17%)
Adaptive wins on friendly (-6%) and mixed (-17%, the policy's
design target). On fully hostile workloads implicit GC fires 982
of 1000 arenas before the dispatcher sees them, so the signal is
drowned and greedy happens to edge adaptive by ~7%. Section
explicitly calls out the collaborative-but-local structure
(shared state on arena_active + EMA + countdown, decisions made
locally by each component) and credits the benchmark work with
surfacing two real correctness bugs in the conservative stack
scan — 24-byte strings misread as env nodes, 40-byte strings
misread as 25-element vectors — both now fixed.
Also:
- meta-gc-policy.dot rewritten to show the adaptive gate
(rate > 50% + probe countdown) before the greedy verify path;
new skip branch, new EMA annotations on edges.
- §6 reproducibility list + Makefile bench-gc-adaptive target.
- PDF rebuilt at 2.64 MB.
137 asm no-GC + 137 asm GC + 189 shared functional tests pass
against the new asm.
Fox flagged that the "A diagram is worth 10,000 words" quote
appeared twice in the paper but nothing was actually illustrated.
Fixed by:
1. Refreshing every .dot source to match current reality:
- docs/asm-architecture.dot: 22 KB (was "13 KB"), 14 syscalls
(was 4), 91 builtins (was 34), djb2 hash (was "linear scan"),
TCP stack + heap-snapshot + portal boxes added.
- docs/benchmark-sumto.dot: sum-to(1M) i5-8350U numbers; C
--fast 238 ms, asm 670 ms, Python --fast 5,136 ms. Was
sum-to(50k) with stale numbers.
- docs/benchmark-ack.dot: ackermann(3,8) i5-8350U numbers. Was
ack(3,4) with stale numbers.
- docs/benchmark-binary-size.dot: asm 22 KB, C 205 KB, busybox
2.1 MB, python3 8.0 MB. Was comparing against different
baselines.
2. Regenerated all PNGs via `make docs`.
3. Embedded in the paper at meaningful points:
- §2 Architecture (Python): python-architecture.png
- §6.4 Three-way bench: benchmark-sumto.png, benchmark-ack.png
- §11 Three Implementations: c-architecture.png, asm-
architecture.png
- §11.3 HTTP + sockets: benchmark-binary-size.png
4. Removed the redundant quote from §12.3; the one in §11
remains because §11 now follows it with two real diagrams.
Prerequisite fox noted: "make sure diagrams are up to date before
using them to code." Done — every embedded figure has the current
numbers/topology, not the old ones.
5 new diagrams (Graphviz DOT → PNG):
benchmark-ack.dot — ack(3,4) across all 5 tiers
benchmark-sumto.dot — sum-to(50k) across all 5 tiers
benchmark-fib.dot — fib(35) iterative across all 5 tiers
benchmark-speedup.dot — JIT speedup ratios (7x-784x)
benchmark-binary-size.dot — 13KB asm vs 171KB C vs ~30MB Python
JIT: 0.19ms ack, 7x faster than CPython, 784x faster than Python VM.
Makefile docs target now auto-discovers all docs/*.dot files.
Fresh in-process benchmarks across all implementations:
JIT: ack 0.19ms, fib 0.09ms, sum 0.55ms
CPython: ack 1.3ms, fib 0.006ms, sum 5.5ms
C interp: ack 20ms, fib 0.06ms, sum 109ms
Python VM: ack 149ms, fib 0.75ms, sum 437ms
Assembly: ack 8ms, fib 0.6ms, sum 43ms
JIT runs Scheme 7-10x faster than CPython runs Python.
Updated: language identified as R7RS Scheme throughout.
Test count updated to 943 across all implementations.
4 architecture diagrams (Graphviz DOT → PNG):
python-architecture.dot — bytecode VM + continuations + portal
c-architecture.dot — tree-walker + VM + JIT tiers
asm-architecture.dot — syscalls-only, 13KB binary
jit-pipeline.dot — AST → x86_64 machine code flow
docs/README.md — full architecture docs with embedded diagrams
and performance summary across all implementations.
Makefile: add asm-repl, docs target, clean-docs. Header comments
document all targets and test suites.
CLAUDE.md: add "A diagram is worth 10,000 words" (russell@unturf.com),
implementation table, test suite inventory.
Assembly is 2.5-4x faster than C interpreter on recursive workloads.
JIT remains 33x faster than hand-written assembly.
