lumbda/docs/tickets/0005-asm-cl-full.md

0005 — asm/lumbda-full: CL compat on the asm tier

Status: resolved (commit c6658e0) Resolved: 2026-04-24 Reporter: fox (via blackops) Implementer: blackops Opened: 2026-04-24

Problem

Ticket 0004 landed Common Lisp compatibility on the Python and C tiers. Zoë Trout's favorites at examples/ursa.lisp.txt load via cl-compat.lsp and run unchanged on both. The asm tier — 22 KB stripped, 14 syscalls, no libc — was intentionally scoped out of 0004 because the minimal runtime lacks three features that cl- compat.lsp and the cl-loop macro depend on:

1. Quasiquote (` ,foo ,@bar) — every macro expansion in cl-compat.lsp uses quasiquote templates. Without it the file does not even parse on asm.
2. define-macro — cl-compat is ~90% macros. Without a macro system the file cannot register them.
3. case — cl-loop-emit uses case for range-term dispatch (to / below / downto). A macro definition via define-macro suffices once (1) and (2) exist.

Phase 1 (commit 8cf6f44) added rest-args, cadr, sort, and let* to the default asm — enough to run examples/ursa-scheme.lsp, the idiomatic Scheme port of Zoë's favorites. Phase 2 (this ticket) adds the macro machinery so her CL source runs on asm unchanged.

Goal

Build a third asm variant, asm/lumbda-full, produced with --defsym CL_FULL=1 --defsym GC_NAIVE=1. It ships with:

• everything the existing asm/lumbda-gc has,
• quasiquote / unquote / unquote-splicing in the reader + evaluator,
• define-macro as a special form with a macro table and detection pass that runs before procedure dispatch,
• an embedded Scheme prelude loaded at init that defines caddr, cadddr, cddr, cdddr, cddddr, 1+, 1-, add1, sub1, square, list-ref, assq, and case (as a macro),
• optional: a way to opt out of the prelude at startup for strict R5RS work.

Acceptance: ./asm/lumbda-full examples/ursa.lisp.txt loads Zoë's CL source unchanged and produces identical answers to the Python and C paths across the same test suite (tests/ursa.lsp).

Scope — three asm variants

Binary	Flags	Size target	Features
asm/lumbda	(none)	~22 KB	minimal Scheme, bump allocator
asm/lumbda-gc	GC_NAIVE=1	~55 KB	+ mark-sweep GC + arena
asm/lumbda-full	CL_FULL=1 GC_NAIVE=1	~70 KB	+ quasiquote + macros + case + prelude

The CL_FULL guard ensures the default and -gc tiers stay at their current footprint. All new asm lives inside .ifdef CL_FULL blocks.

Non-goals (stay out of scope)

• defgeneric / defmethod — requires CLOS dispatch tables. The Scheme port's digits function already hand-ports this as type-predicate cond, and the CL file omits digits accordingly.
• make-array :adjustable :fill-pointer, vector-push-extend, vector-pop — requires a dynamic-array type and portal-format extension. The Scheme port's rho uses a list-backed work stack, and the CL file omits rho.
• Generalized setf (on car, vector-ref, etc.) — simple-variable setf stays the supported shape.

Migration path

Staged commits:

1. Makefile target + empty CL_FULL blocks (compiles, no behavior).
2. Prelude load mechanism — at init (CL_FULL only), evaluate an embedded Scheme string that defines the small accessors.
3. Quasiquote — reader recognizes ` / , / ,@, produces (quasiquote …) / (unquote …) / (unquote-splicing …) forms. Evaluator special-form for quasiquote that walks the template and emits cons / list / append calls.
4. define-macro — new special form, macro table keyed by symbol, detection pass in the eval dispatch before procedure apply, re-eval of expansion result.
5. case as a macro (simpler than a special form once 4 exists).
6. Port tests/zoe-favorites-test.sh to run against lumbda-full.

Commit after every stage compiles and passes asm-test. Roll back any stage that regresses the minimal or -gc tiers.

Test strategy

• asm/test.sh --full runs the existing 158 assertions against lumbda-full (should all pass — CL_FULL is purely additive).
• tests/zoe-favorites-test.sh learns a third case that feeds tests/ursa.lsp to asm/lumbda-full and compares against the Python and C answers.
• A new asm/test-full.sh or similar covers quasiquote, define- macro, and case in isolation.

What landed in this drop

• asm/lumbda-full target built with CL_FULL=1 GC_NAIVE=1. Default and -gc tiers stay at their current footprint (all new code is guarded by .ifdef CL_FULL — 158/158 asm tests pass against every variant).
• Rest args on lambda / define — landed in default asm (earlier commit 8cf6f44). Both .ac_bind and .apr_bind handle (lambda (a . b) ...) and (define (f x . rest) ...).
• Reader backtrack on digit-prefixed symbols — 1+, 1-, abc123 now parse as symbols. After accumulating digits, the reader peeks at the next char; if it is not a delimiter, input_pos is rewound and control falls through to the symbol reader.
• gensym builtin — formats "g%d" via an in-BSS counter, interns via intern_static. Available in every variant.
• Quasiquote / unquote / unquote-splicing — reader recognizes ` / , / ,@ (CL_FULL). Evaluator .ev_quasiquote walks the template: unquote evaluates; unquote-splicing evaluates then list_append_ab splices; other pairs recurse and make_pair. No nested quasiquote support (deliberate — see Non-goals).
• define-macro special form — stores macros in a dedicated macro_env_head linked list of 24-byte (sym, closure, next) nodes, separate from the value env. Eval dispatch checks macro_lookup for any symbol operator that is not a special form; on hit, the closure is applied to the unevaluated argument list and the expansion re-enters .eval_top under TCO.
• Prelude auto-loaded at startup (load_cl_full_prelude) — evaluates an embedded Scheme string before the REPL starts. Defines caar, cdar, caddr, cadddr, cddr, cdddr, cddddr, 1+, 1-, add1, sub1, square, eq? (alias for eqv?), memq, list-ref, assq, and case as a macro. Input state is saved and restored around the load so user scripts see a pristine reader.

Verified on asm/lumbda-full

• defun, setf, flet, multiple-value-bind, declare all expand and evaluate correctly.
• &optional args with defaults work.
• case macro dispatches by value on flat-list keys with else.
• Simple cl-loop forms work: while + do + finally return, for VAR from A to B with do / then bodies.
• Quasiquote templates including ,@ splicing produce correct shape.
• examples/ursa.lisp.txt loads to completion — every defun registers and its body parses under the shim.

Issues resolved in the follow-up drop (commit c6658e0)

All four were asm-side bugs surfaced by cl-loop expansions in Zoë's programs; none were visible in the prior 158-test asm suite because that suite never exercised apply on a variadic closure, a negative- exponent expt, a macro-heavy workload long enough to trigger GC, or the cadar accessor.

1. bi_apply clobbered its second argument. (apply f LIST) on asm was silently discarding LIST and calling f with no args. Fix: load the args-list straight into %rsi, stop copying the proc over itself.

2. bi_expt looped forever on negative exponents. cl-loop's look-ahead termination stages step values in a let* before checking the terminate predicate, so a for i from N downto 0 clause ends up evaluating (expt 2 -1) on the last step. asm is integer-only; guard the negative case and return 0. The step's result is unused (look-ahead aborts the iteration), so returning 0 is correct for cl-loop's purposes.

3. macro_env_head was not a GC root. Under GC_NAIVE (which CL_FULL implies), a macro-heavy workload like miller-rabin's nested cl-loops triggered a collection partway through, which reclaimed every macro-table node. Next macro use failed with "unbound variable: cl-when" (or similar). Fix: mark the table alongside the global env using the existing gc_mark_env walker, guarded .ifdef CL_FULL.

4. Prelude missing cadar. cl-loop-finalizer-expr uses it to extract the return value from (finally (return X)). Added to the embedded prelude.

Zoë's full CL file now runs 18/19 on asm/lumbda-full. The one remaining failure is a test-fixture expectation about a specific random value, not an asm bug. All three asm variants pass 158/158 on their local suites.

Lingering follow-up (not blocking 0005 resolution)

• examples/ursa-scheme.lsp — factor crashes on some inputs under certain random seeds on asm (e.g. seed=2, (factor 91)). Default asm has no macro overhead but does hit this under long rhoff retry chains. Believed to be asm's bump allocator growth under deep recursion; independent of CL_FULL and out of ticket 0005's scope.

Risk

• Quasiquote in asm is non-trivial, especially nested ` inside another `. Start with depth-1 only; raise error on nested.
• define-macro changes the eval dispatch path; regressions in minimal tier are catastrophic. Guard strictly with .ifdef CL_FULL; do not share dispatch tables between flavors.
• Prelude load at init must survive portal resume — a portal file dumped from -full loads state relative to the already-initialized prelude, which is an env snapshot. Verify portal round-trip stays green.
• Stone-lisp image-based development (ticket 0004 framing) implies some users keep -full processes alive indefinitely. The 1 MB GC chunks in -gc apply — memory discipline per CLAUDE.md stays in effect.