lumbda/docs/tickets/0005-asm-cl-full.md

0005 — asm/lumbda-full: CL compat on the asm tier

Status: partially resolved — infrastructure landed, cl-loop-emit edge case open Reporter: fox (via blackops) Implementer: blackops Opened: 2026-04-24

Problem

Ticket 0004 landed Common Lisp compatibility on the Python and C tiers. Zoë Trout's favorites at examples/ursa.lisp.txt load via cl-compat.lsp and run unchanged on both. The asm tier — 22 KB stripped, 14 syscalls, no libc — was intentionally scoped out of 0004 because the minimal runtime lacks three features that cl- compat.lsp and the cl-loop macro depend on:

1. Quasiquote (` ,foo ,@bar) — every macro expansion in cl-compat.lsp uses quasiquote templates. Without it the file does not even parse on asm.
2. define-macro — cl-compat is ~90% macros. Without a macro system the file cannot register them.
3. case — cl-loop-emit uses case for range-term dispatch (to / below / downto). A macro definition via define-macro suffices once (1) and (2) exist.

Phase 1 (commit 8cf6f44) added rest-args, cadr, sort, and let* to the default asm — enough to run examples/ursa-scheme.lsp, the idiomatic Scheme port of Zoë's favorites. Phase 2 (this ticket) adds the macro machinery so her CL source runs on asm unchanged.

Goal

Build a third asm variant, asm/lumbda-full, produced with --defsym CL_FULL=1 --defsym GC_NAIVE=1. It ships with:

• everything the existing asm/lumbda-gc has,
• quasiquote / unquote / unquote-splicing in the reader + evaluator,
• define-macro as a special form with a macro table and detection pass that runs before procedure dispatch,
• an embedded Scheme prelude loaded at init that defines caddr, cadddr, cddr, cdddr, cddddr, 1+, 1-, add1, sub1, square, list-ref, assq, and case (as a macro),
• optional: a way to opt out of the prelude at startup for strict R5RS work.

Acceptance: ./asm/lumbda-full examples/ursa.lisp.txt loads Zoë's CL source unchanged and produces identical answers to the Python and C paths across the same test suite (tests/ursa.lsp).

Scope — three asm variants

Binary	Flags	Size target	Features
asm/lumbda	(none)	~22 KB	minimal Scheme, bump allocator
asm/lumbda-gc	GC_NAIVE=1	~55 KB	+ mark-sweep GC + arena
asm/lumbda-full	CL_FULL=1 GC_NAIVE=1	~70 KB	+ quasiquote + macros + case + prelude

The CL_FULL guard ensures the default and -gc tiers stay at their current footprint. All new asm lives inside .ifdef CL_FULL blocks.

Non-goals (stay out of scope)

• defgeneric / defmethod — requires CLOS dispatch tables. The Scheme port's digits function already hand-ports this as type-predicate cond, and the CL file omits digits accordingly.
• make-array :adjustable :fill-pointer, vector-push-extend, vector-pop — requires a dynamic-array type and portal-format extension. The Scheme port's rho uses a list-backed work stack, and the CL file omits rho.
• Generalized setf (on car, vector-ref, etc.) — simple-variable setf stays the supported shape.

Migration path

Staged commits:

1. Makefile target + empty CL_FULL blocks (compiles, no behavior).
2. Prelude load mechanism — at init (CL_FULL only), evaluate an embedded Scheme string that defines the small accessors.
3. Quasiquote — reader recognizes ` / , / ,@, produces (quasiquote …) / (unquote …) / (unquote-splicing …) forms. Evaluator special-form for quasiquote that walks the template and emits cons / list / append calls.
4. define-macro — new special form, macro table keyed by symbol, detection pass in the eval dispatch before procedure apply, re-eval of expansion result.
5. case as a macro (simpler than a special form once 4 exists).
6. Port tests/zoe-favorites-test.sh to run against lumbda-full.

Commit after every stage compiles and passes asm-test. Roll back any stage that regresses the minimal or -gc tiers.

Test strategy

• asm/test.sh --full runs the existing 158 assertions against lumbda-full (should all pass — CL_FULL is purely additive).
• tests/zoe-favorites-test.sh learns a third case that feeds tests/ursa.lsp to asm/lumbda-full and compares against the Python and C answers.
• A new asm/test-full.sh or similar covers quasiquote, define- macro, and case in isolation.

What landed in this drop

• asm/lumbda-full target built with CL_FULL=1 GC_NAIVE=1. Default and -gc tiers stay at their current footprint (all new code is guarded by .ifdef CL_FULL — 158/158 asm tests pass against every variant).
• Rest args on lambda / define — landed in default asm (earlier commit 8cf6f44). Both .ac_bind and .apr_bind handle (lambda (a . b) ...) and (define (f x . rest) ...).
• Reader backtrack on digit-prefixed symbols — 1+, 1-, abc123 now parse as symbols. After accumulating digits, the reader peeks at the next char; if it is not a delimiter, input_pos is rewound and control falls through to the symbol reader.
• gensym builtin — formats "g%d" via an in-BSS counter, interns via intern_static. Available in every variant.
• Quasiquote / unquote / unquote-splicing — reader recognizes ` / , / ,@ (CL_FULL). Evaluator .ev_quasiquote walks the template: unquote evaluates; unquote-splicing evaluates then list_append_ab splices; other pairs recurse and make_pair. No nested quasiquote support (deliberate — see Non-goals).
• define-macro special form — stores macros in a dedicated macro_env_head linked list of 24-byte (sym, closure, next) nodes, separate from the value env. Eval dispatch checks macro_lookup for any symbol operator that is not a special form; on hit, the closure is applied to the unevaluated argument list and the expansion re-enters .eval_top under TCO.
• Prelude auto-loaded at startup (load_cl_full_prelude) — evaluates an embedded Scheme string before the REPL starts. Defines caar, cdar, caddr, cadddr, cddr, cdddr, cddddr, 1+, 1-, add1, sub1, square, eq? (alias for eqv?), memq, list-ref, assq, and case as a macro. Input state is saved and restored around the load so user scripts see a pristine reader.

Verified on asm/lumbda-full

• defun, setf, flet, multiple-value-bind, declare all expand and evaluate correctly.
• &optional args with defaults work.
• case macro dispatches by value on flat-list keys with else.
• Simple cl-loop forms work: while + do + finally return, for VAR from A to B with do / then bodies.
• Quasiquote templates including ,@ splicing produce correct shape.
• examples/ursa.lisp.txt loads to completion — every defun registers and its body parses under the shim.

Known open issues (follow-up work)

1. cl-loop-emit crashes on some parsed inputs. Specifically, (cl-loop-emit '(() ((simple a 5)) () () () #f () ())) returns (let* () (let g0 () (g0))) — the simple iter's state binding is dropped. Same input on Python and C returns the correct (let* ((g3 5)) (let g0 ((a g3)) ...)). The parse output on asm is correct; the bug is inside the emit's giant let* (30+ bindings). Reproduced in isolation; could not pin down after a few hours — likely an asm-side env or stack interaction that surfaces only inside this specific call shape.

   Consequence: (miller-rabin n) and other defuns whose bodies use cl-loop repeat k for a = ... expand incorrectly and crash at run time. Zoë's full acceptance suite does not run end-to-end yet on asm/lumbda-full.

2. examples/ursa-scheme.lsp — factor crashes on some inputs under certain random seeds on asm (e.g. seed=2, (factor 91)). Default asm has no macro overhead but does hit this under long rhoff retry chains. Believed to be asm's bump allocator / stack growth under deep recursion; out of CL_FULL's scope.

Risk

• Quasiquote in asm is non-trivial, especially nested ` inside another `. Start with depth-1 only; raise error on nested.
• define-macro changes the eval dispatch path; regressions in minimal tier are catastrophic. Guard strictly with .ifdef CL_FULL; do not share dispatch tables between flavors.
• Prelude load at init must survive portal resume — a portal file dumped from -full loads state relative to the already-initialized prelude, which is an env snapshot. Verify portal round-trip stays green.
• Stone-lisp image-based development (ticket 0004 framing) implies some users keep -full processes alive indefinitely. The 1 MB GC chunks in -gc apply — memory discipline per CLAUDE.md stays in effect.