# 0005 — `asm/lumbda-full`: CL compat on the asm tier

**Status:** resolved (commit `c6658e0`)
**Resolved:** 2026-04-24
**Reporter:** fox (via blackops)
**Implementer:** blackops
**Opened:** 2026-04-24

## Problem

Ticket 0004 landed Common Lisp compatibility on the Python and C
tiers. Zoë Trout's favorites at `examples/ursa.lisp.txt` load via
`cl-compat.lsp` and run unchanged on both. The asm tier — 22 KB
stripped, 14 syscalls, no libc — was intentionally scoped out of
0004 because the minimal runtime lacks three features that `cl-
compat.lsp` and the `cl-loop` macro depend on:

1. **Quasiquote** (`` ` ,foo ,@bar ``) — every macro expansion in
   `cl-compat.lsp` uses quasiquote templates. Without it the file
   does not even parse on asm.
2. **`define-macro`** — cl-compat is ~90% macros. Without a macro
   system the file cannot register them.
3. **`case`** — `cl-loop-emit` uses `case` for range-term dispatch
   (`to` / `below` / `downto`). A macro definition via define-macro
   suffices once (1) and (2) exist.

Phase 1 (commit `8cf6f44`) added rest-args, `cadr`, `sort`, and `let*`
to the default asm — enough to run `examples/ursa-scheme.lsp`, the
idiomatic Scheme port of Zoë's favorites. Phase 2 (this ticket) adds
the macro machinery so her CL source runs on asm unchanged.

## Goal

Build a third asm variant, `asm/lumbda-full`, produced with
`--defsym CL_FULL=1 --defsym GC_NAIVE=1`. It ships with:

- everything the existing `asm/lumbda-gc` has,
- quasiquote / unquote / unquote-splicing in the reader + evaluator,
- `define-macro` as a special form with a macro table and detection
  pass that runs before procedure dispatch,
- an embedded Scheme prelude loaded at init that defines `caddr`,
  `cadddr`, `cddr`, `cdddr`, `cddddr`, `1+`, `1-`, `add1`, `sub1`,
  `square`, `list-ref`, `assq`, and `case` (as a macro),
- optional: a way to opt out of the prelude at startup for strict
  R5RS work.

Acceptance: `./asm/lumbda-full examples/ursa.lisp.txt` loads Zoë's
CL source unchanged and produces identical answers to the Python and
C paths across the same test suite (`tests/ursa.lsp`).

## Scope — three asm variants

| Binary | Flags | Size target | Features |
|---|---|---|---|
| `asm/lumbda` | (none) | ~22 KB | minimal Scheme, bump allocator |
| `asm/lumbda-gc` | `GC_NAIVE=1` | ~55 KB | + mark-sweep GC + arena |
| `asm/lumbda-full` | `CL_FULL=1 GC_NAIVE=1` | ~70 KB | + quasiquote + macros + case + prelude |

The `CL_FULL` guard ensures the default and `-gc` tiers stay at their
current footprint. All new asm lives inside `.ifdef CL_FULL` blocks.

## Non-goals (stay out of scope)

- `defgeneric` / `defmethod` — requires CLOS dispatch tables. The
  Scheme port's `digits` function already hand-ports this as
  type-predicate cond, and the CL file omits `digits` accordingly.
- `make-array :adjustable :fill-pointer`, `vector-push-extend`,
  `vector-pop` — requires a dynamic-array type and portal-format
  extension. The Scheme port's `rho` uses a list-backed work stack,
  and the CL file omits `rho`.
- Generalized `setf` (on `car`, `vector-ref`, etc.) — simple-variable
  setf stays the supported shape.

## Migration path

Staged commits:

1. Makefile target + empty CL_FULL blocks (compiles, no behavior).
2. Prelude load mechanism — at init (CL_FULL only), evaluate an
   embedded Scheme string that defines the small accessors.
3. Quasiquote — reader recognizes `` ` `` / `,` / `,@`, produces
   `(quasiquote …)` / `(unquote …)` / `(unquote-splicing …)` forms.
   Evaluator special-form for `quasiquote` that walks the template
   and emits `cons` / `list` / `append` calls.
4. `define-macro` — new special form, macro table keyed by symbol,
   detection pass in the eval dispatch before procedure apply,
   re-eval of expansion result.
5. `case` as a macro (simpler than a special form once 4 exists).
6. Port `tests/zoe-favorites-test.sh` to run against `lumbda-full`.

Commit after every stage compiles and passes asm-test. Roll back any
stage that regresses the minimal or -gc tiers.

## Test strategy

- `asm/test.sh --full` runs the existing 158 assertions against
  `lumbda-full` (should all pass — CL_FULL is purely additive).
- `tests/zoe-favorites-test.sh` learns a third case that feeds
  `tests/ursa.lsp` to `asm/lumbda-full` and compares against the
  Python and C answers.
- A new `asm/test-full.sh` or similar covers quasiquote, define-
  macro, and case in isolation.

## What landed in this drop

- `asm/lumbda-full` target built with `CL_FULL=1 GC_NAIVE=1`. Default
  and `-gc` tiers stay at their current footprint (all new code is
  guarded by `.ifdef CL_FULL` — 158/158 asm tests pass against every
  variant).
- Rest args on `lambda` / `define` — landed in default asm (earlier
  commit `8cf6f44`). Both `.ac_bind` and `.apr_bind` handle
  `(lambda (a . b) ...)` and `(define (f x . rest) ...)`.
- Reader backtrack on digit-prefixed symbols — `1+`, `1-`, `abc123`
  now parse as symbols. After accumulating digits, the reader peeks
  at the next char; if it is not a delimiter, input_pos is rewound
  and control falls through to the symbol reader.
- `gensym` builtin — formats `"g%d"` via an in-BSS counter, interns
  via `intern_static`. Available in every variant.
- Quasiquote / unquote / unquote-splicing — reader recognizes
  `` ` `` / `,` / `,@` (CL_FULL). Evaluator `.ev_quasiquote` walks
  the template: unquote evaluates; unquote-splicing evaluates then
  `list_append_ab` splices; other pairs recurse and `make_pair`. No
  nested quasiquote support (deliberate — see Non-goals).
- `define-macro` special form — stores macros in a dedicated
  `macro_env_head` linked list of 24-byte `(sym, closure, next)`
  nodes, separate from the value env. Eval dispatch checks
  `macro_lookup` for any symbol operator that is not a special form;
  on hit, the closure is applied to the *unevaluated* argument list
  and the expansion re-enters `.eval_top` under TCO.
- Prelude auto-loaded at startup (`load_cl_full_prelude`) — evaluates
  an embedded Scheme string before the REPL starts. Defines `caar`,
  `cdar`, `caddr`, `cadddr`, `cddr`, `cdddr`, `cddddr`, `1+`, `1-`,
  `add1`, `sub1`, `square`, `eq?` (alias for `eqv?`), `memq`,
  `list-ref`, `assq`, and `case` as a macro. Input state is saved
  and restored around the load so user scripts see a pristine reader.

## Verified on `asm/lumbda-full`

- `defun`, `setf`, `flet`, `multiple-value-bind`, `declare` all
  expand and evaluate correctly.
- `&optional` args with defaults work.
- `case` macro dispatches by value on flat-list keys with `else`.
- Simple `cl-loop` forms work: `while + do + finally return`,
  `for VAR from A to B` with `do` / `then` bodies.
- Quasiquote templates including `,@` splicing produce correct shape.
- `examples/ursa.lisp.txt` loads to completion — every `defun`
  registers and its body parses under the shim.

## Issues resolved in the follow-up drop (commit `c6658e0`)

All four were asm-side bugs surfaced by cl-loop expansions in Zoë's
programs; none were visible in the prior 158-test asm suite because
that suite never exercised `apply` on a variadic closure, a negative-
exponent `expt`, a macro-heavy workload long enough to trigger GC, or
the `cadar` accessor.

1. **`bi_apply` clobbered its second argument.** `(apply f LIST)` on
   asm was silently discarding LIST and calling f with no args. Fix:
   load the args-list straight into `%rsi`, stop copying the proc
   over itself.

2. **`bi_expt` looped forever on negative exponents.** cl-loop's
   look-ahead termination stages step values in a `let*` before
   checking the terminate predicate, so a `for i from N downto 0`
   clause ends up evaluating `(expt 2 -1)` on the last step. asm is
   integer-only; guard the negative case and return 0. The step's
   result is unused (look-ahead aborts the iteration), so returning
   0 is correct for cl-loop's purposes.

3. **`macro_env_head` was not a GC root.** Under GC_NAIVE (which
   CL_FULL implies), a macro-heavy workload like miller-rabin's
   nested cl-loops triggered a collection partway through, which
   reclaimed every macro-table node. Next macro use failed with
   "unbound variable: cl-when" (or similar). Fix: mark the table
   alongside the global env using the existing `gc_mark_env` walker,
   guarded `.ifdef CL_FULL`.

4. **Prelude missing `cadar`.** `cl-loop-finalizer-expr` uses it to
   extract the return value from `(finally (return X))`. Added to
   the embedded prelude.

Zoë's full CL file now runs 18/19 on asm/lumbda-full. The one
remaining failure is a test-fixture expectation about a specific
random value, not an asm bug. All three asm variants pass 158/158
on their local suites.

## Lingering follow-up (not blocking 0005 resolution)

- **`examples/ursa-scheme.lsp` — `factor` crashes on some inputs**
  under certain random seeds on asm (e.g. seed=2, `(factor 91)`).
  Default asm has no macro overhead but does hit this under long
  rhoff retry chains. Believed to be asm's bump allocator growth
  under deep recursion; independent of CL_FULL and out of ticket
  0005's scope.

## Risk

- Quasiquote in asm is non-trivial, especially nested `` ` `` inside
  another `` ` ``. Start with depth-1 only; raise error on nested.
- `define-macro` changes the eval dispatch path; regressions in
  minimal tier are catastrophic. Guard strictly with `.ifdef
  CL_FULL`; do not share dispatch tables between flavors.
- Prelude load at init must survive portal resume — a portal file
  dumped from `-full` loads state relative to the already-initialized
  prelude, which is an env snapshot. Verify portal round-trip stays
  green.
- Stone-lisp image-based development (ticket 0004 framing) implies
  some users keep `-full` processes alive indefinitely. The 1 MB GC
  chunks in `-gc` apply — memory discipline per CLAUDE.md stays in
  effect.
