lumbda/docs/tickets/0004-cl-compat.md

0004 — Common Lisp compatibility for Zoë Trout's favorites

Status: resolved Reporter: Zoë Trout (via fox) Implementer: blackops Opened: 2026-04-23 Resolved: 2026-04-24

Problem

Zoë Trout asked a question most systems people never think to ask: do we care for our programs, and how long are they alive for? Common Lisp culture — stone-lisp, image-based development — produces processes that outlive the developer's attention by days, months, sometimes years. Killing them is almost never on purpose.

Her favorite programs (https://wedgewack.org/ursa.lisp.txt) carry that DNA. They lean on CL's loop macro for iteration, setf for mutation, defun with &optional / &key, defgeneric/defmethod for type dispatch, and dynamic arrays (make-array :adjustable :fill-pointer, vector-push-extend, vector-pop) for work queues.

None of these exist in Scheme. A strict port would require Zoë to rewrite every iterative loop as tail-recursion, every setf as a set! cascade, every defgeneric as a cond by type. That strips her programs of the shape she likes.

Goal: keep Zoë's iterative style intact while running her code on lumbda, without compromising the guarantees lumbda already carries (TCO, portal determinism, cross-impl reproducibility).

Scope — four phases

Phase A — idiomatic Scheme ports

Ship examples/ursa-scheme.lsp: every Zoë defun rewritten as named- let + tail recursion + list-backed stacks + type-predicate dispatch. Keeps downstream lumbda machinery pristine. Tests verify correctness against known primes (Mersenne M₃, M₅, M₇, M₁₃), known factorizations (12, 1001, 97), and digit round-trips.

Phase B — CL compat shim (cl-compat.lsp)

A loadable file that provides CL spellings on top of lumbda primitives:

CL form	Expansion	Notes
defun	define macro	supports &optional (var default)
setf	set! cascade	simple variables only; multi-pair supported
flet	let + lambda	local function binding
multiple-value-bind	call-with-values	built on lumbda's values
declare	no-op	compile-time directives ignored
t, nil	#t, #f	nil = #f (not '()) so cond/if work
evenp, oddp, plusp, minusp, zerop	aliases	CL -p → R5RS ?
mod, ash, logbitp, nreverse	aliases / derived	bit ops derived from expt 2 + odd?
cl-when, cl-unless	if with #f on false	plain when/unless in lumbda are special forms returning #<void>, which is truthy in Scheme — cl-when returns #f so CL "nil on false" idioms compose with Scheme cond/if

Not supported (see Phase A for hand-ports):

• &key / &rest / &aux / &body (only &optional)
• defgeneric / defmethod (CLOS)
• make-array :adjustable :fill-pointer, vector-push-extend, vector-pop, fill-pointer (dynamic arrays)
• sbit / bit vectors
• coerce to arbitrary types
• Generalized setf (on car, vector-ref, etc.)

Phase C — cl-loop macro

define-macro implementation of a CL LOOP subset, covering every pattern used in ursa.lisp.txt:

Clause	Example
with VAR = INIT	pre-loop binding
for VAR from A to B	ascending range, inclusive
for VAR from A below B	ascending range, exclusive
for VAR from A downto B	descending range, inclusive
for VAR = INIT	re-evaluated each iter
for VAR = INIT then STEP	do*-style sequential stepping
for VAR across VEC	vector traversal by index
for VAR of-type T	type hint ignored (same as for VAR)
while TEST	pre-body continuation test
until TEST	pre-body termination test
repeat N	counted iterations (look-ahead termination)
do EXPR ...	body forms
when TEST return VAL	mid-loop early exit
unless TEST return VAL	mid-loop early exit (inverted)
finally (return EXPR)	final value expression
finally EXPR ...	final body (no return value)

Every expansion ends in a tail call to a named-let, preserving TCO.

The macro does look-ahead termination for repeat and for ... from ... to/below/downto: after the body runs, we compute new step values into fresh gensym bindings, check termination against those, and either recurse with new values or invoke the finalizer with CURRENT (unstepped) state. This matches CL's "would step past the limit" semantics and matters for patterns like (loop repeat 4 for s = 4 then (- (* s s) 2) finally (return s)) — terminates with s = 37634 after 4 body-runs, not s = 1416317954 after 5.

Phase D — load ursa.lisp.txt with minimal annotations

examples/ursa.lisp.txt preserves Zoë's CL. Minimal edits documented in its header:

• prepend (load "cl-compat.lsp")
• loop → cl-loop (reserved-name avoidance)
• when → cl-when (plain when is a void-returning special form)
• random → random-int (lumbda's RNG)
• &key → &optional (positional args only in the shim)
• omit rho and digits (need make-array / CLOS — see Phase A)

Every other line — every loop for ... from ... to, every setf a 1 b 2 c 3, every declare (optimize ...) — loads and runs unchanged.

Guarantees — what survives

Every shim is syntactic sugar over lumbda's existing functional core. After macroexpansion, all code becomes standard Scheme:

• defun → define
• setf on simple var → set! (existing lumbda capability)
• flet → let + lambda
• cl-loop → let* + named-let + cond + tail recursion

Therefore:

• TCO — every emitted form terminates in a named-let tail call. Stack stays flat for loops of any length.
• Portal determinism — no new mutable types introduced (the queue in rho is a list; adjustable vectors would need new portal serialization and are deliberately out of scope).
• Cross-impl reproducibility — cl-compat.lsp is pure Scheme and loads identically in Python + C. Every Zoë defun runs bit-for-bit identically across tiers (within each tier's integer domain — see ticket 0003).

Defect fixed along the way

Building Phase C surfaced a real lumbda-C defect: env_lookup had a "global shortcut" that checked the global env immediately after missing the local frame, SKIPPING intermediate parent scopes. For code like:

(define s 4)
(let ((s 100))
  (let ((m 0))
    s))   ; returned 4 (global) instead of 100 (outer let)

the inner let body resolved s to the global binding because the shortcut fired before the parent chain was walked. Any code that shadowed a global inside a let, then opened another let whose body referenced the shadowed name, silently read the global instead. This broke cl-loop whenever a state var name collided with a global — common under the stone-lisp development pattern that Zoë's culture leans on.

Fix: remove the global shortcut; walk the parent chain end-to-end. Complexity is O(scope-depth), bounded by code structure not runtime. Full test suite (571 + 83 + 137 + 196 + 196 + 12 + 60 = 1255 assertions) passes unchanged.

File: c/types.c. Commit lands in same drop as this ticket.

Test strategy

• tests/cl-compat.lsp — 44 assertions covering every compat shim and 13 cl-loop patterns. Runs in Python + C.
• tests/ursa.lsp — 28 assertions covering Scheme port (15) and CL-via-shim (13) side-by-side. Runs in Python + C.
• tests/zoe-favorites-test.sh — bundles both above, wired into make test-all via target zoe-favorites-test.

Victory condition: make test-all green; Zoë's original CL programs produce matching answers across Python, C, and the Scheme port path.

Files

• cl-compat.lsp — compat shim (~240 lines)
• examples/ursa.lisp.txt — Zoë's CL (minimally annotated)
• examples/ursa-scheme.lsp — idiomatic Scheme ports (Phase A)
• tests/cl-compat.lsp — shim acceptance tests
• tests/ursa.lsp — Zoë-favorites acceptance tests
• tests/zoe-favorites-test.sh — test runner
• c/types.c — env_lookup defect fix