russell@unturf.com
4ff87920cf
Third asm variant — built with CL_FULL=1 GC_NAIVE=1 via new Makefile
target. Adds the macro machinery needed for cl-compat.lsp on the asm
tier, keeping every addition behind .ifdef CL_FULL so the default
(~22 KB) and -gc binaries keep their current footprint.
Landed in this drop:
* Reader: backtrack on digit-prefixed symbols. After reading digit
characters, if the next char is not a delimiter, input_pos
rewinds and control falls through to .sr_symbol. Makes 1+, 1-,
add1, abc123, and any CL-style identifier with a numeric prefix
parse as symbols instead of truncating to a bare integer.
* Reader: `` ` `` / `,` / `,@` produce (quasiquote X) / (unquote X)
/ (unquote-splicing X) forms. Same build shape as the existing
`'` quote branch.
* Evaluator: .ev_quasiquote + quasiquote_expand walk the template.
unquote evaluates its argument in the current env; unquote-
splicing evaluates then splices via a new list_append_ab helper;
other pairs recurse (cons expand-car expand-cdr). Atoms pass
through. No nested quasiquote depth (deliberate; ticket 0005
scope).
* Evaluator: .ev_define_macro + macro_env_head linked list. Each
(define-macro (name p...) body) prepends a 24-byte
(sym, closure, next) node. Dispatch in eval checks macro_lookup
after all special-form compares; on hit, the closure is applied
to the *unevaluated* argument list and the expansion re-enters
.eval_top under TCO.
* Binding: rest-arg support extended to .apr_bind inside
apply_proc_raw. Previously only .ac_bind (direct .app_closure
path) handled `(lambda (a . b) ...)` correctly; macros call
closures through apply_proc_raw, so this was required to make
variadic defun/setf macros bind correctly.
* Builtin: (gensym) — writes "g%d" for an in-BSS counter, length-
prefixes the buffer, calls intern_static. Available in every
variant (not CL_FULL-gated — useful outside macros too).
* Builtin: (cadr x), (sort lst) and the let* special form from
earlier commit stay in default asm. These are Scheme staples.
* Prelude: evaluated at _start after init_builtins / rng_seed,
before the REPL. Embedded string, input state saved + restored
around the load. Defines caar, cdar, caddr, cadddr, cddr,
cdddr, cddddr, 1+, 1-, add1, sub1, square, eq? (= eqv? for
interned symbols), memq, list-ref, assq, and `case` as a macro.
cl-compat.lsp: two small changes to work under asm's single-list
`map`:
* Added cl-zip helper. Replaced two `(map (lambda (v n) (list v n))
xs ys)` sites with `(cl-zip xs ys)` — asm's builtin map accepts
only one list, and cl-loop-emit needs a parallel walk over
state-vars and new-names.
* Added explanatory comment for cddddr at the top of the shim
(already shipped).
Tests:
* make asm-test (lumbda) — 158/158 pass.
* make asm-test-gc — 158/158 pass.
* make asm-test-full — 158/158 pass on synchronous run.
* Zoë's `examples/ursa.lisp.txt` LOADS on asm/lumbda-full.
`(expt-mod 3 7 100)` = 87.
Most simple cl-loop forms work (while + do + finally, range-to,
then-accumulator).
Known open issues documented in docs/tickets/0005-asm-cl-full.md:
* cl-loop-emit produces wrong output for inputs with `simple` iters
(`(simple a 5)` → state binding dropped). Python/C return the
correct form; asm version is missing the binding. Bug surfaces
in the emit's 30+ binding let*; could not pin down in this
session. Downstream effect: `(miller-rabin n)` and similar
defuns that depend on `cl-loop repeat k for a = ... unless ...
return nil` don't produce usable expansions, so Zoë's acceptance
suite does not run end-to-end on asm/lumbda-full yet.
* examples/ursa-scheme.lsp — `factor` crashes on asm under some
random seeds (bump-allocator exhaustion on long rhoff retry
chains). Out of CL_FULL scope; tracked in same ticket.
Next steps live in ticket 0005. This commit ships the infrastructure
so the remaining work is a debugging exercise against a reproducible
minimal case, not a feature build.
|
||
|---|---|---|
| .. | ||
| tickets | ||
| asm-architecture.dot | ||
| asm-architecture.png | ||
| benchmark-ack.dot | ||
| benchmark-ack.png | ||
| benchmark-binary-size.dot | ||
| benchmark-binary-size.png | ||
| benchmark-fib.dot | ||
| benchmark-fib.png | ||
| benchmark-gc.dot | ||
| benchmark-gc.png | ||
| benchmark-speedup.dot | ||
| benchmark-speedup.png | ||
| benchmark-sumto.dot | ||
| benchmark-sumto.png | ||
| c-architecture.dot | ||
| c-architecture.png | ||
| gpu-architecture.md | ||
| jit-pipeline.dot | ||
| jit-pipeline.png | ||
| meta-gc-policy.dot | ||
| meta-gc-policy.png | ||
| python-architecture.dot | ||
| python-architecture.png | ||
| README.md | ||
lumbda Architecture Documentation
"A diagram is worth 10,000 words." — russell@unturf.com
Three implementations of the same Scheme language, sharing the same .lsp test files.
Python Implementation (lumbda.py)
3,324 lines. Bytecode compiler + stack VM + full continuations + portal.
Execution tiers:
- Tree-walker (
leval): default, handles all forms including macros - Bytecode VM (
--fast): 40 opcodes + superinstructions, 7-19x faster - Python JIT prototype: exec()-based transpilation (labeled as prototype)
Key features:
- Full multi-shot continuations via explicit frame stack
- Portal: serialize VM state to JSON, resume on another machine
- Inline cache, constant folding, peephole optimizer
- Source maps for error reporting with line numbers
- Bytecode serialization (.lspc files)
Tests: 571 unit + integration tests (tests.py)
C Implementation (c/)
8,120 lines. Tree-walker + bytecode VM + x86_64 JIT.
Execution tiers:
- Tree-walker: default, full special form support
- Bytecode VM (
--fast): matching Python's opcodes - x86_64 JIT (
--jit): 10-24x faster than CPython
Key features:
- NaN-boxed 64-bit values (zero-alloc numbers)
- Hash-map environments with parent chain + global shortcut
- Interned symbols
- Real JIT: mmap(PROT_EXEC) + raw x86_64 bytes
Tests: 76 unit + integration + JIT tests (test.c)
Assembly Implementation (asm/)
2,592 lines of GNU assembler. 13KB binary. Zero dependencies.
Design:
- No C. No libc. Only Linux syscalls (read, write, mmap, exit)
- Tag-in-low-3-bits value representation
- Bump allocator on 64MB mmap'd page
- TCO via
jmp .eval_top(never grows the stack) - 34 builtins, all special forms
Tests: 75 unit + integration + functional tests (test.sh)
JIT Pipeline (c/jit.c)
1,309 lines. Compiles Scheme AST directly to x86_64 machine code.
What gets JIT'd:
- if, cond, and, or (conditional jumps)
- +, -, *, =, <, >, <=, >= (native integer ops)
- let, let* (stack-allocated locals)
- Named-let loops (native jmp, zero call overhead)
- car, cdr, cons, null?, pair? (NaN-box pointer ops)
- Self-recursive calls (call/ret) and tail calls (jmp)
What falls back to interpreter:
- call/cc, macros, syntax-rules, quasiquote, modules
- String/vector/hash-table operations
- Any form the AST analyzer can't verify as integer-safe
Performance Summary
All benchmarks measured in-process (no startup overhead) on the same machine.
| Implementation | ack(3,4) | fib(35) | sum-to(50k) | Binary |
|---|---|---|---|---|
| C + x86_64 JIT | 0.19ms | 0.09ms | 0.55ms | 171KB |
| CPython (native) | 1.3ms | 0.006ms | 5.5ms | ~5MB |
| C interpreter | 20ms | 0.06ms | 109ms | 171KB |
| Python bytecode VM | 149ms | 0.75ms | 437ms | 3,324 lines |
| Assembly (13KB) | ~8ms* | ~0.6ms* | ~43ms* | 13KB |
*Assembly times include process startup + tokenizer + parser.
The JIT runs Scheme faster than CPython runs Python on recursive workloads: ack(3,4) is 7x faster, sum-to(50k) is 10x faster. The JIT compiles Scheme AST directly to x86_64 machine code via mmap(PROT_EXEC).
Test Coverage
943 verified assertions across all implementations:
make test-all
Python unit/integration: 571 tests
C unit/integration/JIT: 83 tests
Assembly unit/int/func: 108 tests
Shared functional: 181 tests (Python + C)
Total: 943 assertions
The language is R7RS Scheme. lumbda is the project name — a play on Common Lisp, since this is decidedly uncommon.