whitepaper: extend "feedback is all you need" across 3 scopes
Major revision reflecting current state. The original paper argued feedback (continuations) as a universal within-process primitive. This revision extends the argument to two more scopes discovered while building the cross-impl portal system: - Within process: continuations (live stack) - Across process: portals (serialized VM state) - Across impl: Scheme source itself as wire format Abstract rewritten around the three scopes. New benchmark numbers (asm cross-process round-trip 1.5 ms vs Python 260 ms — 160×). Section 4.3 "Three Scopes of Feedback" added — the key new framing. Section 7 "Portal: Feedback Across Time" rewritten to cover all three formats (S-expression portable, JSON graph-aware, binary heap-dump) with the 3×3 cross-impl matrix, cross-process benchmarks, and mismatch-case behavior table. Section 11 "Three Implementations, One Language" updated with current line counts (4,455 asm, 3,678 py), binary sizes (22 KB asm stripped, 205 KB C), and builtin count (79 asm). Section 11.2 "File I/O Parity" added — the minimal file vocabulary that reaches parity across all three implementations, including the asm port encoding trick (SPECIAL values ≥ PORT_SPECIAL_BASE). Test counts: 974 verified assertions (was 943).
This commit is contained in:
parent
2f8b7dc737
commit
5ce42a7089
2 changed files with 3191 additions and 1299 deletions
File diff suppressed because one or more lines are too long
|
|
@ -13,7 +13,7 @@
|
|||
..
|
||||
.. This software contributes to that vision by proving that feedback
|
||||
.. (the computational primitive of continuations) composes into a universal
|
||||
.. execution model: a complete Scheme in four implementations (Python,
|
||||
.. execution model: a complete Scheme in three implementations (Python,
|
||||
.. C with x86_64 JIT, pure assembly), with a bytecode VM, full
|
||||
.. continuations, machine state serialization, & an antimoad scanner
|
||||
.. that catches the defects our training data encodes,
|
||||
|
|
@ -56,13 +56,23 @@ Abstract
|
|||
|
||||
**License: AGPL-3.0-only** · This implementation, its bytecode VM, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists.
|
||||
|
||||
A programming language needs one primitive to become universal: feedback. A function that receives its own continuation can loop, branch, yield, checkpoint, resume, & migrate. Every control flow pattern reduces to a continuation captured & invoked.
|
||||
A programming language needs one primitive to become universal: feedback. A function that receives its own continuation can loop, branch, yield, checkpoint, resume, & migrate. Every control flow pattern reduces to a continuation captured & invoked. Extend feedback across time (portals) & across implementations (source-as-wire-format) and you recover the full scope of computation without new primitives.
|
||||
|
||||
**uncommonlisp** proves this by implementing R7RS Scheme in three ways: a Python bytecode VM (3,324 lines, full continuations, portal), a C implementation with x86_64 JIT (10,000 lines, 7--10x faster than CPython on recursive workloads), and a pure x86_64 assembly interpreter (3,100 lines, 13KB binary, zero dependencies). 943 tests verify identical behavior across all three. A stack-based bytecode compiler achieves 7--19x speedups over tree-walking interpretation. An explicit frame stack replaces Python's call stack, enabling tail-call optimization of arbitrary depth & full first-class continuations (``call/cc``) that support escape, upward, & multi-shot invocation. A peephole optimizer, inline cache, & constant folder tighten the generated bytecode. A portal system serializes the entire machine state (environment, continuation stack, instruction pointer) to JSON & resumes it on another machine.
|
||||
**uncommonlisp** proves this by implementing R7RS Scheme three ways:
|
||||
|
||||
- **Python bytecode VM** — 3,678 lines, full first-class continuations, JSON portal, reference implementation
|
||||
- **C interpreter + x86_64 JIT** — ~9,000 lines, 7--10× faster than CPython on recursive workloads, JSON portal
|
||||
- **Pure x86_64 assembly** — 4,455 lines, 22 KB stripped binary, zero external dependencies, 79 builtins, binary heap-dump portal
|
||||
|
||||
All three share one interchange format: **Scheme source itself**. An S-expression portal (``(define x 42)``) written by any implementation loads in any other — a 3×3 producer×consumer matrix, 9/9 cells green. The language *is* the wire protocol. This is not a property we added; it is what a parser has always made possible. We report it because most systems forget.
|
||||
|
||||
**The numbers.** The asm implementation saves + resumes 4 variables across two processes in **1.5 ms** (binary portal) or **1.6 ms** (S-expression portal). Python round-trip on the same task: 260 ms. A 160× gap from the same language, same tests, same wire format.
|
||||
|
||||
974+ verified assertions pass identically across the three implementations (571 Python unit, 131 asm, 189 shared Python+C functional, 83 C unit). Every implementation consumes every format it can reach; mismatch cases (wrong format, truncated input, missing file, corrupt header) degrade gracefully with ``#f`` or a clean error.
|
||||
|
||||
The paper further presents the EML universality proof: a single operator ``eml(x, y) = exp(x) - ln(y)`` with the constant 1 generates all elementary functions (exp, ln, arithmetic, negation, complex plane access, trigonometry). Verified numerically in Python, verified in uncommonlisp's own bytecode, & proven formally in Lean 4 with zero ``sorry``.
|
||||
|
||||
Feedback is the primitive. Continuations are the mechanism. One file is the proof.
|
||||
Feedback is the primitive. Continuations are its mechanism in time. Portals are its mechanism across time. S-expressions are its mechanism across implementations. One file is the proof — by three translations.
|
||||
|
||||
|
||||
1. The Problem: Interpreters That Cannot Feed Back
|
||||
|
|
@ -78,7 +88,7 @@ Scheme discovered this in 1975 with ``call-with-current-continuation``. But most
|
|||
2. Architecture: One File, Two Evaluators
|
||||
------------------------------------------
|
||||
|
||||
uncommonlisp implements two evaluation strategies in a single 3,261-line Python file:
|
||||
uncommonlisp implements two evaluation strategies in a single 3,678-line Python file:
|
||||
|
||||
**Tree-walking interpreter** (``leval``): The default mode. Walks the AST directly, handles all forms including macro definitions, record types, & dynamic features. Suitable for interactive development & complex metaprogramming.
|
||||
|
||||
|
|
@ -248,11 +258,32 @@ Every control flow pattern reduces to a continuation operation:
|
|||
- **Loop**: tail-call feeds the function back to itself
|
||||
- **Generator**: ``call/cc`` feeds a value out, saves a resumption point, feeds control back in on next call
|
||||
- **Exception**: ``call/cc`` feeds control to a handler registered via ``dynamic-wind``
|
||||
- **Checkpoint**: portal serializes the continuation to JSON, feeds the machine state to storage
|
||||
- **Checkpoint**: portal serializes the continuation, feeds the machine state to storage
|
||||
- **Migration**: portal deserializes on another machine, feeds the continuation back to a new VM
|
||||
- **Cross-impl exchange**: S-expression serialization feeds bindings across implementations that share no memory layout
|
||||
|
||||
One primitive. Every pattern.
|
||||
|
||||
4.3 Three Scopes of Feedback
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The single word "feedback" covers three nested scopes, each giving rise to one of our core abstractions:
|
||||
|
||||
.. table::
|
||||
:widths: 18 28 28 26
|
||||
|
||||
=============== ========================= ========================= ==========================
|
||||
Scope Mechanism Unit of travel Implementation
|
||||
=============== ========================= ========================= ==========================
|
||||
Within process ``call/cc``, tail-call continuation (live) explicit frame stack
|
||||
Across process portal (JSON or binary) VM state + continuation graph serializer
|
||||
Across impl source-as-interchange bindings (as data) the reader
|
||||
=============== ========================= ========================= ==========================
|
||||
|
||||
Within a single VM, a continuation is the unit — the whole call stack, captured & passed. Across processes on the same machine, a portal is the unit — the whole heap or VM state, serialized to a file, resumed elsewhere. Across implementations that share no binary compatibility, Scheme source itself is the unit: ``(define x 42)`` travels because every parser already knows how to receive it.
|
||||
|
||||
The same abstraction, at three scopes. No new primitive at any level. This is what "feedback is all you need" actually means once you extend it past the process boundary.
|
||||
|
||||
|
||||
5. Optimizations
|
||||
-----------------
|
||||
|
|
@ -349,50 +380,153 @@ The important comparison is not uncommonlisp vs CPython (different languages), b
|
|||
- **hash-table**: 1,000 ``set!`` + ``ref`` operations. Tests Python dict interop overhead
|
||||
|
||||
|
||||
7. Portal: Machine State Serialization
|
||||
7. Portal: Feedback Across Time
|
||||
----------------------------------------
|
||||
|
||||
Portal serializes the complete VM state (environment, continuation stack, instruction pointer, value stack) to JSON & resumes execution on another machine.
|
||||
A continuation is feedback within a process. A portal is feedback across processes. Same primitive, different scope: capture machine state, serialize it, reload it elsewhere, resume. uncommonlisp ships three portal formats with different tradeoffs & constituencies.
|
||||
|
||||
7.1 Serialization
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
.. table::
|
||||
:widths: 22 22 14 14 14 14
|
||||
|
||||
The ``_PortalSerializer`` performs graph-aware serialization:
|
||||
====================== ================================ ======== ======== ======== ===========
|
||||
Format Consumers Size Save Load Resumes
|
||||
====================== ================================ ======== ======== ======== ===========
|
||||
**S-expression** Python, C, asm — all three ~310 B 0.9 ms 0.7 ms bindings
|
||||
**JSON (graph-aware)** Python, C ~26 KB 3.1 ms — full VM + k
|
||||
**Binary heap dump** asm only ~10 KB 0.1 ms 0.1 ms asm heap
|
||||
====================== ================================ ======== ======== ======== ===========
|
||||
|
||||
- Tracks ``id(obj) → ref_id`` for object identity (handles shared references & cycles)
|
||||
Times measured on a 4-binding workload (int + list + string + fib(30) result) excluding process startup. "k" = continuation.
|
||||
|
||||
7.1 S-Expression Portal — the Portable One
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The most boring format is the most portable. An S-expression portal is a sequence of ``define`` forms:
|
||||
|
||||
::
|
||||
|
||||
;; uncommonlisp portable state
|
||||
(define my-int 42)
|
||||
(define my-list '(1 2 3 4 5 6 7 8 9 10))
|
||||
(define my-str "hello world")
|
||||
(define my-result 832040)
|
||||
|
||||
Every Scheme implementation in the world can already read this. No schema, no decoder, no version field. The file is a Scheme program; loading it is evaluating it.
|
||||
|
||||
**The key insight: the language IS the interchange format.** We did not design this. R. Kent Dybvig didn't. John McCarthy didn't. It is a structural consequence of homoiconicity: if the syntax of the language is the syntax of its data structures, then data serialization is language serialization. Portability across implementations is free — it arrives with the parser.
|
||||
|
||||
Producer side: assemble the file with ``(display ...)`` & ``(write ...)`` to an output port. Consumer side: ``(load "file.sexp")``. Both sides exist in every implementation, giving us a 3×3 matrix of valid exchanges.
|
||||
|
||||
7.2 Cross-Implementation Exchange Matrix
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table::
|
||||
:widths: 12 14 14 14
|
||||
|
||||
============ ========= ========= =========
|
||||
Producer → Python C asm
|
||||
Consumer ↓
|
||||
============ ========= ========= =========
|
||||
Python ✓ ✓ ✓
|
||||
C ✓ ✓ ✓
|
||||
asm ✓ ✓ ✓
|
||||
============ ========= ========= =========
|
||||
|
||||
9 of 9. Verified by ``tests/portal-cross-test.sh``. Same file, same semantics, regardless of which process produced it.
|
||||
|
||||
This matters because it defeats the "version lock-in" trap. If the JSON portal were the only option, a Python 3.15 producer could emit structures a C consumer couldn't parse. With S-expression portals, the only dependency is a parser that handles the subset of forms in the file. Every implementation already has one.
|
||||
|
||||
7.3 JSON Portal — Graph-Aware, Continuation-Preserving
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When you need to preserve shared references, cycles, closures, or a *live continuation*, S-expressions aren't enough. The JSON portal (Python & C) performs graph-aware serialization:
|
||||
|
||||
- Tracks ``id(obj) → ref_id`` for object identity (handles shared refs & cycles)
|
||||
- Serializes environments as chains of binding dictionaries with parent pointers
|
||||
- Serializes compiled procedures as ``CodeObj`` + captured environment
|
||||
- Serializes continuations as the full frame stack + machine state
|
||||
- Skips builtins (reconstructed from the prelude on resume)
|
||||
|
||||
7.2 Deserialization
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
Deserialization is two-pass: shell pass creates empty object shells & assigns reference IDs, fill pass populates pointers & values. This handles the closure-that-captures-itself pattern cleanly.
|
||||
|
||||
The ``_PortalDeserializer`` performs two-pass reconstruction:
|
||||
|
||||
1. **Shell pass**: Create empty object shells & assign reference IDs
|
||||
2. **Fill pass**: Populate pointers, values, & environment chains
|
||||
|
||||
This handles circular references (e.g., a closure whose environment contains a reference to the closure itself).
|
||||
|
||||
7.3 Checkpoint & Resume
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
During VM execution, ``portal-checkpoint!`` triggers at ``OP_JUMP`` & ``OP_TAIL_CALL`` instructions. When triggered:
|
||||
|
||||
1. Capture the current continuation (same mechanism as ``call/cc``)
|
||||
2. Serialize the continuation + environment to a ``.portal`` file
|
||||
3. Continue execution (or halt, depending on the use case)
|
||||
|
||||
Resumption:
|
||||
During VM execution, ``portal-checkpoint!`` triggers at ``OP_JUMP`` & ``OP_TAIL_CALL`` instructions, capturing the current continuation, serializing it to a ``.portal`` file, & continuing. Resumption restores the saved state & continues execution from the exact instruction where the checkpoint occurred.
|
||||
|
||||
::
|
||||
|
||||
python3 uncommonlisp.py --portal-resume state.portal
|
||||
|
||||
The VM deserializes the saved state, reconstructs the continuation, & resumes execution from the exact instruction where the checkpoint occurred. The computation does not need to restart from the beginning.
|
||||
The computation does not need to restart from the beginning.
|
||||
|
||||
7.4 Use Case: Distributed Primality Testing
|
||||
7.4 Binary Heap Dump — the Fast One
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The asm implementation ships a direct-dump portal: write the raw heap bytes, the ``r14`` (env) & ``r15`` (bump pointer) registers, the heap base address, & a magic header. No parsing, no encoding.
|
||||
|
||||
::
|
||||
|
||||
bi_portal_save:
|
||||
# Magic | heap_size | heap_base | r14 | r15 | reserved | heap_bytes
|
||||
|
||||
bi_portal_resume:
|
||||
# Read 48-byte header, verify magic + sanity,
|
||||
# mmap MAP_FIXED at saved heap_base so pointers stay valid,
|
||||
# read heap bytes into fixed-address region,
|
||||
# restore r14/r15.
|
||||
|
||||
Sub-millisecond save & resume. The MAP_FIXED trick is why it works across processes: because every pointer inside the heap is an absolute address, resuming at a different address would require relocation. Mapping at the same address keeps pointers live.
|
||||
|
||||
Constraints: same architecture, same binary layout, same process model. An asm binary portal written on one box resumes on another x86_64 Linux box running the same asm binary. It does not resume on a rebuilt binary — the BSS-resident symbol table ``sym_table`` is not in the heap dump, so interned symbols would need to be reinterned. That's the price of trivial serialization.
|
||||
|
||||
7.5 Cross-Process Benchmarks
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Producer process A saves state to a file; consumer process B starts fresh, loads the file, continues. Wall-clock time for both processes end-to-end, 50 iterations, same-laptop:
|
||||
|
||||
.. table::
|
||||
:widths: 36 16
|
||||
|
||||
=================================== ==========
|
||||
Pair (producer → consumer) Time/iter
|
||||
=================================== ==========
|
||||
Python → Python (sexp) 260 ms
|
||||
C → C (sexp) 6 ms
|
||||
**asm → asm (sexp)** **1.6 ms**
|
||||
**asm → asm (binary portal)** **1.5 ms**
|
||||
Python → C (sexp) 133 ms
|
||||
Python → asm (sexp) 145 ms
|
||||
C → Python (sexp) 154 ms
|
||||
C → asm (sexp) 4.1 ms
|
||||
asm → Python (sexp) 167 ms
|
||||
asm → C (sexp) 4.6 ms
|
||||
=================================== ==========
|
||||
|
||||
The asm→asm cross-process is ~160× faster than Python→Python. The binary & S-expression portals are within 10% of each other on this workload — the bottleneck is process startup, not serialization. For larger heaps the binary format pulls further ahead; for portability, S-expression always wins.
|
||||
|
||||
7.6 Mismatch Cases: Graceful Degradation
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
A usable persistence layer fails well. What happens when the consumer meets unexpected input?
|
||||
|
||||
.. table::
|
||||
:widths: 38 14 14 14
|
||||
|
||||
==================================== =========== =========== ===========
|
||||
Input Python C asm
|
||||
==================================== =========== =========== ===========
|
||||
Empty file continue continue continue
|
||||
Missing file ``file not ``error: ``#f`` +
|
||||
found: …`` …`` continue
|
||||
Truncated sexp (``(define x``) ``error: ``error: continue
|
||||
unclosed`` unclosed`` (benign)
|
||||
Binary portal fed to ``(load)`` text-file reader symbol
|
||||
error error errors
|
||||
sexp file fed to ``portal-resume`` — — ``#f``
|
||||
Truncated binary portal — — ``#f``
|
||||
==================================== =========== =========== ===========
|
||||
|
||||
All defects surfaced & fixed during benchmark development: an asm segfault on ``(define x)`` without a value (now binds to ``VOID``), an asm portal-resume that accepted short headers (now verifies ``sys_read`` returned a full 48 bytes & sanity-checks heap metadata), a Python ``file not found`` error reporting the outer script path instead of the inner missing file (now uses ``FileNotFoundError.filename``). Graceful degradation is not free; it is tested.
|
||||
|
||||
7.7 Use Case: Distributed Primality Testing
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
::
|
||||
|
|
@ -409,7 +543,7 @@ The VM deserializes the saved state, reconstructs the continuation, & resumes ex
|
|||
|
||||
(define result (prime? 1000000007))
|
||||
|
||||
Machine A starts the computation. Every 10,000 iterations, it writes a checkpoint. Machine B picks up the ``.portal`` file & continues from the last checkpoint. The computation migrates without either machine needing to know about the other. Feedback (the continuation) carries the entire execution context.
|
||||
Machine A starts the computation. Every 10,000 iterations, it writes a checkpoint. Machine B picks up the ``.portal`` file & continues from the last checkpoint. The computation migrates without either machine needing to know about the other. Feedback — the continuation — carries the entire execution context.
|
||||
|
||||
|
||||
8. The EML Universality Proof
|
||||
|
|
@ -526,7 +660,7 @@ uncommonlisp implements a near-complete R7RS-small Scheme:
|
|||
|
||||
**Standard library** (``stdlib.lsp``, 385 lines): Additional macros (``swap!``, ``fluid-let``, ``while``, ``dotimes``), utility functions, simple object system, SRFI-2/8/64 test framework.
|
||||
|
||||
**Test suite**: 529 tests covering lexing, parsing, special forms, bytecode compilation, macros (hygienic & procedural), continuations, generators, record types, modules, arithmetic, higher-order functions, error handling, & portal serialization.
|
||||
**Test suite**: 974 verified assertions covering lexing, parsing, special forms, bytecode compilation, macros (hygienic & procedural), continuations, generators, record types, modules, arithmetic, higher-order functions, error handling, portal serialization, cross-implementation portal exchange, file I/O parity, & graceful degradation on mismatched or corrupt input.
|
||||
|
||||
|
||||
10. Relationship to Companion Papers
|
||||
|
|
@ -549,7 +683,7 @@ uncommonlisp forms one piece of a larger permacomputer machine learning stack:
|
|||
uncommonlisp provides the runtime layer: a language that can checkpoint its own execution, migrate between machines, & resume from serialized state. The portal system enables distributed computation across permacomputer nodes. Categorization & feedback activities could run inside uncommonlisp's VM, with ``call/cc`` providing the state machine transitions & portal providing persistence.
|
||||
|
||||
|
||||
11. Four Implementations, One Language
|
||||
11. Three Implementations, One Language
|
||||
---------------------------------------
|
||||
|
||||
*"A diagram is worth 10,000 words."* — russell@unturf.com
|
||||
|
|
@ -557,37 +691,63 @@ uncommonlisp provides the runtime layer: a language that can checkpoint its own
|
|||
uncommonlisp implements R7RS Scheme in three implementations sharing the same ``.lsp`` test files. The language is Scheme (a dialect of Lisp, designed 1975). The project name plays on Common Lisp — this is decidedly uncommon.
|
||||
|
||||
.. table::
|
||||
:widths: 25 10 10 10 10 10 10
|
||||
:widths: 24 8 10 10 11 10 9
|
||||
|
||||
===================== ======= ========== ========= =========== ============ ==============
|
||||
Implementation Lines ack(3,4) fib(35) sum-to(50k) Binary Dependencies
|
||||
===================== ======= ========== ========= =========== ============ ==============
|
||||
C + x86_64 JIT 10,000 0.19ms 0.09ms 0.55ms 171KB libc
|
||||
CPython (reference) — 1.3ms 0.006ms 5.5ms — Python 3
|
||||
C interpreter 10,000 20ms 0.06ms 109ms 171KB libc
|
||||
Python bytecode VM 3,324 149ms 0.75ms 437ms interpreted Python 3
|
||||
x86_64 Assembly 3,100 8ms* 0.6ms* 43ms* 13KB none
|
||||
===================== ======= ========== ========= =========== ============ ==============
|
||||
===================== ====== ========== ========= =========== ========= ==============
|
||||
Implementation Lines ack(3,4) fib(35) sum-to(50k) Binary Dependencies
|
||||
===================== ====== ========== ========= =========== ========= ==============
|
||||
C + x86_64 JIT ~9k 0.19 ms 0.09 ms 0.55 ms 205 KB libc
|
||||
CPython (reference) — 1.3 ms 0.006 ms 5.5 ms — Python 3
|
||||
C interpreter ~9k 20 ms 0.06 ms 109 ms 205 KB libc
|
||||
Python bytecode VM 3,678 149 ms 0.75 ms 437 ms — Python 3
|
||||
x86_64 assembly (†) 4,455 8 ms 0.6 ms 43 ms 22 KB none
|
||||
===================== ====== ========== ========= =========== ========= ==============
|
||||
|
||||
All benchmarks measured in-process (no startup overhead) on the same machine. Assembly times (\*) include process startup + tokenizer + parser.
|
||||
All C & Python benchmarks measured in-process (no startup overhead). Assembly times (†) include full process lifetime: startup + tokenizer + parser + eval. "—" = not applicable / interpreted.
|
||||
|
||||
**The JIT runs Scheme faster than CPython runs Python.** ``ack(3,4)`` completes in 0.19ms (JIT) vs 1.3ms (CPython) — 7x faster. ``sum-to(50000)`` completes in 0.55ms (JIT) vs 5.5ms (CPython) — 10x faster. The JIT compiles Scheme AST directly to x86_64 machine code via ``mmap(PROT_EXEC)`` & raw byte emission. It handles ``if``, ``cond``, ``and``, ``or``, ``let``, named-let loops (native ``jmp`` — zero call overhead), ``car``/``cdr``/``cons``, arithmetic, comparisons, & self-recursive calls. Functions that use ``call/cc``, macros, or complex forms fall back to the interpreter.
|
||||
**The JIT runs Scheme faster than CPython runs Python.** ``ack(3,4)`` completes in 0.19 ms (JIT) vs 1.3 ms (CPython) — 7× faster. ``sum-to(50000)`` completes in 0.55 ms (JIT) vs 5.5 ms (CPython) — 10× faster. The JIT compiles Scheme AST directly to x86_64 machine code via ``mmap(PROT_EXEC)`` & raw byte emission. It handles ``if``, ``cond``, ``and``, ``or``, ``let``, named-let loops (native ``jmp`` — zero call overhead), ``car``/``cdr``/``cons``, arithmetic, comparisons, & self-recursive calls. Functions that use ``call/cc``, macros, or complex forms fall back to the interpreter.
|
||||
|
||||
**The assembly implementation proves the language runs on bare metal.** 3,100 lines of GNU assembler, 13KB stripped binary, zero external dependencies. Only four Linux syscalls (``read``, ``write``, ``mmap``, ``exit``). A bump allocator, tag-in-low-3-bits values, 71 builtins, & TCO via ``jmp``. It runs ``(ack 3 4) = 125`` & ``(fib 35) = 9227465`` correctly — 2.5x faster than the C interpreter on recursive workloads because hand-written assembly avoids C's function call overhead for eval dispatch.
|
||||
**The assembly implementation proves the language runs on bare metal.** 4,455 lines of GNU assembler, **22 KB stripped binary**, zero external dependencies. Only seven Linux syscalls (``read``, ``write``, ``open``, ``close``, ``lseek``, ``mmap``, ``munmap``, ``exit``). A bump allocator, tag-in-low-3-bits values, **79 builtins** (including ``load``, ``open-output-file``, ``close-port``, ``port?``, port-aware ``display``/``write``/``newline``, ``write-file``, ``file->string``, ``portal-save``, ``portal-resume``), & TCO via ``jmp``. It runs ``(ack 3 4) = 125`` & ``(fib 35) = 9227465`` correctly — 2.5× faster than the C interpreter on recursive workloads because hand-written assembly avoids C's function call overhead for eval dispatch.
|
||||
|
||||
**The bytecode VM delivers 7--19x speedup over tree-walking.** The Python implementation compiles Scheme to 40 opcodes (plus 20 specialized & 5 superinstructions), executed on an explicit frame stack with inline caching, constant folding, & peephole optimization. Full first-class continuations (multi-shot, upward) enable generators, coroutines, & machine state migration via portal.
|
||||
**The bytecode VM delivers 7--19× speedup over tree-walking.** The Python implementation compiles Scheme to 40 opcodes (plus 20 specialized & 5 superinstructions), executed on an explicit frame stack with inline caching, constant folding, & peephole optimization. Full first-class continuations (multi-shot, upward) enable generators, coroutines, & machine state migration via portal.
|
||||
|
||||
11.1 Test Coverage
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
943 verified assertions across all implementations:
|
||||
**974 verified assertions** across all implementations, all green under ``make test-all``:
|
||||
|
||||
- Python unit + integration: 571 tests (``tests.py``)
|
||||
- C unit + integration + JIT + continuations + portal: 83 tests (``c/test.c``)
|
||||
- Assembly unit + integration + functional: 108 tests (``asm/test.sh``)
|
||||
- Shared functional: 181 tests (``tests/functional.lsp``, runs in Python & C)
|
||||
- Python unit + integration: **571 tests** (``tests.py``)
|
||||
- C unit + integration + JIT + continuations + portal: **83 tests** (``c/test.c``)
|
||||
- Assembly unit + integration + functional: **131 tests** (``asm/test.sh``)
|
||||
- Shared functional (same ``.lsp`` in Python + C): **189 tests** (``tests/functional.lsp``)
|
||||
|
||||
All pass via ``make test-all``.
|
||||
The shared functional suite matters: it runs byte-identical Scheme source through two different runtimes & compares output. Python & C agree 189 times per run. When they disagree, that tells us something specific & actionable.
|
||||
|
||||
Cross-implementation portal exchange is verified separately: ``tests/portal-cross-test.sh`` runs 9 producer×consumer combinations. All 9 green.
|
||||
|
||||
11.2 File I/O Parity
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
All three implementations share the same minimal file I/O vocabulary — discovered necessary while building the cross-impl portal:
|
||||
|
||||
.. table::
|
||||
:widths: 30 20 12 12 12
|
||||
|
||||
============================= ===================== ====== ===== =====
|
||||
Builtin Semantics Py C asm
|
||||
============================= ===================== ====== ===== =====
|
||||
``(load "path")`` read + eval forms ✓ ✓ ✓
|
||||
``(open-output-file p)`` open, return port ✓ ✓ ✓
|
||||
``(close-port p)`` close port ✓ ✓ ✓
|
||||
``(port? x)`` port predicate ✓ ✓ ✓
|
||||
``(display v [port])`` to stdout or port ✓ ✓ ✓
|
||||
``(write v [port])`` quoted / readable ✓ ✓ ✓
|
||||
``(newline [port])`` line break ✓ ✓ ✓
|
||||
``(write-file path str)`` bytes in, #t/#f ✓ ✓ ✓
|
||||
``(file->string path)`` read whole file ✓ ✓ ✓
|
||||
============================= ===================== ====== ===== =====
|
||||
|
||||
The asm port representation deserves a note. Because all three tag bits are consumed by the existing value types (int/pair/sym/closure/builtin/special/string/vector), we encode ports as ``SPECIAL`` values ≥ 1000: ``port_val = ((PORT_SPECIAL_BASE + fd) << 3) | TAG_SPECIAL``. Extraction is ``fd = (val >> 3) - PORT_SPECIAL_BASE``. ``port?`` is a range check. No tag expansion, no heap object, no allocator pressure — the port is the file descriptor, wrapped.
|
||||
|
||||
|
||||
12. MOAD Audit: Fixing What We Built
|
||||
|
|
@ -753,8 +913,9 @@ License
|
|||
|
||||
This software contributes to that vision by proving that feedback
|
||||
(the computational primitive of continuations) composes into a universal
|
||||
execution model. Four implementations, one language, 836 tests, 12,774
|
||||
lines. Code is seeds to sprout on any abandoned technology.
|
||||
execution model, extending across implementations through Scheme source
|
||||
itself as the wire format. Three implementations, one language,
|
||||
974 tests, ~17,000 lines. Code is seeds to sprout on any abandoned technology.
|
||||
|
||||
Learn more: https://www.permacomputer.com
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue