whitepaper: §11.4 S-expressions over sockets — RPC + REPL + chains
Adds the cross-runtime chain story to the abstract and a new §11.4
"S-expressions over Sockets: RPC, REPL, and Chains" documenting:
- read-from-string + eval + symbol->string as the primitives that
close the loop (asm gets these as native builtins in ~100 bytes).
- Whitelisted RPC (examples/rpc-server.lsp, safe dispatch) vs full
remote REPL (examples/repl-server.lsp, persistent global env,
DANGER). 9/9 server×client matrix green across Python/C/asm.
- Transparent byte-forwarding relay (examples/rpc-relay.lsp). Chains
compose naturally because the envelope is Scheme source: Python
client → C relay → asm backend through zero format translation.
Measured chain table (200 ping requests):
Py → asm (direct) 2,061 rps
Py → C → asm 1,234 rps (+605 µs/hop)
Py → Py → C → asm 766 rps
asm → Py → C → asm 796 rps
The existing §11.4 "heap-snapshot" moves to §11.5; no other section
numbers shift.
Abstract + implementation numbers refreshed:
- Python source: 3,743 lines (was 3,678)
- Asm source: 4,968 lines (was 4,527)
- Asm binary: 22 KB stripped (was 45 KB unstripped — the old
number conflated stripped vs unstripped)
- Asm builtins: 91 (was 87), now includes eval, read-from-string,
symbol->string
- Asm syscalls: 14 (adds clock_gettime)
- vs busybox: 96× smaller (was 46× on the unstripped basis)
- vs python3: 360× smaller (was 176×)
The closing "one file is the proof — by three translations" remains
intact; the addition is that the interchange now works over sockets
too, not just files.
This commit is contained in:
parent
ccf86e3c3f
commit
ac2a742bd5
2 changed files with 1032 additions and 728 deletions
File diff suppressed because one or more lines are too long
|
|
@ -60,15 +60,17 @@ A programming language needs one primitive to become universal: feedback. A func
|
|||
|
||||
**uncommonlisp** proves this by implementing R7RS Scheme three ways:
|
||||
|
||||
- **Python bytecode VM** — 3,678 lines, full first-class continuations, JSON portal, reference implementation
|
||||
- **Python bytecode VM** — 3,743 lines, full first-class continuations, JSON portal, reference implementation
|
||||
- **C interpreter + x86_64 JIT** — ~9,000 lines, 7--10× faster than CPython on recursive workloads, JSON portal
|
||||
- **Pure x86_64 assembly** — 4,527 lines, 45 KB stripped binary, zero external dependencies, **87 builtins including a full TCP stack**, binary heap-dump portal
|
||||
- **Pure x86_64 assembly** — 4,968 lines, 22 KB stripped binary, zero external dependencies, **91 builtins including a full TCP stack, `eval`, and `read-from-string`**, binary heap-dump portal
|
||||
|
||||
All three share one interchange format: **Scheme source itself**. An S-expression portal (``(define x 42)``) written by any implementation loads in any other — a 3×3 producer×consumer matrix, 9/9 cells green. The language *is* the wire protocol. This is not a property we added; it is what a parser has always made possible. We report it because most systems forget.
|
||||
|
||||
**The numbers.** The asm implementation saves + resumes 4 variables across two processes in **1.5 ms** (binary portal) or **1.6 ms** (S-expression portal). Python round-trip on the same task: 260 ms. A 160× gap from the same language, same tests, same wire format.
|
||||
|
||||
**Web server, same story.** The asm implementation ships a TCP stack (``tcp-listen``, ``tcp-accept``, ``tcp-connect``, ``tcp-recv``, ``tcp-send``, ``tcp-close``) and a portable 70-line HTTP/1.0 handler in Scheme. The same ``.lsp`` runs identically in all three impls. Asm server + asm client: **2,994 req/s** on a 1 KB body, 45 KB binary, zero libc, seven Linux syscalls plus the socket family. For comparison: busybox httpd (2.1 MB) and ``python3 -m http.server`` (8 MB interpreter) hit the same benchmark at 382 req/s under a curl client and ~2,400 req/s under an in-process client.
|
||||
**Web server, same story.** The asm implementation ships a TCP stack (``tcp-listen``, ``tcp-accept``, ``tcp-connect``, ``tcp-recv``, ``tcp-send``, ``tcp-close``) and a portable 70-line HTTP/1.0 handler in Scheme. The same ``.lsp`` runs identically in all three impls. Asm server + asm client: **2,994 req/s** on a 1 KB body, 22 KB binary, zero libc, seven Linux syscalls plus the socket family. For comparison: busybox httpd (2.1 MB) and ``python3 -m http.server`` (8 MB interpreter) hit the same benchmark at 382 req/s under a curl client and ~2,400 req/s under an in-process client.
|
||||
|
||||
**S-expressions over sockets.** The wire protocol for a 90-line RPC server is one Scheme form per connection. With ``read-from-string`` and ``eval`` added to all three impls (89 bytes of asm for ``eval``, a 20-line asm reader swap for ``read-from-string``), a transparent byte-forwarding relay composes arbitrary chains: a Python client can reach an asm backend through a C relay and a Python relay, four runtimes strung together without any format translation between hops. Each relay adds ~650 µs/request on the same laptop. The language is the envelope.
|
||||
|
||||
975 verified assertions pass identically across the three implementations (571 Python unit, 132 asm, 189 shared Python+C functional, 83 C unit). Every implementation consumes every format it can reach; mismatch cases (wrong format, truncated input, missing file, corrupt header) degrade gracefully with ``#f`` or a clean error.
|
||||
|
||||
|
|
@ -704,14 +706,14 @@ uncommonlisp implements R7RS Scheme in three implementations sharing the same ``
|
|||
CPython (reference) — 1.3 ms 0.006 ms 5.5 ms — Python 3
|
||||
C interpreter ~9k 20 ms 0.06 ms 109 ms 205 KB libc
|
||||
Python bytecode VM 3,678 149 ms 0.75 ms 437 ms — Python 3
|
||||
x86_64 assembly (†) 4,527 8 ms 0.6 ms 43 ms 45 KB none
|
||||
x86_64 assembly (†) 4,968 8 ms 0.6 ms 43 ms 22 KB none
|
||||
===================== ====== ========== ========= =========== ========= ==============
|
||||
|
||||
All C & Python benchmarks measured in-process (no startup overhead). Assembly times (†) include full process lifetime: startup + tokenizer + parser + eval. "—" = not applicable / interpreted.
|
||||
|
||||
**The JIT runs Scheme faster than CPython runs Python.** ``ack(3,4)`` completes in 0.19 ms (JIT) vs 1.3 ms (CPython) — 7× faster. ``sum-to(50000)`` completes in 0.55 ms (JIT) vs 5.5 ms (CPython) — 10× faster. The JIT compiles Scheme AST directly to x86_64 machine code via ``mmap(PROT_EXEC)`` & raw byte emission. It handles ``if``, ``cond``, ``and``, ``or``, ``let``, named-let loops (native ``jmp`` — zero call overhead), ``car``/``cdr``/``cons``, arithmetic, comparisons, & self-recursive calls. Functions that use ``call/cc``, macros, or complex forms fall back to the interpreter.
|
||||
|
||||
**The assembly implementation proves the language runs on bare metal.** 4,527 lines of GNU assembler, **45 KB stripped binary**, zero external dependencies. Thirteen Linux syscalls (``read``, ``write``, ``open``, ``close``, ``lseek``, ``mmap``, ``munmap``, ``socket``, ``connect``, ``accept``, ``bind``, ``listen``, ``clock_gettime``, ``exit``) — no libc, no stdlib. A bump allocator with ``heap-snapshot``/``heap-restore`` arena primitives, tag-in-low-3-bits values, **87 builtins** (including ``load``, ports, ``write-file``, ``file->string``, ``portal-save``, ``portal-resume``, the six ``tcp-*`` socket primitives, ``current-time-ms``), & TCO via ``jmp``. It runs ``(ack 3 4) = 125`` & ``(fib 35) = 9227465`` correctly, serves HTTP at **2,994 req/s**, and survives indefinitely with flat O(1) memory when the programmer uses the snapshot/restore arena in a per-request loop.
|
||||
**The assembly implementation proves the language runs on bare metal.** 4,968 lines of GNU assembler, **22 KB stripped binary**, zero external dependencies. Fourteen Linux syscalls (``read``, ``write``, ``open``, ``close``, ``lseek``, ``mmap``, ``munmap``, ``socket``, ``connect``, ``accept``, ``bind``, ``listen``, ``clock_gettime``, ``exit``) — no libc, no stdlib. A bump allocator with ``heap-snapshot``/``heap-restore`` arena primitives, tag-in-low-3-bits values, **91 builtins** (including ``load``, ports, ``write-file``, ``file->string``, ``portal-save``, ``portal-resume``, the six ``tcp-*`` socket primitives, ``read-from-string``, ``eval``, ``symbol->string``, ``current-time-ms``), & TCO via ``jmp``. It runs ``(ack 3 4) = 125`` & ``(fib 35) = 9227465`` correctly, serves HTTP at **2,994 req/s**, and survives indefinitely with flat O(1) memory when the programmer uses the snapshot/restore arena in a per-request loop.
|
||||
|
||||
**The bytecode VM delivers 7--19× speedup over tree-walking.** The Python implementation compiles Scheme to 40 opcodes (plus 20 specialized & 5 superinstructions), executed on an explicit frame stack with inline caching, constant folding, & peephole optimization. Full first-class continuations (multi-shot, upward) enable generators, coroutines, & machine state migration via portal.
|
||||
|
||||
|
|
@ -797,13 +799,47 @@ The companion ``examples/http-client-bench.lsp`` is a 45-line load generator usi
|
|||
asm server ← C client 2,500 —
|
||||
=============================== ============== ==============
|
||||
|
||||
(The asm-server / asm-client pair is the fastest cell here — 2,994 req/s from a 45 KB binary, served & driven by the same 45 KB binary.)
|
||||
(The asm-server / asm-client pair is the fastest cell here — 2,994 req/s from a 22 KB binary, served & driven by the same 22 KB binary.)
|
||||
|
||||
For external comparison: ``python3 -m http.server`` and busybox httpd both land around 382 req/s under a curl client on the same machine. When driven by an in-process client they hit the same ceiling as our servers — the bottleneck has always been the client fork/exec, not the server.
|
||||
|
||||
**What's remarkable is not the speed.** It is that a 45 KB binary with zero libc dependency and seven canonical Linux syscalls plus the socket family runs HTTP as fast as anything else on the machine, with a protocol handler written in portable Scheme that runs byte-for-byte in all three runtimes. The asm binary is **46× smaller than busybox httpd** (2.1 MB) and **176× smaller than the Python interpreter** alone (8 MB).
|
||||
**What's remarkable is not the speed.** It is that a 22 KB binary with zero libc dependency and seven canonical Linux syscalls plus the socket family runs HTTP as fast as anything else on the machine, with a protocol handler written in portable Scheme that runs byte-for-byte in all three runtimes. The asm binary is **96× smaller than busybox httpd** (2.1 MB) and **360× smaller than the Python interpreter** alone (8 MB).
|
||||
|
||||
11.4 heap-snapshot: The Arena Escape Hatch
|
||||
11.4 S-expressions over Sockets: RPC, REPL, and Chains
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The HTTP example sends request-line bytes and response-line bytes. Those bytes don't have to be HTTP. If both sides of the wire speak Scheme, the wire protocol can be Scheme source itself — the reader is already the parser you need. Two new builtins close the loop in all three impls:
|
||||
|
||||
- ``(read-from-string s)`` — parse one S-expression from a string, return the value
|
||||
- ``(eval expr)`` — evaluate a Scheme value in the global env (Python and C already had this as a special form; asm gains it as an 89-byte builtin)
|
||||
|
||||
Two server patterns emerge:
|
||||
|
||||
**Whitelisted RPC** (``examples/rpc-server.lsp``, 90 lines). The server reads a request sexp, dispatches by car on a closed set (``ping``, ``add``, ``mul``, ``fib``, ``echo``), never calls ``eval`` on client input. Safe by construction. Uses ``heap-snapshot``/``heap-restore`` for O(1) memory on asm.
|
||||
|
||||
**Full remote REPL** (``examples/repl-server.lsp``, 70 lines). The server reads a request sexp and passes it straight to ``eval``. Persistent global env across connections; ``(define x 42)`` from one call is visible from the next. DANGER: any reachable client can run arbitrary Scheme in-process. Deliberately does not use ``heap-snapshot`` because remote ``define`` adds bindings past any snapshot point; the ``ulimit -v`` safety cap (512 MB) backstops the leak.
|
||||
|
||||
Both patterns run byte-identically in Python, C, and asm. The 3×3 server×client matrix is 9/9 green — any runtime can host either side.
|
||||
|
||||
**Chains: relays across runtimes.** A transparent relay (``examples/rpc-relay.lsp``, 50 lines) accepts a connection, forwards the request bytes to a backend without parsing, relays the reply back. Because the envelope is Scheme source and the relay never opens it, chains of arbitrary runtimes compose naturally:
|
||||
|
||||
.. table::
|
||||
:widths: 44 14 14
|
||||
|
||||
============================================== ========== =============
|
||||
Chain (200 ping requests) Requests/s Per-hop cost
|
||||
============================================== ========== =============
|
||||
Py client → asm backend (1 hop) 2,061 baseline
|
||||
Py client → C relay → asm backend (2 hops) 1,234 +605 µs/req
|
||||
Py client → Py relay → C relay → asm (3 hops) 766 +705 µs/req
|
||||
asm client → Py relay → C relay → asm (3 hops) 796 —
|
||||
============================================== ========== =============
|
||||
|
||||
Each relay hop costs ~650 µs (one full TCP round-trip + context switches on the same host, no actual parsing work). The relay never allocates anything beyond a transient buffer; on asm it uses ``heap-snapshot``/``heap-restore`` to keep memory flat under load. Four runtimes strung together through two relay machines, the same ``.lsp`` on every hop.
|
||||
|
||||
The result is not a performance story — it is a composition story. S-expressions are the envelope and the payload. A 22 KB binary can be a backend, a relay, a client, or any point in a chain; the protocol needs no separate definition because the protocol IS the language.
|
||||
|
||||
11.5 heap-snapshot: The Arena Escape Hatch
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The asm bump allocator has no GC. Every ``string-append``, ``tcp-recv``, ``make-pair``, or similar per-request allocation grows r15. Over a long-running server that is an unbounded leak — an incident on 2026-04-16 drove an asm server to 19.3 GB RSS before being killed.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue