russell@unturf.com
78ff89fb8b
gpu-worker.lsp gains a cuda-sim-ops-bin op handler that spawns demo_ops from www.foxhop.net/ecdsa/cuda via spawn-process-stdio, drains stdout, & parses our (cuda-sim-result ...) portal back. Each call now emits two log lines: ;;; bend RECV cuda-sim-ops-bin ops=PATH n-batches=N t-ms=... ;;; bend DONE cuda-sim-ops-bin n-batches=N wall-ms=W cpu-ms=C gpu-ms=G mismatches=0 gpu/cpu=R so we can tell how fast bend jobs run on CPU vs GPU per call. CLAUDE.md & www/index.html mention this integration is now live end-to-end across our fleet.
147 lines
10 KiB
HTML
147 lines
10 KiB
HTML
<!doctype html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||
<title>Lumbda — feedback as a primitive</title>
|
||
<meta name="description" content="Lumbda — a Lisp/Scheme-derived language with four execution backends (Python, Python bytecode VM, C + x86_64 JIT, pure x86_64 GNU asm), full first-class continuations, portal-based state migration, and a formally verified universality proof.">
|
||
<link rel="stylesheet" href="style.css">
|
||
<script src="https://uncloseai.com/uncloseai.js" type="module"></script>
|
||
</head>
|
||
<body>
|
||
|
||
<header>
|
||
<h1 aria-label="lumbda.">lumbda<span class="period" aria-hidden="true">.</span></h1>
|
||
<img class="lambda-mark" src="lumbda-logo-green.png?v=2" alt="" aria-hidden="true">
|
||
<p class="tagline">feedback as a primitive</p>
|
||
</header>
|
||
|
||
<main>
|
||
|
||
<section id="what">
|
||
<p class="lead">
|
||
Lumbda names a Lisp/Scheme-derived language carrying four independently implemented execution backends — one surface syntax, one test suite. A Python tree-walker with an optional bytecode VM, a C implementation that adds an x86_64 JIT, and a pure x86_64 GNU asm interpreter (~6,600 lines, ~23 KB stripped, zero external dependencies). Every backend runs a shared <code>.lsp</code> source byte-identically, with full first-class continuations, exact rationals, records, and hygienic macros.
|
||
</p>
|
||
</section>
|
||
|
||
<section id="quick">
|
||
<h2>Get it</h2>
|
||
<pre><code>git clone https://git.unturf.com/engineering/unturf/lumbda.git
|
||
cd lumbda
|
||
make test-all</code></pre>
|
||
<p>Run a program in any tier:</p>
|
||
<pre><code>python3 lumbda.py --fast examples/fibonacci.lsp
|
||
./c/lumbda examples/fibonacci.lsp
|
||
./asm/lumbda < examples/fibonacci.lsp</code></pre>
|
||
</section>
|
||
|
||
<section id="tiers">
|
||
<h2>Four tiers, one language</h2>
|
||
<table>
|
||
<thead><tr><th>Tier</th><th>Lines</th><th>Binary</th><th>What a tier buys</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Python interpreter + bytecode VM</td><td>3,743</td><td>—</td><td>REPL hackability, debugging, reference</td></tr>
|
||
<tr><td>C tree-walker + bytecode VM</td><td>9,164</td><td>~215 KB</td><td>deep recursion, production workloads</td></tr>
|
||
<tr><td>C + x86_64 JIT</td><td>+patches</td><td>~215 KB</td><td>7–10× faster than CPython on recursive workloads</td></tr>
|
||
<tr><td>Pure x86_64 GNU asm</td><td>6,645</td><td>~23 KB</td><td>zero-dependency boot, auditability, embedded</td></tr>
|
||
<tr><td>GNU asm + naive mark-sweep GC + meta-GC arena</td><td>(same source, <code>GC_NAIVE=1</code>)</td><td>~27 KB</td><td>bounded memory without manual arena discipline</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</section>
|
||
|
||
<section id="portal">
|
||
<h2>Portal: feedback across time</h2>
|
||
<p>A continuation carries feedback within a process. A portal carries feedback across processes. Same primitive, different scope: capture machine state, serialize, reload elsewhere, resume. Lumbda ships three portal formats with distinct trade-offs:</p>
|
||
<ul>
|
||
<li><strong>S-expression portal</strong> — Scheme source as a wire protocol. 16 of 16 producer×consumer cells green across Python, C, asm no-GC, and asm GC.</li>
|
||
<li><strong>JSON portal</strong> — graph-aware, preserves closures and live continuations (Python, C).</li>
|
||
<li><strong>Binary heap dump</strong> — asm only. 1.5 ms save+resume between two processes.</li>
|
||
</ul>
|
||
</section>
|
||
|
||
<section id="bend">
|
||
<h2>bend: dispatch to a GPU without rewriting your code</h2>
|
||
<p>Wrap any registered GPU-able form in <code>(bend …)</code> and lumbda decides per call whether to run it locally or ship it to a CUDA worker over our wire protocol. The decision uses a cost estimator on the argument shape, not the operation name. Tiny inputs stay local; heavy inputs bend to a worker that holds a warm CUDA context across requests.</p>
|
||
|
||
<h3>Start a GPU worker</h3>
|
||
<pre><code># On any host with nvcc + a CUDA-capable GPU:
|
||
make gpu-worker
|
||
# → builds examples/cuda-fanout/shake256-fanout
|
||
# → builds the C tier (~10× faster wire orchestration than Python)
|
||
# → launches gpu-worker.lsp on port 9091
|
||
|
||
# Override tier or port:
|
||
make gpu-worker LUMBDA=python PORT=9001 # easier debugging
|
||
make gpu-worker LUMBDA=asm # smallest footprint</code></pre>
|
||
|
||
<h3>Call it from any tier</h3>
|
||
<pre><code>;; bend works on every tier — Python, C, asm — through the same
|
||
;; tcp-* + portal primitives lumbda already ships.
|
||
(load "examples/cuda-fanout/wire.lsp")
|
||
(load "examples/cuda-fanout/bend.lsp")
|
||
(load "examples/cuda-fanout/bend-macros.lsp") ; Python/C only — asm uses bend-call
|
||
|
||
;; Tiny — cost below threshold, evaluates locally
|
||
(bend (cuda-shake-fanout '("00" "01" "deadbeef") 32))
|
||
|
||
;; Heavy — cost above threshold, ships to the GPU worker
|
||
(bend (cuda-shake-fanout one-million-inputs 32))</code></pre>
|
||
|
||
<p>On a single RTX 3090 with a warm daemon, fan-out matched <code>hashlib.shake_256</code> byte-for-byte and won by 1.5–10× across the workloads we measured. Below the break-even (~100 MB of bulk hash work) host CPU stays faster — the cost estimator picks correctly.</p>
|
||
<h3>Tier choice for the worker host</h3>
|
||
<p>The CUDA kernel runs inside the leaf binary; what the tier choice affects is wire orchestration. The wire has two modes: <strong>S-expression text</strong> (the default — hex strings inside a Scheme list) and <strong>binary</strong> (magic <code>BSHK</code> header + raw bytes, identical layout to the daemon's binary portal). Binary mode bypasses S-expression parsing entirely:</p>
|
||
<table>
|
||
<thead><tr><th>workload</th><th>Py S-exp</th><th>Py binary</th><th>C S-exp</th><th>C binary</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>100 × 16 B</td><td>3.43 ms</td><td>0.74 ms</td><td>0.40 ms</td><td><strong>0.15 ms</strong></td></tr>
|
||
<tr><td>1k × 16 B</td><td>23.24 ms</td><td>0.76 ms</td><td>2.77 ms</td><td><strong>0.22 ms</strong></td></tr>
|
||
<tr><td>10k × 16 B</td><td>218.82 ms</td><td>1.27 ms</td><td>CLIFF</td><td><strong>0.88 ms</strong></td></tr>
|
||
<tr><td>100k × 16 B</td><td>2,219 ms</td><td>10.18 ms</td><td>CLIFF</td><td><strong>10.35 ms</strong></td></tr>
|
||
<tr><td>1M × 16 B</td><td>23,811 ms</td><td>159 ms</td><td>CLIFF</td><td><strong>157 ms</strong></td></tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>Binary mode wins by 30–200× over S-expression mode at scale.</strong> At 1 M × 16 B inputs, C tier binary is <strong>157 ms</strong> end-to-end versus 23,811 ms for the S-exp path — a 150× speedup. The CUDA kernel itself on this 3090 runs in ~47 ms; binary wire adds ~110 ms of file I/O + framing on top, a 2.5× multiplier instead of the 500× multiplier the S-exp path imposed.</p>
|
||
<p>Critically, at huge workloads <strong>bend now beats host hashlib</strong>: host SHAKE256 over 1 M tiny inputs is ~2 s on a single Python core; bend via binary worker is 157 ms — a 12× speedup of host. The cost estimator in <code>bend.lsp</code> should be updated to know about the binary path so the routing decision picks GPU at this scale instead of staying local.</p>
|
||
<p>Binary mode lives behind the <code>BSHK</code> magic byte in the wire payload. S-expression callers see no change; binary callers prepend the magic and send raw bytes. See <code>examples/cuda-fanout/bench_tiers.py --binary</code> for the protocol implementation.</p>
|
||
<p>Three tiers, three operating points (S-expression mode): C tier wins at small & medium scales (8× faster than Python); asm tier hits 0.21 ms at very small inputs (~30% behind C, 6× faster than Python, 70 KB statically linked, zero libc); Python tier scales linearly (~22 µs per input) all the way through 1 M inputs but runs slowly on a single core. The S-exp CLIFFs at 10k (C) and 1k (asm) are tier-internal reader limits — binary mode bypasses them entirely.</p>
|
||
<p>The CUDA toolchain stays isolated to the leaf binary the worker spawns. No tier links libcudart; no tier requires nvcc at build time. Asm tier hosts workers through hand-written <code>pipe2 + fork + execve</code> syscalls — no libc anywhere on the chain.</p>
|
||
<p>See <a href="https://git.unturf.com/engineering/unturf/lumbda/-/blob/master/examples/cuda-fanout/README.md">examples/cuda-fanout/</a> for the wire contract, daemon protocol, bench data, and per-tier integration sketch.</p>
|
||
<h3>Real workload — ecdsafail search</h3>
|
||
<p>Beyond hash fan-out, bend now dispatches quantum-reversible circuit scoring for our secp256k1 point-addition challenge work at <a href="https://www.foxhop.net/ecdsa.html">foxhop.net/ecdsa</a>. Lumbda emits an upstream-format <code>ops.bin</code> from a Phase B Roetteler 12-step circuit, calls <code>(bend!-call '(cuda-sim-ops-bin path 141))</code>, & receives Σ Clifford / Σ Toffoli totals back from a GPU worker over our binary wire — same cross-tier validation, same byte-identical portal contract that the hash demo proves. Search loops on any host tier ship candidate scoring to whichever fleet node holds a warm CUDA context.</p>
|
||
</section>
|
||
|
||
<section id="proof">
|
||
<h2>EML universality proof</h2>
|
||
<p>A single operator <code>eml(x, y) = exp(x) − ln(y)</code> with a constant 1 generates all elementary functions: <code>exp</code>, <code>ln</code>, arithmetic, negation, complex-plane access, trigonometry. Verified numerically in Python, verified in Lumbda's own bytecode, proven formally in Lean 4 with zero <code>sorry</code>. Lumbda's native symbolic-rewrite checker runs five theorems in 46 ms cold or 7 ms cached — roughly 16× faster than Lean's cold rebuild on identical hardware.</p>
|
||
</section>
|
||
|
||
<section id="doc">
|
||
<h2>Whitepaper</h2>
|
||
<p>Full language reference, tier-by-tier architecture, benchmarks, meta-GC design, universality proof.</p>
|
||
<p>
|
||
<a class="cta" href="lumbda-whitepaper.html">Read in browser (HTML)</a>
|
||
<a class="cta" href="lumbda-whitepaper.pdf">Download PDF (~2.7 MB)</a>
|
||
</p>
|
||
</section>
|
||
|
||
<section id="license">
|
||
<h2>License</h2>
|
||
<p>AGPL-3.0-only. Public domain for whitepapers, proofs, and disclosures through <a href="https://undefect.com">undefect.com</a>. Companion to <a href="https://unturf.com">unturf.com</a>’s permacomputer project.</p>
|
||
</section>
|
||
|
||
</main>
|
||
|
||
<footer>
|
||
<p>
|
||
<a href="https://git.unturf.com/engineering/unturf/lumbda">source</a>
|
||
·
|
||
<a href="lumbda-whitepaper.html">whitepaper (HTML)</a>
|
||
·
|
||
<a href="lumbda-whitepaper.pdf">whitepaper (PDF)</a>
|
||
·
|
||
<a href="https://unturf.com">unturf.com</a>
|
||
</p>
|
||
</footer>
|
||
|
||
</body>
|
||
</html>
|