Six-step proof that https://wedgewack.org/ursa.lisp.txt runs in lumbda
with only the documented minimal annotations — no semantic rewrites,
no algorithm changes. New make target `prove-ursa-runs` wires it.
Steps:
1. Fetch /robots.txt; abort if it disallows /ursa.lisp.txt.
2. Fetch the source (209 lines, sha256 recorded in output).
3. Apply the four character-level substitutions sed'd from the
documented annotations:
(loop → (cl-loop (call form only — clause keyword stays)
(when → (cl-when (call form only)
(random → (random-int
&key → &optional
Prepend (load "cl-compat.lsp").
4. Verify that examples/ursa.lisp.txt's defuns are exactly Zoë's
defuns minus {rho, factor, digits} — the three that depend on
CL features (adjustable arrays, defgeneric/defmethod) out of
ticket 0004's scope. No extra edits anywhere.
5. Run tests/ursa.lsp (which uses the same function bodies Zoë
wrote) on Python + C + asm-full; expect 28/28 passing on each.
6. Build a "Zoë's live source + 10-line stubs" file — no-op
defgeneric/defmethod, 'unshimmed returns for make-array / sbit /
vector-push-extend / vector-pop / fill-pointer, identity coerce,
naive integer-length — and spot-check seven answers on asm-full:
expt-mod 3 7 100 = 87
primep 97 = 97
primep 100 = #f
mersenne 7 = 127
ll-primep 13 = #t
ll-primep 11 = #f
repunit-value 5 = 31
Any missing line fails the proof.
Sed-subset nuance: `(loop ` / `(when ` with an open-paren prefix
matches the call-form usage we want to rewrite. Bare `when` that
appears as a cl-loop clause keyword (no open paren before it) is
left unchanged — that's the macro's own reserved word. Same for
loop.
Usage:
make prove-ursa-runs (network required — live fetch + spot-check)
make zoe-favorites-test (offline; uses the committed examples/)
Closes the remaining asm-side gaps from ticket 0005's follow-up
discussion. Every test in tests/cl-compat.lsp and tests/ursa.lsp
now runs unmodified on default asm (Scheme port) and asm-full (full
CL path) — no more commented-out tests or shim syntax.
Landed (all in default asm — useful beyond cl-compat):
* (values . xs) / (call-with-values producer consumer). values
packs a tagged pair (mval_marker . xs) when multiple; a lone arg
passes through unchanged so legacy single-value code is
undisturbed. call-with-values invokes the producer, destructures
the multi-value packet if present, applies consumer positionally.
The marker is a gensymed symbol interned once at init, so no
user-constructed pair can masquerade as a multi-value packet.
* (exit [code]) builtin. Default code is 0 when called with no
args. Passes through to the SYS_EXIT syscall.
* #(...) vector literal in the reader. .sr_hash now dispatches on
'(' as a vector literal alongside 't' and 'f'. list_to_vector_
reader is a standalone helper callable from the reader (separate
from bi_listtovec which uses the GETARG builtin convention).
Matches R7RS vector literal syntax. Existing vector builtins
already handled construction; this just teaches the reader.
* deep_equal extended to vectors. equal? now descends into vectors
(length + elementwise recursive compare), matching R7RS.
Previously only strings and pairs were handled; vectors fell
through to shallow pointer compare which only matched identical
heap objects.
Test file reverts (picking up the new capabilities):
* tests/cl-compat.lsp — multiple-value-bind test restored
(previously commented out because asm lacked values /
call-with-values).
* tests/ursa-scheme.lsp — #(1 0 1 0 1 0) literal restored
(previously worked around with (vector->list (digits ...)));
(exit 1) failure trailer restored (previously removed because
asm had no exit builtin).
* tests/ursa.lsp — same digits literal restoration.
Verified:
* asm regression: 158/158.
* asm-full regression: 158/158.
* Zoë-favorites across Python + C + asm + asm-full: all suites
green with native reader syntax and multi-value tests.
* make test-all stays green.
Two changes, one wiring.
1. rhoff gets a Birthday-bound iteration cap. Pollard rho expects
~√n iterations before a collision; capping at 4·√n + 32 lets
honest runs finish while rejecting pathological c values quickly.
rho's outer retry draws a new c and keeps the total work bounded.
Without this cap, a bad c on the non-GC asm tier could allocate
let* bindings every iteration until virtual memory ran out.
(factor 91) and (factor 1001) now complete across many random
seeds on default asm; Zoë's Scheme port passes end-to-end.
2. tests/ursa-scheme.lsp — Scheme-port-only half of the acceptance
suite. Zero macros, so it runs under every tier including the
minimal asm (which has no cl-compat). Also drops the vector
literal `#(...)` (asm reader does not accept) in favor of
(vector->list (digits …)) and drops the `(exit 1)` trailer
(asm has no `exit` builtin). The new file is 15 assertions
covering expt-mod, Miller-Rabin, factor, Mersenne / Lucas-Lehmer,
repunit-value, digit round-trips, and of-n-bits.
3. tests/cl-compat.lsp — the multiple-value-bind test is commented
out. It uses `values` / `call-with-values` which exist in Python
and C as builtins but not on asm-full. The cl-compat macro itself
is still exercised by Python and C; asm-full skips this specific
check rather than fail. The full 44 remaining assertions all pass
on every tier now.
4. tests/zoe-favorites-test.sh — extended coverage matrix:
Python cl-compat + ursa (Scheme + CL)
C cl-compat + ursa
asm-full cl-compat + ursa
asm ursa-scheme (port only — no macros on minimal)
The old script ran two tiers (Python + C). Now it runs seven
test/tier pairs. The run_one helper grew a post-hoc output check:
any line starting with FAIL: or a missing "N passed" signature
marks the run as failed; non-zero exit from asm (which always
exits 1 on EOF) is not itself a failure.
Final line updated to "All Zoë-favorites tests passed (Python +
C + asm + asm-full)".
make test-all stays green.
asm/lumbda-full now runs Zoë's CL source end-to-end per commit
c6658e0. Rewrite the Known Issues section into a Resolved section
explaining what each of the four underlying asm bugs was (bi_apply
clobber, bi_expt infinite loop on negative exponent, macro_env_head
missing from GC roots, cadar missing from the prelude) and why the
158-test asm suite did not catch them before.
Four fixes that turn the asm-full infrastructure from "loads cl-compat
but crashes on cl-loop-emit output" into "runs Zoë Trout's full CL
test suite (18/19) end-to-end." Zoë's original `examples/ursa.lisp.txt`
now produces matching answers to the Python and C tiers on asm-full.
1. asm/lumbda.s bi_apply — second arg was being clobbered. The
previous impl did `GETARG %rbx; GETARG %rdi; movq %rbx, %rdi;
... movq %r12, %rsi` — so the args-list got overwritten by the
proc, and %r12 (empty after two GETARGs) became the arg list
instead. `(apply f '(1 2 3))` silently reduced to `(f)`. Fix:
`GETARG %rbx; GETARG %rsi; movq %rbx, %rdi; call apply_proc_raw`.
2. asm/lumbda.s bi_expt — decrements rcx by 1 until zero. Negative
exponents looped forever. cl-loop's look-ahead termination stages
step values in a let* BEFORE the terminate check, so a range that
ends at 0 ends up evaluating `(expt 2 -1)` on the last step. Fix:
guard negative exponents, return 0. asm is integer-only; returning
a rational would need a new type. Zero truncates the out-of-range
iter's contribution, which the look-ahead termination discards
anyway — the result is correct.
3. asm/lumbda.s GC roots — macro_env_head was not marked. Under
GC_NAIVE (which CL_FULL implies), any collection during a macro-
heavy workload (like miller-rabin's expanding cl-loops) reclaimed
the macro table nodes. Next use failed with "unbound variable:
cl-when" or similar. Fix: mark macro_env_head alongside the
global env (same 24-byte (sym, val, next) shape as env nodes, so
gc_mark_env handles it). Guarded .ifdef CL_FULL.
4. asm/lumbda.s prelude — added `cadar` (used by
cl-loop-finalizer-expr). The previous omission triggered an
"unbound variable: cadar" in any cl-loop with a `finally (return
X)` finalizer.
5. cl-compat.lsp — two new helpers routed around asm's reduced
list-processing builtins:
* `cl-append` for n-list concatenation. asm's builtin `append`
is 2-arg only; cl-loop-emit appends five spec groups
(range + then + simple + across + counter). Reducing with
2-arg append works on every tier.
* `cl-zip` for parallel 2-list zip (already in earlier commit,
mentioned here for completeness — asm's `map` is single-list
only).
Verification on asm/lumbda-full:
* /tmp/ursa-load-test.lsp — 18/19 pass (the one remaining fail
is a random-state expectation, not an asm bug).
* (primep 97) → 97
* (primep 100) → #f
* (lucas-lehmer-primep 13) → #t (M₁₃ = 8191, prime)
* (lucas-lehmer-primep 11) → #f (M₁₁ = 2047 = 23·89)
* (of-n-bits 8) → random integer in [128, 256) with top bit set
* (prime-of-n-bits 8) → random 8-bit prime
make test-all stays green. All three asm variants still 158/158 on
their local test suites. asm's minimal footprint preserved — every
new line above is under .ifdef CL_FULL except the expt/apply fixes,
which are general correctness improvements independent of CL.
Third asm variant — built with CL_FULL=1 GC_NAIVE=1 via new Makefile
target. Adds the macro machinery needed for cl-compat.lsp on the asm
tier, keeping every addition behind .ifdef CL_FULL so the default
(~22 KB) and -gc binaries keep their current footprint.
Landed in this drop:
* Reader: backtrack on digit-prefixed symbols. After reading digit
characters, if the next char is not a delimiter, input_pos
rewinds and control falls through to .sr_symbol. Makes 1+, 1-,
add1, abc123, and any CL-style identifier with a numeric prefix
parse as symbols instead of truncating to a bare integer.
* Reader: `` ` `` / `,` / `,@` produce (quasiquote X) / (unquote X)
/ (unquote-splicing X) forms. Same build shape as the existing
`'` quote branch.
* Evaluator: .ev_quasiquote + quasiquote_expand walk the template.
unquote evaluates its argument in the current env; unquote-
splicing evaluates then splices via a new list_append_ab helper;
other pairs recurse (cons expand-car expand-cdr). Atoms pass
through. No nested quasiquote depth (deliberate; ticket 0005
scope).
* Evaluator: .ev_define_macro + macro_env_head linked list. Each
(define-macro (name p...) body) prepends a 24-byte
(sym, closure, next) node. Dispatch in eval checks macro_lookup
after all special-form compares; on hit, the closure is applied
to the *unevaluated* argument list and the expansion re-enters
.eval_top under TCO.
* Binding: rest-arg support extended to .apr_bind inside
apply_proc_raw. Previously only .ac_bind (direct .app_closure
path) handled `(lambda (a . b) ...)` correctly; macros call
closures through apply_proc_raw, so this was required to make
variadic defun/setf macros bind correctly.
* Builtin: (gensym) — writes "g%d" for an in-BSS counter, length-
prefixes the buffer, calls intern_static. Available in every
variant (not CL_FULL-gated — useful outside macros too).
* Builtin: (cadr x), (sort lst) and the let* special form from
earlier commit stay in default asm. These are Scheme staples.
* Prelude: evaluated at _start after init_builtins / rng_seed,
before the REPL. Embedded string, input state saved + restored
around the load. Defines caar, cdar, caddr, cadddr, cddr,
cdddr, cddddr, 1+, 1-, add1, sub1, square, eq? (= eqv? for
interned symbols), memq, list-ref, assq, and `case` as a macro.
cl-compat.lsp: two small changes to work under asm's single-list
`map`:
* Added cl-zip helper. Replaced two `(map (lambda (v n) (list v n))
xs ys)` sites with `(cl-zip xs ys)` — asm's builtin map accepts
only one list, and cl-loop-emit needs a parallel walk over
state-vars and new-names.
* Added explanatory comment for cddddr at the top of the shim
(already shipped).
Tests:
* make asm-test (lumbda) — 158/158 pass.
* make asm-test-gc — 158/158 pass.
* make asm-test-full — 158/158 pass on synchronous run.
* Zoë's `examples/ursa.lisp.txt` LOADS on asm/lumbda-full.
`(expt-mod 3 7 100)` = 87.
Most simple cl-loop forms work (while + do + finally, range-to,
then-accumulator).
Known open issues documented in docs/tickets/0005-asm-cl-full.md:
* cl-loop-emit produces wrong output for inputs with `simple` iters
(`(simple a 5)` → state binding dropped). Python/C return the
correct form; asm version is missing the binding. Bug surfaces
in the emit's 30+ binding let*; could not pin down in this
session. Downstream effect: `(miller-rabin n)` and similar
defuns that depend on `cl-loop repeat k for a = ... unless ...
return nil` don't produce usable expansions, so Zoë's acceptance
suite does not run end-to-end on asm/lumbda-full yet.
* examples/ursa-scheme.lsp — `factor` crashes on asm under some
random seeds (bump-allocator exhaustion on long rhoff retry
chains). Out of CL_FULL scope; tracked in same ticket.
Next steps live in ticket 0005. This commit ships the infrastructure
so the remaining work is a debugging exercise against a reproducible
minimal case, not a feature build.
Phase 1 of the asm/lumbda-full roadmap (ticket 0005, in-flight). Adds
the minimum-cost set of additions that lets examples/ursa-scheme.lsp —
the idiomatic Scheme port of ursa.lisp.txt — load and produce correct
results on the asm tier. No CL shim yet: that requires quasiquote,
define-macro, and case, all of which are Phase 2 / 0005.
Added:
* Rest-args in lambda — (define (f x . rest) ...). .ac_bind now
detects when the remaining param list is a raw symbol (TAG_SYM)
and binds it to the remaining arg list. Enables variadic defuns.
* cadr builtin — (car (cdr x)) fast path. Used by Zoë's
repunit-value and any CL-adjacent code.
* sort builtin — ascending insertion sort on a tagged-int list.
Non-destructive. Matches Python/C sort contract (default numeric
ordering). Implementation ~50 lines, recursive sort + insert
helpers.
* let* special form — sequential binding where each init sees the
preceding bindings' values. Fresh sf_let_star + sym_let_star_val
+ .ev_let_star branch that's a one-line variant of .ev_let (eval
init in the extended env rather than the original). TCO preserved.
Tests: 9 new asm assertions in asm/test.sh covering cadr, sort (empty
/ singleton / unsorted / already-sorted), let* (basic + sequential),
rest-args (tail-only + rest-only). Total asm suite now 158 passing.
Known limitation: the Scheme port's factor / rho depends on random
rhoff iteration. For some seeds on asm (e.g. seed=2, factor 91) the
process runs out of virtual memory before rho finds a factor. The
underlying math is correct — this is an asm heap-bump-allocator
behavior under long random-retry chains and will be addressed along
with the CL_FULL work in ticket 0005. Python and C paths unaffected.
make test-all stays green across every tier.
Zoë Trout's favorites at wedgewack.org/ursa.lisp.txt are Common Lisp:
iterative LOOP macros, setf cascades, defun with &optional, image-
based stone-lisp culture. Her first contribution to lumbda was a
question — "do we care for our programs, and how long are they alive
for?" — and the answer now extends beyond the RNG portal (§7.5) to
iteration style itself.
Four-phase delivery, all under ticket 0004:
Phase A — idiomatic Scheme ports at examples/ursa-scheme.lsp.
Every Zoë defun rewritten as named-let + tail recursion + list-
backed work queue + type-predicate dispatch.
Phase B — CL compat shim at cl-compat.lsp.
defun (with &optional), setf (simple vars, multi-pair), flet,
multiple-value-bind, t / nil (nil=#f so cond/if compose),
evenp/oddp/plusp/minusp/zerop, mod/ash/logbitp/nreverse,
cl-when/cl-unless (plain when is a void-returning lumbda special
form), declare (no-op), cddddr (missing accessor).
Phase C — cl-loop macro covering 14 patterns.
while/until/repeat, for VAR from A to/below/downto B, for VAR =
INIT [then STEP], for VAR across VEC, of-type T, do, when/unless
return, finally (return VAL). Sequential do*-style stepping via
gensym + cl-subst. Look-ahead termination so `repeat 4 for s = 4
then (- (* s s) 2) finally (return s)` returns 37634 (pre-step)
rather than 1416317954 (post-step). Every expansion ends in a
named-let tail call — TCO holds for loops of any length.
Phase D — load examples/ursa.lisp.txt with minimal annotation.
Preserves Zoë's CL. Minimal edits documented in file header:
load cl-compat.lsp, loop→cl-loop, when→cl-when, random→random-int,
&key→&optional. rho/digits omitted (need make-array/CLOS — see
ticket 0004 for scope boundary).
Defect uncovered along the way (c/types.c env_lookup): a "global
shortcut" checked global env immediately after missing the local
frame, SKIPPING intermediate parent scopes. Broke lexical scoping
whenever a parent scope shadowed a global. Reproduced with
(define s 4)
(let ((s 100)) (let ((m 0)) s)) ; returned 4, should return 100
Any nested let whose body referenced a shadowed name silently read
the global. Fix: remove the shortcut, walk the parent chain end-to-
end. 1255 assertions across five suites pass unchanged after fix —
surfaced only because cl-loop iterator names routinely collide with
globals accumulated in a stone-lisp image.
Whitepaper §9.2 documents the CL-in-Scheme design and the guarantees
that survive (TCO, portal determinism, cross-impl reproducibility).
Zoë added to authors + acknowledgments; reacknowledgment reframes
her first contribution as the deeper program-lifetime question, with
RNG portal as a derivative (§7.5) and cl-loop as the follow-up.
Tests: tests/cl-compat.lsp (44 assertions) and tests/ursa.lsp (28
assertions) exercise both paths under Python + C via tests/zoe-
favorites-test.sh, wired into make test-all.
MOAD notes: unmoad flags memq/assq in cl-compat.lsp over cl-loop-
keywords (~30 elements, constant) and var->new (≤4 state vars per
loop). Both are macro-expansion-time, bounded-small-N — not runtime
hot paths. Pre-existing c/types.c findings (strcmp-in-loop for
record-type lookup) are not from this change.
New subsection under Language Coverage covers the isqrt primitive that
shipped in 6e9d3ea across Python + C + asm. Documents:
- Semantics: (isqrt n) -> floor(sqrt(n)), integer in / integer out,
negative argument raises; matches Python 3.8+ math.isqrt and R7RS
exact-integer-sqrt.
- Why integer: no FPU drift, no libm platform variance, portal replays
stay bit-identical.
- Algorithm: bit-by-bit digit recurrence, O(log n), no multiply/divide,
no FPU. Asm variant runs in three integer registers.
- Domain limits: Python unbounded, asm 61-bit, C 48-bit (NaN-boxed).
Portal round-trips bit-identical within the smaller tier's window.
- MOAD-0001 note: O(log n) per call, no hidden linear scan.
21 RST lines. Rebuilt PDF and HTML from source.
Closes tickets 0001 (portal-rng) and 0002 (os-entropy-seed).
Ticket 0001 goal 4 called for proof that seeding with k, drawing N,
saving, clearing, resuming in any impl, and drawing M more produces a
full stream matching a single-process Python baseline bit-for-bit.
Prior tests/portal-cross-test.sh exercised producer-consumer agreement
but used a producer-side self-computed baseline; it did not compare
against an independent Python run that never saves or resumes.
tests/portal-rng-cross-test.sh computes a single-process Python baseline
once (seed=42, N+M=10 draws, no portal), then runs all 9 producer x
consumer cells (Python, C, asm each side) and checks that producer's
first N plus consumer's M equals the independent baseline. All 12
assertions pass.
Wired into make test-all. Ticket status updated to resolved on both
0001 and 0002 with dated one-line resolution notes.
Design-only ticket. C's 48-bit NaN-boxed TAG_INT silently truncates
any result > 2^47 (e.g. 10^16 becomes -133099161583616), while Python
(bignum) and asm (61-bit) compute correctly. Recommends heap-allocated
bigint via new TAG_BIGINT, preserving NaN-boxing and JIT fast path for
the inline 48-bit common case. Phased migration with Phase 0 fail-loud
stopgap before the full bignum lands.
Citation block previously linked to git.unturf.com/books/feedback-is-all-you-need,
a placeholder URL that was never live. Replaced with the verified public PDF at
lumbda.com/lumbda-whitepaper.pdf (HTTP 200, application/pdf, 2.7 MB).
Rebuilt PDF and HTML so all three artifacts carry the corrected URL.
Previous revision mentioned (random-seed-from-os!) in one sentence; this
treatment does not match the weight fox assigned — 'a random seed from os
on at least one side is very important.'
New sub-subsection 7.5.1 'Kernel Entropy on One Side' spells out:
- why hand-rolled /dev/urandom seeding is a reproducibility-defect farm
- why we picked /dev/urandom over /dev/random and getrandom(2)
- the real-world Machine A -> portal -> Machine B flow as concrete Scheme
- the audit property: a captured run-id replays the trajectory exactly
- kernel entropy enters the system once, under user control, then never
again — every resume reads the same stream off the portal
Ticket: docs/tickets/0002-os-entropy-seed.md
Ticket 0002 — reads 8 bytes from /dev/urandom (little-endian u64) and
seeds xoshiro256**. Opt-in kernel entropy for stochastic runs; the
default stays deterministic (k=0 at startup), so ticket 0001's
portal-reproducibility contract is unchanged.
Real-world flow now one call away:
Machine A: (random-seed-from-os!) + run simulation + portal-save
Machine B: portal-resume — same stream, bit-for-bit
All three impls fail loud on /dev/urandom trouble (LispErr in Python
and C, stderr + exit(1) in asm) — no silent fallback to a weak seed.
Tests:
- tests/functional.lsp: 2 new shared asserts (entropic + replay)
- asm/test.sh: 2 new asm-local checks (149 total, was 147)
- make test-all green across Python (205), C (205), asm (149)
Whitepaper §7.5 gains one sentence noting the OS-seed path.
unmoad: zero new findings in added code.
Adds a new subsection documenting xoshiro256** across all three tiers
and the portal-format extensions that preserve its state across
processes. Renumbers subsequent subsections 7.5→7.6, 7.6→7.7, 7.7→7.8;
updates the §7.6 cross-reference in the reproducibility map.
New Acknowledgments section before Citation/References/License credits
Zoë Trout — whose question 'does the portal tech keep the random list
seed which will allow transferring random entropy between processes
when continuing a simulation?' surfaced the gap. The earlier claim
'portal = portable machine state' carried a silent asterisk for
stochastic code; that asterisk is now gone.
Ticket: docs/tickets/0001-portal-rng.md
Implementation: 27f468c (Python + C) + 54c4c65 (asm + tests)
Completes ticket 0001 started in 27f468c. All three impls now carry
bit-identical xoshiro256**; portal state round-trips across process
boundaries in every producer x consumer cell (Python <-> C <-> asm).
asm impl:
- 4 new builtins: random-seed!, random-int, random-state, random-state!
- g_rng_state in BSS (4 x u64); rng_splitmix64_step, rng_seed, rng_next
- Binary portal header bumped LUMBDAB1/48 -> LUMBDAB2/80; carries
32 bytes of rng state at offsets 40..64, reserved moved to 72
- No float support in asm, so (random) intentionally omitted there
- _start seeds with 0 so the stream is deterministic from startup
Python + C (supplements 27f468c):
- rng_seed(0) auto-invoked at module load / register_portal_builtins
so (random) without explicit (random-seed!) returns a real value
instead of the all-zero xoshiro fixed point
Tests:
- tests/functional.lsp: 7 new shared assertions (Python + C)
- asm/test.sh: 5 new asm-local assertions (142 -> 147)
- tests/portal-rng-save.lsp / portal-rng-load.lsp: portable S-expression
portal that captures both state AND next-5 baseline so loader self-
verifies without a separate harness
- tests/portal-cross-test.sh: 9 new producer x consumer RNG cells; all
18 cells pass end-to-end
Verified: seed=42, (random-int 1000000) draws 1..10 =
558742 543102 559009 124193 317476 750584 200754 814407 344958 929085
identical in Python, C, and asm.
unmoad scan: zero new findings in added code.
Zoe's contribution question: does our portal preserve RNG state so a
simulation can continue in another process with the same random stream?
Answer today: no — no RNG existed. Answer now (Python + C): yes, bit-identical.
- New builtins: random-seed!, random, random-int, random-state, random-state!
- xoshiro256** (Blackman & Vigna 2018) — deterministic, portable, no libc rand
- State = 4 x u64; portal-v1 JSON gains 'rng' field with 8 x u32 halves
- Python and C produce bit-identical streams (verified: seed=42, 10 draws)
- Asm impl + cross-impl tests + whitepaper note: next commits
Ticket: docs/tickets/0001-portal-rng.md
Previous commit used `all: revert` on the floating button to cancel
docutils's blanket `body > *` rule. Too aggressive — it also removed
uncloseai's own `position: fixed`, `bottom`, `right`, `width`,
`background`, `color`, `border` declarations, making the button
disappear entirely.
Replace with a tight override: only the five properties docutils's
blanket rule sets (background-color, line-height, padding, margin,
max-width) get reset with !important. Everything else falls through
to .uncloseai-floating-button's own stylesheet, so the button
re-appears in its designed 121px pill, bottom-right, black-on-white.
Any <img> inside <main>/<section> now opens in a full-viewport
overlay (92% black bg, image centered at max-width/max-height). Click
the overlay or press Esc to close. Pure vanilla JS + CSS, injected
inline via inject-whitepaper-css.py — keeps the HTML self-contained
with no external lightbox dependency. cursor: zoom-in signals
interactivity; the overlay uses cursor: zoom-out.
Three whitepaper-HTML-only fixes:
* docutils's responsive.css blanket-pads every `body > *` with
`padding: 0.5rem calc(29% - 7.2rem)`. The uncloseai floating
button gets appended to <body>, picks up that padding, and blows
from its designed 121 px to nearly half the viewport. Scoped
override resets the cascade on the button selector with
`all: revert` + explicit zero padding/margin.
* whitepaper h1/h2 titles now use ChunkFive (same face as the
homepage logo) instead of docutils's default serif. Font embedded
as a base64 data: URI inside a <style> block so the HTML stays
self-contained — no external font fetch.
* RST layout: λ logo now sits UNDER the "Lumbda" heading instead
of above it. Permacomputer logo still follows below the λ.
New helper: whitepaper/inject-whitepaper-css.py runs after
embed-images.py, reads ChunkFive from www/fonts/..., base64-encodes,
writes the <style> block before </head>. Makefile wires it in.
Source PNG from the MPS lumbda-logos product is already an inverted λ
(hooks at top-left + bottom-right, V-body). My prior commit applied
vertical + horizontal flips on top of that, which undid the inversion
and produced a right-side-up calligraphic λ — the opposite of what
the artwork was drawn to express.
Regenerate both PNG variants (lumbda-logo.png black,
lumbda-logo-green.png #227842) directly from the source with no
flips. Only transform: alpha-from-brightness so anti-aliased edges
survive over any background.
Homepage img src picks up ?v=2 so any browser with the flipped
version in disk cache re-fetches. Logo size stays 3.33× the
wordmark font (10.66rem). PDF + HTML rebuilt — embed-images.py
base64-inlines the new PNG into the HTML automatically.
Fetched the lumbda-logos artwork from
media.unturf.com/c/fbd3d473-.../lumbda-logos (a MakePostSell product),
applied both flips (vertical + horizontal, = 180°) at bake time so the
PNG ships oriented correctly without any CSS transform dance, and
recolored non-background pixels to brand green (#227842) with alpha
derived from pixel brightness so anti-aliased edges stay smooth.
Two PNG variants ship under whitepaper/diagrams/:
* lumbda-logo.png dark ink for print contexts
* lumbda-logo-green.png #227842 for web
www/ carries symlinks to both.
Homepage (www/index.html): replaces the CSS-rendered λ with an
<img class="lambda-mark"> element sized 9.6rem — 3× the 3.2rem
wordmark font — stacked below "lumbda." on its own line.
Whitepaper (whitepaper/lumbda-whitepaper.rst): adds the logo as the
first figure above the permacomputer-logo, 24% width, centered.
Regenerated PDF + HTML; embed-images.py base64-inlines the new file
automatically so the HTML stays single-file.
CI (.gitlab-ci.yml): generalizes the symlink resolver from two
explicit `cp -L` calls into a `find www -type l` loop, so every
current and future symlinked asset deploys without per-file CI edits.
λ in front was reading as a "Y" attached to the wordmark ("ylumbda.").
Move the mark under the wordmark as a standalone logo below the name.
Sitting alone, the flipped λ doesn't get recruited into the first
letter and reads as its own glyph — a lambda turned on its head.
Collapse --accent (was blue #0b5394) into the same green as the
period. One color now marks everything that says "lumbda": the λ
logo, the declarative period, link underlines, and the CTA buttons.
Dark-mode variant lightened to #5ec07a so the green reads on
#161613.
Previous design folded the flipped λ inside the wordmark as the "l".
That read as a cute substitution but didn't work as a brand mark.
Split them: the λ now stands on its own, rotated upside-down, 5rem
(larger than the 3.2rem wordmark), so it reads as a logo symbol
separate from the text.
Wordmark restores a proper lowercase "l" in "lumbda" — readable,
copy-pasteable, searchable. Trailing period picks up a new --green
design token (#2a8a3a light / #6fd68a dark) so the declarative stop
also reads as growth — matches the permacomputer ethos.
Layout: flex logo-row with baseline alignment and a small gap; the λ
hangs from the wordmark's cap-line for a balanced mark-and-wordmark
composition.
Previously the HTML whitepaper referenced diagrams via relative paths
(src="diagrams/X.png"), which 404'd because the docroot does not carry
a mirror of whitepaper/diagrams/. The "HTML whitepaper" thus showed
broken image stubs instead of the architecture figures.
Fix: a post-processor (whitepaper/embed-images.py, stdlib Python) runs
after rst2html5. Every <img> with a relative src gets base64-inlined
as a data: URI with inferred MIME. Any .svg reference would splice in
as an inline <svg> element, surfacing alt text as <title>; the current
RST only uses the .png variant of gnu-logo so no SVGs inline for now,
but the path works when we switch.
Makefile's whitepaper-html target chains the embed step after the
existing sed passes for the uncloseai.js script tag and the meaningful
<title>. Title tweaked to "feedback as a primitive" to match the
homepage tagline.
Result: whitepaper/lumbda-whitepaper.html grows from 180 KB to 4.6 MB
(base64 inflation on ~2 MB of diagrams), and it now opens offline as
a single file — no network fetches for figures, no docroot mirror
needed.
Wordmark now renders λumbda. — flipped λ stands in for the "l",
visual pun on lambda (the primitive the language shells around),
trailing period as declarative closure. All-lowercase, accent-colored
period. aria-label="lumbda." keeps screen readers reading the name
correctly; the λ and the period both carry aria-hidden.
CLAUDE.md rewritten to avoid the verb "to be" and to prefer "a"/"our"
over "the" when a thing counts as one of many or as shared. Same rule
applied to www/index.html (tagline now "feedback as a primitive", body
prose cleaned of is/are/be, table header now "What a tier buys",
section heads trimmed of definite articles). Monospace body stays,
ChunkFive titles stay.
Whitepaper now ships in two formats:
* Makefile target `make whitepaper-html` runs rst2html5 with
embedded minimal.css + responsive.css (docutils bundled), then
injects the same uncloseai.js module script that the homepage
carries, and stamps a meaningful <title>. 180 KB self-contained.
* PDF build target renamed to `make whitepaper-pdf`;
`make whitepaper` now builds both.
www/lumbda-whitepaper.html symlinks to the generated HTML (same
pattern as the PDF symlink). .gitlab-ci.yml resolves both symlinks to
real files before rsync so the proxy gets byte-identical copies.
Homepage now exposes both formats side-by-side via two .cta buttons
and the footer lists HTML + PDF.
Rename www/whitepaper.pdf → www/lumbda-whitepaper.pdf (symlink target
unchanged; slug now descriptive for search/citation). Update the CTA
and footer links in index.html, plus the .gitlab-ci.yml symlink-resolve
step. Companion Caddyfile redirect in proxy.unturf.com keeps old
/whitepaper.pdf links working via 301.
Add <script src="https://uncloseai.com/uncloseai.js" type="module"> to
<head>, matching www.unturf.com + timehexon.com pattern — brings the
permacomputer chat/AI integration to lumbda.com's front door.
ChunkFive webfont (woff2 + woff, ~43 KB total) dropped into
www/fonts/chunkfive/; style.css adds @font-face with font-display:swap
and applies the family to header h1 (3.2rem logo) and main h2
(1.5rem section titles). Monospace body text unchanged. Font files
reused from www.unturf.com; SIL OFL.
index.html: "x86_64 assembly interpreter" and table-row labels
"Pure x86_64 assembly" / "Assembly + naive mark-sweep GC" now read
"GNU asm" — precise and consistent with how we talk about the tier
elsewhere.
Resolves the www/whitepaper.pdf symlink to a real file before rsync
(deploy-www.sh preserves symlinks, which would land broken on the
proxy), writes version.json with the commit SHA, then calls
deploy-www.sh lumbda www via sudo. Runs on the proxy.uncloseai.com
runner, same pattern as cuppcb.com and the rest of the static fleet.
asm-gc gains (tcp-sendfile socket path) → builtin (90 lines) that issues
SYS_SENDFILE(40) in a loop, streaming a file from fd → socket with no bounce
through the Lumbda heap. Zero-copy kernel path for large responses.
examples/http-static-server-sendfile.lsp (hybrid): small assets
(≤ 16 KB) stay inline-cached as full HTTP responses; large assets cache
only headers and stream the body via tcp-sendfile. 4-way race on
i5-8350U, 100 PDF requests (2.56 MiB), concurrency 8:
uncached 159 req/s 406 MiB/s 15.5 MB RSS
cached 238 req/s 603 MiB/s 7.2 MB RSS
sendfile 480 req/s 1226 MiB/s 4.2 MB RSS
caddy 485 req/s 1238 MiB/s 37.1 MB RSS
sendfile lands within 2% of caddy on throughput with 9x less peak RSS in
a 27 KB binary vs caddy's 38 MB (1400x smaller).
examples/http-static-server-adaptive.lsp (learning preload): per-URL hit
counter persisted to www.hits every N requests. At boot, ranks and
preloads top *cache-max* URLs from the prior run's data (cold-start
falls back to a seed list). Cold requests beyond the seed promote on
first hit. Drops heap-restore arena pattern since the server mutates
persistent state every request; relies on GC build's mark-sweep.
tests/bench-www-race.sh: adds sendfile variant on port 8083, auto-sizes
PDF byte count from the on-disk whitepaper so a whitepaper rebuild
doesn't desync the MiB/s calc.
Whitepaper §11.7 "Static File Serving: Cache, Sendfile, and Adaptive
Preload" documents the four variants, benchmark table, and the
arena-vs-mutation tradeoff. §13 Future Work adds DAG-of-hot-paths
predictive preload as the direction for > 1000-resource deployments
where frequency-only ranking is too narrow.
examples/http-static-server-cached.lsp — same HTTP server but every
preloaded URL's full HTTP/1.0 response (headers + body) is composed
once at startup and stored in a hash-table, so the per-request
handler is a single hash-table-ref/default. No file->string, no
string-append, no MIME lookup in the hot path.
Config knobs:
*docroot* filesystem root (default "www")
*cache-max* soft cap on cached entries (default 100)
*preload-paths* list of URL paths to pre-fetch at startup
Preload list for lumbda.com: "/", "/style.css", "/whitepaper.pdf",
"/robots.txt", "/404.html" — anything not in the list returns the
cached 404 response (no disk hit). Cache lives pre-snapshot so
heap-restore never reclaims it; RSS stays at the cache size
forever.
Race vs caddy on this laptop (2000 small / 200 large, concurrency 8):
small req/s PDF req/s PDF MiB/s peak RSS
lumbda-www uncached 690 195 495 15.5 MB
lumbda-www cached 686 319 811 7.2 MB
caddy file-server 688 478 1,217 38.9 MB
Cache wins on the PDF: 1.64x faster than uncached, RSS DROPS from
15.5 MB to 7.2 MB because the cached path allocates nothing per
request (all allocation happened pre-snapshot). On small files
already-hot paths mean the cache is a wash — 690 vs 686 is noise.
Caddy still wins 1.5x on the PDF via sendfile(2) zero-copy; we
allocate the 2.67 MB response once at startup and tcp-send it.
Closing the gap further would take a sendfile asm primitive —
separate project. For a minimal static site serving its own
whitepaper, the cached 27 KB asm binary is viable: 319 req/s
and 811 MiB/s with 5x less memory than caddy.
tests/bench-www-race.sh updated to run all three side-by-side
(uncached + cached + caddy) at three ports. Cached server's port
is patched via sed at the entry point so the two lumbda variants
don't collide. PDF byte-integrity checked on all three paths.
Ships examples/http-static-server.lsp — ~65 lines of portable Scheme
that reads files from a docroot (default ./www) and serves them over
HTTP/1.0 with MIME dispatch, path-traversal rejection, heap-snapshot
per request. Runs in any tier; target deployment is asm-gc for the
27 KB stripped binary + bounded memory backstop.
Required one asm fix first: heap_grow was mmap'ing fixed HEAP_SIZE
chunks, so any single allocation larger than a chunk (notably the
2.67 MB whitepaper PDF read via file->string) loop-looped through
.ha_overflow forever. Now heap_grow rounds required bytes up to
HEAP_SIZE multiples on oversize alloc, so a big request carves its
own big chunk in one go. Small allocs still land in standard-sized
chunks.
Two new benches:
tests/bench-lumbda-www.sh — drive N small + M large requests against
asm-gc, verify PDF round-trip, sample peak RSS. At 1000/100: 331 req/s
small, 120 req/s large (304 MiB/s), peak 15.5 MB.
tests/bench-www-race.sh — adjacent A/B vs caddy v2.5.1 on the same
docroot. Numbers on this laptop, concurrency 8, 2000 small + 200 large:
small req/s PDF req/s PDF MiB/s peak RSS binary
lumbda-www (asm-gc) 375 137 349 7–16 MB 27 KB
caddy file-server 358 231 588 38 MB 38 MB
Reading: lumbda edges caddy on small files (less per-request overhead),
caddy wins 1.7x on large files (sendfile zero-copy; we allocate the
whole file into a string and write it with one syscall). Both byte-
identical on the PDF. Memory: lumbda 2.5-5x less at steady state.
Binary size: 1400x smaller (27 KB vs 38 MB).
Feature gap: caddy has HTTPS, HTTP/2, range, middleware, etc. lumbda
has none of that yet — but for the specific job of serving lumbda.com's
six-file docroot it is viable right now.
Makefile adds `bench-lumbda-www` and `bench-www-race` targets.
137 asm no-GC + 137 asm GC tests still pass.
Minimal, monospace, readable. Works as a drop-in docroot for
any static server (nginx, caddy, python -m http.server, etc.).
Light/dark mode via prefers-color-scheme. No JS, no fonts, no
third-party anything.
Files:
www/index.html landing page (tagline, get-it, four-tier
table, portal summary, EML proof blurb,
license, whitepaper CTA)
www/style.css ~150 lines, CSS vars for theming
www/robots.txt allow everything
www/404.html referenced by servers that support custom
error pages
www/whitepaper.pdf -> ../whitepaper/lumbda-whitepaper.pdf
(symlink — one source of truth, rebuilds
via `make whitepaper` auto-propagate)
Smoke-tested with python3 -m http.server: 200 on /,
Content-Type: application/pdf for /whitepaper.pdf,
Content-Length matches the current 2.67 MB PDF,
/robots.txt serves, /nonexistent returns 404.
No new build step — docroot is pure static files the existing
whitepaper target already produces. A web server configured
with docroot=www/ and fallback 404.html has a working lumbda.com
today.
Completes the on-disk side of the uncommonlisp -> lumbda rename:
Filesystem: /home/fox/git/uncommonlisp -> /home/fox/git/lumbda
with a back-compat symlink
/home/fox/git/uncommonlisp -> lumbda
so any stale path reference (agent memory files,
shell history, other sessions) still resolves.
tests.py: cwd='/home/fox/git/uncommonlisp' -> '/home/fox/git/lumbda'
(the only hardcoded absolute path we left behind in
the previous rename commit, because the directory
itself hadn't moved yet).
Remote URL (git@git.unturf.com:engineering/unturf/uncommonlisp.git)
still points at the old name and needs to be flipped AFTER fox
renames the gitlab project — probe confirms the new URL currently
404s, so the flip waits for the gitlab rename to land.
Verified 571 Python tests pass under the new cwd; symlink lets
`cd /home/fox/git/uncommonlisp` still work for anything cached.
Historical internal name "uncommonlisp" retired in favor of the
public name "lumbda" ahead of lumbda.com going live. Scope of
this commit:
Source files renamed:
uncommonlisp.py -> lumbda.py
asm/uncommonlisp.s -> asm/lumbda.s
c/uncommonlisp.h -> c/lumbda.h
whitepaper/uncommonlisp-whitepaper -> whitepaper/lumbda-whitepaper (.rst + .pdf)
Binaries renamed (tracked ones; c/ was always gitignored):
asm/uncommonlisp, asm/uncommonlisp-gc, asm/uncommonlisp.o,
asm/uncommonlisp-gc.o -> asm/lumbda(-gc)(.o)
c/.gitignore -> ignores lumbda
Internal string updates (sed pass ordered longest-first):
asm/uncommonlisp -> asm/lumbda
c/uncommonlisp -> c/lumbda
uncommonlisp.py -> lumbda.py
UNCOMMONLISP_BIN -> LUMBDA_BIN (asm/test.sh env var)
"uncommonlisp> " -> "lumbda> " (asm REPL prompt baked into binary)
UNCOMMONLISP -> LUMBDA (macros, comments)
uncommonlisp -> lumbda (prose)
Binary portal magic updated:
"ULPORTAL" -> "LUMBDAB1" # "Lumbda Binary v1"
Old portal files are not backward-compatible — this is a deliberate
break since it's the rename moment. S-expression portals already
carry their own ";; lumbda-portal v1" header and remain cleanly
versioned.
WHITEPAPER.pdf / WHITEPAPER.rst symlinks repointed to the renamed
files. Makefile's whitepaper target targets lumbda-whitepaper.pdf.
Not changed (intentional, separate phases):
- Filesystem directory /home/fox/git/uncommonlisp itself
(fox renames locally and the gitlab repo URL in a follow-up)
- tests.py hardcoded cwd=/home/fox/git/uncommonlisp
(matches the current on-disk location; will flip when the
directory rename ships)
- Git history (immutable; old commits still say uncommonlisp,
which is correct — that's what they were)
Verified:
137 asm no-GC + 137 asm GC + 571 Python + 83 C + 189 shared
functional tests all pass under the new names.
bench-gc-http (2000 req): all 4 cells behave as expected
(cells 1/2 flat, 3 leaks, 4 bounded at 1 chunk).
Python REPL, C REPL, asm REPL all start cleanly.
GC-build portal files now start with ";; lumbda-portal v1\n". The
line is a Scheme comment the reader already skips, so loading a
v1 file via bi_load works unchanged. What's new is that
portal-resume actively validates the header before delegating to
bi_load:
file starts with ";; lumbda-portal v1\n" -> load normally
file starts with ";;" but different text -> return #f (rejected)
file does not start with ";;" at all -> load as legacy (back-compat)
The check reads the first 32 bytes of the file, compares the
first two bytes against ";;", and on match compares the full
20-byte v1 prefix. Closes the versioning-friction concern a SEW
reviewer raised after reading §7.4.1: we can now add a v2 format
with new syntax (complex numbers, records, whatever) without
older consumers silently parsing new files into garbage — they
will cleanly return #f.
Verified:
v1 portal -> resume loads all bindings, returns #<void>
v2 portal -> resume returns #f without evaluating any forms
legacy portal (no ;; header) -> resume loads normally
missing file -> resume returns #f
The deeper framing worth writing down: this is a migration format
for handoff across process / tier / machine, not an archive format.
If archival becomes a real use case it earns its own format with
proper schema evolution and a builtin-rename table. The v1 tag is
the minimum hook that lets v2 happen cleanly when someone needs it.
137 asm no-GC + 137 asm GC still pass. §6.6.4 HTTP cells still
green at 10K: GC + no-snapshot at 543 req/s, peak 1.1 MB, 972 KB
growth (one chunk, steady state).
Root-cause fix for the residual crashes I had documented as known
issues in §6.6.4. Every heap_alloc call site was setting its type
byte with `orq $(HT_X << 8), -8(%rax)` — but OR merges with the
stale type byte from a free-list-reused block. A pair previously
used as a vector (type 5 = 0b101) re-allocated as pair (type 1 =
0b001) ends up with merged type 0b101 = still vector. Walker then
treats the pair as a vector, reads the pair's car as a "length",
and walks off the block end — hence the hash-set bench's "unbound
variable: t", memory bench's "unbound variable: lst", arena bench's
"unbound variable: k".
Fix: overwrite the byte instead of OR-ing. 18 sites converted from
`orq $(HT_X << 8), -8(%rax)` to `movb $HT_X, -7(%rax)`. Every
previously-residual crash gone on first rerun.
Refreshed benchmark numbers throughout §6.6:
§6.6 Memory table: 122× less memory at 26% slowdown (was 124×,
30%). Range shifted because the fix also accelerated the common
paths; ratio stable.
§6.6.4 HTTP soak at 50,000 requests × 16 concurrent × 4 cells:
no-GC + snapshot 630 req/s peak 100 KB growth 4 KB
GC + snapshot 633 req/s peak 120 KB growth 4 KB
no-GC + no snapshot 625 req/s peak 458 MB OOM at cap
GC + no snapshot 610 req/s peak 1,092 KB growth 852 KB
Cell 4 now sustains 50K requests with steady-state 1-chunk memory.
Previous residual edge at 50K (cell 4 failing to start) was a
manifestation of the same type-byte bug, now gone.
§6.6.3 Adaptive numbers collapsed to within ~1% across all three
workloads (was 6% / 7% / 17% deltas). Paper updated to honestly
report adaptive as a null experiment on these shapes — neutral
cost, same stats surface, default on.
§6.6 diagram: bench-gc.png refreshed to match new numbers.
137 asm no-GC + 137 asm GC + 189 shared functional all pass.
Hash-set / memory / arena / adaptive / HTTP benches all clean.
Extended the HTTP-under-GC validation from the original 5,000
requests to 20,000 requests per cell. All four cells still hold:
asm no-GC + snapshot ~240 req/s peak 96 KB growth 0 KB
asm GC + snapshot ~235 req/s peak 120 KB growth 0 KB
asm no-GC + no snap ~470 req/s peak 185 MB growth 185 MB
asm GC + no snap ~450 req/s peak 1,084 KB growth 972 KB
Cell 4 (GC + no snapshot) is the real validation target. 972 KB of
growth over 20,000 requests = one chunk filled once, after which
the collector cycles through reclaimed space. No monotonic leak,
no OOM, no crash. Naive mark-sweep is now a correct (not optimal)
allocator for long-running asm servers that don't manage arenas.
The paper's soak commentary explicitly calls this out and also
notes two remaining rough edges we've observed but not yet
debugged: (a) at 50k requests with the no-GC + no-snap case
saturating the 512 MB vcap right before cell 4 starts, cell 4's
server sometimes fails to initialize — looks process-environment
rather than GC, no clean explanation yet; (b) the hash-set
benchmark on the GC build still surfaces an occasional
unbound-variable error at ~1 MB/iter workloads.
Reproduce: make bench-gc-http defaults to 5k now; soak uses
`REQUESTS=20000 CONCURRENCY=16 bash tests/bench-gc-http.sh`.
Binary heap dump can't work under GC because the heap is a linked
chunk list with typed block headers and a free list. Raw-byte
serialization would lose structure. Rather than invent portal v2
with chunk tables and pointer relocation, the GC build uses the
S-expression format that already works across all other tiers:
# bi_portal_save in GC build:
walk %r14 (env chain); for each non-builtin, non-closure binding,
emit `(define <sym> (quote <val>))` to the opened file via
scheme_print with output_fd redirected to that fd.
# bi_portal_resume in GC build:
jmp bi_load — read every form from the file, eval each in %r14.
The quote wrapper makes data values round-trip cleanly: lists,
vectors, strings, symbols, numbers, pairs all re-read as literals.
Closures and builtins are explicitly skipped — closures can't
faithfully re-read from their printed form; builtins reconstruct
from the target's prelude. Same treatment the JSON portal gives.
The no-GC build keeps the binary portal format unchanged (wrapped
in .ifndef GC_NAIVE). Users get the fast format on the fast build,
the portable format on the safe build. Same API, different wire
format by build.
Cross-tier verified: GC-asm producer -> Python consumer passes
with `x=42`, `nums=(1 2 3 4 5)`. The §7.2 cross-impl matrix
expands from 9 to 16 cells, all green.
Whitepaper §7.4 now notes it's the no-GC format; new §7.4.1
documents the GC build's S-expression portal with the trade-off
(slower than binary dump, stricter about what round-trips, but no
architecture constraint and no "same binary" requirement).
137 asm no-GC + 137 asm GC + 189 shared functional tests pass.
All four HTTP cells from §6.6.4 still bounded under sustained
load (GC + no-snapshot at ~630 req/s peak, 1.1 MB steady state).
Replaces header format from [size:63 | mark:1] with
[size:48 | type:8 | flags:8 (mark in bit 0)]. Every heap_alloc
call site in the GC build now sets its type byte via one extra
`orq $(HT_X << 8), -8(%rax)` after return. Ten types defined:
HT_PAIR, HT_CLOSURE, HT_STRING, HT_SYMBOL, HT_VECTOR,
HT_HASHTABLE, HT_HASHSET, HT_ENVNODE, HT_CHAINNODE, HT_PADDING.
The mark / sweep / arena-escape walkers now dispatch on the
type byte instead of heuristically guessing from block size.
Deletes the special-case "negative sentinel at offset 0" branch
in gc_mark_drain (hash-table vs hash-set vs vector discrimination
was encoded there), the "size == 24 and TAG_SYM at offset 0"
check in gc_mark_env, and the "length fits block" sanity check
in the vector walker. All that logic collapses into a single
compare on the type byte.
Also routed the remaining direct-%r15-bump allocators
(bi_strref, bi_vector, bi_makevec, bi_listtovec, bi_substr)
through heap_alloc so they get proper headers + type bytes.
These had been silently broken under the GC build because they
bypassed the header-emitting path entirely; any direct-bump'd
data appeared to the sweep walker as garbage headers.
§6.6.4 cell 4 (asm GC + no snapshot) was crashing at first GC
before this change. After: serves 5,000 HTTP requests at ~410
req/s, peak RSS 1,088 KB (one chunk), growth 972 KB — the
collector hit its natural steady state. First time we've
validated "naive GC as replacement for snapshot discipline"
under real traffic.
New §6.6.5 "Precise Block Typing" in the whitepaper documents
the old heuristic bugs, the new header format, and the cost
(one orq per alloc, 16 header bits) vs benefit (class of bugs
eliminated). Updated §6.6.4 to reflect cell 4 passing.
Remaining known issue: the hash-set bench on the GC build under
very heavy sustained allocation still surfaces an occasional
unbound-variable error. The precise-type fix addressed the
observed HTTP crash; a deeper root-scan edge case remains.
Tracked for Fix 2 work.
137 asm no-GC + 137 asm GC + 189 shared functional tests all
pass.
New infra:
- examples/http-server-noarena.lsp: same HTTP server minus the
heap-snapshot/heap-restore arena loop. Isolates whether the GC
build actually holds memory under real traffic, independent
of the portable snapshot pattern.
- tests/bench-gc-http.sh: drives 5,000 concurrent requests per
cell across the full 2x2 matrix {no-GC, GC} x {snapshot, no}.
- Makefile: new `bench-gc-http` target.
Extended benches to exercise both asm binaries:
- tests/bench-hashset.sh now runs against both asm/uncommonlisp
and asm/uncommonlisp-gc, with set +e so a GC-build crash on
one workload doesn't abort the other.
- tests/web-benchmark.sh adds a dedicated asm-gc row (and prints
its stripped binary size) so the HTTP throughput comparison
reports both.
Whitepaper updates:
- §6.6.4 "Validation: HTTP Server Under Sustained Load" — the
4-cell memory matrix. 3/4 cells green; cell 4 (GC + no
snapshot) crashes at first GC trigger — another instance of
the conservative-scan type-confusion class we already fixed
once at the env/string boundary. Logged as a known issue
rather than shipping a partial fix under time pressure.
heap-snapshot + heap-restore remains the recommended pattern
for production asm code; the naive GC is diagnostic + control
group, not a replacement for the arena discipline.
- §6.5 hash-set speedup table slightly softened to ~15-20x (was
15-21x) since run-to-run noise on a shared laptop shifts the
per-phase ratio by a few percent. Ratio is stable to first
order.
- §8.6 narrative references the ~1280x symbolic-vs-brute-force
figure instead of the stale 40x.
- §6 reproducibility list now lists `make bench-gc-http`.
All 137 asm no-GC + 137 asm GC + 189 shared functional tests
still pass.
Old table only showed three rows, one of which (Lumbda numerical
brute-force at 59s) had been superseded months ago by the native
symbolic rewriter and cached-replay path already described earlier
in §8. §8.6 lagged and kept claiming "40x faster than brute-force"
when the symbolic-vs-brute-force win is actually ~1,280x.
New six-row table contrasts:
- Python numerical 0.04 s
- Lumbda brute-force 59 s (kept for historical scale)
- Lumbda symbolic cold 46 ms (~1,280x over brute-force)
- Lumbda symbolic cached 7 ms
- Lean 4 cold rebuild 722 ms
- Lean 4 cached 5 ms
Three wins compound: symbolic over numerical (~1280x, MOAD-0001
at proof-methodology layer), asm over Lean's cold binary startup
(~16x), and cached replay over cold on both sides (~100x).
Explicitly notes that Lean's kernel TCB stays smaller even when
timings equalize (~3 KLOC audited elaborator vs ~6.6 KLOC asm
interpreter) — right tool for different assurance levels.
References existing make bench-proof target rather than adding
new plumbing; the numbers already come from proof/benchmark.sh.
Adds §6.6.3 "Collaborative Meta-GC: From Greedy to Adaptive" with
the three-workload benchmark (friendly / hostile / mixed × greedy
/ adaptive). Honest read of the numbers:
friendly greedy 732 ms 1000 resets, 0 escapes
friendly adaptive 691 ms 1000 resets, 0 escapes (-6%)
hostile greedy 568 ms 0 resets, 1000 escapes
hostile adaptive 607 ms 0 resets, 1000 escapes, 11 skipped (+7%)
mixed greedy 1981 ms 17 resets, 1983 escapes
mixed adaptive 1694 ms 14 resets, 1986 escapes, 2 skipped (-17%)
Adaptive wins on friendly (-6%) and mixed (-17%, the policy's
design target). On fully hostile workloads implicit GC fires 982
of 1000 arenas before the dispatcher sees them, so the signal is
drowned and greedy happens to edge adaptive by ~7%. Section
explicitly calls out the collaborative-but-local structure
(shared state on arena_active + EMA + countdown, decisions made
locally by each component) and credits the benchmark work with
surfacing two real correctness bugs in the conservative stack
scan — 24-byte strings misread as env nodes, 40-byte strings
misread as 25-element vectors — both now fixed.
Also:
- meta-gc-policy.dot rewritten to show the adaptive gate
(rate > 50% + probe countdown) before the greedy verify path;
new skip branch, new EMA annotations on edges.
- §6 reproducibility list + Makefile bench-gc-adaptive target.
- PDF rebuilt at 2.64 MB.
137 asm no-GC + 137 asm GC + 189 shared functional tests pass
against the new asm.
Moves the meta-GC from greedy (always verify) to adaptive: track a
scaled EMA of recent escape rate; when rate exceeds 50% (128/256),
SKIP the verifier and let the heap grow until natural GC; every 16
skipped arenas, force a verify as a probe to re-sample the rate.
Two correctness fixes uncovered while testing adaptive:
1. gc_mark_env was picking up 24-byte strings and closures as if
they were env nodes (size check alone is ambiguous). Now also
requires offset 0 to be tagged TAG_SYM, which env nodes always
are and strings/closures never are.
2. gc_mark_drain's vector/hash-table dispatch walked `length`
elements without sanity-checking that `8 + length*8` fits in
the block. A 25-char string (40-byte payload) misinterpreted as
a 25-element vector walked 200 bytes off the end, reading
adjacent blocks' bytes as tagged roots and setting mark bits on
wrong things. Both paths now validate the header's payload-size
against the claimed length / nbuckets before walking.
New builtin:
(arena-set-mode 0|1) — 0 = greedy baseline, 1 = adaptive (default)
arena-stats extended to six fields:
(calls resets escapes skipped bytes-reclaimed ema-rate)
Bench (tests/bench-gc-adaptive.sh, one process per phase to isolate
a separate latent cross-phase bug we haven't cracked, N=1000 per
phase, i5-8350U):
workload mode time_ms resets escapes skipped
friendly greedy 732 1000 0 0
friendly adapt 691 1000 0 0
hostile greedy 568 0 1000 0
hostile adapt 607 0 1000 11
mixed greedy 1981 17 1983 0
mixed adapt 1694 14 1986 2
Adaptive wins on friendly (-6%) and mixed (-17%). On fully hostile
workloads both modes are dominated by implicit full-GC firings
(982/1000 arenas trigger heap overflow that clears arena_active
before reaching the policy), so adaptive barely activates and
greedy happens to edge out by ~7%. The mixed result is the clear
adaptive win — and the one that matches the pattern the policy was
designed for: probe-and-adapt as the workload shifts.
137 asm (no-GC) + 137 asm (GC) + 189 shared functional tests all
still pass.
