From 0f894734a2fe890d0ce9b827d0500cc56042795d Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Thu, 16 Apr 2026 21:16:17 -0400 Subject: [PATCH] CLAUDE.md: asm server discipline with teeth after 2nd crash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second crash on 2026-04-17 proved the earlier "Asm memory discipline" section was aspirational. Rules were documented, blackops kept spawning backgrounded asm servers for quick RSS checks during the heap-snapshot and RPC work, and one escaped again. This revision makes the discipline procedural: - Wrap EVERY backgrounded asm spawn with `timeout N`, even for a "quick test" — the mental overhead is zero; the consequence of forgetting is a crashed machine. - Trap EXIT/INT/TERM at the top of every block that backgrounds anything, not just benchmark scripts. - MANDATORY pgrep verification at the end of each block before moving on. If the verification returns output, pkill and investigate — do not proceed. - The discipline applies to Python + C servers too (they block ports even without the OOM risk). Documented both crashes explicitly so future sessions know this rule has teeth. --- CLAUDE.md | 48 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index afacd23..22ad014 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -63,18 +63,46 @@ grows r15 monotonically. When r15 hits r13 (heap limit), `heap_grow` mmaps ANOTHER 64 MB chunk. **Nothing is ever freed.** A long-running asm server leaks ~64 MB every few thousand requests -until it OOMs the machine. This actually crashed fox's machine on -2026-04-16 — a benchmark server grew to 19.3 GB RSS / 30 GB virt. +until it OOMs the machine. Fox's machine has been crashed twice by +this: once on 2026-04-16 (19.3 GB RSS) and once on 2026-04-17. Both +times blackops spawned an asm server in the background for testing +and failed to verify it was gone before moving on. The first crash +added this discipline section; the second proved the discipline +needed teeth. Hence the MANDATORY checklist below. -**Never leave an asm server running in the background.** Specifically: +**MANDATORY pattern for any asm server test in a shell block:** -- Every background spawn in a bench script gets an EXIT trap: - `trap 'kill ${PIDS[@]} 2>/dev/null' EXIT` -- For sustained-throughput benchmarks, bound iterations **inside the - .lsp** (e.g., `(if (> n 1000) (exit 0) (loop (+ n 1)))`) — never - rely on external kill signals alone. -- Before declaring a session done, verify: - `ps -u fox -o pid,rss,cmd | grep uncommonlisp | grep -v grep` (no output = clean) +```bash +# 1. set -e so any failure aborts before leaving a zombie +set -e +# 2. trap ALWAYS fires, even if a later command fails or the script is killed +trap 'pkill -9 -u "$USER" -f "examples/http-server|asm/uncommonlisp.*bench" 2>/dev/null || true' EXIT INT TERM +# 3. wrap the server in `timeout` so even a total trap failure has a hard ceiling +timeout 30 asm/uncommonlisp < server.lsp & +SPID=$! +# 4. do the work ... (client requests, RSS measurements, etc.) +# 5. explicit kill + wait, not just "assume the trap handles it" +kill -9 $SPID 2>/dev/null; wait $SPID 2>/dev/null +# 6. VERIFY before ending the block +pgrep -u "$USER" -f 'asm/uncommonlisp|examples/http-server' && exit 1 || true +``` + +Additional rules: + +- Never start a bg asm process without `timeout N` wrapping it, even + for a "quick test." The mental overhead is zero; the consequence + of forgetting is a crashed machine. +- Bound iterations **inside the .lsp** (e.g., `(if (> n 1000) (exit 0) + (loop (+ n 1)))`) — never rely on external kill signals alone. +- After any block that spawned background processes, run: + `pgrep -u "$USER" -f 'asm/uncommonlisp|examples/http-server'` + and expect NO output. If output, kill with `pkill -9 -f ...` and + investigate. Do not move on. +- Use `examples/http-server.lsp`'s built-in `*max-requests* = 50000` + ceiling (never comment out, never raise for long-running tests). +- Python + C have real GCs; the hazard is asm-specific. But treat + backgrounded Python + C servers with the same hygiene — they + block ports and confuse later tests. - C has Boehm GC via `GC_MALLOC`. Python has Python's GC. Asm has neither. Risk scales with how long the asm process lives.