CLAUDE.md: asm server discipline with teeth after 2nd crash

Second crash on 2026-04-17 proved the earlier "Asm memory discipline"
section was aspirational. Rules were documented, blackops kept
spawning backgrounded asm servers for quick RSS checks during the
heap-snapshot and RPC work, and one escaped again.

This revision makes the discipline procedural:

- Wrap EVERY backgrounded asm spawn with `timeout N`, even for a
  "quick test" — the mental overhead is zero; the consequence of
  forgetting is a crashed machine.
- Trap EXIT/INT/TERM at the top of every block that backgrounds
  anything, not just benchmark scripts.
- MANDATORY pgrep verification at the end of each block before
  moving on. If the verification returns output, pkill and
  investigate — do not proceed.
- The discipline applies to Python + C servers too (they block
  ports even without the OOM risk).

Documented both crashes explicitly so future sessions know this
rule has teeth.
This commit is contained in:
russell@unturf.com 2026-04-16 21:16:17 -04:00
parent f7ce590099
commit 0f894734a2

View file

@ -63,18 +63,46 @@ grows r15 monotonically. When r15 hits r13 (heap limit),
`heap_grow` mmaps ANOTHER 64 MB chunk. **Nothing is ever freed.**
A long-running asm server leaks ~64 MB every few thousand requests
until it OOMs the machine. This actually crashed fox's machine on
2026-04-16 — a benchmark server grew to 19.3 GB RSS / 30 GB virt.
until it OOMs the machine. Fox's machine has been crashed twice by
this: once on 2026-04-16 (19.3 GB RSS) and once on 2026-04-17. Both
times blackops spawned an asm server in the background for testing
and failed to verify it was gone before moving on. The first crash
added this discipline section; the second proved the discipline
needed teeth. Hence the MANDATORY checklist below.
**Never leave an asm server running in the background.** Specifically:
**MANDATORY pattern for any asm server test in a shell block:**
- Every background spawn in a bench script gets an EXIT trap:
`trap 'kill ${PIDS[@]} 2>/dev/null' EXIT`
- For sustained-throughput benchmarks, bound iterations **inside the
.lsp** (e.g., `(if (> n 1000) (exit 0) (loop (+ n 1)))`) — never
rely on external kill signals alone.
- Before declaring a session done, verify:
`ps -u fox -o pid,rss,cmd | grep uncommonlisp | grep -v grep` (no output = clean)
```bash
# 1. set -e so any failure aborts before leaving a zombie
set -e
# 2. trap ALWAYS fires, even if a later command fails or the script is killed
trap 'pkill -9 -u "$USER" -f "examples/http-server|asm/uncommonlisp.*bench" 2>/dev/null || true' EXIT INT TERM
# 3. wrap the server in `timeout` so even a total trap failure has a hard ceiling
timeout 30 asm/uncommonlisp < server.lsp &
SPID=$!
# 4. do the work ... (client requests, RSS measurements, etc.)
# 5. explicit kill + wait, not just "assume the trap handles it"
kill -9 $SPID 2>/dev/null; wait $SPID 2>/dev/null
# 6. VERIFY before ending the block
pgrep -u "$USER" -f 'asm/uncommonlisp|examples/http-server' && exit 1 || true
```
Additional rules:
- Never start a bg asm process without `timeout N` wrapping it, even
for a "quick test." The mental overhead is zero; the consequence
of forgetting is a crashed machine.
- Bound iterations **inside the .lsp** (e.g., `(if (> n 1000) (exit 0)
(loop (+ n 1)))`) — never rely on external kill signals alone.
- After any block that spawned background processes, run:
`pgrep -u "$USER" -f 'asm/uncommonlisp|examples/http-server'`
and expect NO output. If output, kill with `pkill -9 -f ...` and
investigate. Do not move on.
- Use `examples/http-server.lsp`'s built-in `*max-requests* = 50000`
ceiling (never comment out, never raise for long-running tests).
- Python + C have real GCs; the hazard is asm-specific. But treat
backgrounded Python + C servers with the same hygiene — they
block ports and confuse later tests.
- C has Boehm GC via `GC_MALLOC`. Python has Python's GC. Asm has
neither. Risk scales with how long the asm process lives.