CLAUDE.md: asm server discipline with teeth after 2nd crash
Second crash on 2026-04-17 proved the earlier "Asm memory discipline" section was aspirational. Rules were documented, blackops kept spawning backgrounded asm servers for quick RSS checks during the heap-snapshot and RPC work, and one escaped again. This revision makes the discipline procedural: - Wrap EVERY backgrounded asm spawn with `timeout N`, even for a "quick test" — the mental overhead is zero; the consequence of forgetting is a crashed machine. - Trap EXIT/INT/TERM at the top of every block that backgrounds anything, not just benchmark scripts. - MANDATORY pgrep verification at the end of each block before moving on. If the verification returns output, pkill and investigate — do not proceed. - The discipline applies to Python + C servers too (they block ports even without the OOM risk). Documented both crashes explicitly so future sessions know this rule has teeth.
This commit is contained in:
parent
f7ce590099
commit
0f894734a2
1 changed files with 38 additions and 10 deletions
48
CLAUDE.md
48
CLAUDE.md
|
|
@ -63,18 +63,46 @@ grows r15 monotonically. When r15 hits r13 (heap limit),
|
|||
`heap_grow` mmaps ANOTHER 64 MB chunk. **Nothing is ever freed.**
|
||||
|
||||
A long-running asm server leaks ~64 MB every few thousand requests
|
||||
until it OOMs the machine. This actually crashed fox's machine on
|
||||
2026-04-16 — a benchmark server grew to 19.3 GB RSS / 30 GB virt.
|
||||
until it OOMs the machine. Fox's machine has been crashed twice by
|
||||
this: once on 2026-04-16 (19.3 GB RSS) and once on 2026-04-17. Both
|
||||
times blackops spawned an asm server in the background for testing
|
||||
and failed to verify it was gone before moving on. The first crash
|
||||
added this discipline section; the second proved the discipline
|
||||
needed teeth. Hence the MANDATORY checklist below.
|
||||
|
||||
**Never leave an asm server running in the background.** Specifically:
|
||||
**MANDATORY pattern for any asm server test in a shell block:**
|
||||
|
||||
- Every background spawn in a bench script gets an EXIT trap:
|
||||
`trap 'kill ${PIDS[@]} 2>/dev/null' EXIT`
|
||||
- For sustained-throughput benchmarks, bound iterations **inside the
|
||||
.lsp** (e.g., `(if (> n 1000) (exit 0) (loop (+ n 1)))`) — never
|
||||
rely on external kill signals alone.
|
||||
- Before declaring a session done, verify:
|
||||
`ps -u fox -o pid,rss,cmd | grep uncommonlisp | grep -v grep` (no output = clean)
|
||||
```bash
|
||||
# 1. set -e so any failure aborts before leaving a zombie
|
||||
set -e
|
||||
# 2. trap ALWAYS fires, even if a later command fails or the script is killed
|
||||
trap 'pkill -9 -u "$USER" -f "examples/http-server|asm/uncommonlisp.*bench" 2>/dev/null || true' EXIT INT TERM
|
||||
# 3. wrap the server in `timeout` so even a total trap failure has a hard ceiling
|
||||
timeout 30 asm/uncommonlisp < server.lsp &
|
||||
SPID=$!
|
||||
# 4. do the work ... (client requests, RSS measurements, etc.)
|
||||
# 5. explicit kill + wait, not just "assume the trap handles it"
|
||||
kill -9 $SPID 2>/dev/null; wait $SPID 2>/dev/null
|
||||
# 6. VERIFY before ending the block
|
||||
pgrep -u "$USER" -f 'asm/uncommonlisp|examples/http-server' && exit 1 || true
|
||||
```
|
||||
|
||||
Additional rules:
|
||||
|
||||
- Never start a bg asm process without `timeout N` wrapping it, even
|
||||
for a "quick test." The mental overhead is zero; the consequence
|
||||
of forgetting is a crashed machine.
|
||||
- Bound iterations **inside the .lsp** (e.g., `(if (> n 1000) (exit 0)
|
||||
(loop (+ n 1)))`) — never rely on external kill signals alone.
|
||||
- After any block that spawned background processes, run:
|
||||
`pgrep -u "$USER" -f 'asm/uncommonlisp|examples/http-server'`
|
||||
and expect NO output. If output, kill with `pkill -9 -f ...` and
|
||||
investigate. Do not move on.
|
||||
- Use `examples/http-server.lsp`'s built-in `*max-requests* = 50000`
|
||||
ceiling (never comment out, never raise for long-running tests).
|
||||
- Python + C have real GCs; the hazard is asm-specific. But treat
|
||||
backgrounded Python + C servers with the same hygiene — they
|
||||
block ports and confuse later tests.
|
||||
- C has Boehm GC via `GC_MALLOC`. Python has Python's GC. Asm has
|
||||
neither. Risk scales with how long the asm process lives.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue