The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.
Naming pattern moving forward:
docs/qa-modes-bench.md — living journal (latest run)
docs/qa-modes-bench-2026-04-30.md — historical snapshot
(frozen for the JSON-mode
hardening day)
Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.
Updated references in:
- CLAUDE.md (× 2)
- aborist/qa/prompts.py
- aborist/qa/query.py
- docs/bench-maxing.md
The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
bench-maxing.md gains a 'Bench harness — the speed playbook'
section capturing the 2026-05-02 speed wins as durable doctrine:
- Sample-level shuffled scheduling vs cell-grouped (+58%
throughput, true i.i.d. variance for n>=3)
- Persistent httpx client (TLS handshake amortization)
- Concurrency tuning (vLLM peak at c=3-4, brutal past c=4)
- Per-mode max_context_chars from bench feedback (the bench
is the substrate's voice; let it drive policy)
- --seed for reproducibility
- --resume for stop/start-able bench
- Smoke fixture for inner-loop iteration
- pytest -n auto (3.6× speedup on test suite)
README:
- Updated whitepaper section refs (§6/7/8/9/13) — old refs
pointed at §13.4.11/13.8/13.9 which no longer exist after
the whitepaper rewrite landed.
- 'Tests' section renamed 'Tests & bench' with make targets
for bench-qa, bench-qa-smoke, test-live. Resume + concurrency
semantics surfaced.
Walked the five-step algorithm on CLAUDE.md itself.
Step 1 — requirements check: every section earned its place via a
specific operator-failure context. Sections that had grown into
prose essays got compressed back to load-bearing rules + pointers.
Step 2 — delete:
- Bench-maxing prose section (~100 lines) → moved verbatim to
docs/bench-maxing.md. CLAUDE.md keeps a one-block headline list
pointing at the doc.
- Conventions section: each rule tightened to one paragraph max,
with a `See aborist/qa/verify.py` (or similar) trailing pointer
so the operator can jump straight to the rationale in code.
Verbose duplications between conventions and retrieval-pipeline
sections (e.g. wikitext base prose explained twice) collapsed.
- Retrieval pipeline: each of the 9 stages now references the
relevant module (`qa/concepts.py`, `query.py`) instead of
re-narrating the failure case in prose.
- Architecture tree: minor trim, removed redundant comments
where the filename already names the role.
Step 3 — simplify: source papers section dropped one item that was
duplicate (PDF + RST point to same content); kept the canonical
source.
Steps 4 + 5 — n/a (this is a doc, not a process).
New section: Docs index. Lists every architectural / design doc
in docs/ plus a ticket sub-list (open tickets with their files).
Closes the discoverability gap where TICKETS.md existed but agents
didn't know to look at it.
Net: CLAUDE.md goes from 308 → 311 lines BUT the avg information
density is up — bench prose doesn't bloat the entry-point doc, and
each convention now points to its source. The `docs/bench-maxing.md`
extraction is the real win: separation of "rules I must not break"
(CLAUDE.md) from "discipline I should internalize" (bench-maxing.md).
