v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.
Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).
make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
Single canonical entry point that ties together the four existing
bench-related docs (qa-modes-bench.md / bench-maxing.md /
bench-emergent-design.md / qa-modes-bench-2026-04-30.md) plus the
make targets, fixtures, and bench-row schema.
Sections:
1. Two harnesses, two purposes
bench/qa_sweep.py — curated regression bench
scripts/bench_emergent.py — random-word stress test
2. Four question fixtures (75 / 28 / 9 / 1 / smoke) with cell
sizes + use cases per fixture
3. Signal floor (5pp / n=3 × 9 = 27 / vLLM c=3-4 saturation)
4. Make targets cheat sheet (bench-qa, bench-emergent,
--policy KEY=VALUE A/B pattern, --resume)
5. Bench-row schema — every field a row carries
(identity / verdict / diagnostics / preflight projection /
quantifier classifier / capacity / directive compliance / time)
6. Where headlines live (qa-modes-bench.md addenda, per-ticket
§12/§13 bench sections)
7. bench-emergent log shape + #000006 rolling-amend pattern
8. How to run a focused A/B (the four-cell pattern from #000008)
9. How to interpret results (STRICT-rate, mean-ratio,
UNGROUNDED-rate, FORMAT_COLLAPSED rate, violation kind
distribution, audit-line tails)
10. Operator commands cheat sheet (--show-preflight,
--apply-quantifier-caps, --reject-broad, --soft-preflight,
Makefile shortcuts)
CLAUDE.md docs index updated to point to benchmarks.md as the
"read first" entry for bench work, and to add bench-emergent-design.md
to the index (was missing).
