feat(#000060): bench/jaggedness.py — deterministic retrieval jaggedness instrument

v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.

Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).

make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
This commit is contained in:
russell@unturf.com 2026-05-21 08:38:07 -04:00
parent 7a43ceb699
commit 9d0015b4d4
No known key found for this signature in database
5 changed files with 411 additions and 2 deletions

View file

@ -335,3 +335,41 @@ make query Q="..." BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 \
```
See the `Makefile` directly for the full set.
## 11. Deterministic retrieval instruments (no-LLM)
The bench-maxing rule (CLAUDE.md): when a lever's failure class is
below the n=3 / 5pp noise floor, fix the instrument — mine
ground-truth-carrying questions and grade by deterministic retrieval,
not by the LLM verifier. These instruments take `query --dry-run`
output only: no LLM, no verifier, no judge, no noise floor, scalable
to the corpus. They measure *retrieval surfacing*
necessary-not-sufficient for STRICT, and they **complement, never
replace**, the curated adversarial set (the verifier-honesty gate).
- **`bench/mine_questions.py`** — mines questions whose target article
is known by construction (surface-variant of a corpus title).
Classes: `numeral` (Roman↔ordinal), `accent`, `hyphen`,
`honorific` (Mt↔Mount), `amp` (&↔and), `brit` (US↔UK spelling),
`stale` (temporal fabrication-bait, control_ab only).
- **`bench/recall_at_k.py`** — recall@1/@3/@k of the known target on
a mined fixture. Returns rank, so recall at every k is free from one
retrieval (a too-lenient k hides a rank-only lift — report @1/@3/@k).
- **`bench/jaggedness.py`** (`make bench-jaggedness`, #000060) — does
retrieval surface the SAME target for a question's *canonical*
phrasing (title verbatim) AND its *surface-perturbed* phrasing? A
non-jagged retriever agrees; a jagged one finds "Henry VIII" but
misses "Henry the eighth". Reuses `recall_at_k.probe` +
`mine_questions._surface_variant`.
- `J_norm` (binary) = fraction of titles where canonical & perturbed
DISAGREE on surfacing the target @k (XOR). Lower = more non-jagged.
- graded mean `|Δrank|` over titles where both surfaced — rank
instability the binary metric can't see.
- Distinguishes a true recall-miss (both phrasings miss = not
jagged) from jaggedness (one surfaces, one doesn't). Feeds
#000012's ForkScore `ΔJaggednessReduction`: a retrieval fold that
lowers `J_norm` without lowering recall is a non-jagged win.
- The A-vs-C *answer-quality* jaggedness delta (does the substrate
make answers more consistent across variants than the bare model?)
is the LLM+judge version — rides `bench/control_sweep.py` over
variant groups, gated v2.

View file

@ -1,6 +1,10 @@
# Ticket #000060 — H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + performance curvature)
**Status:** open · awaiting go/no-go
**Status:** in progress · v1 deterministic jaggedness instrument
landed 2026-05-21 (`bench/jaggedness.py` + `make bench-jaggedness`);
A-vs-C answer-quality arm already existed under #000057
(`control_ab.py` / `control_sweep.py`) — NOT rebuilt. Curvature +
LLM-arm jaggedness delta still open (see §8).
**Opened:** 2026-05-20
**Scope:** The report's "decisive proof" — a harness that runs the
SAME base model with and without the arborist substrate over