feat(#000060): bench/jaggedness.py — deterministic retrieval jaggedness instrument
v1 of the same-model substrate-delta harness's non-jagged metric. For one corpus title, surface-perturb its question (numeral / accent / hyphen / honorific / amp / brit) preserving the referent, then ask whether retrieval surfaces the SAME target for canonical vs perturbed phrasing. J_norm = XOR disagreement rate @k (lower = less jagged); graded mean |Δrank| catches rank instability the binary metric misses. Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no 5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe + mine_questions._surface_variant. Feeds #000012 ForkScore ΔJaggednessReduction. A-vs-C answer-quality arm already exists under #000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm jaggedness delta remain open (ticket §8). make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
This commit is contained in:
parent
7a43ceb699
commit
9d0015b4d4
5 changed files with 411 additions and 2 deletions
|
|
@ -335,3 +335,41 @@ make query Q="..." BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 \
|
|||
```
|
||||
|
||||
See the `Makefile` directly for the full set.
|
||||
|
||||
## 11. Deterministic retrieval instruments (no-LLM)
|
||||
|
||||
The bench-maxing rule (CLAUDE.md): when a lever's failure class is
|
||||
below the n=3 / 5pp noise floor, fix the instrument — mine
|
||||
ground-truth-carrying questions and grade by deterministic retrieval,
|
||||
not by the LLM verifier. These instruments take `query --dry-run`
|
||||
output only: no LLM, no verifier, no judge, no noise floor, scalable
|
||||
to the corpus. They measure *retrieval surfacing* —
|
||||
necessary-not-sufficient for STRICT, and they **complement, never
|
||||
replace**, the curated adversarial set (the verifier-honesty gate).
|
||||
|
||||
- **`bench/mine_questions.py`** — mines questions whose target article
|
||||
is known by construction (surface-variant of a corpus title).
|
||||
Classes: `numeral` (Roman↔ordinal), `accent`, `hyphen`,
|
||||
`honorific` (Mt↔Mount), `amp` (&↔and), `brit` (US↔UK spelling),
|
||||
`stale` (temporal fabrication-bait, control_ab only).
|
||||
- **`bench/recall_at_k.py`** — recall@1/@3/@k of the known target on
|
||||
a mined fixture. Returns rank, so recall at every k is free from one
|
||||
retrieval (a too-lenient k hides a rank-only lift — report @1/@3/@k).
|
||||
- **`bench/jaggedness.py`** (`make bench-jaggedness`, #000060) — does
|
||||
retrieval surface the SAME target for a question's *canonical*
|
||||
phrasing (title verbatim) AND its *surface-perturbed* phrasing? A
|
||||
non-jagged retriever agrees; a jagged one finds "Henry VIII" but
|
||||
misses "Henry the eighth". Reuses `recall_at_k.probe` +
|
||||
`mine_questions._surface_variant`.
|
||||
- `J_norm` (binary) = fraction of titles where canonical & perturbed
|
||||
DISAGREE on surfacing the target @k (XOR). Lower = more non-jagged.
|
||||
- graded mean `|Δrank|` over titles where both surfaced — rank
|
||||
instability the binary metric can't see.
|
||||
- Distinguishes a true recall-miss (both phrasings miss = not
|
||||
jagged) from jaggedness (one surfaces, one doesn't). Feeds
|
||||
#000012's ForkScore `ΔJaggednessReduction`: a retrieval fold that
|
||||
lowers `J_norm` without lowering recall is a non-jagged win.
|
||||
- The A-vs-C *answer-quality* jaggedness delta (does the substrate
|
||||
make answers more consistent across variants than the bare model?)
|
||||
is the LLM+judge version — rides `bench/control_sweep.py` over
|
||||
variant groups, gated v2.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,10 @@
|
|||
# Ticket #000060 — H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + performance curvature)
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Status:** in progress · v1 deterministic jaggedness instrument
|
||||
landed 2026-05-21 (`bench/jaggedness.py` + `make bench-jaggedness`);
|
||||
A-vs-C answer-quality arm already existed under #000057
|
||||
(`control_ab.py` / `control_sweep.py`) — NOT rebuilt. Curvature +
|
||||
LLM-arm jaggedness delta still open (see §8).
|
||||
**Opened:** 2026-05-20
|
||||
**Scope:** The report's "decisive proof" — a harness that runs the
|
||||
SAME base model with and without the arborist substrate over
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue