From 9d0015b4d4e663d7bd2912ec02bcf59351bda6f6 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Thu, 21 May 2026 08:38:07 -0400 Subject: [PATCH] =?UTF-8?q?feat(#000060):=20bench/jaggedness.py=20?= =?UTF-8?q?=E2=80=94=20deterministic=20retrieval=20jaggedness=20instrument?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v1 of the same-model substrate-delta harness's non-jagged metric. For one corpus title, surface-perturb its question (numeral / accent / hyphen / honorific / amp / brit) preserving the referent, then ask whether retrieval surfaces the SAME target for canonical vs perturbed phrasing. J_norm = XOR disagreement rate @k (lower = less jagged); graded mean |Δrank| catches rank instability the binary metric misses. Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no 5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe + mine_questions._surface_variant. Feeds #000012 ForkScore ΔJaggednessReduction. A-vs-C answer-quality arm already exists under #000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm jaggedness delta remain open (ticket §8). make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8 --- Makefile | 22 +- bench/jaggedness.py | 200 ++++++++++++++++++ .../jaggedness_2026-05-21T01-56-17Z.json | 147 +++++++++++++ docs/benchmarks.md | 38 ++++ ...defg-same-model-substrate-delta-harness.md | 6 +- 5 files changed, 411 insertions(+), 2 deletions(-) create mode 100644 bench/jaggedness.py create mode 100644 bench/results/jaggedness_2026-05-21T01-56-17Z.json diff --git a/Makefile b/Makefile index 4bffde1..5174aeb 100644 --- a/Makefile +++ b/Makefile @@ -39,7 +39,7 @@ SEARCH_Q ?= computer bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \ bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab control-sweep rapl-access rapl-access-revoke clean clean-db clean-data help \ textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \ - crawl-textbooks crawl-textbooks-stats textbook textbook-list + crawl-textbooks crawl-textbooks-stats textbook textbook-list bench-jaggedness all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats @@ -256,6 +256,26 @@ control-sweep: ## #000057: model×framing control sweep [CONTROL_SWEEP_N / _WORK --shards-dir $(SHARDS_DIR) \ --out-dir $(BENCH_QA_OUT) +# #000060 retrieval jaggedness: does the substrate surface the SAME +# target across surface-preserving question perturbations (numeral / +# accent / hyphen / honorific / amp / brit)? Deterministic — query +# --dry-run only, no LLM / no verifier / no judge / no n=3 noise (the +# recall_at_k discipline). J_norm = canonical-vs-perturbed surfacing +# disagreement rate; lower = more non-jagged. Feeds #000012 ForkScore +# ΔJaggednessReduction. Reuses recall_at_k.probe + mine_questions. +# make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8 +JAGGED_CLASSES ?= numeral,accent,hyphen,honorific,amp,brit +JAGGED_LIMIT ?= 40 +JAGGED_K ?= 8 +JAGGED_CONC ?= 4 +bench-jaggedness: bootstrap ## #000060: deterministic retrieval jaggedness across surface perturbations [JAGGED_LIMIT / _K / _CLASSES ...] + $(PY) bench/jaggedness.py \ + --classes $(JAGGED_CLASSES) \ + --limit $(JAGGED_LIMIT) \ + --k $(JAGGED_K) \ + --conc $(JAGGED_CONC) \ + --shards-dir $(SHARDS_DIR) + # Progressive-AND / DF-filter fixture: 9 questions chosen to exercise # the OR-fallback and progressive-AND drop paths. Use this for any # retrieval-side A/B (alternative search backends, synonym/rerank diff --git a/bench/jaggedness.py b/bench/jaggedness.py new file mode 100644 index 0000000..343fef5 --- /dev/null +++ b/bench/jaggedness.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""#000060 — deterministic retrieval jaggedness instrument. + +Hassabis-style "non-jagged intelligence" (report §73) operationalized +on the cheapest possible substrate-axis: retrieval. For one referent +(a corpus title), a user can phrase the question many surface ways +that ALL preserve the referent — Roman vs ordinal numeral, accented +vs ASCII, hyphen vs space, abbreviated vs spelled-out honorific, & vs +"and", British vs American spelling. A NON-jagged retriever surfaces +the SAME target article for every such phrasing. A jagged one finds +it for "Henry VIII" and misses it for "Henry the eighth". + +Jaggedness here is therefore the *disagreement* between a question's +canonical phrasing (title verbatim) and its surface-perturbed +phrasing, on whether retrieval surfaces the known target — measured +deterministically (`query --dry-run`, no LLM, no verifier, no judge, +no n=3 noise, no 5pp floor — the recall_at_k discipline). + + J_norm (binary) = fraction of titles where canonical & perturbed + DISAGREE on surfacing the target @k (XOR). Lower + is better (more non-jagged). + graded jaggedness = mean |rank_canonical - rank_perturbed| over + titles where BOTH surfaced (rank instability the + binary metric can't see — a fold that shoves the + target from rank 1 to rank 7 is jagged even if + both still land in top-k; cf. recall_at_k's + rank-not-just-presence rule). + +This is necessary-not-sufficient for STRICT (it measures retrieval +surfacing, exactly like recall_at_k) and complements — never replaces +— the curated adversarial set. It feeds #000012's ForkScore +ΔJaggednessReduction term: a retrieval-fold lever that lowers J_norm +without lowering recall is a non-jagged win. + +The A-vs-C answer-quality jaggedness DELTA (does the substrate make +ANSWER quality more consistent across variants than the bare model?) +is the expensive LLM+judge version — that rides bench/control_sweep.py +over variant groups and is gated v2, not built here. This v1 is the +deterministic retrieval floor. + + usage: jaggedness.py [--classes numeral,accent,hyphen,honorific,amp,brit] + [--limit 40] [--k 8] [--conc 4] +""" +from __future__ import annotations + +import argparse +import concurrent.futures as cf +import json +import statistics +import sys +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from bench.mine_questions import mine # noqa: E402 +from bench.recall_at_k import probe # noqa: E402 + +# Orthographic / phrasing classes only. 'stale' is excluded on +# purpose: it is a temporal fabrication-bait class (control_ab), not a +# surface-preserving perturbation, so it carries no jaggedness signal. +_DEFAULT_CLASSES = ["numeral", "accent", "hyphen", "honorific", "amp", "brit"] + + +def _canonical_question(title: str, cls: str) -> str: + """The reference phrasing: the title VERBATIM, same stem the + perturbed form uses, so canonical-vs-perturbed isolates the + surface fold and nothing else.""" + if cls in ("numeral", "stale"): + return f"who was {title}?" + return f"what is {title}?" + + +def _measure(item: dict, cls: str, k: int) -> dict: + """Probe canonical & perturbed phrasings of ONE referent.""" + title = item["target_title"] + canon_q = _canonical_question(title, cls) + pert_q = item["question"] + canon_rank = probe({"question": canon_q, "target_title": title}, k) + pert_rank = probe({"question": pert_q, "target_title": title}, k) + surf_canon = 0 <= canon_rank < k + surf_pert = 0 <= pert_rank < k + return { + "cls": cls, + "title": title, + "canon_q": canon_q, + "pert_q": pert_q, + "canon_rank": canon_rank, + "pert_rank": pert_rank, + "surf_canon": surf_canon, + "surf_pert": surf_pert, + "jagged": surf_canon != surf_pert, # XOR — surfacing disagreement + } + + +def _aggregate(records: list[dict], k: int) -> dict: + n = len(records) + if not n: + return {"n": 0} + jagged = [r for r in records if r["jagged"]] + both = [r for r in records + if r["surf_canon"] and r["surf_pert"]] + rank_gaps = [abs(r["canon_rank"] - r["pert_rank"]) for r in both] + by_cls: dict[str, dict] = {} + for r in records: + c = by_cls.setdefault(r["cls"], {"n": 0, "jagged": 0, + "canon_surf": 0, "pert_surf": 0}) + c["n"] += 1 + c["jagged"] += int(r["jagged"]) + c["canon_surf"] += int(r["surf_canon"]) + c["pert_surf"] += int(r["surf_pert"]) + return { + "n": n, + "k": k, + "j_norm_binary": len(jagged) / n, + "jagged_titles": len(jagged), + "canon_recall@k": sum(r["surf_canon"] for r in records) / n, + "pert_recall@k": sum(r["surf_pert"] for r in records) / n, + "graded_mean_rank_gap": (statistics.mean(rank_gaps) + if rank_gaps else 0.0), + "graded_n_both_surfaced": len(both), + "by_class": by_cls, + } + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--classes", default=",".join(_DEFAULT_CLASSES)) + ap.add_argument("--limit", type=int, default=40, + help="max referents mined per class") + ap.add_argument("--k", type=int, default=8) + ap.add_argument("--conc", type=int, default=4) + ap.add_argument("--shards-dir", + default=str(Path.home() / ".arborist" / "shards")) + ap.add_argument("--out-dir", default="bench/results") + ap.add_argument("--remine", action="store_true", + help="re-mine from shards instead of loading the " + "committed qa_questions__map.json fixtures " + "(slow — rescans the corpus for rare classes)") + a = ap.parse_args() + + classes = [c for c in a.classes.split(",") if c.strip()] + jobs: list[tuple[dict, str]] = [] + for cls in classes: + # Prefer the pre-mined committed fixture (instant) over a fresh + # mine() — rare classes (amp/brit) make mine() rescan the whole + # corpus to fill the limit, which is the slow path. Fall back to + # mining only when the fixture is absent. + fixture = ROOT / "bench" / f"qa_questions_{cls}_map.json" + if fixture.exists() and not a.remine: + items = json.loads(fixture.read_text())[:a.limit] + src = "fixture" + else: + items = mine(a.shards_dir, a.limit, cls) + src = "mined" + jobs.extend((it, cls) for it in items) + print(f" {src} {len(items)} {cls} referents") + if not jobs: + print("no referents — check --classes / fixtures / --shards-dir") + return 1 + + with cf.ThreadPoolExecutor(max_workers=a.conc) as ex: + records = list(ex.map(lambda j: _measure(j[0], j[1], a.k), jobs)) + + agg = _aggregate(records, a.k) + ts = time.strftime("%Y-%m-%dT%H-%M-%SZ", time.gmtime()) + outp = Path(a.out_dir) / f"jaggedness_{ts}.json" + outp.parent.mkdir(parents=True, exist_ok=True) + outp.write_text(json.dumps( + {"meta": {"classes": classes, "limit": a.limit, "k": a.k, + "ts": ts}, "summary": agg, "records": records}, + ensure_ascii=False, indent=2)) + + print(f"\n=== #000060 retrieval jaggedness (n={agg['n']}, " + f"deterministic, no LLM) ===") + print(f"log={outp}") + print(f" canonical recall@{a.k}: {agg['canon_recall@k']:.0%} " + f"perturbed recall@{a.k}: {agg['pert_recall@k']:.0%}") + print(f" J_norm (canon/pert surfacing disagreement): " + f"{agg['j_norm_binary']:.0%} ({agg['jagged_titles']}/{agg['n']} " + f"jagged) — lower is more non-jagged") + print(f" graded mean |Δrank| (both surfaced, n=" + f"{agg['graded_n_both_surfaced']}): " + f"{agg['graded_mean_rank_gap']:.2f}") + print(" per-class jaggedness:") + for cls, c in sorted(agg["by_class"].items()): + jr = c["jagged"] / c["n"] if c["n"] else 0.0 + print(f" {cls:10} {c['jagged']:>2}/{c['n']:<2} jagged " + f"({jr:.0%}) canon {c['canon_surf']}/{c['n']} " + f"pert {c['pert_surf']}/{c['n']}") + print("\nNOTE: retrieval-surfacing jaggedness — necessary-not-" + "sufficient for STRICT; complements (never replaces) the " + "curated adversarial set. A-vs-C answer-quality jaggedness " + "delta = control_sweep over variant groups (v2, gated).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/bench/results/jaggedness_2026-05-21T01-56-17Z.json b/bench/results/jaggedness_2026-05-21T01-56-17Z.json new file mode 100644 index 0000000..7879c33 --- /dev/null +++ b/bench/results/jaggedness_2026-05-21T01-56-17Z.json @@ -0,0 +1,147 @@ +{ + "meta": { + "classes": [ + "numeral", + "accent" + ], + "limit": 5, + "k": 8, + "ts": "2026-05-21T01-56-17Z" + }, + "summary": { + "n": 10, + "k": 8, + "j_norm_binary": 0.0, + "jagged_titles": 0, + "canon_recall@k": 0.8, + "pert_recall@k": 0.8, + "graded_mean_rank_gap": 0.75, + "graded_n_both_surfaced": 8, + "by_class": { + "numeral": { + "n": 5, + "jagged": 0, + "canon_surf": 3, + "pert_surf": 3 + }, + "accent": { + "n": 5, + "jagged": 0, + "canon_surf": 5, + "pert_surf": 5 + } + } + }, + "records": [ + { + "cls": "numeral", + "title": "Albert III", + "canon_q": "who was Albert III?", + "pert_q": "who was Albert the third?", + "canon_rank": -1, + "pert_rank": -1, + "surf_canon": false, + "surf_pert": false, + "jagged": false + }, + { + "cls": "numeral", + "title": "Ahmed III", + "canon_q": "who was Ahmed III?", + "pert_q": "who was Ahmed the third?", + "canon_rank": -1, + "pert_rank": -1, + "surf_canon": false, + "surf_pert": false, + "jagged": false + }, + { + "cls": "numeral", + "title": "Alaric I", + "canon_q": "who was Alaric I?", + "pert_q": "who was Alaric the first?", + "canon_rank": 0, + "pert_rank": 0, + "surf_canon": true, + "surf_pert": true, + "jagged": false + }, + { + "cls": "numeral", + "title": "Alexander I of Epirus", + "canon_q": "who was Alexander I of Epirus?", + "pert_q": "who was Alexander the first of epirus?", + "canon_rank": 0, + "pert_rank": 0, + "surf_canon": true, + "surf_pert": true, + "jagged": false + }, + { + "cls": "numeral", + "title": "Alexander II of Scotland", + "canon_q": "who was Alexander II of Scotland?", + "pert_q": "who was Alexander the second of scotland?", + "canon_rank": 6, + "pert_rank": 6, + "surf_canon": true, + "surf_pert": true, + "jagged": false + }, + { + "cls": "accent", + "title": "Casa Batlló", + "canon_q": "what is Casa Batlló?", + "pert_q": "what is Casa Batllo?", + "canon_rank": 0, + "pert_rank": 0, + "surf_canon": true, + "surf_pert": true, + "jagged": false + }, + { + "cls": "accent", + "title": "André-Marie Ampère", + "canon_q": "what is André-Marie Ampère?", + "pert_q": "what is Andre-Marie Ampere?", + "canon_rank": 0, + "pert_rank": 0, + "surf_canon": true, + "surf_pert": true, + "jagged": false + }, + { + "cls": "accent", + "title": "Antoni Gaudí", + "canon_q": "what is Antoni Gaudí?", + "pert_q": "what is Antoni Gaudi?", + "canon_rank": 0, + "pert_rank": 0, + "surf_canon": true, + "surf_pert": true, + "jagged": false + }, + { + "cls": "accent", + "title": "Alcobaça (Portugal)", + "canon_q": "what is Alcobaça (Portugal)?", + "pert_q": "what is Alcobaca (Portugal)?", + "canon_rank": 0, + "pert_rank": 0, + "surf_canon": true, + "surf_pert": true, + "jagged": false + }, + { + "cls": "accent", + "title": "Bifröst", + "canon_q": "what is Bifröst?", + "pert_q": "what is Bifrost?", + "canon_rank": 0, + "pert_rank": 6, + "surf_canon": true, + "surf_pert": true, + "jagged": false + } + ] +} \ No newline at end of file diff --git a/docs/benchmarks.md b/docs/benchmarks.md index dce0940..53852df 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -335,3 +335,41 @@ make query Q="..." BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 \ ``` See the `Makefile` directly for the full set. + +## 11. Deterministic retrieval instruments (no-LLM) + +The bench-maxing rule (CLAUDE.md): when a lever's failure class is +below the n=3 / 5pp noise floor, fix the instrument — mine +ground-truth-carrying questions and grade by deterministic retrieval, +not by the LLM verifier. These instruments take `query --dry-run` +output only: no LLM, no verifier, no judge, no noise floor, scalable +to the corpus. They measure *retrieval surfacing* — +necessary-not-sufficient for STRICT, and they **complement, never +replace**, the curated adversarial set (the verifier-honesty gate). + +- **`bench/mine_questions.py`** — mines questions whose target article + is known by construction (surface-variant of a corpus title). + Classes: `numeral` (Roman↔ordinal), `accent`, `hyphen`, + `honorific` (Mt↔Mount), `amp` (&↔and), `brit` (US↔UK spelling), + `stale` (temporal fabrication-bait, control_ab only). +- **`bench/recall_at_k.py`** — recall@1/@3/@k of the known target on + a mined fixture. Returns rank, so recall at every k is free from one + retrieval (a too-lenient k hides a rank-only lift — report @1/@3/@k). +- **`bench/jaggedness.py`** (`make bench-jaggedness`, #000060) — does + retrieval surface the SAME target for a question's *canonical* + phrasing (title verbatim) AND its *surface-perturbed* phrasing? A + non-jagged retriever agrees; a jagged one finds "Henry VIII" but + misses "Henry the eighth". Reuses `recall_at_k.probe` + + `mine_questions._surface_variant`. + - `J_norm` (binary) = fraction of titles where canonical & perturbed + DISAGREE on surfacing the target @k (XOR). Lower = more non-jagged. + - graded mean `|Δrank|` over titles where both surfaced — rank + instability the binary metric can't see. + - Distinguishes a true recall-miss (both phrasings miss = not + jagged) from jaggedness (one surfaces, one doesn't). Feeds + #000012's ForkScore `ΔJaggednessReduction`: a retrieval fold that + lowers `J_norm` without lowering recall is a non-jagged win. + - The A-vs-C *answer-quality* jaggedness delta (does the substrate + make answers more consistent across variants than the bare model?) + is the LLM+judge version — rides `bench/control_sweep.py` over + variant groups, gated v2. diff --git a/docs/tickets/ticket-000060-habcdefg-same-model-substrate-delta-harness.md b/docs/tickets/ticket-000060-habcdefg-same-model-substrate-delta-harness.md index e699fc9..e77ecc6 100644 --- a/docs/tickets/ticket-000060-habcdefg-same-model-substrate-delta-harness.md +++ b/docs/tickets/ticket-000060-habcdefg-same-model-substrate-delta-harness.md @@ -1,6 +1,10 @@ # Ticket #000060 — H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + performance curvature) -**Status:** open · awaiting go/no-go +**Status:** in progress · v1 deterministic jaggedness instrument +landed 2026-05-21 (`bench/jaggedness.py` + `make bench-jaggedness`); +A-vs-C answer-quality arm already existed under #000057 +(`control_ab.py` / `control_sweep.py`) — NOT rebuilt. Curvature + +LLM-arm jaggedness delta still open (see §8). **Opened:** 2026-05-20 **Scope:** The report's "decisive proof" — a harness that runs the SAME base model with and without the arborist substrate over