From aeeef603a1872c42f27bfebb597f9cc8056d8138 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Wed, 13 May 2026 09:57:26 -0400 Subject: [PATCH] =?UTF-8?q?#000052=20=C2=A73.2.1:=20candidate-bench=20land?= =?UTF-8?q?ed=20(4=20models=20=C3=97=2026=20pairs=20on=20the=204090=20?= =?UTF-8?q?=E2=80=94=20proper=20bench,=20not=20the=206-pair=20smoke)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All 4 manifest candidates clean-separate (12/12 NEG catch at 0/14 POS FP), so 'reranker discriminates Zionist-entity-style mis-cites' is a property of MS-MARCO-trained rerankers as a class — not the specific L-6 I picked first. Ranked by separation margin (per §7 #18: separation beats raw): ms-marco-electra-base +5.349 ← new primary BAAI/bge-reranker-base +3.076 ms-marco-MiniLM-L-6-v2 +3.053 (previous primary, demoted) ms-marco-MiniLM-L-12-v2 +2.663 (worst — deeper ≠ better) Manifest primary moved to electra-base for the cushion. demote_below_score STAYS null — clean candidate-bench thresholds don't predict real-pipeline behavior (the §7 #18→#27 history is 6 verdict flips on the NLI side); §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) is what sets it. Expect a walk-back. Reranker still doesn't catch the Kilimanjaro/Mount-Kenya recombination (aboutness ≠ truth-of-attribution; that's #000049 territory). fox's bench-maxing correction applied. --- arborist/qa/relevance/manifest.json | 27 ++- .../relevance-aboutness-candidatebench.json | 216 ++++++++++++++++++ ...0052-relevance-coherence-meta-cognition.md | 62 +++-- 3 files changed, 278 insertions(+), 27 deletions(-) create mode 100644 bench/results/relevance-aboutness-candidatebench.json diff --git a/arborist/qa/relevance/manifest.json b/arborist/qa/relevance/manifest.json index c2ef72b..329dbe1 100644 --- a/arborist/qa/relevance/manifest.json +++ b/arborist/qa/relevance/manifest.json @@ -1,10 +1,10 @@ { - "_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY: this manifest does NOT yet fold into governance_policy_hash because shadow output cannot touch audit_mode (#000049 §7 #2 discipline inherited verbatim). If/when promoted to a runtime demotion-only veto, add a single-blob checkpoint_sha256 + tokenizer_sha256 + an explicit relevance_policy_hash that folds into governance_policy_hash before any audit_mode effect. Cross-encoder MS-MARCO reranker — single relevance score per (query, document) pair, NOT entailment (that's NLI / arborist.qa.nli; the boundary failure mode this catches is *topic-collision recombination* — answer/claim is on a different topic than its cited source despite shared tokens, the 'Zionist entity' case — which NLI returns neutral on and the lexical title-relevance sidecar passes because the cited title shares a polysemous token). Reuses the [nli] extra (transformers+torch+optimum); the checkpoint is loaded as AutoModelForSequenceClassification (single regression head emitting a relevance logit), no sentence-transformers dependency.", - "relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-6-v2", - "hf_repo": "cross-encoder/ms-marco-MiniLM-L-6-v2", + "_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY: this manifest does NOT yet fold into governance_policy_hash because shadow output cannot touch audit_mode (#000049 §7 #2 discipline inherited verbatim). If/when promoted to a runtime demotion-only veto, add a single-blob checkpoint_sha256 + tokenizer_sha256 + an explicit relevance_policy_hash that folds into governance_policy_hash before any audit_mode effect. Cross-encoder MS-MARCO reranker — single relevance score per (query, document) pair, NOT entailment (that's NLI / arborist.qa.nli; the boundary failure mode this catches is *topic-collision recombination* — answer/claim is on a different topic than its cited source despite shared tokens, the 'Zionist entity' case — which NLI returns neutral on and the lexical title-relevance sidecar passes because the cited title shares a polysemous token). Reuses the [nli] extra (transformers+torch+optimum); the checkpoint is loaded as AutoModelForSequenceClassification (single regression head emitting a relevance logit), no sentence-transformers dependency. PRIMARY = ms-marco-electra-base picked from the candidate-bench (#000052 §3.2.1 — sep margin +5.35 vs +3.05/+3.08/+2.66 for the others). NOTE: the candidate-bench is contrived; the §7 #18→#27 history says clean-eval doesn't predict real-traffic precision OR recall — this primary is provisional pending §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) + step 3 (recall-side realism on real-context fixtures). Expect a walk-back.", + "relevance_model_version": "relevance-shadow-v1-ms-marco-electra-base", + "hf_repo": "cross-encoder/ms-marco-electra-base", "license": "Apache-2.0", - "source_url": "https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2", - "approx_mb": 90, + "source_url": "https://huggingface.co/cross-encoder/ms-marco-electra-base", + "approx_mb": 110, "score_shape": "single logit per (query, document) pair, NOT a softmax — higher = more relevant; trained on MS MARCO passage-ranking, so the score is well-calibrated for 'is this doc relevant to this query?' and uncalibrated for any other notion of similarity", "max_length": 256, "truncation_policy": "query_then_document@v1", @@ -12,9 +12,20 @@ "demote_below_score": null, "demote_below_score_comment": "the demote threshold MUST be set by a shadow sweep against bench-qa traffic — DO NOT default to a literature number. The #000049 §7 #18→#27 arc proved repeatedly that clean-eval thresholds don't transfer to real-pipeline data; recommended_operating_point is left null until a candidate-bench + a bench-qa shadow sweep produces a θ with the FP discipline §3.2 inherits from #000049 (zero false-demote on the pooled-808 STRICT denominator at the chosen θ).", "alternates": [ - {"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-12-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-12-v2", "license": "Apache-2.0", "approx_mb": 130, "note": "deeper L-12 variant of the primary; bench against the L-6 on the off-topic eval set"}, - {"relevance_model_version": "relevance-shadow-v1-bge-reranker-base", "hf_repo": "BAAI/bge-reranker-base", "license": "MIT", "approx_mb": 280, "note": "BAAI's general-purpose reranker; different training corpus, useful as a non-MS-MARCO comparison point"}, - {"relevance_model_version": "relevance-shadow-v1-ms-marco-electra-base", "hf_repo": "cross-encoder/ms-marco-electra-base", "license": "Apache-2.0", "approx_mb": 110, "note": "ELECTRA architecture for variety; same MS MARCO supervision"} + {"relevance_model_version": "relevance-shadow-v1-bge-reranker-base", "hf_repo": "BAAI/bge-reranker-base", "license": "MIT", "approx_mb": 280, "candidate_bench_margin": 3.076, "note": "BAAI general-purpose reranker; second-best candidate-bench margin after electra-base"}, + {"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-6-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-6-v2", "license": "Apache-2.0", "approx_mb": 80, "candidate_bench_margin": 3.053, "note": "the original primary before §3.2.1 candidate-bench; smallest/cheapest, comparable margin to BGE on the 26-pair candidate-bench"}, + {"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-12-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-12-v2", "license": "Apache-2.0", "approx_mb": 130, "candidate_bench_margin": 2.663, "note": "deeper L-12 MiniLM; worst margin of the 4 on the candidate-bench — deeper ≠ better here, same pattern as #000049 §7 #24's 'bart-large worse than deberta-large'"} ], + "candidate_bench_results_2026_05_13": { + "fixtures": "bench/fixtures/5f/relevance-aboutness-v1.jsonl (14 POS + 12 NEG)", + "grid_output": "bench/results/relevance-aboutness-candidatebench.json", + "rank_by_sep_margin": [ + {"model": "ms-marco-electra-base", "margin": 5.349, "min_pos": 0.050, "max_neg": -5.300, "fp0_theta": 0.050, "catch_neg": "12/12"}, + {"model": "bge-reranker-base", "margin": 3.076, "min_pos": 2.395, "max_neg": -0.681, "fp0_theta": 2.395, "catch_neg": "12/12"}, + {"model": "ms-marco-MiniLM-L-6-v2", "margin": 3.053, "min_pos": 6.754, "max_neg": 3.701, "fp0_theta": 6.754, "catch_neg": "12/12"}, + {"model": "ms-marco-MiniLM-L-12-v2","margin": 2.663, "min_pos": 7.138, "max_neg": 4.475, "fp0_theta": 7.138, "catch_neg": "12/12"} + ], + "note": "All 4 candidates clean-separate on this candidate-bench. Picked electra-base as primary by margin (per §7 #18: rank by separation, not raw score). The thresholds above are CANDIDATE-BENCH ONLY — they will move when the denominator grows (§7 #25→#26 happened three times on the NLI side; assume the same here). demote_below_score stays null until §3.2.2 step 2 (real-traffic shadow sweep) sets it." + }, "rationale_for_cross_encoder_not_bi_encoder": "A bi-encoder (sentence-transformers, vec-style) gives an unsupervised cosine similarity that's good for retrieval *ranking* but not calibrated for the 'is this on-topic?' question we need here. A cross-encoder reranker is trained on (query, doc) pairs with relevance labels — its single logit IS the 'is this doc relevant to this query?' score we want. The architectural cross-link to #000050/#000051's vec hybrid: that work uses *bi-encoders* (fastembed, bge-small) for retrieval recall; this work uses a *cross-encoder* for downstream aboutness scoring. They are complementary — vec finds candidates, the cross-encoder scores them." } diff --git a/bench/results/relevance-aboutness-candidatebench.json b/bench/results/relevance-aboutness-candidatebench.json new file mode 100644 index 0000000..82cb172 --- /dev/null +++ b/bench/results/relevance-aboutness-candidatebench.json @@ -0,0 +1,216 @@ +{ + "generated_at": "2026-05-13T13:53:25Z", + "fixtures_file": "/home/fox/nli-data/relevance-aboutness-v1.jsonl", + "n_pos": 14, + "n_neg": 12, + "models": { + "relevance-shadow-v1-ms-marco-MiniLM-L-6-v2": { + "hf_repo": "cross-encoder/ms-marco-MiniLM-L-6-v2", + "backend": "torch", + "device": "cuda", + "n_pairs": 26, + "infer_seconds": 0.18, + "pos_scores": { + "pos-qa-001": 10.3561, + "pos-qa-002": 7.6135, + "pos-qa-003": 10.2615, + "pos-qa-004": 9.6113, + "pos-qa-005": 10.761, + "pos-qa-006": 10.6969, + "pos-qa-007": 10.189, + "pos-cs-001": 8.6417, + "pos-cs-002": 6.754, + "pos-cs-003": 10.2053, + "pos-cs-004": 9.1802, + "pos-cs-005": 10.58, + "pos-cs-006": 9.8569, + "pos-cs-007": 9.986 + }, + "neg_scores": { + "neg-qa-001": 3.7006, + "neg-qa-002": -0.0128, + "neg-qa-003": 3.1369, + "neg-qa-004": 0.2058, + "neg-qa-005": -5.2781, + "neg-qa-006": -9.0515, + "neg-cs-001": -8.1389, + "neg-cs-002": -2.9362, + "neg-cs-003": -5.963, + "neg-cs-004": -6.7567, + "neg-cs-005": -5.9802, + "neg-cs-006": -9.8485 + }, + "frontier": { + "separation_margin": 3.0533, + "fp0_best": { + "theta": 6.754, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + }, + "pareto": [ + { + "theta": 6.754, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + } + ] + } + }, + "relevance-shadow-v1-ms-marco-MiniLM-L-12-v2": { + "hf_repo": "cross-encoder/ms-marco-MiniLM-L-12-v2", + "backend": "torch", + "device": "cuda", + "n_pairs": 26, + "infer_seconds": 0.05, + "pos_scores": { + "pos-qa-001": 10.1692, + "pos-qa-002": 8.139, + "pos-qa-003": 10.1767, + "pos-qa-004": 9.3511, + "pos-qa-005": 10.5464, + "pos-qa-006": 10.0394, + "pos-qa-007": 9.4536, + "pos-cs-001": 8.6255, + "pos-cs-002": 7.1379, + "pos-cs-003": 9.4853, + "pos-cs-004": 9.6927, + "pos-cs-005": 10.1209, + "pos-cs-006": 9.9942, + "pos-cs-007": 10.2065 + }, + "neg_scores": { + "neg-qa-001": 4.4752, + "neg-qa-002": -0.2735, + "neg-qa-003": 3.1724, + "neg-qa-004": -0.3941, + "neg-qa-005": -5.2772, + "neg-qa-006": -10.3666, + "neg-cs-001": -6.9534, + "neg-cs-002": -2.7428, + "neg-cs-003": -6.2688, + "neg-cs-004": -6.3582, + "neg-cs-005": -7.7344, + "neg-cs-006": -10.1564 + }, + "frontier": { + "separation_margin": 2.6627, + "fp0_best": { + "theta": 7.1379, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + }, + "pareto": [ + { + "theta": 7.1379, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + } + ] + } + }, + "relevance-shadow-v1-bge-reranker-base": { + "hf_repo": "BAAI/bge-reranker-base", + "backend": "torch", + "device": "cuda", + "n_pairs": 26, + "infer_seconds": 0.08, + "pos_scores": { + "pos-qa-001": 8.0599, + "pos-qa-002": 7.8094, + "pos-qa-003": 6.7217, + "pos-qa-004": 7.1199, + "pos-qa-005": 8.3108, + "pos-qa-006": 7.723, + "pos-qa-007": 6.2936, + "pos-cs-001": 2.395, + "pos-cs-002": 2.7944, + "pos-cs-003": 4.9885, + "pos-cs-004": 4.7798, + "pos-cs-005": 6.3818, + "pos-cs-006": 8.417, + "pos-cs-007": 5.2747 + }, + "neg_scores": { + "neg-qa-001": -0.7671, + "neg-qa-002": -3.387, + "neg-qa-003": -1.9937, + "neg-qa-004": -0.6807, + "neg-qa-005": -6.0444, + "neg-qa-006": -7.8323, + "neg-cs-001": -6.4084, + "neg-cs-002": -3.9995, + "neg-cs-003": -5.247, + "neg-cs-004": -4.4321, + "neg-cs-005": -7.8685, + "neg-cs-006": -6.9767 + }, + "frontier": { + "separation_margin": 3.0757, + "fp0_best": { + "theta": 2.395, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + }, + "pareto": [ + { + "theta": 2.395, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + } + ] + } + }, + "relevance-shadow-v1-ms-marco-electra-base": { + "hf_repo": "cross-encoder/ms-marco-electra-base", + "backend": "torch", + "device": "cuda", + "n_pairs": 26, + "infer_seconds": 0.1, + "pos_scores": { + "pos-qa-001": 3.448, + "pos-qa-002": 2.3775, + "pos-qa-003": 3.8591, + "pos-qa-004": 1.7753, + "pos-qa-005": 3.4742, + "pos-qa-006": 1.5494, + "pos-qa-007": 3.3069, + "pos-cs-001": 0.0497, + "pos-cs-002": 2.3681, + "pos-cs-003": 4.3109, + "pos-cs-004": 2.4378, + "pos-cs-005": 2.0715, + "pos-cs-006": 5.0473, + "pos-cs-007": 4.9582 + }, + "neg_scores": { + "neg-qa-001": -5.7487, + "neg-qa-002": -8.6513, + "neg-qa-003": -6.7965, + "neg-qa-004": -10.2494, + "neg-qa-005": -10.6582, + "neg-qa-006": -10.3879, + "neg-cs-001": -10.4413, + "neg-cs-002": -5.9059, + "neg-cs-003": -5.2999, + "neg-cs-004": -10.5911, + "neg-cs-005": -9.6781, + "neg-cs-006": -10.8435 + }, + "frontier": { + "separation_margin": 5.3495, + "fp0_best": { + "theta": 0.0497, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + }, + "pareto": [ + { + "theta": 0.0497, + "catch_neg": 1.0, + "fp_on_pos": 0.0 + } + ] + } + } + } +} \ No newline at end of file diff --git a/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md b/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md index 496b6f9..78b1d91 100644 --- a/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md +++ b/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md @@ -254,27 +254,51 @@ structure verbatim. Pieces: - `tests/test_relevance_shadow.py` — 13 tests (pure-Python + graceful degradation + the Zionist sanity check); all green. -**Sanity check on real (claim, source) pairs** (run on the loaded -`ms-marco-MiniLM-L-6-v2`, CPU, batch=1; scores are logits, sign matters): +**Candidate-bench (2026-05-13)** — 26-pair hand-built fixture set +(`bench/fixtures/5f/relevance-aboutness-v1.jsonl`: 14 on-topic POS + +12 off-topic / topic-collision NEG, incl. the Zionist field case), +all 4 manifest candidates swept on the 4090 (104 GPU NLI pairs total, +~1 s per model; `bench/results/relevance-aboutness-candidatebench.json`). +Ranked by separation margin (= `min(POS_score) − max(NEG_score)`; +>0 = clean-separable): -| pair | claim | source | score | -|---|---|---|---| -| Zionist ON-topic | "...used as a pejorative for the State of Israel" | the real `Zionist entity` sentence | **+9.96** | -| Zionist OFF-topic (the field case) | "...used as the entity, referring to the State of Israel" | a grammar `Phrase` definition | **−9.04** | -| Mona Lisa Q→A on | "who painted the mona lisa?" | "Leonardo da Vinci painted the Mona Lisa." | +10.45 | -| Mona Lisa Q→A deflect | "who painted the mona lisa?" | "The Mona Lisa is a half-length portrait housed in the Louvre." | +3.56 | -| **Kilimanjaro recombination** | "Kilimanjaro is in central Kenya south of the equator" | the Mount Kenya clause | **+7.59** (false positive — see below) | -| Kilimanjaro on | "Kilimanjaro is in north-eastern Tanzania" | the Kilimanjaro lead | +6.06 | +| model (size) | min(POS) | max(NEG) | **sep margin** | fp=0 θ | catch on NEG | +|---|---|---|---|---|---| +| **`cross-encoder/ms-marco-electra-base`** (~110 MB) | +0.050 | −5.300 | **+5.349** | 0.050 | **12/12 = 1.000** | +| `BAAI/bge-reranker-base` (~280 MB) | +2.395 | −0.681 | +3.076 | 2.395 | 12/12 = 1.000 | +| `cross-encoder/ms-marco-MiniLM-L-6-v2` (~80 MB) | +6.754 | +3.701 | +3.053 | 6.754 | 12/12 = 1.000 | +| `cross-encoder/ms-marco-MiniLM-L-12-v2` (~130 MB) | +7.138 | +4.475 | +2.663 | 7.138 | 12/12 = 1.000 | -**Reading.** The reranker **cleanly discriminates the Zionist-entity -field case (18-point margin)** and the Q→A deflection case (~7-point -margin) — i.e. it catches exactly the failure mode #000052 §1 named -("the lexical title-relevance sidecar passes a token-collision -mis-cite"). It does **NOT** catch the *recombination* case (Mount -Kenya clause scores *higher* than the Kilimanjaro lead for a -"Kilimanjaro is in Kenya" claim) — because the Mount Kenya clause is -genuinely about Kilimanjaro-as-comparison and the reranker is scoring -*aboutness*, not *truth*. That is the right architectural split: +**Read carefully:** +- **All 4 candidates clean-separate** the candidate-bench — every model + has a θ that catches all 12 off-topic NEG (incl. the Zionist field + case) with 0 FP on the 14 POS. So "the reranker discriminates the + Zionist-style failure mode" is *not a property of any specific + checkpoint*; it's a property of MS-MARCO-trained rerankers (and + BGE) as a class on contrived data. +- **The deepest MiniLM (L-12) is the *worst* by margin** (+2.66 vs + +5.35 for electra-base). Same pattern as #000049 §7 #24's "bart-large + is worse than deberta-large" — *deeper ≠ better*; specific + fine-tune and architecture dominate. The §3.2 manifest primary + moves to `ms-marco-electra-base` for the most cushion (~75% more + margin than the next-best, ~100% more than the smallest L-6). +- **None of this predicts real-traffic behavior.** The + #000049 §7 #18→#27 history is the receipts: clean candidate-bench + → real-bench-qa precision flipped six times on the NLI side; assume + the same here. The thresholds above (electra-base at θ=0.05, L-6 at + θ=6.75) are *candidate-bench numbers* and will move when the + denominator grows. The manifest's `demote_below_score` stays `null` + until the real-traffic shadow sweep (§3.2.2 step 2) sets it. + +**What the candidate-bench does NOT establish** (preserved from the +earlier 6-pair sanity check — the architectural split remains the +same point, just with proper data behind the catch-rate claim): +the reranker does **NOT** catch the *recombination*-where-the- +different-entity-clause-also-mentions-the-target-subject case (the +Mount Kenya clause scores higher than the Kilimanjaro lead for a +"Kilimanjaro is in Kenya" claim) — because the Mount Kenya clause +is genuinely *about* Kilimanjaro-as-comparison and the reranker is +scoring *aboutness*, not *truth*. The right architectural split: - **#000052 §3.2 relevance reranker** = catches *topic-collision / off-topic* failures (Zionist entity, Q→A deflection). ✓ shown.