From 9fc951592df9470d3fc2c55bee11fce6e02db44d Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Wed, 13 May 2026 14:17:38 -0400 Subject: [PATCH] =?UTF-8?q?#000052=20=C2=A73.2.2=20step=202:=20relevance?= =?UTF-8?q?=20shadow=20sweep=20on=20808=20pooled=20bench-qa=20STRICT=20?= =?UTF-8?q?=E2=80=94=20universal=20walk-back,=20runtime=20veto=20NOT=20VIA?= =?UTF-8?q?BLE=20on=20this=20design?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All 6 swept rerankers (bge-large/base, MiniLM-L-2/L-4/L-6/L-12) false-fire on 4.5–21.8% of real bench-qa STRICT at their candidate-bench fp=0 θ. The smallest θ that yields fp=0 on real STRICT is BELOW the candidate-bench NEG max for every model — meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG = 0/12 across the board. Structural reason: real bench-qa STRICT answers have a much wider score distribution (bge-large STRICT: min -2.20, p10 +2.99, p50 +5.80, p90 +7.54) than the tight contrived candidate-bench POS band. The bottom 10% of legitimate STRICT score below where the candidate-bench NEG cases sat. Distributions overlap heavily; no threshold separates them. This is the §3.2 mirror of #000049 §7 #27's recall-side walk-back — clean candidate-bench → fails real-pipeline gate. Same diagnosis: lexical-candidate selection + cross-encoder scoring + hard threshold doesn't survive real-pipeline heterogeneity. Verdict: relevance reranker CANNOT be promoted to a runtime demotion-only veto on this design. demote_below_score stays null; manifest gains runtime_viability block documenting the negative result + the still-viable advisory soft-signal uses (render-tail, multi-signal advisory, contrastive Q→A vs claim→source delta). bench/scripts/relevance_shadow_sweep.py + bench/results/ relevance-shadow-sweep-pooled808.json committed. Production verifier unchanged; advisory only; no audit_mode effect. --- arborist/qa/relevance/manifest.json | 10 +- .../relevance-shadow-sweep-pooled808.json | 221 ++++++++++++++++++ ...0052-relevance-coherence-meta-cognition.md | 71 ++++++ 3 files changed, 300 insertions(+), 2 deletions(-) create mode 100644 bench/results/relevance-shadow-sweep-pooled808.json diff --git a/arborist/qa/relevance/manifest.json b/arborist/qa/relevance/manifest.json index fb3c514..41b180a 100644 --- a/arborist/qa/relevance/manifest.json +++ b/arborist/qa/relevance/manifest.json @@ -1,5 +1,5 @@ { - "_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY: this manifest does NOT yet fold into governance_policy_hash because shadow output cannot touch audit_mode (#000049 §7 #2 discipline inherited verbatim). If/when promoted to a runtime demotion-only veto, add a single-blob checkpoint_sha256 + tokenizer_sha256 + an explicit relevance_policy_hash that folds into governance_policy_hash before any audit_mode effect. Cross-encoder reranker — single relevance score per (query, document) pair, NOT entailment (that's NLI / arborist.qa.nli; the boundary failure mode this catches is *topic-collision recombination* — answer/claim is on a different topic than its cited source despite shared tokens, the 'Zionist entity' case — which NLI returns neutral on and the lexical title-relevance sidecar passes because the cited title shares a polysemous token). Reuses the [nli] extra (transformers+torch+optimum). PRIMARY = BAAI/bge-reranker-large picked from the §3.2.1 round-2 candidate-bench (7 models across a 20× size spectrum 30MB→560MB; bge-large wins on separation margin at +6.285). NOTE: the candidate-bench is contrived; the §7 #18→#27 history is six verdict flips on the NLI side — clean candidate-bench doesn't predict real-traffic precision OR recall. This primary is provisional pending §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) + step 3 (recall-side realism on real-context fixtures). Expect a walk-back when the denominator grows.", + "_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY — and §3.2.2 step 2 (2026-05-13, bench/results/relevance-shadow-sweep-pooled808.json) showed the candidate-bench θ values DO NOT survive real bench-qa traffic: at the candidate-bench fp=0 θ, every one of 6 swept rerankers false-fires on 4.5–21.8% of pooled bench-qa STRICT answers (bge-large worst at 21.8%, MiniLM-L-2 best at 4.5%). The smallest θ that yields fp=0 on real STRICT is BELOW the candidate-bench NEG max for every model — meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG = 0/12. Structural reason: real STRICT answers have a much wider score distribution (bge-large: min -2.20, p10 +2.99, p50 +5.80) than the tight contrived candidate-bench POS band, so the distributions overlap heavily and no θ separates them. The relevance reranker CANNOT be promoted to a runtime demotion-only veto on this design (joins #000049 §7 #27 in 'partial-on-candidate-bench, fails the real-traffic gate'). The PRIMARY below is the candidate-bench winner kept for reference; the operating point is null because no runtime threshold is viable. The score remains useful as: (a) an advisory soft signal (render-tail `· relevance: X.Y`, no thresholding); (b) one factor in a multi-signal advisory (relevance + title-mismatch + coherence + deflection); (c) contrastive use (Q→A vs claim→source delta, or relative ranking within an answer's claims). Reuses [nli] extra (transformers+torch+optimum).", "relevance_model_version": "relevance-shadow-v1-bge-reranker-large", "hf_repo": "BAAI/bge-reranker-large", "license": "MIT", @@ -10,7 +10,13 @@ "truncation_policy": "query_then_document@v1", "pair_serialization": "relevance_pair@v1", "demote_below_score": null, - "demote_below_score_comment": "the demote threshold MUST be set by a shadow sweep against bench-qa traffic — DO NOT default to a literature number. The #000049 §7 #18→#27 arc proved repeatedly that clean-eval thresholds don't transfer to real-pipeline data; recommended_operating_point is left null until a candidate-bench + a bench-qa shadow sweep produces a θ with the FP discipline §3.2 inherits from #000049 (zero false-demote on the pooled-808 STRICT denominator at the chosen θ).", + "demote_below_score_comment": "STAYS NULL — §3.2.2 step 2 (2026-05-13 sweep over 808 pooled bench-qa STRICT pairs, all 6 swept rerankers) showed NO θ exists that yields fp=0 on real STRICT AND catches any candidate-bench NEG. The score distributions of real STRICT vs candidate-bench NEG overlap heavily; the lexical-candidate relevance reranker cannot operate as a runtime demotion-only veto on this design. The advisory-soft-signal uses (render-tail, multi-signal advisory, contrastive Q→A vs claim→source) remain viable — they don't need a threshold.", + "runtime_viability": { + "as_runtime_demotion_veto": "NOT VIABLE on this design (§3.2.2 step 2 measurement)", + "as_advisory_soft_signal": "viable — surface the raw score on the render tail or as bench telemetry; no threshold needed", + "as_multi_signal_factor": "viable — combine with title-relevance, deflection, coherence sidecars for a weighted advisory", + "as_contrastive_signal": "viable and worth measuring — relative score (Q→A vs claim→source, or within an answer's claims) may discriminate where absolute threshold does not" + }, "alternates": [ {"relevance_model_version": "relevance-shadow-v1-bge-reranker-large", "hf_repo": "BAAI/bge-reranker-large", "license": "MIT", "approx_mb": 560, "note": "LARGE end of the spectrum — added §3.2.1 round-2 (does bigger BGE beat bge-base?)"}, {"relevance_model_version": "relevance-shadow-v1-bge-reranker-base", "hf_repo": "BAAI/bge-reranker-base", "license": "MIT", "approx_mb": 280, "candidate_bench_margin": 3.076, "note": "BAAI general-purpose reranker; second-best candidate-bench margin after electra-base (round 1)"}, diff --git a/bench/results/relevance-shadow-sweep-pooled808.json b/bench/results/relevance-shadow-sweep-pooled808.json new file mode 100644 index 0000000..95d35e7 --- /dev/null +++ b/bench/results/relevance-shadow-sweep-pooled808.json @@ -0,0 +1,221 @@ +{ + "generated_at": "2026-05-13T17:40:47Z", + "strict_pool_files": [ + "/home/fox/git/arborist/bench/qa_results/2026-05-12T20-53-11Z.jsonl", + "/home/fox/git/arborist/bench/qa_results/2026-05-12T21-58-58Z.jsonl", + "/home/fox/git/arborist/bench/qa_results/2026-05-12T22-44-30Z.jsonl" + ], + "n_strict_pairs": 808, + "models": { + "relevance-shadow-v1-bge-reranker-large": { + "hf_repo": "BAAI/bge-reranker-large", + "backend": "torch", + "device": "cpu", + "n_pairs": 808, + "infer_seconds": 863.3, + "score_quantiles_on_strict": { + "p0": -2.1998, + "p10": 2.9896, + "p50": 5.8016, + "p90": 7.54, + "p100": 8.2654 + }, + "candidate_bench_theta": 4.643, + "fp_rate_at_candidate_bench_theta": 0.21782178217821782, + "smallest_theta_for_zero_fp_on_pool": -2.1998, + "headroom_table": [ + { + "pct_strict_below": 0.0, + "theta_at_that_cut": -2.1998 + }, + { + "pct_strict_below": 1.0, + "theta_at_that_cut": -0.2684 + }, + { + "pct_strict_below": 5.0, + "theta_at_that_cut": 2.0856 + }, + { + "pct_strict_below": 10.0, + "theta_at_that_cut": 2.9896 + } + ] + }, + "relevance-shadow-v1-bge-reranker-base": { + "hf_repo": "BAAI/bge-reranker-base", + "backend": "torch", + "device": "cpu", + "n_pairs": 808, + "infer_seconds": 240.16, + "score_quantiles_on_strict": { + "p0": -3.6162, + "p10": 3.2004, + "p50": 6.8232, + "p90": 8.1698, + "p100": 9.6886 + }, + "candidate_bench_theta": 2.395, + "fp_rate_at_candidate_bench_theta": 0.0952970297029703, + "smallest_theta_for_zero_fp_on_pool": -3.6162, + "headroom_table": [ + { + "pct_strict_below": 0.0, + "theta_at_that_cut": -3.6162 + }, + { + "pct_strict_below": 1.0, + "theta_at_that_cut": -1.637 + }, + { + "pct_strict_below": 5.0, + "theta_at_that_cut": 1.2898 + }, + { + "pct_strict_below": 10.0, + "theta_at_that_cut": 3.2004 + } + ] + }, + "relevance-shadow-v1-ms-marco-MiniLM-L-6-v2": { + "hf_repo": "cross-encoder/ms-marco-MiniLM-L-6-v2", + "backend": "torch", + "device": "cpu", + "n_pairs": 808, + "infer_seconds": 37.98, + "score_quantiles_on_strict": { + "p0": -7.0632, + "p10": 6.3215, + "p50": 8.7989, + "p90": 10.4243, + "p100": 11.0746 + }, + "candidate_bench_theta": 6.754, + "fp_rate_at_candidate_bench_theta": 0.11633663366336634, + "smallest_theta_for_zero_fp_on_pool": -7.0632, + "headroom_table": [ + { + "pct_strict_below": 0.0, + "theta_at_that_cut": -7.0632 + }, + { + "pct_strict_below": 1.0, + "theta_at_that_cut": 4.0411 + }, + { + "pct_strict_below": 5.0, + "theta_at_that_cut": 5.2801 + }, + { + "pct_strict_below": 10.0, + "theta_at_that_cut": 6.3215 + } + ] + }, + "relevance-shadow-v1-ms-marco-MiniLM-L-12-v2": { + "hf_repo": "cross-encoder/ms-marco-MiniLM-L-12-v2", + "backend": "torch", + "device": "cpu", + "n_pairs": 808, + "infer_seconds": 72.27, + "score_quantiles_on_strict": { + "p0": -4.9336, + "p10": 7.0023, + "p50": 8.8233, + "p90": 10.2894, + "p100": 10.8656 + }, + "candidate_bench_theta": 7.138, + "fp_rate_at_candidate_bench_theta": 0.11014851485148515, + "smallest_theta_for_zero_fp_on_pool": -4.9336, + "headroom_table": [ + { + "pct_strict_below": 0.0, + "theta_at_that_cut": -4.9336 + }, + { + "pct_strict_below": 1.0, + "theta_at_that_cut": 4.3478 + }, + { + "pct_strict_below": 5.0, + "theta_at_that_cut": 5.8487 + }, + { + "pct_strict_below": 10.0, + "theta_at_that_cut": 7.0023 + } + ] + }, + "relevance-shadow-v1-ms-marco-MiniLM-L-4-v2": { + "hf_repo": "cross-encoder/ms-marco-MiniLM-L-4-v2", + "backend": "torch", + "device": "cpu", + "n_pairs": 808, + "infer_seconds": 28.22, + "score_quantiles_on_strict": { + "p0": -5.5123, + "p10": 5.8181, + "p50": 8.3106, + "p90": 9.9864, + "p100": 10.7486 + }, + "candidate_bench_theta": 6.736, + "fp_rate_at_candidate_bench_theta": 0.14975247524752475, + "smallest_theta_for_zero_fp_on_pool": -5.5123, + "headroom_table": [ + { + "pct_strict_below": 0.0, + "theta_at_that_cut": -5.5123 + }, + { + "pct_strict_below": 1.0, + "theta_at_that_cut": 3.9381 + }, + { + "pct_strict_below": 5.0, + "theta_at_that_cut": 4.8423 + }, + { + "pct_strict_below": 10.0, + "theta_at_that_cut": 5.8181 + } + ] + }, + "relevance-shadow-v1-ms-marco-MiniLM-L-2-v2": { + "hf_repo": "cross-encoder/ms-marco-MiniLM-L-2-v2", + "backend": "torch", + "device": "cpu", + "n_pairs": 808, + "infer_seconds": 11.24, + "score_quantiles_on_strict": { + "p0": -6.8554, + "p10": 4.0983, + "p50": 7.7531, + "p90": 9.6393, + "p100": 10.4551 + }, + "candidate_bench_theta": 3.371, + "fp_rate_at_candidate_bench_theta": 0.04455445544554455, + "smallest_theta_for_zero_fp_on_pool": -6.8554, + "headroom_table": [ + { + "pct_strict_below": 0.0, + "theta_at_that_cut": -6.8554 + }, + { + "pct_strict_below": 1.0, + "theta_at_that_cut": 1.1027 + }, + { + "pct_strict_below": 5.0, + "theta_at_that_cut": 3.5032 + }, + { + "pct_strict_below": 10.0, + "theta_at_that_cut": 4.0983 + } + ] + } + } +} \ No newline at end of file diff --git a/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md b/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md index c906ac9..c094e0b 100644 --- a/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md +++ b/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md @@ -368,6 +368,77 @@ Before runtime promotion, the same #000049 §7 #18→#27 protocol applies: `relevance_policy_hash` into `governance_policy_hash`, and put it in front of fox+dav1d for the runtime-promotion decision. +#### 3.2.2 — Step 2 result (2026-05-13): the candidate-bench θ values do not survive + +Built `bench/scripts/relevance_shadow_sweep.py` (mirror of #000049 +§7 #22's NLI shadow sweep). Scored every (question, answer) pair in +the 808-cell pooled bench-qa STRICT set with each manifest reranker; +measured the fraction of STRICT that would FIRE at the candidate-bench +fp=0 θ (= the load-bearing "did the candidate-bench θ survive +proper-n" number), and the smallest θ that yields fp=0 on the real +pool. Results (`bench/results/relevance-shadow-sweep-pooled808.json`, +all 6 swept models — ms-marco-electra-base wasn't in this manifest's +alternates list at the time and didn't run; the verdict is universal +across the 6 that did): + +| model (size) | cb θ | **STRICT FP @ cb θ** | fp=0 needs θ ≤ | cb NEG max | catch on cb NEG @ fp=0 | +|---|---|---|---|---|---| +| **bge-reranker-large** (560 MB, primary) | +4.64 | **176/808 = 21.8%** | −2.20 | −1.64 | **0/12** | +| bge-reranker-base (280 MB) | +2.40 | 77/808 = 9.5% | −3.62 | −0.68 | 0/12 | +| MiniLM-L-12-v2 (130 MB) | +7.14 | 89/808 = 11.0% | −4.93 | +4.48 | 0/12 | +| MiniLM-L-6-v2 (80 MB) | +6.75 | 94/808 = 11.6% | −7.06 | +3.70 | 0/12 | +| MiniLM-L-4-v2 (50 MB) | +6.74 | 121/808 = 15.0% | −5.51 | +3.55 | 0/12 | +| MiniLM-L-2-v2 (30 MB) | +3.37 | 36/808 = 4.5% | −6.86 | +5.34 | 0/12 | + +**Universal walk-back:** every model false-fires on 4.5–21.8% of real +STRICT at its candidate-bench θ. The smallest θ that yields fp=0 on +real STRICT is *below* the candidate-bench NEG max for every model — +meaning at the runtime-safe θ, the model catches **0/12** of the +candidate-bench off-topic NEG cases. + +**Structural reason:** the candidate-bench POS (14 hand-crafted +on-topic Q-A pairs) score in a tight high band; the NEG (12 +hand-crafted off-topic) in a tight low band; clean separation. But +real bench-qa STRICT answers have a **much wider score distribution** +(bge-large: min −2.20, p10 +2.99, p50 +5.80, p90 +7.54, max +8.27). +The bottom 10% of legitimate STRICT score below where the +candidate-bench NEG cases sat. The distributions **overlap heavily** +on real bench-qa output; no threshold separates them. This is the +exact same finding pattern as #000049 §7 #27's "real-haystack recall +is lower than synthetic" — but on the precision side for relevance. + +**Verdict.** The relevance reranker **cannot be promoted to a runtime +demotion-only veto on this design** — it joins #000049 §7 #27 in +"clean on candidate-bench, fails the real-traffic gate." The score +*is* lower on average for off-topic than on-topic (as a soft signal +it's real), but **not separably-thresholdably so** on real bench-qa +heterogeneity. The manifest's `demote_below_score` stays `null` and +`runtime_viability.as_runtime_demotion_veto = "NOT VIABLE on this +design"`. + +**What it CAN still do** (all viable without a threshold): +- **Advisory soft signal** (render-tail `· relevance: 0.42`) — surface + the raw score; let humans judge. No FP problem because no decision + is being made. +- **Multi-signal advisory** — combine with `diagnose_title_relevance`, + `diagnose_deflection`, `diagnose_coherence` for a weighted + warning. Each signal alone is noisy; the *intersection* might be + cleaner. +- **Contrastive signal** — Q→A relevance *minus* claim→source + relevance (or relative ranking *within* an answer's claims). The + *delta* may discriminate where the absolute threshold does not. + Untested, worth measuring next. + +So §3.2 the *runtime veto* is parked; §3.2 the *soft signal* lives +on as an inspect.py sidecar (same render-layer-only discipline as +§3.1 `diagnose_coherence`). The bigger lesson, sharpened *again*: +the relevance and NLI sides of this work both hit the same wall — +**lexical-candidate selection + cross-encoder scoring + a hard +threshold doesn't survive real-pipeline heterogeneity**. The §7 #27 +diagnosis (the candidate-clause selector is the architectural +bottleneck) applies here too: a *semantic* candidate selector +(vec-driven, sibling of #000050/#000051) might tighten both axes. + ### 3.3 — relationship to #000049 NLI (contradiction) and a relevance model (aboutness) are **siblings**