engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000052 §3.2.2 step 2: relevance shadow sweep on 808 pooled bench-qa STRICT — universal walk-back, runtime veto NOT VIABLE on this design

Browse source 
All 6 swept rerankers (bge-large/base, MiniLM-L-2/L-4/L-6/L-12)
false-fire on 4.5–21.8% of real bench-qa STRICT at their
candidate-bench fp=0 θ. The smallest θ that yields fp=0 on real
STRICT is BELOW the candidate-bench NEG max for every model —
meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG
= 0/12 across the board.

Structural reason: real bench-qa STRICT answers have a much wider
score distribution (bge-large STRICT: min -2.20, p10 +2.99, p50
+5.80, p90 +7.54) than the tight contrived candidate-bench POS
band. The bottom 10% of legitimate STRICT score below where the
candidate-bench NEG cases sat. Distributions overlap heavily; no
threshold separates them.

This is the §3.2 mirror of #000049 §7 #27's recall-side walk-back —
clean candidate-bench → fails real-pipeline gate. Same diagnosis:
lexical-candidate selection + cross-encoder scoring + hard threshold
doesn't survive real-pipeline heterogeneity.

Verdict: relevance reranker CANNOT be promoted to a runtime
demotion-only veto on this design. demote_below_score stays null;
manifest gains runtime_viability block documenting the negative
result + the still-viable advisory soft-signal uses (render-tail,
multi-signal advisory, contrastive Q→A vs claim→source delta).

bench/scripts/relevance_shadow_sweep.py + bench/results/
relevance-shadow-sweep-pooled808.json committed. Production
verifier unchanged; advisory only; no audit_mode effect.
This commit is contained in:
russell@unturf.com 2026-05-13 14:17:38 -04:00
parent 8fd74a2da3
commit 9fc951592d
No known key found for this signature in database
3 changed files with 300 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

10
arborist/qa/relevance/manifest.json
View file

@ -1,5 +1,5 @@
{
"_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY: this manifest does NOT yet fold into governance_policy_hash because shadow output cannot touch audit_mode (#000049 §7 #2 discipline inherited verbatim). If/when promoted to a runtime demotion-only veto, add a single-blob checkpoint_sha256 + tokenizer_sha256 + an explicit relevance_policy_hash that folds into governance_policy_hash before any audit_mode effect. Cross-encoder reranker — single relevance score per (query, document) pair, NOT entailment (that's NLI / arborist.qa.nli; the boundary failure mode this catches is *topic-collision recombination* — answer/claim is on a different topic than its cited source despite shared tokens, the 'Zionist entity' case — which NLI returns neutral on and the lexical title-relevance sidecar passes because the cited title shares a polysemous token). Reuses the [nli] extra (transformers+torch+optimum). PRIMARY = BAAI/bge-reranker-large picked from the §3.2.1 round-2 candidate-bench (7 models across a 20× size spectrum 30MB→560MB; bge-large wins on separation margin at +6.285). NOTE: the candidate-bench is contrived; the §7 #18→#27 history is six verdict flips on the NLI side — clean candidate-bench doesn't predict real-traffic precision OR recall. This primary is provisional pending §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) + step 3 (recall-side realism on real-context fixtures). Expect a walk-back when the denominator grows.",
"_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY — and §3.2.2 step 2 (2026-05-13, bench/results/relevance-shadow-sweep-pooled808.json) showed the candidate-bench θ values DO NOT survive real bench-qa traffic: at the candidate-bench fp=0 θ, every one of 6 swept rerankers false-fires on 4.5–21.8% of pooled bench-qa STRICT answers (bge-large worst at 21.8%, MiniLM-L-2 best at 4.5%). The smallest θ that yields fp=0 on real STRICT is BELOW the candidate-bench NEG max for every model — meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG = 0/12. Structural reason: real STRICT answers have a much wider score distribution (bge-large: min -2.20, p10 +2.99, p50 +5.80) than the tight contrived candidate-bench POS band, so the distributions overlap heavily and no θ separates them. The relevance reranker CANNOT be promoted to a runtime demotion-only veto on this design (joins #000049 §7 #27 in 'partial-on-candidate-bench, fails the real-traffic gate'). The PRIMARY below is the candidate-bench winner kept for reference; the operating point is null because no runtime threshold is viable. The score remains useful as: (a) an advisory soft signal (render-tail `· relevance: X.Y`, no thresholding); (b) one factor in a multi-signal advisory (relevance + title-mismatch + coherence + deflection); (c) contrastive use (Q→A vs claim→source delta, or relative ranking within an answer's claims). Reuses [nli] extra (transformers+torch+optimum).",
"relevance_model_version": "relevance-shadow-v1-bge-reranker-large",
"hf_repo": "BAAI/bge-reranker-large",
"license": "MIT",
@ -10,7 +10,13 @@
"truncation_policy": "query_then_document@v1",
"pair_serialization": "relevance_pair@v1",
"demote_below_score": null,
"demote_below_score_comment": "the demote threshold MUST be set by a shadow sweep against bench-qa traffic — DO NOT default to a literature number. The #000049 §7 #18→#27 arc proved repeatedly that clean-eval thresholds don't transfer to real-pipeline data; recommended_operating_point is left null until a candidate-bench + a bench-qa shadow sweep produces a θ with the FP discipline §3.2 inherits from #000049 (zero false-demote on the pooled-808 STRICT denominator at the chosen θ).",
"demote_below_score_comment": "STAYS NULL — §3.2.2 step 2 (2026-05-13 sweep over 808 pooled bench-qa STRICT pairs, all 6 swept rerankers) showed NO θ exists that yields fp=0 on real STRICT AND catches any candidate-bench NEG. The score distributions of real STRICT vs candidate-bench NEG overlap heavily; the lexical-candidate relevance reranker cannot operate as a runtime demotion-only veto on this design. The advisory-soft-signal uses (render-tail, multi-signal advisory, contrastive Q→A vs claim→source) remain viable — they don't need a threshold.",
"runtime_viability": {
"as_runtime_demotion_veto": "NOT VIABLE on this design (§3.2.2 step 2 measurement)",
"as_advisory_soft_signal": "viable — surface the raw score on the render tail or as bench telemetry; no threshold needed",
"as_multi_signal_factor": "viable — combine with title-relevance, deflection, coherence sidecars for a weighted advisory",
"as_contrastive_signal": "viable and worth measuring — relative score (Q→A vs claim→source, or within an answer's claims) may discriminate where absolute threshold does not"
},
"alternates": [
{"relevance_model_version": "relevance-shadow-v1-bge-reranker-large", "hf_repo": "BAAI/bge-reranker-large", "license": "MIT", "approx_mb": 560, "note": "LARGE end of the spectrum — added §3.2.1 round-2 (does bigger BGE beat bge-base?)"},
{"relevance_model_version": "relevance-shadow-v1-bge-reranker-base", "hf_repo": "BAAI/bge-reranker-base", "license": "MIT", "approx_mb": 280, "candidate_bench_margin": 3.076, "note": "BAAI general-purpose reranker; second-best candidate-bench margin after electra-base (round 1)"},

221
bench/results/relevance-shadow-sweep-pooled808.json Normal file
View file

@ -0,0 +1,221 @@
{
"generated_at": "2026-05-13T17:40:47Z",
"strict_pool_files": [
"/home/fox/git/arborist/bench/qa_results/2026-05-12T20-53-11Z.jsonl",
"/home/fox/git/arborist/bench/qa_results/2026-05-12T21-58-58Z.jsonl",
"/home/fox/git/arborist/bench/qa_results/2026-05-12T22-44-30Z.jsonl"
],
"n_strict_pairs": 808,
"models": {
"relevance-shadow-v1-bge-reranker-large": {
"hf_repo": "BAAI/bge-reranker-large",
"backend": "torch",
"device": "cpu",
"n_pairs": 808,
"infer_seconds": 863.3,
"score_quantiles_on_strict": {
"p0": -2.1998,
"p10": 2.9896,
"p50": 5.8016,
"p90": 7.54,
"p100": 8.2654
},
"candidate_bench_theta": 4.643,
"fp_rate_at_candidate_bench_theta": 0.21782178217821782,
"smallest_theta_for_zero_fp_on_pool": -2.1998,
"headroom_table": [
{
"pct_strict_below": 0.0,
"theta_at_that_cut": -2.1998
},
{
"pct_strict_below": 1.0,
"theta_at_that_cut": -0.2684
},
{
"pct_strict_below": 5.0,
"theta_at_that_cut": 2.0856
},
{
"pct_strict_below": 10.0,
"theta_at_that_cut": 2.9896
}
]
},
"relevance-shadow-v1-bge-reranker-base": {
"hf_repo": "BAAI/bge-reranker-base",
"backend": "torch",
"device": "cpu",
"n_pairs": 808,
"infer_seconds": 240.16,
"score_quantiles_on_strict": {
"p0": -3.6162,
"p10": 3.2004,
"p50": 6.8232,
"p90": 8.1698,
"p100": 9.6886
},
"candidate_bench_theta": 2.395,
"fp_rate_at_candidate_bench_theta": 0.0952970297029703,
"smallest_theta_for_zero_fp_on_pool": -3.6162,
"headroom_table": [
{
"pct_strict_below": 0.0,
"theta_at_that_cut": -3.6162
},
{
"pct_strict_below": 1.0,
"theta_at_that_cut": -1.637
},
{
"pct_strict_below": 5.0,
"theta_at_that_cut": 1.2898
},
{
"pct_strict_below": 10.0,
"theta_at_that_cut": 3.2004
}
]
},
"relevance-shadow-v1-ms-marco-MiniLM-L-6-v2": {
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-6-v2",
"backend": "torch",
"device": "cpu",
"n_pairs": 808,
"infer_seconds": 37.98,
"score_quantiles_on_strict": {
"p0": -7.0632,
"p10": 6.3215,
"p50": 8.7989,
"p90": 10.4243,
"p100": 11.0746
},
"candidate_bench_theta": 6.754,
"fp_rate_at_candidate_bench_theta": 0.11633663366336634,
"smallest_theta_for_zero_fp_on_pool": -7.0632,
"headroom_table": [
{
"pct_strict_below": 0.0,
"theta_at_that_cut": -7.0632
},
{
"pct_strict_below": 1.0,
"theta_at_that_cut": 4.0411
},
{
"pct_strict_below": 5.0,
"theta_at_that_cut": 5.2801
},
{
"pct_strict_below": 10.0,
"theta_at_that_cut": 6.3215
}
]
},
"relevance-shadow-v1-ms-marco-MiniLM-L-12-v2": {
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-12-v2",
"backend": "torch",
"device": "cpu",
"n_pairs": 808,
"infer_seconds": 72.27,
"score_quantiles_on_strict": {
"p0": -4.9336,
"p10": 7.0023,
"p50": 8.8233,
"p90": 10.2894,
"p100": 10.8656
},
"candidate_bench_theta": 7.138,
"fp_rate_at_candidate_bench_theta": 0.11014851485148515,
"smallest_theta_for_zero_fp_on_pool": -4.9336,
"headroom_table": [
{
"pct_strict_below": 0.0,
"theta_at_that_cut": -4.9336
},
{
"pct_strict_below": 1.0,
"theta_at_that_cut": 4.3478
},
{
"pct_strict_below": 5.0,
"theta_at_that_cut": 5.8487
},
{
"pct_strict_below": 10.0,
"theta_at_that_cut": 7.0023
}
]
},
"relevance-shadow-v1-ms-marco-MiniLM-L-4-v2": {
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-4-v2",
"backend": "torch",
"device": "cpu",
"n_pairs": 808,
"infer_seconds": 28.22,
"score_quantiles_on_strict": {
"p0": -5.5123,
"p10": 5.8181,
"p50": 8.3106,
"p90": 9.9864,
"p100": 10.7486
},
"candidate_bench_theta": 6.736,
"fp_rate_at_candidate_bench_theta": 0.14975247524752475,
"smallest_theta_for_zero_fp_on_pool": -5.5123,
"headroom_table": [
{
"pct_strict_below": 0.0,
"theta_at_that_cut": -5.5123
},
{
"pct_strict_below": 1.0,
"theta_at_that_cut": 3.9381
},
{
"pct_strict_below": 5.0,
"theta_at_that_cut": 4.8423
},
{
"pct_strict_below": 10.0,
"theta_at_that_cut": 5.8181
}
]
},
"relevance-shadow-v1-ms-marco-MiniLM-L-2-v2": {
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-2-v2",
"backend": "torch",
"device": "cpu",
"n_pairs": 808,
"infer_seconds": 11.24,
"score_quantiles_on_strict": {
"p0": -6.8554,
"p10": 4.0983,
"p50": 7.7531,
"p90": 9.6393,
"p100": 10.4551
},
"candidate_bench_theta": 3.371,
"fp_rate_at_candidate_bench_theta": 0.04455445544554455,
"smallest_theta_for_zero_fp_on_pool": -6.8554,
"headroom_table": [
{
"pct_strict_below": 0.0,
"theta_at_that_cut": -6.8554
},
{
"pct_strict_below": 1.0,
"theta_at_that_cut": 1.1027
},
{
"pct_strict_below": 5.0,
"theta_at_that_cut": 3.5032
},
{
"pct_strict_below": 10.0,
"theta_at_that_cut": 4.0983
}
]
}
}
}

71
docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md
View file

@ -368,6 +368,77 @@ Before runtime promotion, the same #000049 §7 #18→#27 protocol applies:
`relevance_policy_hash` into `governance_policy_hash`, and put it
in front of fox+dav1d for the runtime-promotion decision.
#### 3.2.2 — Step 2 result (2026-05-13): the candidate-bench θ values do not survive
Built `bench/scripts/relevance_shadow_sweep.py` (mirror of #000049
§7 #22's NLI shadow sweep). Scored every (question, answer) pair in
the 808-cell pooled bench-qa STRICT set with each manifest reranker;
measured the fraction of STRICT that would FIRE at the candidate-bench
fp=0 θ (= the load-bearing "did the candidate-bench θ survive
proper-n" number), and the smallest θ that yields fp=0 on the real
pool. Results (`bench/results/relevance-shadow-sweep-pooled808.json`,
all 6 swept models — ms-marco-electra-base wasn't in this manifest's
alternates list at the time and didn't run; the verdict is universal
across the 6 that did):
| model (size) | cb θ | **STRICT FP @ cb θ** | fp=0 needs θ ≤ | cb NEG max | catch on cb NEG @ fp=0 |
|---|---|---|---|---|---|
| **bge-reranker-large** (560 MB, primary) | +4.64 | **176/808 = 21.8%** | −2.20 | −1.64 | **0/12** |
| bge-reranker-base (280 MB) | +2.40 | 77/808 = 9.5% | −3.62 | −0.68 | 0/12 |
| MiniLM-L-12-v2 (130 MB) | +7.14 | 89/808 = 11.0% | −4.93 | +4.48 | 0/12 |
| MiniLM-L-6-v2 (80 MB) | +6.75 | 94/808 = 11.6% | −7.06 | +3.70 | 0/12 |
| MiniLM-L-4-v2 (50 MB) | +6.74 | 121/808 = 15.0% | −5.51 | +3.55 | 0/12 |
| MiniLM-L-2-v2 (30 MB) | +3.37 | 36/808 = 4.5% | −6.86 | +5.34 | 0/12 |
**Universal walk-back:** every model false-fires on 4.5–21.8% of real
STRICT at its candidate-bench θ. The smallest θ that yields fp=0 on
real STRICT is *below* the candidate-bench NEG max for every model —
meaning at the runtime-safe θ, the model catches **0/12** of the
candidate-bench off-topic NEG cases.
**Structural reason:** the candidate-bench POS (14 hand-crafted
on-topic Q-A pairs) score in a tight high band; the NEG (12
hand-crafted off-topic) in a tight low band; clean separation. But
real bench-qa STRICT answers have a **much wider score distribution**
(bge-large: min −2.20, p10 +2.99, p50 +5.80, p90 +7.54, max +8.27).
The bottom 10% of legitimate STRICT score below where the
candidate-bench NEG cases sat. The distributions **overlap heavily**
on real bench-qa output; no threshold separates them. This is the
exact same finding pattern as #000049 §7 #27's "real-haystack recall
is lower than synthetic" — but on the precision side for relevance.
**Verdict.** The relevance reranker **cannot be promoted to a runtime
demotion-only veto on this design** — it joins #000049 §7 #27 in
"clean on candidate-bench, fails the real-traffic gate." The score
*is* lower on average for off-topic than on-topic (as a soft signal
it's real), but **not separably-thresholdably so** on real bench-qa
heterogeneity. The manifest's `demote_below_score` stays `null` and
`runtime_viability.as_runtime_demotion_veto = "NOT VIABLE on this
design"`.
**What it CAN still do** (all viable without a threshold):
- **Advisory soft signal** (render-tail `· relevance: 0.42`) — surface
the raw score; let humans judge. No FP problem because no decision
is being made.
- **Multi-signal advisory** — combine with `diagnose_title_relevance`,
`diagnose_deflection`, `diagnose_coherence` for a weighted
warning. Each signal alone is noisy; the *intersection* might be
cleaner.
- **Contrastive signal** — Q→A relevance *minus* claim→source
relevance (or relative ranking *within* an answer's claims). The
*delta* may discriminate where the absolute threshold does not.
Untested, worth measuring next.
So §3.2 the *runtime veto* is parked; §3.2 the *soft signal* lives
on as an inspect.py sidecar (same render-layer-only discipline as
§3.1 `diagnose_coherence`). The bigger lesson, sharpened *again*:
the relevance and NLI sides of this work both hit the same wall —
**lexical-candidate selection + cross-encoder scoring + a hard
threshold doesn't survive real-pipeline heterogeneity**. The §7 #27
diagnosis (the candidate-clause selector is the architectural
bottleneck) applies here too: a *semantic* candidate selector
(vec-driven, sibling of #000050/#000051) might tighten both axes.
### 3.3 — relationship to #000049
NLI (contradiction) and a relevance model (aboutness) are **siblings**