#000052 §3.2.2 step 2: relevance shadow sweep on 808 pooled bench-qa STRICT — universal walk-back, runtime veto NOT VIABLE on this design

All 6 swept rerankers (bge-large/base, MiniLM-L-2/L-4/L-6/L-12)
false-fire on 4.5–21.8% of real bench-qa STRICT at their
candidate-bench fp=0 θ. The smallest θ that yields fp=0 on real
STRICT is BELOW the candidate-bench NEG max for every model —
meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG
= 0/12 across the board.

Structural reason: real bench-qa STRICT answers have a much wider
score distribution (bge-large STRICT: min -2.20, p10 +2.99, p50
+5.80, p90 +7.54) than the tight contrived candidate-bench POS
band. The bottom 10% of legitimate STRICT score below where the
candidate-bench NEG cases sat. Distributions overlap heavily; no
threshold separates them.

This is the §3.2 mirror of #000049 §7 #27's recall-side walk-back —
clean candidate-bench → fails real-pipeline gate. Same diagnosis:
lexical-candidate selection + cross-encoder scoring + hard threshold
doesn't survive real-pipeline heterogeneity.

Verdict: relevance reranker CANNOT be promoted to a runtime
demotion-only veto on this design. demote_below_score stays null;
manifest gains runtime_viability block documenting the negative
result + the still-viable advisory soft-signal uses (render-tail,
multi-signal advisory, contrastive Q→A vs claim→source delta).

bench/scripts/relevance_shadow_sweep.py + bench/results/
relevance-shadow-sweep-pooled808.json committed. Production
verifier unchanged; advisory only; no audit_mode effect.

docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md

@@ -368,6 +368,77 @@ Before runtime promotion, the same #000049 §7 #18→#27 protocol applies:
    `relevance_policy_hash` into `governance_policy_hash`, and put it
    in front of fox+dav1d for the runtime-promotion decision.
 
+#### 3.2.2 — Step 2 result (2026-05-13): the candidate-bench θ values do not survive
+
+Built `bench/scripts/relevance_shadow_sweep.py` (mirror of #000049
+§7 #22's NLI shadow sweep). Scored every (question, answer) pair in
+the 808-cell pooled bench-qa STRICT set with each manifest reranker;
+measured the fraction of STRICT that would FIRE at the candidate-bench
+fp=0 θ (= the load-bearing "did the candidate-bench θ survive
+proper-n" number), and the smallest θ that yields fp=0 on the real
+pool. Results (`bench/results/relevance-shadow-sweep-pooled808.json`,
+all 6 swept models — ms-marco-electra-base wasn't in this manifest's
+alternates list at the time and didn't run; the verdict is universal
+across the 6 that did):
+
+| model (size) | cb θ | **STRICT FP @ cb θ** | fp=0 needs θ ≤ | cb NEG max | catch on cb NEG @ fp=0 |
+|---|---|---|---|---|---|
+| **bge-reranker-large** (560 MB, primary) | +4.64 | **176/808 = 21.8%** | −2.20 | −1.64 | **0/12** |
+| bge-reranker-base (280 MB) | +2.40 | 77/808 = 9.5% | −3.62 | −0.68 | 0/12 |
+| MiniLM-L-12-v2 (130 MB) | +7.14 | 89/808 = 11.0% | −4.93 | +4.48 | 0/12 |
+| MiniLM-L-6-v2 (80 MB) | +6.75 | 94/808 = 11.6% | −7.06 | +3.70 | 0/12 |
+| MiniLM-L-4-v2 (50 MB) | +6.74 | 121/808 = 15.0% | −5.51 | +3.55 | 0/12 |
+| MiniLM-L-2-v2 (30 MB) | +3.37 | 36/808 = 4.5% | −6.86 | +5.34 | 0/12 |
+
+**Universal walk-back:** every model false-fires on 4.5–21.8% of real
+STRICT at its candidate-bench θ. The smallest θ that yields fp=0 on
+real STRICT is *below* the candidate-bench NEG max for every model —
+meaning at the runtime-safe θ, the model catches **0/12** of the
+candidate-bench off-topic NEG cases.
+
+**Structural reason:** the candidate-bench POS (14 hand-crafted
+on-topic Q-A pairs) score in a tight high band; the NEG (12
+hand-crafted off-topic) in a tight low band; clean separation. But
+real bench-qa STRICT answers have a **much wider score distribution**
+(bge-large: min −2.20, p10 +2.99, p50 +5.80, p90 +7.54, max +8.27).
+The bottom 10% of legitimate STRICT score below where the
+candidate-bench NEG cases sat. The distributions **overlap heavily**
+on real bench-qa output; no threshold separates them. This is the
+exact same finding pattern as #000049 §7 #27's "real-haystack recall
+is lower than synthetic" — but on the precision side for relevance.
+
+**Verdict.** The relevance reranker **cannot be promoted to a runtime
+demotion-only veto on this design** — it joins #000049 §7 #27 in
+"clean on candidate-bench, fails the real-traffic gate." The score
+*is* lower on average for off-topic than on-topic (as a soft signal
+it's real), but **not separably-thresholdably so** on real bench-qa
+heterogeneity. The manifest's `demote_below_score` stays `null` and
+`runtime_viability.as_runtime_demotion_veto = "NOT VIABLE on this
+design"`.
+
+**What it CAN still do** (all viable without a threshold):
+- **Advisory soft signal** (render-tail `· relevance: 0.42`) — surface
+  the raw score; let humans judge. No FP problem because no decision
+  is being made.
+- **Multi-signal advisory** — combine with `diagnose_title_relevance`,
+  `diagnose_deflection`, `diagnose_coherence` for a weighted
+  warning. Each signal alone is noisy; the *intersection* might be
+  cleaner.
+- **Contrastive signal** — Q→A relevance *minus* claim→source
+  relevance (or relative ranking *within* an answer's claims). The
+  *delta* may discriminate where the absolute threshold does not.
+  Untested, worth measuring next.
+
+So §3.2 the *runtime veto* is parked; §3.2 the *soft signal* lives
+on as an inspect.py sidecar (same render-layer-only discipline as
+§3.1 `diagnose_coherence`). The bigger lesson, sharpened *again*:
+the relevance and NLI sides of this work both hit the same wall —
+**lexical-candidate selection + cross-encoder scoring + a hard
+threshold doesn't survive real-pipeline heterogeneity**. The §7 #27
+diagnosis (the candidate-clause selector is the architectural
+bottleneck) applies here too: a *semantic* candidate selector
+(vec-driven, sibling of #000050/#000051) might tighten both axes.
+
 ### 3.3 — relationship to #000049
 
 NLI (contradiction) and a relevance model (aboutness) are **siblings**