russell@unturf.com
9fc951592d
All 6 swept rerankers (bge-large/base, MiniLM-L-2/L-4/L-6/L-12) false-fire on 4.5–21.8% of real bench-qa STRICT at their candidate-bench fp=0 θ. The smallest θ that yields fp=0 on real STRICT is BELOW the candidate-bench NEG max for every model — meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG = 0/12 across the board. Structural reason: real bench-qa STRICT answers have a much wider score distribution (bge-large STRICT: min -2.20, p10 +2.99, p50 +5.80, p90 +7.54) than the tight contrived candidate-bench POS band. The bottom 10% of legitimate STRICT score below where the candidate-bench NEG cases sat. Distributions overlap heavily; no threshold separates them. This is the §3.2 mirror of #000049 §7 #27's recall-side walk-back — clean candidate-bench → fails real-pipeline gate. Same diagnosis: lexical-candidate selection + cross-encoder scoring + hard threshold doesn't survive real-pipeline heterogeneity. Verdict: relevance reranker CANNOT be promoted to a runtime demotion-only veto on this design. demote_below_score stays null; manifest gains runtime_viability block documenting the negative result + the still-viable advisory soft-signal uses (render-tail, multi-signal advisory, contrastive Q→A vs claim→source delta). bench/scripts/relevance_shadow_sweep.py + bench/results/ relevance-shadow-sweep-pooled808.json committed. Production verifier unchanged; advisory only; no audit_mode effect. |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||