russell@unturf.com
e4cc3293b5
Hand-inspection of the bottom-15 STRICT-fires from the raw §3.2.2 step 2 sweep showed claim-lattice overlay markup ([E\d+ | title | hash: '…']) depressing scores on correct concise answers (the 6× Henry-VIII case), while true-positive deflections (broad-question / narrow-answer like 'winners of all major sports?' → just-one-sport) remained correctly low-scored. So the noise FP class is the bracket metadata; cleaning it should reduce FP without losing true-positive signal. Built clean_for_relevance() in arborist/qa/relevance/shadow.py — strips [E\d+ | ... ] blocks + trailing '...']' tails. Baked into ShadowRelevance.check_question_answer / check_claim_source by default (opt out with clean_input=False). relevance_shadow_sweep.py applies it to inputs before _score_batch (opt out with --no-clean). Re-ran the full 6-model sweep on the 808-cell pooled STRICT with cleaning: bge-reranker-large 21.8% → 18.6% (-3.2) MiniLM-L-4-v2 15.0% → 9.4% (-5.6 pts, -37% rel) MiniLM-L-6-v2 11.6% → 9.0% (-2.6) MiniLM-L-12-v2 11.0% → 8.0% (-3.0) bge-reranker-base 9.5% → 9.0% (-0.5) MiniLM-L-2-v2 4.5% → 1.5% (-3.0 pts, -67% rel) Universal improvement, every model better. Big surprise: MiniLM-L-2-v2 — the model that FAILED the candidate-bench separability (margin -1.97, declared 'capacity floor') — has the LOWEST real-traffic FP rate at its own cb θ (1.5%). Because L-2's compressed score range gives it a low cb θ which few real STRICT pairs score below. SEVENTH instance of 'candidate-bench doesn't predict real-traffic'. Runtime-veto verdict UNCHANGED — still not viable; smallest fp=0 θ on real STRICT is below the cb NEG max for every model, so at any runtime-safe θ the catch on cb NEG is 0/12. But cleaning is now FREE improvement for any soft-signal / advisory / contrastive use of the relevance score. Hand-inspected bottom-10 post-cleaning confirms true- positive deflection signal preserved. |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||