russell@unturf.com
4352b84508
Built bench/fixtures/5f/recombination-realcontext-v1.jsonl: 25 hand-built recombination claims (deliberate re-attributions within real bench-qa 30 KB Wikipedia haystacks the pipeline produced — Mount Kenya into a Kilimanjaro answer, Galileo onto all 63 Jupiter moons, Joshua Reynolds onto the Mona Lisa, etc.). Ran the full 7-model × full-grid sweep vs the 808 pooled STRICT. Result: - bart-large-mnli / k=12 / max / θc=0.999 → 12/25 = 0.48 catch · 0/808 FP (the real-haystack fp=0 leader) - deberta-large-mnli / k=3 / margin / θc=0.995 → 6/25 = 0.24 (§7 #26's 'settled' config — 28/28 synthetic, 0.24 real-haystack: 4× over-estimate) - roberta-large 0.12, MiniLM 0.08, deberta-base 0.04 So §7 #26's 'boundary closed' walks back to 'boundary PARTIALLY closed' on real haystacks. The bottleneck is architectural: top-k by token overlap misses the contradicting clause when it shares few subject-area tokens with the answer (e.g. the Mount Kenya clause only shares 'Kenya' with a Kilimanjaro claim — ranked low, NLI never sees it). Threshold tuning doesn't lift the ceiling; a SEMANTIC candidate selector (vec-driven, sibling of #000050/#000051's hybrid retrieval) does. bart's pareto above fp=0: fp=0.011 catch=0.52, fp=0.057 catch=0.84, fp=0.068 catch=0.92 — permissive operating points are on the menu if fox+dav1d sign off. recommended_operating_point updated to bart-large-mnli/k=12/max/θc=0.999; deberta-large/margin kept as the synthetic-eval reference. Sixth meta-lesson instance: clean synthetic eval doesn't predict bench-qa precision OR recall — neither contrived dataset axis is load-bearing, only the real pipeline shape is. Production verifier unchanged; falsification-hard stays 10/12. Still SHADOW; runtime promotion fox+dav1d-decides. |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||