russell@unturf.com
4352b84508
Built bench/fixtures/5f/recombination-realcontext-v1.jsonl: 25 hand-built recombination claims (deliberate re-attributions within real bench-qa 30 KB Wikipedia haystacks the pipeline produced — Mount Kenya into a Kilimanjaro answer, Galileo onto all 63 Jupiter moons, Joshua Reynolds onto the Mona Lisa, etc.). Ran the full 7-model × full-grid sweep vs the 808 pooled STRICT. Result: - bart-large-mnli / k=12 / max / θc=0.999 → 12/25 = 0.48 catch · 0/808 FP (the real-haystack fp=0 leader) - deberta-large-mnli / k=3 / margin / θc=0.995 → 6/25 = 0.24 (§7 #26's 'settled' config — 28/28 synthetic, 0.24 real-haystack: 4× over-estimate) - roberta-large 0.12, MiniLM 0.08, deberta-base 0.04 So §7 #26's 'boundary closed' walks back to 'boundary PARTIALLY closed' on real haystacks. The bottleneck is architectural: top-k by token overlap misses the contradicting clause when it shares few subject-area tokens with the answer (e.g. the Mount Kenya clause only shares 'Kenya' with a Kilimanjaro claim — ranked low, NLI never sees it). Threshold tuning doesn't lift the ceiling; a SEMANTIC candidate selector (vec-driven, sibling of #000050/#000051's hybrid retrieval) does. bart's pareto above fp=0: fp=0.011 catch=0.52, fp=0.057 catch=0.84, fp=0.068 catch=0.92 — permissive operating points are on the menu if fox+dav1d sign off. recommended_operating_point updated to bart-large-mnli/k=12/max/θc=0.999; deberta-large/margin kept as the synthetic-eval reference. Sixth meta-lesson instance: clean synthetic eval doesn't predict bench-qa precision OR recall — neither contrived dataset axis is load-bearing, only the real pipeline shape is. Production verifier unchanged; falsification-hard stays 10/12. Still SHADOW; runtime promotion fox+dav1d-decides. |
||
|---|---|---|
| .. | ||
| _source | ||
| diagrams | ||
| tickets | ||
| bench-maxing.md | ||
| benchmarks.md | ||
| calculator-test-patterns.md | ||
| cti-architecture.md | ||
| lexical-first-rationale.md | ||
| mesh.md | ||
| pi-star-composition.md | ||
| qa-modes-bench.md | ||
| seven-point-program.md | ||
| soft-hash-channel-analysis.md | ||
| soft-hash-channel-t3-bound.md | ||
| spec-methodology.md | ||
| TICKETS.md | ||
| tool-action-dag-design.md | ||
| v7w-frontier-catalog.md | ||
| v8-fork-score.md | ||
| warrant-substrate-cookbook.md | ||
| zk-frontier-bench.md | ||
| zk-wire-protocol.md | ||