russell@unturf.com
d57e5c1435
(a) Mined the pooled n=1+3+5 bench-qa runs (808 distinct STRICT answers) for natural recombinations at lowered θc≥0.7 → 37 would-fires, ALL token-collision FPs on inspection (Mount Kenya pulled into a Kilimanjaro answer, Dalí into da Vinci, Donovan into Superman). ZERO genuine recombination errors — the boundary is theoretical-in-practice; the failure mode is the candidate selector (top-k by token overlap) pulling different-entity same-subject-area clauses. (b) Re-ran the grid against the 808-cell pooled STRICT set: the §7 #25 'max@0.96' was itself a small-sample artifact — the n=5 444-cell set lacked the high-confidence spurious hits the pooled set has. On 808 cells θc goes back to ~0.995, and at θc=0.995 only agg=margin still catches 28/28 (max gets 27/28). microsoft/deberta-large-mnli / k=3 / agg=margin / θc=0.995 → 28/28 synthetic recombinations · 0/808 pooled real STRICT FP · 0/26 synthetic legit — the ONLY config in the 7-model×full-grid sweep that hits 1.0/0.0 on 808 cells, held at n=3 too. recommended_operating_point reverted to margin@0.995. Realistic next check: ~20-30 hand-built synthetic-recombination-vs- real-bench-qa-context fixtures (real haystack, deliberate re-attribution). Still SHADOW; runtime promotion fox+dav1d-decides. Production verifier unchanged; falsification-hard stays 10/12. Meta-lesson instance five: a bigger sample can vindicate a config a smaller one made look unnecessary — re-confirm the config choice (not just the threshold) each time the denominator grows. |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||