russell@unturf.com
421b2c5fd5
The §7 #22 'fails the gate' was the verdict for the DEFAULT config (k=6/θc=0.5/θe=0.9, tuned on the clean synthetic set), not the approach. A {model × candidate-cap k × aggregation × θc × θe} grid sweep (bench/scripts/nli_shadow_grid.py — NLI runs once per (model,record) over the top-12 candidate clauses, the k/agg/θ grid is then arithmetic on cached scores; ~10s on the 4090 for 4 models) finds clean passing configs: on the 89 real STRICT cells (n=1 bench-qa), deberta-base-184M / k=2 / agg=max / θc=0.99 / θe=0.9 → 27/28 synthetic recombinations caught (incl. both 5f-fal-hard fixtures), 0/89 STRICT FP. MiniLM-82M passes too (24/28 · 0/89). Model science: 184M > 82M > 407M for fp=0 recombination recall; int8-ONNX costs ~1 catch vs fp32. Caveats: FP side is n=1 (BENCH_QA_N=3 run in flight); recall is on the synthetic set; flipping to a runtime demotion-only veto is fox-decides (then nli_policy_hash folds into governance_policy_hash per §7 #2). Manifest active defaults stay k=6/θc=0.5; recommended_operating_point (deberta-base, k=2, agg=max, θc=0.99, θe=0.9) documented in the manifest. 3 grid result JSONs committed. Standing lesson: neither the clean synthetic eval NOR the default config predicts bench-qa precision — you have to sweep. Production verifier unchanged; falsification-hard stays 10/12. |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||