russell@unturf.com
f4de936ff9
Live hook: ARBORIST_NLI_SHADOW=1 makes query() surface the verifier-input text (gated off-by-default, never a cache_key/governance/audit_mode input); qa_sweep.py carries it + the answer into bench rows; the shadow sweep reads them and buckets by audit_mode. ARBORIST_NLI_SHADOW=1 make bench-qa-smoke (15 cells) → the naive 'NLI on every context clause' scaffold has a ~30% would-demote rate on STRICT answers — a haystack / multiple-comparisons artifact (real contexts segment into 100-336 clauses; max-over-all almost always finds a tangential clause the model reads as contradicting; a paraphrased STRICT answer often isn't verbatim- entailed by any single clause so the entailment guard doesn't rescue it). Lesson: the §7 #5 'candidate source clauses' + recombination-risk gating is load-bearing, not optional. Do NOT enable runtime NLI demotion on the current scaffold; next step is the candidate-clause restriction, then re-run, then gate item 4 is meaningful. Production verifier unchanged; falsification-hard stays 10/12. |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||