#000049 §7 #20: first real-traffic NLI shadow sweep — the haystack problem

Live hook: ARBORIST_NLI_SHADOW=1 makes query() surface the verifier-input
text (gated off-by-default, never a cache_key/governance/audit_mode
input); qa_sweep.py carries it + the answer into bench rows; the shadow
sweep reads them and buckets by audit_mode. ARBORIST_NLI_SHADOW=1 make
bench-qa-smoke (15 cells) → the naive 'NLI on every context clause'
scaffold has a ~30% would-demote rate on STRICT answers — a haystack /
multiple-comparisons artifact (real contexts segment into 100-336
clauses; max-over-all almost always finds a tangential clause the model
reads as contradicting; a paraphrased STRICT answer often isn't verbatim-
entailed by any single clause so the entailment guard doesn't rescue it).
Lesson: the §7 #5 'candidate source clauses' + recombination-risk gating
is load-bearing, not optional. Do NOT enable runtime NLI demotion on the
current scaffold; next step is the candidate-clause restriction, then
re-run, then gate item 4 is meaningful. Production verifier unchanged;
falsification-hard stays 10/12.

bench/scripts/nli_shadow_sweep.py

@@ -55,8 +55,16 @@ def _records(path: Path):
         source = obj.get("context") or obj.get("source")
         if not claim or not source:
             continue
-        bucket = obj.get("expected_reason") or obj.get("want") or "unlabeled"
-        is_fp_probe = (obj.get("want") == "not_contradiction")
+        bucket = (obj.get("expected_reason") or obj.get("want")
+                  or obj.get("audit_mode") or "unlabeled")
+        # On real bench-qa rows there is no ground truth; treat a
+        # would_demote on a STRICT row as a *potential* false positive
+        # worth surfacing (the lexical verifier was confident, NLI
+        # disagrees). On nli-bench eval rows `want=not_contradiction`
+        # is the authoritative FP probe.
+        is_fp_probe = (obj.get("want") == "not_contradiction"
+                       or (obj.get("want") is None and obj.get("expected_reason") is None
+                           and obj.get("audit_mode") == "STRICT"))
         yield {"id": obj.get("id", path.stem), "claim": claim, "source": source,
                "bucket": bucket, "is_fp_probe": is_fp_probe, "src_file": path.name}