#000049 Phase 2: NLI shadow scaffold in arborist (§7 #19)

arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not
yet in governance_policy_hash per §7 #2). manifest.json pins
cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the
bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO;
shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new
[nli] extra, clauses() segmenter, the §7 #5 clause-level Demote()
decision, degrades to available=False when [nli] absent);
bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow
(the gate-item-4 instrument); 16 tests.

First sweep (116 records — 5f-falsification packs + the arborist-nli-bench
eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries,
0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the
contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear
on available data; item 4 — shadow FP rate on a real live-bench-qa
sample — remains the open measurement. Production verifier unchanged;
falsification-hard stays 10/12.
This commit is contained in:
russell@unturf.com 2026-05-12 14:14:21 -04:00
parent 87c92162a1
commit 70ecda3d6c
No known key found for this signature in database
10 changed files with 2543 additions and 8 deletions

View file

@ -79,6 +79,23 @@ vec = [
"sqlite-vec>=0.1.9",
"fastembed>=0.4",
]
nli = [
# Sentence-pair NLI for the #000049 Phase-2 *shadow* path
# (arborist/qa/nli/) — measures whether a clause-level contradiction
# veto would demote a weakly-grounded answer; never touches
# audit_mode. transformers + a CPU torch is ~600 MB installed, so it
# is gated hard out of core / dev — a fresh checkout stays
# python3.12 + venv + sqlite3, and the default test suite skips the
# NLI tests via pytest.importorskip when this extra is absent.
# Install with:
# pip install 'arborist[nli]'
# Phase 3 (if it happens) should ONNX-export the pinned checkpoint
# and switch this to onnxruntime-cpu to drop torch (cf. [vec]).
"transformers>=4.40",
"torch>=2.2",
"sentencepiece>=0.2",
"protobuf>=4.0",
]
dev = [
"pytest>=8",
"pytest-asyncio>=0.23",