arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not yet in governance_policy_hash per §7 #2). manifest.json pins cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO; shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new [nli] extra, clauses() segmenter, the §7 #5 clause-level Demote() decision, degrades to available=False when [nli] absent); bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow (the gate-item-4 instrument); 16 tests. First sweep (116 records — 5f-falsification packs + the arborist-nli-bench eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries, 0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear on available data; item 4 — shadow FP rate on a real live-bench-qa sample — remains the open measurement. Production verifier unchanged; falsification-hard stays 10/12.
This commit is contained in:
parent
87c92162a1
commit
70ecda3d6c
10 changed files with 2543 additions and 8 deletions
|
|
@ -79,6 +79,23 @@ vec = [
|
|||
"sqlite-vec>=0.1.9",
|
||||
"fastembed>=0.4",
|
||||
]
|
||||
nli = [
|
||||
# Sentence-pair NLI for the #000049 Phase-2 *shadow* path
|
||||
# (arborist/qa/nli/) — measures whether a clause-level contradiction
|
||||
# veto would demote a weakly-grounded answer; never touches
|
||||
# audit_mode. transformers + a CPU torch is ~600 MB installed, so it
|
||||
# is gated hard out of core / dev — a fresh checkout stays
|
||||
# python3.12 + venv + sqlite3, and the default test suite skips the
|
||||
# NLI tests via pytest.importorskip when this extra is absent.
|
||||
# Install with:
|
||||
# pip install 'arborist[nli]'
|
||||
# Phase 3 (if it happens) should ONNX-export the pinned checkpoint
|
||||
# and switch this to onnxruntime-cpu to drop torch (cf. [vec]).
|
||||
"transformers>=4.40",
|
||||
"torch>=2.2",
|
||||
"sentencepiece>=0.2",
|
||||
"protobuf>=4.0",
|
||||
]
|
||||
dev = [
|
||||
"pytest>=8",
|
||||
"pytest-asyncio>=0.23",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue