engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000049 Phase 2: NLI shadow scaffold in arborist (§7 #19)

Browse source 
arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not
yet in governance_policy_hash per §7 #2). manifest.json pins
cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the
bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO;
shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new
[nli] extra, clauses() segmenter, the §7 #5 clause-level Demote()
decision, degrades to available=False when [nli] absent);
bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow
(the gate-item-4 instrument); 16 tests.

First sweep (116 records — 5f-falsification packs + the arborist-nli-bench
eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries,
0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the
contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear
on available data; item 4 — shadow FP rate on a real live-bench-qa
sample — remains the open measurement. Production verifier unchanged;
falsification-hard stays 10/12.
This commit is contained in:
russell@unturf.com 2026-05-12 14:14:21 -04:00
parent 87c92162a1
commit 70ecda3d6c
No known key found for this signature in database
10 changed files with 2543 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

24
Makefile
View file

@ -36,7 +36,7 @@ SEARCH_Q ?= computer
prometheus-trigger-probe bench-5f-threshold-calibration \
bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \
bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
bootstrap-math clean clean-db clean-data help \
bootstrap-math bootstrap-nli bench-nli-shadow clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list
@ -779,6 +779,28 @@ bootstrap-crawler: bootstrap ## install [crawler] extras into the venv
bootstrap-math: bootstrap ## install [math] extras (sympy) into the venv
$(PIP) install -e '.[math]'
# #000049 Phase 2 — sentence-pair NLI runtime for the SHADOW path
# (arborist/qa/nli/). ~600 MB (transformers + CPU torch); hard out of
# core / dev so a fresh checkout stays python3.12 + venv + sqlite3.
# Installs the extra, then warms the pinned checkpoint download so the
# first `make bench-nli-shadow` doesn't pay for it. Operator target —
# NOT part of `make bootstrap`, `make test`, or a fresh checkout.
bootstrap-nli: bootstrap ## install [nli] extras + warm the pinned NLI checkpoint
$(PIP) install -e '.[nli]'
$(PY) -c "from arborist.qa.nli import ShadowNLI; r=ShadowNLI(); r._ensure_loaded(); print('nli runtime ·', 'available:', r.available, '·', r._reason)"
# #000049 Phase 2 / §7 #12 gate item 4 — measure the would-demote rate
# of the clause-level shadow check over (answer, context) records.
# SHADOW ONLY: writes only bench/results/nli-shadow-sweep.json, never an
# audit_mode. With no INPUT it sweeps the 5f falsification packs (every
# record there is a FALSE claim, so a high would-demote rate = the model
# working). Point INPUT at a legit-answer sample for the real gate-item-4
# false-positive number. Renders a structural report even without the
# [nli] extra (marked available:false). Run `make bootstrap-nli` first
# for real numbers.
bench-nli-shadow: bootstrap ## #000049 Phase 2 — NLI shadow would-demote sweep (INPUT="f1.jsonl f2.jsonl" optional)
PYTHONUNBUFFERED=1 $(PY) bench/scripts/nli_shadow_sweep.py $(foreach f,$(INPUT),--input $(f))
# ---------------------------------------------------------------------------
# Public-domain + open-licensed textbooks (#000031 / surface-ingest scope).
#