engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000049 §7 #22: speedup (batch + cuda auto-detect + ONNX-int8 export) + the gate-item-4 verdict at proper n

Browse source 
Speedup (§3 plan): ShadowNLI._nli_batch batches forwards
(ARBORIST_NLI_BATCH=64); device auto-detect (ARBORIST_NLI_DEVICE, else
cuda-if-available); auto-prefer an ONNX export — bench/scripts/export_nli_onnx.py
/ make export-nli-onnx exports + int8-dynamic-quantizes the pinned
checkpoint into ~/.arborist/models/nli/<ver>/onnx/ (operator state, NOT
committed), _ensure_loaded loads model_quantized.onnx via
optimum.onnxruntime (backend onnx-int8), falls back to torch silently.
torch-cpu-batch1 ~120ms/pair → onnx-int8-cpu-batched ~32ms/pair (~4x);
seconds on a 4090. optimum[onnxruntime] added to the [nli] extra; 24
tests.

Gate-item-4 verdict at proper n: ARBORIST_NLI_SHADOW=1 make bench-qa
BENCH_QA_N=1 → 223 cells (89 STRICT / 90 HYBRID / 44 UNGROUNDED; also
surfaced + fixed a lone-surrogate bug). Shadow sweep over those: NLI-as-
runtime-veto on STRICT has ~26% FP at θc 0.5, ~8% at θc 0.90, ~0% only
at θc 0.99 — and θc 0.99 gives up most recombination recall (hard
synthetic recombinations bottom out ~0.76). FAILS the §7 #12 gate on
this design. Only untried path that might pass: a Phase-3 runtime hook
running NLI on the verifier's actual matched clauses (1-3), not
top-6-by-overlap. Until then: runtime NLI demotion stays off; the 2
fixtures stay permanent boundary markers; θc stays 0.5. Production
verifier unchanged; falsification-hard stays 10/12.
This commit is contained in:
russell@unturf.com 2026-05-12 17:21:32 -04:00
parent 96ff586da1
commit 87d9db15c7
No known key found for this signature in database
8 changed files with 4587 additions and 30 deletions
Download patch file Download diff file Expand all files Collapse all files

13
Makefile
View file

@ -36,7 +36,7 @@ SEARCH_Q ?= computer
prometheus-trigger-probe bench-5f-threshold-calibration \
bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \
bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
bootstrap-math bootstrap-nli bench-nli-shadow clean clean-db clean-data help \
bootstrap-math bootstrap-nli bench-nli-shadow export-nli-onnx clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list
@ -787,7 +787,16 @@ bootstrap-math: bootstrap ## install [math] extras (sympy) into the venv
# NOT part of `make bootstrap`, `make test`, or a fresh checkout.
bootstrap-nli: bootstrap ## install [nli] extras + warm the pinned NLI checkpoint
$(PIP) install -e '.[nli]'
$(PY) -c "from arborist.qa.nli import ShadowNLI; r=ShadowNLI(); r._ensure_loaded(); print('nli runtime ·', 'available:', r.available, '·', r._reason)"
$(PY) -c "from arborist.qa.nli import ShadowNLI; r=ShadowNLI(); r._ensure_loaded(); print('nli runtime ·', 'available:', r.available, '· backend:', r.backend, '· device:', r.device, '·', r._reason)"
# #000049 §3 speedup — export the pinned shadow-NLI checkpoint to ONNX
# (+ int8 dynamic quantization), into ~/.arborist/models/nli/<ver>/onnx/.
# ShadowNLI._ensure_loaded auto-prefers the export if present (~2-4x on
# CPU, drops the torch forward path; on cuda uses CUDAExecutionProvider).
# Run once after `make bootstrap-nli`. SHADOW infra — same checkpoint,
# same labels, nothing about audit_mode changes.
export-nli-onnx: bootstrap ## #000049 §3 — export the pinned shadow-NLI checkpoint to ONNX (int8)
PYTHONUNBUFFERED=1 $(PY) bench/scripts/export_nli_onnx.py
# #000049 Phase 2 / §7 #12 gate item 4 — measure the would-demote rate
# of the clause-level shadow check over (answer, context) records.