engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000049 §7 #22: make bootstrap-nli-only (lean GPU/CPU NLI compute-node setup) + GPU benchmark

Browse source 
bootstrap-nli-only: a venv + [nli] only — no [dev] extras (no crawler/
hessian/vec/sympy). On a CUDA host PyPI's torch wheel is the CUDA build,
so ShadowNLI auto-detects cuda and bench-nli-shadow / export-nli-onnx /
the candidate bench all run on the GPU with no further wiring. Verified
on the ai box (RTX 4090): torch 2.11+cu130, cuda True; 82M cross-encoder
batched ≈ 0.09 ms/pair (512 pairs in 0.047s — ~350x onnx-int8-cpu,
~1300x torch-cpu-batch1); 62-record synthetic shadow sweep in 2.4s wall;
24/24 tests pass. The GPU only accelerates the NLI half — make bench-qa
(Hermes LLM) still runs wherever the shard corpus is.
This commit is contained in:
russell@unturf.com 2026-05-12 17:55:54 -04:00
parent 87d9db15c7
commit 68c6665db9
No known key found for this signature in database
2 changed files with 31 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

16
Makefile
View file

@ -36,7 +36,7 @@ SEARCH_Q ?= computer
prometheus-trigger-probe bench-5f-threshold-calibration \
bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \
bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
bootstrap-math bootstrap-nli bench-nli-shadow export-nli-onnx clean clean-db clean-data help \
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list
@ -789,6 +789,20 @@ bootstrap-nli: bootstrap ## install [nli] extras + warm the pinned NLI checkpoin
$(PIP) install -e '.[nli]'
$(PY) -c "from arborist.qa.nli import ShadowNLI; r=ShadowNLI(); r._ensure_loaded(); print('nli runtime ·', 'available:', r.available, '· backend:', r.backend, '· device:', r.device, '·', r._reason)"
# #000049 §7 #22 — lean NLI compute-node bootstrap: a venv + `[nli]` ONLY
# (no `[dev]` extras — no crawler/hessian/vec/sympy). For a GPU box that
# just runs NLI sweeps: on a CUDA host PyPI's torch wheel is the CUDA
# build, so `ShadowNLI` auto-detects cuda (ARBORIST_NLI_DEVICE overrides)
# and `make bench-nli-shadow` / `make export-nli-onnx` / the candidate
# bench all run on the GPU with no further wiring. Measured: an 82M
# cross-encoder, batched, on a 4090 ≈ 0.09 ms/pair (~350x onnx-int8-cpu,
# ~1300x torch-cpu-batch1) → a full bench-qa shadow sweep is <1 s of NLI.
bootstrap-nli-only: ## #000049 §7 #22 — minimal venv + [nli] only (GPU/CPU compute-node setup; no [dev])
[ -d $(VENV) ] || $(PYTHON) -m venv $(VENV)
$(PIP) install -q -U pip
$(PIP) install -e '.[nli]' pytest
$(PY) -c "import torch; from arborist.qa.nli import ShadowNLI; r=ShadowNLI(); r._ensure_loaded(); print('torch', torch.__version__, '· cuda', torch.cuda.is_available(), (torch.cuda.get_device_name(0) if torch.cuda.is_available() else '')); print('nli runtime ·', 'available:', r.available, '· backend:', r.backend, '· device:', r.device, '·', r._reason)"
# #000049 §3 speedup — export the pinned shadow-NLI checkpoint to ONNX
# (+ int8 dynamic quantization), into ~/.arborist/models/nli/<ver>/onnx/.
# ShadowNLI._ensure_loaded auto-prefers the export if present (~2-4x on

20
docs/tickets/ticket-000049-attribution-aware-grounding-check.md
View file

@ -844,10 +844,22 @@ state, **not** committed — same discipline as the textbook manifest /
`[vec]`), and `_ensure_loaded` loads `model_quantized.onnx` via
`optimum.onnxruntime` (backend `onnx-int8`), falling back silently to
torch when no export / no `optimum`. Measured: torch-cpu-batch1 ≈ 120
ms/pair → **onnx-int8-cpu-batched ≈ 32 ms/pair** (~4×); on a CUDA box
(the 4090) batched inference is ~10⁴–10⁵ pairs/s — the whole sweep is
seconds. `optimum[onnxruntime]` added to the `[nli]` extra; 24 tests in
`tests/test_nli_shadow.py`.
ms/pair → **onnx-int8-cpu-batched ≈ 32 ms/pair** (~4×) → **torch-cuda-
batched ≈ 0.09 ms/pair on a 4090** (512 pairs in 0.047 s — ~350×
onnx-int8-cpu, ~1300× the original), so a full `bench-qa` n=3 shadow
sweep (~8 k clause-pairs) is **< 1 s** of NLI compute on the GPU
(2.4 s wall for a 62-record synthetic sweep, model load included).
`optimum[onnxruntime]` added to the `[nli]` extra; 24 tests in
`tests/test_nli_shadow.py`. **GPU compute-node workflow** (verified on
`ai`, RTX 4090): `make bootstrap-nli-only` (a venv + `[nli]` only — no
`[dev]` bloat; PyPI's torch wheel is the CUDA build on a CUDA host) →
`ShadowNLI` auto-detects cuda (`ARBORIST_NLI_DEVICE` overrides) → `make
bench-nli-shadow [INPUT=…]` / `make export-nli-onnx` / the candidate
bench all run on the GPU with no further wiring. The pattern: ship the
laptop's `bench-qa` JSONL to the GPU box, run the sweep there in
seconds. (`ai` doesn't have the ~6 GB shard corpus, so `make bench-qa`
itself still runs on the laptop — that's the Hermes LLM, not local
GPU; the GPU only accelerates the NLI half.)
*The gate-item-4 number, at proper n.* `ARBORIST_NLI_SHADOW=1 make
bench-qa BENCH_QA_N=1` → 75 q × 3 modes, of which 223 cells completed