arborist/bench/scripts
russell@unturf.com a4f3e126f7
feat(#000049 §7 #28): tinygrad NLI backend + deterministic engine-agreement A/B; ONNX-immunity rationale
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.

bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.

First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).

docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.

Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
2026-05-19 12:34:04 -04:00
..
__init__.py #000026 Phase 2: real-shard baseline + lazy concept loading 2026-05-08 13:10:58 -04:00
demo_plot.py fan-out: witness audit + 5F extractor + function-sampled demo + docs 2026-05-09 13:19:30 -04:00
export_nli_onnx.py #000049 §7 #22: speedup (batch + cuda auto-detect + ONNX-int8 export) + the gate-item-4 verdict at proper n 2026-05-12 17:21:32 -04:00
fivef_threshold_calibration.py #000047 — close: delta_aggregator knob on ForkScore (Option D) 2026-05-11 08:27:38 -04:00
harvest_falsification_proposals.py #000037 follow-through: persist falsification proposals + harvest live stream 2026-05-10 19:40:00 -04:00
nli_backend_ab.py feat(#000049 §7 #28): tinygrad NLI backend + deterministic engine-agreement A/B; ONNX-immunity rationale 2026-05-19 12:34:04 -04:00
nli_shadow_grid.py nli_shadow_grid: expand to no-stone-unturned sweep — 7 aggregations (incl. margin + paired-entail guard), finer θc grid (20 pts up to 0.999), --extra-models flag, global-best-fp=0 + Pareto-frontier output per model 2026-05-12 18:30:40 -04:00
nli_shadow_sweep.py #000049 §7 #21: candidate-clause restriction in ShadowNLI.check — helps, doesn't close it 2026-05-12 14:34:44 -04:00
phi_alignment_probe.py ticket #000034 Phase 1a: φ_linear alignment probe + KAT fixture 2026-05-10 10:23:38 -04:00
prometheus_sigma_sweep_dryrun.py #000037 §22: dry-run swap to sweep_weights() + §15.4 added 2026-05-10 19:22:03 -04:00
real_shard_baseline.py #000026 Phase 2: real-shard baseline + lazy concept loading 2026-05-08 13:10:58 -04:00
relevance_shadow_grid.py #000052 §3.2 step 1: candidate-bench fixtures (13 POS + 13 NEG) + relevance_shadow_grid.py — sweep manifest models on aboutness, rank by separation margin (not raw score, per §7 #18) 2026-05-13 09:53:10 -04:00
relevance_shadow_sweep.py #000052 §3.2.2 refinement: claim-lattice metadata cleaning — STRICT FP drops universally (-0.5 to -5.6 pts), L-2 down to 1.5% 2026-05-13 15:01:50 -04:00
selfmodel_chain_snapshot.py #000025 §10.11 + §10.13 + §10.14 — close the 5F battery 2026-05-11 07:41:37 -04:00
t3_bound_calculator.py ticket #000036 Tier-2: dav1d Option B (conservative B1 envelope) applied in v1 2026-05-11 07:06:50 -04:00
textbooks_manifest.py ticket #000031 follow-ups B-1 + B-2: alias attribution + source-side title author 2026-05-10 09:35:49 -04:00
witness_sweep.py fan-out: close #000030 · composition fixtures · witness end-to-end 2026-05-09 13:29:59 -04:00
witness_sweep_cron.sh fan-out: witness automation · #000018 analysis · #000033 pillar VII (Claude) 2026-05-09 14:36:39 -04:00
witness_to_5f.py fan-out: witness audit + 5F extractor + function-sampled demo + docs 2026-05-09 13:19:30 -04:00