feat(#000049 §7 #28): tinygrad NLI backend + deterministic engine-agreement A/B; ONNX-immunity rationale

Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.

bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.

First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).

docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.

Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
This commit is contained in:
russell@unturf.com 2026-05-19 12:34:04 -04:00
parent 031634c69c
commit a4f3e126f7
No known key found for this signature in database
7 changed files with 551 additions and 4 deletions

View file

@ -1368,3 +1368,90 @@ buried — those are *different* problems, and only the second one is
the production problem. The 4090 made this measurable in ~4 min
(7 models × full grid × 833 records); without it we would have
shipped a 24%-recall config under a "1.000" label.
---
**§7 #28 (2026-05-19) — tinygrad as a third NLI inference backend;
deterministic engine-agreement A/B (instrument built in-repo; numbers
pending a producer-box run).** Context: the Sherief/tiny-corp thread
(ONNX Runtime's ~2,500-line many-EP build vs tinygrad's minimalist
ONNX frontend that reportedly passes more of the official ONNX test
suite than ORT itself) raised a fox question — can the shadow-NLI
checkpoint run under tinygrad's ONNX frontend instead of / alongside
`optimum.onnxruntime`? This is a legitimate §7 sub-item, not a new
ticket: it is the *same* speedup/backend axis as §7 #22 (the
ONNX-int8 export), one more interchangeable accelerator below the
cage. It also makes the abstract immunity argument concrete — see the
new `docs/onnx-vendor-capture-immunity.md` (the engine is never a
trust dependency; this bench is the demonstration, not a migration).
**The gate is agreement, not speed.** The primary acceptance metric
is **per-pair logit/probability agreement with the `transformers`+
`torch` reference within tolerance** over the same pinned checkpoint
and the same input pairs — *then* p50/p95 latency as the secondary
axis. Rationale, in the exact shape of this ticket's standing lesson:
a faster engine that produces *different* `(p_entail, p_neutral,
p_contra)` is not a faster shadow — it is a *different* shadow signal.
Under the deliberately-unwired Phase-3 demotion hook that would mean a
different `nli_policy_hash` folding into `governance_policy_hash` (§7
#2) and a silently different demotion surface. So: engine-swap is only
admissible if it is *numerically* the same model, measured, not
assumed. "Honest verdicts beat optimistic ones" applies to the
instrument too.
**Honest risk, stated up front (don't trust the contrived assumption
until confirmed on the real shape).** The current
`recommended_operating_point` is `bart-large-mnli` (§7 #27); the
synthetic alternate is `deberta-large-mnli`. tinygrad's ONNX frontend
op-coverage for *these large MNLI cross-encoders* (Erf/GELU, fused
attention, `GatherElements`, dynamic shapes) is **unverified** — it
is an empirical question with a real chance of "frontend rejects the
graph" or "loads but diverges." That outcome is itself a valid,
publishable result (it sharpens the immunity doc: torch is the
always-works floor, ORT the proven accelerator, tinygrad a *candidate*
whose coverage is checkpoint-specific). No assumption of success is
baked into the plan.
**What landed in this commit (instrument before experiment):**
(1) a third backend branch in `arborist/qa/nli/shadow.py`,
`backend == "tinygrad"`, parallel to the onnx/torch branches —
optional `import`, graceful `available=False` on any failure,
SHADOW-only, never an `audit_mode` input, GPU-producer-box only (the
#000051 "heavy passes never in arborist's python+sqlite3 core"
pattern; arborist core stays python3.12+venv+sqlite3); selectable via
`ARBORIST_NLI_BACKEND`. (2) A deterministic A/B instrument
`bench/scripts/nli_backend_ab.py` + `make bench-nli-backends`: loads
the pinned checkpoint under each *available* backend, runs an
identical fixed pair set, reports max/mean abs probability delta vs
the torch reference and p50/p95 latency per backend — no LLM, no
verifier, no n=3 noise, no 5pp floor (the `recall_at_k` deterministic-
instrument discipline). **Pending:** the numbers. tinygrad is not in
arborist's deps and the run belongs on the GPU producer box, not the
core dev box — so this is fox/producer-box-decides to (a) install
tinygrad there and (b) `make bench-nli-backends`. SHADOW throughout;
manifest active defaults and the proof path are byte-unchanged by
construction (the immunity property, exercised).
*First instrument run (2026-05-19, CPU dev box, tinygrad absent — a
smoke of the instrument, not the experiment):* the A/B already
produced a real finding before tinygrad is even in scope. The
**currently-deployed §7 #22 int8 ONNX export diverges from the torch
reference by Δmax ≈ 0.42 per-class probability** on the fixed 24-pair
probe (24 pairs × 5 repeats; mean abs delta smaller but the max is
large). This quantifies, with a deterministic instrument, the prior
hand-wave "int8-ONNX costs ≈1 catch": int8 is a *lossy-by-
construction different numerical model*, and the A/B honestly reports
it as `DIVERGES` at a tight (2e-3) faithfulness tolerance. This is
**not a defect** — it is the immunity property made measurable: even
the shipped accelerator is a numerical surrogate, which is safe only
because it is a shadow that never reaches the proof path
(`docs/onnx-vendor-capture-immunity.md`). It does sharpen the eventual
Phase-3 question: if NLI is ever wired to demote, the engine+
quantization must be pinned into `nli_policy_hash` (an int8 export and
its fp32 source are *different policies*, and this instrument now
proves it numerically). Instrument-honesty caveat encoded: when
`ARBORIST_NLI_BACKEND=tinygrad` is requested but the frontend cannot
load the checkpoint, ShadowNLI correctly degrades to the proven path,
but the A/B reports **tinygrad-unavailable**, never relabels the
fallback's numbers as tinygrad's. Pending unchanged: real tinygrad
numbers need the producer box.