feat(#000049 §7 #28): tinygrad NLI backend + deterministic engine-agreement A/B; ONNX-immunity rationale
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049 cage: shadow-only, never an audit_mode input, never auto-preempts the proven path (guarded so it cannot regress torch/onnx). Loads the ONNX export through tinygrad's frontend wrapped to the existing model(**enc).logits contract so _nli_batch is byte-unchanged. bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic A/B, gate is numerical agreement with the torch reference first, latency second (a divergent engine = a different shadow signal = a different nli_policy_hash). Instrument is honest — reports a requested backend as unavailable rather than relabelling a fallback's numbers. First CPU-smoke run already quantified that the deployed §7 #22 int8 export diverges Δmax≈0.42 from torch — the immunity property made measurable, not a defect. Real tinygrad numbers pending a producer-box run (tinygrad not an arborist dep; frontend op-coverage for the large MNLI checkpoints unverified by design). docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage makes the inference engine an interchangeable sidecar, never a trust dependency — public-domain positioning capital. Indexed in CLAUDE.md. Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
This commit is contained in:
parent
031634c69c
commit
a4f3e126f7
7 changed files with 551 additions and 4 deletions
|
|
@ -1368,3 +1368,90 @@ buried — those are *different* problems, and only the second one is
|
|||
the production problem. The 4090 made this measurable in ~4 min
|
||||
(7 models × full grid × 833 records); without it we would have
|
||||
shipped a 24%-recall config under a "1.000" label.
|
||||
|
||||
---
|
||||
|
||||
**§7 #28 (2026-05-19) — tinygrad as a third NLI inference backend;
|
||||
deterministic engine-agreement A/B (instrument built in-repo; numbers
|
||||
pending a producer-box run).** Context: the Sherief/tiny-corp thread
|
||||
(ONNX Runtime's ~2,500-line many-EP build vs tinygrad's minimalist
|
||||
ONNX frontend that reportedly passes more of the official ONNX test
|
||||
suite than ORT itself) raised a fox question — can the shadow-NLI
|
||||
checkpoint run under tinygrad's ONNX frontend instead of / alongside
|
||||
`optimum.onnxruntime`? This is a legitimate §7 sub-item, not a new
|
||||
ticket: it is the *same* speedup/backend axis as §7 #22 (the
|
||||
ONNX-int8 export), one more interchangeable accelerator below the
|
||||
cage. It also makes the abstract immunity argument concrete — see the
|
||||
new `docs/onnx-vendor-capture-immunity.md` (the engine is never a
|
||||
trust dependency; this bench is the demonstration, not a migration).
|
||||
|
||||
**The gate is agreement, not speed.** The primary acceptance metric
|
||||
is **per-pair logit/probability agreement with the `transformers`+
|
||||
`torch` reference within tolerance** over the same pinned checkpoint
|
||||
and the same input pairs — *then* p50/p95 latency as the secondary
|
||||
axis. Rationale, in the exact shape of this ticket's standing lesson:
|
||||
a faster engine that produces *different* `(p_entail, p_neutral,
|
||||
p_contra)` is not a faster shadow — it is a *different* shadow signal.
|
||||
Under the deliberately-unwired Phase-3 demotion hook that would mean a
|
||||
different `nli_policy_hash` folding into `governance_policy_hash` (§7
|
||||
#2) and a silently different demotion surface. So: engine-swap is only
|
||||
admissible if it is *numerically* the same model, measured, not
|
||||
assumed. "Honest verdicts beat optimistic ones" applies to the
|
||||
instrument too.
|
||||
|
||||
**Honest risk, stated up front (don't trust the contrived assumption
|
||||
until confirmed on the real shape).** The current
|
||||
`recommended_operating_point` is `bart-large-mnli` (§7 #27); the
|
||||
synthetic alternate is `deberta-large-mnli`. tinygrad's ONNX frontend
|
||||
op-coverage for *these large MNLI cross-encoders* (Erf/GELU, fused
|
||||
attention, `GatherElements`, dynamic shapes) is **unverified** — it
|
||||
is an empirical question with a real chance of "frontend rejects the
|
||||
graph" or "loads but diverges." That outcome is itself a valid,
|
||||
publishable result (it sharpens the immunity doc: torch is the
|
||||
always-works floor, ORT the proven accelerator, tinygrad a *candidate*
|
||||
whose coverage is checkpoint-specific). No assumption of success is
|
||||
baked into the plan.
|
||||
|
||||
**What landed in this commit (instrument before experiment):**
|
||||
(1) a third backend branch in `arborist/qa/nli/shadow.py`,
|
||||
`backend == "tinygrad"`, parallel to the onnx/torch branches —
|
||||
optional `import`, graceful `available=False` on any failure,
|
||||
SHADOW-only, never an `audit_mode` input, GPU-producer-box only (the
|
||||
#000051 "heavy passes never in arborist's python+sqlite3 core"
|
||||
pattern; arborist core stays python3.12+venv+sqlite3); selectable via
|
||||
`ARBORIST_NLI_BACKEND`. (2) A deterministic A/B instrument
|
||||
`bench/scripts/nli_backend_ab.py` + `make bench-nli-backends`: loads
|
||||
the pinned checkpoint under each *available* backend, runs an
|
||||
identical fixed pair set, reports max/mean abs probability delta vs
|
||||
the torch reference and p50/p95 latency per backend — no LLM, no
|
||||
verifier, no n=3 noise, no 5pp floor (the `recall_at_k` deterministic-
|
||||
instrument discipline). **Pending:** the numbers. tinygrad is not in
|
||||
arborist's deps and the run belongs on the GPU producer box, not the
|
||||
core dev box — so this is fox/producer-box-decides to (a) install
|
||||
tinygrad there and (b) `make bench-nli-backends`. SHADOW throughout;
|
||||
manifest active defaults and the proof path are byte-unchanged by
|
||||
construction (the immunity property, exercised).
|
||||
|
||||
*First instrument run (2026-05-19, CPU dev box, tinygrad absent — a
|
||||
smoke of the instrument, not the experiment):* the A/B already
|
||||
produced a real finding before tinygrad is even in scope. The
|
||||
**currently-deployed §7 #22 int8 ONNX export diverges from the torch
|
||||
reference by Δmax ≈ 0.42 per-class probability** on the fixed 24-pair
|
||||
probe (24 pairs × 5 repeats; mean abs delta smaller but the max is
|
||||
large). This quantifies, with a deterministic instrument, the prior
|
||||
hand-wave "int8-ONNX costs ≈1 catch": int8 is a *lossy-by-
|
||||
construction different numerical model*, and the A/B honestly reports
|
||||
it as `DIVERGES` at a tight (2e-3) faithfulness tolerance. This is
|
||||
**not a defect** — it is the immunity property made measurable: even
|
||||
the shipped accelerator is a numerical surrogate, which is safe only
|
||||
because it is a shadow that never reaches the proof path
|
||||
(`docs/onnx-vendor-capture-immunity.md`). It does sharpen the eventual
|
||||
Phase-3 question: if NLI is ever wired to demote, the engine+
|
||||
quantization must be pinned into `nli_policy_hash` (an int8 export and
|
||||
its fp32 source are *different policies*, and this instrument now
|
||||
proves it numerically). Instrument-honesty caveat encoded: when
|
||||
`ARBORIST_NLI_BACKEND=tinygrad` is requested but the frontend cannot
|
||||
load the checkpoint, ShadowNLI correctly degrades to the proven path,
|
||||
but the A/B reports **tinygrad-unavailable**, never relabels the
|
||||
fallback's numbers as tinygrad's. Pending unchanged: real tinygrad
|
||||
numbers need the producer box.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue