engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#000049 §7 #28): tinygrad NLI backend + deterministic engine-agreement A/B; ONNX-immunity rationale

Browse source 
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.

bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.

First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).

docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.

Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
This commit is contained in:
russell@unturf.com 2026-05-19 12:34:04 -04:00
parent 031634c69c
commit a4f3e126f7
No known key found for this signature in database
7 changed files with 551 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

4
CLAUDE.md
View file

@ -562,6 +562,10 @@ Architecture / ongoing work:
of soft-hash covert channel risks under M0/M1/M2 threat models.
- `docs/soft-hash-channel-t3-bound.md` — #000036 closed-form
per-window budget bound; pairs with t3_bound_calculator.
- `docs/onnx-vendor-capture-immunity.md` — why the model-in-proof-
path cage (#000049) makes the inference engine (ORT / torch /
tinygrad) an interchangeable sidecar, never a trust dependency;
public-domain positioning capital.
- `docs/calculator-test-patterns.md` — sister discipline for
testing calculator-style code (12 patterns; see #000044 for
numeric drift discipline).

12
Makefile
View file

@ -37,7 +37,7 @@ SEARCH_Q ?= computer
prometheus-trigger-probe bench-5f-threshold-calibration \
bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \
bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx judge-self-test control-ab clean clean-db clean-data help \
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list
@ -836,6 +836,16 @@ bootstrap-nli-only: ## #000049 §7 #22 — minimal venv + [nli] only (GPU/CPU co
export-nli-onnx: bootstrap ## #000049 §3 — export the pinned shadow-NLI checkpoint to ONNX (int8)
PYTHONUNBUFFERED=1 $(PY) bench/scripts/export_nli_onnx.py
# #000049 §7 #28 — deterministic inference-backend A/B: same pinned
# checkpoint under torch / onnx-int8 / tinygrad. Gate is AGREEMENT with
# the torch reference first, latency second. SHADOW ONLY (writes only
# bench/results/nli-backend-ab.json). tinygrad is not an arborist dep —
# run this on the GPU producer box after installing tinygrad + an ONNX
# export (`make export-nli-onnx`). Reports backends honestly as
# unavailable when they don't load (e.g. tinygrad frontend op-coverage).
bench-nli-backends: bootstrap ## #000049 §7 #28 — torch vs onnx-int8 vs tinygrad agreement+latency A/B
PYTHONUNBUFFERED=1 $(PY) bench/scripts/nli_backend_ab.py $(if $(REPEATS),--repeats $(REPEATS),)
# #000049 Phase 2 / §7 #12 gate item 4 — measure the would-demote rate
# of the clause-level shadow check over (answer, context) records.
# SHADOW ONLY: writes only bench/results/nli-shadow-sweep.json, never an

102
arborist/qa/nli/shadow.py
View file

@ -152,7 +152,14 @@ class ShadowNLI:
self.device_pref: str = device or os.environ.get("ARBORIST_NLI_DEVICE") or "auto"
self.device: Optional[str] = None # resolved at load
self.batch_size: int = int(os.environ.get("ARBORIST_NLI_BATCH", "64"))
self.backend: Optional[str] = None # "onnx" | "torch", set at load
# opt-in engine override (§7 #28). Default "" = the proven auto path
# (onnx-int8 export if present, else torch). "tinygrad" is opt-in
# only: its ONNX-frontend op-coverage for the large MNLI
# cross-encoders is unverified, so it never auto-preempts the
# proven path — honest-verdicts discipline + the cage (engine is
# never a trust dependency; see docs/onnx-vendor-capture-immunity.md).
self.backend_pref: str = (os.environ.get("ARBORIST_NLI_BACKEND") or "").strip().lower()
self.backend: Optional[str] = None # "onnx" | "onnx-int8" | "torch" | "tinygrad", set at load
self.available = False
self._reason = "uninitialised"
self._tok = None
@ -186,12 +193,27 @@ class ShadowNLI:
self.device = "cuda" if torch.cuda.is_available() else "cpu"
else:
self.device = self.device_pref
# §7 #28 — opt-in tinygrad ONNX-frontend backend. Tried ONLY when
# ARBORIST_NLI_BACKEND=tinygrad is set; never auto-preferred (its
# op-coverage for these checkpoints is unverified — measured by
# `make bench-nli-backends`, not assumed). SHADOW only; the engine
# swap is invisible to the proof path by construction. Any failure
# falls through to the proven onnx/torch auto path below.
if self.backend_pref == "tinygrad":
try:
self._tok, self._model = self._try_tinygrad()
if self._model is not None:
self.backend = "tinygrad"
except Exception: # noqa: BLE001 — best-effort; fall through
self._tok = self._model = None
self.backend = None
# prefer an ONNX export if one exists and `optimum` is importable
# (§3.1 of the speedup plan — ~2-4x on CPU, drops the torch fwd path;
# on cuda uses CUDAExecutionProvider). Falls back to torch silently.
onnx_dir = self._onnx_dir()
try:
if onnx_dir and onnx_dir.exists():
if self._model is None and onnx_dir and onnx_dir.exists():
from optimum.onnxruntime import ORTModelForSequenceClassification
provider = ("CUDAExecutionProvider" if self.device == "cuda"
else "CPUExecutionProvider")
@ -229,6 +251,82 @@ class ShadowNLI:
self.available = True
self._reason = "ok"
def _try_tinygrad(self):
"""§7 #28 — load the pinned checkpoint's ONNX export through
tinygrad's ONNX frontend, wrapped so it satisfies the exact
``model(**enc).logits`` (torch tensor) contract ``_nli_batch``
already uses — so the proven batched forward path is byte-
unchanged (zero regression risk to torch/onnx).
Returns ``(tokenizer, adapter)`` or ``(None, None)``. tinygrad
is *not* an arborist dependency; this fails closed (op-coverage
for the large MNLI cross-encoders is unverified by design —
`make bench-nli-backends` measures it, this does not assume it).
Prefers the fp32 ``model.onnx`` (int8 QLinear* ops are unlikely
to be covered by the frontend).
"""
onnx_dir = self._onnx_dir()
if not (onnx_dir and onnx_dir.exists()):
return None, None
onnx_path = onnx_dir / "model.onnx"
if not onnx_path.exists():
return None, None
import numpy as np
import onnx as _onnx
import torch
from transformers import AutoTokenizer, AutoConfig
proto = _onnx.load(str(onnx_path))
graph_inputs = [i.name for i in proto.graph.input]
out_names = [o.name for o in proto.graph.output]
logit_out = next((n for n in out_names if "logit" in n.lower()),
out_names[0] if out_names else None)
# Defensive multi-version frontend import. tinygrad has moved
# this module across releases; try the known paths in order.
runner = None
try:
from tinygrad.frontend.onnx import OnnxRunner # newer tinygrad
runner = OnnxRunner(proto)
_kind = "OnnxRunner"
except Exception: # noqa: BLE001
try:
from extra.onnx import get_run_onnx # older tinygrad layout
runner = get_run_onnx(proto)
_kind = "get_run_onnx"
except Exception: # noqa: BLE001
return None, None
tok = AutoTokenizer.from_pretrained(str(onnx_dir))
cfg = AutoConfig.from_pretrained(str(onnx_dir))
class _TinygradAdapter:
"""Mimics a HF seq-classification model's call contract."""
def __init__(self):
self.config = cfg
def __call__(self, **enc):
feed = {}
for k, v in enc.items():
if k not in graph_inputs:
continue
arr = v.detach().cpu().numpy() if hasattr(v, "detach") else np.asarray(v)
feed[k] = arr
out = runner(feed)
if isinstance(out, dict):
val = out.get(logit_out, next(iter(out.values())))
else:
val = out
if hasattr(val, "numpy"): # tinygrad Tensor
val = val.numpy()
logits = torch.as_tensor(np.asarray(val, dtype=np.float32))
import types
return types.SimpleNamespace(logits=logits)
return tok, _TinygradAdapter()
def _nli_batch(self, pairs: list[tuple[str, str]]) -> list[tuple[float, float, float]]:
"""One forward pass over a list of (premise, hypothesis) pairs.
Replaces N batch-1 forwards with ⌈N/batch_size⌉ batched ones —

228
bench/scripts/nli_backend_ab.py Normal file
View file

@ -0,0 +1,228 @@
#!/usr/bin/env python3
"""NLI inference-backend A/B — #000049 §7 #28.
Loads the *same pinned shadow-NLI checkpoint* under each available
inference backend and answers two questions, in priority order:
1. AGREEMENT (the gate): does the backend produce the same
``(p_entail, p_neutral, p_contra)`` as the ``torch`` reference,
within tolerance, over an identical fixed pair set? A faster
engine that *diverges* is not a faster shadow — it is a
*different* shadow signal (and, under the deliberately-unwired
Phase-3 demotion hook, a different ``nli_policy_hash``). So
divergence fails the A/B regardless of speed.
2. LATENCY (secondary): p50/p95 per-pair wall time.
Backends probed (only those that actually load are reported):
• ``torch`` — the reference (`transformers` + torch fwd path)
• ``onnx-int8`` — the §7 #22 `optimum.onnxruntime` export, if present
• ``tinygrad`` — the §7 #28 tinygrad ONNX-frontend backend, if
tinygrad is importable AND its frontend covers
this checkpoint's ops (UNVERIFIED by design —
this script *measures* it, never assumes it)
Deterministic: a fixed in-repo pair set, no LLM, no verifier, no n=3
noise, no 5pp floor — the `recall_at_k` instrument discipline. SHADOW
only: writes nothing but ``bench/results/nli-backend-ab.json``.
This belongs on the GPU producer box (the #000051 "heavy passes never
in arborist's python+sqlite3 core" pattern); tinygrad is not an
arborist dependency. Install tinygrad there, then:
make bench-nli-backends
python3 bench/scripts/nli_backend_ab.py --repeats 5
"""
from __future__ import annotations
import argparse
import json
import os
import statistics
import sys
import time
from pathlib import Path
REPO = Path(__file__).resolve().parents[2]
# Fixed, deterministic (premise, hypothesis) probe set — a spread of
# entailment / contradiction / neutral over factual content. Not a
# quality benchmark; an engine-agreement fixture (same input every run).
PAIRS = [
("The Eiffel Tower is located in Paris, France.", "The Eiffel Tower is in Paris."),
("The Eiffel Tower is located in Paris, France.", "The Eiffel Tower is in Berlin."),
("The Eiffel Tower is located in Paris, France.", "Paris has many museums."),
("Water boils at 100 degrees Celsius at sea level.", "Water boils at 100 C at sea level."),
("Water boils at 100 degrees Celsius at sea level.", "Water boils at 50 degrees Celsius at sea level."),
("Water boils at 100 degrees Celsius at sea level.", "Ice is frozen water."),
("Mount Everest is the highest mountain above sea level.", "Mount Everest is the tallest mountain above sea level."),
("Mount Everest is the highest mountain above sea level.", "K2 is the highest mountain above sea level."),
("Mount Everest is the highest mountain above sea level.", "Many climbers attempt Everest each year."),
("The novel was written by George Orwell in 1949.", "George Orwell wrote the novel in 1949."),
("The novel was written by George Orwell in 1949.", "The novel was written by Aldous Huxley."),
("The novel was written by George Orwell in 1949.", "The novel has been widely translated."),
("Photosynthesis converts sunlight into chemical energy in plants.", "Plants use sunlight to make chemical energy."),
("Photosynthesis converts sunlight into chemical energy in plants.", "Photosynthesis converts chemical energy into sunlight."),
("Photosynthesis converts sunlight into chemical energy in plants.", "Some plants grow in shade."),
("The company reported a net profit of two billion dollars last year.", "The company was profitable last year."),
("The company reported a net profit of two billion dollars last year.", "The company reported a net loss last year."),
("The company reported a net profit of two billion dollars last year.", "The company has offices worldwide."),
("Light travels faster than sound.", "Sound travels faster than light."),
("Light travels faster than sound.", "Light is faster than sound."),
("The treaty was signed by both nations in 1815.", "Both nations signed the treaty in 1815."),
("The treaty was signed by both nations in 1815.", "Neither nation signed the treaty."),
("The species is native to the islands of Southeast Asia.", "The species comes from Southeast Asian islands."),
("The species is native to the islands of Southeast Asia.", "The species is native to northern Europe."),
]
def _fresh_shadow(env_overrides: dict):
"""Construct a ShadowNLI with a scoped os.environ, force-load it,
restore the environ. Returns the (loaded) instance."""
from arborist.qa.nli.shadow import ShadowNLI
saved = {k: os.environ.get(k) for k in env_overrides}
try:
for k, v in env_overrides.items():
if v is None:
os.environ.pop(k, None)
else:
os.environ[k] = v
nli = ShadowNLI()
nli._ensure_loaded()
return nli
finally:
for k, old in saved.items():
if old is None:
os.environ.pop(k, None)
else:
os.environ[k] = old
def _probe(nli, repeats: int):
"""Run the fixed PAIRS through _nli_batch `repeats` times. Returns
(per_pair_probs, per_pair_seconds_list)."""
_ = nli._nli_batch(PAIRS[:2]) # warmup (graph build / kernel JIT)
per_iter_secs = []
probs = None
for _ in range(repeats):
t0 = time.perf_counter()
out = nli._nli_batch(PAIRS)
per_iter_secs.append((time.perf_counter() - t0) / len(PAIRS))
probs = out # deterministic — last iter is representative
return probs, per_iter_secs
def _max_abs_delta(a, b):
m = 0.0
s = 0.0
n = 0
for (ae, an, ac), (be, bn, bc) in zip(a, b):
for x, y in ((ae, be), (an, bn), (ac, bc)):
d = abs(x - y)
m = max(m, d)
s += d
n += 1
return m, (s / n if n else 0.0)
def main(argv=None) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repeats", type=int, default=5, help="timed iterations per backend")
ap.add_argument("--tol", type=float, default=2e-3,
help="max abs probability delta vs torch to PASS agreement")
ap.add_argument("--out", default=str(REPO / "bench" / "results" / "nli-backend-ab.json"))
args = ap.parse_args(argv)
sys.path.insert(0, str(REPO))
# A nonexistent dir forces the torch path even if an export exists.
no_onnx = str(REPO / ".nonexistent-onnx-dir")
plans = [
("torch", {"ARBORIST_NLI_BACKEND": None, "ARBORIST_NLI_ONNX_DIR": no_onnx}),
("onnx", {"ARBORIST_NLI_BACKEND": None, "ARBORIST_NLI_ONNX_DIR": None}),
("tinygrad", {"ARBORIST_NLI_BACKEND": "tinygrad", "ARBORIST_NLI_ONNX_DIR": None}),
]
results = {}
ref_probs = None
for name, env in plans:
print(f"[nli-ab] probing {name} ...", flush=True)
try:
nli = _fresh_shadow(env)
except Exception as e: # noqa: BLE001
results[name] = {"available": False, "reason": f"{type(e).__name__}: {e}"}
print(f"[nli-ab] {name}: load raised — {e}", flush=True)
continue
if not nli.available:
results[name] = {"available": False, "reason": nli._reason}
print(f"[nli-ab] {name}: unavailable — {nli._reason}", flush=True)
continue
# Instrument honesty: ShadowNLI *correctly* degrades to a working
# backend in production, but for an A/B that means "requested X,
# got Y" must be reported as X-unavailable — never relabel Y's
# numbers as X's (false-positive-is-corruption; #000049's own
# discipline). torch=exactly torch; onnx=either onnx variant;
# tinygrad=exactly tinygrad.
ok = {"torch": nli.backend == "torch",
"onnx": nli.backend in ("onnx", "onnx-int8"),
"tinygrad": nli.backend == "tinygrad"}.get(name, True)
if not ok:
reason = f"requested {name} but ShadowNLI loaded {nli.backend} (engine not usable here)"
results[name] = {"available": False, "reason": reason}
print(f"[nli-ab] {name}: unavailable — {reason}", flush=True)
continue
probs, secs = _probe(nli, args.repeats)
p50 = statistics.median(secs)
p95 = sorted(secs)[max(0, int(len(secs) * 0.95) - 1)]
rec = {
"available": True,
"backend": nli.backend,
"device": nli.device,
"p50_ms_per_pair": round(p50 * 1e3, 3),
"p95_ms_per_pair": round(p95 * 1e3, 3),
}
if name == "torch":
ref_probs = probs
rec["agreement_vs_torch"] = "reference"
elif ref_probs is not None and probs is not None:
mx, mean = _max_abs_delta(ref_probs, probs)
rec["max_abs_delta_vs_torch"] = round(mx, 6)
rec["mean_abs_delta_vs_torch"] = round(mean, 6)
rec["agreement_pass"] = bool(mx <= args.tol)
else:
rec["agreement_vs_torch"] = "no torch reference to compare against"
results[name] = rec
print(f"[nli-ab] {name}: backend={nli.backend} device={nli.device} "
f"p50={rec['p50_ms_per_pair']}ms/pair "
f"{('Δmax=' + str(rec.get('max_abs_delta_vs_torch'))) if 'max_abs_delta_vs_torch' in rec else ''}",
flush=True)
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
payload = {
"ticket": "#000049 §7 #28",
"n_pairs": len(PAIRS),
"repeats": args.repeats,
"tolerance": args.tol,
"shadow_only": True,
"results": results,
}
out.write_text(json.dumps(payload, indent=2))
print(f"\n[nli-ab] wrote {out}", flush=True)
# Verdict (informational; SHADOW — never gates a build red).
print("\n=== verdict ===", flush=True)
print("Gate is AGREEMENT first, latency second:", flush=True)
for name in ("onnx", "tinygrad"):
r = results.get(name, {})
if not r.get("available"):
print(f" {name}: not available — {r.get('reason','?')}", flush=True)
continue
if "agreement_pass" in r:
tag = "PASS" if r["agreement_pass"] else "DIVERGES"
print(f" {name}: agreement {tag} (Δmax={r['max_abs_delta_vs_torch']}, "
f"tol={args.tol}) · {r['p50_ms_per_pair']}ms/pair p50", flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())

2
docs/TICKETS.md
View file

File diff suppressed because one or more lines are too long

120
docs/onnx-vendor-capture-immunity.md Normal file
View file

@ -0,0 +1,120 @@
# ONNX vendor-capture immunity — why the engine is not a trust dependency
**What this is:** the rationale for why arborist is *structurally*
immune to ONNX-format and ONNX-Runtime vendor capture — and why that
immunity was a free side-effect of a discipline adopted for honesty,
not for procurement reasons. Not a ticket; a positioning/architecture
reference. Public-domain intellectual capital: the argument is
reproducible and inheritable by any fork or downstream client that
keeps the same cage. The headline: **arborist puts zero neural
inference in its proof path, so whichever vendor owns the inference
engine owns none of arborist's trust boundary.**
---
## 1. The capture vector, stated honestly
ONNX is an open model-exchange format; ONNX Runtime (ORT) is its
dominant engine. Both are governed and overwhelmingly resourced by
Microsoft. The systems-programming critique that motivated this note
(ORT's ~2,500-line, many-flag, many-execution-provider build) is real,
but it is the *small* version of the risk. The large version is the
trust one:
> Any system that places a neural model — its embedder, its NLI head,
> its reranker, its LLM — *inside* the chain of custody for its
> answers inherits that model's runtime, and therefore that runtime's
> vendor. The vendor's roadmap, deprecation policy, license drift,
> build complexity, and execution-provider politics all become load-
> bearing for *your* correctness guarantee.
This is the same defect class as a logged secret or a leaked context:
a dependency that should have stayed at the edge has reached into the
core. The fix is not "pick a better engine." The fix is **not putting
the engine in the core at all.**
## 2. Where models are, and are not, in arborist
| layer | what runs there | model? | vendor-exposed? |
|---|---|---|---|
| Merkle commit (`merkle.py`) | SHA-256, fixed combine conventions | no | no |
| Content store / cache key (`store.py`, `qa/keys.py`) | canonical encodings, 8-dim key | no | no |
| Verifier (`qa/verify.py`) | byte-identical lexical matching (quote / span / entity / paraphrase) | **no** | **no** |
| Audit chain (`audit_events`) | `sha256(prev ‖ canonical(body))` | no | no |
| NLI shadow (`qa/nli/shadow.py`) | entailment contradiction veto | yes | yes — but **sidecar** |
| Relevance shadow (`qa/relevance/shadow.py`) | cross-encoder aboutness | yes | yes — but **sidecar** |
| MT edges (`qa/mt/translator.py`) | opus-mt query/display translation | yes | yes — but **display-only** |
Everything above the rule is the proof path: deterministic, hash-
anchored, model-free, reproducible from `python + sqlite3` alone.
Everything below is a sidecar that is **demote-only, never an
`audit_mode` input, never writes `providence_cache` / `audit_events` /
`run_dag_root`** (the #000049 model-in-proof-path cage; the
verifier-stays-binary and "no soft signal in the hard chain" rules in
CLAUDE.md). A model in arborist can only ever *lower* confidence as an
advisory signal — it can never mint, promote, or sign a result.
## 3. The immunity, precisely
Because no model output is an input to any commitment, proof, cache
key, or audit row:
- **The inference engine is interchangeable.** arborist already swaps
it silently: `qa/nli/shadow.py` prefers an `optimum.onnxruntime`
int8 graph if one exists, picks `CUDAExecutionProvider` /
`CPUExecutionProvider`, and **falls through to the plain
`transformers` + `torch` path on any failure** — with no effect on
any hash. ORT is an *accelerator*, not an authority.
- **A hostile vendor pivot is bounded and cheap.** If Microsoft
deprecated ORT, relicensed it, fragmented the ONNX opset, or its
build complexity became untenable, the blast radius is: re-export
the *pinned* checkpoint to whatever engine survives (torch always
works; tinygrad's ONNX frontend is a candidate — #000049 §7). The
checkpoint is already version-pinned in the NLI manifest; nothing
in the proof path observes the swap. Cost is hours of producer-box
work, not a correctness migration.
- **Cache validity is engine-independent.** A record's
`falsification_state` and `audit_mode` were decided by the
deterministic verifier. Re-running tomorrow on a different engine
cannot change a prior STRICT/HYBRID/UNGROUNDED verdict, because the
engine never produced it. (Contrast: a system whose answer is
graded by an LLM must re-grade — and silently re-trust — on every
model or runtime change.)
The one place a model could ever touch authority is the deliberately-
unwired Phase-3 demotion hook in #000049. Even there the design
forces the exposure to be *named*: if NLI is ever allowed to demote
`audit_mode`, `nli_policy_hash` (model + k + agg + guard + θ) folds
into `governance_policy_hash`, so the model's identity becomes a
first-class, cache-partitioning, audited policy input — capture made
visible and versioned, never silent.
## 4. This was free — and that is the point
We did not design the cage to dodge Microsoft. We designed it because
a model in the proof path is a honesty defect: a false-positive STRICT
is corruption, and a model that can promote will eventually hallucinate
a promotion. The deletion-first guardrails (delete the part: the model
does not get to be in the hard chain) were adopted for *truth*.
Vendor-capture immunity fell out for free. That is the recurring shape
of the right architectural call: a discipline taken for one good
reason pays a second, unbid dividend. A system built to never lie about
grounding turns out to also be a system no inference-engine vendor can
hold hostage.
## 5. Why this is published intellectual capital
Per the Prime Mission, this argument is a gift to the ecosystem, not a
moat. Any RAG / retrieval / agent system can adopt the same split:
deterministic verifier in the proof path, models as demote-only
sidecars on a producer box, engine identity versioned into a policy
hash only if it is ever allowed to touch authority. A fork that keeps
the cage inherits the immunity verbatim — no license, no dependency,
no permission required. The cage *is* the gift.
See also: `docs/lexical-first-rationale.md` (the cost-asymmetry
sibling — why the cheap deterministic path is the default),
`docs/soft-hash-channel-analysis.md` (soft-vs-hard channel threat
model), and ticket #000049 §7 (the NLI cage, shadow path, and the
tinygrad/ORT engine-swap bench).

87
docs/tickets/ticket-000049-attribution-aware-grounding-check.md
View file

@ -1368,3 +1368,90 @@ buried — those are *different* problems, and only the second one is
the production problem. The 4090 made this measurable in ~4 min
(7 models × full grid × 833 records); without it we would have
shipped a 24%-recall config under a "1.000" label.
---
**§7 #28 (2026-05-19) — tinygrad as a third NLI inference backend;
deterministic engine-agreement A/B (instrument built in-repo; numbers
pending a producer-box run).** Context: the Sherief/tiny-corp thread
(ONNX Runtime's ~2,500-line many-EP build vs tinygrad's minimalist
ONNX frontend that reportedly passes more of the official ONNX test
suite than ORT itself) raised a fox question — can the shadow-NLI
checkpoint run under tinygrad's ONNX frontend instead of / alongside
`optimum.onnxruntime`? This is a legitimate §7 sub-item, not a new
ticket: it is the *same* speedup/backend axis as §7 #22 (the
ONNX-int8 export), one more interchangeable accelerator below the
cage. It also makes the abstract immunity argument concrete — see the
new `docs/onnx-vendor-capture-immunity.md` (the engine is never a
trust dependency; this bench is the demonstration, not a migration).
**The gate is agreement, not speed.** The primary acceptance metric
is **per-pair logit/probability agreement with the `transformers`+
`torch` reference within tolerance** over the same pinned checkpoint
and the same input pairs — *then* p50/p95 latency as the secondary
axis. Rationale, in the exact shape of this ticket's standing lesson:
a faster engine that produces *different* `(p_entail, p_neutral,
p_contra)` is not a faster shadow — it is a *different* shadow signal.
Under the deliberately-unwired Phase-3 demotion hook that would mean a
different `nli_policy_hash` folding into `governance_policy_hash` (§7
#2) and a silently different demotion surface. So: engine-swap is only
admissible if it is *numerically* the same model, measured, not
assumed. "Honest verdicts beat optimistic ones" applies to the
instrument too.
**Honest risk, stated up front (don't trust the contrived assumption
until confirmed on the real shape).** The current
`recommended_operating_point` is `bart-large-mnli` (§7 #27); the
synthetic alternate is `deberta-large-mnli`. tinygrad's ONNX frontend
op-coverage for *these large MNLI cross-encoders* (Erf/GELU, fused
attention, `GatherElements`, dynamic shapes) is **unverified** — it
is an empirical question with a real chance of "frontend rejects the
graph" or "loads but diverges." That outcome is itself a valid,
publishable result (it sharpens the immunity doc: torch is the
always-works floor, ORT the proven accelerator, tinygrad a *candidate*
whose coverage is checkpoint-specific). No assumption of success is
baked into the plan.
**What landed in this commit (instrument before experiment):**
(1) a third backend branch in `arborist/qa/nli/shadow.py`,
`backend == "tinygrad"`, parallel to the onnx/torch branches —
optional `import`, graceful `available=False` on any failure,
SHADOW-only, never an `audit_mode` input, GPU-producer-box only (the
#000051 "heavy passes never in arborist's python+sqlite3 core"
pattern; arborist core stays python3.12+venv+sqlite3); selectable via
`ARBORIST_NLI_BACKEND`. (2) A deterministic A/B instrument
`bench/scripts/nli_backend_ab.py` + `make bench-nli-backends`: loads
the pinned checkpoint under each *available* backend, runs an
identical fixed pair set, reports max/mean abs probability delta vs
the torch reference and p50/p95 latency per backend — no LLM, no
verifier, no n=3 noise, no 5pp floor (the `recall_at_k` deterministic-
instrument discipline). **Pending:** the numbers. tinygrad is not in
arborist's deps and the run belongs on the GPU producer box, not the
core dev box — so this is fox/producer-box-decides to (a) install
tinygrad there and (b) `make bench-nli-backends`. SHADOW throughout;
manifest active defaults and the proof path are byte-unchanged by
construction (the immunity property, exercised).
*First instrument run (2026-05-19, CPU dev box, tinygrad absent — a
smoke of the instrument, not the experiment):* the A/B already
produced a real finding before tinygrad is even in scope. The
**currently-deployed §7 #22 int8 ONNX export diverges from the torch
reference by Δmax ≈ 0.42 per-class probability** on the fixed 24-pair
probe (24 pairs × 5 repeats; mean abs delta smaller but the max is
large). This quantifies, with a deterministic instrument, the prior
hand-wave "int8-ONNX costs ≈1 catch": int8 is a *lossy-by-
construction different numerical model*, and the A/B honestly reports
it as `DIVERGES` at a tight (2e-3) faithfulness tolerance. This is
**not a defect** — it is the immunity property made measurable: even
the shipped accelerator is a numerical surrogate, which is safe only
because it is a shadow that never reaches the proof path
(`docs/onnx-vendor-capture-immunity.md`). It does sharpen the eventual
Phase-3 question: if NLI is ever wired to demote, the engine+
quantization must be pinned into `nli_policy_hash` (an int8 export and
its fp32 source are *different policies*, and this instrument now
proves it numerically). Instrument-honesty caveat encoded: when
`ARBORIST_NLI_BACKEND=tinygrad` is requested but the frontend cannot
load the checkpoint, ShadowNLI correctly degrades to the proven path,
but the A/B reports **tinygrad-unavailable**, never relabels the
fallback's numbers as tinygrad's. Pending unchanged: real tinygrad
numbers need the producer box.