# Ticket #000049 — Attribution-aware grounding check (the recombination boundary)

**Status:** open · boundary accepted · production no-go · shadow-path
approved (de novo review 2026-05-13 — see §7) · Phase-1 + Phase-2-scaffold
landed 2026-05-12 (§7 #18/#19 — candidate bench in `~/git/arborist-nli-bench/`
[MiniLM-82M cost-pick, §7 #5 entailment guard confirmed load-bearing];
`arborist/qa/nli/` shadow module + `[nli]` extra + `make bench-nli-shadow`
in-repo, SHADOW ONLY — never an `audit_mode` input; first sweep:
28/28 recombination demoted, 0/26 FP on legit summaries, never fires on
already-STRICT records). First bench-qa-traffic shadow sweep done (§7 #20, `ARBORIST_NLI_SHADOW=1 make
bench-qa-smoke` → 15 cells): the *naive* "NLI on every context clause"
version has a **~30% would-demote rate on STRICT answers** — a haystack /
multiple-comparisons artifact (real contexts segment into 100–336 clauses;
`max`-over-all-of-them almost always finds a tangential clause the model
reads as contradicting). Lesson: the §7 #5 "candidate source clauses" +
recombination-risk gating is **load-bearing, not optional** — Phase 3 (and
the next Phase-2 step) must restrict the NLI call to the clauses the
lexical verifier actually matched and/or a deterministic recombination-risk
trigger; do NOT enable runtime NLI demotion on the current scaffold.
Candidate-clause restriction (§7 #21) + speedup (§7 #22 — batched
forwards, device auto-detect, ONNX-int8 export, ~4× on CPU / seconds
on a 4090) + a `{model × k × agg × guard × θc × θe}` grid sweep
(§7 #23–#26, `bench/scripts/nli_shadow_grid.py`, run against bench-qa
STRICT samples of growing size: n=1=89 → n=3=275 → n=5=444 → pooled
n=1+3+5 = 808) + a mining pass over the 808 real STRICT answers for
natural recombinations + a recall-side realism check (§7 #27 — 25
hand-built recombination claims against real bench-qa 30 KB haystacks)
landed; the verdict moved **six times**: §7 #22 "fails" = default-
config artifact; §7 #23 "deberta-base passes" = small-FP-sample
artifact; §7 #24/#25/#26 = three more samples relocating the
configuration target (large model + `margin`@0.995 settled at pooled
808 STRICT); **§7 #27 shows even §7 #26's "settled" was a synthetic-
*recall* artifact** — the same deberta-large-mnli/`margin`@0.995 config
that gets 28/28 on the 28-case synthetic set gets **6/25 = 0.24** on
the 25-case real-haystack set. **Honest current state: the lexical-
candidate NLI veto *partially* closes the boundary** — the new fp=0
leader on real haystacks is `bart-large-mnli / k=12 / agg=max / θc=0.999`
→ **12/25 = 0.48 recall · 0/808 STRICT FP**. *Strictly better than
the current 0%* but not the §7 #26 "full closure." The remaining gap
is **architectural, not threshold-tuning**: the candidate-clause
selector (top-k by content-token overlap) misses the contradicting
clause when it shares few subject-area tokens with the answer (e.g.
the Mount Kenya clause only shares "Kenya" with a "Kilimanjaro is in
central Kenya" claim against the real Kilimanjaro article — gets
ranked low, NLI never sees it, can't fire). The Phase-3 lever that
lifts the ceiling is **a semantic candidate selector** (vec-driven,
sibling of #000050/#000051's hybrid retrieval — embedding similarity
finds *meaning-similar* clauses, not just *token-similar*); raising
threshold doesn't help. Other findings: `bart-large-mnli` is the new
real-context winner (k=12/max/θc=0.999 stays FP-safe because its
score distribution is more spread) — both the §7 #18 "MiniLM
cost-pick" *and* the §7 #26 "deberta-large + margin" are overturned;
the FP-side analysis (`margin@0.995` discriminates spurious from real
on the *FP* side) is unaffected; mining the 808 STRICT answers for
*natural* recombinations found zero (the boundary is contrived-in-
practice on factual-Wikipedia QA, but the synthetic-vs-real-haystack
gap on the recall side is what matters for any *future* recombination
that emerges in production). Caveats: still SHADOW; the 25 real-
context fixtures are hand-built (representative but not exhaustive);
the pareto above fp=0 buys substantial recall for small FP cost
(bart-large at fp=0.057 catches 0.84; deberta-large at *single* STRICT
FP buys catch 0.56) — a permissive operating point is on the menu if
fox+dav1d sign off. Runtime promotion is fox+dav1d-decides (then
`nli_policy_hash` = model+k+agg+guard+θc folds into
`governance_policy_hash` per §7 #2). Manifest active defaults stay
k=6/θc=0.5; `recommended_operating_point` updated to
`bart-large-mnli / k=12 / max / θc=0.999` (the real-haystack fp=0
leader); the deberta-large/margin config kept as a synthetic-eval
reference. Standing lesson, sharpened *six* times (codified in
CLAUDE.md bench-maxing): clean synthetic eval doesn't predict bench-qa
**precision *or* recall** — neither contrived dataset axis is
load-bearing; only the real pipeline shape is (§7 #18→#20→#22→#23→
#24→#25→#26→**#27**). Production verifier unchanged; `falsification-hard`
stays 10/12.
**Opened:** 2026-05-12
**Scope:** Decide whether — and if so how — to add a verifier check
that catches a *recombination*: a claim whose content tokens are all
present in the source but **paired with different subjects/predicates
than the answer asserts**, so the answer is false even though it
"covers" the source lexically. Doc-only proposal: the problem, the
design options (dependency-attribution heuristic vs. mini-NLI model),
the discipline trade-off, a recommendation. No code in this ticket.
**Audience:** fox + maintainers of `arborist/qa/verify.py` + #000048
follow-up readers + anyone weighing "do we put an ML model in the
verifier path?".
**Hard constraint:** whatever lands here is bench-gated (`make
bench-qa` n=3, before/after — STRICT-rate must not regress on legit
answers). It MUST NOT break the existing layered verifier
(quote→span→entity→paraphrase, first-to-classify); a contradiction
signal *demotes*, never promotes. **The "no LLM-as-judge in the
verifier" rule still holds** — a small purpose-built NLI/entailment
*model* is a discipline question this ticket has to settle explicitly
(it is not the same as "ask the answering LLM to grade itself", but
it is an ML model in the proof-adjacent path); a dependency-parse
heuristic sidesteps that but is brittle. **Governance-hash rule
(corrected by the 2026-05-13 review — §7 #2):** *while this stays
doc-only / Option 2.3 / shadow-only* there is no schema /
governance-hash / canonicalization-version change; but **if an NLI
output ever influences `audit_mode`, then `nli_model_version` + the
checkpoint/tokenizer/threshold/pair-serialization policy MUST fold
into `governance_policy_hash`** — otherwise two deployments could
return different `audit_mode` for the same (answer, source) pair while
sharing cache identity, which breaks the Merkle/SQD reproducibility
discipline.

---

## 1. Problem statement

`#000048` (closed 2026-05-12) closed 12 of the 16 residual
over-grounds/mis-segments via *lexical* surface improvements: the
numeric-agreement gate (`#000046` Phase 3), the entity
salient-token-disagreement gate (`#000048` step 2.1 — an *absence*
signal: a digit / capitalized token the source doesn't contain), and
the `parse_pointer_claims` clause segmenter (`#000048` step 2.4 — a
*structure-of-the-model's-own-output* signal). What it could **not**
close — and `#000048` §5 "What we learned" spells out why — is the
**recombination** class:

- `5f-fal-hard-003`: "Mercury is the largest planet in the Solar
  System." against source "Jupiter is the largest planet in the
  Solar System; **Mercury is the smallest**." — the answer reuses
  the source's "largest planet in the Solar System" (which is the
  source's predicate for *Jupiter*) and the source's "Mercury"
  (whose predicate is *smallest*), pairing them into a falsehood.
  Token-coverage = 1.0 (every content token is *somewhere* in the
  source); the entity gate doesn't fire (no token is *absent*);
  `verify_quotes` returns STRICT_PARAPHRASE or (post-#000048 step
  2.1, since the paraphrase path now falls through) HYBRID_ENTITY.
- `5f-fal-hard-005`: "Albert Einstein developed the theory of
  evolution." against "Albert Einstein developed the theory of
  relativity; **Charles Darwin developed evolution**." — same
  shape: the source's "Albert Einstein developed the theory of"
  paired with the source's "evolution" (whose subject is *Darwin*).

`#000048` §5 established that this is **not a lexical question**:
a recombination and a legitimate cross-sentence summary ("Batman,
who is the alias of Bruce Wayne, lives in Gotham City." over "Batman
is the alias of Bruce Wayne. Batman lives in Gotham City.") have the
*same* lexical profile (tokens scattered across source clauses), and
the recombination's best-single-clause coverage (4/5) sits *above*
the legit summary's (4/6) — no token-coverage / clause-containment /
bigram threshold separates them in the safe direction. The
discriminating fact is **attribution**: in the source, is the
answer's token X attached to the same subject/predicate the answer
attaches it to? That requires either parsing the source's
dependency/predicate structure or an entailment judgment — i.e., a
capability the lexical verifier deliberately doesn't have. This
ticket is the home for "do we want that capability, and at what
cost?".

(Out-of-scope-of-#000048 was the conclusion; here it's the subject.)

---

## 2. Design options

### 2.1 — dependency-attribution heuristic (no model)

A small hand-rolled check, run on the paraphrase path (and the entity
path's weakest slot) when a span token-covers the source but spans
multiple source clauses: for each *content token* in the answer span,
find its occurrence(s) in the source; identify the nearest preceding
*subject-like* token (a capitalized noun phrase) and the nearest
*predicate-like* head; if the answer pairs token X with subject S but
in the source X's nearest subject is S′ ≠ S, flag a contradiction →
the span is not grounded. No new dependency: just regex/heuristic
parsing of "S … (verb) … X" patterns. **Pro:** stays within the
"no-model" verifier discipline; deterministic. **Con:** brittle —
"nearest capitalized noun before X" is a crude proxy for "X's
syntactic subject"; passive voice, appositives, coordination, and
list structure all break it; tuning it without over-firing on legit
paraphrases (which reorder freely *within* a clause) is the same
threshold-can't-separate problem one rung up the abstraction ladder.
Likely catches the 2 contrived fixtures with hand-tuning but provides
little assurance on the general class.

### 2.2 — small purpose-built NLI / entailment model (a model dependency)

Run a tiny sentence-pair entailment model (a few-hundred-MB DeBERTa-
or RoBERTa-NLI checkpoint, CPU-fast) on `(source_clause, answer_span)`
pairs the lexical path is unsure about; if the best-matching source
clause *contradicts* (not just "doesn't entail") the answer span,
demote. **Pro:** this is the *right* capability for the recombination
class — "Mercury is the largest" vs "Mercury is the smallest" is a
textbook contradiction; "Batman lives in Gotham" vs "Batman is the
alias of Bruce Wayne" is neutral/entailed, not contradicted. **Con:**
(a) **a model in the verifier path** — even though it's *not* "the
answering LLM grading itself" (it's a fixed, purpose-built classifier
with no shared parameters), it is an ML model whose output influences
`audit_mode`, which the "no LLM-as-judge" rule was written to keep
out; settling whether a small fixed NLI model is allowed is the
*first* decision this ticket needs. (b) a new optional dependency
(`transformers` / `onnxruntime` + a checkpoint) — keep it behind a
`[nli]` extra so a fresh checkout still needs only python+venv+sqlite3,
gate it behind `entity_policy`-style policy so it's off-by-default
until benched, and **don't put the weights in the repo** — see §2.4
for the checkpoint provenance & distribution (a tiny manifest in the
repo, the weights fetched on demand into `~/.arborist/models/` from
an origin URL or, once mesh-blob-sync exists, a peer; hash-pinned).
(c) non-determinism risk — pin the checkpoint + run on CPU with a
fixed dtype, and fold `nli_model_version` into `governance_policy_hash`
(§2.4 #4) so a swap stales the cache; entailment classifiers are
deterministic given a pinned checkpoint + CPU + fixed dtype, but it's
a thing to verify and pin. (d) latency — only run it on the small
subset the lexical path can't resolve.

### 2.3 — do nothing; the 2 fixtures stand as a marker

`falsification-hard` at 10/12 is already a real below-ceiling
baseline; the 2 recombination fixtures *are* the documentation of
where the lexical verifier stops (#000048 §5). Closing them buys
2/12 on a contrived pack at the cost of (heuristic brittleness, or a
model dependency). If no bench-qa evidence shows recombination
over-grounds happening at scale on actual traffic, this is the
honest choice. (No bench evidence of that exists today — the #000048
step-2.1/2.4 per-row diffs found 0 gate-attributable and 0
segmenter-attributable *legit* harms; they didn't surface
recombination-over-grounds in real answers either, because the
paraphrase path is a last resort that rarely fires on real Wikipedia
QA.)

### 2.4 — where would the NLI weights live? (Option 2.2 detail — provenance & distribution)

If Option 2.2 is ever chosen, the *checkpoint* has to come from
somewhere and reach every deployment that enables the policy. Three
shapes, two of them dead ends:

- **Trained just-in-time, per deployment — ❌.** Training a
  DeBERTa-/RoBERTa-NLI-class classifier is GPU-hours over MNLI / SNLI
  / ANLI; it is *not* a per-deployment step. Worse, two training runs
  give different weights → different `audit_mode` on the same input →
  this breaks the verifier's reproducibility invariant
  (`audit_mode` must be a deterministic function of the inputs +
  policy). "Training" is a *one-time act by someone*; what gets
  distributed is the resulting fixed blob, not the recipe. So this
  isn't really an option — it collapses into one of the other two
  for "where does the blob come from", with the worst reproducibility
  story.
- **Embedded in the git repo as data — ❌.** A few-hundred-MB binary
  in `.git` blows the standing invariant that a fresh `git clone`
  needs only `python3.12 + venv + sqlite3` (now it's "+ ~400 MB of
  model weights for an off-by-default feature most deployments never
  enable"). Git LFS would add an LFS dependency and still bloats the
  clone. Wrong fit — the repo is source + tiny fixtures + docs, not a
  model registry. (Contrast `bench/fixtures/` JSONL packs: a few KB
  each, the *data* IS the test; a model checkpoint is neither small
  nor a test.)
- **Obtained once, content-addressed, fetched on demand, mesh-
  distributable — ✓.** The right shape, and the one consistent with
  arborist's existing patterns:
  1. **Source.** Either a published off-the-shelf NLI checkpoint (a
     known DeBERTa-/RoBERTa-NLI fine-tune with a redistribution-OK
     license — most are MIT/Apache; record it) **or** a one-time
     fox/blackops-trained-and-published one if arborist wants its
     own. Either way the result is a single fixed blob with a
     sha256.
  2. **Manifest in the repo (tiny).** `arborist/qa/nli/manifest.json`
     (say) records `{checkpoint_sha256, source_url, license,
     nli_model_version, onnx: true|false}`. The repo carries the
     *pointer + hash + license*, not the weights — exactly the
     pattern of `bench/fixtures/textbooks/manifest-v1.jsonl` (textbook
     URLs + license tokens, not the textbook text) and the `[vec]`
     extra (#000039 — `fastembed`'s `bge-small` checkpoint fetched
     on first use, never committed). Precedent: "an ML model behind a
     `[…]` extra, fetched not committed" is already how the vec
     backend ships.
  3. **Fetched on first use into `~/.arborist/models/nli/<hash>/`.**
     Operator-state lives under `~/.arborist/` (like `shards/`,
     `crawl/`, `textbooks.db`), not the repo. A `make fetch-nli` (or
     auto-fetch the first time a `--enable-nli` query runs) downloads
     it and **verifies the sha256 against the manifest, fail-closed
     on mismatch** — the "artifact over instruction" / "md5sum
     downloads" discipline applies to the checkpoint exactly as it
     does to a Wikipedia dump. The `[nli]` extra pulls the runtime
     (`onnxruntime` if the manifest's checkpoint is ONNX-exported —
     no torch, smaller; else `transformers` + a CPU torch).
  4. **`nli_model_version` folds into `governance_policy_hash`.** A
     checkpoint swap stales prior `providence_cache` records on
     lookup, exactly like bumping `chunking_version` /
     `canonicalization_version`. So the verifier stays reproducible:
     `audit_mode` is a deterministic function of (the answer, the
     source, the policy, **and the pinned checkpoint hash**) — run
     it twice with the same hash on the same shard → identical
     verdict. The audit chain can record an `nli_checkpoint_loaded`
     event carrying the hash so provenance is on the chain.
  5. **Mesh-distributable as a content-addressed blob (the gossip
     path).** Once arborist's mesh (`arborist/mesh/`, `docs/mesh.md`
     — today: audit-event / group-key sync) grows blob-sync, a
     deployment with peers pulls the checkpoint *from a peer*
     (AXFR-style — same shape as ClouDNS slaves pulling a zone from
     the PowerDNS master, or a fresh shard rehydrating from a
     `snapshot.db`) instead of the origin URL. Content-addressing
     makes peer-pull and origin-pull interchangeable: verify the
     sha256 either way. The mesh is an **optimization** (faster, no
     external dependency at runtime, works on an air-gapped
     permacomputer cluster once one node has it), **not the canonical
     source** — the manifest's hash is. Cold start (a fresh
     deployment with no peers, no local copy) falls back to the
     origin fetch in step 3.

**Bottom line:** *not* JIT-trained (non-deterministic, GPU-hours, not
a per-deployment step), *not* repo-embedded weights (blows the lean-
checkout invariant); **a one-time-obtained, content-addressed, hash-
pinned checkpoint — the repo carries a tiny manifest, the weights
live under `~/.arborist/models/`, fetched on demand (origin URL or,
once mesh-blob-sync exists, a peer), the hash folds into
`governance_policy_hash`.** All of which presupposes the §2.2/§5
discipline call came back `yes`; if `no`, none of this is built.

---

## 3. Recommendation

**Updated by the 2026-05-13 de novo review (§7):** production stays
**Option 2.3** today; **additionally, an Option-2.2 shadow path is
approved** — a fixed NLI model may be evaluated as a *demotion-only
contradiction veto* in bench/shadow mode, but it must not affect
`audit_mode` until it passes the deterministic bench gate (§7 #6) and
its checkpoint/tokenizer/threshold/pair-serialization policy folds
into `governance_policy_hash`. NLI never *promotes* an answer to
grounded; it can only demote a *weak* paraphrase/entity outcome when
recombination risk is detected and no stronger deterministic warrant
exists. Heuristic demotion (Option 2.1) is **not** recommended even
then — only as a deterministic *recombination-risk trigger* that
decides when to call shadow NLI, never as the contradiction decider.

Original rationale (still holds for the production no-go):

1. The lexical verifier is honest about its boundary (the 2 fixtures
   document it); there's no observed bench-qa-traffic harm.
2. Option 2.1 (heuristic) is the same threshold-can't-separate
   problem dressed up — low assurance, real over-fire risk on the
   QA bench.
3. Option 2.2 (NLI model) is the right *capability* but the wrong
   *cost* until there's evidence it's needed: it forces the
   "is a fixed NLI model an LLM-judge?" decision, adds a dependency,
   and adds a non-determinism surface to pin — all for a contrived
   2/12.
4. The trigger to revisit: a `make bench-qa` per-row diff (or a
   field report) showing a *legitimate-looking but false* answer
   getting STRICT/HYBRID via the paraphrase/entity path because it
   recombined source tokens. At that point this ticket reopens with
   real data, and Option 2.2 (behind a `[nli]` extra, policy-gated,
   off by default, bench-gated) becomes the path — *if* fox rules a
   fixed purpose-built NLI model is acceptable in the proof-adjacent
   path (it's a genuine discipline call; the safe default is "no",
   in which case 2.3 stands permanently and the 2 fixtures are just
   permanent boundary markers).

---

## 4. Out of scope

- **The `falsification-hard` / `formulate-hard` packs themselves** —
  #000046 built them; this ticket would change the verifier surface,
  not the fixtures.
- **`fork_score` / the aggregator / the bench Δ-rate mechanics** —
  unchanged; this only affects which `audit_mode` `verify_quotes`
  returns on a recombination span, hence `falsification-hard`'s rate.
- **Replacing the lexical verifier wholesale with an embedding/LLM
  one** — a far bigger architectural move; this ticket is narrowly
  about an *added* contradiction check, lexical path untouched as the
  first-line.
- **A harder fixture tier** — #000046's territory if the current
  packs go saturated again.

---

## 5. Status

**Open · boundary accepted · production no-go · shadow-path approved**
(de novo review 2026-05-13, §7). The diagnosis (recombination is an
*attribution* failure, not lexical) is confirmed. Production verifier
is unchanged; the 2 recombination fixtures (`falsification-hard`
stays 10/12) are accepted as honest boundary markers. The **discipline
question** in §2.2's con (a) — may a small, fixed, purpose-built
NLI/entailment model influence `audit_mode`? — is answered **"not by
default; yes only as an opt-in, hash-pinned, governance-hashed,
demotion-only contradiction veto after shadow-mode evidence"** (§7
#19). Next concrete steps: the §7 implementation roadmap — Phase 0
(this amendment), Phase 1 (shadow design: manifest format,
fetch/verify command, `nli_pair@v1` canonicalization, recombination-
risk trigger), Phase 2 (optional bench-only shadow implementation,
`[nli]` extra, `make fetch-nli`), Phase 3 (demotion-only runtime,
gated), Phase 4 (mesh blob sync). Nothing in Phases 2–4 lands without
the §7 #6 bench gate passing.

---

## 6. References

- `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md`
  (closed) — §5 "What we learned" is the argument that recombination
  isn't lexical; §2.3 there sketched this; this ticket is its home.
- `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md` (closed) —
  built `falsification-hard` (`5f-fal-hard-003` Mercury, `-005`
  Einstein are the 2 residual recombination fixtures this ticket
  would close).
- `bench/fixtures/5f/falsification-hard-v1.jsonl` — the 2 fixtures.
- **Provenance/distribution precedents for §2.4:** `bench/fixtures/textbooks/manifest-v1.jsonl`
  (a repo manifest of *pointers + license tokens*, not the texts);
  `arborist/search/vec.py` + the `[vec]` extra (#000039 — an ML model,
  `fastembed`'s `bge-small` checkpoint, fetched on first use, never
  committed); `~/.arborist/{shards,crawl}` + `textbooks.db`
  (operator-state lives under `~/.arborist/`, not the repo);
  `docs/mesh.md` + `arborist/mesh/` (the mesh — today audit-event /
  group-key sync; a content-addressed blob-sync extension is what
  the §2.4 "gossip/peer-pull" path needs); `arborist/snapshot.py`
  (snapshot rehydrate — the same "pull a content-addressed artifact"
  shape, peer or origin).
- `arborist/qa/verify.py` — the layered verifier; a contradiction
  check would attach to the paraphrase path + the entity path's
  weakest slot (the same places #000046 / #000048 step 2.1's gates
  attach).
- `docs/bench-maxing.md` — the bench discipline (n=3, 5-pp floor,
  honest verdicts over optimistic) every step here honors.
- `docs/seven-point-program.md` D1 ("no LLM in the verifier") — the
  rule §2.2's con (a) tests against.

---

## 7. De novo review — incorporated (2026-05-13)

An independent de novo review (`~/Downloads/response_ticket-000049-*.txt`,
2026-05-13 Asia/Kuala_Lumpur) confirmed the diagnosis and the §2.4
weights answer, and added the corrections below. They are **additive
and no-regression** — production verifier unchanged.

**1. Core diagnosis confirmed.** The 2 residuals are *attribution
recombination* failures (every token in the source, but the answer
attaches a subject to the wrong predicate/object). Token coverage,
clause containment, and bigram thresholds cannot separate a
recombination from a legitimate cross-sentence summary in the safe
direction — the discriminating fact is attribution, which needs a
dependency structure or an entailment judgment. Not a failure of
#000046/#000048; the next boundary after lexical hardening.

**2. Governance-hash correction (most important).** The §-header
"no governance-hash change" constraint is valid *only while this is
doc-only / Option 2.3 / shadow-only*. If a fixed NLI checkpoint can
demote a claim, then `audit_mode` becomes a function of {answer span,
source span/clause, verifier policy, NLI checkpoint hash, NLI runtime
policy, NLI tokenizer policy, NLI threshold policy} — so
`nli_model_version` + that policy bundle (or an explicit
`nli_policy_hash`) **must** fold into `governance_policy_hash`.
Otherwise two deployments give different `audit_mode` for the same
(answer, source) pair while sharing cache identity → violates the
Merkle/SQD reproducibility discipline. Corrected three-state rule:
doc-only/2.3 → no hash change; shadow mode with no `audit_mode`
effect → no cache-identity change, but record shadow metadata
separately; NLI affecting `audit_mode` → `governance_policy_hash`
MUST include NLI model + runtime + tokenizer + threshold + pair
policy.

**3. Discipline call.** A fixed NLI classifier is *not* "the
answering LLM grading itself" (no shared parameters, purpose-built,
content-addressed) — so it isn't the exact failure mode D1 names —
but it is still an ML model in the proof-adjacent verifier path, so
it violates the *spirit* of a deterministic symbolic/lexical verifier
unless tightly bounded. Permit it only under a new category
**`MODEL_ASSISTED_DEMOTION`**, never `MODEL_ASSISTED_PROMOTION`: NLI
may say "this weakly grounded claim appears contradicted; demote it",
never "this unsupported claim is true; promote it to grounded". No
model manufactures proof; at most a pinned model adds a contradiction
veto to a claim the lexical verifier was already uncertain about.

**4. Placement.** NLI runs *only* after a lexical path reaches a weak
grounding state (paraphrase path; weak entity path; high lexical
coverage but multi-clause source; source has multiple candidate
subject/predicate pairings; recombination-risk heuristic fires).
Never on: direct quote match, exact span match, strong
pointer-warranted claim, or any case a deterministic verifier already
proves. Layered-first-to-classify invariant preserved; contradiction
demotes, never promotes.

**5. Algorithm (clause-level, not "NLI on whole source").** Coarse
`NLI(source_document, answer_claim)` is wrong — a source can hold
both a contradictory background statement and a later supporting one.
Use:
1. segment source into clauses (existing clause/pointer segmenter);
2. extract answer claim span;
3. detect recombination risk (high content-token coverage; span
   crosses multiple source clauses; answer subject/object tokens
   occur in *different* source clauses; weak verifier path only);
4. for each candidate source clause, run `NLI(source_clause,
   answer_claim)`;
5. compute `max_entailment_score`, `max_contradiction_score` over
   clauses;
6. demote only if `max_contradiction ≥ θ_c` AND `max_entailment <
   θ_e` AND current label is weak enough to be vetoed AND no stronger
   deterministic warrant exists elsewhere;
7. else leave the lexical/verifier result unchanged.

Formal: `Demote(claim) = WeakGrounding(claim) ∧
RecombinationRisk(claim, source) ∧ MaxContradiction(source_clauses,
claim) ≥ θ_c ∧ MaxEntailment(source_clauses, claim) < θ_e ∧ ¬
StrongDeterministicSupport(claim)`.

**6. Audit label.** Don't silently convert a contradiction into a
generic failure — emit an explicit reason:
`{audit_mode: "UNSUPPORTED"/"UNGROUNDED", demotion_reason:
"NLI_CONTRADICTION_VETO", nli_model_version, nli_checkpoint_sha256,
source_clause_id, answer_claim_id, max_contradiction, max_entailment}`.
If the schema can't carry this yet → run shadow mode only, don't
implement runtime demotion.

**7. NLI manifest fields (extends §2.4 #2).** `arborist/qa/nli/manifest.json`
should carry: `nli_model_version`, `checkpoint_sha256`, `source_url`,
`license`, `license_verified_at`, `model_family`, `onnx` (bool),
`runtime` (`onnxruntime-cpu`), `runtime_version_range`,
`tokenizer_sha256`, `label_map` (`{"0":"entailment","1":"neutral",
"2":"contradiction"}` — pin the actual ordering of the chosen
checkpoint), `max_length`, `truncation_policy`
(`source_clause_then_claim@v1`), `pair_serialization` (`nli_pair@v1`),
`thresholds` (`{contradiction_veto: 0.90, entailment_block_veto:
0.50}` — illustrative; calibrate). The manifest hash (or an explicit
`nli_policy_hash`) folds into `governance_policy_hash` once
`audit_mode` is affected.

**8. `nli_pair@v1` canonicalization (SQD interaction).** The NLI
input pair must be canonicalized — whitespace / clause-boundary
jitter changes tokenization. `nli_pair@v1` = canonical serialization
of {`source_clause_text`, `answer_claim_text`, `source_clause_id`,
`answer_claim_id`, `truncation_policy`, `tokenizer_policy`};
`nli_pair_hash = H(canonical_json(nli_pair@v1))`, recorded in the
run-DAG / audit sidecar.

**9. Mesh / #000051 relation.** NLI-checkpoint distribution can
reuse the vecpack blob-sync *pattern* but needs stronger handling: a
vecpack is a soft retrieval sidecar (a wrong blob can't directly
touch `audit_mode`); an NLI checkpoint is a proof-adjacent verifier
artifact (a wrong blob *can*, if NLI is enabled). So: structural
sanity + trust tier is enough for vecpacks; for NLI, **exact sha256
match against the manifest is mandatory, fail-closed on mismatch,
and `governance_policy_hash` includes the checkpoint identity**. Peer
gossip distributes the blob only as transport — origin URL and peer
blob are equivalent iff the sha256 matches; mesh sender trust is not
sufficient, the manifest hash is canonical.

**10. Reproducibility KATs.** If NLI ever affects `audit_mode`: fixed
checkpoint hash, fixed tokenizer hash, fixed ONNX/runtime backend,
CPU inference, fixed dtype, fixed pair serialization, fixed
truncation policy, fixed label map, fixed thresholds, behavior KATs,
all folded into `governance_policy_hash`. KATs should be *label*-level
(`expected_label: "contradiction"` for Mercury/Jupiter;
`"entailment_or_not_contradiction"` for the Batman cross-sentence
summary), not exact-logit, unless the runtime is proven stable. The
load-bearing invariant: *same checkpoint + policy → same
demote/no-demote decision*.

**11. Shadow mode before runtime mode.** Add a middle path between
"do nothing" and "runtime NLI": **NLI shadow mode** — optional, off
by default, no `audit_mode` effect, no cache effect, writes only
bench/shadow telemetry (`{shadow_nli_enabled, would_demote,
current_audit_mode, nli_policy_hash, checkpoint_sha256,
source_clause_id, answer_claim_id}`). Lets us measure the
false-positive rate on bench-qa output *before* the model is allowed to
affect `audit_mode`. This is the no-regression path.

**12. Bench gate (explicit) — before enabling runtime demotion:**
(1) `falsification-hard` catches the 2 recombination fixtures; (2)
legit cross-sentence summaries — no demotion; (3) paraphrase/entity
legit cases — no proof-quality regression; (4) bench-qa sample —
shadow-mode false-positive rate below threshold; (5) latency — NLI
called only on a small unresolved subset; (6) determinism — same run
×3 → identical demote/no-demote decisions. Suggested thresholds:
closes both known fixtures; 0 regressions on curated legit-summary
fixtures; 0 STRICT/EVIDENCE-WARRANTED regressions in n=3 `bench-qa`;
NLI invocation rate < 5% of verified answers unless explicitly
configured; p95 added latency bounded and reported. Any fail → keep
NLI shadow-only.

**13. CTI / claim-lattice framing.** Represent a recombination as a
clause-level attribution failure: claim `LargestPlanet(Mercury)` vs
source clauses `LargestPlanet(Jupiter)` + `SmallestPlanet(Mercury)`
— lexical coverage true, attribution consistency false → demote weak
grounding. A CTI-style trace records `{claim, claim_tuple,
source_tuples, recombination_risk, contradiction_veto}`. Even
approximate tuple extraction makes this the conceptual target; NLI is
the model-assisted approximation to the missing attribution relation.

**14. Prometheus-Σ framing.** Branch-selection: A = lexical only,
recombination stays a documented boundary; B = deterministic
heuristic demotion; C = NLI shadow mode only; D = NLI demotion-only
runtime; E = full model-based verifier replacement. `utility =
recombination-catch-rate − legit-answer-regression − latency-cost −
dependency-cost − reproducibility-risk − governance-complexity`.
Decision: **A for production, C optionally for measurement, D only
on real evidence, B not recommended except as triage, E rejected.**

**15. SelfModel / MemoryRoot.** The system should record the
limitation honestly: "lexical verifier cannot generally detect
attribution recombination where all answer tokens appear in the
source but source predicates attach to different subjects; known
fixtures Mercury/Jupiter, Einstein/Darwin; current production policy
= documented boundary, not automatically closed." If shadow mode
lands, SelfModel records the NLI checker's availability /
shadow-vs-demotion state / checkpoint version / bench status.

**16. Implementation roadmap.**
- *Phase 0 — keep boundary marker (this amendment).* #000049 stays
  open; `falsification-hard` stays 10/12; documentation explicitly
  states the residual 2/12 are attribution recombination, not lexical.
- *Phase 1 — shadow design (doc-only).* NLI manifest format;
  fetch/verify command design; `nli_pair@v1` canonicalization design;
  recombination-risk trigger design. No runtime `audit_mode` effect.
- *Phase 2 — optional bench-only shadow implementation.* `[nli]`
  extra; `make fetch-nli`; model under `~/.arborist/models/nli/<hash>/`;
  shadow NLI runs only in bench or with `--nli-shadow`; records
  `would_demote`, does not demote.
- *Phase 3 — demotion-only runtime, gated.* Only if Phase-2 bench
  supports it: `governance_policy_hash` includes `nli_policy_hash`;
  NLI runs only on recombination-risk weak labels; contradiction
  demotes only, never promotes; audit output records reason/hash/pair.
- *Phase 4 — mesh blob sync.* Peer-pull the checkpoint by sha256;
  origin URL remains canonical fallback; hash verification
  fail-closed.

**17. Discipline question — final answer.** *May a small fixed NLI
model influence `audit_mode`?* — **Not by default. Yes only as an
opt-in, hash-pinned, governance-hashed, demotion-only contradiction
veto after shadow-mode evidence.** Respects D1 (the answering LLM
isn't grading itself; the NLI model is fixed and content-addressed;
it can't promote truth; it only vetoes weak grounding; every
checkpoint/policy change invalidates prior cache identity; the output
is bench-gated and auditable) while respecting the danger (a model in
the verifier path is still a model in the verifier path — a
controlled exception, not a casual dependency).

**18. Phase-2 candidate bench (2026-05-12).** Harness:
`~/git/arborist-nli-bench/` (sibling repo — keeps `transformers`/
`torch` out of arborist's `python+sqlite3` core; commits `829f9a4`
[harness] + `a1cb28d` [harder eval + 5-candidate scorecard]). Runs
the §7 #5 clause-level algorithm checkpoint-agnostically (reads
`id2label` from each model's config — handles the cross-encoder vs
MNLI vs BART label-ordering differences with no per-model wiring)
over **28 synthesized recombination cases** (incl. the two
`5f-fal-hard-003/-005` fixtures + harder shapes: longer sources with
the swap buried mid-paragraph, paraphrased predicates, numeric swaps,
qualifier swaps — the veto *should* fire) + **26 legit cases** (true
cross-sentence summaries + near-miss decoys: same-subject-across-
clauses, qualifier-dropping-but-still-true, competing-superlative,
negation-paraphrase — the veto must *not* fire). CPU only, float32,
no sampling.

| candidate | params | p50 / p95 (CPU) | catch / FP (guarded) | raw legit-contra | threshold-free gap min(c\|rec)−max(c\|leg) |
|---|---|---|---|---|---|
| `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` (Apache) | 184M | 223 / 279 ms | **28/28 · 0/26** @ θc 0.5, θe 0.9 | 1/26 | 0.977 − 0.986 = −0.01 |
| `cross-encoder/nli-deberta-v3-base` (Apache) | 184M | 235 / 275 ms | 27/28 · 0/26 @ θc 0.5, θe 0.3 | 1/26 | 0.991 − 0.999 = −0.01 |
| `cross-encoder/nli-MiniLM2-L6-H768` (Apache) | 82M | **45 / 60 ms** | **28/28 · 0/26** @ θc 0.5, θe 0.9 | 1/26 | 0.759 − 0.946 = −0.19 |
| `facebook/bart-large-mnli` (MIT) | 407M | 259 / 350 ms | **28/28 · 0/26** @ θc 0.5, θe 1.01 | **0/26** | 0.648 − 0.412 = **+0.24** |
| `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli` (MIT) | — | — | repo id failed to resolve under transformers 5.x (TODO — base variant of the same family loaded fine, so it's an id/availability hiccup, not fundamental) | — | — |

All four working candidates deterministic across re-runs.

**Key finding — the §7 #5 two-threshold rule is load-bearing, not
decorative.** On the *threshold-free* argmax view, three of four
candidates argmax-label one legit case as `contradiction` (1/26 ≈
4%): the model picks the *wrong source clause* — e.g. `legit-025`
("The Sahara is the largest hot desert on Earth. Antarctica is the
largest desert overall." / claim "The Sahara is the largest hot
desert on Earth.") triggers a spurious contradiction on the
*Antarctica* clause because the model reads "largest hot desert" vs
"largest desert overall" as competing superlatives. **The
entailment guard catches every one of these**: another source clause
(here the Sahara one) directly *entails* the claim, so `max_entail`
is ≈0.99 ≥ θe and the `Demote()` rule does not fire. That is exactly
the failure mode §7 #5 step 6's `max_entail < θ_e` conjunct was
written for ("contradiction alone should not demote if another clause
entails the answer"), now empirically confirmed. A single-threshold
contradiction veto would carry ~4% FP; the two-threshold rule
carries 0% on this set.

**Picture.** Contradiction recombination is an *easy* task for any
modern sentence-pair NLI checkpoint — the differentiator is **cost
and threshold-robustness, not capability**. The 82M
`nli-MiniLM2-L6-H768` (45 ms p50, 6× faster) hits full catch · 0 FP
with the standard θe=0.9 guard — the natural pick for a check that
runs only on a small unresolved subset. `bart-large-mnli` (407M, 259
ms) is the only one with a *clean threshold-free* separation (no
entail guard needed, +0.24 gap) — the pick if latency budget allows
and threshold-robustness is prized over speed. `cross-encoder/
nli-deberta-v3-base` is the weakest of the working four (needs θe=0.3,
loses one catch). Null baseline (current lexical verifier, no NLI):
0/2 catch on the fixtures · 0 FP — so all four working candidates
beat it *net* on this eval set.

**Caveats (why this clears §7 #12 gate items 1–3 + 6 only — not
item 4).** (a) The eval set is *synthesized and still relatively
clean* — single-/few-sentence sources, encyclopedia-shaped facts;
real Wikipedia QA has long messy contexts, partial overlaps, hedged
claims, and the paraphrase/entity path it'd attach to rarely fires
on bench-qa traffic anyway. **Gate item 4 — shadow-mode FP rate on a
full `bench-qa` sample — is the actual test and is not done here.**
(b) n = 54 total — better than n = 2, still small; the harder shapes
help but a bench-qa-traffic-derived set would be more load-bearing. (c)
4 of 5 manifest candidates ran (deberta-large repo-id TODO) — but the
picture is already clear enough that "which checkpoint" is a
cost/robustness trade, not an accuracy one. **None of Phases 2 (full
shadow implementation *in arborist*) / 3 / 4 are unblocked by this
run** — it unblocks the *decision to build a shadow mode at all*,
with real numbers replacing a contrived 2/12, and the answer it
points to is: yes, a shadow mode is worth building; the §7 #5
two-threshold algorithm is the right shape (the entailment guard
earns its place); a small cross-encoder NLI suffices; the one open
risk is bench-qa-traffic false positives, measurable only by a shadow
run on actual QA (Phase 2 → gate item 4).

**19. Phase-2 shadow scaffold landed in arborist (2026-05-12).**
`arborist/qa/nli/` — SHADOW ONLY (writes nothing to `providence_cache`
/ `audit_events`, never an `audit_mode` input; per §7 #2 the manifest
does *not* yet fold into `governance_policy_hash` because shadow
output can't touch `audit_mode`). Pieces: `manifest.json` (pins
`cross-encoder/nli-MiniLM2-L6-H768` @ a fixed HF revision + the
bench-validated θc 0.5 / θe 0.9 + two alternates; carries the Phase-3
TODO — add single-blob `checkpoint_sha256` + `tokenizer_sha256` +
`nli_policy_hash` and ONNX-export to drop torch); `shadow.py`
(`ShadowNLI` / `shadow_check` — lazy `transformers`+`torch` import
behind the new `[nli]` extra, `clauses()` segmenter, the §7 #5
clause-level `Demote()` decision; construction never raises, degrades
to `available=False` when `[nli]` absent); `[nli]` extra in
`pyproject.toml` (hard out of core/dev — fresh checkout stays
`python3.12 + venv + sqlite3`; `make bootstrap-nli` to opt in);
`bench/scripts/nli_shadow_sweep.py` + `make bench-nli-shadow` (the
gate-item-4 instrument — sweeps `(answer, context)` records, reports
the *would-demote* rate bucketed by verifier label; renders even
without `[nli]`, marked `available:false`); <!--AUTOCOUNT:tests:tests/test_nli_shadow.py-->24<!--/AUTOCOUNT--> tests in
`tests/test_nli_shadow.py` (pure-Python parts + graceful degradation
+ the bench-sweep parser — run in the default suite).

**First shadow sweep (2026-05-12, `nli-shadow-v1-minilm2-l6-h768`,
θc 0.5 / θe 0.9, 116 records, ~14 s CPU; `bench/results/nli-shadow-sweep.json`)** —
inputs: `bench/fixtures/5f/falsification-{hard,live}-v1.jsonl` (all
*false* claims the lexical verifier already rejects) + the
`arborist-nli-bench` 28 recombination / 26 legit-summary eval sets:

| verifier-label bucket | n | would_demote | rate | reading |
|---|---|---|---|---|
| `contradiction` (synth recombination, incl. the 2 fixtures) | 28 | 28 | **1.00** | every recombination falsehood demoted — closes the fixtures (gate item 1) |
| `not_contradiction` (synth legit summaries + decoys) | 26 | 0 | **0.00** | zero false positives (gate item 2) |
| `STRICT_SPAN` | 9 | 0 | 0.00 | never fires on an already-STRICT-verified record (gate item 3, in this sample) |
| `STRICT_PARAPHRASE` | 1 | 0 | 0.00 | " |
| `UNGROUNDED` (5f-fal-live, all false) | 50 | 25 | 0.50 | NLI demotes the *attribution-contradiction* half, stays quiet on the *non-sequitur* half ("random claim" vs "unrelated context" is *neutral*, not contradicted — correct: NLI is a contradiction veto, not a grounding check) |
| `HYBRID_ENTITY` | 2 | 1 | 0.50 | n too small to read |

So gate items 1, 2, 3, 5 (latency ≈ 60 ms/pair CPU, and it'd run only
on the unresolved subset), 6 (deterministic — `ShadowNLI` is) all look
clear on available data. **Item 4 — the shadow FP rate on a *real*
`bench-qa` sample (live-LLM answers that actually took the
paraphrase/entity path) — remains the one open measurement**; this
sweep used synthetic legit + the 5f packs, not real Wikipedia QA
output. The instrument is in place (`make bench-nli-shadow
INPUT="…"`); feeding it a live `bench-qa` run is the next concrete
step, and a slow/live one (fox-decides). Nothing here changes the
production verifier; `falsification-hard` stays 10/12.

**20. First *bench-qa-traffic* shadow sweep (2026-05-12) — the haystack
problem.** *Terminology:* "bench-qa traffic" here means **arborist's
own query pipeline exercised by the curated `bench/qa_questions.txt`
set** — real FTS5 retrieval over the real ~3.4 M-article enwiki
corpus → real assembled 8–64 KB contexts → the real Hermes LLM →
the real verifier — *not* live human users (there are none). It is
the messiest input we actually produce, and the right proxy for "what
would real questions look like"; the `ARBORIST_NLI_SHADOW=1` hook is
the same machinery that would accumulate telemetry on live queries
if there ever were any. "Synthetic" (§7 #18) = the contrived
`arborist-nli-bench/eval/*.jsonl` + 5f fixtures (short, clean,
designed-to-test). Added the live hook: `ARBORIST_NLI_SHADOW=1` makes
`query()` surface the verifier-input text (gated, off by default,
never a cache_key / governance / `audit_mode` input — `query.py`
~`return result`), `bench/qa_sweep.py` carries it + the answer into
its rows when present, and `bench/scripts/nli_shadow_sweep.py` reads
those rows (buckets by the row's `audit_mode`; treats a `would_demote`
on a `STRICT` row as a candidate false positive). Ran
`ARBORIST_NLI_SHADOW=1 make bench-qa-smoke` (5 questions × 3 modes =
15 cells against Hermes) → `bench/results/nli-shadow-sweep-smoke-live.json`:

| audit_mode bucket | n | would_demote | rate |
|---|---|---|---|
| STRICT | 10 | **3** | **0.30** |
| HYBRID | 3 | 2 | 0.67 |
| UNGROUNDED | 2 | 2 | 1.00 |

**A 30% would-demote rate on STRICT answers — i.e. a ~30% false-
positive rate.** That is *far* above any acceptable §7 #12 gate
threshold (item 2/4 want ≈ 0), and it is the opposite of the §7 #18
synthetic result (0/26 FP). The cause is the **haystack**: a real
retrieved context is 8–64 KB → 100–336 *clauses* per the §7 #5
step-1 segmenter; running `NLI(clause, answer)` over *all* of them
and taking `max_contradiction` is a massive multiple-comparisons
problem — somewhere in 200 tangential Wikipedia clauses there is
almost always one the model reads as contradicting a short
synthesized answer, and the entailment guard doesn't always rescue it
because a *paraphrased/aggregated* STRICT answer often isn't verbatim-
entailed by any single clause (max_entail stays < θe even though the
answer is perfectly grounded). On the clean synthetic set (1–3 clause
sources) `max`-over-2 is fine; `max`-over-300 on real context is a
different regime. (Also: 417 s for 15 records ≈ 28 s/record on CPU —
running NLI on hundreds of clauses per answer is too slow for inline
use regardless.)

**Consequence for the design — the §7 #5 "candidate source clauses" +
"recombination-risk heuristic fires" gating is not optional, it is
load-bearing.** The shadow scaffold's `ShadowNLI.check` deliberately
ran NLI on *every* clause (the simplest thing that could measure
anything); that version is now empirically shown to be unusable on
bench-qa traffic. Phase 3 (and a refined Phase-2 shadow) MUST restrict the
NLI call to a *small* candidate set — only the clauses the lexical
verifier's quote/span/entity matching actually touched, and/or only
when a deterministic recombination-risk trigger fires (high
content-token coverage, answer subject/object tokens occurring in
*different* matched clauses, weak verifier path only — §7 #5 step 3,
§7 #12). The §7 #18 framing ("recombination is an *easy* task for any
modern NLI checkpoint") holds for the *isolated clause-pair* judgment;
it does **not** hold for "find the one contradicting clause in a
30 KB haystack" — that is a precision/triage problem the NLI model
cannot solve alone, and the deterministic recombination-risk pre-
filter is what makes it tractable. So: do not enable any runtime NLI
demotion on the current scaffold; the next Phase-2 step is to add the
candidate-clause restriction to `ShadowNLI.check`, re-run the
bench-qa-traffic sweep, and only then is gate item 4 measurable as a
meaningful number. Production verifier unchanged; `falsification-hard`
stays 10/12.

**21. Candidate-clause restriction added — helps, doesn't close it
(2026-05-12).** `ShadowNLI.check` now runs NLI only on the *top-N
source clauses by content-token overlap with the answer claim*
(`candidate_clauses()`; `max_candidate_clauses=6` in the manifest;
zero-overlap clauses dropped; if nothing overlaps → `would_demote=False,
reason="no_candidate_clauses"`), and records `n_candidate_clauses`,
`best_clause_overlap`, `recombination_risk` (claim tokens scattered
across ≥2 candidate clauses, none dominant). Re-ran both sweeps:

- *Synthetic* (`bench/results/nli-shadow-sweep.json`, 116 records,
  ~9 s — mean **1.45** candidate clauses/record): `contradiction`
  28/28, `not_contradiction` **0/26**, `STRICT_SPAN`/`STRICT_PARAPHRASE`
  0 — the clean-set numbers are unchanged (short sources → top-6 ≈ all
  clauses anyway).
- *Real traffic* (`bench/results/nli-shadow-sweep-smoke-live.json`,
  the 15-cell smoke re-run): **STRICT would-demote 30% → 20%** (2/10),
  overall 47% → 33%. Better, **not fixed**. The recombination-risk
  split doesn't separate either (risk 4/10 = 0.40, no_risk 1/5 = 0.20
  — a no-risk case still FPs).

The residual STRICT FPs land at `max_contradiction` ≈ 0.83 / 0.88
(and a HYBRID at 0.92) — the model is *moderately* confident a
top-overlap Wikipedia clause contradicts a *synthesized/aggregated*
STRICT answer (a dropped qualifier, a tangential date, a sibling
fact). On the clean synthetic set the recombination contras are ≈
0.99 and the legit ones ≈ 0.005, so the manifest's θc = 0.5 was fine
*there*; on bench-qa traffic the false-contra mass sits in 0.6–0.92, so
**θc would need to rise to ≈ 0.90**. At θc = 0.90 on the data in hand:
synthetic recombination catch = 27/28 (loses 1 hard case), synthetic
legit FP = 0/26 (the one 0.946 legit case is rescued by the entailment
guard, `max_entail` ≈ 0.99), smoke STRICT FP = **0/10**. So the
*combination* {candidate-clause restriction + θc ≈ 0.90 + entailment
guard} looks viable — but n = 10 STRICT in the smoke is far too small
to set a threshold on. **Next step (slow/live, fox-decides): a fuller
`ARBORIST_NLI_SHADOW=1 make bench-qa` run (n=1 or n=3 over the full
question set) → ~hundreds of STRICT cells → sweep θc/θe over that,
confirm the FP rate at θc ≈ 0.90, *then* set it in the manifest.**
Until that confirmation: θc stays 0.5 (the clean-set default), runtime
NLI demotion stays off, and the standalone-lexical candidate proxy
stays a proxy — a Phase-3 runtime hook would pass the verifier's
*actual* matched clauses, tighter than top-6-by-overlap. The deeper
standing lesson: the clean synthetic eval (§7 #18) does **not** predict
bench-qa-traffic precision — every gate number that matters has to come
from a shadow run on `bench-qa` pipeline output, not from contrived
fixtures. Production verifier unchanged; `falsification-hard` stays
10/12.

**22. Full-ish bench-qa shadow sweep + speedup (2026-05-12).**

*Speedup (the §3 plan).* `ShadowNLI` now (1) **batches** the forward
passes — `_nli_batch(pairs)` runs ⌈N/`batch_size`⌉ batched forwards
instead of N batch-1 ones (`ARBORIST_NLI_BATCH=64` default); (2)
**auto-detects device** (`ARBORIST_NLI_DEVICE` env, else cuda if
available else cpu — `.to("cuda")` + cuda inputs on the torch path,
`CUDAExecutionProvider` on the ONNX path); (3) **prefers an ONNX
export** if one exists — `bench/scripts/export_nli_onnx.py` /
`make export-nli-onnx` exports the pinned checkpoint to ONNX + int8-
dynamic-quantizes it into `~/.arborist/models/nli/<ver>/onnx/` (operator
state, **not** committed — same discipline as the textbook manifest /
`[vec]`), and `_ensure_loaded` loads `model_quantized.onnx` via
`optimum.onnxruntime` (backend `onnx-int8`), falling back silently to
torch when no export / no `optimum`. Measured: torch-cpu-batch1 ≈ 120
ms/pair → **onnx-int8-cpu-batched ≈ 32 ms/pair** (~4×) → **torch-cuda-
batched ≈ 0.09 ms/pair on a 4090** (512 pairs in 0.047 s — ~350×
onnx-int8-cpu, ~1300× the original), so a full `bench-qa` n=3 shadow
sweep (~8 k clause-pairs) is **< 1 s** of NLI compute on the GPU
(2.4 s wall for a 62-record synthetic sweep, model load included).
`optimum[onnxruntime]` added to the `[nli]` extra; 24 tests in
`tests/test_nli_shadow.py`. **GPU compute-node workflow** (verified on
`ai`, RTX 4090): `make bootstrap-nli-only` (a venv + `[nli]` only — no
`[dev]` bloat; PyPI's torch wheel is the CUDA build on a CUDA host) →
`ShadowNLI` auto-detects cuda (`ARBORIST_NLI_DEVICE` overrides) → `make
bench-nli-shadow [INPUT=…]` / `make export-nli-onnx` / the candidate
bench all run on the GPU with no further wiring. The pattern: ship the
laptop's `bench-qa` JSONL to the GPU box, run the sweep there in
seconds. (`ai` doesn't have the ~6 GB shard corpus, so `make bench-qa`
itself still runs on the laptop — that's the Hermes LLM, not local
GPU; the GPU only accelerates the NLI half.)

*The gate-item-4 number, at proper n.* `ARBORIST_NLI_SHADOW=1 make
bench-qa BENCH_QA_N=1` → 75 q × 3 modes, of which 223 cells completed
(89 STRICT / 90 HYBRID / 44 UNGROUNDED — the run also surfaced a
lone-surrogate bug in real Wikipedia context that `qa_sweep` now
scrubs). Shadow sweep over those 223 (`bench/results/nli-shadow-sweep-benchqa-n1.json`,
candidate-clause restriction on, θe = 0.9):

| audit_mode | n | would_demote @ θc 0.5 | reading |
|---|---|---|---|
| STRICT | 89 | **23 (25.8%)** | the false-positive rate — confirms the smoke at proper n |
| HYBRID | 90 | 36 (40%) | already-demoted; further demotion less harmful |
| UNGROUNDED | 44 | 28 (64%) | already-rejected; NLI agreeing is fine |

θc-sweep on the 89 STRICT cells (FP rate): 0.5→25.8%, 0.7→19.1%,
0.8→12.4%, 0.85→11.2%, **0.90→7.9%**, 0.95→5.6%, 0.97→2.2%, 0.99→0%.
`max_contradiction` on STRICT: p50 0.29, p90 0.88, p95 0.96, max
0.99. So **even with the candidate-clause restriction, NLI-as-runtime-
veto on STRICT answers is *not* gate-passable at any θc that still
catches recombination** — the synthetic recombinations bottom out at
~0.76 on the hard cases (~0.98 on easy), so a θc high enough to get the
STRICT FP rate to ≈0 (0.99) gives up most of the recall the veto exists
for; θc = 0.90 is the least-bad point (~8% STRICT FP, ~96% synthetic
recall) but ~8% false-demote on confidently-grounded answers is well
above any acceptable gate. The recombination-risk split doesn't rescue
it (STRICT FP: 20/65 risk vs 3/24 no-risk — both nonzero).

**Verdict.** This *hardens* §7 #21's conclusion at real n:
NLI-as-runtime-demotion-veto, on the current standalone-lexical
candidate-clause design, **fails the §7 #12 gate** (item 2/4: a
~8–26% false-positive rate on STRICT). The recombination boundary is
*not* closed by this approach. What remains untried — and the only
path that might pass: a **Phase-3 runtime hook** that runs NLI on the
clauses the *lexical verifier actually matched a span/quote/entity
against* (1–3 clauses, the right ones), not "top-6 by token overlap"
(6 clauses, several merely lexically-overlapping) — a verifier-side
change, not a standalone proxy. Until that's built and re-measured:
**runtime NLI demotion stays off; the 2 fixtures stay permanent
boundary markers; the shadow telemetry is a monitoring signal, not a
gate.** θc stays 0.5 in the manifest. Production verifier unchanged;
`falsification-hard` stays 10/12.

**23. Grid sweep flips the verdict — the §7 #22 "fails" was config-
specific, not approach-specific (2026-05-12).** §7 #22's "fails the
gate" was the verdict for the *default config* (`k=6, θc=0.5, θe=0.9`)
— which was tuned on the clean synthetic set, where everything is
≈0.99-vs-≈0.005, and is *not* the right config for bench-qa traffic.
A grid sweep over `{model × candidate-clause-cap k ∈ {1,2,3,4,6,12} ×
aggregation ∈ {max, mean, top2} × θc × θe}` (`bench/scripts/nli_shadow_grid.py`
— NLI runs once per (model, record) over the top-12 candidate clauses;
the k/agg/θ grid is then pure arithmetic on the cached per-clause
scores; the full 4-model grid is ~10 s on the 4090 vs ~3 min on CPU)
finds clean passing configs. Three sweeps (n=1 bench-qa = 89 STRICT
cells; results in `bench/results/nli-shadow-grid-n1-*.json`):

- *Sweep B — synthetic-28 recombination recall vs synthetic-26 legit
  FP*: every model, **catch 1.00 · FP 0.00** at `k≥2`, low θc — the
  clean set is trivially solved (reconfirms §7 #18; says nothing about
  real traffic).
- *Sweep A — the 12 `falsification-hard` false-claims (incl. both
  recombination fixtures) recall vs the 89 real STRICT cells FP*:
  best fp=0 configs catch **8-9 / 12** — MiniLM-82M `k=1/θc=0.93` →
  0.75, deberta-base-184M `k=2/θc=0.99` → 0.75, bart-large-407M → 0.67;
  both `5f-fal-hard-003` (Mercury, contra 0.98) and `-005` (Einstein,
  contra 0.98) fire — **the 2 fixtures #000049 exists for are caught
  at 0 real-STRICT FP**.
- *Sweep C — the actual gate question: synthetic-28 recombination
  recall vs the 89 real STRICT cells FP*:

| NLI model | best `fp=0` config | recombination catch | real-STRICT FP |
|---|---|---|---|
| MiniLM-82M | `k=2, agg=max, θc=0.95, θe=0.9` | **24/28 = 0.857** | **0/89** |
| **deberta-base-184M** | **`k=2, agg=max, θc=0.99, θe=0.9`** | **27/28 = 0.964** | **0/89** |
| bart-large-407M | `k=3, agg=mean, θc=0.95, θe=0.5` | 20/28 = 0.714 | 0/89 |
| bart-large-407M (relaxed) | `k=2, agg=max, θc=0.95, θe=1.01` | 27/28 = 0.964 | 4/89 = 0.045 |

**`deberta-base-184M` at `k=2 / agg=max / θc=0.99 / θe=0.9` catches 27
of 28 synthetic recombinations with ZERO false positives on 89 real
STRICT answers — that is a passing config.** MiniLM-82M passes too at
24/28. The structure that works: **k=2** (two candidate clauses — k=1
misses cases where the best-token-overlap clause isn't the
contradicting one; k≥3 with `max` reintroduces haystack; `mean` over
more clauses dilutes), **`max` aggregation**, **θc ≈ 0.95-0.99** (not
0.5), **θe = 0.9** (the manifest default for the guard is fine; the
tighter 0.5 isn't needed once θc is high). Model-size science: **184M
> 82M > 407M for fp=0 recombination recall** — bigger isn't
monotonically better, bart-large's contradiction scores are more
spread so it needs a lower θc that costs FP; deberta-base is the sweet
spot (much better recall than MiniLM at the same fp=0 bar, ~3 s for
~1100 pairs on GPU). int8-ONNX quantization costs ≈1 catch vs fp32
(MiniLM 0.667 int8 vs 0.75 fp32 on Sweep A) — a small but real tax.

**Revised verdict.** The lexical-candidate NLI veto **clears the §7
#12 gate** (catch the recombinations incl. both fixtures, ≈0 false-
demote on confidently-grounded answers) — at **~86-96% recombination
recall depending on model**, with `deberta-base / k=2 / agg=max /
θc=0.99 / θe=0.9` the recommended operating point. The Phase-3
verifier-matched-clause hook would only push recall higher (a refined
candidate selector), not rescue a failing approach. **Caveats before
this is final:** (a) the FP side is n=1 = 89 STRICT cells; a full
`ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=3` run (~250 STRICT
cells, in flight) must confirm fp stays 0 at the chosen θc — until
then the manifest defaults stay `k=2`-untuned (`max_candidate_clauses=6`,
θc 0.5); (b) the "27/28" / "24/28" recall is on the *synthetic*
recombination set — a bench-qa-derived recombination set (real false
answers that took the paraphrase/entity path) would be more
load-bearing, but those are rare in real traffic (the whole point of
the boundary); (c) this is still SHADOW — flipping it to a runtime
demotion-only veto is a fox-decides (`MODEL_ASSISTED_DEMOTION`, and
`nli_policy_hash` — model + k + agg + θc + θe — folds into
`governance_policy_hash` at that point, per §7 #2). What changes
*now*: the manifest gains `agg` + the k=2/θc=0.99 candidate config as
the recommended-but-not-yet-default operating point, pending the n=3
confirmation. Production verifier unchanged; `falsification-hard`
stays 10/12.

**24. The n=3 confirmation — §7 #23 was a small-sample artifact; the
no-stone-unturned mega-grid found the config that *does* survive
(2026-05-12).** Ran `ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=3`
→ 675 cells, **275 STRICT** (vs the 89 from n=1 — 3× the FP-side
denominator). Re-ran the grid with the expanded sweep
(`nli_shadow_grid.py` now: 7 aggregations incl. `margin` [= max over
clauses of `p_contra − p_entail`, a single score folding the
entailment guard in] + `paired_entail` guard variant; θc grid to
0.999; `--extra-models`) over all manifest models **+ 4 extra
checkpoints** spanning xsmall→large (`microsoft/deberta-large-mnli`,
`roberta-large-mnli`, `cross-encoder/nli-deberta-v3-small`,
`MoritzLaurer/DeBERTa-v3-xsmall-mnli-fever-anli`), against the
synthetic-28 recombination set (recall) vs the 275 real STRICT cells
(FP). 3355 NLI pairs/model, ~3.5 min total on the 4090. Result
(`bench/results/nli-shadow-grid-n3-synth-vs-realstrict.json`):

| NLI model (size) | best `fp=0` config | recombination catch | real-STRICT FP (n=3, 275 cells) |
|---|---|---|---|
| **`microsoft/deberta-large-mnli` (~400M)** | **k=3, agg=`margin`, θc=0.995** | **28/28 = 1.000** | **0/275** |
| `roberta-large-mnli` (~355M) | k=2, agg=max, θc=0.998 | 21/28 = 0.750 | 0/275 |
| `bart-large-mnli` (~407M) | k=3, agg=mean, θc=0.94 | 20/28 = 0.714 | 0/275 |
| MiniLM-82M | k=4, agg=mean, θc=0.96 | 14/28 = 0.500 | 0/275 |
| `deberta-base-184M` *(the §7 #23 hero)* | k=3, agg=mean, θc=0.995 | **11/28 = 0.393** | 0/275 |
| `nli-deberta-v3-small` (~140M) | — | 0/28 | 0/275 |

**The §7 #23 "deberta-base k=2/θc=0.99 → 27/28 · 0/89" does NOT
survive n=3.** The 89-STRICT n=1 sample under-counted the spurious-
contradiction tail; the 275-cell sample surfaces ~14% more of it,
forcing θc up to ~0.995, and at θc=0.995 deberta-base only catches
11/28 recombinations. So §7 #23's "the lexical-candidate veto passes"
was over-optimistic by exactly the amount §7 #23's own caveat warned
about. **However** — the *broader* sweep (the "no stone unturned"
that this run was) found the config that *does* survive at n=3:
**`microsoft/deberta-large-mnli` / k=3 / agg=`margin` / θc=0.995 →
catches all 28 synthetic recombinations (incl. both `5f-fal-hard`
fixtures, margins 0.997/0.998) + 8/12 of the `falsification-hard`
pack, with 0 false positives on 275 real STRICT answers and 0 on the
26 synthetic legit summaries.** A passing config at proper n.

Two findings from the mega-grid:
- **The `margin` aggregation is the right score-shape.** `margin =
  max_clause(p_contra − p_entail)` collapses the two-threshold rule
  (`p_contra ≥ θc ∧ p_entail < θe`) into a *single* threshold and is
  what lets the strong model separate cleanly at n=3 — every other
  aggregation needs the more fragile two-threshold dance.
- **The *specific checkpoint* matters more than parameter count.**
  `microsoft/deberta-large-mnli` (a plain MNLI fine-tune, ~400M) is
  the clear winner; `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli`
  (similar size) won't even load under transformers 5.x;
  `deberta-base-184M` (the n=1 hero) collapses at n=3;
  `roberta-large-mnli` / `bart-large-mnli` (similar size to the
  winner) only reach ~0.71-0.75. There is no "bigger is better" law —
  you have to sweep the actual checkpoints.

**Revised verdict.** The lexical-candidate NLI veto **can clear the
§7 #12 gate at proper n** — but only with the right checkpoint
(`microsoft/deberta-large-mnli`) + the right score-shape (`margin`) +
a high θc (~0.995), and at "100% of the 28 synthetic recombinations"
recall (which is a 28-case denominator — 100%-of-28 ≠ 100%-in-general;
a bench-qa-derived recombination set would be more load-bearing). The
honest line for dav1d: *the §7 #22 "fails" was a default-config
artifact; the §7 #23 "passes" was a small-FP-sample artifact; the
n=3-confirmed truth is `deberta-large-mnli + margin + k=3 + θc=0.995`
passes — 28/28 synthetic recombination · 0/275 real STRICT — but the
margin (θc=0.995, a 28-case recall denominator) is thin enough that a
bigger STRICT sample and a bigger recombination set should be the next
checks before runtime promotion.* Caveats unchanged: still SHADOW;
flipping to runtime demotion-only is fox+dav1d-decides; `nli_policy_hash`
(model + k + agg + θc) folds into `governance_policy_hash` at that
point; deberta-large is ~400M (GPU-fast, slow on CPU — and ONNX-int8
would cost a catch or two). The manifest's `recommended_operating_point`
is updated to this config (still not the active default). Production
verifier unchanged; `falsification-hard` stays 10/12.

*Meta-lesson, sharpened twice now:* the clean synthetic eval doesn't
predict bench-qa precision (§7 #18→#20), the default config doesn't
predict the best config (§7 #22→#23), **and a small FP-side sample
doesn't predict the large-sample FP rate (§7 #23→#24).** Every gate
number is provisional until the denominators are big enough — and the
sweep has to be wide enough to include the config that survives them.
The 4090 is what makes that affordable: this whole §7 #24 run — 7
models × the full {k × agg × guard × θc × θe} grid × 303 records — was
~3.5 min.

**25. The n=5 confirmation — it firms up, and the right config + model
crystallize (2026-05-12).** `ARBORIST_NLI_SHADOW=1 make bench-qa
BENCH_QA_N=5` → 1125 cells, **444 STRICT** (5× the n=1 sample, 1.6×
the n=3). Re-ran the full 7-model mega-grid against the synthetic-28
recombination set vs the 444 real STRICT cells
(`bench/results/nli-shadow-grid-n5-synth-vs-realstrict.json`):

| NLI model (size) | best `fp=0` config | recombination catch | real-STRICT FP (n=5, 444 cells) | θc margin |
|---|---|---|---|---|
| `roberta-large-mnli` (~355M) | k=2, agg=max, θc=0.95 | **28/28 = 1.000** | **0/444** | ~5 pts |
| `microsoft/deberta-large-mnli` (~400M) | k=2, agg=max, θc=0.96 | **28/28 = 1.000** | **0/444** | ~4 pts |
| `deberta-base-184M` | k=2, agg=max, θc=0.99 | 27/28 = 0.964 | 0/444 | **edge** (28/28 needs θc=0.975 → fp 0.004) |
| MiniLM-82M | k=2, agg=max, θc=0.98 | 23/28 = 0.821 | 0/444 | — |
| `nli-deberta-v3-small` (~140M) | k=2, agg=max, θc=0.998 | 23/28 = 0.821 | 0/444 | — |
| `bart-large-mnli` (~407M) | k=3, agg=mean, θc=0.94 | 20/28 = 0.714 | 0/444 | — |

**What resolved:**
- *Model.* The **large models (~350–400M) are the robust ones**:
  `roberta-large-mnli` and `microsoft/deberta-large-mnli` both hit
  **28/28 · 0/444 at modest θc (0.95–0.96)** — i.e. ~4–5 points of θc
  margin before they'd start false-demoting. `deberta-large-mnli` is
  the more-validated pick (1.0/0.0 at *both* n=3 [θc=0.995, margin
  agg] and n=5 [θc=0.96, plain max]). **`deberta-base-184M` sits right
  at the cliff** — 27/28 at θc=0.99 on this sample, but the §7 #24 n=3
  sample drove it to 11/28; its config is sample-sensitive. The small
  models cap at ~0.82. So the §7 #18 "MiniLM-82M is the cost-pick" is
  **overturned by the proper-n evidence** — at the fp=0 bar MiniLM
  caps at 0.82, the large model gets 1.0; cost was the wrong axis to
  optimize once you demand 0 false-demote.
- *Aggregation.* `agg=max` + the `max_entail` guard + θc≈0.95 is the
  robust config across *all* models and both proper-n samples; §7 #24's
  `margin` win was a sample-specific tie — keep `max`.
- *k.* `k=2` is the consistent winner everywhere.

**Verdict (n=5-confirmed).** The lexical-candidate NLI veto **passes
the §7 #12 gate, robustly**, with **`microsoft/deberta-large-mnli` /
k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9** → 28/28 synthetic
recombinations (incl. both `5f-fal-hard` fixtures) · 0/444 real STRICT
FP, ~4 pts of θc headroom; `roberta-large-mnli` is an equally-good
alternate. The remaining caveats are now narrow: (a) the 28/28 recall
is still a *28-case* denominator — a bench-qa-derived recombination set
(real false answers that took the paraphrase/entity path) is the one
load-bearing check not yet done (those are rare in real traffic — the
whole reason this is a "boundary"); (b) the FP side is 444 cells at 0
FP, a solid denominator — n=9 (~825 cells) would add confidence but,
given the ~4–5 pt θc margin, is unlikely to move the verdict (n=9 is
"if dav1d wants more"); (c) still SHADOW — flipping to a runtime
demotion-only veto is fox+dav1d-decides, then `nli_policy_hash` =
model+k+agg+guard+θc folds into `governance_policy_hash` (§7 #2);
deberta-large is ~400M (GPU-fast; ~200–300 ms/pair on CPU on the now-
small k=2 candidate set; ONNX-int8 would cost a catch or two). The
manifest's `recommended_operating_point` is updated to this config;
active defaults stay k=6/θc=0.5 until fox promotes. Production verifier
unchanged; `falsification-hard` stays 10/12.

**This `make bench-qa BENCH_QA_N=5` run (~50 min Hermes) + the 7-model
mega-grid on it (~3 min on the 4090) is the worked example behind
CLAUDE.md's new bench-maxing line** ("when researching a config/model
decision, sweep wide+deep on real-traffic-sized samples; the GPU makes
it cheap; burn the electricity"). The n=1→n=3→n=5 sequence is exactly
why: n=1 said MiniLM/deberta-base pass at high recall; n=3 said
deberta-base collapses; n=5 said the *large* models are the robust
ones and the small ones cap at 0.82 — three samples, three sharper
answers, ~2 hours of GPU+LLM total. A guessed config (MiniLM, per the
n=1 read) would have shipped a 0.82-recall veto when a 1.0-recall one
was available.

**26. Mining for *real* recombinations + the pooled-808-STRICT check
— the §7 #25 "max@0.96" was *itself* a small-sample artifact; `margin`
is reinstated as the right aggregation, and the boundary is
theoretical-in-practice (2026-05-12).** §7 #25 left one open item: the
28/28 recall is on a 28-case *synthetic* set; do real LLM answers
actually recombine? Two passes:

*(a) Mine the pooled n=1+n=3+n=5 bench-qa runs (808 distinct STRICT
answers) for natural recombinations.* Ran the candidate veto over all
808 at a *lowered* θc≥0.7: 37 STRICT-fires (`bench/results/nli-strict-fires-mining-pooled.jsonl`).
On inspection, **all 37 are token-collision false positives, not
genuine recombination errors** — the candidate-clause selector
(top-k by token overlap) pulls in *different-entity, same-subject-area*
clauses and NLI false-fires: e.g. "where is Kilimanjaro located?" /
"Kilimanjaro is in Tanzania" gets the clause "Mount Kenya is located
in central Kenya" pulled in (shares `Mount`, `located`, a country
name) → NLI reads it as a contradiction (contra 0.99); "who painted
the Mona Lisa?" / "da Vinci painted the Mona Lisa" gets "Salvador
Dalí … painted …" (contra 0.95); "Superman's girlfriend" gets a
Donovan *Sunshine Superman* album-tracklist clause (contra 0.99).
**Zero of the 808 real STRICT answers look like an LLM-recombined
falsehood the verifier missed** — the STRICT label holds; the
recombination boundary is, in practice on well-covered factual
questions, a *contrived/theoretical* concern (the 2 fixtures are
hand-built). What the mining *does* prove is that the failure mode is
**systematic in the candidate selector** — the token-overlap proxy
pulls subject-area-sharing-but-different-entity clauses, and NLI then
reads "statement about a different entity" as "contradicts the answer".

*(b) Re-run the grid against the pooled 808-STRICT set (the biggest
real FP denominator we have)* (`bench/results/nli-shadow-grid-pooled808-synth-vs-realstrict.json`):

| config | recombination catch | real-STRICT FP (pooled, 808 cells) |
|---|---|---|
| **`deberta-large-mnli` / k=3 / agg=`margin` / θc=0.995** | **28/28 = 1.000** | **0/808** |
| `deberta-large-mnli` / k=2 / agg=`max` / θc=0.995 | 27/28 = 0.964 | 0/808 |
| `deberta-large-mnli` / k=2 / agg=`max` / θc=**0.96** *(the §7 #25 config)* | — | **FP > 0** (the Superman/Kilimanjaro/Dalí hits at contra 0.95–0.99) |
| `roberta-large-mnli` (best fp=0) | 21/28 = 0.750 | 0/808 |
| `bart-large-mnli` / `deberta-base` / MiniLM (best fp=0) | ≤ 20/28 | 0/808 |

**The §7 #25 "`max` @ θc=0.96" was the small-sample artifact this
time** — the n=5 444-cell STRICT set happened not to contain the
high-confidence spurious hits the pooled n=1+n=3 runs do, so it let
θc drop to 0.96; on 808 cells θc has to go back up to ~0.995. And at
θc=0.995, **`agg=margin` is the only thing that still catches 28/28**
(`max` gets 27/28; `mean` and the others ≤0.75). So **§7 #24's
`margin` finding is reinstated**: `margin = max_clause(p_contra −
p_entail)` is the right score-shape because it discriminates a *real*
recombination (the source genuinely contradicts → that clause's
entail ≈ 0 → margin ≈ 0.996–0.998) from a *spurious token-collision*
FP (the source *also* contains the entailing clause → its margin is
dragged to ~0.95–0.99) — a narrow gap, but it held on 808 cells where
plain contradiction-thresholding did not.

**Settled verdict.** The lexical-candidate NLI veto clears the §7 #12
gate with **`microsoft/deberta-large-mnli` / k=3 / agg=`margin` /
θc=0.995** → **28/28 synthetic recombinations (incl. both `5f-fal-hard`
fixtures) · 0/808 pooled real STRICT FP · 0/26 synthetic legit FP** —
this is the *only* config (across the 7-model × full-grid sweep) that
hits 1.0/0.0 on the 808-cell denominator, and it has held across n=3
(275 cells) and the pooled 808; the §7 #25 "max@0.96" only held on the
n=5 444-cell sample. **Open items now minimal:** (a) the 28/28 recall
is still a 28-case denominator — but since there are *zero* natural
recombinations in 808 real STRICT answers, a "bench-qa-derived
recombination set" can't be mined (there's nothing to mine); the
realistic next step is *synthetic recombination claims constructed
against real bench-qa contexts* (real haystack, deliberate
re-attribution) — that tests "the candidate-selector + veto vs a real
30 KB haystack" on the recall side, and is doable as ~20–30
hand-built fixtures. (b) The systematic candidate-selector FP failure
(different-entity clauses pulled in by token overlap) is *survived*
by `margin@0.995` on real traffic but would be *eliminated* by a
Phase-3 verifier-matched-clause hook (pass NLI only the clause the
verifier actually grounded the answer against) — `margin@0.995` is
robust enough to ship as-is, the Phase-3 hook is a "clean it up
later". (c) Still SHADOW; runtime promotion is fox+dav1d-decides
(then `nli_policy_hash` = model+k+agg+guard+θc folds into
`governance_policy_hash`); deberta-large is ~400M (GPU-fast).
Manifest active defaults stay k=6/θc=0.5; `recommended_operating_point`
reverted to the `margin`@0.995 config. Production verifier unchanged;
`falsification-hard` stays 10/12.

*Meta-lesson, instance five (and the one that cuts the other way):*
§7 #18→#20→#22→#23→#24→#25→#26 — clean eval ≠ bench-qa precision; the
default config ≠ the best config; a small FP sample ≠ the large-sample
FP rate **— and a *bigger* sample can vindicate a config a smaller
one made look unnecessary** (n=5=444 said "drop θc to 0.96, use plain
`max`"; pooled=808 said "no — `margin`@0.995 was right all along").
The honest discipline: don't trust *any* denominator until it's the
biggest you can afford, and re-confirm the *config choice* (not just
the threshold) every time the denominator grows. Pooled 808 STRICT ×
the full grid was ~4 min on the 4090.

**27. The real-context recombination set — synthetic recall *also*
inflated; the candidate-selector is the recall-side bottleneck
(2026-05-13).** §7 #26's "the boundary is closeable; deberta-large-mnli
+ margin@0.995 → 28/28 · 0/808" had one remaining caveat: the 28/28
recall was on a *28-case synthetic* set with 1–3-sentence sources. To
test recall against a *real* haystack, built `bench/fixtures/5f/recombination-realcontext-v1.jsonl`
— 25 hand-crafted recombination claims, each a deliberate
*re-attribution within a real bench-qa context* (8–64 KB Wikipedia
haystack the pipeline actually produced for the curated bench
questions; every claim FALSE; every content token present in the
source somewhere): "Mount Kilimanjaro is in central Kenya, just south
of the equator" (against the real Kilimanjaro article — re-attributes
Mount Kenya's location); "Galileo Galilei discovered all 63 of
Jupiter's confirmed moons" (against the real "Moons of Jupiter"
article — Galileo discovered 4); "Joshua Reynolds painted the Mona
Lisa" (against the real Mona Lisa article — Reynolds is mentioned
there as a critic); and so on across 25 questions × domains. Then ran
the full 7-model × full-grid sweep against these 25 (recall) vs the
808 pooled real STRICT (FP):

| NLI model (size) | best `fp=0` config | recall on *real-context* recombinations (25) | FP (808 pooled STRICT) |
|---|---|---|---|
| **`bart-large-mnli` (~407M)** | **k=12, agg=max, θc=0.999, θe=0.5** | **12/25 = 0.480** | **0/808** |
| `microsoft/deberta-large-mnli` (~400M) | k=2/3, agg=max, θc=0.995 | 6/25 = 0.240 | 0/808 |
| `roberta-large-mnli` (~355M) | k=8, agg=max, θc=0.998 | 3/25 = 0.120 | 0/808 |
| MiniLM-82M | k=4, agg=top3, θc=0.975 | 2/25 = 0.080 | 0/808 |
| `deberta-base-184M` | k=3, agg=mean, θc=0.995 | 1/25 = 0.040 | 0/808 |

**The §7 #26 synthetic 28/28 was a 4× over-estimate of real-haystack
recall.** Same model (`deberta-large-mnli`), same config (k=3/margin/
θc=0.995) — synthetic recall 28/28 = 1.000; real-context recall 6/25
= **0.240**. **The clean synthetic eval didn't predict bench-qa
precision *or* real-haystack recall** — sixth instance of the
denominator/realism lesson (instances 1–5 were all on the FP side;
this one is the recall-side mirror).

**Diagnosis: the candidate-clause selector is the recall-side
bottleneck.** Top-k by content-token overlap picks clauses *similar to
the answer's subject area*; for a recombination claim like
"Kilimanjaro is in central Kenya", the overlap-ranked top-k of the
30 KB Kilimanjaro article are about the mountain itself (height,
climbing routes, climate, glaciers) — clauses that share many tokens
with the claim but *don't contradict it* — while the *Mount Kenya*
clause (which would contradict by re-attribution) shares only "Kenya"
and gets ranked low. So NLI never sees the contradicting clause and
can't fire. This is the *recall mirror* of §7 #26's FP failure mode
(the selector pulling subject-area-sharing-but-different-entity
clauses): the same lexical proxy that pulls in spurious entities
*also fails to pull in genuinely-contradicting ones*. Token overlap
is the wrong tool for both.

**Model-science update at proper recall-side realism:** `bart-large-mnli`
is **the new fp=0 leader on real haystacks** (12/25 at k=12/max/θc=0.999)
— a complete reversal of §7 #24's "bart is the threshold-robust pick"
narrative and §7 #26's "deberta-large-mnli is the winner." Why bart
wins on real haystacks: its score distribution is more spread, so at
very high θc (0.999) it stays FP-safe even at k=12 — and k=12 is what
*finds* the contradicting clause when token overlap doesn't rank it
top-2. The other large models can't use k=12 without false-firing.
The §7 #18 "MiniLM cost-pick" remains overturned; the §7 #26
"deberta-large is the winner" is now *also* overturned at recall-side
realism.

**The precision/recall frontier on real haystacks** (bart-large-mnli
pareto): fp=0.000 catch 0.48; fp=0.011 catch 0.52; fp=0.022 catch 0.60;
fp=0.035 catch 0.64; fp=0.057 catch 0.80; fp=0.067 catch 0.88;
fp=0.068 catch 0.92. So the *gate-passable* (fp=0) operating point
catches ~half of real-haystack recombinations; pushing to 80–90%
recall costs 3–7% false-demote — that's not gate-passable.
deberta-large-mnli's pareto is similar: fp=0 catch 0.24 → fp=0.001
catch 0.56 (a single STRICT FP buys huge recall via k=8) → fp=0.013
catch 0.72.

**Revised verdict (the honest one for dav1d).** The lexical-candidate
NLI veto **partially closes the recombination boundary on real
haystacks** — ~48% recall at strictly 0 FP on 808 real STRICT cells,
with `bart-large-mnli / k=12 / agg=max / θc=0.999`. That is *strictly
better than the current verifier's 0%* but well short of the §7 #26
"settled, 28/28" claim, which was a synthetic-set artifact. The
remaining gap is **architectural, not threshold-tuning**: the
candidate-clause selector (top-k by token overlap) misses the
contradicting clause for the majority of real-haystack recombinations.
The Phase-3 lever that lifts the ceiling is **a semantic candidate
selector** (e.g. vec-driven, sibling of #000050/#000051's hybrid
retrieval — embedding similarity finds *meaning-similar* clauses, not
just *token-similar*); a verifier-matched-clause hook *also* doesn't
help recall directly (for a false claim the verifier may not match
cleanly) — but a better selector does. Caveats: still SHADOW; the 25
real-context fixtures are hand-built (representative but not
exhaustive); runtime promotion is fox+dav1d-decides. Manifest active
defaults stay k=6/θc=0.5; `recommended_operating_point` updated to
`bart-large-mnli / k=12 / max / θc=0.999` (the fp=0 leader on real
haystacks), with the synthetic-set deberta-large config kept as a
documented alternate.

*Meta-lesson, instance six (and the one that closes the loop):*
the discipline isn't only "don't trust small *FP* denominators" — it
is **"don't trust *any* contrived dataset on *either* axis until it
has been re-confirmed on the real pipeline shape."** Synthetic eval
predicts neither real precision (§7 #18→#20) nor real recall (§7
#26→#27). The 28 synthetic recombinations had short 1–3-sentence
sources where every clause was a candidate; the 25 real-context
recombinations have 30 KB sources where the contradicting clause is
buried — those are *different* problems, and only the second one is
the production problem. The 4090 made this measurable in ~4 min
(7 models × full grid × 833 records); without it we would have
shipped a 24%-recall config under a "1.000" label.

---

**§7 #28 (2026-05-19) — tinygrad as a third NLI inference backend;
deterministic engine-agreement A/B (instrument built in-repo; numbers
pending a producer-box run).** Context: the Sherief/tiny-corp thread
(ONNX Runtime's ~2,500-line many-EP build vs tinygrad's minimalist
ONNX frontend that reportedly passes more of the official ONNX test
suite than ORT itself) raised a fox question — can the shadow-NLI
checkpoint run under tinygrad's ONNX frontend instead of / alongside
`optimum.onnxruntime`? This is a legitimate §7 sub-item, not a new
ticket: it is the *same* speedup/backend axis as §7 #22 (the
ONNX-int8 export), one more interchangeable accelerator below the
cage. It also makes the abstract immunity argument concrete — see the
new `docs/onnx-vendor-capture-immunity.md` (the engine is never a
trust dependency; this bench is the demonstration, not a migration).

**The gate is agreement, not speed.** The primary acceptance metric
is **per-pair logit/probability agreement with the `transformers`+
`torch` reference within tolerance** over the same pinned checkpoint
and the same input pairs — *then* p50/p95 latency as the secondary
axis. Rationale, in the exact shape of this ticket's standing lesson:
a faster engine that produces *different* `(p_entail, p_neutral,
p_contra)` is not a faster shadow — it is a *different* shadow signal.
Under the deliberately-unwired Phase-3 demotion hook that would mean a
different `nli_policy_hash` folding into `governance_policy_hash` (§7
#2) and a silently different demotion surface. So: engine-swap is only
admissible if it is *numerically* the same model, measured, not
assumed. "Honest verdicts beat optimistic ones" applies to the
instrument too.

**Honest risk, stated up front (don't trust the contrived assumption
until confirmed on the real shape).** The current
`recommended_operating_point` is `bart-large-mnli` (§7 #27); the
synthetic alternate is `deberta-large-mnli`. tinygrad's ONNX frontend
op-coverage for *these large MNLI cross-encoders* (Erf/GELU, fused
attention, `GatherElements`, dynamic shapes) is **unverified** — it
is an empirical question with a real chance of "frontend rejects the
graph" or "loads but diverges." That outcome is itself a valid,
publishable result (it sharpens the immunity doc: torch is the
always-works floor, ORT the proven accelerator, tinygrad a *candidate*
whose coverage is checkpoint-specific). No assumption of success is
baked into the plan.

**What landed in this commit (instrument before experiment):**
(1) a third backend branch in `arborist/qa/nli/shadow.py`,
`backend == "tinygrad"`, parallel to the onnx/torch branches —
optional `import`, graceful `available=False` on any failure,
SHADOW-only, never an `audit_mode` input, GPU-producer-box only (the
#000051 "heavy passes never in arborist's python+sqlite3 core"
pattern; arborist core stays python3.12+venv+sqlite3); selectable via
`ARBORIST_NLI_BACKEND`. (2) A deterministic A/B instrument
`bench/scripts/nli_backend_ab.py` + `make bench-nli-backends`: loads
the pinned checkpoint under each *available* backend, runs an
identical fixed pair set, reports max/mean abs probability delta vs
the torch reference and p50/p95 latency per backend — no LLM, no
verifier, no n=3 noise, no 5pp floor (the `recall_at_k` deterministic-
instrument discipline). **Pending:** the numbers. tinygrad is not in
arborist's deps and the run belongs on the GPU producer box, not the
core dev box — so this is fox/producer-box-decides to (a) install
tinygrad there and (b) `make bench-nli-backends`. SHADOW throughout;
manifest active defaults and the proof path are byte-unchanged by
construction (the immunity property, exercised).

*First instrument run (2026-05-19, CPU dev box, tinygrad absent — a
smoke of the instrument, not the experiment):* the A/B already
produced a real finding before tinygrad is even in scope. The
**currently-deployed §7 #22 int8 ONNX export diverges from the torch
reference by Δmax ≈ 0.42 per-class probability** on the fixed 24-pair
probe (24 pairs × 5 repeats; mean abs delta smaller but the max is
large). This quantifies, with a deterministic instrument, the prior
hand-wave "int8-ONNX costs ≈1 catch": int8 is a *lossy-by-
construction different numerical model*, and the A/B honestly reports
it as `DIVERGES` at a tight (2e-3) faithfulness tolerance. This is
**not a defect** — it is the immunity property made measurable: even
the shipped accelerator is a numerical surrogate, which is safe only
because it is a shadow that never reaches the proof path
(`docs/onnx-vendor-capture-immunity.md`). It does sharpen the eventual
Phase-3 question: if NLI is ever wired to demote, the engine+
quantization must be pinned into `nli_policy_hash` (an int8 export and
its fp32 source are *different policies*, and this instrument now
proves it numerically). Instrument-honesty caveat encoded: when
`ARBORIST_NLI_BACKEND=tinygrad` is requested but the frontend cannot
load the checkpoint, ShadowNLI correctly degrades to the proven path,
but the A/B reports **tinygrad-unavailable**, never relabels the
fallback's numbers as tinygrad's. Pending unchanged: real tinygrad
numbers need the producer box.
