diff --git a/docs/tickets/ticket-000049-attribution-aware-grounding-check.md b/docs/tickets/ticket-000049-attribution-aware-grounding-check.md index b118ba9..cc11dde 100644 --- a/docs/tickets/ticket-000049-attribution-aware-grounding-check.md +++ b/docs/tickets/ticket-000049-attribution-aware-grounding-check.md @@ -112,13 +112,19 @@ with no shared parameters), it is an ML model whose output influences `audit_mode`, which the "no LLM-as-judge" rule was written to keep out; settling whether a small fixed NLI model is allowed is the *first* decision this ticket needs. (b) a new optional dependency -(`transformers` + a checkpoint) — keep it behind a `[nli]` extra so a -fresh checkout still needs only python+venv+sqlite3, and gate it -behind `entity_policy`-style policy so it's off-by-default until -benched. (c) non-determinism risk — pin the checkpoint + run on CPU -with a fixed dtype; entailment classifiers are deterministic given -that, but it's a thing to verify and pin. (d) latency — only run it -on the small subset the lexical path can't resolve. +(`transformers` / `onnxruntime` + a checkpoint) — keep it behind a +`[nli]` extra so a fresh checkout still needs only python+venv+sqlite3, +gate it behind `entity_policy`-style policy so it's off-by-default +until benched, and **don't put the weights in the repo** — see §2.4 +for the checkpoint provenance & distribution (a tiny manifest in the +repo, the weights fetched on demand into `~/.arborist/models/` from +an origin URL or, once mesh-blob-sync exists, a peer; hash-pinned). +(c) non-determinism risk — pin the checkpoint + run on CPU with a +fixed dtype, and fold `nli_model_version` into `governance_policy_hash` +(§2.4 #4) so a swap stales the cache; entailment classifiers are +deterministic given a pinned checkpoint + CPU + fixed dtype, but it's +a thing to verify and pin. (d) latency — only run it on the small +subset the lexical path can't resolve. ### 2.3 — do nothing; the 2 fixtures stand as a marker @@ -135,6 +141,94 @@ recombination-over-grounds in real answers either, because the paraphrase path is a last resort that rarely fires on real Wikipedia QA.) +### 2.4 — where would the NLI weights live? (Option 2.2 detail — provenance & distribution) + +If Option 2.2 is ever chosen, the *checkpoint* has to come from +somewhere and reach every deployment that enables the policy. Three +shapes, two of them dead ends: + +- **Trained just-in-time, per deployment — ❌.** Training a + DeBERTa-/RoBERTa-NLI-class classifier is GPU-hours over MNLI / SNLI + / ANLI; it is *not* a per-deployment step. Worse, two training runs + give different weights → different `audit_mode` on the same input → + this breaks the verifier's reproducibility invariant + (`audit_mode` must be a deterministic function of the inputs + + policy). "Training" is a *one-time act by someone*; what gets + distributed is the resulting fixed blob, not the recipe. So this + isn't really an option — it collapses into one of the other two + for "where does the blob come from", with the worst reproducibility + story. +- **Embedded in the git repo as data — ❌.** A few-hundred-MB binary + in `.git` blows the standing invariant that a fresh `git clone` + needs only `python3.12 + venv + sqlite3` (now it's "+ ~400 MB of + model weights for an off-by-default feature most deployments never + enable"). Git LFS would add an LFS dependency and still bloats the + clone. Wrong fit — the repo is source + tiny fixtures + docs, not a + model registry. (Contrast `bench/fixtures/` JSONL packs: a few KB + each, the *data* IS the test; a model checkpoint is neither small + nor a test.) +- **Obtained once, content-addressed, fetched on demand, mesh- + distributable — ✓.** The right shape, and the one consistent with + arborist's existing patterns: + 1. **Source.** Either a published off-the-shelf NLI checkpoint (a + known DeBERTa-/RoBERTa-NLI fine-tune with a redistribution-OK + license — most are MIT/Apache; record it) **or** a one-time + fox/blackops-trained-and-published one if arborist wants its + own. Either way the result is a single fixed blob with a + sha256. + 2. **Manifest in the repo (tiny).** `arborist/qa/nli/manifest.json` + (say) records `{checkpoint_sha256, source_url, license, + nli_model_version, onnx: true|false}`. The repo carries the + *pointer + hash + license*, not the weights — exactly the + pattern of `bench/fixtures/textbooks/manifest-v1.jsonl` (textbook + URLs + license tokens, not the textbook text) and the `[vec]` + extra (#000039 — `fastembed`'s `bge-small` checkpoint fetched + on first use, never committed). Precedent: "an ML model behind a + `[…]` extra, fetched not committed" is already how the vec + backend ships. + 3. **Fetched on first use into `~/.arborist/models/nli//`.** + Operator-state lives under `~/.arborist/` (like `shards/`, + `crawl/`, `textbooks.db`), not the repo. A `make fetch-nli` (or + auto-fetch the first time a `--enable-nli` query runs) downloads + it and **verifies the sha256 against the manifest, fail-closed + on mismatch** — the "artifact over instruction" / "md5sum + downloads" discipline applies to the checkpoint exactly as it + does to a Wikipedia dump. The `[nli]` extra pulls the runtime + (`onnxruntime` if the manifest's checkpoint is ONNX-exported — + no torch, smaller; else `transformers` + a CPU torch). + 4. **`nli_model_version` folds into `governance_policy_hash`.** A + checkpoint swap stales prior `providence_cache` records on + lookup, exactly like bumping `chunking_version` / + `canonicalization_version`. So the verifier stays reproducible: + `audit_mode` is a deterministic function of (the answer, the + source, the policy, **and the pinned checkpoint hash**) — run + it twice with the same hash on the same shard → identical + verdict. The audit chain can record an `nli_checkpoint_loaded` + event carrying the hash so provenance is on the chain. + 5. **Mesh-distributable as a content-addressed blob (the gossip + path).** Once arborist's mesh (`arborist/mesh/`, `docs/mesh.md` + — today: audit-event / group-key sync) grows blob-sync, a + deployment with peers pulls the checkpoint *from a peer* + (AXFR-style — same shape as ClouDNS slaves pulling a zone from + the PowerDNS master, or a fresh shard rehydrating from a + `snapshot.db`) instead of the origin URL. Content-addressing + makes peer-pull and origin-pull interchangeable: verify the + sha256 either way. The mesh is an **optimization** (faster, no + external dependency at runtime, works on an air-gapped + permacomputer cluster once one node has it), **not the canonical + source** — the manifest's hash is. Cold start (a fresh + deployment with no peers, no local copy) falls back to the + origin fetch in step 3. + +**Bottom line:** *not* JIT-trained (non-deterministic, GPU-hours, not +a per-deployment step), *not* repo-embedded weights (blows the lean- +checkout invariant); **a one-time-obtained, content-addressed, hash- +pinned checkpoint — the repo carries a tiny manifest, the weights +live under `~/.arborist/models/`, fetched on demand (origin URL or, +once mesh-blob-sync exists, a peer), the hash folds into +`governance_policy_hash`.** All of which presupposes the §2.2/§5 +discipline call came back `yes`; if `no`, none of this is built. + --- ## 3. Recommendation @@ -191,7 +285,11 @@ purpose-built NLI/entailment model influence `audit_mode`? — `yes` → Option 2.2 becomes viable when triggered (§3 #4); `no` → Option 2.3 stands and the 2 recombination fixtures are permanent boundary markers. The second decision (only if `yes` and triggered) is the -implementation: a `[nli]` extra + a pinned CPU checkpoint + policy- +implementation, whose *shape* is already settled in §2.4: a `[nli]` +extra + a *content-addressed, hash-pinned checkpoint* (manifest in +the repo, weights under `~/.arborist/models/`, fetched on demand from +an origin URL or — once arborist's mesh grows blob-sync — a peer; +`nli_model_version` folds into `governance_policy_hash`) + policy- gated off-by-default + the `make bench-qa` before/after gate. Until then: nothing lands; `falsification-hard` stays 10/12. @@ -207,6 +305,17 @@ then: nothing lands; `falsification-hard` stays 10/12. Einstein are the 2 residual recombination fixtures this ticket would close). - `bench/fixtures/5f/falsification-hard-v1.jsonl` — the 2 fixtures. +- **Provenance/distribution precedents for §2.4:** `bench/fixtures/textbooks/manifest-v1.jsonl` + (a repo manifest of *pointers + license tokens*, not the texts); + `arborist/search/vec.py` + the `[vec]` extra (#000039 — an ML model, + `fastembed`'s `bge-small` checkpoint, fetched on first use, never + committed); `~/.arborist/{shards,crawl}` + `textbooks.db` + (operator-state lives under `~/.arborist/`, not the repo); + `docs/mesh.md` + `arborist/mesh/` (the mesh — today audit-event / + group-key sync; a content-addressed blob-sync extension is what + the §2.4 "gossip/peer-pull" path needs); `arborist/snapshot.py` + (snapshot rehydrate — the same "pull a content-addressed artifact" + shape, peer or origin). - `arborist/qa/verify.py` — the layered verifier; a contradiction check would attach to the paraphrase path + the entity path's weakest slot (the same places #000046 / #000048 step 2.1's gates