#000049: expand — where would the NLI weights live? (§2.4 provenance & distribution)
fox asked: JIT-trained, gossip-pulled, or repo-embedded? Added §2.4 — three shapes, two dead ends: - JIT-trained per deployment: ❌ — GPU-hours over MNLI/SNLI/ANLI, not a per-deployment step; non-deterministic across runs → different audit_mode → breaks the verifier's reproducibility invariant. "Training" is a one-time act by someone; the distributed thing is the resulting fixed blob, not the recipe. - Embedded in the git repo as data: ❌ — a few-hundred-MB binary in .git blows the standing "fresh clone = python3.12 + venv + sqlite3" invariant; Git LFS adds a dep and still bloats; the repo is source + tiny fixtures + docs, not a model registry. - One-time obtained, content-addressed, fetched on demand, mesh- distributable: ✓ — the repo carries a tiny manifest (arborist/qa/nli/manifest.json: {checkpoint_sha256, source_url, license, nli_model_version}); the weights live under ~/.arborist/models/nli/<hash>/, fetched on first use (make fetch-nli or auto-fetch on first --enable-nli query) with the sha256 verified fail-closed against the manifest; nli_model_version folds into governance_policy_hash (a swap stales the cache, like chunking_version / canonicalization_version), keeping audit_mode a deterministic function of (answer, source, policy, pinned hash); and once arborist's mesh grows blob-sync, a deployment with peers pulls the checkpoint from a peer (AXFR-style, like ClouDNS slaves pulling a zone, like a shard rehydrating from snapshot.db) instead of the origin URL — content-addressing makes peer-pull and origin-pull interchangeable, the mesh is the optimization not the canonical source, cold start falls back to origin. Source: a published off-the-shelf NLI checkpoint (MIT/Apache, recorded) or a one-time fox/blackops-trained-and-published one. Bottom line: not JIT-trained, not repo-embedded weights — a content-addressed hash-pinned checkpoint, manifest in the repo, weights under ~/.arborist/, fetched on demand (origin or peer), hash in governance_policy_hash. Precedents cited: the textbook manifest (pointers + licenses not texts), the [vec] extra (fastembed's bge-small fetched not committed — #000039), ~/.arborist/ operator- state, docs/mesh.md + snapshot rehydrate. All presupposes the §2.2/§5 discipline call came back "yes" (is a fixed NLI model allowed in the audit_mode path?); if "no", none of it builds. Also wired §2.2 con (b)/(c), §3, §5, §6 to point at §2.4. Doc-only.
This commit is contained in:
parent
0d55fea3a6
commit
e391f1f01c
1 changed files with 117 additions and 8 deletions
|
|
@ -112,13 +112,19 @@ with no shared parameters), it is an ML model whose output influences
|
|||
`audit_mode`, which the "no LLM-as-judge" rule was written to keep
|
||||
out; settling whether a small fixed NLI model is allowed is the
|
||||
*first* decision this ticket needs. (b) a new optional dependency
|
||||
(`transformers` + a checkpoint) — keep it behind a `[nli]` extra so a
|
||||
fresh checkout still needs only python+venv+sqlite3, and gate it
|
||||
behind `entity_policy`-style policy so it's off-by-default until
|
||||
benched. (c) non-determinism risk — pin the checkpoint + run on CPU
|
||||
with a fixed dtype; entailment classifiers are deterministic given
|
||||
that, but it's a thing to verify and pin. (d) latency — only run it
|
||||
on the small subset the lexical path can't resolve.
|
||||
(`transformers` / `onnxruntime` + a checkpoint) — keep it behind a
|
||||
`[nli]` extra so a fresh checkout still needs only python+venv+sqlite3,
|
||||
gate it behind `entity_policy`-style policy so it's off-by-default
|
||||
until benched, and **don't put the weights in the repo** — see §2.4
|
||||
for the checkpoint provenance & distribution (a tiny manifest in the
|
||||
repo, the weights fetched on demand into `~/.arborist/models/` from
|
||||
an origin URL or, once mesh-blob-sync exists, a peer; hash-pinned).
|
||||
(c) non-determinism risk — pin the checkpoint + run on CPU with a
|
||||
fixed dtype, and fold `nli_model_version` into `governance_policy_hash`
|
||||
(§2.4 #4) so a swap stales the cache; entailment classifiers are
|
||||
deterministic given a pinned checkpoint + CPU + fixed dtype, but it's
|
||||
a thing to verify and pin. (d) latency — only run it on the small
|
||||
subset the lexical path can't resolve.
|
||||
|
||||
### 2.3 — do nothing; the 2 fixtures stand as a marker
|
||||
|
||||
|
|
@ -135,6 +141,94 @@ recombination-over-grounds in real answers either, because the
|
|||
paraphrase path is a last resort that rarely fires on real Wikipedia
|
||||
QA.)
|
||||
|
||||
### 2.4 — where would the NLI weights live? (Option 2.2 detail — provenance & distribution)
|
||||
|
||||
If Option 2.2 is ever chosen, the *checkpoint* has to come from
|
||||
somewhere and reach every deployment that enables the policy. Three
|
||||
shapes, two of them dead ends:
|
||||
|
||||
- **Trained just-in-time, per deployment — ❌.** Training a
|
||||
DeBERTa-/RoBERTa-NLI-class classifier is GPU-hours over MNLI / SNLI
|
||||
/ ANLI; it is *not* a per-deployment step. Worse, two training runs
|
||||
give different weights → different `audit_mode` on the same input →
|
||||
this breaks the verifier's reproducibility invariant
|
||||
(`audit_mode` must be a deterministic function of the inputs +
|
||||
policy). "Training" is a *one-time act by someone*; what gets
|
||||
distributed is the resulting fixed blob, not the recipe. So this
|
||||
isn't really an option — it collapses into one of the other two
|
||||
for "where does the blob come from", with the worst reproducibility
|
||||
story.
|
||||
- **Embedded in the git repo as data — ❌.** A few-hundred-MB binary
|
||||
in `.git` blows the standing invariant that a fresh `git clone`
|
||||
needs only `python3.12 + venv + sqlite3` (now it's "+ ~400 MB of
|
||||
model weights for an off-by-default feature most deployments never
|
||||
enable"). Git LFS would add an LFS dependency and still bloats the
|
||||
clone. Wrong fit — the repo is source + tiny fixtures + docs, not a
|
||||
model registry. (Contrast `bench/fixtures/` JSONL packs: a few KB
|
||||
each, the *data* IS the test; a model checkpoint is neither small
|
||||
nor a test.)
|
||||
- **Obtained once, content-addressed, fetched on demand, mesh-
|
||||
distributable — ✓.** The right shape, and the one consistent with
|
||||
arborist's existing patterns:
|
||||
1. **Source.** Either a published off-the-shelf NLI checkpoint (a
|
||||
known DeBERTa-/RoBERTa-NLI fine-tune with a redistribution-OK
|
||||
license — most are MIT/Apache; record it) **or** a one-time
|
||||
fox/blackops-trained-and-published one if arborist wants its
|
||||
own. Either way the result is a single fixed blob with a
|
||||
sha256.
|
||||
2. **Manifest in the repo (tiny).** `arborist/qa/nli/manifest.json`
|
||||
(say) records `{checkpoint_sha256, source_url, license,
|
||||
nli_model_version, onnx: true|false}`. The repo carries the
|
||||
*pointer + hash + license*, not the weights — exactly the
|
||||
pattern of `bench/fixtures/textbooks/manifest-v1.jsonl` (textbook
|
||||
URLs + license tokens, not the textbook text) and the `[vec]`
|
||||
extra (#000039 — `fastembed`'s `bge-small` checkpoint fetched
|
||||
on first use, never committed). Precedent: "an ML model behind a
|
||||
`[…]` extra, fetched not committed" is already how the vec
|
||||
backend ships.
|
||||
3. **Fetched on first use into `~/.arborist/models/nli/<hash>/`.**
|
||||
Operator-state lives under `~/.arborist/` (like `shards/`,
|
||||
`crawl/`, `textbooks.db`), not the repo. A `make fetch-nli` (or
|
||||
auto-fetch the first time a `--enable-nli` query runs) downloads
|
||||
it and **verifies the sha256 against the manifest, fail-closed
|
||||
on mismatch** — the "artifact over instruction" / "md5sum
|
||||
downloads" discipline applies to the checkpoint exactly as it
|
||||
does to a Wikipedia dump. The `[nli]` extra pulls the runtime
|
||||
(`onnxruntime` if the manifest's checkpoint is ONNX-exported —
|
||||
no torch, smaller; else `transformers` + a CPU torch).
|
||||
4. **`nli_model_version` folds into `governance_policy_hash`.** A
|
||||
checkpoint swap stales prior `providence_cache` records on
|
||||
lookup, exactly like bumping `chunking_version` /
|
||||
`canonicalization_version`. So the verifier stays reproducible:
|
||||
`audit_mode` is a deterministic function of (the answer, the
|
||||
source, the policy, **and the pinned checkpoint hash**) — run
|
||||
it twice with the same hash on the same shard → identical
|
||||
verdict. The audit chain can record an `nli_checkpoint_loaded`
|
||||
event carrying the hash so provenance is on the chain.
|
||||
5. **Mesh-distributable as a content-addressed blob (the gossip
|
||||
path).** Once arborist's mesh (`arborist/mesh/`, `docs/mesh.md`
|
||||
— today: audit-event / group-key sync) grows blob-sync, a
|
||||
deployment with peers pulls the checkpoint *from a peer*
|
||||
(AXFR-style — same shape as ClouDNS slaves pulling a zone from
|
||||
the PowerDNS master, or a fresh shard rehydrating from a
|
||||
`snapshot.db`) instead of the origin URL. Content-addressing
|
||||
makes peer-pull and origin-pull interchangeable: verify the
|
||||
sha256 either way. The mesh is an **optimization** (faster, no
|
||||
external dependency at runtime, works on an air-gapped
|
||||
permacomputer cluster once one node has it), **not the canonical
|
||||
source** — the manifest's hash is. Cold start (a fresh
|
||||
deployment with no peers, no local copy) falls back to the
|
||||
origin fetch in step 3.
|
||||
|
||||
**Bottom line:** *not* JIT-trained (non-deterministic, GPU-hours, not
|
||||
a per-deployment step), *not* repo-embedded weights (blows the lean-
|
||||
checkout invariant); **a one-time-obtained, content-addressed, hash-
|
||||
pinned checkpoint — the repo carries a tiny manifest, the weights
|
||||
live under `~/.arborist/models/`, fetched on demand (origin URL or,
|
||||
once mesh-blob-sync exists, a peer), the hash folds into
|
||||
`governance_policy_hash`.** All of which presupposes the §2.2/§5
|
||||
discipline call came back `yes`; if `no`, none of this is built.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
|
|
@ -191,7 +285,11 @@ purpose-built NLI/entailment model influence `audit_mode`? — `yes` →
|
|||
Option 2.2 becomes viable when triggered (§3 #4); `no` → Option 2.3
|
||||
stands and the 2 recombination fixtures are permanent boundary
|
||||
markers. The second decision (only if `yes` and triggered) is the
|
||||
implementation: a `[nli]` extra + a pinned CPU checkpoint + policy-
|
||||
implementation, whose *shape* is already settled in §2.4: a `[nli]`
|
||||
extra + a *content-addressed, hash-pinned checkpoint* (manifest in
|
||||
the repo, weights under `~/.arborist/models/`, fetched on demand from
|
||||
an origin URL or — once arborist's mesh grows blob-sync — a peer;
|
||||
`nli_model_version` folds into `governance_policy_hash`) + policy-
|
||||
gated off-by-default + the `make bench-qa` before/after gate. Until
|
||||
then: nothing lands; `falsification-hard` stays 10/12.
|
||||
|
||||
|
|
@ -207,6 +305,17 @@ then: nothing lands; `falsification-hard` stays 10/12.
|
|||
Einstein are the 2 residual recombination fixtures this ticket
|
||||
would close).
|
||||
- `bench/fixtures/5f/falsification-hard-v1.jsonl` — the 2 fixtures.
|
||||
- **Provenance/distribution precedents for §2.4:** `bench/fixtures/textbooks/manifest-v1.jsonl`
|
||||
(a repo manifest of *pointers + license tokens*, not the texts);
|
||||
`arborist/search/vec.py` + the `[vec]` extra (#000039 — an ML model,
|
||||
`fastembed`'s `bge-small` checkpoint, fetched on first use, never
|
||||
committed); `~/.arborist/{shards,crawl}` + `textbooks.db`
|
||||
(operator-state lives under `~/.arborist/`, not the repo);
|
||||
`docs/mesh.md` + `arborist/mesh/` (the mesh — today audit-event /
|
||||
group-key sync; a content-addressed blob-sync extension is what
|
||||
the §2.4 "gossip/peer-pull" path needs); `arborist/snapshot.py`
|
||||
(snapshot rehydrate — the same "pull a content-addressed artifact"
|
||||
shape, peer or origin).
|
||||
- `arborist/qa/verify.py` — the layered verifier; a contradiction
|
||||
check would attach to the paraphrase path + the entity path's
|
||||
weakest slot (the same places #000046 / #000048 step 2.1's gates
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue