#000049: incorporate de novo review (2026-05-13) — shadow-path approved, governance-hash correction, demotion-only rule, NLI manifest fields, nli_pair@v1, bench gate, roadmap
This commit is contained in:
parent
7ff37f8d70
commit
57bb26ba4e
2 changed files with 253 additions and 20 deletions
|
|
@ -103,7 +103,7 @@ Newest first. Update on every open/close.
|
|||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — |
|
||||
| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — |
|
||||
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · awaiting go/no-go · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). First decision = the discipline question: may a small fixed purpose-built NLI/entailment *model* influence `audit_mode`? (vs the "no LLM-as-judge" rule). Recommends Option 2.3 (do nothing — the 2 fixtures are a boundary marker) until real-traffic recombination-over-grounds show up, then Option 2.2 (`[nli]` extra, policy-gated, off-by-default, bench-gated) *if* fox rules a fixed NLI model is acceptable. #000048 follow-up | 2026-05-12 | — |
|
||||
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. #000048 follow-up | 2026-05-12 | — |
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, now tracked as **#000049**, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 closed | 2026-05-11 | — |
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
# Ticket #000049 — Attribution-aware grounding check (the recombination boundary)
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Status:** open · boundary accepted · production no-go · shadow-path
|
||||
approved (de novo review 2026-05-13 — see §7)
|
||||
**Opened:** 2026-05-12
|
||||
**Scope:** Decide whether — and if so how — to add a verifier check
|
||||
that catches a *recombination*: a claim whose content tokens are all
|
||||
|
|
@ -21,8 +22,16 @@ verifier" rule still holds** — a small purpose-built NLI/entailment
|
|||
*model* is a discipline question this ticket has to settle explicitly
|
||||
(it is not the same as "ask the answering LLM to grade itself", but
|
||||
it is an ML model in the proof-adjacent path); a dependency-parse
|
||||
heuristic sidesteps that but is brittle. No schema / governance-hash /
|
||||
canonicalization-version change.
|
||||
heuristic sidesteps that but is brittle. **Governance-hash rule
|
||||
(corrected by the 2026-05-13 review — §7 #2):** *while this stays
|
||||
doc-only / Option 2.3 / shadow-only* there is no schema /
|
||||
governance-hash / canonicalization-version change; but **if an NLI
|
||||
output ever influences `audit_mode`, then `nli_model_version` + the
|
||||
checkpoint/tokenizer/threshold/pair-serialization policy MUST fold
|
||||
into `governance_policy_hash`** — otherwise two deployments could
|
||||
return different `audit_mode` for the same (answer, source) pair while
|
||||
sharing cache identity, which breaks the Merkle/SQD reproducibility
|
||||
discipline.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -233,9 +242,20 @@ discipline call came back `yes`; if `no`, none of this is built.
|
|||
|
||||
## 3. Recommendation
|
||||
|
||||
**Option 2.3 (do nothing) for now — but settle the Option-2.2
|
||||
discipline question if/when recombination over-grounds show up on
|
||||
real traffic.** Rationale:
|
||||
**Updated by the 2026-05-13 de novo review (§7):** production stays
|
||||
**Option 2.3** today; **additionally, an Option-2.2 shadow path is
|
||||
approved** — a fixed NLI model may be evaluated as a *demotion-only
|
||||
contradiction veto* in bench/shadow mode, but it must not affect
|
||||
`audit_mode` until it passes the deterministic bench gate (§7 #6) and
|
||||
its checkpoint/tokenizer/threshold/pair-serialization policy folds
|
||||
into `governance_policy_hash`. NLI never *promotes* an answer to
|
||||
grounded; it can only demote a *weak* paraphrase/entity outcome when
|
||||
recombination risk is detected and no stronger deterministic warrant
|
||||
exists. Heuristic demotion (Option 2.1) is **not** recommended even
|
||||
then — only as a deterministic *recombination-risk trigger* that
|
||||
decides when to call shadow NLI, never as the contradiction decider.
|
||||
|
||||
Original rationale (still holds for the production no-go):
|
||||
|
||||
1. The lexical verifier is honest about its boundary (the 2 fixtures
|
||||
document it); there's no observed real-traffic harm.
|
||||
|
|
@ -279,19 +299,22 @@ real traffic.** Rationale:
|
|||
|
||||
## 5. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Doc-only. The first thing fox decides
|
||||
is the **discipline question** in §2.2's con (a): may a small, fixed,
|
||||
purpose-built NLI/entailment model influence `audit_mode`? — `yes` →
|
||||
Option 2.2 becomes viable when triggered (§3 #4); `no` → Option 2.3
|
||||
stands and the 2 recombination fixtures are permanent boundary
|
||||
markers. The second decision (only if `yes` and triggered) is the
|
||||
implementation, whose *shape* is already settled in §2.4: a `[nli]`
|
||||
extra + a *content-addressed, hash-pinned checkpoint* (manifest in
|
||||
the repo, weights under `~/.arborist/models/`, fetched on demand from
|
||||
an origin URL or — once arborist's mesh grows blob-sync — a peer;
|
||||
`nli_model_version` folds into `governance_policy_hash`) + policy-
|
||||
gated off-by-default + the `make bench-qa` before/after gate. Until
|
||||
then: nothing lands; `falsification-hard` stays 10/12.
|
||||
**Open · boundary accepted · production no-go · shadow-path approved**
|
||||
(de novo review 2026-05-13, §7). The diagnosis (recombination is an
|
||||
*attribution* failure, not lexical) is confirmed. Production verifier
|
||||
is unchanged; the 2 recombination fixtures (`falsification-hard`
|
||||
stays 10/12) are accepted as honest boundary markers. The **discipline
|
||||
question** in §2.2's con (a) — may a small, fixed, purpose-built
|
||||
NLI/entailment model influence `audit_mode`? — is answered **"not by
|
||||
default; yes only as an opt-in, hash-pinned, governance-hashed,
|
||||
demotion-only contradiction veto after shadow-mode evidence"** (§7
|
||||
#19). Next concrete steps: the §7 implementation roadmap — Phase 0
|
||||
(this amendment), Phase 1 (shadow design: manifest format,
|
||||
fetch/verify command, `nli_pair@v1` canonicalization, recombination-
|
||||
risk trigger), Phase 2 (optional bench-only shadow implementation,
|
||||
`[nli]` extra, `make fetch-nli`), Phase 3 (demotion-only runtime,
|
||||
gated), Phase 4 (mesh blob sync). Nothing in Phases 2–4 lands without
|
||||
the §7 #6 bench gate passing.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -324,3 +347,213 @@ then: nothing lands; `falsification-hard` stays 10/12.
|
|||
honest verdicts over optimistic) every step here honors.
|
||||
- `docs/seven-point-program.md` D1 ("no LLM in the verifier") — the
|
||||
rule §2.2's con (a) tests against.
|
||||
|
||||
---
|
||||
|
||||
## 7. De novo review — incorporated (2026-05-13)
|
||||
|
||||
An independent de novo review (`~/Downloads/response_ticket-000049-*.txt`,
|
||||
2026-05-13 Asia/Kuala_Lumpur) confirmed the diagnosis and the §2.4
|
||||
weights answer, and added the corrections below. They are **additive
|
||||
and no-regression** — production verifier unchanged.
|
||||
|
||||
**1. Core diagnosis confirmed.** The 2 residuals are *attribution
|
||||
recombination* failures (every token in the source, but the answer
|
||||
attaches a subject to the wrong predicate/object). Token coverage,
|
||||
clause containment, and bigram thresholds cannot separate a
|
||||
recombination from a legitimate cross-sentence summary in the safe
|
||||
direction — the discriminating fact is attribution, which needs a
|
||||
dependency structure or an entailment judgment. Not a failure of
|
||||
#000046/#000048; the next boundary after lexical hardening.
|
||||
|
||||
**2. Governance-hash correction (most important).** The §-header
|
||||
"no governance-hash change" constraint is valid *only while this is
|
||||
doc-only / Option 2.3 / shadow-only*. If a fixed NLI checkpoint can
|
||||
demote a claim, then `audit_mode` becomes a function of {answer span,
|
||||
source span/clause, verifier policy, NLI checkpoint hash, NLI runtime
|
||||
policy, NLI tokenizer policy, NLI threshold policy} — so
|
||||
`nli_model_version` + that policy bundle (or an explicit
|
||||
`nli_policy_hash`) **must** fold into `governance_policy_hash`.
|
||||
Otherwise two deployments give different `audit_mode` for the same
|
||||
(answer, source) pair while sharing cache identity → violates the
|
||||
Merkle/SQD reproducibility discipline. Corrected three-state rule:
|
||||
doc-only/2.3 → no hash change; shadow mode with no `audit_mode`
|
||||
effect → no cache-identity change, but record shadow metadata
|
||||
separately; NLI affecting `audit_mode` → `governance_policy_hash`
|
||||
MUST include NLI model + runtime + tokenizer + threshold + pair
|
||||
policy.
|
||||
|
||||
**3. Discipline call.** A fixed NLI classifier is *not* "the
|
||||
answering LLM grading itself" (no shared parameters, purpose-built,
|
||||
content-addressed) — so it isn't the exact failure mode D1 names —
|
||||
but it is still an ML model in the proof-adjacent verifier path, so
|
||||
it violates the *spirit* of a deterministic symbolic/lexical verifier
|
||||
unless tightly bounded. Permit it only under a new category
|
||||
**`MODEL_ASSISTED_DEMOTION`**, never `MODEL_ASSISTED_PROMOTION`: NLI
|
||||
may say "this weakly grounded claim appears contradicted; demote it",
|
||||
never "this unsupported claim is true; promote it to grounded". No
|
||||
model manufactures proof; at most a pinned model adds a contradiction
|
||||
veto to a claim the lexical verifier was already uncertain about.
|
||||
|
||||
**4. Placement.** NLI runs *only* after a lexical path reaches a weak
|
||||
grounding state (paraphrase path; weak entity path; high lexical
|
||||
coverage but multi-clause source; source has multiple candidate
|
||||
subject/predicate pairings; recombination-risk heuristic fires).
|
||||
Never on: direct quote match, exact span match, strong
|
||||
pointer-warranted claim, or any case a deterministic verifier already
|
||||
proves. Layered-first-to-classify invariant preserved; contradiction
|
||||
demotes, never promotes.
|
||||
|
||||
**5. Algorithm (clause-level, not "NLI on whole source").** Coarse
|
||||
`NLI(source_document, answer_claim)` is wrong — a source can hold
|
||||
both a contradictory background statement and a later supporting one.
|
||||
Use:
|
||||
1. segment source into clauses (existing clause/pointer segmenter);
|
||||
2. extract answer claim span;
|
||||
3. detect recombination risk (high content-token coverage; span
|
||||
crosses multiple source clauses; answer subject/object tokens
|
||||
occur in *different* source clauses; weak verifier path only);
|
||||
4. for each candidate source clause, run `NLI(source_clause,
|
||||
answer_claim)`;
|
||||
5. compute `max_entailment_score`, `max_contradiction_score` over
|
||||
clauses;
|
||||
6. demote only if `max_contradiction ≥ θ_c` AND `max_entailment <
|
||||
θ_e` AND current label is weak enough to be vetoed AND no stronger
|
||||
deterministic warrant exists elsewhere;
|
||||
7. else leave the lexical/verifier result unchanged.
|
||||
|
||||
Formal: `Demote(claim) = WeakGrounding(claim) ∧
|
||||
RecombinationRisk(claim, source) ∧ MaxContradiction(source_clauses,
|
||||
claim) ≥ θ_c ∧ MaxEntailment(source_clauses, claim) < θ_e ∧ ¬
|
||||
StrongDeterministicSupport(claim)`.
|
||||
|
||||
**6. Audit label.** Don't silently convert a contradiction into a
|
||||
generic failure — emit an explicit reason:
|
||||
`{audit_mode: "UNSUPPORTED"/"UNGROUNDED", demotion_reason:
|
||||
"NLI_CONTRADICTION_VETO", nli_model_version, nli_checkpoint_sha256,
|
||||
source_clause_id, answer_claim_id, max_contradiction, max_entailment}`.
|
||||
If the schema can't carry this yet → run shadow mode only, don't
|
||||
implement runtime demotion.
|
||||
|
||||
**7. NLI manifest fields (extends §2.4 #2).** `arborist/qa/nli/manifest.json`
|
||||
should carry: `nli_model_version`, `checkpoint_sha256`, `source_url`,
|
||||
`license`, `license_verified_at`, `model_family`, `onnx` (bool),
|
||||
`runtime` (`onnxruntime-cpu`), `runtime_version_range`,
|
||||
`tokenizer_sha256`, `label_map` (`{"0":"entailment","1":"neutral",
|
||||
"2":"contradiction"}` — pin the actual ordering of the chosen
|
||||
checkpoint), `max_length`, `truncation_policy`
|
||||
(`source_clause_then_claim@v1`), `pair_serialization` (`nli_pair@v1`),
|
||||
`thresholds` (`{contradiction_veto: 0.90, entailment_block_veto:
|
||||
0.50}` — illustrative; calibrate). The manifest hash (or an explicit
|
||||
`nli_policy_hash`) folds into `governance_policy_hash` once
|
||||
`audit_mode` is affected.
|
||||
|
||||
**8. `nli_pair@v1` canonicalization (SQD interaction).** The NLI
|
||||
input pair must be canonicalized — whitespace / clause-boundary
|
||||
jitter changes tokenization. `nli_pair@v1` = canonical serialization
|
||||
of {`source_clause_text`, `answer_claim_text`, `source_clause_id`,
|
||||
`answer_claim_id`, `truncation_policy`, `tokenizer_policy`};
|
||||
`nli_pair_hash = H(canonical_json(nli_pair@v1))`, recorded in the
|
||||
run-DAG / audit sidecar.
|
||||
|
||||
**9. Mesh / #000051 relation.** NLI-checkpoint distribution can
|
||||
reuse the vecpack blob-sync *pattern* but needs stronger handling: a
|
||||
vecpack is a soft retrieval sidecar (a wrong blob can't directly
|
||||
touch `audit_mode`); an NLI checkpoint is a proof-adjacent verifier
|
||||
artifact (a wrong blob *can*, if NLI is enabled). So: structural
|
||||
sanity + trust tier is enough for vecpacks; for NLI, **exact sha256
|
||||
match against the manifest is mandatory, fail-closed on mismatch,
|
||||
and `governance_policy_hash` includes the checkpoint identity**. Peer
|
||||
gossip distributes the blob only as transport — origin URL and peer
|
||||
blob are equivalent iff the sha256 matches; mesh sender trust is not
|
||||
sufficient, the manifest hash is canonical.
|
||||
|
||||
**10. Reproducibility KATs.** If NLI ever affects `audit_mode`: fixed
|
||||
checkpoint hash, fixed tokenizer hash, fixed ONNX/runtime backend,
|
||||
CPU inference, fixed dtype, fixed pair serialization, fixed
|
||||
truncation policy, fixed label map, fixed thresholds, behavior KATs,
|
||||
all folded into `governance_policy_hash`. KATs should be *label*-level
|
||||
(`expected_label: "contradiction"` for Mercury/Jupiter;
|
||||
`"entailment_or_not_contradiction"` for the Batman cross-sentence
|
||||
summary), not exact-logit, unless the runtime is proven stable. The
|
||||
load-bearing invariant: *same checkpoint + policy → same
|
||||
demote/no-demote decision*.
|
||||
|
||||
**11. Shadow mode before runtime mode.** Add a middle path between
|
||||
"do nothing" and "runtime NLI": **NLI shadow mode** — optional, off
|
||||
by default, no `audit_mode` effect, no cache effect, writes only
|
||||
bench/shadow telemetry (`{shadow_nli_enabled, would_demote,
|
||||
current_audit_mode, nli_policy_hash, checkpoint_sha256,
|
||||
source_clause_id, answer_claim_id}`). Lets us measure the
|
||||
false-positive rate on real QA *before* the model is allowed to
|
||||
affect `audit_mode`. This is the no-regression path.
|
||||
|
||||
**12. Bench gate (explicit) — before enabling runtime demotion:**
|
||||
(1) `falsification-hard` catches the 2 recombination fixtures; (2)
|
||||
legit cross-sentence summaries — no demotion; (3) paraphrase/entity
|
||||
legit cases — no proof-quality regression; (4) real-QA sample —
|
||||
shadow-mode false-positive rate below threshold; (5) latency — NLI
|
||||
called only on a small unresolved subset; (6) determinism — same run
|
||||
×3 → identical demote/no-demote decisions. Suggested thresholds:
|
||||
closes both known fixtures; 0 regressions on curated legit-summary
|
||||
fixtures; 0 STRICT/EVIDENCE-WARRANTED regressions in n=3 `bench-qa`;
|
||||
NLI invocation rate < 5% of verified answers unless explicitly
|
||||
configured; p95 added latency bounded and reported. Any fail → keep
|
||||
NLI shadow-only.
|
||||
|
||||
**13. CTI / claim-lattice framing.** Represent a recombination as a
|
||||
clause-level attribution failure: claim `LargestPlanet(Mercury)` vs
|
||||
source clauses `LargestPlanet(Jupiter)` + `SmallestPlanet(Mercury)`
|
||||
— lexical coverage true, attribution consistency false → demote weak
|
||||
grounding. A CTI-style trace records `{claim, claim_tuple,
|
||||
source_tuples, recombination_risk, contradiction_veto}`. Even
|
||||
approximate tuple extraction makes this the conceptual target; NLI is
|
||||
the model-assisted approximation to the missing attribution relation.
|
||||
|
||||
**14. Prometheus-Σ framing.** Branch-selection: A = lexical only,
|
||||
recombination stays a documented boundary; B = deterministic
|
||||
heuristic demotion; C = NLI shadow mode only; D = NLI demotion-only
|
||||
runtime; E = full model-based verifier replacement. `utility =
|
||||
recombination-catch-rate − legit-answer-regression − latency-cost −
|
||||
dependency-cost − reproducibility-risk − governance-complexity`.
|
||||
Decision: **A for production, C optionally for measurement, D only
|
||||
on real evidence, B not recommended except as triage, E rejected.**
|
||||
|
||||
**15. SelfModel / MemoryRoot.** The system should record the
|
||||
limitation honestly: "lexical verifier cannot generally detect
|
||||
attribution recombination where all answer tokens appear in the
|
||||
source but source predicates attach to different subjects; known
|
||||
fixtures Mercury/Jupiter, Einstein/Darwin; current production policy
|
||||
= documented boundary, not automatically closed." If shadow mode
|
||||
lands, SelfModel records the NLI checker's availability /
|
||||
shadow-vs-demotion state / checkpoint version / bench status.
|
||||
|
||||
**16. Implementation roadmap.**
|
||||
- *Phase 0 — keep boundary marker (this amendment).* #000049 stays
|
||||
open; `falsification-hard` stays 10/12; documentation explicitly
|
||||
states the residual 2/12 are attribution recombination, not lexical.
|
||||
- *Phase 1 — shadow design (doc-only).* NLI manifest format;
|
||||
fetch/verify command design; `nli_pair@v1` canonicalization design;
|
||||
recombination-risk trigger design. No runtime `audit_mode` effect.
|
||||
- *Phase 2 — optional bench-only shadow implementation.* `[nli]`
|
||||
extra; `make fetch-nli`; model under `~/.arborist/models/nli/<hash>/`;
|
||||
shadow NLI runs only in bench or with `--nli-shadow`; records
|
||||
`would_demote`, does not demote.
|
||||
- *Phase 3 — demotion-only runtime, gated.* Only if Phase-2 bench
|
||||
supports it: `governance_policy_hash` includes `nli_policy_hash`;
|
||||
NLI runs only on recombination-risk weak labels; contradiction
|
||||
demotes only, never promotes; audit output records reason/hash/pair.
|
||||
- *Phase 4 — mesh blob sync.* Peer-pull the checkpoint by sha256;
|
||||
origin URL remains canonical fallback; hash verification
|
||||
fail-closed.
|
||||
|
||||
**17. Discipline question — final answer.** *May a small fixed NLI
|
||||
model influence `audit_mode`?* — **Not by default. Yes only as an
|
||||
opt-in, hash-pinned, governance-hashed, demotion-only contradiction
|
||||
veto after shadow-mode evidence.** Respects D1 (the answering LLM
|
||||
isn't grading itself; the NLI model is fixed and content-addressed;
|
||||
it can't promote truth; it only vetoes weak grounding; every
|
||||
checkpoint/policy change invalidates prior cache identity; the output
|
||||
is bench-gated and auditable) while respecting the danger (a model in
|
||||
the verifier path is still a model in the verifier path — a
|
||||
controlled exception, not a casual dependency).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue