#000049: fix loose 'real traffic' wording → 'bench-qa traffic' (defined in §7 #20: arborist's own query pipeline driven by the curated bench question set, not live users)
This commit is contained in:
parent
59d8a17a14
commit
f955082ebe
2 changed files with 28 additions and 18 deletions
|
|
@ -104,7 +104,7 @@ Newest first. Update on every open/close.
|
|||
| #000052 | Relevance + coherence meta-cognition (answer-*shape* signals) | open · awaiting go/no-go · doc-only scaffold. Two more read-only, demote-only, never-in-proof-path sidecars joining the `diagnose_deflection` / `diagnose_metaphor_deflection` / `diagnose_title_relevance` / soft-preflight family: **(1) `diagnose_coherence`** — word-salad / circular ("X is X") / vacuous / underspecified answers; lexical/parse-shape, **no model** — the near-term win (recommended now). **(2) `diagnose_relevance`** — semantic (not just lexical) "aboutness": does the answer address the question; is each claim about its cited source? Today's checks (subject-anchor token overlap, stemmed title-stem overlap) are *lexical* and a token collision defeats them — a small *aboutness/reranker* model (NOT NLI — entailment ≠ topicality) under #000049 §7's discipline cage verbatim (demotion-only, hash-pinned, `relevance_model_version`→`governance_policy_hash` iff it touches `audit_mode`, shadow-first, `[…]` extra, the §7 #20 haystack lesson — never over the whole context); gated on evidence, travels with #000049's model question. Motivating field case (2026-05-12, fox): the `claim_lattice` query that returned *"the phrase 'Zionist entity' is sometimes used as the entity, referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL 2/3` — incoherent + token-collision recombination that NLI can't catch (returns *neutral*, not *contradiction*) and both lexical relevance checks waved through. Flags an upstream retrieval ticket (polysemy / title-token-soup) as the root-cause fix, not scoped here. #000049 sibling | 2026-05-12 | — |
|
||||
| #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — |
|
||||
| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — |
|
||||
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. **Phase-2 candidate bench done 2026-05-12** (`~/git/arborist-nli-bench/`, commits `829f9a4` + `a1cb28d`; ticket §7 #18): checkpoint-agnostic harness runs the §7 #5 clause-level algorithm over 28 synth recombination cases (incl. the 2 fixtures + harder shapes) + 26 legit cases (true summaries + near-miss decoys). 4 working candidates; `nli-MiniLM2-L6-H768` (82M, 45ms p50 CPU), `deberta-v3-base-mnli-fever-anli` (184M, 223ms), `bart-large-mnli` (407M, 259ms) all 28/28 catch · 0/26 FP with the standard θe=0.9 entailment guard; `cross-encoder/nli-deberta-v3-base` 27/28; deberta-large repo-id TODO. **Key finding: the §7 #5 two-threshold rule is load-bearing** — 3 of 4 candidates argmax-contradict 1/26 legit cases on the *wrong* source clause (competing-superlative confusion, e.g. "largest hot desert" vs "largest desert overall"); the entailment guard filters every one because another clause restates the claim → 0% guarded FP vs ~4% single-threshold. Picture: recombination is *easy* for any modern NLI checkpoint — differentiator is cost/robustness, MiniLM is the cost-pick, bart-large the threshold-robust pick. **Phase-2 shadow scaffold landed in arborist 2026-05-12** (ticket §7 #19): `arborist/qa/nli/` (manifest pins MiniLM @ a fixed HF revision + θc 0.5/θe 0.9 + 2 alternates; `ShadowNLI`/`shadow_check` lazy-imports `transformers`+`torch` behind a new `[nli]` extra, degrades to `available=False` when absent — SHADOW ONLY, never an `audit_mode` input, manifest not yet in `governance_policy_hash` per §7 #2) + `bench/scripts/nli_shadow_sweep.py` + `make bootstrap-nli` / `make bench-nli-shadow` + 16 tests. Synthetic sweep (116 records): 28/28 recombination demoted, 0/26 FP on legit summaries, 0/9 fires on already-`STRICT_SPAN`. **First real-traffic sweep** (§7 #20 — `ARBORIST_NLI_SHADOW=1 make bench-qa-smoke`, 15 cells; `query.py` surfaces verifier-input text gated off-by-default, `qa_sweep.py` carries it, `nli_shadow_sweep.py` reads it): the *naive* "NLI on every context clause" scaffold has a **~30% would-demote rate on STRICT answers** — a haystack/multiple-comparisons artifact (real contexts → 100–336 clauses; `max`-over-all almost always hits a tangential "contradiction"). Candidate-clause restriction added (§7 #21 — NLI runs on the top-6 source clauses by content-token overlap, not the whole context; `max_candidate_clauses=6`): STRICT would-demote 30% → 20% on the smoke, overall 47% → 33% — **helps, not fixed**; recombination-risk split doesn't separate either. Residual STRICT false-contras land at ~0.83–0.92, so θc would need ≈ 0.90 (up from the clean-set 0.5): at θc=0.90 the data in hand gives 27/28 synthetic recombination recall, 0/26 synthetic legit FP, 0/10 smoke STRICT FP — but n=10 is far too small to set a threshold on. Remaining: a fuller `ARBORIST_NLI_SHADOW=1 make bench-qa` run → ~hundreds of STRICT cells → sweep θc/θe → confirm → set it; until then θc stays 0.5 and runtime NLI demotion stays off. Standing lesson: the clean synthetic eval (§7 #18) does not predict real-traffic precision. Production verifier unchanged; `falsification-hard` stays 10/12. #000048 follow-up | 2026-05-12 | — |
|
||||
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. **Phase-2 candidate bench done 2026-05-12** (`~/git/arborist-nli-bench/`, commits `829f9a4` + `a1cb28d`; ticket §7 #18): checkpoint-agnostic harness runs the §7 #5 clause-level algorithm over 28 synth recombination cases (incl. the 2 fixtures + harder shapes) + 26 legit cases (true summaries + near-miss decoys). 4 working candidates; `nli-MiniLM2-L6-H768` (82M, 45ms p50 CPU), `deberta-v3-base-mnli-fever-anli` (184M, 223ms), `bart-large-mnli` (407M, 259ms) all 28/28 catch · 0/26 FP with the standard θe=0.9 entailment guard; `cross-encoder/nli-deberta-v3-base` 27/28; deberta-large repo-id TODO. **Key finding: the §7 #5 two-threshold rule is load-bearing** — 3 of 4 candidates argmax-contradict 1/26 legit cases on the *wrong* source clause (competing-superlative confusion, e.g. "largest hot desert" vs "largest desert overall"); the entailment guard filters every one because another clause restates the claim → 0% guarded FP vs ~4% single-threshold. Picture: recombination is *easy* for any modern NLI checkpoint — differentiator is cost/robustness, MiniLM is the cost-pick, bart-large the threshold-robust pick. **Phase-2 shadow scaffold landed in arborist 2026-05-12** (ticket §7 #19): `arborist/qa/nli/` (manifest pins MiniLM @ a fixed HF revision + θc 0.5/θe 0.9 + 2 alternates; `ShadowNLI`/`shadow_check` lazy-imports `transformers`+`torch` behind a new `[nli]` extra, degrades to `available=False` when absent — SHADOW ONLY, never an `audit_mode` input, manifest not yet in `governance_policy_hash` per §7 #2) + `bench/scripts/nli_shadow_sweep.py` + `make bootstrap-nli` / `make bench-nli-shadow` + 16 tests. Synthetic sweep (116 records): 28/28 recombination demoted, 0/26 FP on legit summaries, 0/9 fires on already-`STRICT_SPAN`. **First bench-qa-traffic sweep** (§7 #20 — `ARBORIST_NLI_SHADOW=1 make bench-qa-smoke`, 15 cells; `query.py` surfaces verifier-input text gated off-by-default, `qa_sweep.py` carries it, `nli_shadow_sweep.py` reads it): the *naive* "NLI on every context clause" scaffold has a **~30% would-demote rate on STRICT answers** — a haystack/multiple-comparisons artifact (real Wikipedia contexts → 100–336 clauses; `max`-over-all almost always hits a tangential "contradiction"). Candidate-clause restriction added (§7 #21 — NLI runs on the top-6 source clauses by content-token overlap, not the whole context; `max_candidate_clauses=6`): STRICT would-demote 30% → 20% on the smoke, overall 47% → 33% — **helps, not fixed**; recombination-risk split doesn't separate either. Residual STRICT false-contras land at ~0.83–0.92, so θc would need ≈ 0.90 (up from the clean-set 0.5): at θc=0.90 the data in hand gives 27/28 synthetic recombination recall, 0/26 synthetic legit FP, 0/10 smoke STRICT FP — but n=10 is far too small to set a threshold on. Remaining: a fuller `ARBORIST_NLI_SHADOW=1 make bench-qa` run → ~hundreds of STRICT cells → sweep θc/θe → confirm → set it; until then θc stays 0.5 and runtime NLI demotion stays off. Standing lesson: the clean synthetic eval (§7 #18) does not predict bench-qa-traffic precision. Production verifier unchanged; `falsification-hard` stays 10/12. #000048 follow-up | 2026-05-12 | — |
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, now tracked as **#000049**, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 closed | 2026-05-11 | — |
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ landed 2026-05-12 (§7 #18/#19 — candidate bench in `~/git/arborist-nli-bench/
|
|||
`arborist/qa/nli/` shadow module + `[nli]` extra + `make bench-nli-shadow`
|
||||
in-repo, SHADOW ONLY — never an `audit_mode` input; first sweep:
|
||||
28/28 recombination demoted, 0/26 FP on legit summaries, never fires on
|
||||
already-STRICT records). First real-traffic shadow sweep done (§7 #20, `ARBORIST_NLI_SHADOW=1 make
|
||||
already-STRICT records). First bench-qa-traffic shadow sweep done (§7 #20, `ARBORIST_NLI_SHADOW=1 make
|
||||
bench-qa-smoke` → 15 cells): the *naive* "NLI on every context clause"
|
||||
version has a **~30% would-demote rate on STRICT answers** — a haystack /
|
||||
multiple-comparisons artifact (real contexts segment into 100–336 clauses;
|
||||
|
|
@ -20,7 +20,7 @@ trigger; do NOT enable runtime NLI demotion on the current scaffold.
|
|||
Candidate-clause restriction added (§7 #21 — `ShadowNLI.check` now NLI's
|
||||
only the top-6 source clauses by overlap, not the whole context): STRICT
|
||||
would-demote 30% → 20% on the smoke — *helps, not fixed*; the data points
|
||||
to θc ≈ 0.90 (up from the clean-set 0.5) to zero out the real-traffic
|
||||
to θc ≈ 0.90 (up from the clean-set 0.5) to zero out the bench-qa-traffic
|
||||
STRICT FPs at 27/28 synthetic recombination recall, but n=10 is too small
|
||||
to set it on. Remaining: a fuller `ARBORIST_NLI_SHADOW=1 make bench-qa`
|
||||
run → sweep θc on hundreds of STRICT cells → confirm → set it. Runtime NLI
|
||||
|
|
@ -165,7 +165,7 @@ subset the lexical path can't resolve.
|
|||
baseline; the 2 recombination fixtures *are* the documentation of
|
||||
where the lexical verifier stops (#000048 §5). Closing them buys
|
||||
2/12 on a contrived pack at the cost of (heuristic brittleness, or a
|
||||
model dependency). If no real QA-bench evidence shows recombination
|
||||
model dependency). If no bench-qa evidence shows recombination
|
||||
over-grounds happening at scale on actual traffic, this is the
|
||||
honest choice. (No bench evidence of that exists today — the #000048
|
||||
step-2.1/2.4 per-row diffs found 0 gate-attributable and 0
|
||||
|
|
@ -282,7 +282,7 @@ decides when to call shadow NLI, never as the contradiction decider.
|
|||
Original rationale (still holds for the production no-go):
|
||||
|
||||
1. The lexical verifier is honest about its boundary (the 2 fixtures
|
||||
document it); there's no observed real-traffic harm.
|
||||
document it); there's no observed bench-qa-traffic harm.
|
||||
2. Option 2.1 (heuristic) is the same threshold-can't-separate
|
||||
problem dressed up — low assurance, real over-fire risk on the
|
||||
QA bench.
|
||||
|
|
@ -509,13 +509,13 @@ by default, no `audit_mode` effect, no cache effect, writes only
|
|||
bench/shadow telemetry (`{shadow_nli_enabled, would_demote,
|
||||
current_audit_mode, nli_policy_hash, checkpoint_sha256,
|
||||
source_clause_id, answer_claim_id}`). Lets us measure the
|
||||
false-positive rate on real QA *before* the model is allowed to
|
||||
false-positive rate on bench-qa output *before* the model is allowed to
|
||||
affect `audit_mode`. This is the no-regression path.
|
||||
|
||||
**12. Bench gate (explicit) — before enabling runtime demotion:**
|
||||
(1) `falsification-hard` catches the 2 recombination fixtures; (2)
|
||||
legit cross-sentence summaries — no demotion; (3) paraphrase/entity
|
||||
legit cases — no proof-quality regression; (4) real-QA sample —
|
||||
legit cases — no proof-quality regression; (4) bench-qa sample —
|
||||
shadow-mode false-positive rate below threshold; (5) latency — NLI
|
||||
called only on a small unresolved subset; (6) determinism — same run
|
||||
×3 → identical demote/no-demote decisions. Suggested thresholds:
|
||||
|
|
@ -645,10 +645,10 @@ item 4).** (a) The eval set is *synthesized and still relatively
|
|||
clean* — single-/few-sentence sources, encyclopedia-shaped facts;
|
||||
real Wikipedia QA has long messy contexts, partial overlaps, hedged
|
||||
claims, and the paraphrase/entity path it'd attach to rarely fires
|
||||
on real traffic anyway. **Gate item 4 — shadow-mode FP rate on a
|
||||
real `bench-qa` sample — is the actual test and is not done here.**
|
||||
on bench-qa traffic anyway. **Gate item 4 — shadow-mode FP rate on a
|
||||
full `bench-qa` sample — is the actual test and is not done here.**
|
||||
(b) n = 54 total — better than n = 2, still small; the harder shapes
|
||||
help but a real-traffic-derived set would be more load-bearing. (c)
|
||||
help but a bench-qa-traffic-derived set would be more load-bearing. (c)
|
||||
4 of 5 manifest candidates ran (deberta-large repo-id TODO) — but the
|
||||
picture is already clear enough that "which checkpoint" is a
|
||||
cost/robustness trade, not an accuracy one. **None of Phases 2 (full
|
||||
|
|
@ -658,7 +658,7 @@ with real numbers replacing a contrived 2/12, and the answer it
|
|||
points to is: yes, a shadow mode is worth building; the §7 #5
|
||||
two-threshold algorithm is the right shape (the entailment guard
|
||||
earns its place); a small cross-encoder NLI suffices; the one open
|
||||
risk is real-traffic false positives, measurable only by a shadow
|
||||
risk is bench-qa-traffic false positives, measurable only by a shadow
|
||||
run on actual QA (Phase 2 → gate item 4).
|
||||
|
||||
**19. Phase-2 shadow scaffold landed in arborist (2026-05-12).**
|
||||
|
|
@ -709,8 +709,18 @@ INPUT="…"`); feeding it a live `bench-qa` run is the next concrete
|
|||
step, and a slow/live one (fox-decides). Nothing here changes the
|
||||
production verifier; `falsification-hard` stays 10/12.
|
||||
|
||||
**20. First *real-traffic* shadow sweep (2026-05-12) — the haystack
|
||||
problem.** Added the live hook: `ARBORIST_NLI_SHADOW=1` makes
|
||||
**20. First *bench-qa-traffic* shadow sweep (2026-05-12) — the haystack
|
||||
problem.** *Terminology:* "bench-qa traffic" here means **arborist's
|
||||
own query pipeline exercised by the curated `bench/qa_questions.txt`
|
||||
set** — real FTS5 retrieval over the real ~3.4 M-article enwiki
|
||||
corpus → real assembled 8–64 KB contexts → the real Hermes LLM →
|
||||
the real verifier — *not* live human users (there are none). It is
|
||||
the messiest input we actually produce, and the right proxy for "what
|
||||
would real questions look like"; the `ARBORIST_NLI_SHADOW=1` hook is
|
||||
the same machinery that would accumulate telemetry on live queries
|
||||
if there ever were any. "Synthetic" (§7 #18) = the contrived
|
||||
`arborist-nli-bench/eval/*.jsonl` + 5f fixtures (short, clean,
|
||||
designed-to-test). Added the live hook: `ARBORIST_NLI_SHADOW=1` makes
|
||||
`query()` surface the verifier-input text (gated, off by default,
|
||||
never a cache_key / governance / `audit_mode` input — `query.py`
|
||||
~`return result`), `bench/qa_sweep.py` carries it + the answer into
|
||||
|
|
@ -749,7 +759,7 @@ use regardless.)
|
|||
load-bearing.** The shadow scaffold's `ShadowNLI.check` deliberately
|
||||
ran NLI on *every* clause (the simplest thing that could measure
|
||||
anything); that version is now empirically shown to be unusable on
|
||||
real traffic. Phase 3 (and a refined Phase-2 shadow) MUST restrict the
|
||||
bench-qa traffic. Phase 3 (and a refined Phase-2 shadow) MUST restrict the
|
||||
NLI call to a *small* candidate set — only the clauses the lexical
|
||||
verifier's quote/span/entity matching actually touched, and/or only
|
||||
when a deterministic recombination-risk trigger fires (high
|
||||
|
|
@ -763,7 +773,7 @@ cannot solve alone, and the deterministic recombination-risk pre-
|
|||
filter is what makes it tractable. So: do not enable any runtime NLI
|
||||
demotion on the current scaffold; the next Phase-2 step is to add the
|
||||
candidate-clause restriction to `ShadowNLI.check`, re-run the
|
||||
real-traffic sweep, and only then is gate item 4 measurable as a
|
||||
bench-qa-traffic sweep, and only then is gate item 4 measurable as a
|
||||
meaningful number. Production verifier unchanged; `falsification-hard`
|
||||
stays 10/12.
|
||||
|
||||
|
|
@ -793,7 +803,7 @@ top-overlap Wikipedia clause contradicts a *synthesized/aggregated*
|
|||
STRICT answer (a dropped qualifier, a tangential date, a sibling
|
||||
fact). On the clean synthetic set the recombination contras are ≈
|
||||
0.99 and the legit ones ≈ 0.005, so the manifest's θc = 0.5 was fine
|
||||
*there*; on real traffic the false-contra mass sits in 0.6–0.92, so
|
||||
*there*; on bench-qa traffic the false-contra mass sits in 0.6–0.92, so
|
||||
**θc would need to rise to ≈ 0.90**. At θc = 0.90 on the data in hand:
|
||||
synthetic recombination catch = 27/28 (loses 1 hard case), synthetic
|
||||
legit FP = 0/26 (the one 0.946 legit case is rescued by the entailment
|
||||
|
|
@ -809,7 +819,7 @@ NLI demotion stays off, and the standalone-lexical candidate proxy
|
|||
stays a proxy — a Phase-3 runtime hook would pass the verifier's
|
||||
*actual* matched clauses, tighter than top-6-by-overlap. The deeper
|
||||
standing lesson: the clean synthetic eval (§7 #18) does **not** predict
|
||||
real-traffic precision — every gate number that matters has to come
|
||||
from a shadow run on real `bench-qa` output, not from contrived
|
||||
bench-qa-traffic precision — every gate number that matters has to come
|
||||
from a shadow run on `bench-qa` pipeline output, not from contrived
|
||||
fixtures. Production verifier unchanged; `falsification-hard` stays
|
||||
10/12.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue