diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 06ceb63..6100a71 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -104,7 +104,7 @@ Newest first. Update on every open/close. | #000053 | Acronym-aware verifier content tokens | **closed · 2026-05-13** — `arborist.qa.evidence._content_tokens` now keeps all-caps 2-3-char acronyms (CPU/GPU/DNA/FBI/USB…) as content tokens instead of dropping every <4-char token; fixes the field case where "what is a CPU?" cited to "CPU design" tripped `TITLE_MISMATCH` spuriously (claim & title share "CPU" but neither registered) — also affects `SUBJECT_TOKENS_ABSENT` (Rule 9), `BARE_NAME_CLAIM`, spotlight-excerpt token pick. Versioned: `content_token_rules: "v2-acronym-aware"` in both default policies + `_VERIFIER_POLICY_FIELDS` → folds into `verifier_policy_hash`, prior cache records orphan on lookup (by design, same discipline as `base_version` / `hyphen_fold_v1`). Monotone toward *fewer* spurious demotes (only relaxes overlap checks, never tightens). 8 new tests; full suite green; `bench-qa-smoke` clean. Does NOT fix the *retrieval* abbreviation→expansion gap (`CPU`→`Central processing unit` = #000050 vec hybrid / `concepts/` synonym edges — the root cause of the satellite-article retrieval). | 2026-05-13 | — | | #000052 | Relevance + coherence meta-cognition (answer-*shape* signals) | in progress — **§3.1 `diagnose_coherence` landed** (lexical, no model: `circular` / `phrase_component_reuse` / `vacuous`; in `arborist/qa/inspect.py`, surfaced via `inspect_cache_key` + `arborist inspect` `· incoherent: `; 9 tests; demote-policy hook deliberately not wired — advisory only). Joins the `diagnose_deflection` / `diagnose_metaphor_deflection` / `diagnose_title_relevance` / soft-preflight family of read-only, demote-only, never-in-proof-path sidecars; `phrase_component_reuse` catches the motivating field case (a subject quoting a phrase, a predicate reusing one of that phrase's own tokens as a bare `the ` referent). **Still open: (2) `diagnose_relevance`** — semantic (not just lexical) "aboutness": does the answer address the question; is each claim about its cited source? Today's checks (subject-anchor token overlap, stemmed title-stem overlap) are *lexical* and a token collision defeats them — a small *aboutness/reranker* model (NOT NLI — entailment ≠ topicality) under #000049 §7's discipline cage verbatim (demotion-only, hash-pinned, `relevance_model_version`→`governance_policy_hash` iff it touches `audit_mode`, shadow-first, `[…]` extra, the §7 #20 haystack lesson — never over the whole context); gated on evidence, travels with #000049's model question. Motivating field case (2026-05-12, fox): the `claim_lattice` query that returned *"the phrase 'Zionist entity' is sometimes used as the entity, referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL 2/3` — incoherent + token-collision recombination that NLI can't catch (returns *neutral*, not *contradiction*) and both lexical relevance checks waved through. Flags an upstream retrieval ticket (polysemy / title-token-soup) as the root-cause fix, not scoped here. #000049 sibling | 2026-05-12 | — | | #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — | -| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — | +| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on the **§2a semantic-allusion fixture set** (running list of query-words-share-zero-tokens-with-target-title cases: Orwell→Eastasia, "what is a CPU?"→Central processing unit, "what is a GPU?"→Graphics processing unit — abbreviation ones also fixable upstream via `concepts/` synonym edges; the Orwell one is the genuine-conceptual-allusion case that justifies the vec layer) + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — | | #000049 | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. **Phase-2 candidate bench done 2026-05-12** (`~/git/arborist-nli-bench/`, commits `829f9a4` + `a1cb28d`; ticket §7 #18): checkpoint-agnostic harness runs the §7 #5 clause-level algorithm over 28 synth recombination cases (incl. the 2 fixtures + harder shapes) + 26 legit cases (true summaries + near-miss decoys). 4 working candidates; `nli-MiniLM2-L6-H768` (82M, 45ms p50 CPU), `deberta-v3-base-mnli-fever-anli` (184M, 223ms), `bart-large-mnli` (407M, 259ms) all 28/28 catch · 0/26 FP with the standard θe=0.9 entailment guard; `cross-encoder/nli-deberta-v3-base` 27/28; deberta-large repo-id TODO. **Key finding: the §7 #5 two-threshold rule is load-bearing** — 3 of 4 candidates argmax-contradict 1/26 legit cases on the *wrong* source clause (competing-superlative confusion, e.g. "largest hot desert" vs "largest desert overall"); the entailment guard filters every one because another clause restates the claim → 0% guarded FP vs ~4% single-threshold. Picture: recombination is *easy* for any modern NLI checkpoint — differentiator is cost/robustness, MiniLM is the cost-pick, bart-large the threshold-robust pick. **Phase-2 shadow scaffold landed in arborist 2026-05-12** (ticket §7 #19): `arborist/qa/nli/` (manifest pins MiniLM @ a fixed HF revision + θc 0.5/θe 0.9 + 2 alternates; `ShadowNLI`/`shadow_check` lazy-imports `transformers`+`torch` behind a new `[nli]` extra, degrades to `available=False` when absent — SHADOW ONLY, never an `audit_mode` input, manifest not yet in `governance_policy_hash` per §7 #2) + `bench/scripts/nli_shadow_sweep.py` + `make bootstrap-nli` / `make bench-nli-shadow` + 16 tests. Synthetic sweep (116 records): 28/28 recombination demoted, 0/26 FP on legit summaries, 0/9 fires on already-`STRICT_SPAN`. **First bench-qa-traffic sweep** (§7 #20 — `ARBORIST_NLI_SHADOW=1 make bench-qa-smoke`, 15 cells; `query.py` surfaces verifier-input text gated off-by-default, `qa_sweep.py` carries it, `nli_shadow_sweep.py` reads it): the *naive* "NLI on every context clause" scaffold has a **~30% would-demote rate on STRICT answers** — a haystack/multiple-comparisons artifact (real Wikipedia contexts → 100–336 clauses; `max`-over-all almost always hits a tangential "contradiction"). Candidate-clause restriction (§7 #21 — NLI runs on the top-6 source clauses by content-token overlap, `max_candidate_clauses=6`) + speedup (§7 #22 — batched forwards, `ARBORIST_NLI_DEVICE` cuda auto-detect, ONNX-int8 export via `make export-nli-onnx`: torch-cpu-batch1 ~120ms/pair → onnx-int8-cpu-batched ~32ms/pair, seconds on a 4090; `optimum[onnxruntime]` added to `[nli]`; 24 tests) landed. **Verdict moved three times then settled** (§7 #22→#23→#24→#25; `bench/scripts/nli_shadow_grid.py` — `{model × k × agg × guard × θc × θe}` sweep, ~3.5min for 7 models on the 4090, run against bench-qa STRICT samples of growing size n=1=89 → n=3=275 → n=5=444): §7 #22's "fails the gate" was a *default-config* artifact (k=6/θc=0.5/θe=0.9); §7 #23's "deberta-base k=2/θc=0.99 → 27/28 · 0/89 passes" was a *small-FP-sample* artifact (it's 27/28 again at n=5 but the n=3 sample drove it to 11/28 — sample-sensitive, sits at the cliff); §7 #24 (n=3) found `microsoft/deberta-large-mnli` passes; **§7 #25 (n=5-confirmed, 444 real STRICT cells)**: the lexical-candidate NLI veto **robustly clears the §7 #12 gate** with **`microsoft/deberta-large-mnli` / k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9 → catches all 28 synthetic recombinations (incl. both `5f-fal-hard` fixtures) · 0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts of θc headroom**; `roberta-large-mnli` is an equally-good alternate (k=2/max/θc=0.95). Resolved findings: `agg=max + max_entail guard` is the robust score-shape across both proper-n samples (§7 #24's `margin` win was a sample tie); the *specific large checkpoint* is what matters — deberta-large-mnli / roberta-large-mnli (~350-400M) hit 1.0/0.0, `bart-large` (similar size) only ~0.71, `deberta-base-184M` at the cliff, the small models (MiniLM-82M, deberta-v3-small) cap at ~0.82 — so the §7 #18 "MiniLM is the cost-pick" is **overturned by the proper-n evidence**; k=2 is the consistent winner; int8-ONNX costs ≈1 catch vs fp32. Remaining caveats (narrow now): the 28/28 recall is still a 28-case denominator — a bench-qa-derived recombination set is the one load-bearing check not yet done; the FP side is 444 cells at 0 FP (solid); n=9 (~825 cells) would add confidence but, given the ~4-5 pt θc margin, is "if dav1d wants more"; still SHADOW — flipping to a runtime demotion-only veto is fox+dav1d-decides (then `nli_policy_hash` = model+k+agg+guard+θc folds into `governance_policy_hash` per §7 #2); deberta-large is ~400M (GPU-fast). Manifest active defaults stay k=6/θc=0.5; the `recommended_operating_point` (deberta-large-mnli / k=2 / max / θc=0.96 / θe=0.9) is documented in the manifest. Standing lesson, sharpened three times — clean eval ≠ bench-qa precision (§7 #18→#20), default config ≠ best config (§7 #22→#23), small FP sample ≠ large-sample FP rate (§7 #23→#24→#25): every gate number is provisional until the denominators are big enough, and the sweep has to be wide enough to include the config that survives them — codified in CLAUDE.md's bench-maxing section. Production verifier unchanged; `falsification-hard` stays 10/12. #000048 follow-up | 2026-05-12 | — | | #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | — | | #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — | diff --git a/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md b/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md index 4e43aeb..c29eeff 100644 --- a/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md +++ b/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md @@ -61,9 +61,10 @@ Phase 2 implementation does **not** open until both: **#000051** — this prereq is "a vecpack exists and has been imported on the bench box", not "fox embedded the corpus locally". 2. **A recall bench clears the gate.** Run **four conditions** on the - bench fixtures (bench-emergent semantic-allusion shapes + the - curated set + the adversarial semantic-neighbor pack from §3 - below): **(A)** FTS5-only baseline, **(B)** vec-only, **(C)** FTS5 + bench fixtures (the **§2a semantic-allusion fixture set** + the + bench-emergent shapes + the curated set + the adversarial + semantic-neighbor pack from §3 below): **(A)** FTS5-only baseline, + **(B)** vec-only, **(C)** FTS5 + vec RRF hybrid, **(D)** FTS5 + vec candidate-union but the *existing* reranker only (no RRF). D is the control that distinguishes "RRF helped" from "more candidates helped" — if C @@ -87,6 +88,33 @@ Phase 2 implementation does **not** open until both: (`--backend vec`), hybrid does NOT become the default, and the ~2.4 GB tax isn't paid for nothing. +## 2a. Semantic-allusion fixture set (the cases the §2 bench must include) + +The §2 gate's "semantic-allusion fixtures" aren't a vague category — +this is the running list of concrete cases where the query's words +share *zero tokens* with the target article's title, so the four FTS5 +routes (body BM25, title-LIKE, title-token, phrase-pattern) structurally +can't surface it, but a dense-vector route should. The vec-only bench +(condition B) must beat FTS5-only (A) by ≥ 5pp recall@K on **≥ 1** of +these *and* surface ≥ 1 correct candidate absent from A's top-K; the +hybrid bench (C) must lift the proof-quality-label rate on the set. +When a new field case of this shape shows up, add a row here. + +| query | target article (title shares 0 tokens with query) | why FTS5 misses it | +|---|---|---| +| "what did Orwell call the country at war with Oceania?" | *Nineteen Eighty-Four* (→ Eastasia / Eurasia) | query tokens {orwell, country, war, oceania}; the article's distinctive content ("Eastasia", "always been at war with") isn't in the title, and the title shares no token with the query — only the verbatim *phrase* route ("always been at war") rescues a piece of it today (CLAUDE.md retrieval-pipeline §1) | +| "what is a CPU?" | *Central processing unit* | query token "CPU" ≠ the title "Central processing unit" — FTS5 pulls the "CPU *" satellites (CPU design, CPU socket, CPU time, CPU cache, CPU-Z, CPU multiplier, CPU disambiguation) where "CPU" is *in* the title; the canonical article never surfaces. (Field case 2026-05-13, fox. Note: #000053 fixed the *verifier*'s separate acronym blind spot — `EVIDENCE-WARRANTED` instead of a spurious `TITLE_MISMATCH` — but the answer is still grounded in "CPU design", not the real article; that's *this* gap.) | +| "what is a GPU?" | *Graphics processing unit* | identical shape to CPU — FTS5 returns GPU cluster / GPU (disambiguation) / Molecular modeling on GPU / Evergreen (GPU family) / etc.; observed 0/1 UNGROUNDED because the model recited the (modern) GPU-lead definition and *no* 2003-dump GPU-* satellite carried that text. (Field case 2026-05-13, fox.) | + +(The abbreviation→expansion subclass — `CPU`↔`Central processing unit`, +`GPU`↔`Graphics processing unit` — is *also* addressable upstream by a +`concepts/` synonym edge, cheaper than vec; both fixes are +complementary and the bench should record which one (or both) closes +each row. The Orwell row is *not* an abbreviation case — it's genuine +conceptual allusion, the kind only dense vectors reach — so it's the +one that actually justifies the vec layer if the abbreviation cases +turn out to be all `concepts/` could-have-fixed.) + ## 3. Implementation sketch (when the gate clears) **RRF merge.** `arborist/qa/query.py`: after the four FTS5 routes