From 59d8a17a14cc0c253ffd48ff9f715dca6ae887ae Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 12 May 2026 14:49:10 -0400 Subject: [PATCH] =?UTF-8?q?#000052:=20open=20=E2=80=94=20relevance=20+=20c?= =?UTF-8?q?oherence=20meta-cognition=20(answer-shape=20sidecars)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Doc-only scaffold. Two more read-only/demote-only/never-in-proof-path sidecars joining the diagnose_deflection family: (1) diagnose_coherence — word-salad/circular/vacuous answers; lexical, no model; the near-term win. (2) diagnose_relevance — semantic 'aboutness' (does the answer address the question / is each claim about its cited source?); today's checks are lexical and a token collision defeats them; a small aboutness/reranker model (NOT NLI) under #000049 §7's discipline cage verbatim; gated on evidence, travels with #000049's model question. Motivating field case: the 'Zionist entity' claim_lattice query — incoherent token-collision recombination NLI can't catch (returns neutral) and both lexical relevance checks waved through. Flags an upstream retrieval (polysemy/title-soup) root-cause ticket, not scoped here. Next ID 000052 → 000053. #000049 sibling --- docs/TICKETS.md | 3 +- ...0052-relevance-coherence-meta-cognition.md | 310 ++++++++++++++++++ 2 files changed, 312 insertions(+), 1 deletion(-) create mode 100644 docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 004c5dc..3606ea6 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -101,6 +101,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| +| #000052 | Relevance + coherence meta-cognition (answer-*shape* signals) | open · awaiting go/no-go · doc-only scaffold. Two more read-only, demote-only, never-in-proof-path sidecars joining the `diagnose_deflection` / `diagnose_metaphor_deflection` / `diagnose_title_relevance` / soft-preflight family: **(1) `diagnose_coherence`** — word-salad / circular ("X is X") / vacuous / underspecified answers; lexical/parse-shape, **no model** — the near-term win (recommended now). **(2) `diagnose_relevance`** — semantic (not just lexical) "aboutness": does the answer address the question; is each claim about its cited source? Today's checks (subject-anchor token overlap, stemmed title-stem overlap) are *lexical* and a token collision defeats them — a small *aboutness/reranker* model (NOT NLI — entailment ≠ topicality) under #000049 §7's discipline cage verbatim (demotion-only, hash-pinned, `relevance_model_version`→`governance_policy_hash` iff it touches `audit_mode`, shadow-first, `[…]` extra, the §7 #20 haystack lesson — never over the whole context); gated on evidence, travels with #000049's model question. Motivating field case (2026-05-12, fox): the `claim_lattice` query that returned *"the phrase 'Zionist entity' is sometimes used as the entity, referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL 2/3` — incoherent + token-collision recombination that NLI can't catch (returns *neutral*, not *contradiction*) and both lexical relevance checks waved through. Flags an upstream retrieval ticket (polysemy / title-token-soup) as the root-cause fix, not scoped here. #000049 sibling | 2026-05-12 | — | | #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — | | #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — | | #000049 | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. **Phase-2 candidate bench done 2026-05-12** (`~/git/arborist-nli-bench/`, commits `829f9a4` + `a1cb28d`; ticket §7 #18): checkpoint-agnostic harness runs the §7 #5 clause-level algorithm over 28 synth recombination cases (incl. the 2 fixtures + harder shapes) + 26 legit cases (true summaries + near-miss decoys). 4 working candidates; `nli-MiniLM2-L6-H768` (82M, 45ms p50 CPU), `deberta-v3-base-mnli-fever-anli` (184M, 223ms), `bart-large-mnli` (407M, 259ms) all 28/28 catch · 0/26 FP with the standard θe=0.9 entailment guard; `cross-encoder/nli-deberta-v3-base` 27/28; deberta-large repo-id TODO. **Key finding: the §7 #5 two-threshold rule is load-bearing** — 3 of 4 candidates argmax-contradict 1/26 legit cases on the *wrong* source clause (competing-superlative confusion, e.g. "largest hot desert" vs "largest desert overall"); the entailment guard filters every one because another clause restates the claim → 0% guarded FP vs ~4% single-threshold. Picture: recombination is *easy* for any modern NLI checkpoint — differentiator is cost/robustness, MiniLM is the cost-pick, bart-large the threshold-robust pick. **Phase-2 shadow scaffold landed in arborist 2026-05-12** (ticket §7 #19): `arborist/qa/nli/` (manifest pins MiniLM @ a fixed HF revision + θc 0.5/θe 0.9 + 2 alternates; `ShadowNLI`/`shadow_check` lazy-imports `transformers`+`torch` behind a new `[nli]` extra, degrades to `available=False` when absent — SHADOW ONLY, never an `audit_mode` input, manifest not yet in `governance_policy_hash` per §7 #2) + `bench/scripts/nli_shadow_sweep.py` + `make bootstrap-nli` / `make bench-nli-shadow` + 16 tests. Synthetic sweep (116 records): 28/28 recombination demoted, 0/26 FP on legit summaries, 0/9 fires on already-`STRICT_SPAN`. **First real-traffic sweep** (§7 #20 — `ARBORIST_NLI_SHADOW=1 make bench-qa-smoke`, 15 cells; `query.py` surfaces verifier-input text gated off-by-default, `qa_sweep.py` carries it, `nli_shadow_sweep.py` reads it): the *naive* "NLI on every context clause" scaffold has a **~30% would-demote rate on STRICT answers** — a haystack/multiple-comparisons artifact (real contexts → 100–336 clauses; `max`-over-all almost always hits a tangential "contradiction"). Candidate-clause restriction added (§7 #21 — NLI runs on the top-6 source clauses by content-token overlap, not the whole context; `max_candidate_clauses=6`): STRICT would-demote 30% → 20% on the smoke, overall 47% → 33% — **helps, not fixed**; recombination-risk split doesn't separate either. Residual STRICT false-contras land at ~0.83–0.92, so θc would need ≈ 0.90 (up from the clean-set 0.5): at θc=0.90 the data in hand gives 27/28 synthetic recombination recall, 0/26 synthetic legit FP, 0/10 smoke STRICT FP — but n=10 is far too small to set a threshold on. Remaining: a fuller `ARBORIST_NLI_SHADOW=1 make bench-qa` run → ~hundreds of STRICT cells → sweep θc/θe → confirm → set it; until then θc stays 0.5 and runtime NLI demotion stays off. Standing lesson: the clean synthetic eval (§7 #18) does not predict real-traffic precision. Production verifier unchanged; `falsification-hard` stays 10/12. #000048 follow-up | 2026-05-12 | — | @@ -155,4 +156,4 @@ Newest first. Update on every open/close. ## Next ID -`000052` +`000053` diff --git a/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md b/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md new file mode 100644 index 0000000..07d5cb3 --- /dev/null +++ b/docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md @@ -0,0 +1,310 @@ +# Ticket #000052 — Relevance + coherence meta-cognition (answer-*shape* signals) + +**Status:** open · awaiting go/no-go · doc-only scaffold +**Opened:** 2026-05-12 +**Scope:** Decide whether — and if so how — to add two more read-only +meta-cognition sidecars to the family that already includes +`diagnose_deflection` (topic-drift), `diagnose_metaphor_deflection`, +`diagnose_title_relevance` (Rule-8 hard check + legacy sidecar), the +soft-preflight hints, and the `QuestionState` metacognition: + +1. **`diagnose_relevance`** — is the answer actually *about* the + question, and *about* its cited sources? Today's checks are + *lexical* (subject-anchor token overlap, stemmed title-stem + overlap) — they pass on token collisions (a source titled "Zionist + entity" shares the token "entity" with a garbled claim about the + abstract noun "entity"). A small *aboutness / relevance* model (a + cross-encoder reranker or sentence-similarity model — **not** NLI; + NLI is entailment, this is topicality) would give a semantic + relevance score, under the same discipline cage as #000049's NLI. +2. **`diagnose_coherence`** — is the answer a well-formed assertion, + or word-salad / circular ("X is X") / vacuously true / too + underspecified to be a claim? Likely lexical/parse-shape, no model + needed — stays in the existing sidecar discipline. + +Both are **read-only signals: they demote, never promote, never enter +`providence_cache` / `audit_events` / `run_dag_root` / the proof +path** — exactly like the existing `inspect.py` diagnostics. Doc-only +proposal: the problem, the two capabilities, the sidecar-vs-model +split, the discipline, a recommendation. No code in this ticket. +**Audience:** fox + maintainers of `arborist/qa/inspect.py` + +`arborist/qa/verify.py` + #000049 readers + anyone weighing "another +ML model in the proof-adjacent path". +**Hard constraint:** if a *model* lands here (the relevance one), it +inherits #000049 §7's cage verbatim — demotion-only +(`MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`), +hash-pinned + reproducible (fixed checkpoint/tokenizer/dtype/CPU), +`relevance_model_version` folds into `governance_policy_hash` *iff* it +ever influences `audit_mode`, shadow-first + bench-gated, behind a +`[…]` extra so a fresh checkout stays `python3.12 + venv + sqlite3`, +and the #000049 §7 #20 "haystack" lesson applies — never run it over +the whole assembled context, only over candidate clauses / cited +sources. The lexical `diagnose_coherence` inherits the *existing* +sidecar discipline (no model, no proof-path write, advisory render- +tail only). + +--- + +## 1. Problem statement — groundedness ≠ epistemic well-formedness + +arborist's verifier answers exactly one question: **are the answer's +tokens present in the cited source?** (`verify_quotes` → STRICT / +HYBRID / UNGROUNDED, the four-rung ladder for lattice modes.) It is +deliberately, correctly narrow. But an answer can be perfectly +*grounded* and still be a bad answer along orthogonal axes: + +- **off-topic** — the answer addresses something other than the + question (`diagnose_deflection` catches the obvious cases via + subject-anchor token overlap; it's lexical, so a token collision + defeats it); +- **mis-sourced** — the answer's claim is cited to a structurally + unrelated source (Rule 8 / `_claim_title_overlap` + + `diagnose_title_relevance` catch this via stemmed title-stem + overlap; also lexical — a source titled with a shared token passes); +- **incoherent / underspecified** — the answer is word-salad, + circular, or vacuously true; *nothing* checks this today. + +**The motivating field case (2026-05-12, fox).** A vague 2-token-ish +query about "phrase" + "entity" → title-token-soup retrieval over +everything titled `Phrase` or `Entity` (`Zionist entity`, `Phrase`, +`Entity`, `Sensual Phrase`, `Entity Paradigm`, `Phrase book`, +`23 skidoo (phrase)`, `Group entity`). The `Zionist entity` article +was pulled in purely because its title contains "entity" and its body +opens "...is **a phrase** sometimes used by Arabs and Muslims as a +pejorative for the State of Israel." The `claim_lattice` LLM then +garbled that into the rendered claim *"the phrase 'Zionist entity' is +sometimes used **as the entity**, referring to the State of Israel"* +— grammatically and semantically incoherent (it conflated the +*string* "entity" inside the phrase with the *abstract noun* "entity" +the query was about). The verifier scored it +`EVIDENCE-WARRANTED-PARTIAL 2/3` because every content token of the +rendered claim *is* in the cited span — the lexical machinery cannot +tell the recombination is meaningless. `diagnose_deflection` didn't +fire (the answer does contain the question's tokens), and +`diagnose_title_relevance` didn't fire (the cited title is literally +"Zionist entity", which stem-overlaps the claim). Two lexical +relevance checks, both defeated by the same token collision. + +**The split #000049 surfaced.** #000049's NLI (contradiction veto) +catches *contradiction-recombination* ("Mercury is the largest" from a +source saying Jupiter is largest and Mercury smallest) — but it +*cannot* catch this case: feed NLI `(premise = the Zionist-entity +sentence, hypothesis = the garbled claim)` and it returns **neutral**, +not contradiction — the garbled claim isn't *contradicted* by the +source, it's just incoherent / off-topic. NLI checks "the source says +the opposite"; it does not check "the answer is word-salad" or "the +answer is about a different thing than the question." So the +recombination class has (at least) two sub-types, needing different +tools: + +| sub-type | example | tool | +|---|---|---| +| contradiction-recombination | "Mercury is the largest" ⟂ "Jupiter is largest; Mercury smallest" | NLI veto (#000049 — *if* its haystack/threshold problems get solved) | +| incoherence / topic-collision recombination | "the phrase ... is used as the entity" (token salad) | **this ticket** — relevance + coherence sidecars | + +This ticket is the home for the second column. + +--- + +## 2. The meta-cognition family — where these two fit + +arborist already does meta-cognition: signals *about* the answer that +are orthogonal to whether its tokens are grounded. The verifier checks +**groundedness** (tokens present, attached to a cited span). The +sidecars check **shape**: + +- `diagnose_deflection` — does the answer's *subject anchor* match the + question's? (topic-drift) +- `diagnose_metaphor_deflection` — did the answer slide into a + figurative reading of a literal question? +- `diagnose_title_relevance` / Rule 8 — is the cited source's *title* + topically related to the claim? +- soft-preflight hints, `QuestionState` — false-premise hints, + contradiction-pair detection, quantifier intensity, temporal + sensitivity. + +#000049's NLI adds an **attribution** signal (tokens attached to the +*right* subject in the source). This ticket adds two more **shape** +signals: + +- **relevance** — semantic (not just lexical) "aboutness": does the + answer address the question; is each claim about its cited source? +- **coherence** — is the answer a well-formed assertion at all? + +All of these are the same kind of thing: **read-only, demote-only, +never-in-the-proof-path advisory signals** that surface on the render +tail (`· deflection`, `· title mismatch`, `· soft: …`) and feed bench +tracking. The verifier stays binary; the sidecars carry the soft +signal. (CLAUDE.md "Conventions — Verifier stays binary; falsifications +carry soft signal" / "Soft hash vs hard hash".) + +--- + +## 3. Design — two capabilities, probably two sidecars + +### 3.1 — `diagnose_coherence(answer_text)` — lexical, no model + +Detect answers that are not well-formed assertions: + +- **circular / tautological** — "X is X", "a phrase is a phrase", + "the entity is the entity referring to …" (subject NP ≈ a predicate + NP, modulo articles/stopwords); +- **token-salad recombination** — a sentence whose content tokens are + a permutation of a source span's tokens but whose subject-predicate + structure is incoherent ("the phrase ... is used as the entity" — a + subject "the phrase" with a predicate "used as the entity" that has + no referent). Heuristic: shallow subject/predicate extraction + + "does the predicate's head noun appear as a free-standing referent + with no determiner/role"? +- **vacuous / underspecified** — "X is a thing", "X is something used + in some contexts", "X refers to a concept" (predicate is a pure + hypernym placeholder with no differentia); +- **self-reference loop** — the answer defines a term using the term. + +Pure lexical / parse-shape — reuses the stemming + content-token +machinery already in `inspect.py`. Same discipline as +`diagnose_deflection`: returns a dict (`{coherence_kind: "circular" | +"token_salad" | "vacuous" | "ok", evidence: …}`); the renderer can +surface `· incoherent: `; *optionally* the verifier could demote +STRICT → HYBRID on a hard `coherence_kind` (the same way Rule 8 +demotes on `TITLE_MISMATCH`) — that's a policy flag, off by default, +bench-gated, folds into `governance_policy_hash` if enabled. Likely +the cheapest, highest-confidence win here; deletion-first — try this +before reaching for a model. + +### 3.2 — `diagnose_relevance(question, answer, cited)` — the model question + +Today's relevance checks are lexical (token/stem overlap); they fail +on collisions. A small *aboutness* model would give a semantic score: + +- **input pairs**: `(question, answer)` for question-relevance; + `(claim, cited_source_span_or_title)` for source-relevance — run + **only** on the candidate clauses / cited sources, never the whole + context (#000049 §7 #20 haystack lesson — same multiple-comparisons + trap); +- **model**: a sentence-similarity / cross-encoder *reranker* + (`ms-marco`-style, or a small `sentence-transformers` bi/cross- + encoder) — **not** an NLI model. NLI labels entailment/contradiction; + this needs a topicality score. (If #000049's `[nli]` extra already + pulls `transformers`+`torch`, this can share it — record which + checkpoint, separate `relevance_model_version`.) +- **decision**: a demotion-only veto — if the answer's *max* relevance + to the question across its sentences is below θ_rel, or a claim's + relevance to *every* cited source is below θ_src, flag (and, + policy-permitting, demote STRICT → HYBRID). Mirror #000049's + two-threshold caution and shadow-first protocol exactly. +- **cage**: identical to #000049 §7 — demotion-only, hash-pinned + + reproducible, `relevance_model_version` + thresholds into + `governance_policy_hash` iff it touches `audit_mode`, shadow mode + (`ARBORIST_RELEVANCE_SHADOW=1` → telemetry, no `audit_mode` effect) + before any runtime use, bench-gated on a real `bench-qa` sample + (false-positive rate on legit on-topic answers must be near zero — + and #000049 §7 #21's lesson stands: the clean-eval number won't + predict the real-traffic one). + +### 3.3 — relationship to #000049 + +NLI (contradiction) and a relevance model (aboutness) are **siblings** +— two small ML models in the proof-adjacent path, both inside the same +discipline cage, both shadow-first. Reuse #000049 §7's manifest / +governance-hash / shadow-sweep / `make bench-…-shadow` pattern as the +template; do not re-derive it here. If #000049's runtime NLI never +ships (its real-traffic FP rate is the open question), the relevance +model is independently decidable — and `diagnose_coherence` (no model) +is independent of both. + +--- + +## 4. Recommendation + +**Build `diagnose_coherence` (lexical, §3.1) first; treat +`diagnose_relevance` (§3.2) as the model question to settle alongside +#000049's, not before it.** Rationale (the five-step algorithm — +delete/simplify before reaching for a model): + +1. The motivating case ("Zionist entity") is *incoherence* + *token- + collision*, and a chunk of it (the "used as the entity" salad) is + catchable by a *lexical* coherence-shape check — no model, no + dependency, fits the existing `inspect.py` discipline. Do that. +2. The token-collision-in-retrieval root cause (a vague polysemous + query → title-token-soup retrieval) is arguably better fixed + *upstream* in the retrieval pipeline (a polysemy/disambiguation + pass, or a query-specificity guard that refuses to answer a + 2-token query that fans out to N unrelated articles) than + downstream by a relevance veto. Worth its own retrieval ticket if + fox wants it tracked — note here, don't scope-creep. +3. A relevance *model* is the right tool for "semantically off-topic + despite token overlap", but it carries the same costs as #000049's + NLI (dependency, non-determinism surface, governance-hash, the + real-traffic-FP unknown) — only worth it if the lexical coherence + check + the existing deflection/title sidecars + an upstream + retrieval fix together still leave a measurable real-traffic + incoherence/off-topic rate. Park it behind that evidence; when it's + warranted, it ships under #000049's cage, shadow-first. + +So: §3.1 is a near-term, model-free win; §3.2 is gated on evidence and +travels with #000049's discipline; the upstream retrieval angle (#4.2) +is a flag, not a deliverable here. + +--- + +## 5. Out of scope + +- **The retrieval-side polysemy / title-token-soup fix** — the root + cause of the motivating case; a separate retrieval ticket if + tracked. This ticket is downstream answer-shape signals only. +- **Replacing the lexical verifier** — untouched; these are *added* + read-only sidecars, the verifier stays binary and first-line. +- **#000049's NLI contradiction veto** — sibling, not this; cross-ref + only. +- **A general "answer quality" score / RLHF-style reward model** — far + bigger; these are two narrow, named, demote-only signals. +- **Touching `providence_cache` / `run_dag_root`** — sidecars never + do; if a *hard* coherence/relevance demotion is enabled, it changes + which `audit_mode` `verify_quotes` returns (like Rule 8) and folds + into `governance_policy_hash`, but the *sidecar dicts* stay + render-layer + bench-tracking only. + +--- + +## 6. Status + +**Open · awaiting go/no-go.** Doc-only scaffold. First decision: build +the lexical `diagnose_coherence` (§3.1) now? — low-risk, model-free, +fits `inspect.py`; the recommendation is yes. Second decision (gated +on evidence, travels with #000049): is a relevance *model* (§3.2) +warranted, and if so it ships under #000049 §7's cage, shadow-first. +Third (a flag, not this ticket): does fox want the upstream retrieval +polysemy fix tracked? Until then: nothing lands; the existing +deflection / title-relevance / metaphor sidecars stand as the only +shape signals. + +--- + +## 7. References + +- `arborist/qa/inspect.py` — `diagnose_deflection`, + `diagnose_metaphor_deflection`, `diagnose_title_relevance` (the + sidecar family these two join); `_stem_for_deflection` / + `_content_tokens_for_deflection` / `_content_tokens_in_order` (the + lexical machinery `diagnose_coherence` would reuse). +- `arborist/qa/verify.py` — Rule 8 / `_claim_title_overlap` (the + hard-check precedent: a sidecar signal promoted to a demote-only + verifier rule, folded into `governance_policy_hash`). +- `docs/tickets/ticket-000049-attribution-aware-grounding-check.md` + §7 — the discipline cage (manifest, governance-hash rule, + shadow-first, `[…]` extra, the haystack lesson) this ticket's + §3.2 inherits verbatim; §7 #20/#21 = why "clean eval ≠ real-traffic + precision" and "never run over the whole context". +- `docs/tickets/ticket-000002-reference-frame-polarity-contract.md`, + `docs/tickets/ticket-000003-anchor-class-warrant-generalization.md` + — earlier "NLI deferred, do the cheap lexical thing first" + decisions; same instinct applies to §3.1-before-§3.2. +- The motivating field case: a `claim_lattice` query that returned + *"the phrase 'Zionist entity' is sometimes used as the entity, + referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL + 2/3` (2026-05-12, fox) — incoherent + token-collision recombination + that every existing lexical relevance check waved through. +- CLAUDE.md "Conventions — Verifier stays binary; falsifications carry + soft signal" / "Soft hash vs hard hash" — the read-only-sidecar + discipline both new signals obey.