# Ticket #000052 — Relevance + coherence meta-cognition (answer-*shape* signals)

**Status:** in progress — §3.1 `diagnose_coherence` landed (lexical, no
model: `circular` / `phrase_component_reuse` / `vacuous`; sidecar in
`arborist/qa/inspect.py`, surfaced in `inspect_cache_key` + the
`arborist inspect` human view as `· incoherent: <kind>`; 9 tests in
`tests/test_inspect.py`; demote-policy hook deliberately NOT wired —
advisory only, off the proof path). **§3.2 SHADOW SCAFFOLD LANDED
2026-05-13** (`arborist/qa/relevance/` mirroring `arborist/qa/nli/` —
manifest pins `cross-encoder/ms-marco-MiniLM-L-6-v2` as primary + 3
alternates; `ShadowRelevance` w/ `check_question_answer` and
`check_claim_source`; shares the `[nli]` extra; `demote_below_score`
intentionally `null` until a bench-gated shadow sweep sets it per the
#000049 §7 #18→#27 discipline; 13 tests). Sanity-checked on the
motivating field case: the Zionist-entity ON-topic / OFF-topic pair
scores +9.96 vs −9.04 — 18-point margin, the reranker cleanly catches
exactly the topic-collision mis-cite failure mode (and the Mona Lisa
Q→A deflection case at +10.45 vs +3.56). It does NOT catch the
*recombination*-where-the-different-entity-clause-mentions-the-target-
subject case (the Kilimanjaro/Mount Kenya pattern from #000049 §7 #26)
— and that's the right architectural split: §3.2 is *aboutness*, §000049
is *entailment*; they're orthogonal axes. §3.2 *threshold-setting and
runtime promotion* still gated on the bench-protocol in §3.2.2. §4.2
(upstream retrieval polysemy fix) still a flag, not scoped here.
**Opened:** 2026-05-12
**Scope:** Decide whether — and if so how — to add two more read-only
meta-cognition sidecars to the family that already includes
`diagnose_deflection` (topic-drift), `diagnose_metaphor_deflection`,
`diagnose_title_relevance` (Rule-8 hard check + legacy sidecar), the
soft-preflight hints, and the `QuestionState` metacognition:

1. **`diagnose_relevance`** — is the answer actually *about* the
   question, and *about* its cited sources? Today's checks are
   *lexical* (subject-anchor token overlap, stemmed title-stem
   overlap) — they pass on token collisions (a source titled "Zionist
   entity" shares the token "entity" with a garbled claim about the
   abstract noun "entity"). A small *aboutness / relevance* model (a
   cross-encoder reranker or sentence-similarity model — **not** NLI;
   NLI is entailment, this is topicality) would give a semantic
   relevance score, under the same discipline cage as #000049's NLI.
2. **`diagnose_coherence`** — is the answer a well-formed assertion,
   or word-salad / circular ("X is X") / vacuously true / too
   underspecified to be a claim? Likely lexical/parse-shape, no model
   needed — stays in the existing sidecar discipline.

Both are **read-only signals: they demote, never promote, never enter
`providence_cache` / `audit_events` / `run_dag_root` / the proof
path** — exactly like the existing `inspect.py` diagnostics. Doc-only
proposal: the problem, the two capabilities, the sidecar-vs-model
split, the discipline, a recommendation. No code in this ticket.
**Audience:** fox + maintainers of `arborist/qa/inspect.py` +
`arborist/qa/verify.py` + #000049 readers + anyone weighing "another
ML model in the proof-adjacent path".
**Hard constraint:** if a *model* lands here (the relevance one), it
inherits #000049 §7's cage verbatim — demotion-only
(`MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`),
hash-pinned + reproducible (fixed checkpoint/tokenizer/dtype/CPU),
`relevance_model_version` folds into `governance_policy_hash` *iff* it
ever influences `audit_mode`, shadow-first + bench-gated, behind a
`[…]` extra so a fresh checkout stays `python3.12 + venv + sqlite3`,
and the #000049 §7 #20 "haystack" lesson applies — never run it over
the whole assembled context, only over candidate clauses / cited
sources. The lexical `diagnose_coherence` inherits the *existing*
sidecar discipline (no model, no proof-path write, advisory render-
tail only).

---

## 1. Problem statement — groundedness ≠ epistemic well-formedness

arborist's verifier answers exactly one question: **are the answer's
tokens present in the cited source?** (`verify_quotes` → STRICT /
HYBRID / UNGROUNDED, the four-rung ladder for lattice modes.) It is
deliberately, correctly narrow. But an answer can be perfectly
*grounded* and still be a bad answer along orthogonal axes:

- **off-topic** — the answer addresses something other than the
  question (`diagnose_deflection` catches the obvious cases via
  subject-anchor token overlap; it's lexical, so a token collision
  defeats it);
- **mis-sourced** — the answer's claim is cited to a structurally
  unrelated source (Rule 8 / `_claim_title_overlap` +
  `diagnose_title_relevance` catch this via stemmed title-stem
  overlap; also lexical — a source titled with a shared token passes);
- **incoherent / underspecified** — the answer is word-salad,
  circular, or vacuously true; *nothing* checks this today.

**The motivating field case (2026-05-12, fox).** A vague 2-token-ish
query about "phrase" + "entity" → title-token-soup retrieval over
everything titled `Phrase` or `Entity` (`Zionist entity`, `Phrase`,
`Entity`, `Sensual Phrase`, `Entity Paradigm`, `Phrase book`,
`23 skidoo (phrase)`, `Group entity`). The `Zionist entity` article
was pulled in purely because its title contains "entity" and its body
opens "...is **a phrase** sometimes used by Arabs and Muslims as a
pejorative for the State of Israel." The `claim_lattice` LLM then
garbled that into the rendered claim *"the phrase 'Zionist entity' is
sometimes used **as the entity**, referring to the State of Israel"*
— grammatically and semantically incoherent (it conflated the
*string* "entity" inside the phrase with the *abstract noun* "entity"
the query was about). The verifier scored it
`EVIDENCE-WARRANTED-PARTIAL 2/3` because every content token of the
rendered claim *is* in the cited span — the lexical machinery cannot
tell the recombination is meaningless. `diagnose_deflection` didn't
fire (the answer does contain the question's tokens), and
`diagnose_title_relevance` didn't fire (the cited title is literally
"Zionist entity", which stem-overlaps the claim). Two lexical
relevance checks, both defeated by the same token collision.

**The split #000049 surfaced.** #000049's NLI (contradiction veto)
catches *contradiction-recombination* ("Mercury is the largest" from a
source saying Jupiter is largest and Mercury smallest) — but it
*cannot* catch this case: feed NLI `(premise = the Zionist-entity
sentence, hypothesis = the garbled claim)` and it returns **neutral**,
not contradiction — the garbled claim isn't *contradicted* by the
source, it's just incoherent / off-topic. NLI checks "the source says
the opposite"; it does not check "the answer is word-salad" or "the
answer is about a different thing than the question." So the
recombination class has (at least) two sub-types, needing different
tools:

| sub-type | example | tool |
|---|---|---|
| contradiction-recombination | "Mercury is the largest" ⟂ "Jupiter is largest; Mercury smallest" | NLI veto (#000049 — *if* its haystack/threshold problems get solved) |
| incoherence / topic-collision recombination | "the phrase ... is used as the entity" (token salad) | **this ticket** — relevance + coherence sidecars |

This ticket is the home for the second column.

---

## 2. The meta-cognition family — where these two fit

arborist already does meta-cognition: signals *about* the answer that
are orthogonal to whether its tokens are grounded. The verifier checks
**groundedness** (tokens present, attached to a cited span). The
sidecars check **shape**:

- `diagnose_deflection` — does the answer's *subject anchor* match the
  question's? (topic-drift)
- `diagnose_metaphor_deflection` — did the answer slide into a
  figurative reading of a literal question?
- `diagnose_title_relevance` / Rule 8 — is the cited source's *title*
  topically related to the claim?
- soft-preflight hints, `QuestionState` — false-premise hints,
  contradiction-pair detection, quantifier intensity, temporal
  sensitivity.

#000049's NLI adds an **attribution** signal (tokens attached to the
*right* subject in the source). This ticket adds two more **shape**
signals:

- **relevance** — semantic (not just lexical) "aboutness": does the
  answer address the question; is each claim about its cited source?
- **coherence** — is the answer a well-formed assertion at all?

All of these are the same kind of thing: **read-only, demote-only,
never-in-the-proof-path advisory signals** that surface on the render
tail (`· deflection`, `· title mismatch`, `· soft: …`) and feed bench
tracking. The verifier stays binary; the sidecars carry the soft
signal. (CLAUDE.md "Conventions — Verifier stays binary; falsifications
carry soft signal" / "Soft hash vs hard hash".)

---

## 3. Design — two capabilities, probably two sidecars

### 3.1 — `diagnose_coherence(answer_text)` — lexical, no model

Detect answers that are not well-formed assertions:

- **circular / tautological** — "X is X", "a phrase is a phrase",
  "the entity is the entity referring to …" (subject NP ≈ a predicate
  NP, modulo articles/stopwords);
- **token-salad recombination** — a sentence whose content tokens are
  a permutation of a source span's tokens but whose subject-predicate
  structure is incoherent ("the phrase ... is used as the entity" — a
  subject "the phrase" with a predicate "used as the entity" that has
  no referent). Heuristic: shallow subject/predicate extraction +
  "does the predicate's head noun appear as a free-standing referent
  with no determiner/role"?
- **vacuous / underspecified** — "X is a thing", "X is something used
  in some contexts", "X refers to a concept" (predicate is a pure
  hypernym placeholder with no differentia);
- **self-reference loop** — the answer defines a term using the term.

Pure lexical / parse-shape — reuses the stemming + content-token
machinery already in `inspect.py`. Same discipline as
`diagnose_deflection`: returns a dict (`{coherence_kind: "circular" |
"token_salad" | "vacuous" | "ok", evidence: …}`); the renderer can
surface `· incoherent: <kind>`; *optionally* the verifier could demote
STRICT → HYBRID on a hard `coherence_kind` (the same way Rule 8
demotes on `TITLE_MISMATCH`) — that's a policy flag, off by default,
bench-gated, folds into `governance_policy_hash` if enabled. Likely
the cheapest, highest-confidence win here; deletion-first — try this
before reaching for a model.

### 3.2 — `diagnose_relevance(question, answer, cited)` — the model question

Today's relevance checks are lexical (token/stem overlap); they fail
on collisions. A small *aboutness* model would give a semantic score:

- **input pairs**: `(question, answer)` for question-relevance;
  `(claim, cited_source_span_or_title)` for source-relevance — run
  **only** on the candidate clauses / cited sources, never the whole
  context (#000049 §7 #20 haystack lesson — same multiple-comparisons
  trap);
- **model**: a sentence-similarity / cross-encoder *reranker*
  (`ms-marco`-style, or a small `sentence-transformers` bi/cross-
  encoder) — **not** an NLI model. NLI labels entailment/contradiction;
  this needs a topicality score. (If #000049's `[nli]` extra already
  pulls `transformers`+`torch`, this can share it — record which
  checkpoint, separate `relevance_model_version`.)
- **decision**: a demotion-only veto — if the answer's *max* relevance
  to the question across its sentences is below θ_rel, or a claim's
  relevance to *every* cited source is below θ_src, flag (and,
  policy-permitting, demote STRICT → HYBRID). Mirror #000049's
  two-threshold caution and shadow-first protocol exactly.
- **cage**: identical to #000049 §7 — demotion-only, hash-pinned +
  reproducible, `relevance_model_version` + thresholds into
  `governance_policy_hash` iff it touches `audit_mode`, shadow mode
  (`ARBORIST_RELEVANCE_SHADOW=1` → telemetry, no `audit_mode` effect)
  before any runtime use, bench-gated on a real `bench-qa` sample
  (false-positive rate on legit on-topic answers must be near zero —
  and #000049 §7 #21's lesson stands: the clean-eval number won't
  predict the real-traffic one).

#### 3.2.1 — scaffold landed (2026-05-13)

The §3.2 *shadow scaffold* is in place, mirroring `arborist/qa/nli/`'s
structure verbatim. Pieces:

- `arborist/qa/relevance/manifest.json` — pins
  `cross-encoder/ms-marco-MiniLM-L-6-v2` (~80 MB, Apache-2.0) as the
  primary candidate; three alternates documented
  (`ms-marco-MiniLM-L-12-v2`, `BAAI/bge-reranker-base`,
  `cross-encoder/ms-marco-electra-base`). Crucially, **the manifest's
  `demote_below_score` is `null`** — *no hardcoded threshold*. The
  bench-maxing discipline from #000049 §7 #18→#27 (proved six times
  over that clean-eval thresholds don't transfer to bench-qa
  precision *or* recall) requires the demote threshold to be set by a
  *shadow sweep against pooled real STRICT*, not by a literature
  number. Tests assert `demote_below_score is None` so it can't drift
  back to a guess.
- `arborist/qa/relevance/__init__.py` + `shadow.py` — `ShadowRelevance`
  class (lazy `transformers`+`torch` import behind the *shared* `[nli]`
  extra; same graceful-degrade-to-`available=False` contract; `cuda`
  auto-detect from `ARBORIST_RELEVANCE_DEVICE` or `ARBORIST_NLI_DEVICE`;
  batched `_score_batch`), with two surface methods:
  `check_question_answer(question, answer)` (catches deflection /
  topic-drift the lexical `diagnose_deflection` sidecar misses on
  token-collision questions) and `check_claim_source(claim, source)`
  (the Zionist-entity-style mis-cite). Both return a single relevance
  *logit* (higher = more relevant; **not a probability**) and a
  `would_demote` flag that stays `False` until a threshold is set.
- `tests/test_relevance_shadow.py` — <!--AUTOCOUNT:tests:tests/test_relevance_shadow.py-->20<!--/AUTOCOUNT--> tests (pure-Python + graceful
  degradation + the Zionist sanity check); all green.

**Candidate-bench round 2 (2026-05-13)** — 26-pair hand-built fixture
set (`bench/fixtures/5f/relevance-aboutness-v1.jsonl`: 14 on-topic POS
+ 12 off-topic / topic-collision NEG, incl. the Zionist field case),
**7 candidates spanning ~20× in size (30 MB → 560 MB)** swept on the
4090 (~1 s per model; `bench/results/relevance-aboutness-candidatebench-round2.json`).
Ranked by separation margin (= `min(POS_score) − max(NEG_score)`;
>0 = clean-separable):

| model | size | min(POS) | max(NEG) | **sep margin** | catch on NEG @ fp=0 |
|---|---|---|---|---|---|
| **`BAAI/bge-reranker-large`** | 560 MB | +4.64 | −1.64 | **+6.285** | 12/12 |
| `cross-encoder/ms-marco-electra-base` | 110 MB | +0.05 | −5.30 | +5.349 | 12/12 |
| `cross-encoder/ms-marco-MiniLM-L-4-v2` | 50 MB | +6.74 | +3.55 | +3.187 | 12/12 |
| `BAAI/bge-reranker-base` | 280 MB | +2.40 | −0.68 | +3.076 | 12/12 |
| `cross-encoder/ms-marco-MiniLM-L-6-v2` | 80 MB | +6.75 | +3.70 | +3.053 | 12/12 |
| `cross-encoder/ms-marco-MiniLM-L-12-v2` | 130 MB | +7.14 | +4.48 | +2.663 | 12/12 |
| **`cross-encoder/ms-marco-MiniLM-L-2-v2`** | **30 MB** | +3.37 | +5.34 | **−1.970** | **10/12 (NOT separable)** |

**Real findings from extending the spectrum** (the 4-model round-1
table earlier in this section was the cost-pick story; round-2 is
the *science* story):

- **Biggest is best within a family** — `bge-reranker-large` (560 MB)
  has +6.285 margin vs `bge-reranker-base` (280 MB) at +3.076.
  Doubling size doubled margin within BGE. **But not across families**:
  `electra-base` (110 MB) at +5.349 *beats* `bge-reranker-base`
  (280 MB) at +3.076 — 60% smaller, ~75% more margin. *Specific
  checkpoint / architecture / training corpus dominates parameter
  count in this size range.*
- **Within the MS-MARCO MiniLM family, depth is non-monotonic**:
  **L-4 (50 MB, +3.187) > L-6 (80 MB, +3.053) > L-12 (130 MB, +2.663)**.
  Adding depth from L-6 to L-12 *hurts*. Same "deeper ≠ better"
  pattern as #000049 §7 #24's bart-large-vs-deberta-large finding,
  reproduced on a different model family.
- **Capacity floor is between L-2 and L-4**. `MiniLM-L-2-v2` (30 MB)
  is the *only* model that fails — margin **−1.970**, NOT
  clean-separable. Some NEG outscore some POS. At its best fp=0
  threshold it catches 10/12, not 12/12. L-4 (50 MB) separates
  cleanly. So somewhere around 40–50 MB / ≥4 layers there's a phase
  transition for this task.
- **All 6 non-floor candidates clean-separate** the 26-pair
  candidate-bench. So the "reranker catches Zionist-style topic
  mis-cites" claim is a *property of competent rerankers as a class*,
  not the specific checkpoint — once you're above the capacity floor.

**Manifest primary moves to `BAAI/bge-reranker-large`** for the most
cushion (+6.285 — the model most likely to survive a denominator
growth per #000049 §7 #25's lesson), with `ms-marco-electra-base` as
the cost alternate (5× smaller, +5.349 margin — 87% of bge-large's
cushion at 20% of the size; if the real-traffic shadow sweep at
§3.2.2 step 2 shows bge-large isn't worth the size, walk back).

**None of this predicts real-traffic behavior.** The #000049 §7
#18→#27 history is six verdict flips on the NLI side from candidate-
bench → real-bench-qa-pipeline; assume the same here. The thresholds
above (bge-large at θ=4.64, electra-base at θ=0.05) are
*candidate-bench numbers* and *will* move when the denominator
grows. The manifest's `demote_below_score` stays `null` until the
real-traffic shadow sweep (§3.2.2 step 2) sets it. **Expect a
walk-back.**

**What the candidate-bench does NOT establish** (preserved from the
earlier 6-pair sanity check — the architectural split remains the
same point, just with proper data behind the catch-rate claim):
the reranker does **NOT** catch the *recombination*-where-the-
different-entity-clause-also-mentions-the-target-subject case (the
Mount Kenya clause scores higher than the Kilimanjaro lead for a
"Kilimanjaro is in Kenya" claim) — because the Mount Kenya clause
is genuinely *about* Kilimanjaro-as-comparison and the reranker is
scoring *aboutness*, not *truth*. The right architectural split:

- **#000052 §3.2 relevance reranker** = catches *topic-collision /
  off-topic* failures (Zionist entity, Q→A deflection). ✓ shown.
- **#000049 NLI veto** = catches *contradiction-recombination*
  failures (Mercury / Einstein synthetic fixtures). ✓ shown (§7 #24).
- *Neither* by itself catches the *recombination-where-the-different-
  entity-clause-also-mentions-the-target-subject* case (the Mount
  Kenya / Kilimanjaro pattern from §7 #26's mining) — that needs the
  **semantic candidate selector** (vec-driven; #000050/#000051) so
  NLI doesn't get handed the wrong clause in the first place.

So §3.2's *aboutness* signal complements #000049's *entailment*
signal — they're orthogonal axes (topicality vs. truth-of-attribution)
and the two failure modes the sidecar family didn't previously cover
(incoherence and topic-collision) get cleanly handled by:
`diagnose_coherence` (§3.1, lexical, landed) + this relevance reranker
(§3.2, model, scaffold landed; threshold-setting and bench-gated
shadow run still TODO).

#### 3.2.2 — Remaining work (the bench-gated decision)

The scaffold is in place; the **threshold is intentionally unset**.
Before runtime promotion, the same #000049 §7 #18→#27 protocol applies:

1. *Build a small candidate-bench eval set* (~20-30 fixtures) of
   (question, answer) deflection cases + (claim, source) mis-cite
   cases (the Zionist-entity case is fixture 1); also a negative-
   control set of legit on-topic pairs. Measure each alternate model
   (L-6, L-12, BGE, electra) on it — pick the winner by *separation
   margin*, not raw score (per §7 #18's "model science").
2. *Shadow-run the winner against pooled real bench-qa STRICT*
   (~800+ cells) with a θ sweep; pick the θ where false-demote rate
   on STRICT is zero. (And expect a walk-back the moment the
   denominator grows — §7 #25→#26 → #27 happened three times on the
   NLI side; assume it will here too.)
3. *Recall-side realism check* against hand-built deflection/mis-cite
   claims constructed against *real* bench-qa contexts (#000049 §7
   #27 pattern — *don't* trust the candidate-bench's recall numbers
   to transfer).
4. Only after 1–3 set a real `demote_below_score`, switch the
   `recommended_operating_point` to active, fold
   `relevance_policy_hash` into `governance_policy_hash`, and put it
   in front of fox+dav1d for the runtime-promotion decision.

#### 3.2.2 — Step 2 result (2026-05-13): the candidate-bench θ values do not survive

Built `bench/scripts/relevance_shadow_sweep.py` (mirror of #000049
§7 #22's NLI shadow sweep). Scored every (question, answer) pair in
the 808-cell pooled bench-qa STRICT set with each manifest reranker;
measured the fraction of STRICT that would FIRE at the candidate-bench
fp=0 θ (= the load-bearing "did the candidate-bench θ survive
proper-n" number), and the smallest θ that yields fp=0 on the real
pool. Results (`bench/results/relevance-shadow-sweep-pooled808.json`,
all 6 swept models — ms-marco-electra-base wasn't in this manifest's
alternates list at the time and didn't run; the verdict is universal
across the 6 that did):

| model (size) | cb θ | **STRICT FP @ cb θ** | fp=0 needs θ ≤ | cb NEG max | catch on cb NEG @ fp=0 |
|---|---|---|---|---|---|
| **bge-reranker-large** (560 MB, primary) | +4.64 | **176/808 = 21.8%** | −2.20 | −1.64 | **0/12** |
| bge-reranker-base (280 MB) | +2.40 | 77/808 = 9.5% | −3.62 | −0.68 | 0/12 |
| MiniLM-L-12-v2 (130 MB) | +7.14 | 89/808 = 11.0% | −4.93 | +4.48 | 0/12 |
| MiniLM-L-6-v2 (80 MB) | +6.75 | 94/808 = 11.6% | −7.06 | +3.70 | 0/12 |
| MiniLM-L-4-v2 (50 MB) | +6.74 | 121/808 = 15.0% | −5.51 | +3.55 | 0/12 |
| MiniLM-L-2-v2 (30 MB) | +3.37 | 36/808 = 4.5% | −6.86 | +5.34 | 0/12 |

**Universal walk-back:** every model false-fires on 4.5–21.8% of real
STRICT at its candidate-bench θ. The smallest θ that yields fp=0 on
real STRICT is *below* the candidate-bench NEG max for every model —
meaning at the runtime-safe θ, the model catches **0/12** of the
candidate-bench off-topic NEG cases.

**Structural reason:** the candidate-bench POS (14 hand-crafted
on-topic Q-A pairs) score in a tight high band; the NEG (12
hand-crafted off-topic) in a tight low band; clean separation. But
real bench-qa STRICT answers have a **much wider score distribution**
(bge-large: min −2.20, p10 +2.99, p50 +5.80, p90 +7.54, max +8.27).
The bottom 10% of legitimate STRICT score below where the
candidate-bench NEG cases sat. The distributions **overlap heavily**
on real bench-qa output; no threshold separates them. This is the
exact same finding pattern as #000049 §7 #27's "real-haystack recall
is lower than synthetic" — but on the precision side for relevance.

**Verdict.** The relevance reranker **cannot be promoted to a runtime
demotion-only veto on this design** — it joins #000049 §7 #27 in
"clean on candidate-bench, fails the real-traffic gate." The score
*is* lower on average for off-topic than on-topic (as a soft signal
it's real), but **not separably-thresholdably so** on real bench-qa
heterogeneity. The manifest's `demote_below_score` stays `null` and
`runtime_viability.as_runtime_demotion_veto = "NOT VIABLE on this
design"`.

**What it CAN still do** (all viable without a threshold):
- **Advisory soft signal** (render-tail `· relevance: 0.42`) — surface
  the raw score; let humans judge. No FP problem because no decision
  is being made.
- **Multi-signal advisory** — combine with `diagnose_title_relevance`,
  `diagnose_deflection`, `diagnose_coherence` for a weighted
  warning. Each signal alone is noisy; the *intersection* might be
  cleaner.
- **Contrastive signal** — Q→A relevance *minus* claim→source
  relevance (or relative ranking *within* an answer's claims). The
  *delta* may discriminate where the absolute threshold does not.
  Untested, worth measuring next.

So §3.2 the *runtime veto* is parked; §3.2 the *soft signal* lives
on as an inspect.py sidecar (same render-layer-only discipline as
§3.1 `diagnose_coherence`). The bigger lesson, sharpened *again*:
the relevance and NLI sides of this work both hit the same wall —
**lexical-candidate selection + cross-encoder scoring + a hard
threshold doesn't survive real-pipeline heterogeneity**. The §7 #27
diagnosis (the candidate-clause selector is the architectural
bottleneck) applies here too: a *semantic* candidate selector
(vec-driven, sibling of #000050/#000051) might tighten both axes.

### 3.3 — relationship to #000049

NLI (contradiction) and a relevance model (aboutness) are **siblings**
— two small ML models in the proof-adjacent path, both inside the same
discipline cage, both shadow-first. Reuse #000049 §7's manifest /
governance-hash / shadow-sweep / `make bench-…-shadow` pattern as the
template; do not re-derive it here. If #000049's runtime NLI never
ships (its real-traffic FP rate is the open question), the relevance
model is independently decidable — and `diagnose_coherence` (no model)
is independent of both.

---

## 4. Recommendation

**Build `diagnose_coherence` (lexical, §3.1) first; treat
`diagnose_relevance` (§3.2) as the model question to settle alongside
#000049's, not before it.** Rationale (the five-step algorithm —
delete/simplify before reaching for a model):

1. The motivating case ("Zionist entity") is *incoherence* + *token-
   collision*, and a chunk of it (the "used as the entity" salad) is
   catchable by a *lexical* coherence-shape check — no model, no
   dependency, fits the existing `inspect.py` discipline. Do that.
2. The token-collision-in-retrieval root cause (a vague polysemous
   query → title-token-soup retrieval) is arguably better fixed
   *upstream* in the retrieval pipeline (a polysemy/disambiguation
   pass, or a query-specificity guard that refuses to answer a
   2-token query that fans out to N unrelated articles) than
   downstream by a relevance veto. Worth its own retrieval ticket if
   fox wants it tracked — note here, don't scope-creep.
3. A relevance *model* is the right tool for "semantically off-topic
   despite token overlap", but it carries the same costs as #000049's
   NLI (dependency, non-determinism surface, governance-hash, the
   real-traffic-FP unknown) — only worth it if the lexical coherence
   check + the existing deflection/title sidecars + an upstream
   retrieval fix together still leave a measurable real-traffic
   incoherence/off-topic rate. Park it behind that evidence; when it's
   warranted, it ships under #000049's cage, shadow-first.

So: §3.1 is a near-term, model-free win; §3.2 is gated on evidence and
travels with #000049's discipline; the upstream retrieval angle (#4.2)
is a flag, not a deliverable here.

---

## 5. Out of scope

- **The retrieval-side polysemy / title-token-soup fix** — the root
  cause of the motivating case; a separate retrieval ticket if
  tracked. This ticket is downstream answer-shape signals only.
- **Replacing the lexical verifier** — untouched; these are *added*
  read-only sidecars, the verifier stays binary and first-line.
- **#000049's NLI contradiction veto** — sibling, not this; cross-ref
  only.
- **A general "answer quality" score / RLHF-style reward model** — far
  bigger; these are two narrow, named, demote-only signals.
- **Touching `providence_cache` / `run_dag_root`** — sidecars never
  do; if a *hard* coherence/relevance demotion is enabled, it changes
  which `audit_mode` `verify_quotes` returns (like Rule 8) and folds
  into `governance_policy_hash`, but the *sidecar dicts* stay
  render-layer + bench-tracking only.

---

## 6. Status

**In progress.**

- **§3.1 `diagnose_coherence` — landed.** Lexical, model-free, in
  `arborist/qa/inspect.py`: per-sentence shape check emitting
  `kind ∈ {phrase_component_reuse, circular, vacuous, ok, empty}`
  (severity `phrase_component_reuse > circular > vacuous`).
  `phrase_component_reuse` is the one that catches the motivating
  field case — a subject that quotes a phrase, a predicate that reuses
  one of that phrase's own content tokens as a bare `the/a/an <token>`
  referent (copulas inside a quoted span are skipped so `'war is
  peace'` doesn't break the subject/predicate split). Surfaced in
  `inspect_cache_key`'s return dict (`"coherence": {...}`) and the
  `arborist inspect` human view (`· incoherent: <kind>`). 9 tests in
  `tests/test_inspect.py`. **Demote-policy hook deliberately NOT
  wired** — like every sibling sidecar this is advisory render-tail +
  bench telemetry only; if fox later wants the verifier to demote
  STRICT → HYBRID on a hard coherence_kind, that's a separate policy
  field that folds into `governance_policy_hash`.
- **§3.2 relevance *model* — still the open question.** Gated on
  evidence (does the lexical coherence check + the existing
  deflection/title sidecars + an upstream retrieval fix together still
  leave a measurable real-traffic incoherence/off-topic rate?) and
  travels with #000049's "may a fixed small model touch `audit_mode`?"
  decision. When warranted it ships under #000049 §7's cage,
  shadow-first.
- **§4.2 upstream retrieval polysemy fix — a flag, not this ticket.**
  Open its own retrieval ticket if fox wants it tracked.

---

## 7. References

- `arborist/qa/inspect.py` — `diagnose_deflection`,
  `diagnose_metaphor_deflection`, `diagnose_title_relevance` (the
  sidecar family these two join); `_stem_for_deflection` /
  `_content_tokens_for_deflection` / `_content_tokens_in_order` (the
  lexical machinery `diagnose_coherence` would reuse).
- `arborist/qa/verify.py` — Rule 8 / `_claim_title_overlap` (the
  hard-check precedent: a sidecar signal promoted to a demote-only
  verifier rule, folded into `governance_policy_hash`).
- `docs/tickets/ticket-000049-attribution-aware-grounding-check.md`
  §7 — the discipline cage (manifest, governance-hash rule,
  shadow-first, `[…]` extra, the haystack lesson) this ticket's
  §3.2 inherits verbatim; §7 #20/#21 = why "clean eval ≠ real-traffic
  precision" and "never run over the whole context".
- `docs/tickets/ticket-000002-reference-frame-polarity-contract.md`,
  `docs/tickets/ticket-000003-anchor-class-warrant-generalization.md`
  — earlier "NLI deferred, do the cheap lexical thing first"
  decisions; same instinct applies to §3.1-before-§3.2.
- The motivating field case: a `claim_lattice` query that returned
  *"the phrase 'Zionist entity' is sometimes used as the entity,
  referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL
  2/3` (2026-05-12, fox) — incoherent + token-collision recombination
  that every existing lexical relevance check waved through.
- CLAUDE.md "Conventions — Verifier stays binary; falsifications carry
  soft signal" / "Soft hash vs hard hash" — the read-only-sidecar
  discipline both new signals obey.
