#000052: open — relevance + coherence meta-cognition (answer-shape sidecars)
Doc-only scaffold. Two more read-only/demote-only/never-in-proof-path sidecars joining the diagnose_deflection family: (1) diagnose_coherence — word-salad/circular/vacuous answers; lexical, no model; the near-term win. (2) diagnose_relevance — semantic 'aboutness' (does the answer address the question / is each claim about its cited source?); today's checks are lexical and a token collision defeats them; a small aboutness/reranker model (NOT NLI) under #000049 §7's discipline cage verbatim; gated on evidence, travels with #000049's model question. Motivating field case: the 'Zionist entity' claim_lattice query — incoherent token-collision recombination NLI can't catch (returns neutral) and both lexical relevance checks waved through. Flags an upstream retrieval (polysemy/title-soup) root-cause ticket, not scoped here. Next ID 000052 → 000053. #000049 sibling
This commit is contained in:
parent
02f8dfec07
commit
59d8a17a14
2 changed files with 312 additions and 1 deletions
|
|
@ -101,6 +101,7 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000052 | Relevance + coherence meta-cognition (answer-*shape* signals) | open · awaiting go/no-go · doc-only scaffold. Two more read-only, demote-only, never-in-proof-path sidecars joining the `diagnose_deflection` / `diagnose_metaphor_deflection` / `diagnose_title_relevance` / soft-preflight family: **(1) `diagnose_coherence`** — word-salad / circular ("X is X") / vacuous / underspecified answers; lexical/parse-shape, **no model** — the near-term win (recommended now). **(2) `diagnose_relevance`** — semantic (not just lexical) "aboutness": does the answer address the question; is each claim about its cited source? Today's checks (subject-anchor token overlap, stemmed title-stem overlap) are *lexical* and a token collision defeats them — a small *aboutness/reranker* model (NOT NLI — entailment ≠ topicality) under #000049 §7's discipline cage verbatim (demotion-only, hash-pinned, `relevance_model_version`→`governance_policy_hash` iff it touches `audit_mode`, shadow-first, `[…]` extra, the §7 #20 haystack lesson — never over the whole context); gated on evidence, travels with #000049's model question. Motivating field case (2026-05-12, fox): the `claim_lattice` query that returned *"the phrase 'Zionist entity' is sometimes used as the entity, referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL 2/3` — incoherent + token-collision recombination that NLI can't catch (returns *neutral*, not *contradiction*) and both lexical relevance checks waved through. Flags an upstream retrieval ticket (polysemy / title-token-soup) as the root-cause fix, not scoped here. #000049 sibling | 2026-05-12 | — |
|
||||
| #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — |
|
||||
| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — |
|
||||
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. **Phase-2 candidate bench done 2026-05-12** (`~/git/arborist-nli-bench/`, commits `829f9a4` + `a1cb28d`; ticket §7 #18): checkpoint-agnostic harness runs the §7 #5 clause-level algorithm over 28 synth recombination cases (incl. the 2 fixtures + harder shapes) + 26 legit cases (true summaries + near-miss decoys). 4 working candidates; `nli-MiniLM2-L6-H768` (82M, 45ms p50 CPU), `deberta-v3-base-mnli-fever-anli` (184M, 223ms), `bart-large-mnli` (407M, 259ms) all 28/28 catch · 0/26 FP with the standard θe=0.9 entailment guard; `cross-encoder/nli-deberta-v3-base` 27/28; deberta-large repo-id TODO. **Key finding: the §7 #5 two-threshold rule is load-bearing** — 3 of 4 candidates argmax-contradict 1/26 legit cases on the *wrong* source clause (competing-superlative confusion, e.g. "largest hot desert" vs "largest desert overall"); the entailment guard filters every one because another clause restates the claim → 0% guarded FP vs ~4% single-threshold. Picture: recombination is *easy* for any modern NLI checkpoint — differentiator is cost/robustness, MiniLM is the cost-pick, bart-large the threshold-robust pick. **Phase-2 shadow scaffold landed in arborist 2026-05-12** (ticket §7 #19): `arborist/qa/nli/` (manifest pins MiniLM @ a fixed HF revision + θc 0.5/θe 0.9 + 2 alternates; `ShadowNLI`/`shadow_check` lazy-imports `transformers`+`torch` behind a new `[nli]` extra, degrades to `available=False` when absent — SHADOW ONLY, never an `audit_mode` input, manifest not yet in `governance_policy_hash` per §7 #2) + `bench/scripts/nli_shadow_sweep.py` + `make bootstrap-nli` / `make bench-nli-shadow` + 16 tests. Synthetic sweep (116 records): 28/28 recombination demoted, 0/26 FP on legit summaries, 0/9 fires on already-`STRICT_SPAN`. **First real-traffic sweep** (§7 #20 — `ARBORIST_NLI_SHADOW=1 make bench-qa-smoke`, 15 cells; `query.py` surfaces verifier-input text gated off-by-default, `qa_sweep.py` carries it, `nli_shadow_sweep.py` reads it): the *naive* "NLI on every context clause" scaffold has a **~30% would-demote rate on STRICT answers** — a haystack/multiple-comparisons artifact (real contexts → 100–336 clauses; `max`-over-all almost always hits a tangential "contradiction"). Candidate-clause restriction added (§7 #21 — NLI runs on the top-6 source clauses by content-token overlap, not the whole context; `max_candidate_clauses=6`): STRICT would-demote 30% → 20% on the smoke, overall 47% → 33% — **helps, not fixed**; recombination-risk split doesn't separate either. Residual STRICT false-contras land at ~0.83–0.92, so θc would need ≈ 0.90 (up from the clean-set 0.5): at θc=0.90 the data in hand gives 27/28 synthetic recombination recall, 0/26 synthetic legit FP, 0/10 smoke STRICT FP — but n=10 is far too small to set a threshold on. Remaining: a fuller `ARBORIST_NLI_SHADOW=1 make bench-qa` run → ~hundreds of STRICT cells → sweep θc/θe → confirm → set it; until then θc stays 0.5 and runtime NLI demotion stays off. Standing lesson: the clean synthetic eval (§7 #18) does not predict real-traffic precision. Production verifier unchanged; `falsification-hard` stays 10/12. #000048 follow-up | 2026-05-12 | — |
|
||||
|
|
@ -155,4 +156,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000052`
|
||||
`000053`
|
||||
|
|
|
|||
310
docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md
Normal file
310
docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md
Normal file
|
|
@ -0,0 +1,310 @@
|
|||
# Ticket #000052 — Relevance + coherence meta-cognition (answer-*shape* signals)
|
||||
|
||||
**Status:** open · awaiting go/no-go · doc-only scaffold
|
||||
**Opened:** 2026-05-12
|
||||
**Scope:** Decide whether — and if so how — to add two more read-only
|
||||
meta-cognition sidecars to the family that already includes
|
||||
`diagnose_deflection` (topic-drift), `diagnose_metaphor_deflection`,
|
||||
`diagnose_title_relevance` (Rule-8 hard check + legacy sidecar), the
|
||||
soft-preflight hints, and the `QuestionState` metacognition:
|
||||
|
||||
1. **`diagnose_relevance`** — is the answer actually *about* the
|
||||
question, and *about* its cited sources? Today's checks are
|
||||
*lexical* (subject-anchor token overlap, stemmed title-stem
|
||||
overlap) — they pass on token collisions (a source titled "Zionist
|
||||
entity" shares the token "entity" with a garbled claim about the
|
||||
abstract noun "entity"). A small *aboutness / relevance* model (a
|
||||
cross-encoder reranker or sentence-similarity model — **not** NLI;
|
||||
NLI is entailment, this is topicality) would give a semantic
|
||||
relevance score, under the same discipline cage as #000049's NLI.
|
||||
2. **`diagnose_coherence`** — is the answer a well-formed assertion,
|
||||
or word-salad / circular ("X is X") / vacuously true / too
|
||||
underspecified to be a claim? Likely lexical/parse-shape, no model
|
||||
needed — stays in the existing sidecar discipline.
|
||||
|
||||
Both are **read-only signals: they demote, never promote, never enter
|
||||
`providence_cache` / `audit_events` / `run_dag_root` / the proof
|
||||
path** — exactly like the existing `inspect.py` diagnostics. Doc-only
|
||||
proposal: the problem, the two capabilities, the sidecar-vs-model
|
||||
split, the discipline, a recommendation. No code in this ticket.
|
||||
**Audience:** fox + maintainers of `arborist/qa/inspect.py` +
|
||||
`arborist/qa/verify.py` + #000049 readers + anyone weighing "another
|
||||
ML model in the proof-adjacent path".
|
||||
**Hard constraint:** if a *model* lands here (the relevance one), it
|
||||
inherits #000049 §7's cage verbatim — demotion-only
|
||||
(`MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`),
|
||||
hash-pinned + reproducible (fixed checkpoint/tokenizer/dtype/CPU),
|
||||
`relevance_model_version` folds into `governance_policy_hash` *iff* it
|
||||
ever influences `audit_mode`, shadow-first + bench-gated, behind a
|
||||
`[…]` extra so a fresh checkout stays `python3.12 + venv + sqlite3`,
|
||||
and the #000049 §7 #20 "haystack" lesson applies — never run it over
|
||||
the whole assembled context, only over candidate clauses / cited
|
||||
sources. The lexical `diagnose_coherence` inherits the *existing*
|
||||
sidecar discipline (no model, no proof-path write, advisory render-
|
||||
tail only).
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement — groundedness ≠ epistemic well-formedness
|
||||
|
||||
arborist's verifier answers exactly one question: **are the answer's
|
||||
tokens present in the cited source?** (`verify_quotes` → STRICT /
|
||||
HYBRID / UNGROUNDED, the four-rung ladder for lattice modes.) It is
|
||||
deliberately, correctly narrow. But an answer can be perfectly
|
||||
*grounded* and still be a bad answer along orthogonal axes:
|
||||
|
||||
- **off-topic** — the answer addresses something other than the
|
||||
question (`diagnose_deflection` catches the obvious cases via
|
||||
subject-anchor token overlap; it's lexical, so a token collision
|
||||
defeats it);
|
||||
- **mis-sourced** — the answer's claim is cited to a structurally
|
||||
unrelated source (Rule 8 / `_claim_title_overlap` +
|
||||
`diagnose_title_relevance` catch this via stemmed title-stem
|
||||
overlap; also lexical — a source titled with a shared token passes);
|
||||
- **incoherent / underspecified** — the answer is word-salad,
|
||||
circular, or vacuously true; *nothing* checks this today.
|
||||
|
||||
**The motivating field case (2026-05-12, fox).** A vague 2-token-ish
|
||||
query about "phrase" + "entity" → title-token-soup retrieval over
|
||||
everything titled `Phrase` or `Entity` (`Zionist entity`, `Phrase`,
|
||||
`Entity`, `Sensual Phrase`, `Entity Paradigm`, `Phrase book`,
|
||||
`23 skidoo (phrase)`, `Group entity`). The `Zionist entity` article
|
||||
was pulled in purely because its title contains "entity" and its body
|
||||
opens "...is **a phrase** sometimes used by Arabs and Muslims as a
|
||||
pejorative for the State of Israel." The `claim_lattice` LLM then
|
||||
garbled that into the rendered claim *"the phrase 'Zionist entity' is
|
||||
sometimes used **as the entity**, referring to the State of Israel"*
|
||||
— grammatically and semantically incoherent (it conflated the
|
||||
*string* "entity" inside the phrase with the *abstract noun* "entity"
|
||||
the query was about). The verifier scored it
|
||||
`EVIDENCE-WARRANTED-PARTIAL 2/3` because every content token of the
|
||||
rendered claim *is* in the cited span — the lexical machinery cannot
|
||||
tell the recombination is meaningless. `diagnose_deflection` didn't
|
||||
fire (the answer does contain the question's tokens), and
|
||||
`diagnose_title_relevance` didn't fire (the cited title is literally
|
||||
"Zionist entity", which stem-overlaps the claim). Two lexical
|
||||
relevance checks, both defeated by the same token collision.
|
||||
|
||||
**The split #000049 surfaced.** #000049's NLI (contradiction veto)
|
||||
catches *contradiction-recombination* ("Mercury is the largest" from a
|
||||
source saying Jupiter is largest and Mercury smallest) — but it
|
||||
*cannot* catch this case: feed NLI `(premise = the Zionist-entity
|
||||
sentence, hypothesis = the garbled claim)` and it returns **neutral**,
|
||||
not contradiction — the garbled claim isn't *contradicted* by the
|
||||
source, it's just incoherent / off-topic. NLI checks "the source says
|
||||
the opposite"; it does not check "the answer is word-salad" or "the
|
||||
answer is about a different thing than the question." So the
|
||||
recombination class has (at least) two sub-types, needing different
|
||||
tools:
|
||||
|
||||
| sub-type | example | tool |
|
||||
|---|---|---|
|
||||
| contradiction-recombination | "Mercury is the largest" ⟂ "Jupiter is largest; Mercury smallest" | NLI veto (#000049 — *if* its haystack/threshold problems get solved) |
|
||||
| incoherence / topic-collision recombination | "the phrase ... is used as the entity" (token salad) | **this ticket** — relevance + coherence sidecars |
|
||||
|
||||
This ticket is the home for the second column.
|
||||
|
||||
---
|
||||
|
||||
## 2. The meta-cognition family — where these two fit
|
||||
|
||||
arborist already does meta-cognition: signals *about* the answer that
|
||||
are orthogonal to whether its tokens are grounded. The verifier checks
|
||||
**groundedness** (tokens present, attached to a cited span). The
|
||||
sidecars check **shape**:
|
||||
|
||||
- `diagnose_deflection` — does the answer's *subject anchor* match the
|
||||
question's? (topic-drift)
|
||||
- `diagnose_metaphor_deflection` — did the answer slide into a
|
||||
figurative reading of a literal question?
|
||||
- `diagnose_title_relevance` / Rule 8 — is the cited source's *title*
|
||||
topically related to the claim?
|
||||
- soft-preflight hints, `QuestionState` — false-premise hints,
|
||||
contradiction-pair detection, quantifier intensity, temporal
|
||||
sensitivity.
|
||||
|
||||
#000049's NLI adds an **attribution** signal (tokens attached to the
|
||||
*right* subject in the source). This ticket adds two more **shape**
|
||||
signals:
|
||||
|
||||
- **relevance** — semantic (not just lexical) "aboutness": does the
|
||||
answer address the question; is each claim about its cited source?
|
||||
- **coherence** — is the answer a well-formed assertion at all?
|
||||
|
||||
All of these are the same kind of thing: **read-only, demote-only,
|
||||
never-in-the-proof-path advisory signals** that surface on the render
|
||||
tail (`· deflection`, `· title mismatch`, `· soft: …`) and feed bench
|
||||
tracking. The verifier stays binary; the sidecars carry the soft
|
||||
signal. (CLAUDE.md "Conventions — Verifier stays binary; falsifications
|
||||
carry soft signal" / "Soft hash vs hard hash".)
|
||||
|
||||
---
|
||||
|
||||
## 3. Design — two capabilities, probably two sidecars
|
||||
|
||||
### 3.1 — `diagnose_coherence(answer_text)` — lexical, no model
|
||||
|
||||
Detect answers that are not well-formed assertions:
|
||||
|
||||
- **circular / tautological** — "X is X", "a phrase is a phrase",
|
||||
"the entity is the entity referring to …" (subject NP ≈ a predicate
|
||||
NP, modulo articles/stopwords);
|
||||
- **token-salad recombination** — a sentence whose content tokens are
|
||||
a permutation of a source span's tokens but whose subject-predicate
|
||||
structure is incoherent ("the phrase ... is used as the entity" — a
|
||||
subject "the phrase" with a predicate "used as the entity" that has
|
||||
no referent). Heuristic: shallow subject/predicate extraction +
|
||||
"does the predicate's head noun appear as a free-standing referent
|
||||
with no determiner/role"?
|
||||
- **vacuous / underspecified** — "X is a thing", "X is something used
|
||||
in some contexts", "X refers to a concept" (predicate is a pure
|
||||
hypernym placeholder with no differentia);
|
||||
- **self-reference loop** — the answer defines a term using the term.
|
||||
|
||||
Pure lexical / parse-shape — reuses the stemming + content-token
|
||||
machinery already in `inspect.py`. Same discipline as
|
||||
`diagnose_deflection`: returns a dict (`{coherence_kind: "circular" |
|
||||
"token_salad" | "vacuous" | "ok", evidence: …}`); the renderer can
|
||||
surface `· incoherent: <kind>`; *optionally* the verifier could demote
|
||||
STRICT → HYBRID on a hard `coherence_kind` (the same way Rule 8
|
||||
demotes on `TITLE_MISMATCH`) — that's a policy flag, off by default,
|
||||
bench-gated, folds into `governance_policy_hash` if enabled. Likely
|
||||
the cheapest, highest-confidence win here; deletion-first — try this
|
||||
before reaching for a model.
|
||||
|
||||
### 3.2 — `diagnose_relevance(question, answer, cited)` — the model question
|
||||
|
||||
Today's relevance checks are lexical (token/stem overlap); they fail
|
||||
on collisions. A small *aboutness* model would give a semantic score:
|
||||
|
||||
- **input pairs**: `(question, answer)` for question-relevance;
|
||||
`(claim, cited_source_span_or_title)` for source-relevance — run
|
||||
**only** on the candidate clauses / cited sources, never the whole
|
||||
context (#000049 §7 #20 haystack lesson — same multiple-comparisons
|
||||
trap);
|
||||
- **model**: a sentence-similarity / cross-encoder *reranker*
|
||||
(`ms-marco`-style, or a small `sentence-transformers` bi/cross-
|
||||
encoder) — **not** an NLI model. NLI labels entailment/contradiction;
|
||||
this needs a topicality score. (If #000049's `[nli]` extra already
|
||||
pulls `transformers`+`torch`, this can share it — record which
|
||||
checkpoint, separate `relevance_model_version`.)
|
||||
- **decision**: a demotion-only veto — if the answer's *max* relevance
|
||||
to the question across its sentences is below θ_rel, or a claim's
|
||||
relevance to *every* cited source is below θ_src, flag (and,
|
||||
policy-permitting, demote STRICT → HYBRID). Mirror #000049's
|
||||
two-threshold caution and shadow-first protocol exactly.
|
||||
- **cage**: identical to #000049 §7 — demotion-only, hash-pinned +
|
||||
reproducible, `relevance_model_version` + thresholds into
|
||||
`governance_policy_hash` iff it touches `audit_mode`, shadow mode
|
||||
(`ARBORIST_RELEVANCE_SHADOW=1` → telemetry, no `audit_mode` effect)
|
||||
before any runtime use, bench-gated on a real `bench-qa` sample
|
||||
(false-positive rate on legit on-topic answers must be near zero —
|
||||
and #000049 §7 #21's lesson stands: the clean-eval number won't
|
||||
predict the real-traffic one).
|
||||
|
||||
### 3.3 — relationship to #000049
|
||||
|
||||
NLI (contradiction) and a relevance model (aboutness) are **siblings**
|
||||
— two small ML models in the proof-adjacent path, both inside the same
|
||||
discipline cage, both shadow-first. Reuse #000049 §7's manifest /
|
||||
governance-hash / shadow-sweep / `make bench-…-shadow` pattern as the
|
||||
template; do not re-derive it here. If #000049's runtime NLI never
|
||||
ships (its real-traffic FP rate is the open question), the relevance
|
||||
model is independently decidable — and `diagnose_coherence` (no model)
|
||||
is independent of both.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation
|
||||
|
||||
**Build `diagnose_coherence` (lexical, §3.1) first; treat
|
||||
`diagnose_relevance` (§3.2) as the model question to settle alongside
|
||||
#000049's, not before it.** Rationale (the five-step algorithm —
|
||||
delete/simplify before reaching for a model):
|
||||
|
||||
1. The motivating case ("Zionist entity") is *incoherence* + *token-
|
||||
collision*, and a chunk of it (the "used as the entity" salad) is
|
||||
catchable by a *lexical* coherence-shape check — no model, no
|
||||
dependency, fits the existing `inspect.py` discipline. Do that.
|
||||
2. The token-collision-in-retrieval root cause (a vague polysemous
|
||||
query → title-token-soup retrieval) is arguably better fixed
|
||||
*upstream* in the retrieval pipeline (a polysemy/disambiguation
|
||||
pass, or a query-specificity guard that refuses to answer a
|
||||
2-token query that fans out to N unrelated articles) than
|
||||
downstream by a relevance veto. Worth its own retrieval ticket if
|
||||
fox wants it tracked — note here, don't scope-creep.
|
||||
3. A relevance *model* is the right tool for "semantically off-topic
|
||||
despite token overlap", but it carries the same costs as #000049's
|
||||
NLI (dependency, non-determinism surface, governance-hash, the
|
||||
real-traffic-FP unknown) — only worth it if the lexical coherence
|
||||
check + the existing deflection/title sidecars + an upstream
|
||||
retrieval fix together still leave a measurable real-traffic
|
||||
incoherence/off-topic rate. Park it behind that evidence; when it's
|
||||
warranted, it ships under #000049's cage, shadow-first.
|
||||
|
||||
So: §3.1 is a near-term, model-free win; §3.2 is gated on evidence and
|
||||
travels with #000049's discipline; the upstream retrieval angle (#4.2)
|
||||
is a flag, not a deliverable here.
|
||||
|
||||
---
|
||||
|
||||
## 5. Out of scope
|
||||
|
||||
- **The retrieval-side polysemy / title-token-soup fix** — the root
|
||||
cause of the motivating case; a separate retrieval ticket if
|
||||
tracked. This ticket is downstream answer-shape signals only.
|
||||
- **Replacing the lexical verifier** — untouched; these are *added*
|
||||
read-only sidecars, the verifier stays binary and first-line.
|
||||
- **#000049's NLI contradiction veto** — sibling, not this; cross-ref
|
||||
only.
|
||||
- **A general "answer quality" score / RLHF-style reward model** — far
|
||||
bigger; these are two narrow, named, demote-only signals.
|
||||
- **Touching `providence_cache` / `run_dag_root`** — sidecars never
|
||||
do; if a *hard* coherence/relevance demotion is enabled, it changes
|
||||
which `audit_mode` `verify_quotes` returns (like Rule 8) and folds
|
||||
into `governance_policy_hash`, but the *sidecar dicts* stay
|
||||
render-layer + bench-tracking only.
|
||||
|
||||
---
|
||||
|
||||
## 6. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Doc-only scaffold. First decision: build
|
||||
the lexical `diagnose_coherence` (§3.1) now? — low-risk, model-free,
|
||||
fits `inspect.py`; the recommendation is yes. Second decision (gated
|
||||
on evidence, travels with #000049): is a relevance *model* (§3.2)
|
||||
warranted, and if so it ships under #000049 §7's cage, shadow-first.
|
||||
Third (a flag, not this ticket): does fox want the upstream retrieval
|
||||
polysemy fix tracked? Until then: nothing lands; the existing
|
||||
deflection / title-relevance / metaphor sidecars stand as the only
|
||||
shape signals.
|
||||
|
||||
---
|
||||
|
||||
## 7. References
|
||||
|
||||
- `arborist/qa/inspect.py` — `diagnose_deflection`,
|
||||
`diagnose_metaphor_deflection`, `diagnose_title_relevance` (the
|
||||
sidecar family these two join); `_stem_for_deflection` /
|
||||
`_content_tokens_for_deflection` / `_content_tokens_in_order` (the
|
||||
lexical machinery `diagnose_coherence` would reuse).
|
||||
- `arborist/qa/verify.py` — Rule 8 / `_claim_title_overlap` (the
|
||||
hard-check precedent: a sidecar signal promoted to a demote-only
|
||||
verifier rule, folded into `governance_policy_hash`).
|
||||
- `docs/tickets/ticket-000049-attribution-aware-grounding-check.md`
|
||||
§7 — the discipline cage (manifest, governance-hash rule,
|
||||
shadow-first, `[…]` extra, the haystack lesson) this ticket's
|
||||
§3.2 inherits verbatim; §7 #20/#21 = why "clean eval ≠ real-traffic
|
||||
precision" and "never run over the whole context".
|
||||
- `docs/tickets/ticket-000002-reference-frame-polarity-contract.md`,
|
||||
`docs/tickets/ticket-000003-anchor-class-warrant-generalization.md`
|
||||
— earlier "NLI deferred, do the cheap lexical thing first"
|
||||
decisions; same instinct applies to §3.1-before-§3.2.
|
||||
- The motivating field case: a `claim_lattice` query that returned
|
||||
*"the phrase 'Zionist entity' is sometimes used as the entity,
|
||||
referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL
|
||||
2/3` (2026-05-12, fox) — incoherent + token-collision recombination
|
||||
that every existing lexical relevance check waved through.
|
||||
- CLAUDE.md "Conventions — Verifier stays binary; falsifications carry
|
||||
soft signal" / "Soft hash vs hard hash" — the read-only-sidecar
|
||||
discipline both new signals obey.
|
||||
Loading…
Add table
Add a link
Reference in a new issue