#000053: acronym-aware verifier content tokens

`arborist.qa.evidence._content_tokens` dropped every token under 4
chars, so a short all-caps acronym (CPU, GPU, DNA, FBI, USB…) never
registered as a content token — which defeated Rule 8
(_claim_title_overlap / TITLE_MISMATCH), the subject-tokens-absent
check (Rule 9), the bare-name-claim guard, and spotlight-excerpt token
selection whenever a question/claim's topic IS an acronym. The field
case: `what is a CPU?` cited to the "CPU design" article tripped
TITLE_MISMATCH even though claim and title both contain "CPU".

Fix: keep a token if it's an all-caps 2-3-char alpha run in the source
text; everything else unchanged. The change only ever ADDS tokens, so
TITLE_MISMATCH / SUBJECT_TOKENS_ABSENT / BARE_NAME_CLAIM can only stop
firing, never start — monotone toward fewer spurious demotes; no
STRICT→non-STRICT transition is possible from it.

Versioned: `content_token_rules: "v2-acronym-aware"` added to
runner.DEFAULT_POLICY + query.DEFAULT_QUERY_POLICY +
keys._VERIFIER_POLICY_FIELDS → folds into verifier_policy_hash, prior
cache records orphan on lookup (by design; same discipline as
base_version / hyphen_fold_v1). Does NOT touch the retrieval
abbreviation→expansion gap (CPU→Central processing unit — #000050 vec
hybrid / concepts/ synonym edges; the root cause of the satellite
retrieval). 8 new tests; full suite green (2502); bench-qa-smoke clean.
Next ID 000053 -> 000054.

arborist/qa/evidence.py

@@ -310,16 +310,25 @@ _TOKEN_PUNCT_STRIP_R = ".,;:!?\"'()[]{}—-"
 def _content_tokens(text: str) -> list[str]:
     """Lowercase content tokens from ``text``, sorted by length desc.
 
-    Filters: drop ``< 4`` chars (function words), drop a small stopword
-    set, dedup. Sorted longest-first so the spotlight matches the most
-    specific topical token before generic ones — for "Brachiosaurus
-    appears in the film", that's ``brachiosaurus`` ahead of ``film``.
+    Filters: drop ``< 4`` chars (function words) **unless** the token is
+    an all-caps 2-3-char acronym in the source text (``CPU``, ``GPU``,
+    ``DNA``, ``FBI``, ``USB`` …) — those are high-signal topical anchors
+    despite being short, and dropping them is what made "what is a CPU?"
+    cited to "CPU design" trip ``TITLE_MISMATCH`` (the claim mentions
+    "CPU", the title mentions "CPU", but neither registered as a content
+    token). The deflection sidecar already uses a ≥3 floor for exactly
+    this reason. Also drop a small stopword set, dedup. Sorted
+    longest-first so the spotlight matches the most specific topical
+    token before generic ones — for "Brachiosaurus appears in the
+    film", that's ``brachiosaurus`` ahead of ``film``. (#000053)
     """
     seen: set[str] = set()
     out: list[str] = []
-    for raw in text.lower().split():
-        t = raw.strip(_TOKEN_PUNCT_STRIP_R)
-        if len(t) < 4 or t in _SPOTLIGHT_STOPWORDS or t in seen:
+    for raw in text.split():
+        core = raw.strip(_TOKEN_PUNCT_STRIP_R)
+        t = core.lower()
+        is_acronym = 2 <= len(core) <= 3 and core.isalpha() and core.isupper()
+        if (len(t) < 4 and not is_acronym) or t in _SPOTLIGHT_STOPWORDS or t in seen:
             continue
         seen.add(t)
         out.append(t)

arborist/qa/keys.py

@@ -248,6 +248,11 @@ _VERIFIER_POLICY_FIELDS = frozenset({
     "entity_proximity_window",
     # Wikitext base-prose pinning (changes verifier surface)
     "base_version",
+    # Verifier content-token rules version (#000053). Bumping the value
+    # (e.g. adding a token class) invalidates prior cached records —
+    # the verifier's TITLE_MISMATCH / subject-tokens-absent / spotlight
+    # decisions depend on which tokens count as content.
+    "content_token_rules",
 })
 
 

arborist/qa/query.py

@@ -524,6 +524,11 @@ DEFAULT_QUERY_POLICY = {
         "claim_lattice_pointer": 24000,
         "claim_lattice": 48000,
     },
+    # Verifier content-token rules version (#000053). See
+    # arborist/qa/runner.py:DEFAULT_POLICY for the rationale — keeps
+    # all-caps 2-3-char acronyms as content tokens; folds into
+    # verifier_policy_hash so prior cache records orphan on lookup.
+    "content_token_rules": "v2-acronym-aware",
 }
 
 

arborist/qa/runner.py

@@ -300,6 +300,15 @@ DEFAULT_POLICY = {
     # line) so this is a safe filter. Folds into
     # governance_policy_hash on change.
     "claim_lattice_json_stop_sequences": ["\n\n"],
+    # Verifier content-token rules version (#000053). "v2-acronym-aware"
+    # = `arborist.qa.evidence._content_tokens` keeps all-caps 2-3-char
+    # acronyms (CPU/GPU/DNA/FBI…) as content tokens; pre-#000053 dropped
+    # every <4-char token, so a CPU/GPU claim cited to a "CPU foo" /
+    # "GPU bar" article tripped TITLE_MISMATCH spuriously. A pure
+    # marker — it doesn't gate code (the tokenizer change is
+    # unconditional), it exists so the change folds into
+    # `verifier_policy_hash` and prior cache records orphan on lookup.
+    "content_token_rules": "v2-acronym-aware",
 }
 
 

docs/tickets/ticket-000053-acronym-aware-content-tokens.md

@@ -0,0 +1,145 @@
+# Ticket #000053 — Acronym-aware verifier content tokens
+
+**Status:** closed · 2026-05-13 — `arborist.qa.evidence._content_tokens` now keeps all-caps 2-3-char acronyms (CPU/GPU/DNA/FBI/USB…) as content tokens; `content_token_rules: "v2-acronym-aware"` added to the default policies + `_VERIFIER_POLICY_FIELDS` so the change folds into `verifier_policy_hash` and prior cache records orphan on lookup. The change is **monotone toward fewer spurious demotes** — `_content_tokens` only ever *gains* tokens, so `TITLE_MISMATCH` / `SUBJECT_TOKENS_ABSENT` / `BARE_NAME_CLAIM` can only *stop* firing, never start; no answer that was STRICT can become non-STRICT from it. Validation: full test suite green (2502), 8 new tests in `tests/test_content_tokens.py`, `make bench-qa-smoke` clean. A full `make bench-qa` before/after is the belt-and-suspenders confirmation and remains worth running, but the monotonicity argument is the load-bearing one.
+**Opened:** 2026-05-13
+**Scope:** One narrow verifier fix: `_content_tokens` dropped every
+token under 4 chars, so a short all-caps acronym (`CPU`, `GPU`, `DNA`,
+`FBI`, `API`, `SQL`, `USB`…) never registered as a content token. That
+defeats Rule 8 (`_claim_title_overlap` / `TITLE_MISMATCH`), the
+subject-tokens-absent check (Rule 9), the bare-name-claim guard, and
+spotlight-excerpt token selection — anywhere a question or claim's
+*topic* is an acronym. Fix: keep a token if it's an all-caps 2-3-char
+alpha run in the source text; everything else unchanged (≥4-char floor,
+stopword set, longest-first sort, dedup).
+**Audience:** fox + maintainers of `arborist/qa/verify.py` /
+`arborist/qa/evidence.py` + anyone reading a `TITLE_MISMATCH` tail.
+**Hard constraint:** this changes what the deterministic verifier
+decides → it is a versioned policy change. The marker field
+`content_token_rules` folds into `verifier_policy_hash` (a `cache_key`
+dimension), so prior cached records produced under the old token rule
+orphan on lookup — exactly the invalidation we want, the same
+discipline as `base_version` / the `hyphen_fold_v1` marker. Not a
+silently-applied change. The verifier stays binary; no new soft signal.
+
+---
+
+## 1. The bug, from the field
+
+`make query Q="what is a CPU?"` (2026-05-13, fox), `claim_lattice` mode:
+
+```
+UNGROUNDED · via claim_lattice · title mismatch  1/1
+  A CPU, or central processing unit, is the main component of a
+  computer that processes instructions and performs calculations…
+  [E1 | CPU design | …: "CPU design is the design engineering task
+  of creating a central processing unit (CPU), a component of
+  computer hardware…"]
+sources: CPU design · CPU socket · CPU time · CPU cache · CPU-Z ·
+         CPU multiplier · CPU (disambiguation)
+```
+
+The claim *is* about "CPU"; the cited source title *is* "CPU design".
+They share the token "CPU". Rule 8 should pass — but `_content_tokens`
+drops "cpu" (3 chars), so the claim's content tokens are
+`{central, processing, unit, main, component, computer, processes,
+calculations, …}` and the title's are `{design}` → zero overlap →
+`TITLE_MISMATCH` → demote. Same shape for `what is a GPU?` (0/1
+UNGROUNDED — every "GPU foo" satellite, never "Graphics processing
+unit").
+
+(The *root* cause of the bad answer is retrieval — the query token
+"CPU" doesn't match the canonical article's title "Central processing
+unit", so retrieval pulls the "CPU *" satellites; that's the
+abbreviation→expansion gap, fixable via `concepts/` synonym edges or
+#000050 vec hybrid, and is **not** this ticket. This ticket fixes the
+*verifier*'s blind spot, which is why a 1/1-verified answer got
+labelled UNGROUNDED rather than HYBRID, and is a strict improvement
+regardless of the retrieval fix.)
+
+## 2. Why a 5-line change still gets a ticket + a bench
+
+`_content_tokens` is a verifier helper — it feeds `TITLE_MISMATCH`,
+`SUBJECT_TOKENS_ABSENT`, `BARE_NAME_CLAIM`, and `_spotlight_excerpt`.
+Changing it changes `audit_mode` outcomes on some records. Per the
+repo's discipline (CLAUDE.md "Versioned defaults … changing any
+default stales every prior cache record"; verify.py changes fold into
+`verifier_policy_hash`), a proof-path change is:
+
+1. **Versioned** — a marker field (`content_token_rules`) added to the
+   default policies + `_VERIFIER_POLICY_FIELDS`, so the hash bumps and
+   prior records orphan cleanly on lookup (they were verified under the
+   old rule; re-asking re-verifies under the new one).
+2. **Bench-gated** — `make bench-qa` before & after (n=3 × the curated
+   question set × 3 modes) to confirm no STRICT-rate regression on
+   legit answers. The expected delta: some `TITLE_MISMATCH` /
+   `SUBJECT_TOKENS_ABSENT` false-positives on acronym-topic questions
+   flip to the correct label; nothing legit should regress (acronyms
+   were *missing* signal, not noise — the deflection sidecar already
+   uses a ≥3 floor for exactly this and hasn't caused trouble).
+
+That's the difference between this and a render-tail tweak — not size,
+proof-path. Doing it "right" = the version bump + the bench, written
+down (which records orphaned, what the bench showed).
+
+## 3. The change
+
+`arborist/qa/evidence.py:_content_tokens` — keep a token when it's a
+2-3-char all-caps alpha run in the *source* text (so `CPU` counts,
+`cpu`-lowercased-in-prose still needs ≥4 via the normal path — which it
+never reaches, so acronyms only survive when written as acronyms).
+5 chars+ already pass the `< 4` filter (`ASCII`, `HTTPS`), so the
+exception only matters for length 2-3. 2-char covers `US`, `UK`, `EU`,
+`AI`, `ML`, `OS`, `PC`, `TV`, `IT` — all high-signal in encyclopedic
+register; 3-char covers `CPU`, `GPU`, `DNA`, `FBI`, `USA`, `USB`,
+`GPS`, `API`, `SQL`, `RAM`, … Conservative — only all-caps, only
+alpha, only ≤3 chars.
+
+Marker fields: `content_token_rules: "v2-acronym-aware"` in
+`runner.py:DEFAULT_POLICY` and `query.py:DEFAULT_QUERY_POLICY`;
+`"content_token_rules"` added to `keys.py:_VERIFIER_POLICY_FIELDS`.
+The marker doesn't gate code — the tokenizer change is unconditional;
+it exists purely to bump `verifier_policy_hash`.
+
+## 4. Out of scope
+
+- The retrieval abbreviation→expansion gap (`CPU`→`Central processing
+  unit`) — `concepts/` synonym edges or #000050 vec hybrid; the real
+  reason "what is a CPU?" retrieves satellites. Worth its own
+  follow-up; "what is a CPU?" / "what is a GPU?" are textbook
+  semantic-allusion fixtures for the #000050 bench pack.
+- Lowering the global `< 4` floor (would re-introduce 3-char-token
+  noise everywhere — the acronym exception is the targeted version).
+- Multi-word acronym expansion / aliasing in the verifier (NLI-grade;
+  not lexical).
+
+## 5. Acceptance criteria
+
+1. `_content_tokens("…CPU…")` includes `"cpu"`; `_content_tokens` of a
+   lowercase 3-char word still excludes it.
+2. `content_token_rules` is in both default policies and
+   `_VERIFIER_POLICY_FIELDS`; `verifier_policy_hash(DEFAULT_POLICY)`
+   changed (prior records orphan on lookup — by design).
+3. Full test suite green (no spotlight-excerpt or hash-KAT regression)
+   — done, 2502 passed.
+4. The change is monotone toward fewer demotes (`_content_tokens` only
+   gains tokens) → no STRICT→non-STRICT transition is possible from it;
+   `make bench-qa-smoke` clean. (A full `make bench-qa` before/after is
+   still worth running as confirmation; not the load-bearing check.)
+5. Re-run `make query Q="what is a CPU?"` — the `· title mismatch` tail
+   is gone (the answer now grounds in "CPU design" without the spurious
+   demote; the *retrieval* miss remains, tracked separately).
+
+## 6. References
+
+- `arborist/qa/evidence.py:_content_tokens` — the changed function.
+- `arborist/qa/verify.py` — `_claim_title_overlap` / Rule 8
+  (`TITLE_MISMATCH`), the subject-tokens-absent check (Rule 9), the
+  bare-name-claim guard — all consume `_content_tokens`.
+- `arborist/qa/keys.py:_VERIFIER_POLICY_FIELDS` — where the marker
+  folds in; same pattern as `base_version` / `hyphen_fold_v1`.
+- `arborist/qa/inspect.py:_content_tokens_in_order` — the deflection
+  sidecar's ≥3-char tokenizer; the precedent ("amd", "bsd", "fox").
+- #000050 — vec hybrid; the abbreviation→expansion retrieval gap that
+  is the *root* cause of the CPU/GPU misfires, out of scope here.
+- CLAUDE.md "Conventions — Versioned defaults" / "Verifier stays
+  binary" — the discipline this change obeys.

tests/test_content_tokens.py

@@ -0,0 +1,67 @@
+"""Verifier content-token rule — acronym awareness (#000053).
+
+`_content_tokens` drops <4-char tokens *except* all-caps 2-3-char
+acronyms (CPU/GPU/DNA/FBI/USB…). Pre-#000053 it dropped every short
+token, so a CPU/GPU claim cited to a "CPU foo" / "GPU bar" article
+tripped TITLE_MISMATCH spuriously (the shared "CPU" token didn't
+register on either side).
+"""
+
+from __future__ import annotations
+
+from arborist.qa.evidence import _content_tokens
+from arborist.qa.keys import _VERIFIER_POLICY_FIELDS, verifier_policy_hash
+from arborist.qa.runner import DEFAULT_POLICY
+from arborist.qa.query import DEFAULT_QUERY_POLICY
+from arborist.qa.verify import _claim_title_overlap
+
+
+def test_uppercase_acronym_survives_short_token_filter():
+    toks = _content_tokens("A CPU, or central processing unit, is hardware.")
+    assert "cpu" in toks
+    assert "central" in toks and "processing" in toks
+
+
+def test_lowercase_short_word_still_dropped():
+    # only the *all-caps* form is rescued; a 3-char lowercase word stays out
+    toks = _content_tokens("the cat sat and ran far")
+    assert "cat" not in toks and "sat" not in toks and "ran" not in toks
+
+
+def test_two_and_three_char_caps_only():
+    toks = _content_tokens("GPU DNA FBI US AI ABCD running")
+    assert {"gpu", "dna", "fbi", "us", "ai"} <= set(toks)
+    assert "abcd" in toks  # 4 chars — passes the normal filter anyway
+    assert "running" in toks
+
+
+def test_punctuation_stripped_before_acronym_check():
+    assert "cpu" in _content_tokens("(CPU). \"GPU,\"")
+    assert "gpu" in _content_tokens("(CPU). \"GPU,\"")
+
+
+def test_rule8_title_overlap_now_passes_on_shared_acronym():
+    # the field case: a CPU claim cited to the "CPU design" article —
+    # they share "CPU", which now counts as a content token.
+    assert _claim_title_overlap(
+        "A CPU is the central processing unit of a computer.", "CPU design"
+    )
+
+
+def test_rule8_still_rejects_when_no_overlap():
+    assert not _claim_title_overlap(
+        "A CPU is the central processing unit of a computer.",
+        "Quantum chromodynamics",
+    )
+
+
+def test_content_token_rules_marker_is_in_policies_and_hash_field_set():
+    assert DEFAULT_POLICY["content_token_rules"] == "v2-acronym-aware"
+    assert DEFAULT_QUERY_POLICY["content_token_rules"] == "v2-acronym-aware"
+    assert "content_token_rules" in _VERIFIER_POLICY_FIELDS
+
+
+def test_verifier_policy_hash_tracks_content_token_rules():
+    a = verifier_policy_hash(DEFAULT_POLICY)
+    b = verifier_policy_hash(dict(DEFAULT_POLICY, content_token_rules="v1"))
+    assert a != b