#000054: acronym-parens concept extractor (closes abbreviation→expansion retrieval gap)

`arborist/concepts/extract.py:acronym_parens_synonym` — new
corpus-agnostic extractor. Scans each doc's lead chunk (first 4000
chars) for `<Multi-Word Phrase> (ACRO)` where the all-caps acronym's
letters strictly match the content-word initials of the phrase, in
order, after function-word filtering. Emits bidirectional synonym
edges between the lowercased acronym and each ≥3-char content token
of the phrase, evidence_kind="acronym_parens", anchored to that doc's
document_root. Idempotent like link_reciprocity_synonym.

Why this complements link_reciprocity: Wikipedia represents
abbreviation→expansion as a one-way *redirect* (CPU →
Central processing unit), which the ingest does not record as an
edge — so the existing reciprocal-link extractor never learned the
relation. The relation IS in body text by near-universal convention
("Central processing unit (CPU) is..."), which this extractor reads.
Corpus-agnostic: HTML, blogs, textbooks benefit equally.

Conservative: strict 1:1 acronym-to-atom match (rejects HTTP-shape,
where letters land mid-word), function words filtered, repeated
definitions deduped per doc, ≥3-char target floor. 8 new tests
covering CPU bidirectional emit, RAM idempotency, FBI function-word
filter, HTTP length-mismatch reject, XYZ initial-mismatch reject,
ROM hyphenated-word handling, per-doc dedupe, registry presence.

Retrieval-side only — synonym edges reshape FTS5 candidate selection
via synonym_expand at query time, never enter audit_mode / cache_key
/ audit_event_hash. No governance hash bump, no cache invalidation.

Closes #000050 §2a's CPU/GPU abbreviation rows *upstream* of vec;
the Orwell-shape conceptual-allusion row remains the genuine #000050
justification. Operational follow-up (not code): run on each shard
via `arborist concepts derive --extractor acronym_parens` (CLI
surface itself is aspirational in docstrings; extractors are called
programmatically today). Next ID 000054 -> 000055.

docs/TICKETS.md

@@ -101,6 +101,7 @@ Newest first. Update on every open/close.
 
 | ID       | Title                                          | Status                | Opened     | Directive |
 |----------|------------------------------------------------|-----------------------|------------|-----------|
+| #000054  | Acronym-parens concept extractor (closes the abbreviation→expansion retrieval gap) | **closed · 2026-05-13** — `arborist/concepts/extract.py:acronym_parens_synonym` lands as a new corpus-agnostic extractor in `EXTRACTORS` (`evidence_kind="acronym_parens"`). Scans each doc's lead chunk for `<Multi-Word Phrase> (ACRO)` where the all-caps acronym's letters match the content-word initials of the phrase in order; emits bidirectional synonym edges between the lowercased acronym and each ≥3-char content token of the phrase. Conservative (strict 1:1 initials, function words filtered, repeated definitions deduped per doc). Closes the *retrieval-side* abbreviation gap (`CPU↔central processing unit`, `GPU↔graphics processing unit`, `RAM↔random access memory`, `FBI↔federal bureau of investigation`, `WHO↔world health organization`, …) that `link_reciprocity_synonym` can't reach because the relation lives in body text, not the wiki link graph (Wikipedia represents abbreviation→expansion as a *redirect* — not an edge). Per-shard like all `concept_relations` data; corpus-agnostic so HTML/blogs/textbooks benefit equally. Retrieval-side only — never proof-path. 8 new tests; full suite green. Closes #000050 §2a's CPU/GPU fixture rows *upstream* of vec; the Orwell-shape conceptual-allusion row remains the genuine #000050 justification. Operational follow-up (not code): `arborist concepts derive --extractor acronym_parens` on each shard. | 2026-05-13 | —         |
 | #000053  | Acronym-aware verifier content tokens | **closed · 2026-05-13** — `arborist.qa.evidence._content_tokens` now keeps all-caps 2-3-char acronyms (CPU/GPU/DNA/FBI/USB…) as content tokens instead of dropping every <4-char token; fixes the field case where "what is a CPU?" cited to "CPU design" tripped `TITLE_MISMATCH` spuriously (claim & title share "CPU" but neither registered) — also affects `SUBJECT_TOKENS_ABSENT` (Rule 9), `BARE_NAME_CLAIM`, spotlight-excerpt token pick. Versioned: `content_token_rules: "v2-acronym-aware"` in both default policies + `_VERIFIER_POLICY_FIELDS` → folds into `verifier_policy_hash`, prior cache records orphan on lookup (by design, same discipline as `base_version` / `hyphen_fold_v1`). Monotone toward *fewer* spurious demotes (only relaxes overlap checks, never tightens). 8 new tests; full suite green; `bench-qa-smoke` clean. Does NOT fix the *retrieval* abbreviation→expansion gap (`CPU`→`Central processing unit` = #000050 vec hybrid / `concepts/` synonym edges — the root cause of the satellite-article retrieval). | 2026-05-13 | —         |
 | #000052  | Relevance + coherence meta-cognition (answer-*shape* signals) | in progress — **§3.1 `diagnose_coherence` landed** (lexical, no model: `circular` / `phrase_component_reuse` / `vacuous`; in `arborist/qa/inspect.py`, surfaced via `inspect_cache_key` + `arborist inspect` `· incoherent: <kind>`; 9 tests; demote-policy hook deliberately not wired — advisory only). Joins the `diagnose_deflection` / `diagnose_metaphor_deflection` / `diagnose_title_relevance` / soft-preflight family of read-only, demote-only, never-in-proof-path sidecars; `phrase_component_reuse` catches the motivating field case (a subject quoting a phrase, a predicate reusing one of that phrase's own tokens as a bare `the <token>` referent). **Still open: (2) `diagnose_relevance`** — semantic (not just lexical) "aboutness": does the answer address the question; is each claim about its cited source? Today's checks (subject-anchor token overlap, stemmed title-stem overlap) are *lexical* and a token collision defeats them — a small *aboutness/reranker* model (NOT NLI — entailment ≠ topicality) under #000049 §7's discipline cage verbatim (demotion-only, hash-pinned, `relevance_model_version`→`governance_policy_hash` iff it touches `audit_mode`, shadow-first, `[…]` extra, the §7 #20 haystack lesson — never over the whole context); gated on evidence, travels with #000049's model question. Motivating field case (2026-05-12, fox): the `claim_lattice` query that returned *"the phrase 'Zionist entity' is sometimes used as the entity, referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL 2/3` — incoherent + token-collision recombination that NLI can't catch (returns *neutral*, not *contradiction*) and both lexical relevance checks waved through. Flags an upstream retrieval ticket (polysemy / title-token-soup) as the root-cause fix, not scoped here. #000049 sibling | 2026-05-12 | —         |
 | #000051  | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | —         |
@@ -157,4 +158,4 @@ Newest first. Update on every open/close.
 
 ## Next ID
 
-`000054`
+`000055`

docs/tickets/ticket-000054-acronym-parens-concept-extractor.md

@@ -0,0 +1,151 @@
+# Ticket #000054 — Acronym-parens concept extractor (closing the abbreviation→expansion retrieval gap)
+
+**Status:** closed · 2026-05-13 — `arborist/concepts/extract.py:acronym_parens_synonym` lands as a new corpus-agnostic extractor in the existing `EXTRACTORS` registry (`evidence_kind = "acronym_parens"`). Scans the lead chunk of every document for the pattern `<Multi-Word Phrase> (ACRO)` where the all-caps parenthesized acronym's letters match the content-word initials of the phrase in order; emits bidirectional synonym edges between the (lowercased) acronym and each ≥3-char content token of the phrase. Conservative: strict 1:1 initial match, function words filtered, repeated definitions deduped per doc. 8 new tests in `tests/test_concepts_extract.py` (28 → 36); full suite green. Closes the *retrieval-side* abbreviation gap (`CPU↔central processing unit`, `GPU↔graphics processing unit`, `RAM↔random access memory`, `FBI↔federal bureau of investigation`, `WHO↔world health organization`, …) that `link_reciprocity_synonym` structurally can't reach because the relation lives in body text, not the wiki link graph (Wikipedia represents abbreviation→expansion as a *redirect*, which the ingest does not record as an edge). Per-shard like all `concept_relations` data; corpus-agnostic so HTML / blogs / textbooks benefit the same way as Wikipedia. Complements #000050 (vec hybrid) without overlap — the abbreviation cases the vec layer would otherwise have to carry are now closable cheaply; the Orwell-shape *conceptual* allusion remains a vec-only case.
+**Opened:** 2026-05-13
+**Scope:** One new extractor in `arborist/concepts/extract.py` + registry entry + tests. No schema change, no proof-path change (synonym edges are a *retrieval-side* soft signal — they reshape which candidates `qa/query.py` considers but never enter `audit_mode` / `cache_key` / `audit_event_hash`).
+**Audience:** fox + anyone maintaining the retrieval pipeline + future shifts that wonder "why didn't `CPU` find `Central processing unit`?"
+**Hard constraint:** synonym edges are **retrieval-side, never proof-path** — same discipline as `link_reciprocity_synonym` (CLAUDE.md "soft hash vs hard hash"). The extractor is idempotent (`INSERT OR IGNORE` via `add_concept_relation`) and writes only `concept_relations` rows. No verifier change, no `governance_policy_hash` bump, no cache invalidation.
+
+---
+
+## 1. The gap, from the field
+
+`make query Q="what is a CPU?"` retrieves "CPU design", "CPU socket",
+"CPU time", "CPU cache", "CPU-Z", "CPU multiplier", "CPU
+(disambiguation)" — every "CPU *" satellite — but **not** "Central
+processing unit", the canonical definitional article. Same shape for
+"what is a GPU?" → satellites only, never "Graphics processing
+unit". The four FTS5 routes (body BM25, title-LIKE, title-token,
+phrase-pattern) all key on the query token `CPU`; the canonical
+article's title shares *zero tokens* with that query (its title is
+"Central processing unit"), so it never ranks. Body-BM25 *could*
+surface it on raw frequency, but the satellite articles where `CPU` is
+in *both* title and body outrank it on title-boost. (2026-05-13 field
+case, fox.)
+
+This is exactly what `concepts/synonym_expand` exists to fix: a query
+token gets expanded to a set including its synonyms, FTS5 then
+retrieves on the union. But the existing extractor
+(`link_reciprocity_synonym`) finds synonyms only from *reciprocal
+wikilinks* — and Wikipedia represents abbreviation→expansion as a
+**one-way redirect** (`CPU → Central processing unit`), which the
+ingest does not record as an edge. So `concepts/` never learned the
+relation.
+
+It is, however, in *body text* of every article that uses the term, by
+near-universal convention: "Central processing unit (CPU) is the most
+important component of a computer." That's the signal the new
+extractor reads.
+
+## 2. The extractor
+
+`arborist/concepts/extract.py:acronym_parens_synonym`. Scans
+`chunks.content` at `idx = 0` (the lead chunk; encyclopedic and
+reference prose introduces its abbreviation in the first paragraph by
+convention; bounded to the first 4000 chars for cheapness). Pattern:
+
+```
+<word> (<word>){1..6}  (  <ACRO 2..6 caps>  )
+```
+
+For each candidate `(phrase, ACRO)` pair: split the phrase into atomic
+words (whitespace + hyphens both split), drop function words (`of`,
+`the`, `and`, `or`, `for`, `on`, `in`, `to`, `a`, `an`, `by`, `at`,
+`as`, `is`, `are`, `was`, `be`), then require **strict 1:1 match** —
+`len(atoms) == len(ACRO)` and each acronym letter equals the first
+letter of the corresponding atom (case-insensitive). On match, emit a
+bidirectional synonym edge between the lowercased acronym and each
+≥3-char content token of the phrase, anchored to the doc's
+`document_root` and tagged `evidence_kind = "acronym_parens"`.
+
+Deliberately conservative:
+
+- **Strict 1:1.** `Hypertext Transfer Protocol (HTTP)` (3 atoms, 4
+  letters) is *rejected* — the catch isn't worth the false-positive
+  risk that comes with allowing letters to land mid-word. Same for
+  `USA`-shape acronyms where letters skip atoms.
+- **Function-word filter for atoms only.** "Federal Bureau of
+  Investigation (FBI)" → atoms `[Federal, Bureau, Investigation]`, FBI
+  matches (length 3 to 3). Atom-skipwords don't appear as edges either
+  (no `of ↔ fbi`).
+- **Per-doc dedupe.** Same `(phrase, ACRO)` repeated in the same doc
+  produces one edge set, not N.
+- **≥3-char target floor.** "We" or 2-char particles never become a
+  target.
+
+Registry: `EXTRACTORS["acronym_parens"] = acronym_parens_synonym`.
+
+## 3. Why this complements #000050 (and didn't get folded into it)
+
+#000050 (vec RRF hybrid) is the right tool for *genuine conceptual*
+allusion — the Orwell→Eastasia case, where the query and target share
+no semantic structure beyond meaning. Abbreviation→expansion is a
+*different* shape: the relation is explicit in body text under a
+universal convention, lexical and deterministic. A vec embedder
+*could* close it (CPU and "Central processing unit" are semantically
+adjacent), but at ~10–100× the ingest cost and a transformer in the
+query path. The cheap, lexical, per-shard `concept_relations` table is
+the right home; vec stays the answer to the cases lexical structurally
+can't reach. #000050's §2a fixture set names the CPU/GPU cases — this
+ticket closes them upstream; the bench should record *which* fix
+closes each row, and the Orwell case is the one that genuinely
+*requires* the vec layer.
+
+## 4. Out of scope
+
+- Backfilling the existing shards (`arborist concepts derive
+  --extractor acronym_parens` on each `~/.arborist/shards/*.db` —
+  operational, not code; one-shot, idempotent, cheap).
+- Permissive acronym matching (HTTP-shape, letters mid-word) — would
+  fire #000054.1 if the false-positive rate of the strict matcher
+  proves limiting in practice.
+- Cross-shard synonym indices (per-shard is the existing discipline;
+  cross-shard requires schema work that isn't justified yet — the
+  query-time expansion already merges across shards by lookup).
+- Wiring an actual `arborist concepts derive` CLI subcommand (the
+  docstrings reference one but the surface isn't wired in `cli.py`
+  today; extractors are called programmatically. Worth its own ticket
+  if operations grows past one-off scripting.)
+
+## 5. Acceptance criteria
+
+1. `acronym_parens_synonym` lands in `arborist/concepts/extract.py`
+   with a stable `evidence_kind = "acronym_parens"` and is registered
+   in `EXTRACTORS`.
+2. The CPU case (`"central processing unit (CPU)"` in body) emits
+   bidirectional edges `cpu ↔ central`, `cpu ↔ processing`,
+   `cpu ↔ unit` and is idempotent on re-run.
+3. False-positive rejection: `Hypertext Transfer Protocol (HTTP)`
+   (mismatched length) and `Apple Banana Carrot (XYZ)` (mismatched
+   initials) emit zero edges.
+4. Function-word filter: `Federal Bureau of Investigation (FBI)` emits
+   `fbi ↔ federal/bureau/investigation`; `of` never appears as a
+   target.
+5. Per-doc dedupe: the same `(phrase, ACRO)` repeated in one doc emits
+   one edge set.
+6. 8 new tests in `tests/test_concepts_extract.py` (covering CPU /
+   idempotency / function-word filter / length mismatch / initial
+   mismatch / hyphenated words / repeat dedupe / registry presence);
+   full suite green.
+
+## 6. References
+
+- `arborist/concepts/extract.py` — sibling `link_reciprocity_synonym`
+  is the design precedent (per-shard, evidence-kind-tagged, idempotent,
+  same return-dict shape).
+- `arborist/concepts/store.py:add_concept_relation` — the write API.
+  `RELATION_KINDS = ("synonym", "antonym", "rivalry", "category")`.
+- `arborist/qa/concepts.py` + `arborist/concepts/query.py` — the
+  read-side: `synonym_expand(qtokens, shards_dir=…)` walks every
+  shard's `concept_relations` and unions the matches into the query
+  token set before FTS5 retrieval runs.
+- #000050 §2a (semantic-allusion fixture set) — CPU/GPU named there
+  as the two abbreviation rows; this extractor closes them *upstream*
+  of vec, leaving the Orwell row as the genuine vec justification.
+- #000053 — verifier acronym-aware `_content_tokens`; orthogonal
+  (verifier-side, not retrieval-side) but in the same field-case
+  thread (a CPU/GPU query exposed *both* gaps; closing one didn't
+  close the other).
+- CLAUDE.md "Retrieval pipeline" §5 (`concepts/` rivalry +
+  synonym layer) / "soft hash vs hard hash" — the discipline this
+  extractor obeys.