# Ticket #000054 — Acronym-parens concept extractor (closing the abbreviation→expansion retrieval gap)

**Status:** in progress — Phase 1 + Phase 2 landed; bench in flight. Phase 1 (`58027e9`): the extractor + 481K edges across 4 shards. Phase 2 (pending commit) wraps four interlocking consumer-side fixes that the end-to-end gap-close exposed in flight, all folded into this ticket per the "don't proliferate" discipline (CLAUDE.md / `feedback_ticket_proliferation`): **(a)** `synonym_expand` rank-and-truncate over the per-token cap (source-frequency descending) instead of hard-skipping — CPU has 13 legitimate homonym expansions across the corpus, Phase 1 alone hit the `MAX_NEIGHBORS_PER_TOKEN=8` cap → zero expansion → no surfacing; **(b)** `_search_titles` orders by FTS5 `bm25` (multi-token-match-favoring) instead of `LENGTH(title) ASC` on the FTS5 path — the length-asc fix the 2026-05-02 "Back to the Future" case introduced was tokenization-needed for the LIKE path but counter-productive on FTS5; **(c)** retrieval routes (title-search, core-keyword) + the title-token additive rerank use the synonym-expanded `accept_tokens`, not qtokens-only; **(d)** the architecturally load-bearing one — `synonym_expand_strict()` (new), a high-trust evidence-kind subset (manual + acronym_parens, **excludes** `link_reciprocity`), used by `accept_tokens` and `_rerank_by_title_purity` (the multiplicative ranker). Reciprocal-wikilink edges express *topical adjacency*, not synonymy (a `Dinosaurs` page reciprocally links to a `Curious George Brigade` page → an edge that should not amplify retrieval — observed 2026-05-13 dinosaurs regression where Phase-2-broad pulled CGB into the top), and a multiplicative ranker over them blows up. The strict view preserves the acronym-parens surfacing (those edges ARE the phrase=expansion identity) while keeping link-reciprocity restricted to additive retrieval-route boosts. Also (e): tightened extractor regex `[A-Z]{2,6}` → `[A-Z]{3,6}` and purged ~21K 2-letter acronym edges (`AI`/`ML`/`OS`/`US`/`UK`/`IT`/`PC`/`TV`) that homonym-collided with 2-char query tokens like `go`/`is`/`am`. **End-to-end verified live:** `what is a CPU?` retrieval pulls *Central processing unit* at #1 (was: only CPU-* satellites); `what is a GPU?` → *Graphics processing unit* at #1 + EVIDENCE-WARRANTED 1/1; `where is mount kilimanjaro located?` → Mount Kilimanjaro at #1; `when did the soviet union dissolve?` → Soviet Union at #1 (regressions from the no-strict intermediate state went away). **Bench:** `make bench-qa BENCH_QA_N=3 BENCH_QA_LIMIT=5` (2026-05-13T14:24Z) — 30/45 STRICT (67%), zero regressions on `mona lisa` / `capital of france` / `new london bridge` (each 9/9 STRICT across all three modes); the bench-qa-smoke n=1 flickers ("dinosaurs", "soviet union" going S→H/U on a single run) were LLM stochasticity, not a retrieval bug. Phase 1 prior closure was premature — the extractor produced edges but didn't change retrieval outcomes; this is the actual end-to-end-working bar.
**Opened:** 2026-05-13
**Scope:** One new extractor in `arborist/concepts/extract.py` + registry entry + tests. No schema change, no proof-path change (synonym edges are a *retrieval-side* soft signal — they reshape which candidates `qa/query.py` considers but never enter `audit_mode` / `cache_key` / `audit_event_hash`).
**Audience:** fox + anyone maintaining the retrieval pipeline + future shifts that wonder "why didn't `CPU` find `Central processing unit`?"
**Hard constraint:** synonym edges are **retrieval-side, never proof-path** — same discipline as `link_reciprocity_synonym` (CLAUDE.md "soft hash vs hard hash"). The extractor is idempotent (`INSERT OR IGNORE` via `add_concept_relation`) and writes only `concept_relations` rows. No verifier change, no `governance_policy_hash` bump, no cache invalidation.

---

## 1. The gap, from the field

`make query Q="what is a CPU?"` retrieves "CPU design", "CPU socket",
"CPU time", "CPU cache", "CPU-Z", "CPU multiplier", "CPU
(disambiguation)" — every "CPU *" satellite — but **not** "Central
processing unit", the canonical definitional article. Same shape for
"what is a GPU?" → satellites only, never "Graphics processing
unit". The four FTS5 routes (body BM25, title-LIKE, title-token,
phrase-pattern) all key on the query token `CPU`; the canonical
article's title shares *zero tokens* with that query (its title is
"Central processing unit"), so it never ranks. Body-BM25 *could*
surface it on raw frequency, but the satellite articles where `CPU` is
in *both* title and body outrank it on title-boost. (2026-05-13 field
case, fox.)

This is exactly what `concepts/synonym_expand` exists to fix: a query
token gets expanded to a set including its synonyms, FTS5 then
retrieves on the union. But the existing extractor
(`link_reciprocity_synonym`) finds synonyms only from *reciprocal
wikilinks* — and Wikipedia represents abbreviation→expansion as a
**one-way redirect** (`CPU → Central processing unit`), which the
ingest does not record as an edge. So `concepts/` never learned the
relation.

It is, however, in *body text* of every article that uses the term, by
near-universal convention: "Central processing unit (CPU) is the most
important component of a computer." That's the signal the new
extractor reads.

## 2. The extractor

`arborist/concepts/extract.py:acronym_parens_synonym`. Scans
`chunks.content` at `idx = 0` (the lead chunk; encyclopedic and
reference prose introduces its abbreviation in the first paragraph by
convention; bounded to the first 4000 chars for cheapness). Pattern:

```
<word> (<word>){1..6}  (  <ACRO 2..6 caps>  )
```

For each candidate `(phrase, ACRO)` pair: split the phrase into atomic
words (whitespace + hyphens both split), drop function words (`of`,
`the`, `and`, `or`, `for`, `on`, `in`, `to`, `a`, `an`, `by`, `at`,
`as`, `is`, `are`, `was`, `be`), then require **strict 1:1 match** —
`len(atoms) == len(ACRO)` and each acronym letter equals the first
letter of the corresponding atom (case-insensitive). On match, emit a
bidirectional synonym edge between the lowercased acronym and each
≥3-char content token of the phrase, anchored to the doc's
`document_root` and tagged `evidence_kind = "acronym_parens"`.

Deliberately conservative:

- **Strict 1:1.** `Hypertext Transfer Protocol (HTTP)` (3 atoms, 4
  letters) is *rejected* — the catch isn't worth the false-positive
  risk that comes with allowing letters to land mid-word. Same for
  `USA`-shape acronyms where letters skip atoms.
- **Function-word filter for atoms only.** "Federal Bureau of
  Investigation (FBI)" → atoms `[Federal, Bureau, Investigation]`, FBI
  matches (length 3 to 3). Atom-skipwords don't appear as edges either
  (no `of ↔ fbi`).
- **Per-doc dedupe.** Same `(phrase, ACRO)` repeated in the same doc
  produces one edge set, not N.
- **≥3-char target floor.** "We" or 2-char particles never become a
  target.

Registry: `EXTRACTORS["acronym_parens"] = acronym_parens_synonym`.

## 3. Why this complements #000050 (and didn't get folded into it)

#000050 (vec RRF hybrid) is the right tool for *genuine conceptual*
allusion — the Orwell→Eastasia case, where the query and target share
no semantic structure beyond meaning. Abbreviation→expansion is a
*different* shape: the relation is explicit in body text under a
universal convention, lexical and deterministic. A vec embedder
*could* close it (CPU and "Central processing unit" are semantically
adjacent), but at ~10–100× the ingest cost and a transformer in the
query path. The cheap, lexical, per-shard `concept_relations` table is
the right home; vec stays the answer to the cases lexical structurally
can't reach. #000050's §2a fixture set names the CPU/GPU cases — this
ticket closes them upstream; the bench should record *which* fix
closes each row, and the Orwell case is the one that genuinely
*requires* the vec layer.

## 4. Out of scope

- Backfilling the existing shards (`arborist concepts derive
  --extractor acronym_parens` on each `~/.arborist/shards/*.db` —
  operational, not code; one-shot, idempotent, cheap).
- Permissive acronym matching (HTTP-shape, letters mid-word) — would
  fire #000054.1 if the false-positive rate of the strict matcher
  proves limiting in practice.
- Cross-shard synonym indices (per-shard is the existing discipline;
  cross-shard requires schema work that isn't justified yet — the
  query-time expansion already merges across shards by lookup).
- Wiring an actual `arborist concepts derive` CLI subcommand (the
  docstrings reference one but the surface isn't wired in `cli.py`
  today; extractors are called programmatically. Worth its own ticket
  if operations grows past one-off scripting.)

## 5. Acceptance criteria

1. `acronym_parens_synonym` lands in `arborist/concepts/extract.py`
   with a stable `evidence_kind = "acronym_parens"` and is registered
   in `EXTRACTORS`.
2. The CPU case (`"central processing unit (CPU)"` in body) emits
   bidirectional edges `cpu ↔ central`, `cpu ↔ processing`,
   `cpu ↔ unit` and is idempotent on re-run.
3. False-positive rejection: `Hypertext Transfer Protocol (HTTP)`
   (mismatched length) and `Apple Banana Carrot (XYZ)` (mismatched
   initials) emit zero edges.
4. Function-word filter: `Federal Bureau of Investigation (FBI)` emits
   `fbi ↔ federal/bureau/investigation`; `of` never appears as a
   target.
5. Per-doc dedupe: the same `(phrase, ACRO)` repeated in one doc emits
   one edge set.
6. 8 new tests in `tests/test_concepts_extract.py` (covering CPU /
   idempotency / function-word filter / length mismatch / initial
   mismatch / hyphenated words / repeat dedupe / registry presence);
   full suite green.

## 6. References

- `arborist/concepts/extract.py` — sibling `link_reciprocity_synonym`
  is the design precedent (per-shard, evidence-kind-tagged, idempotent,
  same return-dict shape).
- `arborist/concepts/store.py:add_concept_relation` — the write API.
  `RELATION_KINDS = ("synonym", "antonym", "rivalry", "category")`.
- `arborist/qa/concepts.py` + `arborist/concepts/query.py` — the
  read-side: `synonym_expand(qtokens, shards_dir=…)` walks every
  shard's `concept_relations` and unions the matches into the query
  token set before FTS5 retrieval runs.
- #000050 §2a (semantic-allusion fixture set) — CPU/GPU named there
  as the two abbreviation rows; this extractor closes them *upstream*
  of vec, leaving the Orwell row as the genuine vec justification.
- #000053 — verifier acronym-aware `_content_tokens`; orthogonal
  (verifier-side, not retrieval-side) but in the same field-case
  thread (a CPU/GPU query exposed *both* gaps; closing one didn't
  close the other).
- CLAUDE.md "Retrieval pipeline" §5 (`concepts/` rivalry +
  synonym layer) / "soft hash vs hard hash" — the discipline this
  extractor obeys.
