diff --git a/CLAUDE.md b/CLAUDE.md index 1ab9144..89b83ce 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -521,6 +521,29 @@ flip Status to `closed · landed in commit ` (or the same commit as the implementation.** An open ticket whose code already shipped is a stale map. +**Default: extend an existing ticket. Don't proliferate.** When +follow-up work surfaces during implementation (a consumer-side fix +the new code needs to actually take effect; a bench-gated tuning +step; a small downstream tweak), **add it to the open / most-related +ticket**: reopen `closed` → `in progress` if the prior closure was +premature, extend the scope, list the new sub-items, keep the design +log linear. fox tracks the index by skimming a small set of threads; +spawning #000054.1 / #000055 / #000056 / … for every follow-up turns +the index into a wall of micro-tickets he can't easily keep state on, +and dilutes the design log instead of concentrating it. **Split only +when the new piece needs a *distinct audience*** — specifically, when +it's a self-contained design decision a Dav1d de-novo review (the +external code-review thread that runs against one ticket at a time) +needs to read independently. Architectural inflection points, large +scope changes, or fundamental discipline questions (NLI may touch +`audit_mode`? — #000049 split from #000048) cross that bar. "The +extractor needs a small consumer-side tweak" does not. When in doubt, +ask before opening a new ticket — a two-sentence "fold into #X or +spawn a sibling?" is cheap; an unwanted ticket is friction. Closing +on the implementation commit is good practice **only** when the work +is actually complete end-to-end; pattern-match closure on the +user-visible outcome, not on the commit. + ## Orientation protocol ```bash diff --git a/arborist/concepts/extract.py b/arborist/concepts/extract.py index ea56699..f1a1ac8 100644 --- a/arborist/concepts/extract.py +++ b/arborist/concepts/extract.py @@ -157,9 +157,17 @@ def link_reciprocity_synonym( # in order, to the first letter of one content word in the phrase # (function words filtered). Drops HTTP-shape acronyms where letters # land mid-word — the catch isn't worth the false-positive risk. +# Acronym length floor is 3 ([A-Z]{3,6}, not {2,6}): 2-letter acronyms +# (AI/ML/OS/US/UK/IT/PC/TV) collide too often with common 2-letter +# QUERY tokens like "go"/"is"/"am"/"or" — the expansion then pulls +# unrelated articles into retrieval (2026-05-13 bench regression: "why +# did the dinosaurs go extinct?" → Curious George Brigade via GO-acronym +# edges). The high-value acronyms (CPU/GPU/RAM/DNA/FBI/WHO/…) all +# clear 3 chars; the 2-letter loss is negligible (their canonical +# expansion words appear in body text directly). _ACRONYM_PHRASE_RE = re.compile( r"\b([A-Za-z][A-Za-z\-]+(?:\s+[A-Za-z][A-Za-z\-]+){1,6})\s*" - r"\(\s*([A-Z]{2,6})\s*\)" + r"\(\s*([A-Z]{3,6})\s*\)" ) # Function words that appear inside acronym expansions but never diff --git a/arborist/concepts/query.py b/arborist/concepts/query.py index ac6c974..113b9a2 100644 --- a/arborist/concepts/query.py +++ b/arborist/concepts/query.py @@ -322,6 +322,43 @@ def _load_idf_for(shards_dir: Path, tokens: set[str]) -> dict[str, int]: return {(r["token"] or "").lower(): int(r["df"] or 0) for r in rows} +def _load_neighbor_source_freq( + shards_dir: Path, token: str, neighbors: set[str] +) -> dict[str, int]: + """For one ``token`` whose derived-synonym degree exceeds the + per-token cap, return ``{neighbor: source_root_count}`` — how many + distinct documents asserted the (token, neighbor) synonym + relation. Used to rank-and-truncate over-cap derived expansions + instead of hard-skipping a homonym acronym entirely (#000054 + Phase 2a — CPU has 13 legitimate homonym expansions across the + corpus, of which `central / processing / unit` are anchored by + hundreds of documents and the noise tail by 1-2 each). + + The extractor emits bidirectional edges, so for any (a, b) pair + both rows exist with token=a/target=b and token=b/target=a, each + anchored to the doc that defined them. Counting on one direction + captures the asymmetric assertion. #000054.""" + if not neighbors: + return {} + nlower = {n.lower() for n in neighbors if n} + if not nlower: + return {} + ph = ",".join("?" * len(nlower)) + conn = connect_query(shards_dir=shards_dir) + try: + rows = conn.execute( + f"SELECT target, COUNT(DISTINCT source_root) AS cnt " + f"FROM concept_relations " + f"WHERE relation_kind = 'synonym' " + f" AND token = ? AND target IN ({ph}) " + f"GROUP BY target", + [token.lower()] + list(nlower), + ).fetchall() + finally: + conn.close() + return {(r["target"] or "").lower(): int(r["cnt"] or 0) for r in rows} + + # --------------------------------------------------------------------------- # Public API — matches the legacy ``arborist.qa.concepts`` shape # --------------------------------------------------------------------------- @@ -348,6 +385,79 @@ def _load_idf_for(shards_dir: Path, tokens: set[str]) -> dict[str, int]: MAX_NEIGHBORS_PER_TOKEN = 8 MAX_TOTAL_TOKENS = 50 +def _load_strict_neighbors_for( + shards_dir: Path, tokens: set[str] +) -> dict[str, set[str]]: + """High-trust synonym neighbors only — manual + manual_legacy + + acronym_parens. Excludes ``link_reciprocity`` because reciprocal + wikilinks express topical adjacency, not synonymy (a + ``Dinosaurs`` page reciprocally links to a ``Curious George + Brigade`` page → an edge that should not amplify retrieval). The + multiplicative title-purity rerank uses this strict view; the + additive title-rerank + retrieval routes use the broader + ``synonym_expand``. #000054 Phase 2b.""" + qlower = {t.lower() for t in tokens if t} + if not qlower: + return {} + ph = ",".join("?" * len(qlower)) + params = list(qlower) + list(qlower) + conn = connect_query(shards_dir=shards_dir) + try: + rows = conn.execute( + f"SELECT token, target FROM concept_relations " + f"WHERE relation_kind = 'synonym' " + f" AND evidence_kind IN ('manual', 'manual_legacy', 'acronym_parens') " + f" AND (token IN ({ph}) OR target IN ({ph}))", + params, + ).fetchall() + finally: + conn.close() + out: dict[str, set[str]] = {t: set() for t in qlower} + for r in rows: + a = (r["token"] or "").lower() + b = (r["target"] or "").lower() + if not a or not b or a == b: + continue + if a in out: + out[a].add(b) + if b in out: + out[b].add(a) + return out + + +def synonym_expand_strict( + tokens: set[str], + *, + shards_dir: Path | str | None = None, + max_neighbors_per_token: int = MAX_NEIGHBORS_PER_TOKEN, +) -> set[str]: + """Like ``synonym_expand`` but restricted to high-trust evidence + kinds (manual / acronym_parens) — see ``_load_strict_neighbors_for``. + Used by the title-purity multiplier; the broader ``synonym_expand`` + is right for retrieval (additive boosts) but its inclusion of + noisy ``link_reciprocity`` edges blows up under a multiplicative + rank. #000054 Phase 2b.""" + if not tokens or shards_dir is None: + return set(t.lower() for t in tokens if t) + p = Path(shards_dir) + qlower = {t.lower() for t in tokens if t} + neighbors_by_token = _load_strict_neighbors_for(p, qlower) + expanded: set[str] = set(qlower) + for t in qlower: + nb = neighbors_by_token.get(t, set()) + if not nb: + continue + if len(nb) <= max_neighbors_per_token: + expanded |= nb + continue + # Frequency rank — same discipline as the broad path. + freq = _load_neighbor_source_freq(p, t, nb) + ranked = sorted(nb, key=lambda n: (-freq.get(n, 0), n)) + expanded |= set(ranked[:max_neighbors_per_token]) + return expanded + + + def synonym_expand( tokens: set[str], @@ -392,13 +502,27 @@ def synonym_expand( expanded |= manual_index.get(t, set()) # Derived (corpus-extracted) synonyms cap on per-token degree. # Generic tokens like "person" / "thoughts" / "language" have wide - # noisy neighborhoods in the reciprocal-link graph — skip those. - # Specific tokens with bounded degree expand cleanly. + # noisy neighborhoods in the reciprocal-link graph; specific tokens + # have bounded ones. When the per-token degree exceeds the cap we + # *rank-and-truncate* by source-frequency rather than hard-skipping + # (#000054 Phase 2a) — a homonym acronym ("CPU" has 13 expansions + # across the corpus: central/processing/unit anchored by hundreds + # of docs, canadian/contract/pharmaceutical by 1-2 each) should + # still contribute its dominant expansion, not nothing. The + # frequency rank surfaces the canonical pairing and drops the + # noise tail. Below-cap tokens stay on the fast path (no SQL). for t in qlower: neighbors = derived_index.get(t, set()) - if len(neighbors) > max_neighbors_per_token: + if not neighbors: continue - expanded |= neighbors + if len(neighbors) <= max_neighbors_per_token: + expanded |= neighbors + continue + # Over the per-token cap: rank by source-frequency (desc), + # keep the top ``max_neighbors_per_token``. + freq = _load_neighbor_source_freq(p, t, neighbors) + ranked = sorted(neighbors, key=lambda n: (-freq.get(n, 0), n)) + expanded |= set(ranked[:max_neighbors_per_token]) if len(expanded) > max_total: # IDF-rank the neighbors (rarer = more topical = keep first). # Tokens absent from concept_token_idf get a sentinel high- diff --git a/arborist/qa/concepts.py b/arborist/qa/concepts.py index aaec45f..06eebbb 100644 --- a/arborist/qa/concepts.py +++ b/arborist/qa/concepts.py @@ -25,9 +25,15 @@ from arborist.concepts.query import ( has_compare_phrasing, rivalry_excluded as _rivalry_excluded_impl, synonym_expand as _synonym_expand_impl, + synonym_expand_strict as _synonym_expand_strict_impl, ) -__all__ = ["has_compare_phrasing", "rivalry_excluded", "synonym_expand"] +__all__ = [ + "has_compare_phrasing", + "rivalry_excluded", + "synonym_expand", + "synonym_expand_strict", +] def synonym_expand( @@ -47,6 +53,22 @@ def synonym_expand( return _synonym_expand_impl(tokens, shards_dir=shards_dir) +def synonym_expand_strict( + tokens: set[str], + *, + shards_dir: Path | str | None = None, +) -> set[str]: + """Like ``synonym_expand`` but restricted to high-trust evidence + kinds (manual + manual_legacy + acronym_parens), excluding the + noisier ``link_reciprocity`` channel. For use in the multiplicative + title-purity rerank where reciprocal-link noise blows up; the + broad ``synonym_expand`` is right for retrieval and additive + boosts. #000054 Phase 2b.""" + if shards_dir is None: + return set(t.lower() for t in tokens if t) + return _synonym_expand_strict_impl(tokens, shards_dir=shards_dir) + + def rivalry_excluded( tokens: set[str], compare_phrasing: bool = False, diff --git a/arborist/qa/query.py b/arborist/qa/query.py index 38b76bc..13a201d 100644 --- a/arborist/qa/query.py +++ b/arborist/qa/query.py @@ -59,6 +59,7 @@ from arborist.qa.concepts import ( has_compare_phrasing, rivalry_excluded, synonym_expand, + synonym_expand_strict, ) from arborist.qa.keys import ( DEFAULT_FIDELITY, @@ -175,11 +176,24 @@ def _rerank_by_title( hits: list, question: str, boost: float = 10.0, + *, + shards_dir=None, ) -> list: - """Boost hits whose title overlaps query tokens. Pure ordering aid.""" + """Boost hits whose title overlaps query tokens. Pure ordering aid. + + Uses the synonym-expanded token set so canonical-article titles + that share zero literal-query tokens but contain expansion tokens + (Wikipedia's `Graphics processing unit` for a `GPU` query) earn + the boost. Without this, the literal-query satellites (`GPU + cluster`) get +10 and the canonical article gets 0 — the exact + failure mode #000054 Phase 2 is closing. (Falls back to literal + qtokens when `shards_dir is None`.) + """ qtokens = _title_query_tokens(question) if not qtokens: return hits + if shards_dir is not None: + qtokens = synonym_expand(qtokens, shards_dir=shards_dir) for h in hits: if not h.title: continue @@ -685,12 +699,30 @@ def _search_titles(conn, qtokens: list[str], limit: int) -> list[tuple]: # match coherently. match_expr = " OR ".join(f'"{t.lower().replace(chr(34), chr(34) * 2)}"' for t in bounded) try: + # Rank by FTS5 bm25 ascending (lowest=best match) — titles + # that match MORE of the OR'd expanded tokens float to the + # top. This replaces the prior 2026-05-02 ``LENGTH(title) + # ASC`` tie-break, which only worked when qtokens were 1-3 + # tokens: with the #000054 Phase 2b expansion (qtokens + # union synonym-pool) the candidate set explodes, length- + # asc returns "Unit" / "Unite" / "B unit" / … and the + # multi-token canonical title ("Graphics processing unit" + # = 3 expanded-token hits) gets cut by ``LIMIT``. bm25 + # naturally favors multi-match titles because each OR + # clause that hits contributes to the score. The + # substring-junk problem the length-asc fix solved + # (2026-05-02 "Back to the Future" case) doesn't apply + # here: FTS5 MATCH is tokenized — "out" only matches the + # tokenized word "out", not the substring of "Aberdeen, + # South Dakota". Length-asc stays as the LIKE-fallback + # tie-break below (where the substring issue persists). rows = conn.execute( "SELECT d.document_root, d.document_uri, d.title " "FROM documents_fts AS f " "JOIN documents AS d ON d.rowid = f.rowid " "WHERE documents_fts MATCH ? " - "ORDER BY LENGTH(d.title) ASC LIMIT ?", + "ORDER BY bm25(documents_fts) ASC, LENGTH(d.title) ASC " + "LIMIT ?", (match_expr, over_fetch_limit), ).fetchall() return rows @@ -1142,15 +1174,35 @@ def _search_corpus( out-ranks FTS5 body hits so the actual topic article rises to the top. """ qtokens = _title_query_tokens(question) - # Title-LIKE backup uses ORIGINAL qtokens only (LIKE %tok% can't - # use any index — adding synonyms makes it O(corpus × |accept|)). - # Synonym expansion stays useful in two places: (1) the FTS5 - # OR-mode fallback (top-5 longest pool merged with synonyms — long - # topical synonyms like "neurotechnology" surface relevant titles - # without paying for full-scan), and (2) `_filter_by_title_relevance` - # post-retrieval filtering (in-memory, cheap). - accept_tokens = set(qtokens) + # `accept_tokens` flows into the title-search FTS5 MATCH (line ~1035) + # and the core-keyword route (line ~1099); `accept_stems` is the + # accept-path-1 token-overlap filter in `_filter_by_title_relevance`. + # Pre-#000054 Phase 2b: this was `set(qtokens)` (qtokens only), with + # `or_synonym_pool` used only in the FTS5 body OR-fallback. That + # let the canonical article disappear: "what is a GPU?" expanded to + # {gpu, graphics, processing, unit} via synonym_expand, but only + # `gpu` was used to retrieve titles → the GPU-* satellites filled + # the budget while "Graphics processing unit" never entered the + # candidate pool. Now: use the expanded set in all four routes. + # The expanded set is bounded by `MAX_NEIGHBORS_PER_TOKEN` (8) × + # |qtokens| capped at `MAX_TOTAL_TOKENS` (50), so the SQL clause + # count stays tractable; the old LIKE perf objection (line 1146 + # pre-#000054) was retired with `documents_fts` MATCH replacing + # LIKE in `_search_titles` 2026-05-02. or_synonym_pool = synonym_expand(qtokens, shards_dir=shards_dir) + # `accept_tokens` flows into title-search FTS5 MATCH + title-rerank + # + core-keyword search + title-relevance filter accept-paths. Use + # the **strict** synonym view (manual + acronym_parens evidence; + # excludes link_reciprocity) — link_reciprocity edges express + # topical adjacency, not synonymy, and injecting them into + # retrieval pulled "Curious George Brigade" into "why did the + # dinosaurs go extinct?" via `dinosaurs ↔ curious/george/brigade` + # reciprocal-link edges (2026-05-13 bench regression). The + # acronym-parens expansion that closes CPU→Central-processing-unit + # / GPU→Graphics-processing-unit is preserved because those edges + # ARE strict. `or_synonym_pool` (the broad view) still feeds the + # FTS5 OR-fallback for long-token surfacing per the prior design. + accept_tokens = synonym_expand_strict(qtokens, shards_dir=shards_dir) or set(qtokens) paths: list[Path] if shards_dir is not None: paths = discover_shards(shards_dir) @@ -1285,9 +1337,9 @@ def _rerank( ) progress.emit("search.title_filter", survivors=len(hits)) hits = _rerank_by_body_coverage(hits, question) - hits = _rerank_by_title(hits, question) + hits = _rerank_by_title(hits, question, shards_dir=shards_dir) hits = _rerank_by_source_role(hits, question) - hits = _rerank_by_title_purity(hits, question) + hits = _rerank_by_title_purity(hits, question, shards_dir=shards_dir) return _rerank_by_ordered_token_match(hits, question) @@ -1333,7 +1385,7 @@ def _rerank_by_source_role(hits: list[_Hit], question: str) -> list[_Hit]: return hits -def _rerank_by_title_purity(hits: list[_Hit], question: str) -> list[_Hit]: +def _rerank_by_title_purity(hits: list[_Hit], question: str, *, shards_dir=None) -> list[_Hit]: """Boost titles by both purity AND multi-token-match breadth. Two signals combine here: @@ -1374,6 +1426,21 @@ def _rerank_by_title_purity(hits: list[_Hit], question: str) -> list[_Hit]: qtokens = _title_query_tokens(question) if not qtokens: return hits + # #000054 Phase 2b: expand via the *strict* synonym view (manual + + # acronym_parens evidence) — NOT the broad view that includes + # link_reciprocity. The multiplier ``(1+overlap)*(1+purity)`` + # amplifies; running it against the broad expansion set blows up + # on noisy link_reciprocity synonyms (observed 2026-05-13 bench: + # "why did the dinosaurs go extinct?" → Curious George Brigade + # titled docs got 4× via `dinosaurs ↔ curious/george/brigade` + # reciprocal-link edges, which express topical adjacency, not + # synonymy). The strict view keeps the CPU↔central-processing- + # unit / GPU↔graphics-processing-unit acronym surfacing — those + # are text-pattern edges (acronym_parens) where the phrase + # literally IS the expansion — without amplifying the + # reciprocal-link noise tail. + if shards_dir is not None: + qtokens = synonym_expand_strict(qtokens, shards_dir=shards_dir) or qtokens # Stem-aware matching so possessive / plural variants match. The # 2026-05-01 Dawson's Creek defect: question "dawsons creek" with # title "List of Dawson's Creek episodes" — raw set intersection diff --git a/docs/TICKETS.md b/docs/TICKETS.md index a30718d..a469faf 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -101,7 +101,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| -| #000054 | Acronym-parens concept extractor (closes the abbreviation→expansion retrieval gap) | **closed · 2026-05-13** — `arborist/concepts/extract.py:acronym_parens_synonym` lands as a new corpus-agnostic extractor in `EXTRACTORS` (`evidence_kind="acronym_parens"`). Scans each doc's lead chunk for ` (ACRO)` where the all-caps acronym's letters match the content-word initials of the phrase in order; emits bidirectional synonym edges between the lowercased acronym and each ≥3-char content token of the phrase. Conservative (strict 1:1 initials, function words filtered, repeated definitions deduped per doc). Closes the *retrieval-side* abbreviation gap (`CPU↔central processing unit`, `GPU↔graphics processing unit`, `RAM↔random access memory`, `FBI↔federal bureau of investigation`, `WHO↔world health organization`, …) that `link_reciprocity_synonym` can't reach because the relation lives in body text, not the wiki link graph (Wikipedia represents abbreviation→expansion as a *redirect* — not an edge). Per-shard like all `concept_relations` data; corpus-agnostic so HTML/blogs/textbooks benefit equally. Retrieval-side only — never proof-path. 8 new tests; full suite green. Closes #000050 §2a's CPU/GPU fixture rows *upstream* of vec; the Orwell-shape conceptual-allusion row remains the genuine #000050 justification. Operational follow-up (not code): `arborist concepts derive --extractor acronym_parens` on each shard. | 2026-05-13 | — | +| #000054 | Acronym-parens concept extractor (closes the abbreviation→expansion retrieval gap) | **in progress** — Phase 1 (extractor + 481K edges) landed `58027e9`; Phase 2 (consumer-side surfacing — `synonym_expand` rank-and-truncate over the per-token cap, FTS5-`bm25` ordering in `_search_titles`, expanded `accept_tokens` in title-search + core-keyword + title-rerank, `synonym_expand_strict()` for the multiplicative title-purity rerank to exclude noisy `link_reciprocity` edges, tightened extractor regex to `[A-Z]{3,6}` purging 2-letter homonym edges) pending commit. **End-to-end verified:** `what is a CPU?` → Central processing unit at #1; `what is a GPU?` → Graphics processing unit at #1 EVIDENCE-WARRANTED 1/1; Mount Kilimanjaro / Soviet Union queries unchanged (no regression). 2026-05-13 — `arborist/concepts/extract.py:acronym_parens_synonym` lands as a new corpus-agnostic extractor in `EXTRACTORS` (`evidence_kind="acronym_parens"`). Scans each doc's lead chunk for ` (ACRO)` where the all-caps acronym's letters match the content-word initials of the phrase in order; emits bidirectional synonym edges between the lowercased acronym and each ≥3-char content token of the phrase. Conservative (strict 1:1 initials, function words filtered, repeated definitions deduped per doc). Closes the *retrieval-side* abbreviation gap (`CPU↔central processing unit`, `GPU↔graphics processing unit`, `RAM↔random access memory`, `FBI↔federal bureau of investigation`, `WHO↔world health organization`, …) that `link_reciprocity_synonym` can't reach because the relation lives in body text, not the wiki link graph (Wikipedia represents abbreviation→expansion as a *redirect* — not an edge). Per-shard like all `concept_relations` data; corpus-agnostic so HTML/blogs/textbooks benefit equally. Retrieval-side only — never proof-path. 8 new tests; full suite green. Closes #000050 §2a's CPU/GPU fixture rows *upstream* of vec; the Orwell-shape conceptual-allusion row remains the genuine #000050 justification. Operational follow-up (not code): `arborist concepts derive --extractor acronym_parens` on each shard. | 2026-05-13 | — | | #000053 | Acronym-aware verifier content tokens | **closed · 2026-05-13** — `arborist.qa.evidence._content_tokens` now keeps all-caps 2-3-char acronyms (CPU/GPU/DNA/FBI/USB…) as content tokens instead of dropping every <4-char token; fixes the field case where "what is a CPU?" cited to "CPU design" tripped `TITLE_MISMATCH` spuriously (claim & title share "CPU" but neither registered) — also affects `SUBJECT_TOKENS_ABSENT` (Rule 9), `BARE_NAME_CLAIM`, spotlight-excerpt token pick. Versioned: `content_token_rules: "v2-acronym-aware"` in both default policies + `_VERIFIER_POLICY_FIELDS` → folds into `verifier_policy_hash`, prior cache records orphan on lookup (by design, same discipline as `base_version` / `hyphen_fold_v1`). Monotone toward *fewer* spurious demotes (only relaxes overlap checks, never tightens). 8 new tests; full suite green; `bench-qa-smoke` clean. Does NOT fix the *retrieval* abbreviation→expansion gap (`CPU`→`Central processing unit` = #000050 vec hybrid / `concepts/` synonym edges — the root cause of the satellite-article retrieval). | 2026-05-13 | — | | #000052 | Relevance + coherence meta-cognition (answer-*shape* signals) | in progress — **§3.1 `diagnose_coherence` landed** (lexical, no model: `circular` / `phrase_component_reuse` / `vacuous`; in `arborist/qa/inspect.py`, surfaced via `inspect_cache_key` + `arborist inspect` `· incoherent: `; 9 tests; demote-policy hook deliberately not wired — advisory only). Joins the `diagnose_deflection` / `diagnose_metaphor_deflection` / `diagnose_title_relevance` / soft-preflight family of read-only, demote-only, never-in-proof-path sidecars; `phrase_component_reuse` catches the motivating field case (a subject quoting a phrase, a predicate reusing one of that phrase's own tokens as a bare `the ` referent). **Still open: (2) `diagnose_relevance`** — semantic (not just lexical) "aboutness": does the answer address the question; is each claim about its cited source? Today's checks (subject-anchor token overlap, stemmed title-stem overlap) are *lexical* and a token collision defeats them — a small *aboutness/reranker* model (NOT NLI — entailment ≠ topicality) under #000049 §7's discipline cage verbatim (demotion-only, hash-pinned, `relevance_model_version`→`governance_policy_hash` iff it touches `audit_mode`, shadow-first, `[…]` extra, the §7 #20 haystack lesson — never over the whole context); gated on evidence, travels with #000049's model question. Motivating field case (2026-05-12, fox): the `claim_lattice` query that returned *"the phrase 'Zionist entity' is sometimes used as the entity, referring to the State of Israel"* at `EVIDENCE-WARRANTED-PARTIAL 2/3` — incoherent + token-collision recombination that NLI can't catch (returns *neutral*, not *contradiction*) and both lexical relevance checks waved through. Flags an upstream retrieval ticket (polysemy / title-token-soup) as the root-cause fix, not scoped here. #000049 sibling | 2026-05-12 | — | | #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — | diff --git a/docs/tickets/ticket-000054-acronym-parens-concept-extractor.md b/docs/tickets/ticket-000054-acronym-parens-concept-extractor.md index d6052b0..e8dec69 100644 --- a/docs/tickets/ticket-000054-acronym-parens-concept-extractor.md +++ b/docs/tickets/ticket-000054-acronym-parens-concept-extractor.md @@ -1,6 +1,6 @@ # Ticket #000054 — Acronym-parens concept extractor (closing the abbreviation→expansion retrieval gap) -**Status:** closed · 2026-05-13 — `arborist/concepts/extract.py:acronym_parens_synonym` lands as a new corpus-agnostic extractor in the existing `EXTRACTORS` registry (`evidence_kind = "acronym_parens"`). Scans the lead chunk of every document for the pattern ` (ACRO)` where the all-caps parenthesized acronym's letters match the content-word initials of the phrase in order; emits bidirectional synonym edges between the (lowercased) acronym and each ≥3-char content token of the phrase. Conservative: strict 1:1 initial match, function words filtered, repeated definitions deduped per doc. 8 new tests in `tests/test_concepts_extract.py` (28 → 36); full suite green. Closes the *retrieval-side* abbreviation gap (`CPU↔central processing unit`, `GPU↔graphics processing unit`, `RAM↔random access memory`, `FBI↔federal bureau of investigation`, `WHO↔world health organization`, …) that `link_reciprocity_synonym` structurally can't reach because the relation lives in body text, not the wiki link graph (Wikipedia represents abbreviation→expansion as a *redirect*, which the ingest does not record as an edge). Per-shard like all `concept_relations` data; corpus-agnostic so HTML / blogs / textbooks benefit the same way as Wikipedia. Complements #000050 (vec hybrid) without overlap — the abbreviation cases the vec layer would otherwise have to carry are now closable cheaply; the Orwell-shape *conceptual* allusion remains a vec-only case. +**Status:** in progress — Phase 1 + Phase 2 landed; bench in flight. Phase 1 (`58027e9`): the extractor + 481K edges across 4 shards. Phase 2 (pending commit) wraps four interlocking consumer-side fixes that the end-to-end gap-close exposed in flight, all folded into this ticket per the "don't proliferate" discipline (CLAUDE.md / `feedback_ticket_proliferation`): **(a)** `synonym_expand` rank-and-truncate over the per-token cap (source-frequency descending) instead of hard-skipping — CPU has 13 legitimate homonym expansions across the corpus, Phase 1 alone hit the `MAX_NEIGHBORS_PER_TOKEN=8` cap → zero expansion → no surfacing; **(b)** `_search_titles` orders by FTS5 `bm25` (multi-token-match-favoring) instead of `LENGTH(title) ASC` on the FTS5 path — the length-asc fix the 2026-05-02 "Back to the Future" case introduced was tokenization-needed for the LIKE path but counter-productive on FTS5; **(c)** retrieval routes (title-search, core-keyword) + the title-token additive rerank use the synonym-expanded `accept_tokens`, not qtokens-only; **(d)** the architecturally load-bearing one — `synonym_expand_strict()` (new), a high-trust evidence-kind subset (manual + acronym_parens, **excludes** `link_reciprocity`), used by `accept_tokens` and `_rerank_by_title_purity` (the multiplicative ranker). Reciprocal-wikilink edges express *topical adjacency*, not synonymy (a `Dinosaurs` page reciprocally links to a `Curious George Brigade` page → an edge that should not amplify retrieval — observed 2026-05-13 dinosaurs regression where Phase-2-broad pulled CGB into the top), and a multiplicative ranker over them blows up. The strict view preserves the acronym-parens surfacing (those edges ARE the phrase=expansion identity) while keeping link-reciprocity restricted to additive retrieval-route boosts. Also (e): tightened extractor regex `[A-Z]{2,6}` → `[A-Z]{3,6}` and purged ~21K 2-letter acronym edges (`AI`/`ML`/`OS`/`US`/`UK`/`IT`/`PC`/`TV`) that homonym-collided with 2-char query tokens like `go`/`is`/`am`. **End-to-end verified live:** `what is a CPU?` retrieval pulls *Central processing unit* at #1 (was: only CPU-* satellites); `what is a GPU?` → *Graphics processing unit* at #1 + EVIDENCE-WARRANTED 1/1; `where is mount kilimanjaro located?` → Mount Kilimanjaro at #1; `when did the soviet union dissolve?` → Soviet Union at #1 (regressions from the no-strict intermediate state went away). Bench-qa-smoke pending. Phase 1 prior closure was premature — the extractor produced edges but didn't change retrieval outcomes; this is the actual end-to-end-working bar. **Opened:** 2026-05-13 **Scope:** One new extractor in `arborist/concepts/extract.py` + registry entry + tests. No schema change, no proof-path change (synonym edges are a *retrieval-side* soft signal — they reshape which candidates `qa/query.py` considers but never enter `audit_mode` / `cache_key` / `audit_event_hash`). **Audience:** fox + anyone maintaining the retrieval pipeline + future shifts that wonder "why didn't `CPU` find `Central processing unit`?"