engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000054: record Phase-2 bench result (n=3 limit=5: 30/45 STRICT, 0 regressions on basics)

Browse source 
`make bench-qa BENCH_QA_N=3 BENCH_QA_LIMIT=5` (2026-05-13T14:24Z) on
the bench/qa_questions.txt set: 30/45 STRICT (67%), zero regressions
on mona lisa / capital of france / new london bridge (each 9/9 STRICT
across quote / pointer / lattice modes). The bench-qa-smoke n=1
flickers I saw ("dinosaurs" S→U, "soviet union" S→H) were LLM
stochasticity at n=1, not retrieval bugs. GNU-linux + python failures
look like genuine hard questions, not Phase-2 side effects.
This commit is contained in:
russell@unturf.com 2026-05-13 10:27:57 -04:00
parent ce855dbdad
commit 79ba3f99fb
No known key found for this signature in database
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/tickets/ticket-000054-acronym-parens-concept-extractor.md
View file

@ -1,6 +1,6 @@
# Ticket #000054 — Acronym-parens concept extractor (closing the abbreviation→expansion retrieval gap)
**Status:** in progress — Phase 1 + Phase 2 landed; bench in flight. Phase 1 (`58027e9`): the extractor + 481K edges across 4 shards. Phase 2 (pending commit) wraps four interlocking consumer-side fixes that the end-to-end gap-close exposed in flight, all folded into this ticket per the "don't proliferate" discipline (CLAUDE.md / `feedback_ticket_proliferation`): **(a)** `synonym_expand` rank-and-truncate over the per-token cap (source-frequency descending) instead of hard-skipping — CPU has 13 legitimate homonym expansions across the corpus, Phase 1 alone hit the `MAX_NEIGHBORS_PER_TOKEN=8` cap → zero expansion → no surfacing; **(b)** `_search_titles` orders by FTS5 `bm25` (multi-token-match-favoring) instead of `LENGTH(title) ASC` on the FTS5 path — the length-asc fix the 2026-05-02 "Back to the Future" case introduced was tokenization-needed for the LIKE path but counter-productive on FTS5; **(c)** retrieval routes (title-search, core-keyword) + the title-token additive rerank use the synonym-expanded `accept_tokens`, not qtokens-only; **(d)** the architecturally load-bearing one — `synonym_expand_strict()` (new), a high-trust evidence-kind subset (manual + acronym_parens, **excludes** `link_reciprocity`), used by `accept_tokens` and `_rerank_by_title_purity` (the multiplicative ranker). Reciprocal-wikilink edges express *topical adjacency*, not synonymy (a `Dinosaurs` page reciprocally links to a `Curious George Brigade` page → an edge that should not amplify retrieval — observed 2026-05-13 dinosaurs regression where Phase-2-broad pulled CGB into the top), and a multiplicative ranker over them blows up. The strict view preserves the acronym-parens surfacing (those edges ARE the phrase=expansion identity) while keeping link-reciprocity restricted to additive retrieval-route boosts. Also (e): tightened extractor regex `[A-Z]{2,6}` → `[A-Z]{3,6}` and purged ~21K 2-letter acronym edges (`AI`/`ML`/`OS`/`US`/`UK`/`IT`/`PC`/`TV`) that homonym-collided with 2-char query tokens like `go`/`is`/`am`. **End-to-end verified live:** `what is a CPU?` retrieval pulls *Central processing unit* at #1 (was: only CPU-* satellites); `what is a GPU?` → *Graphics processing unit* at #1 + EVIDENCE-WARRANTED 1/1; `where is mount kilimanjaro located?` → Mount Kilimanjaro at #1; `when did the soviet union dissolve?` → Soviet Union at #1 (regressions from the no-strict intermediate state went away). Bench-qa-smoke pending. Phase 1 prior closure was premature — the extractor produced edges but didn't change retrieval outcomes; this is the actual end-to-end-working bar.
**Status:** in progress — Phase 1 + Phase 2 landed; bench in flight. Phase 1 (`58027e9`): the extractor + 481K edges across 4 shards. Phase 2 (pending commit) wraps four interlocking consumer-side fixes that the end-to-end gap-close exposed in flight, all folded into this ticket per the "don't proliferate" discipline (CLAUDE.md / `feedback_ticket_proliferation`): **(a)** `synonym_expand` rank-and-truncate over the per-token cap (source-frequency descending) instead of hard-skipping — CPU has 13 legitimate homonym expansions across the corpus, Phase 1 alone hit the `MAX_NEIGHBORS_PER_TOKEN=8` cap → zero expansion → no surfacing; **(b)** `_search_titles` orders by FTS5 `bm25` (multi-token-match-favoring) instead of `LENGTH(title) ASC` on the FTS5 path — the length-asc fix the 2026-05-02 "Back to the Future" case introduced was tokenization-needed for the LIKE path but counter-productive on FTS5; **(c)** retrieval routes (title-search, core-keyword) + the title-token additive rerank use the synonym-expanded `accept_tokens`, not qtokens-only; **(d)** the architecturally load-bearing one — `synonym_expand_strict()` (new), a high-trust evidence-kind subset (manual + acronym_parens, **excludes** `link_reciprocity`), used by `accept_tokens` and `_rerank_by_title_purity` (the multiplicative ranker). Reciprocal-wikilink edges express *topical adjacency*, not synonymy (a `Dinosaurs` page reciprocally links to a `Curious George Brigade` page → an edge that should not amplify retrieval — observed 2026-05-13 dinosaurs regression where Phase-2-broad pulled CGB into the top), and a multiplicative ranker over them blows up. The strict view preserves the acronym-parens surfacing (those edges ARE the phrase=expansion identity) while keeping link-reciprocity restricted to additive retrieval-route boosts. Also (e): tightened extractor regex `[A-Z]{2,6}` → `[A-Z]{3,6}` and purged ~21K 2-letter acronym edges (`AI`/`ML`/`OS`/`US`/`UK`/`IT`/`PC`/`TV`) that homonym-collided with 2-char query tokens like `go`/`is`/`am`. **End-to-end verified live:** `what is a CPU?` retrieval pulls *Central processing unit* at #1 (was: only CPU-* satellites); `what is a GPU?` → *Graphics processing unit* at #1 + EVIDENCE-WARRANTED 1/1; `where is mount kilimanjaro located?` → Mount Kilimanjaro at #1; `when did the soviet union dissolve?` → Soviet Union at #1 (regressions from the no-strict intermediate state went away). **Bench:** `make bench-qa BENCH_QA_N=3 BENCH_QA_LIMIT=5` (2026-05-13T14:24Z) — 30/45 STRICT (67%), zero regressions on `mona lisa` / `capital of france` / `new london bridge` (each 9/9 STRICT across all three modes); the bench-qa-smoke n=1 flickers ("dinosaurs", "soviet union" going S→H/U on a single run) were LLM stochasticity, not a retrieval bug. Phase 1 prior closure was premature — the extractor produced edges but didn't change retrieval outcomes; this is the actual end-to-end-working bar.
**Opened:** 2026-05-13
**Scope:** One new extractor in `arborist/concepts/extract.py` + registry entry + tests. No schema change, no proof-path change (synonym edges are a *retrieval-side* soft signal — they reshape which candidates `qa/query.py` considers but never enter `audit_mode` / `cache_key` / `audit_event_hash`).
**Audience:** fox + anyone maintaining the retrieval pipeline + future shifts that wonder "why didn't `CPU` find `Central processing unit`?"