arborist/docs/tickets/ticket-000007-query-layer-hyphen-fold.md

Ticket #000007 — Query-layer hyphen folding

Status: closed · landed 2026-05-02 Opened: 2026-05-02 Closed: 2026-05-02 Scope: Add hyphen-folded query variants at retrieval time so bi-polar matches both Bi-Polar Blues (hyphen-tokenized) and Bipolar disorder (joined-tokenized) without re-indexing the corpus. Audience: fox + future blackops shifts. Hard constraint: No canonicalization_version, chunking_version, or schema bump. Existing cache_keys and providence_cache records remain valid. Query-layer only.

---

1. Problem statement

Stock FTS5 with tokenize = 'porter unicode61' (store.py:317, store.py:337) splits on hyphen at index time and at query time. _title_query_tokens (query.py:108) uses [A-Za-z][A-Za-z0-9]*, which also splits on hyphen. Net result:

• Title Bi-Polar (Chilli album) indexes as [bi, polar, chilli, album].
• Title Bipolar disorder indexes as [bipolar, disorder].
• Query bi-polar is rare? produces qtokens {bi, polar, rare}.

Two non-overlapping token sets. The query lands in the album/ disambiguation cluster and misses the medical-condition cluster entirely.

1.1 Reproduction (2026-05-02)

make query Q="bi-polar is rare?" BURN=1

returns EVIDENCE-WARRANTED on a claim grounded in the BI disambiguation page, while the corpus contains every medical- condition article (Bipolar disorder, Bipolar I disorder, Bipolar II disorder, Treatment of bipolar disorder, History of bipolar disorder, International Society for Bipolar Disorders, List of people affected by bipolar disorder, Bipolar spectrum). None retrieved.

The verifier is honest — given that evidence, the claim is grounded. The defect is upstream of the verifier in the retrieval tokenizer asymmetry.

1.2 Why existing knobs don't catch it

• Stem-aware match (_stem_token_for_match) folds possessive/plural only.
• Concept relations are corpus-derived semantic rivalries, not mechanical orthography.
• Title-relevance Rule 8 passes — BI stems to bi, claim text contains Bi-polar. Lexically valid, semantically wrong.
• Phrase-pattern route (n=5/n=6) requires longer questions. A four- word question is below threshold.
• Deflection sidecar passes — subject-anchor rare appears in the answer.

---

2. Design choices

2.1 Option A — query-side dual-emit + hyphen-anchor accept path (RECOMMENDED)

Two-part query-layer change:

1. _title_query_tokens additively emits joined-no-hyphen variants for every hyphenated run in its input. Applies symmetrically to queries AND titles (the function is called on both). Hyphenated query bi-polar produces qtokens {bi, polar, bipolar, rare}; hyphenated title Bi-Polar Blues produces ttokens {bi, polar, blues, bipolar}; non-hyphen title Bipolar disorder is unchanged at {bipolar, disorder}. Fold is purely additive — no existing call site loses tokens it relied on (preserves the Coca-Cola style 2-token-overlap match).

2. _filter_by_title_relevance gains accept-path 5 — a hyphen_fold_anchors set (the joined-form-only variants from the query). Any title whose stem set intersects with the anchor set passes the filter even when title-breadth fails. This rescues non-hyphen titles like Bipolar disorder (which only matches one of four qtokens after fold) from the breadth gate. Empty when the query has no hyphens — zero effect on existing queries.

Without (2), the breadth filter rejects Bipolar disorder (1 of 4 qtokens overlap, breadth threshold = 3) while accepting Bi-Polar Blues (3 of 4 because hyphenated titles double-count via fold). The accept-path 5 closes the asymmetry without disrupting the breadth metric for non-hyphen queries.

Pros:

• No re-index, no version bump, no cache invalidation.
• Symmetric: queries hit hyphenated AND non-hyphenated indexed forms.
• Single helper (_hyphen_fold_variants) + single new accept path.
• Zero blast radius for non-hyphen queries.

Cons:

• Body-FTS still uses arborist/search/fts5.py:_query_tokens (a separate function) and stays on AND-mode bi AND polar AND rare. Body-FTS won't pull in Bipolar disorder directly. Title-route + accept-path 5 + rerank carries the recall load. Acceptable: title route is independent and pulls the right doc into top-K via documents_fts MATCH on bipolar.
• One-way: query bipolar does NOT also probe bi+polar at the body route. Different problem; no observed demand.
• Slight noise risk on coincidence collisions: query high-school also probes highschool. Empirically rare; bench will measure.

2.2 Option B — title-route only fold, no accept path

Apply hyphen-folding only at _search_titles, not at _filter_by_title_relevance.

Cons: Title-FTS pulls Bipolar disorder in but the breadth filter rejects it (1 of N qtoken overlap < threshold). Half-fix.

2.3 Option C — substring trigram tokenizer

Switch to trigram for either or both FTS5 indexes.

Cons: BM25 ranking degrades meaningfully; index size inflates. Forces full re-index; bumps canonicalization_version. Out of scope per ticket's hard constraint.

2.4 Option D — custom tokenizer with hyphen-fold at index time

Right answer in the long run, but bumps canonicalization_version and stales every prior cache record. Tracked separately as research.

2.5 Recommendation

A. Cheapest, reversible, no migration. Ships as the immediate fix; index-time work earns a separate research ticket if and when bench shows it's worth the version-bump cost.

---

3. Implementation sketch

1. arborist/qa/query.py — new module-level constant _HYPHEN_RUN_RE = re.compile(r"[A-Za-z][A-Za-z0-9]*(?:-[A-Za-z][A-Za-z0-9]*)+") plus pure helper:

   def _hyphen_fold_variants(s: str) -> set[str]:
       out: set[str] = set()
       for run in _HYPHEN_RUN_RE.findall(s):
           joined = run.replace("-", "").lower()
           if len(joined) > 1 and joined not in _TITLE_STOPWORDS:
               out.add(joined)
       return out
   

2. _title_query_tokens — fold the variants into the returned set:

   def _title_query_tokens(s: str) -> set[str]:
       base = {
           t.lower()
           for t in _TITLE_TOKEN_RE.findall(s)
           if t.lower() not in _TITLE_STOPWORDS and len(t) > 1
       }
       base |= _hyphen_fold_variants(s)
       return base
   

3. _filter_by_title_relevance — accept new optional kwarg hyphen_fold_anchors: set[str] | None = None. Add accept-path 5 inside the per-hit loop:

   if hyphen_fold_anchors:
       anchor_stems = {_stem_token_for_match(a) for a in hyphen_fold_anchors}
       if anchor_stems & ttokens_stem:
           kept.append(h)
           continue
   

4. _rerank — pass-through hyphen_fold_anchors to _filter_by_title_relevance.

5. Caller at query.py:1543 ish — compute hyphen_fold_anchors = _hyphen_fold_variants(retrieval_query) and pass to _rerank.

6. DEFAULT_QUERY_POLICY (and runner.py:DEFAULT_POLICY) — add "hyphen_fold_v1": True. Folds into governance_policy_hash automatically; flipping to False later produces a clean cache split. Code unconditionally applies the fold; the flag is a policy-hash marker so an auditor can tell which records were produced under the new rule.

7. Tests (tests/test_query.py):

   • Unit: _hyphen_fold_variants("bi-polar is rare?") returns {"bipolar"}.
   • Unit: _hyphen_fold_variants("plain query") returns set().
   • Unit: _title_query_tokens("bi-polar is rare?") returns {"bi", "polar", "bipolar", "rare"} (after stopword strip).
   • Integration: synthetic shard with one hyphenated-title doc and one joined-title doc; query the joined form, both surface in top-K.
   • Integration: query bi-polar against a fixture whose only relevant doc is titled Bipolar disorder; confirm it lands in result["sources"].

8. Bench: add four hyphen-stress questions to bench/qa_sweep.py (bi-polar is rare?, co-operative banking, e-mail history, re-enter atmosphere). Baseline + post-patch strict-rate.

---

4. Out of scope

• Body-FTS hyphen handling (see §2.1 cons). Title-route carries retrieval; revisit if a bench fixture shows title-route alone is insufficient.
• Symmetric joined → split variant (bipolar query also probing bi-polar). Demand-driven; revisit if a corpus query exhibits the inverse failure.
• Index-time hyphen handling. Forbidden by hard constraint; future research ticket.
• Multilingual / compound / diacritic / numeric-internal classes. Same research-ticket scope.
• Bumping canonicalization_version. Forbidden.

---

5. Status

Closed 2026-05-02. Landed Option A as designed:

• arborist/qa/query.py — _HYPHEN_RUN_RE constant + _hyphen_fold_variants(s) helper. _title_query_tokens(s) additively merges the variants. _filter_by_title_relevance gains optional hyphen_fold_anchors kwarg + accept-path 5 (title stem-overlap with anchor stems passes the filter even when title-breadth fails). _rerank threads the kwarg. _search_corpus caller computes _hyphen_fold_variants(retrieval_query) and passes through.
• arborist/qa/query.py:DEFAULT_QUERY_POLICY and arborist/qa/runner.py:DEFAULT_POLICY — hyphen_fold_v1: True marker. Folds into governance_policy_hash so records produced under the new rule cache-split cleanly.
• tests/test_query.py — 4 new tests: test_unit_hyphen_fold_variants_emits_joined_form, test_unit_title_query_tokens_includes_hyphen_fold_additively, test_unit_title_query_tokens_no_hyphen_unchanged, test_integration_hyphenated_query_retrieves_joined_title.

Full suite: 760 passed, 34 skipped. Live shard repro for bi-polar is rare? now retrieves Bipolar disorder (source #5) and Bipolar disambiguation (source #7); answer cites both: "Bi-polar disorder is not rare; it affects approximately 2.8% of the U.S. population, or about 5.7 million adults." EVIDENCE-WARRANTED, 2/2 claims, properly grounded — defect closed.

Bench-stress fixtures and the body-FTS hyphen handling stay deferred per §4.