# Ticket #000007 — Query-layer hyphen folding

**Status:** closed · landed 2026-05-02
**Opened:** 2026-05-02
**Closed:** 2026-05-02
**Scope:** Add hyphen-folded query variants at retrieval time so
`bi-polar` matches both `Bi-Polar Blues` (hyphen-tokenized) and
`Bipolar disorder` (joined-tokenized) without re-indexing the corpus.
**Audience:** fox + future blackops shifts.
**Hard constraint:** No `canonicalization_version`, `chunking_version`,
or schema bump. Existing `cache_key`s and `providence_cache` records
remain valid. Query-layer only.

---

## 1. Problem statement

Stock FTS5 with `tokenize = 'porter unicode61'` (`store.py:317`,
`store.py:337`) splits on hyphen at index time and at query time.
`_title_query_tokens` (`query.py:108`) uses `[A-Za-z][A-Za-z0-9]*`,
which also splits on hyphen. Net result:

- Title `Bi-Polar (Chilli album)` indexes as `[bi, polar, chilli, album]`.
- Title `Bipolar disorder` indexes as `[bipolar, disorder]`.
- Query `bi-polar is rare?` produces qtokens `{bi, polar, rare}`.

Two non-overlapping token sets. The query lands in the album/
disambiguation cluster and misses the medical-condition cluster
entirely.

### 1.1 Reproduction (2026-05-02)

```
make query Q="bi-polar is rare?" BURN=1
```

returns EVIDENCE-WARRANTED on a claim grounded in the `BI`
disambiguation page, while the corpus contains every medical-
condition article (`Bipolar disorder`, `Bipolar I disorder`,
`Bipolar II disorder`, `Treatment of bipolar disorder`,
`History of bipolar disorder`, `International Society for Bipolar
Disorders`, `List of people affected by bipolar disorder`,
`Bipolar spectrum`). None retrieved.

The verifier is honest — given that evidence, the claim is grounded.
The defect is upstream of the verifier in the retrieval tokenizer
asymmetry.

### 1.2 Why existing knobs don't catch it

- Stem-aware match (`_stem_token_for_match`) folds possessive/plural
  only.
- Concept relations are corpus-derived semantic rivalries, not
  mechanical orthography.
- Title-relevance Rule 8 passes — `BI` stems to `bi`, claim text
  contains `Bi-polar`. Lexically valid, semantically wrong.
- Phrase-pattern route (n=5/n=6) requires longer questions. A four-
  word question is below threshold.
- Deflection sidecar passes — subject-anchor `rare` appears in the
  answer.

---

## 2. Design choices

### 2.1 Option A — query-side dual-emit + hyphen-anchor accept path (RECOMMENDED)

Two-part query-layer change:

1. `_title_query_tokens` additively emits joined-no-hyphen variants
   for every hyphenated run in its input. Applies symmetrically to
   queries AND titles (the function is called on both). Hyphenated
   query `bi-polar` produces qtokens `{bi, polar, bipolar, rare}`;
   hyphenated title `Bi-Polar Blues` produces ttokens `{bi, polar,
   blues, bipolar}`; non-hyphen title `Bipolar disorder` is
   unchanged at `{bipolar, disorder}`. Fold is purely additive — no
   existing call site loses tokens it relied on (preserves the
   `Coca-Cola` style 2-token-overlap match).

2. `_filter_by_title_relevance` gains accept-path 5 — a
   `hyphen_fold_anchors` set (the joined-form-only variants from the
   query). Any title whose stem set intersects with the anchor set
   passes the filter even when title-breadth fails. This rescues
   non-hyphen titles like `Bipolar disorder` (which only matches one
   of four qtokens after fold) from the breadth gate. Empty when the
   query has no hyphens — zero effect on existing queries.

Without (2), the breadth filter rejects `Bipolar disorder` (1 of 4
qtokens overlap, breadth threshold = 3) while accepting `Bi-Polar
Blues` (3 of 4 because hyphenated titles double-count via fold).
The accept-path 5 closes the asymmetry without disrupting the
breadth metric for non-hyphen queries.

**Pros:**
- No re-index, no version bump, no cache invalidation.
- Symmetric: queries hit hyphenated AND non-hyphenated indexed forms.
- Single helper (`_hyphen_fold_variants`) + single new accept path.
- Zero blast radius for non-hyphen queries.

**Cons:**
- Body-FTS still uses `arborist/search/fts5.py:_query_tokens` (a
  separate function) and stays on AND-mode `bi AND polar AND rare`.
  Body-FTS won't pull in `Bipolar disorder` directly. Title-route +
  accept-path 5 + rerank carries the recall load. Acceptable: title
  route is independent and pulls the right doc into top-K via
  documents_fts MATCH on `bipolar`.
- One-way: query `bipolar` does NOT also probe `bi+polar` at the
  body route. Different problem; no observed demand.
- Slight noise risk on coincidence collisions: query `high-school`
  also probes `highschool`. Empirically rare; bench will measure.

### 2.2 Option B — title-route only fold, no accept path

Apply hyphen-folding only at `_search_titles`, not at
`_filter_by_title_relevance`.

**Cons:** Title-FTS pulls `Bipolar disorder` in but the breadth
filter rejects it (1 of N qtoken overlap < threshold). Half-fix.

### 2.3 Option C — substring `trigram` tokenizer

Switch to `trigram` for either or both FTS5 indexes.

**Cons:** BM25 ranking degrades meaningfully; index size inflates.
Forces full re-index; bumps `canonicalization_version`. Out of scope
per ticket's hard constraint.

### 2.4 Option D — custom tokenizer with hyphen-fold at index time

Right answer in the long run, but bumps `canonicalization_version`
and stales every prior cache record. Tracked separately as research.

### 2.5 Recommendation

**A.** Cheapest, reversible, no migration. Ships as the immediate
fix; index-time work earns a separate research ticket if and when
bench shows it's worth the version-bump cost.

---

## 3. Implementation sketch

1. **`arborist/qa/query.py`** — new module-level constant
   `_HYPHEN_RUN_RE = re.compile(r"[A-Za-z][A-Za-z0-9]*(?:-[A-Za-z][A-Za-z0-9]*)+")`
   plus pure helper:

   ```python
   def _hyphen_fold_variants(s: str) -> set[str]:
       out: set[str] = set()
       for run in _HYPHEN_RUN_RE.findall(s):
           joined = run.replace("-", "").lower()
           if len(joined) > 1 and joined not in _TITLE_STOPWORDS:
               out.add(joined)
       return out
   ```

2. **`_title_query_tokens`** — fold the variants into the returned
   set:

   ```python
   def _title_query_tokens(s: str) -> set[str]:
       base = {
           t.lower()
           for t in _TITLE_TOKEN_RE.findall(s)
           if t.lower() not in _TITLE_STOPWORDS and len(t) > 1
       }
       base |= _hyphen_fold_variants(s)
       return base
   ```

3. **`_filter_by_title_relevance`** — accept new optional kwarg
   `hyphen_fold_anchors: set[str] | None = None`. Add accept-path 5
   inside the per-hit loop:

   ```python
   if hyphen_fold_anchors:
       anchor_stems = {_stem_token_for_match(a) for a in hyphen_fold_anchors}
       if anchor_stems & ttokens_stem:
           kept.append(h)
           continue
   ```

4. **`_rerank`** — pass-through `hyphen_fold_anchors` to
   `_filter_by_title_relevance`.

5. **Caller at `query.py:1543` ish** — compute
   `hyphen_fold_anchors = _hyphen_fold_variants(retrieval_query)`
   and pass to `_rerank`.

6. **`DEFAULT_QUERY_POLICY`** (and `runner.py:DEFAULT_POLICY`) —
   add `"hyphen_fold_v1": True`. Folds into
   `governance_policy_hash` automatically; flipping to `False`
   later produces a clean cache split. Code unconditionally applies
   the fold; the flag is a policy-hash marker so an auditor can
   tell which records were produced under the new rule.

7. **Tests** (`tests/test_query.py`):
   - Unit: `_hyphen_fold_variants("bi-polar is rare?")` returns
     `{"bipolar"}`.
   - Unit: `_hyphen_fold_variants("plain query")` returns `set()`.
   - Unit: `_title_query_tokens("bi-polar is rare?")` returns
     `{"bi", "polar", "bipolar", "rare"}` (after stopword strip).
   - Integration: synthetic shard with one hyphenated-title doc and
     one joined-title doc; query the joined form, both surface in
     top-K.
   - Integration: query `bi-polar` against a fixture whose only
     relevant doc is titled `Bipolar disorder`; confirm it lands in
     `result["sources"]`.

8. **Bench**: add four hyphen-stress questions to
   `bench/qa_sweep.py` (`bi-polar is rare?`, `co-operative banking`,
   `e-mail history`, `re-enter atmosphere`). Baseline + post-patch
   strict-rate.

---

## 4. Out of scope

- Body-FTS hyphen handling (see §2.1 cons). Title-route carries
  retrieval; revisit if a bench fixture shows title-route alone is
  insufficient.
- Symmetric joined → split variant (`bipolar` query also probing
  `bi-polar`). Demand-driven; revisit if a corpus query exhibits
  the inverse failure.
- Index-time hyphen handling. Forbidden by hard constraint; future
  research ticket.
- Multilingual / compound / diacritic / numeric-internal classes.
  Same research-ticket scope.
- Bumping `canonicalization_version`. Forbidden.

---

## 5. Status

Closed 2026-05-02. Landed Option A as designed:

- `arborist/qa/query.py` — `_HYPHEN_RUN_RE` constant +
  `_hyphen_fold_variants(s)` helper. `_title_query_tokens(s)`
  additively merges the variants. `_filter_by_title_relevance`
  gains optional `hyphen_fold_anchors` kwarg + accept-path 5
  (title stem-overlap with anchor stems passes the filter even
  when title-breadth fails). `_rerank` threads the kwarg.
  `_search_corpus` caller computes
  `_hyphen_fold_variants(retrieval_query)` and passes through.
- `arborist/qa/query.py:DEFAULT_QUERY_POLICY` and
  `arborist/qa/runner.py:DEFAULT_POLICY` — `hyphen_fold_v1: True`
  marker. Folds into `governance_policy_hash` so records
  produced under the new rule cache-split cleanly.
- `tests/test_query.py` — 4 new tests:
  `test_unit_hyphen_fold_variants_emits_joined_form`,
  `test_unit_title_query_tokens_includes_hyphen_fold_additively`,
  `test_unit_title_query_tokens_no_hyphen_unchanged`,
  `test_integration_hyphenated_query_retrieves_joined_title`.

Full suite: 760 passed, 34 skipped. Live shard repro for
`bi-polar is rare?` now retrieves `Bipolar disorder` (source
#5) and `Bipolar` disambiguation (source #7); answer cites
both: *"Bi-polar disorder is not rare; it affects approximately
2.8% of the U.S. population, or about 5.7 million adults."*
EVIDENCE-WARRANTED, 2/2 claims, properly grounded — defect closed.

Bench-stress fixtures and the body-FTS hyphen handling stay
deferred per §4.
