From 5f2d0b3fa790a1b07540c6455c1499439bdf71a5 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 2 May 2026 14:26:47 -0400 Subject: [PATCH] bench: add hyphenation/orthography fixture (bi-polar is rare?) Pins the Ticket #000007 reproduction case as a permanent bench fixture. Stresses the FTS5 hyphen-tokenization asymmetry where query orthography (`bi-polar`) and indexed orthography (`Bipolar disorder`) use different hyphen conventions for the same concept. Pre-fix the medical-condition cluster never surfaced; post-fix EVIDENCE-WARRANTED 2/2 grounded in `Bipolar disorder` + `Bipolar` disambiguation. --- bench/qa_questions.txt | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/bench/qa_questions.txt b/bench/qa_questions.txt index 786062f..dbccbd4 100644 --- a/bench/qa_questions.txt +++ b/bench/qa_questions.txt @@ -118,3 +118,17 @@ what does winter is coming mean? what is the meaning of rosebud? who said to be or not to be that is the question? what does the cake is a lie reference? + +# hyphenation / orthography — query and corpus use different hyphen +# conventions for the same concept. FTS5 `unicode61` splits on hyphen +# at index AND query time, so `bi-polar` (the orthography many +# English readers actually use) tokenizes to [bi, polar] while the +# Wikipedia article `Bipolar disorder` tokenizes to [bipolar]. Two +# non-overlapping token sets for the same medical condition. The +# query-layer hyphen-fold (Ticket #000007, 2026-05-02) closes this +# class by emitting joined-no-hyphen variants on the query side and +# adding a hyphen-anchor accept path in `_filter_by_title_relevance`. +# Pre-fix: only Bi-Polar (album) / Bi-Polar Blues / BI disambiguation +# surfaced; the medical-condition cluster never reached the LLM. +# Post-fix: `Bipolar disorder` lands in top-K and grounds the answer. +bi-polar is rare?