engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

bench: add hyphenation/orthography fixture (bi-polar is rare?)

Browse source 
Pins the Ticket #000007 reproduction case as a permanent bench
fixture. Stresses the FTS5 hyphen-tokenization asymmetry where
query orthography (`bi-polar`) and indexed orthography (`Bipolar
disorder`) use different hyphen conventions for the same concept.
Pre-fix the medical-condition cluster never surfaced; post-fix
EVIDENCE-WARRANTED 2/2 grounded in `Bipolar disorder` + `Bipolar`
disambiguation.
This commit is contained in:
russell@unturf.com 2026-05-02 14:26:47 -04:00
parent 8fb1fe51d7
commit 5f2d0b3fa7
No known key found for this signature in database
1 changed files with 14 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

14
bench/qa_questions.txt
View file

@ -118,3 +118,17 @@ what does winter is coming mean?
what is the meaning of rosebud?
who said to be or not to be that is the question?
what does the cake is a lie reference?
# hyphenation / orthography — query and corpus use different hyphen
# conventions for the same concept. FTS5 `unicode61` splits on hyphen
# at index AND query time, so `bi-polar` (the orthography many
# English readers actually use) tokenizes to [bi, polar] while the
# Wikipedia article `Bipolar disorder` tokenizes to [bipolar]. Two
# non-overlapping token sets for the same medical condition. The
# query-layer hyphen-fold (Ticket #000007, 2026-05-02) closes this
# class by emitting joined-no-hyphen variants on the query side and
# adding a hyphen-anchor accept path in `_filter_by_title_relevance`.
# Pre-fix: only Bi-Polar (album) / Bi-Polar Blues / BI disambiguation
# surfaced; the medical-condition cluster never reached the LLM.
# Post-fix: `Bipolar disorder` lands in top-K and grounds the answer.
bi-polar is rare?