Phase 2.2 surfaced real categories but left noise:
- Color (119), Number (100), Day (63), Room (36) — unigrams fully covered
by multiple bigrams, but the prior supersession only considered one
bigram at a time so "Day" stayed even though "Valentine's Day" +
"Patrick's Day" + … collectively cover all its products.
- "Valentine Day" / "Patrick Day" labels read as typo-broken because
apostrophes were stripped during cleaning.
- 50 candidates wasn't long-tail enough on a 481-product catalog.
Three fixes:
- Multi-bigram supersession: a unigram drops when the UNION of bigrams
containing it covers ≥ 80% of its product set. Iterates all bigrams
for the unigram's stem, unions their product sets, computes coverage
once.
- Apostrophe-preserving tokeniser + stemmer: `_MD_PUNCT` no longer
strips `'`; `_WORD` regex accepts a trailing `(?:'[a-z]+)?` so
"valentine's" and "patrick's" survive as surface forms.
`simple_stem` drops the apostrophe tail before suffix-stripping so
"valentine's" stems to "valentine" — the cluster groups correctly
while the label vote wins with the readable surface form. Stopword
check uses the apostrophe-less base so possessives can't slip past
the list.
- top_n default 50 → 100. CLI default also bumped.
Tested with a Valentine's/Patrick's-heavy sample: bigrams render as
"Valentine's Day", "Patrick's Day" with proper apostrophes; the bare
"Day" unigram drops because the bigrams together cover all its
products. 1080 tests passing.
russell@unturf.com
81c051e3fc
