russell@unturf.com
81c051e3fc
Phase 2.2 surfaced real categories but left noise: - Color (119), Number (100), Day (63), Room (36) — unigrams fully covered by multiple bigrams, but the prior supersession only considered one bigram at a time so "Day" stayed even though "Valentine's Day" + "Patrick's Day" + … collectively cover all its products. - "Valentine Day" / "Patrick Day" labels read as typo-broken because apostrophes were stripped during cleaning. - 50 candidates wasn't long-tail enough on a 481-product catalog. Three fixes: - Multi-bigram supersession: a unigram drops when the UNION of bigrams containing it covers ≥ 80% of its product set. Iterates all bigrams for the unigram's stem, unions their product sets, computes coverage once. - Apostrophe-preserving tokeniser + stemmer: `_MD_PUNCT` no longer strips `'`; `_WORD` regex accepts a trailing `(?:'[a-z]+)?` so "valentine's" and "patrick's" survive as surface forms. `simple_stem` drops the apostrophe tail before suffix-stripping so "valentine's" stems to "valentine" — the cluster groups correctly while the label vote wins with the readable surface form. Stopword check uses the apostrophe-less base so possessives can't slip past the list. - top_n default 50 → 100. CLI default also bumped. Tested with a Valentine's/Patrick's-heavy sample: bigrams render as "Valentine's Day", "Patrick's Day" with proper apostrophes; the bare "Day" unigram drops because the bigrams together cover all its products. 1080 tests passing. |
||
|---|---|---|
| .. | ||
| lib | ||
| models | ||
| scripts | ||
| static | ||
| templates | ||
| tests | ||
| views | ||
| __init__.py | ||
| config_jinja2.py | ||
| request_methods.py | ||
| routes.py | ||