feat: MPS-24 Phase 2.3 — multi-bigram supersession, apostrophe labels, top_n 100
Phase 2.2 surfaced real categories but left noise: - Color (119), Number (100), Day (63), Room (36) — unigrams fully covered by multiple bigrams, but the prior supersession only considered one bigram at a time so "Day" stayed even though "Valentine's Day" + "Patrick's Day" + … collectively cover all its products. - "Valentine Day" / "Patrick Day" labels read as typo-broken because apostrophes were stripped during cleaning. - 50 candidates wasn't long-tail enough on a 481-product catalog. Three fixes: - Multi-bigram supersession: a unigram drops when the UNION of bigrams containing it covers ≥ 80% of its product set. Iterates all bigrams for the unigram's stem, unions their product sets, computes coverage once. - Apostrophe-preserving tokeniser + stemmer: `_MD_PUNCT` no longer strips `'`; `_WORD` regex accepts a trailing `(?:'[a-z]+)?` so "valentine's" and "patrick's" survive as surface forms. `simple_stem` drops the apostrophe tail before suffix-stripping so "valentine's" stems to "valentine" — the cluster groups correctly while the label vote wins with the readable surface form. Stopword check uses the apostrophe-less base so possessives can't slip past the list. - top_n default 50 → 100. CLI default also bumped. Tested with a Valentine's/Patrick's-heavy sample: bigrams render as "Valentine's Day", "Patrick's Day" with proper apostrophes; the bare "Day" unigram drops because the bigrams together cover all its products. 1080 tests passing.
This commit is contained in:
parent
4e203c4001
commit
81c051e3fc
3 changed files with 121 additions and 29 deletions
|
|
@ -319,6 +319,31 @@ picker is Phase 2).
|
|||
| `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
|
||||
| `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
|
||||
|
||||
### Phase 2.3 — multi-bigram supersession + apostrophe labels + top_n 100 (shipped 2026-05-15)
|
||||
|
||||
Phase 2.2 surfaced real categories but left residue: `Color` (119),
|
||||
`Number` (100), `Day` (63), `Room` (36) — all unigrams that are fully
|
||||
covered by multiple bigrams (e.g. `Day` is covered by `Valentine's Day`
|
||||
+ `Patrick's Day` + others). And bigram labels like `Valentine Day` /
|
||||
`Patrick Day` lost their apostrophes — operators read them as
|
||||
typo-broken. Three fixes:
|
||||
|
||||
- **Multi-bigram supersession**: a unigram drops when the *union* of
|
||||
bigrams containing it covers ≥ 80% of its products. Phase 2.2 only
|
||||
considered single-bigram coverage; now `Day` drops because the
|
||||
combined set of `Valentine's Day` ∪ `Patrick's Day` ∪ … covers it.
|
||||
- **Apostrophe-preserving labels**: tokeniser keeps the possessive /
|
||||
contraction tail (`valentine's`, `patrick's`); stemmer strips it
|
||||
*before* matching but the label vote still wins with the readable
|
||||
surface form. `_MD_PUNCT` no longer kills apostrophes. Stopword
|
||||
check uses the apostrophe-less base so possessives can't slip in.
|
||||
- **`top_n` default 50 → 100** for the long tail of niche categories.
|
||||
|
||||
Result on a Valentine's/Patrick's-heavy sample: bigrams render as
|
||||
`Valentine's Day`, `Patrick's Day` (readable possessives), and the
|
||||
catch-all `Day` unigram disappears because the two bigrams together
|
||||
cover all its products.
|
||||
|
||||
### Phase 2.2 — bigrams + title-required + bigger stopwords (shipped 2026-05-15)
|
||||
|
||||
Phase 2.1's `max_share=0.4` filter only caught one of printableprompts'
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue