feat: MPS-24 Phase 2.3 — multi-bigram supersession, apostrophe labels, top_n 100

Phase 2.2 surfaced real categories but left noise:
- Color (119), Number (100), Day (63), Room (36) — unigrams fully covered
  by multiple bigrams, but the prior supersession only considered one
  bigram at a time so "Day" stayed even though "Valentine's Day" +
  "Patrick's Day" + … collectively cover all its products.
- "Valentine Day" / "Patrick Day" labels read as typo-broken because
  apostrophes were stripped during cleaning.
- 50 candidates wasn't long-tail enough on a 481-product catalog.

Three fixes:

- Multi-bigram supersession: a unigram drops when the UNION of bigrams
  containing it covers ≥ 80% of its product set. Iterates all bigrams
  for the unigram's stem, unions their product sets, computes coverage
  once.
- Apostrophe-preserving tokeniser + stemmer: `_MD_PUNCT` no longer
  strips `'`; `_WORD` regex accepts a trailing `(?:'[a-z]+)?` so
  "valentine's" and "patrick's" survive as surface forms.
  `simple_stem` drops the apostrophe tail before suffix-stripping so
  "valentine's" stems to "valentine" — the cluster groups correctly
  while the label vote wins with the readable surface form. Stopword
  check uses the apostrophe-less base so possessives can't slip past
  the list.
- top_n default 50 → 100. CLI default also bumped.

Tested with a Valentine's/Patrick's-heavy sample: bigrams render as
"Valentine's Day", "Patrick's Day" with proper apostrophes; the bare
"Day" unigram drops because the bigrams together cover all its
products. 1080 tests passing.
This commit is contained in:
russell@unturf.com 2026-05-15 11:41:46 -04:00
parent 4e203c4001
commit 81c051e3fc
No known key found for this signature in database
3 changed files with 121 additions and 29 deletions

View file

@ -319,6 +319,31 @@ picker is Phase 2).
| `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
| `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
### Phase 2.3 — multi-bigram supersession + apostrophe labels + top_n 100 (shipped 2026-05-15)
Phase 2.2 surfaced real categories but left residue: `Color` (119),
`Number` (100), `Day` (63), `Room` (36) — all unigrams that are fully
covered by multiple bigrams (e.g. `Day` is covered by `Valentine's Day`
+ `Patrick's Day` + others). And bigram labels like `Valentine Day` /
`Patrick Day` lost their apostrophes — operators read them as
typo-broken. Three fixes:
- **Multi-bigram supersession**: a unigram drops when the *union* of
bigrams containing it covers ≥ 80% of its products. Phase 2.2 only
considered single-bigram coverage; now `Day` drops because the
combined set of `Valentine's Day` `Patrick's Day` … covers it.
- **Apostrophe-preserving labels**: tokeniser keeps the possessive /
contraction tail (`valentine's`, `patrick's`); stemmer strips it
*before* matching but the label vote still wins with the readable
surface form. `_MD_PUNCT` no longer kills apostrophes. Stopword
check uses the apostrophe-less base so possessives can't slip in.
- **`top_n` default 50 → 100** for the long tail of niche categories.
Result on a Valentine's/Patrick's-heavy sample: bigrams render as
`Valentine's Day`, `Patrick's Day` (readable possessives), and the
catch-all `Day` unigram disappears because the two bigrams together
cover all its products.
### Phase 2.2 — bigrams + title-required + bigger stopwords (shipped 2026-05-15)
Phase 2.1's `max_share=0.4` filter only caught one of printableprompts'