feat: MPS-24 Phase 2.2 — bigrams + title-required + supersession dedup

Phase 2.1's max_share=0.4 filter only caught Students (53%); the other
four printableprompts generics (Resource / Activities / Writing /
Practice, each 30-32%) slipped through. And single-word "First" was
collapsing the real phrase "First Grade" into noise. Three compounding
fixes plus a dedup pass:

- Bigram detection: adjacent non-stopword tokens cluster as phrases.
  "Write the Room" → bigram "write room"; "First Grade Math" →
  "first grade"; "Valentine's Day Color" → "valentine day"; "Novel
  Study" → "novel study". Bigrams get 2× unigram weight per product —
  phrases out-rank single words when both cluster equally well.
- Title-required filter (min_title_share, default 0.3): candidate must
  appear in title of at least 30% of carrier products. Kills
  description-only marketing noise like "versions", "engaged",
  "offered", "during", "these", "check", "right", "well", "help",
  "time" — words that live in body copy but never in product titles.
- Expanded English stopword list (~80 → ~200): adds generic verbs
  ("see", "ask", "give", "tell", "show"), marketing fluff ("perfect",
  "best", "lovely", "amazing", "favorite"), content-medium nouns
  ("version", "sheet", "page", "draw", "line", "color", "theme",
  "graphic", "answer", "picture"), and their inflections.
- Bigram supersession: when a bigram and one of its component
  unigrams overlap ≥ 80% of products, drop the unigram. Operator sees
  "Write Room" once, not "Write" + "Room" + "Write Room" three times.

URL knobs: ?max_share=0.3 / ?max_share=1 / ?min_title=0.5 /
?min_title=0 / ?bigrams=0 / ?top_n=200. CLI: --min-title-share,
--no-bigrams flags on scripts/backfill_tags.py.

On a printableprompts-shaped fixture the new defaults surface
Write Room, Novel Study, Valentine Day as bigram phrases plus Math,
Counting, Addition, Literacy, Fall — 13 clean candidates instead of
the original 50 noisy ones.

1078 total tests passing; 5 new pure-function tests cover bigrams,
title-required filter, and supersession dedup.
This commit is contained in:
russell@unturf.com 2026-05-15 10:49:54 -04:00
parent 1c43f467fc
commit df65536c17
No known key found for this signature in database
8 changed files with 363 additions and 87 deletions

View file

@ -319,6 +319,39 @@ picker is Phase 2).
| `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
| `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
### Phase 2.2 — bigrams + title-required + bigger stopwords (shipped 2026-05-15)
Phase 2.1's `max_share=0.4` filter only caught one of printableprompts'
five generic candidates (`Students` 53%, the others 3032%). And single
words like `First` (121 products) were collapsing the natural phrase
`First Grade`. Three compounding fixes:
- **Bigrams** in `lib/tag_suggest.py`: adjacent non-stopword tokens
cluster as phrases. `Write the Room` → bigram `write room`,
`First Grade Math``first grade`, `Valentine's Day`
`valentine day`. Bigrams get `BIGRAM_WEIGHT_MULTIPLIER × ` (2×) the
unigram score per product — phrases out-rank single words when both
cluster equally well. URL toggle: `?bigrams=0` to disable.
- **Title-required filter** (`min_title_share`, default `0.3`): a
candidate must appear in the *title* of at least 30% of products
carrying it. Kills description-only marketing noise that doesn't
belong as a category (`versions`, `offered`, `engaged`, `during`,
`these`, `check`, `right`, `well`, `web`, `help`, `build`, `time`).
URL knob: `?min_title=0.5` (stricter), `?min_title=0` (disable).
- **Expanded English stopword list** (~80 → ~200 entries): adds common
filler / generic verbs / marketing fluff / content-medium words like
`see`, `please`, `way`, `well`, `kind`, `type`, `set`, `lot`, `part`,
`time`, `version`, `picture`, `sheet`, `page`, `theme`, `color`,
`draw`, `line`, `cut`, `learn`, `teach`, `offer`, `engage`, `check`,
`build`, `work`, `play`, `help`, `find`, `see`, `look`, `ask`,
`give`, `take`, `tell`, `say` — and their inflections.
Result on a printableprompts-like sample: bigrams `Write Room`,
`Novel Study`, `Valentine Day` rise to the top alongside unigrams
`Math`, `Counting`, `Addition`, `Literacy`. The description-only
noise (`Versions`, `Offered`, `Engaged`, `During`, `These`, `Check`,
`Right`) gets filtered before reaching the operator's screen.
### Phase 2.1 — shop-vocabulary filter + top-N bump (shipped 2026-05-15)
Initial Phase 2 deploy surfaced the wrong candidates on