feat: MPS-24 Phase 2.2 — bigrams + title-required + supersession dedup
Phase 2.1's max_share=0.4 filter only caught Students (53%); the other
four printableprompts generics (Resource / Activities / Writing /
Practice, each 30-32%) slipped through. And single-word "First" was
collapsing the real phrase "First Grade" into noise. Three compounding
fixes plus a dedup pass:
- Bigram detection: adjacent non-stopword tokens cluster as phrases.
"Write the Room" → bigram "write room"; "First Grade Math" →
"first grade"; "Valentine's Day Color" → "valentine day"; "Novel
Study" → "novel study". Bigrams get 2× unigram weight per product —
phrases out-rank single words when both cluster equally well.
- Title-required filter (min_title_share, default 0.3): candidate must
appear in title of at least 30% of carrier products. Kills
description-only marketing noise like "versions", "engaged",
"offered", "during", "these", "check", "right", "well", "help",
"time" — words that live in body copy but never in product titles.
- Expanded English stopword list (~80 → ~200): adds generic verbs
("see", "ask", "give", "tell", "show"), marketing fluff ("perfect",
"best", "lovely", "amazing", "favorite"), content-medium nouns
("version", "sheet", "page", "draw", "line", "color", "theme",
"graphic", "answer", "picture"), and their inflections.
- Bigram supersession: when a bigram and one of its component
unigrams overlap ≥ 80% of products, drop the unigram. Operator sees
"Write Room" once, not "Write" + "Room" + "Write Room" three times.
URL knobs: ?max_share=0.3 / ?max_share=1 / ?min_title=0.5 /
?min_title=0 / ?bigrams=0 / ?top_n=200. CLI: --min-title-share,
--no-bigrams flags on scripts/backfill_tags.py.
On a printableprompts-shaped fixture the new defaults surface
Write Room, Novel Study, Valentine Day as bigram phrases plus Math,
Counting, Addition, Literacy, Fall — 13 clean candidates instead of
the original 50 noisy ones.
1078 total tests passing; 5 new pure-function tests cover bigrams,
title-required filter, and supersession dedup.
This commit is contained in:
parent
1c43f467fc
commit
df65536c17
8 changed files with 363 additions and 87 deletions
|
|
@ -319,6 +319,39 @@ picker is Phase 2).
|
|||
| `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
|
||||
| `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
|
||||
|
||||
### Phase 2.2 — bigrams + title-required + bigger stopwords (shipped 2026-05-15)
|
||||
|
||||
Phase 2.1's `max_share=0.4` filter only caught one of printableprompts'
|
||||
five generic candidates (`Students` 53%, the others 30–32%). And single
|
||||
words like `First` (121 products) were collapsing the natural phrase
|
||||
`First Grade`. Three compounding fixes:
|
||||
|
||||
- **Bigrams** in `lib/tag_suggest.py`: adjacent non-stopword tokens
|
||||
cluster as phrases. `Write the Room` → bigram `write room`,
|
||||
`First Grade Math` → `first grade`, `Valentine's Day` →
|
||||
`valentine day`. Bigrams get `BIGRAM_WEIGHT_MULTIPLIER × ` (2×) the
|
||||
unigram score per product — phrases out-rank single words when both
|
||||
cluster equally well. URL toggle: `?bigrams=0` to disable.
|
||||
- **Title-required filter** (`min_title_share`, default `0.3`): a
|
||||
candidate must appear in the *title* of at least 30% of products
|
||||
carrying it. Kills description-only marketing noise that doesn't
|
||||
belong as a category (`versions`, `offered`, `engaged`, `during`,
|
||||
`these`, `check`, `right`, `well`, `web`, `help`, `build`, `time`).
|
||||
URL knob: `?min_title=0.5` (stricter), `?min_title=0` (disable).
|
||||
- **Expanded English stopword list** (~80 → ~200 entries): adds common
|
||||
filler / generic verbs / marketing fluff / content-medium words like
|
||||
`see`, `please`, `way`, `well`, `kind`, `type`, `set`, `lot`, `part`,
|
||||
`time`, `version`, `picture`, `sheet`, `page`, `theme`, `color`,
|
||||
`draw`, `line`, `cut`, `learn`, `teach`, `offer`, `engage`, `check`,
|
||||
`build`, `work`, `play`, `help`, `find`, `see`, `look`, `ask`,
|
||||
`give`, `take`, `tell`, `say` — and their inflections.
|
||||
|
||||
Result on a printableprompts-like sample: bigrams `Write Room`,
|
||||
`Novel Study`, `Valentine Day` rise to the top alongside unigrams
|
||||
`Math`, `Counting`, `Addition`, `Literacy`. The description-only
|
||||
noise (`Versions`, `Offered`, `Engaged`, `During`, `These`, `Check`,
|
||||
`Right`) gets filtered before reaching the operator's screen.
|
||||
|
||||
### Phase 2.1 — shop-vocabulary filter + top-N bump (shipped 2026-05-15)
|
||||
|
||||
Initial Phase 2 deploy surfaced the wrong candidates on
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue