Phase 2.1's max_share=0.4 filter only caught Students (53%); the other
four printableprompts generics (Resource / Activities / Writing /
Practice, each 30-32%) slipped through. And single-word "First" was
collapsing the real phrase "First Grade" into noise. Three compounding
fixes plus a dedup pass:
- Bigram detection: adjacent non-stopword tokens cluster as phrases.
"Write the Room" → bigram "write room"; "First Grade Math" →
"first grade"; "Valentine's Day Color" → "valentine day"; "Novel
Study" → "novel study". Bigrams get 2× unigram weight per product —
phrases out-rank single words when both cluster equally well.
- Title-required filter (min_title_share, default 0.3): candidate must
appear in title of at least 30% of carrier products. Kills
description-only marketing noise like "versions", "engaged",
"offered", "during", "these", "check", "right", "well", "help",
"time" — words that live in body copy but never in product titles.
- Expanded English stopword list (~80 → ~200): adds generic verbs
("see", "ask", "give", "tell", "show"), marketing fluff ("perfect",
"best", "lovely", "amazing", "favorite"), content-medium nouns
("version", "sheet", "page", "draw", "line", "color", "theme",
"graphic", "answer", "picture"), and their inflections.
- Bigram supersession: when a bigram and one of its component
unigrams overlap ≥ 80% of products, drop the unigram. Operator sees
"Write Room" once, not "Write" + "Room" + "Write Room" three times.
URL knobs: ?max_share=0.3 / ?max_share=1 / ?min_title=0.5 /
?min_title=0 / ?bigrams=0 / ?top_n=200. CLI: --min-title-share,
--no-bigrams flags on scripts/backfill_tags.py.
On a printableprompts-shaped fixture the new defaults surface
Write Room, Novel Study, Valentine Day as bigram phrases plus Math,
Counting, Addition, Literacy, Fall — 13 clean candidates instead of
the original 50 noisy ones.
1078 total tests passing; 5 new pure-function tests cover bigrams,
title-required filter, and supersession dedup.
russell@unturf.com
df65536c17
