feat: MPS-24 Phase 2.1 — drop shop-vocabulary stems, surface more candidates

First Phase 2 deploy surfaced the wrong candidates on
shop.printableprompts.com: Students (53%), Resource (32%), Activities
(32%), Writing (31%), Practice (30%). These are shop vocabulary —
words that describe the whole shop, not categories within it. A stem
in 53% of products gives a shopper almost no information about which
subset a product belongs to.

- lib/tag_suggest.py: new max_share filter (default 0.4). Stems whose
  product share exceeds this fraction auto-drop as shop vocabulary.
  suggest_clusters now returns (clusters, filtered_count) so the UI
  can show how many stems were filtered.
- top_n default 20 → 50 so the long tail of niche categories surfaces.
- views/shop.py: ?max_share=0.3 (stricter), ?max_share=1 (disable),
  ?top_n=200 URL knobs on the suggestions endpoint — power users tune
  in the browser without redeploying. Floats over 1.0 are interpreted
  as percentages (40 → 0.4) so the URL accepts either form.
- templates/shop_tags.j2: filtered-count hint with copy-paste tuning
  knobs ("?max_share=0.3 stricter, ?max_share=1 to disable").
- scripts/backfill_tags.py: --max-share=0.4 CLI flag.
- Tests: test_suggest_clusters_filters_shop_vocabulary +
  test_suggest_clusters_max_share_one_disables_filter. Existing pure-
  function tests pass max_share=1.0 since their tiny fixtures would
  otherwise be penalised for being small. 1067 total passing.
This commit is contained in:
russell@unturf.com 2026-05-15 09:51:16 -04:00
parent 60a6f02cbc
commit 546e85416e
No known key found for this signature in database
8 changed files with 241 additions and 35 deletions

View file

@ -481,13 +481,19 @@ Phase 2 (shipped): deterministic title-plus-description auto-tagger
in `lib/tag_suggest.py`. Title tokens weight × 3, description × 1 (capped
at 100 unique tokens per product). Pipeline: tokenize → English + per-shop
stopwords → suffix-strip stem → group by stem → drop stems matching
existing tag slugs → rank by product count → label = most frequent
original word for that stem. Surface: button on `/s/{id}/tags`
"Suggested categories" well with one-click Apply / Dismiss per cluster
(`action=apply_suggestion` / `action=dismiss_suggestion`). CLI:
existing tag slugs → **drop stems appearing in > 40% of products as shop
vocabulary** (`max_share` filter — words like "students" / "resource" /
"activity" describe the whole shop, not categories) → rank by product
count → label = most frequent original word for that stem. Returns a
`(clusters, filtered_count)` tuple so the UI can report how many stems
got filtered. Surface: button on `/s/{id}/tags` → "Suggested categories"
well with one-click Apply / Dismiss per cluster
(`action=apply_suggestion` / `action=dismiss_suggestion`). URL knobs:
`?max_share=0.3` (stricter), `?max_share=1` (disable),
`?top_n=200` (show more). CLI:
`python -m make_post_sell.scripts.backfill_tags data/development.ini
--shop=<id> [--apply]`. **Never auto-commits** — operator approves
every cluster.
--shop=<id> [--max-share=0.4] [--apply]`. **Never auto-commits**
operator approves every cluster.
Phase 3 (this ticket, gated): ML categorization via uncloseai endpoint
behind `app.features.ml_categorization.enabled` kill switch (mirror MPS-22).