feat: MPS-24 Phase 2.1 — drop shop-vocabulary stems, surface more candidates

First Phase 2 deploy surfaced the wrong candidates on
shop.printableprompts.com: Students (53%), Resource (32%), Activities
(32%), Writing (31%), Practice (30%). These are shop vocabulary —
words that describe the whole shop, not categories within it. A stem
in 53% of products gives a shopper almost no information about which
subset a product belongs to.

- lib/tag_suggest.py: new max_share filter (default 0.4). Stems whose
  product share exceeds this fraction auto-drop as shop vocabulary.
  suggest_clusters now returns (clusters, filtered_count) so the UI
  can show how many stems were filtered.
- top_n default 20 → 50 so the long tail of niche categories surfaces.
- views/shop.py: ?max_share=0.3 (stricter), ?max_share=1 (disable),
  ?top_n=200 URL knobs on the suggestions endpoint — power users tune
  in the browser without redeploying. Floats over 1.0 are interpreted
  as percentages (40 → 0.4) so the URL accepts either form.
- templates/shop_tags.j2: filtered-count hint with copy-paste tuning
  knobs ("?max_share=0.3 stricter, ?max_share=1 to disable").
- scripts/backfill_tags.py: --max-share=0.4 CLI flag.
- Tests: test_suggest_clusters_filters_shop_vocabulary +
  test_suggest_clusters_max_share_one_disables_filter. Existing pure-
  function tests pass max_share=1.0 since their tiny fixtures would
  otherwise be penalised for being small. 1067 total passing.
This commit is contained in:
russell@unturf.com 2026-05-15 09:51:16 -04:00
parent 60a6f02cbc
commit 546e85416e
No known key found for this signature in database
8 changed files with 241 additions and 35 deletions

View file

@ -319,6 +319,32 @@ picker is Phase 2).
| `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
| `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
### Phase 2.1 — shop-vocabulary filter + top-N bump (shipped 2026-05-15)
Initial Phase 2 deploy surfaced the wrong candidates on
`shop.printableprompts.com`: `Students`, `Resource`, `Activities`,
`Writing`, `Practice` (each in 3053% of products). These are *shop
vocabulary* — words that describe the whole shop, not categories
within it. A stem in 53% of products tells a shopper almost nothing
about which subset a product belongs to. Fix:
- **`max_share` filter** in `lib/tag_suggest.py:suggest_clusters`
default `0.4` drops any stem in more than 40% of products as shop
vocabulary. Returns a `(clusters, filtered_count)` tuple so callers
can show "auto-dropped N common words."
- **`top_n` default 20 → 50** so the long tail of niche categories
surfaces on a 481-product catalog. Backfill CLI default also bumped.
- **URL knobs** on `/s/{shop_id}/tags`: `?max_share=0.3` (stricter),
`?max_share=1` (disable), `?top_n=200` (show more). No DB column —
power users tune in the browser.
- **Template note** under the suggestions well reports how many stems
got filtered as shop vocabulary plus the tuning hints.
- **CLI flag** `--max-share=0.4` on `scripts/backfill_tags.py`.
- Tests: `test_suggest_clusters_filters_shop_vocabulary` +
`test_suggest_clusters_max_share_one_disables_filter`. Existing
`TestTagSuggestPureFunctions` tests pass `max_share=1.0` (their tiny
fixtures would otherwise be penalised for being small).
## Tests (Phase 1)
### Unit (`test_models.py`)