feat: MPS-24 Phase 2.1 — drop shop-vocabulary stems, surface more candidates

First Phase 2 deploy surfaced the wrong candidates on
shop.printableprompts.com: Students (53%), Resource (32%), Activities
(32%), Writing (31%), Practice (30%). These are shop vocabulary —
words that describe the whole shop, not categories within it. A stem
in 53% of products gives a shopper almost no information about which
subset a product belongs to.

- lib/tag_suggest.py: new max_share filter (default 0.4). Stems whose
  product share exceeds this fraction auto-drop as shop vocabulary.
  suggest_clusters now returns (clusters, filtered_count) so the UI
  can show how many stems were filtered.
- top_n default 20 → 50 so the long tail of niche categories surfaces.
- views/shop.py: ?max_share=0.3 (stricter), ?max_share=1 (disable),
  ?top_n=200 URL knobs on the suggestions endpoint — power users tune
  in the browser without redeploying. Floats over 1.0 are interpreted
  as percentages (40 → 0.4) so the URL accepts either form.
- templates/shop_tags.j2: filtered-count hint with copy-paste tuning
  knobs ("?max_share=0.3 stricter, ?max_share=1 to disable").
- scripts/backfill_tags.py: --max-share=0.4 CLI flag.
- Tests: test_suggest_clusters_filters_shop_vocabulary +
  test_suggest_clusters_max_share_one_disables_filter. Existing pure-
  function tests pass max_share=1.0 since their tiny fixtures would
  otherwise be penalised for being small. 1067 total passing.

CLAUDE.md

@@ -481,13 +481,19 @@ Phase 2 (shipped): deterministic title-plus-description auto-tagger
 in `lib/tag_suggest.py`. Title tokens weight × 3, description × 1 (capped
 at 100 unique tokens per product). Pipeline: tokenize → English + per-shop
 stopwords → suffix-strip stem → group by stem → drop stems matching
-existing tag slugs → rank by product count → label = most frequent
+existing tag slugs → **drop stems appearing in > 40% of products as shop
-original word for that stem. Surface: button on `/s/{id}/tags` →
+vocabulary** (`max_share` filter — words like "students" / "resource" /
-"Suggested categories" well with one-click Apply / Dismiss per cluster
+"activity" describe the whole shop, not categories) → rank by product
-(`action=apply_suggestion` / `action=dismiss_suggestion`). CLI:
+count → label = most frequent original word for that stem. Returns a
+`(clusters, filtered_count)` tuple so the UI can report how many stems
+got filtered. Surface: button on `/s/{id}/tags` → "Suggested categories"
+well with one-click Apply / Dismiss per cluster
+(`action=apply_suggestion` / `action=dismiss_suggestion`). URL knobs:
+`?max_share=0.3` (stricter), `?max_share=1` (disable),
+`?top_n=200` (show more). CLI:
 `python -m make_post_sell.scripts.backfill_tags data/development.ini
---shop=<id> [--apply]`. **Never auto-commits** — operator approves
+--shop=<id> [--max-share=0.4] [--apply]`. **Never auto-commits** —
-every cluster.
+operator approves every cluster.
 
 Phase 3 (this ticket, gated): ML categorization via uncloseai endpoint
 behind `app.features.ml_categorization.enabled` kill switch (mirror MPS-22).

docs/tickets/mps-24.md

@@ -319,6 +319,32 @@ picker is Phase 2).
 | `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
 | `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
 
+### Phase 2.1 — shop-vocabulary filter + top-N bump (shipped 2026-05-15)
+
+Initial Phase 2 deploy surfaced the wrong candidates on
+`shop.printableprompts.com`: `Students`, `Resource`, `Activities`,
+`Writing`, `Practice` (each in 30–53% of products). These are *shop
+vocabulary* — words that describe the whole shop, not categories
+within it. A stem in 53% of products tells a shopper almost nothing
+about which subset a product belongs to. Fix:
+
+- **`max_share` filter** in `lib/tag_suggest.py:suggest_clusters` —
+  default `0.4` drops any stem in more than 40% of products as shop
+  vocabulary. Returns a `(clusters, filtered_count)` tuple so callers
+  can show "auto-dropped N common words."
+- **`top_n` default 20 → 50** so the long tail of niche categories
+  surfaces on a 481-product catalog. Backfill CLI default also bumped.
+- **URL knobs** on `/s/{shop_id}/tags`: `?max_share=0.3` (stricter),
+  `?max_share=1` (disable), `?top_n=200` (show more). No DB column —
+  power users tune in the browser.
+- **Template note** under the suggestions well reports how many stems
+  got filtered as shop vocabulary plus the tuning hints.
+- **CLI flag** `--max-share=0.4` on `scripts/backfill_tags.py`.
+- Tests: `test_suggest_clusters_filters_shop_vocabulary` +
+  `test_suggest_clusters_max_share_one_disables_filter`. Existing
+  `TestTagSuggestPureFunctions` tests pass `max_share=1.0` (their tiny
+  fixtures would otherwise be penalised for being small).
+
 ## Tests (Phase 1)
 
 ### Unit (`test_models.py`)

make_post_sell/lib/tag_suggest.py

@@ -31,7 +31,16 @@ DESCRIPTION_WEIGHT = 1
 DEFAULT_MIN_PRODUCTS = 2
 
 # Default cap on how many candidate clusters we return per call.
-DEFAULT_TOP_N = 20
+# Set generously — large catalogues (printableprompts has 481) carry many
+# valid niche categories beyond the obvious top 20.
+DEFAULT_TOP_N = 50
+
+# A stem appearing in more than this fraction of products is treated as
+# **shop vocabulary** — words the operator uses to describe everything
+# they sell, not differentiators between products. E.g. on a K-1 printables
+# shop: students / resource / activity / practice — every product is one
+# of those. Auto-drop them so the next 20 candidates are actually useful.
+DEFAULT_MAX_SHARE = 0.4
 
 # English stopwords — small, hand-tuned for product copy. Operators add
 # shop-specific extras via shop.tag_stopwords_json (e.g. printableprompts
@@ -185,6 +194,7 @@ def suggest_clusters(
     existing_tag_slugs=None,
     min_products=DEFAULT_MIN_PRODUCTS,
     top_n=DEFAULT_TOP_N,
+    max_share=DEFAULT_MAX_SHARE,
 ):
     """Group products by shared stems and rank candidate categories.
 
@@ -197,16 +207,23 @@ def suggest_clusters(
                             already-applied categories.
         min_products: minimum products sharing a stem to qualify.
         top_n: maximum candidate clusters returned.
+        max_share: fraction (0..1). A stem appearing in more than this
+                   share of the catalog is dropped as shop vocabulary.
+                   `1.0` disables the filter.
 
     Returns:
-        List of dicts (ranked, longest first):
+        Tuple of (clusters, filtered_count):
-            {
+            clusters: list of dicts ranked by product count desc:
-                "stem":  "season",
+                {
-                "label": "Seasonal",  # most frequent original word
+                    "stem":  "season",
-                "score": 117,         # sum of weighted stem hits
+                    "label": "Seasonal",  # most frequent original word
-                "product_ids": [<uuid>, <uuid>, ...],
+                    "score": 117,         # sum of weighted stem hits
-                "sample_titles": ["Spring Math", "Autumn Read-Alouds", ...],
+                    "product_ids": [<uuid>, <uuid>, ...],
-            }
+                    "sample_titles": ["Spring Math", "Autumn Reading", ...],
+                }
+            filtered_count: number of stems auto-dropped as shop vocabulary
+                            (so callers can show "n stems filtered as shop
+                            vocabulary").
     """
     from slugify import slugify  # local import; only needed here
 
@@ -218,7 +235,9 @@ def suggest_clusters(
     stem_labels = defaultdict(Counter)  # stem -> Counter(original_word)
     stem_titles = defaultdict(list)     # stem -> [title, ...] for samples
 
+    total_products = 0
     for product in products:
+        total_products += 1
         title = product.title or ""
         description = product.description or ""
         bag, labels = stem_bag(title, description, stopwords=stopwords)
@@ -230,9 +249,16 @@ def suggest_clusters(
             stem_titles[stem].append(title)
 
     candidates = []
+    shop_vocab_filtered = 0
     for stem, ids in stem_products.items():
         if len(ids) < min_products:
             continue
+        # Drop stems that describe the whole shop, not a category subset.
+        if total_products > 0 and max_share < 1.0:
+            share = len(ids) / total_products
+            if share > max_share:
+                shop_vocab_filtered += 1
+                continue
         # Best-vote original word becomes the candidate label.
         label_word = (
             stem_labels[stem].most_common(1)[0][0] if stem_labels[stem] else stem
@@ -254,4 +280,4 @@ def suggest_clusters(
         key=lambda c: (len(c["product_ids"]), c["score"]),
         reverse=True,
     )
-    return candidates[:top_n]
+    return candidates[:top_n], shop_vocab_filtered

make_post_sell/scripts/backfill_tags.py

@@ -34,8 +34,13 @@ def main(argv=None):
         help="Minimum products sharing a stem to surface (default: 2)",
     )
     parser.add_argument(
-        "--top-n", type=int, default=20,
+        "--top-n", type=int, default=50,
-        help="Max candidate clusters to print (default: 20)",
+        help="Max candidate clusters to print (default: 50)",
+    )
+    parser.add_argument(
+        "--max-share", type=float, default=0.4,
+        help="Drop stems in more than this share of products as shop "
+             "vocabulary; pass 1.0 to disable (default: 0.4)",
     )
     parser.add_argument(
         "--apply", action="store_true",
@@ -63,14 +68,20 @@ def main(argv=None):
         print(f"Mode:               {'APPLY' if args.apply else 'DRY-RUN (preview)'}")
         print("-" * 60)
 
-        suggestions = suggest_clusters(
+        suggestions, vocab_filtered = suggest_clusters(
             products,
             stopwords=shop.tag_stopwords,
             existing_tag_slugs=existing_slugs,
             min_products=args.min_products,
             top_n=args.top_n,
+            max_share=args.max_share,
         )
 
+        if vocab_filtered:
+            print(
+                f"({vocab_filtered} stem(s) dropped as shop vocabulary — "
+                f"appeared in >{int(args.max_share * 100)}% of products)"
+            )
         if not suggestions:
             print("No candidate categories surfaced.")
             return 0

make_post_sell/static/css/common.css

@@ -4926,6 +4926,40 @@ textarea {
     padding: 0 10px;
 }
 
+.analytics-header {
+    display: grid;
+    grid-template-columns: 1fr auto;
+    gap: var(--space-3, 12px);
+    align-items: center;
+    margin-bottom: var(--space-3, 12px);
+}
+
+.analytics-header h3 {
+    margin: 0;
+}
+
+.analytics-range-form {
+    display: grid;
+    grid-template-columns: auto auto auto;
+    gap: var(--space-2, 8px);
+    align-items: center;
+    margin: 0;
+}
+
+.analytics-range-form label {
+    font-size: var(--font-size-sm, 0.875rem);
+    color: var(--text-muted, #666);
+}
+
+.analytics-range-form select {
+    padding: var(--space-1, 4px) var(--space-2, 8px);
+    border-radius: var(--radius-sm, 4px);
+    border: 1px solid var(--border-color, #dee2e6);
+    background: var(--surface-color, #fff);
+    color: var(--text-color, #222);
+    font-size: var(--font-size-sm, 0.875rem);
+}
+
 .analytics-overview {
     display: grid;
     grid-template-columns: repeat(2, 1fr);

make_post_sell/templates/shop_tags.j2

@@ -35,6 +35,15 @@
   </a>
 
   {% if show_suggestions %}
+    {% if shop_vocab_filtered and shop_vocab_filtered > 0 %}
+    <p class="type-body-sm tag-suggest-filtered-note">
+      Auto-dropped <b>{{ shop_vocab_filtered }}</b> common word{% if shop_vocab_filtered != 1 %}s{% endif %} as shop vocabulary
+      (appeared in too many products to be a useful category).
+      Tune with <code>?show_suggestions=1&amp;max_share=0.3</code> (lower = stricter)
+      or <code>&amp;max_share=1</code> to disable.
+      Show more with <code>&amp;top_n=200</code>.
+    </p>
+    {% endif %}
     {% if suggestions %}
     <h3 class="type-title tag-suggest-heading">Candidate categories ({{ suggestions|length }})</h3>
     <ul class="tag-suggest-list">

make_post_sell/tests/test_models.py

@@ -5138,11 +5138,15 @@ class TestTagSuggestPureFunctions(unittest.TestCase):
             SimpleNamespace(id="e", title="Stone Fox Novel Study", description="Reading novel."),
             SimpleNamespace(id="f", title="Chocolate Touch Novel Study", description="Reading novel."),
         ]
-        clusters = suggest_clusters(
+        # max_share=1.0 disables the shop-vocabulary filter (covered by its
+        # own dedicated test below) so this fixture isn't penalised for
+        # being tiny — "math" naturally appears in 3 of 6 sample products.
+        clusters, _ = suggest_clusters(
             products,
             stopwords=["write", "room", "activity", "day"],
             existing_tag_slugs=[],
             min_products=2,
+            max_share=1.0,
         )
         labels = [c["label"] for c in clusters]
         self.assertIn("Math", labels)
@@ -5159,8 +5163,8 @@ class TestTagSuggestPureFunctions(unittest.TestCase):
             SimpleNamespace(id="a", title="Math Worksheet One", description=""),
             SimpleNamespace(id="b", title="Math Worksheet Two", description=""),
         ]
-        clusters = suggest_clusters(
+        clusters, _ = suggest_clusters(
-            products, stopwords=[], existing_tag_slugs=["math"]
+            products, stopwords=[], existing_tag_slugs=["math"], max_share=1.0
         )
         labels = [c["label"] for c in clusters]
         self.assertNotIn("Math", labels)
@@ -5174,13 +5178,10 @@ class TestTagSuggestPureFunctions(unittest.TestCase):
             SimpleNamespace(id="b", title="Valentine's Day Reading", description=""),
             SimpleNamespace(id="c", title="Valentine's Day Crafts", description=""),
         ]
-        clusters = suggest_clusters(
+        clusters, _ = suggest_clusters(
-            products, stopwords=["day"], existing_tag_slugs=[]
+            products, stopwords=["day"], existing_tag_slugs=[], max_share=1.0,
         )
-        # The label should be the most common original word, not the stem.
         labels = [c["label"] for c in clusters]
-        # "Valentine" comes back as the readable label — the apostrophe is
-        # stripped at the markdown-cleaning step.
         self.assertTrue(
             any("alentin" in l.lower() for l in labels),
             f"expected valentine-ish label in {labels}",
@@ -5196,8 +5197,9 @@ class TestTagSuggestPureFunctions(unittest.TestCase):
             SimpleNamespace(id="d", title="Delta Reading", description=""),
             SimpleNamespace(id="e", title="Epsilon Reading", description=""),
         ]
-        clusters = suggest_clusters(
+        clusters, _ = suggest_clusters(
-            products, stopwords=[], existing_tag_slugs=[], min_products=2
+            products, stopwords=[], existing_tag_slugs=[],
+            min_products=2, max_share=1.0,
         )
         # "Math" (3 products) ranks above "Reading" (2 products).
         self.assertEqual(clusters[0]["label"], "Math")
@@ -5210,10 +5212,67 @@ class TestTagSuggestPureFunctions(unittest.TestCase):
             SimpleNamespace(id="a", title="Unique Title One", description=""),
             SimpleNamespace(id="b", title="Different Title Two", description=""),
         ]
-        clusters = suggest_clusters(
+        clusters, _ = suggest_clusters(
-            products, stopwords=["title"], existing_tag_slugs=[], min_products=2
+            products, stopwords=["title"], existing_tag_slugs=[],
+            min_products=2, max_share=1.0,
         )
-        # No stem appears in both products → no clusters.
         self.assertEqual(clusters, [])
 
+    def test_suggest_clusters_filters_shop_vocabulary(self):
+        """Stems present in too many products auto-drop as shop vocabulary.
+
+        Mirrors the printableprompts gotcha: 'students', 'resource',
+        'activity' appear in 53%/32%/32% of products — they describe
+        the entire shop, not subsets, and confuse shoppers.
+        """
+        from types import SimpleNamespace
+        from ..lib.tag_suggest import suggest_clusters
+        # 10 products: "resource" in 6/10 (60%); "math" in 3/10 (30%);
+        # "reading" in 3/10 (30%).
+        products = [
+            SimpleNamespace(id=f"p{i}", title=t, description="")
+            for i, t in enumerate([
+                "Math Resource Alpha",
+                "Math Resource Beta",
+                "Math Worksheet Gamma",
+                "Reading Resource Delta",
+                "Reading Resource Epsilon",
+                "Reading Resource Zeta",
+                "Eta Resource",
+                "Theta Resource",
+                "Iota Practice",
+                "Kappa Practice",
+            ])
+        ]
+        # max_share=0.4 → "resource" (60%) drops, "math" (30%) and
+        # "reading" (30%) survive.
+        clusters, filtered = suggest_clusters(
+            products,
+            stopwords=[],
+            existing_tag_slugs=[],
+            min_products=2,
+            max_share=0.4,
+        )
+        labels = [c["label"] for c in clusters]
+        self.assertNotIn("Resource", labels)
+        self.assertIn("Math", labels)
+        self.assertIn("Reading", labels)
+        self.assertGreaterEqual(filtered, 1)
+
+    def test_suggest_clusters_max_share_one_disables_filter(self):
+        from types import SimpleNamespace
+        from ..lib.tag_suggest import suggest_clusters
+        products = [
+            SimpleNamespace(id="a", title="Foo Bar", description=""),
+            SimpleNamespace(id="b", title="Foo Bar", description=""),
+        ]
+        # "foo" and "bar" each in 100% of products — only filtered when
+        # max_share < 1.0.
+        clusters, filtered = suggest_clusters(
+            products, stopwords=[], existing_tag_slugs=[],
+            min_products=2, max_share=1.0,
+        )
+        self.assertEqual(filtered, 0)
+        self.assertTrue(len(clusters) > 0)
+
 

make_post_sell/views/shop.py

@@ -2108,17 +2108,51 @@ def shop_tags(request):
     # MPS-24 Phase 2: compute candidate clusters on demand. We always
     # compute (cheap O(N × tokens) over the shop catalog), but the
     # template only renders the well when the operator clicks the
-    # button (?show_suggestions=1).
+    # button (?show_suggestions=1). Power-user knobs `max_share` and
+    # `top_n` accept URL overrides so the operator can tune without
+    # redeploying.
     show_suggestions = (request.params.get("show_suggestions") or "") == "1"
     suggestions = []
+    shop_vocab_filtered = 0
     if show_suggestions:
-        from ..lib.tag_suggest import suggest_clusters
+        from ..lib.tag_suggest import (
+            DEFAULT_MAX_SHARE,
+            DEFAULT_TOP_N,
+            suggest_clusters,
+        )
+
+        def _float_param(name, default, lo=0.01, hi=1.0):
+            raw = (request.params.get(name) or "").strip()
+            if not raw:
+                return default
+            try:
+                v = float(raw)
+                if v > 1.0:
+                    v = v / 100.0  # accept "40" for 0.4
+            except (TypeError, ValueError):
+                return default
+            return max(lo, min(hi, v))
+
+        def _int_param(name, default, lo=1, hi=500):
+            raw = (request.params.get(name) or "").strip()
+            if not raw:
+                return default
+            try:
+                v = int(float(raw))
+            except (TypeError, ValueError):
+                return default
+            return max(lo, min(hi, v))
+
+        max_share = _float_param("max_share", DEFAULT_MAX_SHARE)
+        top_n = _int_param("top_n", DEFAULT_TOP_N)
 
         existing_slugs = [t.slug for t in all_tags]
-        suggestions = suggest_clusters(
+        suggestions, shop_vocab_filtered = suggest_clusters(
             list(all_products),
             stopwords=shop.tag_stopwords,
             existing_tag_slugs=existing_slugs,
+            top_n=top_n,
+            max_share=max_share,
         )
 
     return {
@@ -2126,6 +2160,7 @@ def shop_tags(request):
         "focus_tag": focus_tag,
         "all_products": all_products,
         "suggestions": suggestions,
+        "shop_vocab_filtered": shop_vocab_filtered,
         "show_suggestions": show_suggestions,
     }