feat: MPS-24 Phase 2 — auto-suggest tags from title + description

Operator with 481 untagged products (printableprompts.com) gets a
one-click path to a usable categorization without hand-tagging each
product. Strictly suggest-then-approve — nothing writes Tag or
ProductTag rows until the operator clicks Apply on a cluster.

- lib/tag_suggest.py: pure-function clusterer. Tokenize title (weight 3)
  + description (weight 1, capped at 100 unique tokens per product),
  strip markdown / URLs / HTML, English + per-shop stopwords, simple
  suffix-strip stemmer, group by stem, drop stems matching existing
  tag slugs, rank by product count, label each cluster with the most
  frequent original word for its stem. No new deps, no ML.
- scripts/backfill_tags.py: CLI preview + --apply for a single shop.
- views/shop.py: shop_tags gains action=apply_suggestion (creates tag +
  bulk-attaches every product in cluster) and action=dismiss_suggestion
  (adds the cluster's words to shop.tag_stopwords_json so it never
  resurfaces). ?show_suggestions=1 triggers the cluster compute.
- templates/shop_tags.j2: "Suggest categories from titles + descriptions"
  button + suggestions well with per-cluster sample titles, Apply, and
  Dismiss buttons.
- 15 new tests (11 unit over tokenize / stem / cluster + 4 functional
  over the suggest/apply/dismiss flow). 1064 total passing.

On a printableprompts-style sample the clusterer surfaces Math, Reading,
Literacy, Seasonal, Novel, Activities, Comprehension — matching what an
operator would manually pick.

2026-05-15 09:09:40 -04:00

15 KiB

Raw Blame History

MPS Architecture Overview

System Diagram

                                    Internet
                                       |
                              ┌────────┴────────┐
                              │  Caddy (HTTPS)   │
                              │  reverse proxy   │
                              │  :443 → :6001    │
                              └────────┬────────┘
                                       |
                              ┌────────┴────────┐
                              │   uWSGI          │
                              │   2 proc, 8 thr  │
                              │   reload@512MB   │
                              └────┬───────┬────┘
                                   |       |
                    ┌──────────────┘       └──────────────┐
                    |                                      |
           ┌────────┴────────┐                   ┌────────┴────────┐
           │  Pyramid / WSGI  │                   │   Background    │
           │  Request Cycle   │                   │   Threads       │
           │                  │                   │                 │
           │  views/          │                   │  S3 mirror      │
           │  models/         │                   │  Karaoke vocal  │
           │  templates/      │                   │  isolation      │
           │  lib/            │                   │                 │
           └───┬────┬────┬───┘                   └────────┬────────┘
               |    |    |                                 |
     ┌─────────┘    |    └─────────┐                      |
     |              |              |                      |
┌────┴────┐   ┌─────┴─────┐  ┌────┴────────┐    ┌───────┴───────┐
│ SQLite  │   │  DO Spaces │  │  Payment     │    │  DO Spaces    │
│ DB      │   │  (CDN)     │  │  Providers   │    │  + User S3    │
│         │   │            │  │              │    │  + Mirror S3  │
│ models  │   │  media     │  │  Stripe      │    │               │
│ sessions│   │  thumbs    │  │  PayPal      │    │  Presigned    │
│ signals │   │  assets    │  │  Crypto      │    │  URLs only    │
│         │   │            │  │  Gift Cards  │    │               │
└─────────┘   └────────────┘  └──────────────┘    └───────────────┘

Request Flow

Browser GET /s/{shop_id}/{slug}
  │
  ├─ Pyramid route dispatch
  │   └─ views/content.py or views/product.py
  │       ├─ Query product + shop from SQLite
  │       ├─ Compute discovery ring related products
  │       ├─ Generate presigned URLs for media (15 min TTL)
  │       └─ Render Jinja2 template
  │
  ├─ Template layers:
  │   ├─ base.j2 (theme, nav, conditional sandbox toolbar)
  │   ├─ product.j2 / content.j2 (media, metadata, CTA)
  │   ├─ snippets/related_content.j2 (ring sidebar)
  │   ├─ snippets/comments.j2 (comment form + list)
  │   └─ snippets/analytics.j2 (optional Plausible/GA)
  │
  ├─ Client JS (progressive enhancement):
  │   ├─ signals.js → anonymous beacon on unload
  │   ├─ watch.js → SPA navigation (if watch_mode_enabled)
  │   ├─ sandbox.js → filter toolbar (if sandbox_mode)
  │   └─ comments.js → AJAX comment submission
  │
  └─ Media served from CDN via presigned URLs (never through uwsgi)

Data Collection Pipeline

Page visit (browser)
  │
  │ signals.js collects:
  │  - presence (wall_clock, visible, active ms)
  │  - scroll (depth, direction changes)
  │  - media (play, pause, seek, speed, completion)
  │  - viewport width
  │
  ▼
navigator.sendBeacon("/signals/beacon")
  │  ~300 bytes JSON, one per page visit
  │
  ▼
views/signals.py
  │
  ├─ classify_referrer(Referer header)
  │   → (class, domain, query) tuple
  │   Stores: referrer_class, referrer_domain, referrer_query
  │
  ├─ classify_device(viewport_width)
  │   → 0=mobile, 1=tablet, 2=desktop
  │
  ├─ Insert mps_page_session row
  │
  └─ If visible_ms >= 7000:
      increment product.view_count

Analytics Pipeline

mps_page_session (raw rows)
  │
  ├─ views/analytics.py
  │   │
  │   ├─ Daily bucketing functions (28-day windows):
  │   │   _daily_buckets()        → view counts (bar chart)
  │   │   _daily_avg_duration()   → avg session duration
  │   │   _daily_engagement()     → engagement ratio
  │   │   _daily_bounce_rate()    → bounce rate
  │   │   _daily_referrer_counts()→ external referrer volume
  │   │
  │   ├─ Ranked queries:
  │   │   _top_referrer_domains() → top external domains
  │   │   _top_referrer_queries() → top search engine queries
  │   │   _top_search_keywords()  → top internal search terms
  │   │
  │   └─ Aggregate queries:
  │       Overview strip (7d)
  │       Top products by views (7/14/21d)
  │       Ring entry points
  │       Engagement/attention leaders
  │       Study material / background favorites
  │       Traffic sources / device split
  │
  ├─ templates/analytics.j2 (shop-level dashboard)
  │   └─ SVG line_chart macro (server-rendered polyline)
  │
  └─ templates/analytics_product.j2 (per-product dashboard)
      └─ SVG line_chart macro

S3 Storage Architecture

┌─────────────────────────────────────────────────────────┐
│                    MPS Main Bucket                        │
│            (DigitalOcean Spaces + CDN)                   │
│                                                          │
│  {shop_id}/products/{product_id}/{filename}              │
│  {shop_id}/products/{product_id}/thumb/{filename}        │
│  {shop_id}/shop/{logo|banner}                            │
│  {shop_id}/products/{product_id}/karaoke/{filename}      │
└───────────┬─────────────────────────────────┬────────────┘
            │                                 │
     Presigned URLs                    Mirror sync
     (15 min TTL)                   (daemon thread)
            │                                 │
            ▼                                 ▼
    Browser / CDN                 ┌────────────────────┐
                                  │  Shop Mirror Bucket │
                                  │  (shop.has_s3_mirror)│
                                  │                     │
                                  │  Same key structure  │
                                  │  Passive copy        │
                                  │  Backfill on setup   │
                                  └─────────────────────┘

┌─────────────────────────────────────────────────────────┐
│              Shop Primary Bucket (BYOB)                  │
│            (shop.has_primary_s3 — MPS-16)                │
│                                                          │
│  When enabled, REPLACES MPS Main Bucket for this shop:   │
│  - All presigned URLs use shop's S3 client               │
│  - All CDN URLs use shop's cdn_endpoint                  │
│  - request.shop_uploads_client / shop_bucket_name /      │
│    shop_cdn_endpoint fall back to MPS default when off   │
│                                                          │
│  Configured via bucket-settings form section             │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│                  User Artifact Bucket                     │
│               (user.has_s3_bucket)                        │
│                                                          │
│  sandbox/{user_id}/{timestamp}-{filename}                │
│                                                          │
│  Presigned POST from /u/sandbox/upload                   │
│  Browser uploads directly (never through MPS server)     │
│  50MB max per file                                       │
└─────────────────────────────────────────────────────────┘

Feature Toggle Matrix

Feature	Model Column	Form Section	Default
Watch mode	`shop.watch_mode_enabled`	`ribbon-settings`	Off
Sandbox mode	`shop.sandbox_mode`	`ribbon-settings`	Off
Show dates	`shop.show_dates`	`ribbon-settings`	On
Grid lanes	`shop.grid_lanes_enabled`	`ribbon-settings`	Off
Color filter	`shop.color_filter`	`ribbon-settings`	0 (none)
Comments	`shop.comments_enabled`	`comment-settings`	On
Stripe	`shop.stripe_enabled`	`stripe-settings`	On
PayPal	`shop.paypal_*`	`paypal-settings`	Off
Crypto	`shop.monero_` / `shop.dogecoin_`	`crypto-settings`	Off
Gift cards	`shop.gift_card_enabled`	`gift-card-settings`	Off
S3 mirror	`shop.mirror_s3_*`	`mirror-settings`	Off
BYOB (primary S3)	`shop.primary_s3_*`	`bucket-settings`	Off
Environment	`shop.environment`	`environment-settings`	0 (production)
Trial	`shop.trial_started_timestamp`	Auto on creation	21 days
Discovery ring	`shop.discovery_ring`	Automatic	Auto-computed
Subscriptions	`shop.subscription_*`	`ribbon-settings`	Off
Karaoke (vocal isolation)	`shop.unsandbox_*_key`	`unsandbox-settings`	Off
Make-an-offer	`shop.offer_enabled` + `product.allow_offers`	`offer-settings`	Off
Offer expiration (hours)	`shop.offer_expiration_hours`	`offer-settings`	48
Offer pay window post-accept (hours)	`shop.offer_acceptance_payment_hours`	`offer-settings`	24
In-app notifications	(always on, no toggle — every txn email also drops a row)	n/a	On
Home page layout (MPS-24)	`shop.home_layout` (0=flat / 1=chips / 2=lanes)	`home-layout-settings`	0 (flat)
Tag chip / lane caps (MPS-24)	`shop.home_layout_tag_limit` / `shop.home_layout_per_lane_limit`	`home-layout-settings`	8 / 10
Featured products strip (MPS-24)	`shop.featured_product_ids_json`	`home-layout-settings`	empty
Product tags (MPS-24)	`Tag` + `ProductTag` association	product edit + `/s/{id}/tags` bulk editor	—
Tag auto-suggest (MPS-24 Phase 2)	`lib/tag_suggest.py` over `Product.title` + `Product.description`	`?show_suggestions=1` on `/s/{id}/tags` + `scripts/backfill_tags.py`	Never auto-applies

Ticket Index

Ticket	Title	Status
MPS-0	AJAX Comment Submission	Complete
MPS-1	YouTube-Style Watch Experience	Complete
MPS-2	Anonymous Signal Gathering & View Count	Complete
MPS-3	Creator Analytics Dashboard	Complete
MPS-4	Eliminate 502s from uWSGI Worker Recycling	Complete
MPS-5	Investigate uWSGI Worker Memory Growth	Open
MPS-6	Referrer Analytics — Domain, Query, Trend Lines	Complete
MPS-7	Sandbox Mode — Creative Filter System	Complete
MPS-8	User S3 Bucket + Artifact Storage	Complete
MPS-9	Shop S3 Mirror Bucket	Complete
MPS-10	Gift Card — Models & Migration	Complete
MPS-11	Gift Card — Purchase Flow	Complete
MPS-12	Gift Card — Redemption at Checkout	Complete
MPS-13	Gift Card — Shop Admin & Settings	Complete
MPS-14	Shop Environment — Dev & Stage Shops	Complete
MPS-15	21-Day Free Trial	Complete
MPS-16	Bring Your Own Bucket (BYOB)	Complete
MPS-17	REST API v1 — HMAC-signed product/content + upload	Open
MPS-18	Karaoke Mode — Diagnose & Fix Vocal Isolation	Open (Broken in prod)
MPS-19	BitTorrent / Magnet Link — Diagnose & Fix Distribution	Open (Broken in prod)
MPS-20	Auction House Mode (eBay-style Bidding)	Complete
MPS-21	Make-an-Offer Mode	Complete
MPS-22	Kill-Switch Feature Flags — Karaoke + Torrent Off by Default	Complete
MPS-23	Consolidated Transactional Sender Identity + Shop Contact Email	Open
MPS-24	Shop home page overhaul + product categorization (tags + chips + lanes + auto-suggest)	In progress (Phases 1 + 2 landed)

Doc	Purpose
Design System	Design tokens, CSS architecture, component library
Notifications	In-app notification table, kinds, and per-transition wiring
Auction House	MPS-20 — bidding, soft-close, payment deadline
Make Offer	MPS-21 — offer state machine, counter rounds, expiry
JavaScript	Client-side JS architecture
Karaoke Pipeline	Vocal isolation: pipeline, on-demand, streaming architecture
Sandbox Mode	Creative filter system
Auction House	MPS-20: state machine, bidding logic, soft-close, proxy
Make-an-Offer	MPS-21: state machine, counter rounds, auto-accept/decline
Testing Performance	Test suite optimization

15 KiB Raw Blame History

MPS Architecture Overview

System Diagram

Request Flow

Data Collection Pipeline

Analytics Pipeline

S3 Storage Architecture

Feature Toggle Matrix

Ticket Index

Related Docs

15 KiB

Raw Blame History