engineering/make_post_sell
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions
make_post_sell/docs/architecture.md
russell@unturf.com 155f7ff66f
fix: MPS-24 Phase 2.8 — bulk tagger AJAX tag-focus + real drag-to-reorder 
Operator (printableprompts.com, 481 products) reported the bulk tagger
'still refreshing the whole screen' and 'dragging tags doesn't work'
after 2.7. Two real defects the 2.7 static audit missed:

1. Tag-focus was a full-page navigation: clicking a tag chip is
   <a href=?focus=slug>, and the view loaded+rendered ALL products on
   EVERY GET. On a 481-product catalog every tag click reloaded a
   multi-MB page. The forms were AJAX; the dominant workflow was not.
2. Drag-to-reorder never existed: shop_tags.j2 shipped draggable=true +
   a handle + help text, but tag_bulk.js had ZERO drag handlers.

Fix:
- shop.py:shop_tags — all_products loads only when focus_tag or
  show_suggestions (bare GET is light). New AJAX branch: is_ajax +
  ?focus=slug -> JSON {focus, products:[{id,title,url,attached}]}.
- shop_tags.j2 — stable [data-focus-section] (always in DOM, hidden
  until focused); ?focus= chips carry data-tag-focus-link. No-JS
  unchanged (real navigation, server renders the section).
- tag_bulk.js — wireFocusLinks() intercepts chip clicks, fetchFocus()
  + renderFocus() swap the list in place, active-chip + history
  pushState/popstate, real-navigation fallback. wireDragAndDrop()
  HTML5 DnD -> persistOrder() POSTs action=set_order&tag_slugs=…
  (view already supported it) + re-syncs up/down disabled states.
  .tag-list-dragging CSS added.
- Tests: TestProductTagsSpa +4 (ajax focus json, unknown-slug null,
  set_order persists positions, bare GET no catalog). 1128 passed.

Docs: mps-24.md Phase 2.8, architecture.md, design-system.md, CLAUDE.md.
Deferred: AJAX 'Suggest categories' link (occasional click, not hot path).
2026-05-16 08:31:03 -04:00

270 lines
16 KiB
Markdown
Raw Blame History

# MPS Architecture Overview
## System Diagram
```
Internet
|
┌────────┴────────┐
│ Caddy (HTTPS) │
│ reverse proxy │
│ :443 → :6001 │
└────────┬────────┘
|
┌────────┴────────┐
│ uWSGI │
│ 2 proc, 8 thr │
│ reload@512MB │
└────┬───────┬────┘
| |
┌──────────────┘ └──────────────┐
| |
┌────────┴────────┐ ┌────────┴────────┐
│ Pyramid / WSGI │ │ Background │
│ Request Cycle │ │ Threads │
│ │ │ │
│ views/ │ │ S3 mirror │
│ models/ │ │ Karaoke vocal │
│ templates/ │ │ isolation │
│ lib/ │ │ │
└───┬────┬────┬───┘ └────────┬────────┘
| | | |
┌─────────┘ | └─────────┐ |
| | | |
┌────┴────┐ ┌─────┴─────┐ ┌────┴────────┐ ┌───────┴───────┐
│ SQLite │ │ DO Spaces │ │ Payment │ │ DO Spaces │
│ DB │ │ (CDN) │ │ Providers │ │ + User S3 │
│ │ │ │ │ │ │ + Mirror S3 │
│ models │ │ media │ │ Stripe │ │ │
│ sessions│ │ thumbs │ │ PayPal │ │ Presigned │
│ signals │ │ assets │ │ Crypto │ │ URLs only │
│ │ │ │ │ Gift Cards │ │ │
└─────────┘ └────────────┘ └──────────────┘ └───────────────┘
```
## Request Flow
```
Browser GET /s/{shop_id}/{slug}
│
├─ Pyramid route dispatch
│ └─ views/content.py or views/product.py
│ ├─ Query product + shop from SQLite
│ ├─ Compute discovery ring related products
│ ├─ Generate presigned URLs for media (15 min TTL)
│ └─ Render Jinja2 template
│
├─ Template layers:
│ ├─ base.j2 (theme, nav, conditional sandbox toolbar)
│ ├─ product.j2 / content.j2 (media, metadata, CTA)
│ ├─ snippets/related_content.j2 (ring sidebar)
│ ├─ snippets/comments.j2 (comment form + list)
│ └─ snippets/analytics.j2 (optional Plausible/GA)
│
├─ Client JS (progressive enhancement):
│ ├─ signals.js → anonymous beacon on unload
│ ├─ watch.js → SPA navigation (if watch_mode_enabled)
│ ├─ sandbox.js → filter toolbar (if sandbox_mode)
│ └─ comments.js → AJAX comment submission
│
└─ Media served from CDN via presigned URLs (never through uwsgi)
```
## Data Collection Pipeline
```
Page visit (browser)
│
│ signals.js collects:
│ - presence (wall_clock, visible, active ms)
│ - scroll (depth, direction changes)
│ - media (play, pause, seek, speed, completion)
│ - viewport width
│
▼
navigator.sendBeacon("/signals/beacon")
│ ~300 bytes JSON, one per page visit
│
▼
views/signals.py
│
├─ classify_referrer(Referer header)
│ → (class, domain, query) tuple
│ Stores: referrer_class, referrer_domain, referrer_query
│
├─ classify_device(viewport_width)
│ → 0=mobile, 1=tablet, 2=desktop
│
├─ Insert mps_page_session row
│
└─ If visible_ms >= 7000:
increment product.view_count
```
## Analytics Pipeline
```
mps_page_session (raw rows)
│
├─ views/analytics.py
│ │
│ ├─ Daily bucketing functions (28-day windows):
│ │ _daily_buckets() → view counts (bar chart)
│ │ _daily_avg_duration() → avg session duration
│ │ _daily_engagement() → engagement ratio
│ │ _daily_bounce_rate() → bounce rate
│ │ _daily_referrer_counts()→ external referrer volume
│ │
│ ├─ Ranked queries:
│ │ _top_referrer_domains() → top external domains
│ │ _top_referrer_queries() → top search engine queries
│ │ _top_search_keywords() → top internal search terms
│ │
│ └─ Aggregate queries:
│ Overview strip (7d)
│ Top products by views (7/14/21d)
│ Ring entry points
│ Engagement/attention leaders
│ Study material / background favorites
│ Traffic sources / device split
│
├─ templates/analytics.j2 (shop-level dashboard)
│ └─ SVG line_chart macro (server-rendered polyline)
│
└─ templates/analytics_product.j2 (per-product dashboard)
└─ SVG line_chart macro
```
## S3 Storage Architecture
```
┌─────────────────────────────────────────────────────────┐
│ MPS Main Bucket │
│ (DigitalOcean Spaces + CDN) │
│ │
│ {shop_id}/products/{product_id}/{filename} │
│ {shop_id}/products/{product_id}/thumb/{filename} │
│ {shop_id}/shop/{logo|banner} │
│ {shop_id}/products/{product_id}/karaoke/{filename} │
└───────────┬─────────────────────────────────┬────────────┘
│ │
Presigned URLs Mirror sync
(15 min TTL) (daemon thread)
│ │
▼ ▼
Browser / CDN ┌────────────────────┐
│ Shop Mirror Bucket │
│ (shop.has_s3_mirror)│
│ │
│ Same key structure │
│ Passive copy │
│ Backfill on setup │
└─────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Shop Primary Bucket (BYOB) │
│ (shop.has_primary_s3 — MPS-16) │
│ │
│ When enabled, REPLACES MPS Main Bucket for this shop: │
│ - All presigned URLs use shop's S3 client │
│ - All CDN URLs use shop's cdn_endpoint │
│ - request.shop_uploads_client / shop_bucket_name / │
│ shop_cdn_endpoint fall back to MPS default when off │
│ │
│ Configured via bucket-settings form section │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ User Artifact Bucket │
│ (user.has_s3_bucket) │
│ │
│ sandbox/{user_id}/{timestamp}-{filename} │
│ │
│ Presigned POST from /u/sandbox/upload │
│ Browser uploads directly (never through MPS server) │
│ 50MB max per file │
└─────────────────────────────────────────────────────────┘
```
## Feature Toggle Matrix
| Feature | Model Column | Form Section | Default |
|---------|-------------|--------------|---------|
| Watch mode | `shop.watch_mode_enabled` | `ribbon-settings` | Off |
| Sandbox mode | `shop.sandbox_mode` | `ribbon-settings` | Off |
| Show price history (MPS-24 Phase 2.5) | `shop.show_price_history` | `ribbon-settings` | Off |
| Show dates | `shop.show_dates` | `ribbon-settings` | On |
| Grid lanes | `shop.grid_lanes_enabled` | `ribbon-settings` | Off |
| Color filter | `shop.color_filter` | `ribbon-settings` | 0 (none) |
| Comments | `shop.comments_enabled` | `comment-settings` | On |
| Stripe | `shop.stripe_enabled` | `stripe-settings` | On |
| PayPal | `shop.paypal_*` | `paypal-settings` | Off |
| Crypto | `shop.monero_*` / `shop.dogecoin_*` | `crypto-settings` | Off |
| Gift cards | `shop.gift_card_enabled` | `gift-card-settings` | Off |
| S3 mirror | `shop.mirror_s3_*` | `mirror-settings` | Off |
| BYOB (primary S3) | `shop.primary_s3_*` | `bucket-settings` | Off |
| Environment | `shop.environment` | `environment-settings` | 0 (production) |
| Trial | `shop.trial_started_timestamp` | Auto on creation | 21 days |
| Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed |
| Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off |
| Karaoke (vocal isolation) | `shop.unsandbox_*_key` | `unsandbox-settings` | Off |
| Make-an-offer | `shop.offer_enabled` + `product.allow_offers` | `offer-settings` | Off |
| Offer expiration (hours) | `shop.offer_expiration_hours` | `offer-settings` | 48 |
| Offer pay window post-accept (hours) | `shop.offer_acceptance_payment_hours` | `offer-settings` | 24 |
| In-app notifications | (always on, no toggle — every txn email also drops a row) | n/a | On |
| Home page layout (MPS-24) | `shop.home_layout` (0=flat / 1=chips / 2=lanes) | `home-layout-settings` | 0 (flat) |
| Tag chip / lane caps (MPS-24) | `shop.home_layout_tag_limit` / `shop.home_layout_per_lane_limit` | `home-layout-settings` | 8 / 10 |
| Featured products strip (MPS-24) | `shop.featured_product_ids_json` | `home-layout-settings` | empty |
| Product tags (MPS-24) | `Tag` + `ProductTag` association | product edit + `/s/{id}/tags` bulk editor | — |
| Tag auto-suggest (MPS-24 Phase 2) | `lib/tag_suggest.py` over `Product.title` + `Product.description` | `?show_suggestions=1` on `/s/{id}/tags` + `scripts/backfill_tags.py` | Never auto-applies |
| Tag-detail facet sidebar (MPS-24 Phase 2.6) | `shop_tag.j2` + `views/shop.py` `_price_range_from_request` / `_filter_by_price_range` | Sort + price range + categories list, GET form, sidebar ≥800px, top chip strip <800px | Always on for tag detail pages |
| 6-sentence SERP excerpt (MPS-24 Phase 2.6) | `Product.excerpt_sentences(n=6, max_chars=1500)` via shared `_strip_markdown()` helper | `shop_tag.j2` row description | Always on |
| Facet nav on shop home (MPS-24 Phase 2.6b) | `_facet_nav.j2` macros + `home.j2` / `shop.j2` wrappers | Desktop sidebar + mobile `<details>` accordion when `shop.home_layout >= 1` | Opt-in via home_layout |
| Mobile SERP rows under each lane (MPS-24 Phase 2.6b) | `home.j2` / `shop.j2` `.tag-lane-rows` markup + CSS visibility swap | Horizontal tiles ≥800px; vertical SERP rows with 6-sentence excerpts <800px | On for layout 2 |
| Per-product SPA tag chips (MPS-24 Phase 2.7) | `product_tags` view (`/p/{id}/tags`) + `static/js/product_tags.js` + `views/__init__.py:is_ajax()` | Chip add/remove on product edit, AJAX persist, no full page reload; no-JS keeps the comma `tags` field on the main form | Always on (JS-enhanced; no-JS fallback) |
| Bulk tagger AJAX focus + DnD (MPS-24 Phase 2.8) | `shop.py:shop_tags` AJAX `?focus=` branch + `tag_bulk.js` `wireFocusLinks`/`wireDragAndDrop` | Tag chip click swaps product list in place (no reload); HTML5 drag-to-reorder POSTs `set_order`; bare GET no longer loads the whole catalog | Always on (JS-enhanced; no-JS = real navigation) |
## Ticket Index
| Ticket | Title | Status |
|--------|-------|--------|
| [MPS-0](tickets/mps-0.md) | AJAX Comment Submission | Complete |
| [MPS-1](tickets/mps-1.md) | YouTube-Style Watch Experience | Complete |
| [MPS-2](tickets/mps-2.md) | Anonymous Signal Gathering & View Count | Complete |
| [MPS-3](tickets/mps-3.md) | Creator Analytics Dashboard | Complete |
| [MPS-4](tickets/mps-4.md) | Eliminate 502s from uWSGI Worker Recycling | Complete |
| [MPS-5](tickets/mps-5.md) | Investigate uWSGI Worker Memory Growth | Open |
| [MPS-6](tickets/mps-6.md) | Referrer Analytics — Domain, Query, Trend Lines | Complete |
| [MPS-7](tickets/mps-7.md) | Sandbox Mode — Creative Filter System | Complete |
| [MPS-8](tickets/mps-8.md) | User S3 Bucket + Artifact Storage | Complete |
| [MPS-9](tickets/mps-9.md) | Shop S3 Mirror Bucket | Complete |
| [MPS-10](tickets/mps-10.md) | Gift Card — Models & Migration | Complete |
| [MPS-11](tickets/mps-11.md) | Gift Card — Purchase Flow | Complete |
| [MPS-12](tickets/mps-12.md) | Gift Card — Redemption at Checkout | Complete |
| [MPS-13](tickets/mps-13.md) | Gift Card — Shop Admin & Settings | Complete |
| [MPS-14](tickets/mps-14.md) | Shop Environment — Dev & Stage Shops | Complete |
| [MPS-15](tickets/mps-15.md) | 21-Day Free Trial | Complete |
| [MPS-16](tickets/mps-16.md) | Bring Your Own Bucket (BYOB) | Complete |
| [MPS-17](tickets/mps-17.md) | REST API v1 — HMAC-signed product/content + upload | Open |
| [MPS-18](tickets/mps-18.md) | Karaoke Mode — Diagnose & Fix Vocal Isolation | Open (Broken in prod) |
| [MPS-19](tickets/mps-19.md) | BitTorrent / Magnet Link — Diagnose & Fix Distribution | Open (Broken in prod) |
| [MPS-20](tickets/mps-20.md) | Auction House Mode (eBay-style Bidding) | Complete |
| [MPS-21](tickets/mps-21.md) | Make-an-Offer Mode | Complete |
| [MPS-22](tickets/mps-22.md) | Kill-Switch Feature Flags — Karaoke + Torrent Off by Default | Complete |
| [MPS-23](tickets/mps-23.md) | Consolidated Transactional Sender Identity + Shop Contact Email | Open |
| [MPS-24](tickets/mps-24.md) | Shop home page overhaul + product categorization (tags + chips + lanes + auto-suggest + per-product SPA tag chips + bulk-tagger AJAX/DnD) | In progress (Phases 1 + 2 + 2.6 + 2.7 + 2.8 landed) |
## Related Docs
| Doc | Purpose |
|-----|---------|
| [Design System](design-system.md) | Design tokens, CSS architecture, component library |
| [Notifications](notifications.md) | In-app notification table, kinds, and per-transition wiring |
| [Auction House](auction-house.md) | MPS-20 — bidding, soft-close, payment deadline |
| [Make Offer](make-offer.md) | MPS-21 — offer state machine, counter rounds, expiry |
| [JavaScript](JAVASCRIPT.md) | Client-side JS architecture |
| [Karaoke Pipeline](karaoke-pipeline.md) | Vocal isolation: pipeline, on-demand, streaming architecture |
| [Sandbox Mode](sandbox-mode.md) | Creative filter system |
| [Auction House](auction-house.md) | MPS-20: state machine, bidding logic, soft-close, proxy |
| [Make-an-Offer](make-offer.md) | MPS-21: state machine, counter rounds, auto-accept/decline |
| [Testing Performance](testing-performance.md) | Test suite optimization |
Reference in a new issue View git blame Copy permalink