# MPS Architecture Overview ## System Diagram ``` Internet | ┌────────┴────────┐ │ Caddy (HTTPS) │ │ reverse proxy │ │ :443 → :6001 │ └────────┬────────┘ | ┌────────┴────────┐ │ uWSGI │ │ 2 proc, 8 thr │ │ reload@512MB │ └────┬───────┬────┘ | | ┌──────────────┘ └──────────────┐ | | ┌────────┴────────┐ ┌────────┴────────┐ │ Pyramid / WSGI │ │ Background │ │ Request Cycle │ │ Threads │ │ │ │ │ │ views/ │ │ S3 mirror │ │ models/ │ │ Karaoke vocal │ │ templates/ │ │ isolation │ │ lib/ │ │ │ └───┬────┬────┬───┘ └────────┬────────┘ | | | | ┌─────────┘ | └─────────┐ | | | | | ┌────┴────┐ ┌─────┴─────┐ ┌────┴────────┐ ┌───────┴───────┐ │ SQLite │ │ DO Spaces │ │ Payment │ │ DO Spaces │ │ DB │ │ (CDN) │ │ Providers │ │ + User S3 │ │ │ │ │ │ │ │ + Mirror S3 │ │ models │ │ media │ │ Stripe │ │ │ │ sessions│ │ thumbs │ │ PayPal │ │ Presigned │ │ signals │ │ assets │ │ Crypto │ │ URLs only │ │ │ │ │ │ Gift Cards │ │ │ └─────────┘ └────────────┘ └──────────────┘ └───────────────┘ ``` ## Request Flow ``` Browser GET /s/{shop_id}/{slug} │ ├─ Pyramid route dispatch │ └─ views/content.py or views/product.py │ ├─ Query product + shop from SQLite │ ├─ Compute discovery ring related products │ ├─ Generate presigned URLs for media (15 min TTL) │ └─ Render Jinja2 template │ ├─ Template layers: │ ├─ base.j2 (theme, nav, conditional sandbox toolbar) │ ├─ product.j2 / content.j2 (media, metadata, CTA) │ ├─ snippets/related_content.j2 (ring sidebar) │ ├─ snippets/comments.j2 (comment form + list) │ └─ snippets/analytics.j2 (optional Plausible/GA) │ ├─ Client JS (progressive enhancement): │ ├─ signals.js → anonymous beacon on unload │ ├─ watch.js → SPA navigation (if watch_mode_enabled) │ ├─ sandbox.js → filter toolbar (if sandbox_mode) │ └─ comments.js → AJAX comment submission │ └─ Media served from CDN via presigned URLs (never through uwsgi) ``` ## Data Collection Pipeline ``` Page visit (browser) │ │ signals.js collects: │ - presence (wall_clock, visible, active ms) │ - scroll (depth, direction changes) │ - media (play, pause, seek, speed, completion) │ - viewport width │ ▼ navigator.sendBeacon("/signals/beacon") │ ~300 bytes JSON, one per page visit │ ▼ views/signals.py │ ├─ classify_referrer(Referer header) │ → (class, domain, query) tuple │ Stores: referrer_class, referrer_domain, referrer_query │ ├─ classify_device(viewport_width) │ → 0=mobile, 1=tablet, 2=desktop │ ├─ Insert mps_page_session row │ └─ If visible_ms >= 7000: increment product.view_count ``` ## Analytics Pipeline ``` mps_page_session (raw rows) │ ├─ views/analytics.py │ │ │ ├─ Daily bucketing functions (28-day windows): │ │ _daily_buckets() → view counts (bar chart) │ │ _daily_avg_duration() → avg session duration │ │ _daily_engagement() → engagement ratio │ │ _daily_bounce_rate() → bounce rate │ │ _daily_referrer_counts()→ external referrer volume │ │ │ ├─ Ranked queries: │ │ _top_referrer_domains() → top external domains │ │ _top_referrer_queries() → top search engine queries │ │ _top_search_keywords() → top internal search terms │ │ │ └─ Aggregate queries: │ Overview strip (7d) │ Top products by views (7/14/21d) │ Ring entry points │ Engagement/attention leaders │ Study material / background favorites │ Traffic sources / device split │ ├─ templates/analytics.j2 (shop-level dashboard) │ └─ SVG line_chart macro (server-rendered polyline) │ └─ templates/analytics_product.j2 (per-product dashboard) └─ SVG line_chart macro ``` ## S3 Storage Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ MPS Main Bucket │ │ (DigitalOcean Spaces + CDN) │ │ │ │ {shop_id}/products/{product_id}/{filename} │ │ {shop_id}/products/{product_id}/thumb/{filename} │ │ {shop_id}/shop/{logo|banner} │ │ {shop_id}/products/{product_id}/karaoke/{filename} │ └───────────┬─────────────────────────────────┬────────────┘ │ │ Presigned URLs Mirror sync (15 min TTL) (daemon thread) │ │ ▼ ▼ Browser / CDN ┌────────────────────┐ │ Shop Mirror Bucket │ │ (shop.has_s3_mirror)│ │ │ │ Same key structure │ │ Passive copy │ │ Backfill on setup │ └─────────────────────┘ ┌─────────────────────────────────────────────────────────┐ │ Shop Primary Bucket (BYOB) │ │ (shop.has_primary_s3 — MPS-16) │ │ │ │ When enabled, REPLACES MPS Main Bucket for this shop: │ │ - All presigned URLs use shop's S3 client │ │ - All CDN URLs use shop's cdn_endpoint │ │ - request.shop_uploads_client / shop_bucket_name / │ │ shop_cdn_endpoint fall back to MPS default when off │ │ │ │ Configured via bucket-settings form section │ └─────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────┐ │ User Artifact Bucket │ │ (user.has_s3_bucket) │ │ │ │ sandbox/{user_id}/{timestamp}-{filename} │ │ │ │ Presigned POST from /u/sandbox/upload │ │ Browser uploads directly (never through MPS server) │ │ 50MB max per file │ └─────────────────────────────────────────────────────────┘ ``` ## Feature Toggle Matrix | Feature | Model Column | Form Section | Default | |---------|-------------|--------------|---------| | Watch mode | `shop.watch_mode_enabled` | `ribbon-settings` | Off | | Sandbox mode | `shop.sandbox_mode` | `ribbon-settings` | Off | | Show price history (MPS-24 Phase 2.5) | `shop.show_price_history` | `ribbon-settings` | Off | | Show dates | `shop.show_dates` | `ribbon-settings` | On | | Grid lanes | `shop.grid_lanes_enabled` | `ribbon-settings` | Off | | Color filter | `shop.color_filter` | `ribbon-settings` | 0 (none) | | Comments | `shop.comments_enabled` | `comment-settings` | On | | Stripe | `shop.stripe_enabled` | `stripe-settings` | On | | PayPal | `shop.paypal_*` | `paypal-settings` | Off | | Crypto | `shop.monero_*` / `shop.dogecoin_*` | `crypto-settings` | Off | | Gift cards | `shop.gift_card_enabled` | `gift-card-settings` | Off | | S3 mirror | `shop.mirror_s3_*` | `mirror-settings` | Off | | BYOB (primary S3) | `shop.primary_s3_*` | `bucket-settings` | Off | | Environment | `shop.environment` | `environment-settings` | 0 (production) | | Trial | `shop.trial_started_timestamp` | Auto on creation | 21 days | | Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed | | Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off | | Karaoke (vocal isolation) | `shop.unsandbox_*_key` | `unsandbox-settings` | Off | | Make-an-offer | `shop.offer_enabled` + `product.allow_offers` | `offer-settings` | Off | | Offer expiration (hours) | `shop.offer_expiration_hours` | `offer-settings` | 48 | | Offer pay window post-accept (hours) | `shop.offer_acceptance_payment_hours` | `offer-settings` | 24 | | In-app notifications | (always on, no toggle — every txn email also drops a row) | n/a | On | | Home page layout (MPS-24) | `shop.home_layout` (0=flat / 1=chips / 2=lanes) | `home-layout-settings` | 0 (flat) | | Tag chip / lane caps (MPS-24) | `shop.home_layout_tag_limit` / `shop.home_layout_per_lane_limit` | `home-layout-settings` | 8 / 10 | | Featured products strip (MPS-24) | `shop.featured_product_ids_json` | `home-layout-settings` | empty | | Product tags (MPS-24) | `Tag` + `ProductTag` association | product edit + `/s/{id}/tags` bulk editor | — | | Tag auto-suggest (MPS-24 Phase 2) | `lib/tag_suggest.py` over `Product.title` + `Product.description` | `?show_suggestions=1` on `/s/{id}/tags` + `scripts/backfill_tags.py` | Never auto-applies | | Tag-detail facet sidebar (MPS-24 Phase 2.6) | `shop_tag.j2` + `views/shop.py` `_price_range_from_request` / `_filter_by_price_range` | Sort + price range + categories list, GET form, sidebar ≥800px, top chip strip <800px | Always on for tag detail pages | | 6-sentence SERP excerpt (MPS-24 Phase 2.6) | `Product.excerpt_sentences(n=6, max_chars=1500)` via shared `_strip_markdown()` helper | `shop_tag.j2` row description | Always on | | Facet nav on shop home (MPS-24 Phase 2.6b) | `_facet_nav.j2` macros + `home.j2` / `shop.j2` wrappers | Desktop sidebar + mobile `
` accordion when `shop.home_layout >= 1` | Opt-in via home_layout | | Mobile SERP rows under each lane (MPS-24 Phase 2.6b) | `home.j2` / `shop.j2` `.tag-lane-rows` markup + CSS visibility swap | Horizontal tiles ≥800px; vertical SERP rows with 6-sentence excerpts <800px | On for layout 2 | | Per-product SPA tag chips (MPS-24 Phase 2.7) | `product_tags` view (`/p/{id}/tags`) + `static/js/product_tags.js` + `views/__init__.py:is_ajax()` | Chip add/remove on product edit, AJAX persist, no full page reload; no-JS keeps the comma `tags` field on the main form | Always on (JS-enhanced; no-JS fallback) | | Bulk tagger AJAX focus + DnD (MPS-24 Phase 2.8) | `shop.py:shop_tags` AJAX `?focus=` branch + `tag_bulk.js` `wireFocusLinks`/`wireDragAndDrop` | Tag chip click swaps product list in place (no reload); HTML5 drag-to-reorder POSTs `set_order`; bare GET no longer loads the whole catalog | Always on (JS-enhanced; no-JS = real navigation) | ## Ticket Index | Ticket | Title | Status | |--------|-------|--------| | [MPS-0](tickets/mps-0.md) | AJAX Comment Submission | Complete | | [MPS-1](tickets/mps-1.md) | YouTube-Style Watch Experience | Complete | | [MPS-2](tickets/mps-2.md) | Anonymous Signal Gathering & View Count | Complete | | [MPS-3](tickets/mps-3.md) | Creator Analytics Dashboard | Complete | | [MPS-4](tickets/mps-4.md) | Eliminate 502s from uWSGI Worker Recycling | Complete | | [MPS-5](tickets/mps-5.md) | Investigate uWSGI Worker Memory Growth | Open | | [MPS-6](tickets/mps-6.md) | Referrer Analytics — Domain, Query, Trend Lines | Complete | | [MPS-7](tickets/mps-7.md) | Sandbox Mode — Creative Filter System | Complete | | [MPS-8](tickets/mps-8.md) | User S3 Bucket + Artifact Storage | Complete | | [MPS-9](tickets/mps-9.md) | Shop S3 Mirror Bucket | Complete | | [MPS-10](tickets/mps-10.md) | Gift Card — Models & Migration | Complete | | [MPS-11](tickets/mps-11.md) | Gift Card — Purchase Flow | Complete | | [MPS-12](tickets/mps-12.md) | Gift Card — Redemption at Checkout | Complete | | [MPS-13](tickets/mps-13.md) | Gift Card — Shop Admin & Settings | Complete | | [MPS-14](tickets/mps-14.md) | Shop Environment — Dev & Stage Shops | Complete | | [MPS-15](tickets/mps-15.md) | 21-Day Free Trial | Complete | | [MPS-16](tickets/mps-16.md) | Bring Your Own Bucket (BYOB) | Complete | | [MPS-17](tickets/mps-17.md) | REST API v1 — HMAC-signed product/content + upload | Open | | [MPS-18](tickets/mps-18.md) | Karaoke Mode — Diagnose & Fix Vocal Isolation | Open (Broken in prod) | | [MPS-19](tickets/mps-19.md) | BitTorrent / Magnet Link — Diagnose & Fix Distribution | Open (Broken in prod) | | [MPS-20](tickets/mps-20.md) | Auction House Mode (eBay-style Bidding) | Complete | | [MPS-21](tickets/mps-21.md) | Make-an-Offer Mode | Complete | | [MPS-22](tickets/mps-22.md) | Kill-Switch Feature Flags — Karaoke + Torrent Off by Default | Complete | | [MPS-23](tickets/mps-23.md) | Consolidated Transactional Sender Identity + Shop Contact Email | Open | | [MPS-24](tickets/mps-24.md) | Shop home page overhaul + product categorization (tags + chips + lanes + auto-suggest + per-product SPA tag chips + bulk-tagger AJAX/DnD) | In progress (Phases 1 + 2 + 2.6 + 2.7 + 2.8 landed) | ## Related Docs | Doc | Purpose | |-----|---------| | [Design System](design-system.md) | Design tokens, CSS architecture, component library | | [Notifications](notifications.md) | In-app notification table, kinds, and per-transition wiring | | [Auction House](auction-house.md) | MPS-20 — bidding, soft-close, payment deadline | | [Make Offer](make-offer.md) | MPS-21 — offer state machine, counter rounds, expiry | | [JavaScript](JAVASCRIPT.md) | Client-side JS architecture | | [Karaoke Pipeline](karaoke-pipeline.md) | Vocal isolation: pipeline, on-demand, streaming architecture | | [Sandbox Mode](sandbox-mode.md) | Creative filter system | | [Auction House](auction-house.md) | MPS-20: state machine, bidding logic, soft-close, proxy | | [Make-an-Offer](make-offer.md) | MPS-21: state machine, counter rounds, auto-accept/decline | | [Testing Performance](testing-performance.md) | Test suite optimization |