engineering/make_post_sell
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions
make_post_sell/docs/architecture.md
russell@unturf.com 8649e6aaae docs: karaoke pipeline architecture with dot diagrams 
Add docs/karaoke-pipeline.md covering the full streaming pipeline from
MPS through unsandbox API to zerotrust container and back. Includes two
Graphviz dot diagrams (rendered to SVG):

- karaoke-pipeline.dot: full system flow across MPS, API, pool, container
- karaoke-ondemand.dot: watch mode on-demand user flow

Update architecture.md feature toggle matrix and related docs table.
Update CLAUDE.md karaoke section with streaming path and on-demand info.
2026-03-11 17:49:57 -04:00

241 lines
13 KiB
Markdown
Raw Blame History

# MPS Architecture Overview
## System Diagram
```
Internet
|
┌────────┴────────┐
│ Caddy (HTTPS) │
│ reverse proxy │
│ :443 → :6001 │
└────────┬────────┘
|
┌────────┴────────┐
│ uWSGI │
│ 2 proc, 8 thr │
│ reload@512MB │
└────┬───────┬────┘
| |
┌──────────────┘ └──────────────┐
| |
┌────────┴────────┐ ┌────────┴────────┐
│ Pyramid / WSGI │ │ Background │
│ Request Cycle │ │ Threads │
│ │ │ │
│ views/ │ │ S3 mirror │
│ models/ │ │ Karaoke vocal │
│ templates/ │ │ isolation │
│ lib/ │ │ │
└───┬────┬────┬───┘ └────────┬────────┘
| | | |
┌─────────┘ | └─────────┐ |
| | | |
┌────┴────┐ ┌─────┴─────┐ ┌────┴────────┐ ┌───────┴───────┐
│ SQLite │ │ DO Spaces │ │ Payment │ │ DO Spaces │
│ DB │ │ (CDN) │ │ Providers │ │ + User S3 │
│ │ │ │ │ │ │ + Mirror S3 │
│ models │ │ media │ │ Stripe │ │ │
│ sessions│ │ thumbs │ │ PayPal │ │ Presigned │
│ signals │ │ assets │ │ Crypto │ │ URLs only │
│ │ │ │ │ Gift Cards │ │ │
└─────────┘ └────────────┘ └──────────────┘ └───────────────┘
```
## Request Flow
```
Browser GET /s/{shop_id}/{slug}
│
├─ Pyramid route dispatch
│ └─ views/content.py or views/product.py
│ ├─ Query product + shop from SQLite
│ ├─ Compute discovery ring related products
│ ├─ Generate presigned URLs for media (15 min TTL)
│ └─ Render Jinja2 template
│
├─ Template layers:
│ ├─ base.j2 (theme, nav, conditional sandbox toolbar)
│ ├─ product.j2 / content.j2 (media, metadata, CTA)
│ ├─ snippets/related_content.j2 (ring sidebar)
│ ├─ snippets/comments.j2 (comment form + list)
│ └─ snippets/analytics.j2 (optional Plausible/GA)
│
├─ Client JS (progressive enhancement):
│ ├─ signals.js → anonymous beacon on unload
│ ├─ watch.js → SPA navigation (if watch_mode_enabled)
│ ├─ sandbox.js → filter toolbar (if sandbox_mode)
│ └─ comments.js → AJAX comment submission
│
└─ Media served from CDN via presigned URLs (never through uwsgi)
```
## Data Collection Pipeline
```
Page visit (browser)
│
│ signals.js collects:
│ - presence (wall_clock, visible, active ms)
│ - scroll (depth, direction changes)
│ - media (play, pause, seek, speed, completion)
│ - viewport width
│
▼
navigator.sendBeacon("/signals/beacon")
│ ~300 bytes JSON, one per page visit
│
▼
views/signals.py
│
├─ classify_referrer(Referer header)
│ → (class, domain, query) tuple
│ Stores: referrer_class, referrer_domain, referrer_query
│
├─ classify_device(viewport_width)
│ → 0=mobile, 1=tablet, 2=desktop
│
├─ Insert mps_page_session row
│
└─ If visible_ms >= 7000:
increment product.view_count
```
## Analytics Pipeline
```
mps_page_session (raw rows)
│
├─ views/analytics.py
│ │
│ ├─ Daily bucketing functions (28-day windows):
│ │ _daily_buckets() → view counts (bar chart)
│ │ _daily_avg_duration() → avg session duration
│ │ _daily_engagement() → engagement ratio
│ │ _daily_bounce_rate() → bounce rate
│ │ _daily_referrer_counts()→ external referrer volume
│ │
│ ├─ Ranked queries:
│ │ _top_referrer_domains() → top external domains
│ │ _top_referrer_queries() → top search engine queries
│ │ _top_search_keywords() → top internal search terms
│ │
│ └─ Aggregate queries:
│ Overview strip (7d)
│ Top products by views (7/14/21d)
│ Ring entry points
│ Engagement/attention leaders
│ Study material / background favorites
│ Traffic sources / device split
│
├─ templates/analytics.j2 (shop-level dashboard)
│ └─ SVG line_chart macro (server-rendered polyline)
│
└─ templates/analytics_product.j2 (per-product dashboard)
└─ SVG line_chart macro
```
## S3 Storage Architecture
```
┌─────────────────────────────────────────────────────────┐
│ MPS Main Bucket │
│ (DigitalOcean Spaces + CDN) │
│ │
│ {shop_id}/products/{product_id}/{filename} │
│ {shop_id}/products/{product_id}/thumb/{filename} │
│ {shop_id}/shop/{logo|banner} │
│ {shop_id}/products/{product_id}/karaoke/{filename} │
└───────────┬─────────────────────────────────┬────────────┘
│ │
Presigned URLs Mirror sync
(15 min TTL) (daemon thread)
│ │
▼ ▼
Browser / CDN ┌────────────────────┐
│ Shop Mirror Bucket │
│ (shop.has_s3_mirror)│
│ │
│ Same key structure │
│ Passive copy │
│ Backfill on setup │
└─────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Shop Primary Bucket (BYOB) │
│ (shop.has_primary_s3 — MPS-16) │
│ │
│ When enabled, REPLACES MPS Main Bucket for this shop: │
│ - All presigned URLs use shop's S3 client │
│ - All CDN URLs use shop's cdn_endpoint │
│ - request.shop_uploads_client / shop_bucket_name / │
│ shop_cdn_endpoint fall back to MPS default when off │
│ │
│ Configured via bucket-settings form section │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ User Artifact Bucket │
│ (user.has_s3_bucket) │
│ │
│ sandbox/{user_id}/{timestamp}-{filename} │
│ │
│ Presigned POST from /u/sandbox/upload │
│ Browser uploads directly (never through MPS server) │
│ 50MB max per file │
└─────────────────────────────────────────────────────────┘
```
## Feature Toggle Matrix
| Feature | Model Column | Form Section | Default |
|---------|-------------|--------------|---------|
| Watch mode | `shop.watch_mode_enabled` | `ribbon-settings` | Off |
| Sandbox mode | `shop.sandbox_mode` | `ribbon-settings` | Off |
| Show dates | `shop.show_dates` | `ribbon-settings` | On |
| Grid lanes | `shop.grid_lanes_enabled` | `ribbon-settings` | Off |
| Color filter | `shop.color_filter` | `ribbon-settings` | 0 (none) |
| Comments | `shop.comments_enabled` | `comment-settings` | On |
| Stripe | `shop.stripe_enabled` | `stripe-settings` | On |
| PayPal | `shop.paypal_*` | `paypal-settings` | Off |
| Crypto | `shop.monero_*` / `shop.dogecoin_*` | `crypto-settings` | Off |
| Gift cards | `shop.gift_card_enabled` | `gift-card-settings` | Off |
| S3 mirror | `shop.mirror_s3_*` | `mirror-settings` | Off |
| BYOB (primary S3) | `shop.primary_s3_*` | `bucket-settings` | Off |
| Environment | `shop.environment` | `environment-settings` | 0 (production) |
| Trial | `shop.trial_started_timestamp` | Auto on creation | 21 days |
| Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed |
| Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off |
| Karaoke (vocal isolation) | `shop.unsandbox_*_key` | `unsandbox-settings` | Off |
## Ticket Index
| Ticket | Title | Status |
|--------|-------|--------|
| [MPS-0](tickets/mps-0.md) | AJAX Comment Submission | Complete |
| [MPS-1](tickets/mps-1.md) | YouTube-Style Watch Experience | Complete |
| [MPS-2](tickets/mps-2.md) | Anonymous Signal Gathering & View Count | Complete |
| [MPS-3](tickets/mps-3.md) | Creator Analytics Dashboard | Complete |
| [MPS-4](tickets/mps-4.md) | Eliminate 502s from uWSGI Worker Recycling | Complete |
| [MPS-5](tickets/mps-5.md) | Investigate uWSGI Worker Memory Growth | Open |
| [MPS-6](tickets/mps-6.md) | Referrer Analytics — Domain, Query, Trend Lines | Complete |
| [MPS-7](tickets/mps-7.md) | Sandbox Mode — Creative Filter System | Complete |
| [MPS-8](tickets/mps-8.md) | User S3 Bucket + Artifact Storage | Complete |
| [MPS-9](tickets/mps-9.md) | Shop S3 Mirror Bucket | Complete |
| [MPS-10](tickets/mps-10.md) | Gift Card — Models & Migration | Complete |
| [MPS-11](tickets/mps-11.md) | Gift Card — Purchase Flow | Complete |
| [MPS-12](tickets/mps-12.md) | Gift Card — Redemption at Checkout | Complete |
| [MPS-13](tickets/mps-13.md) | Gift Card — Shop Admin & Settings | Complete |
| [MPS-14](tickets/mps-14.md) | Shop Environment — Dev & Stage Shops | Complete |
| [MPS-15](tickets/mps-15.md) | 21-Day Free Trial | Complete |
| [MPS-16](tickets/mps-16.md) | Bring Your Own Bucket (BYOB) | Complete |
## Related Docs
| Doc | Purpose |
|-----|---------|
| [Design System](design-system.md) | Design tokens, CSS architecture, component library |
| [JavaScript](JAVASCRIPT.md) | Client-side JS architecture |
| [Karaoke Pipeline](karaoke-pipeline.md) | Vocal isolation: pipeline, on-demand, streaming architecture |
| [Sandbox Mode](sandbox-mode.md) | Creative filter system |
| [Testing Performance](testing-performance.md) | Test suite optimization |
Reference in a new issue View git blame Copy permalink