diff --git a/docs/JAVASCRIPT.md b/docs/JAVASCRIPT.md index 45edb69..9e11751 100644 --- a/docs/JAVASCRIPT.md +++ b/docs/JAVASCRIPT.md @@ -37,6 +37,26 @@ Located in `make_post_sell/static/js/`: - Autoplay toggle persisted in localStorage - Ring state (position, direction, history) persisted in localStorage +### sandbox.js +- Client-side creative filter engine (IIFE, ~700 lines) +- Used by: `base.j2` when shop has `sandbox_mode` enabled +- 32 CSS filter presets across 7 categories (basic, warm, cool, dramatic, color, Instagram-style, SVG) +- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia, grayscale) +- Image export via canvas `toBlob()` (PNG) +- Video export via `MediaRecorder` + `captureStream(30fps)` (WebM) +- On-demand face detection via MediaPipe Face Mesh (468 landmarks, eye glow, face mask overlays) +- Upload-to-bucket: presigned POST to user's S3 bucket via `/u/sandbox/upload` +- localStorage persistence: preset, slider values, panel state, filter mode +- Exposes `window.sandboxReapply()` for watch.js SPA integration + +### signals.js +- Anonymous page visit signal collector +- Used by: all product/content pages +- Collects presence (wall_clock, visible, active), scroll (depth, direction changes), and media playback signals +- Sends single JSON beacon (~300 bytes) via `navigator.sendBeacon()` on page unload +- No cookies, no IPs, no fingerprints — purely behavioral signals +- Viewport width sent for device class derivation (server-side) + ### comments.js - AJAX comment submission (progressive enhancement) - Intercepts comment form POST, submits via `fetch()` with `X-Requested-With: XMLHttpRequest` @@ -138,6 +158,7 @@ Located in `make_post_sell/static/js/`: | Stripe | `js.stripe.com/v3/` | Card payment form | | Plausible | Shop-configured domain | Privacy analytics | | Google Analytics | `googletagmanager.com` | Google analytics | +| MediaPipe | `cdn.jsdelivr.net` | Face detection (sandbox mode, on-demand) | ## CSRF Protection diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..dae60bd --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,205 @@ +# MPS Architecture Overview + +## System Diagram + +``` + Internet + | + ┌────────┴────────┐ + │ Caddy (HTTPS) │ + │ reverse proxy │ + │ :443 → :6001 │ + └────────┬────────┘ + | + ┌────────┴────────┐ + │ uWSGI │ + │ 2 proc, 8 thr │ + │ reload@512MB │ + └────┬───────┬────┘ + | | + ┌──────────────┘ └──────────────┐ + | | + ┌────────┴────────┐ ┌────────┴────────┐ + │ Pyramid / WSGI │ │ Background │ + │ Request Cycle │ │ Threads │ + │ │ │ │ + │ views/ │ │ S3 mirror │ + │ models/ │ │ Karaoke vocal │ + │ templates/ │ │ isolation │ + │ lib/ │ │ │ + └───┬────┬────┬───┘ └────────┬────────┘ + | | | | + ┌─────────┘ | └─────────┐ | + | | | | +┌────┴────┐ ┌─────┴─────┐ ┌────┴────────┐ ┌───────┴───────┐ +│ SQLite │ │ DO Spaces │ │ Payment │ │ DO Spaces │ +│ DB │ │ (CDN) │ │ Providers │ │ + User S3 │ +│ │ │ │ │ │ │ + Mirror S3 │ +│ models │ │ media │ │ Stripe │ │ │ +│ sessions│ │ thumbs │ │ PayPal │ │ Presigned │ +│ signals │ │ assets │ │ Crypto │ │ URLs only │ +└─────────┘ └────────────┘ └──────────────┘ └───────────────┘ +``` + +## Request Flow + +``` +Browser GET /s/{shop_id}/{slug} + │ + ├─ Pyramid route dispatch + │ └─ views/content.py or views/product.py + │ ├─ Query product + shop from SQLite + │ ├─ Compute discovery ring related products + │ ├─ Generate presigned URLs for media (15 min TTL) + │ └─ Render Jinja2 template + │ + ├─ Template layers: + │ ├─ base.j2 (theme, nav, conditional sandbox toolbar) + │ ├─ product.j2 / content.j2 (media, metadata, CTA) + │ ├─ snippets/related_content.j2 (ring sidebar) + │ ├─ snippets/comments.j2 (comment form + list) + │ └─ snippets/analytics.j2 (optional Plausible/GA) + │ + ├─ Client JS (progressive enhancement): + │ ├─ signals.js → anonymous beacon on unload + │ ├─ watch.js → SPA navigation (if watch_mode_enabled) + │ ├─ sandbox.js → filter toolbar (if sandbox_mode) + │ └─ comments.js → AJAX comment submission + │ + └─ Media served from CDN via presigned URLs (never through uwsgi) +``` + +## Data Collection Pipeline + +``` +Page visit (browser) + │ + │ signals.js collects: + │ - presence (wall_clock, visible, active ms) + │ - scroll (depth, direction changes) + │ - media (play, pause, seek, speed, completion) + │ - viewport width + │ + ▼ +navigator.sendBeacon("/signals/beacon") + │ ~300 bytes JSON, one per page visit + │ + ▼ +views/signals.py + │ + ├─ classify_referrer(Referer header) + │ → (class, domain, query) tuple + │ Stores: referrer_class, referrer_domain, referrer_query + │ + ├─ classify_device(viewport_width) + │ → 0=mobile, 1=tablet, 2=desktop + │ + ├─ Insert mps_page_session row + │ + └─ If visible_ms >= 7000: + increment product.view_count +``` + +## Analytics Pipeline + +``` +mps_page_session (raw rows) + │ + ├─ views/analytics.py + │ │ + │ ├─ Daily bucketing functions (28-day windows): + │ │ _daily_buckets() → view counts (bar chart) + │ │ _daily_avg_duration() → avg session duration + │ │ _daily_engagement() → engagement ratio + │ │ _daily_bounce_rate() → bounce rate + │ │ _daily_referrer_counts()→ external referrer volume + │ │ + │ ├─ Ranked queries: + │ │ _top_referrer_domains() → top external domains + │ │ _top_referrer_queries() → top search engine queries + │ │ _top_search_keywords() → top internal search terms + │ │ + │ └─ Aggregate queries: + │ Overview strip (7d) + │ Top products by views (7/14/21d) + │ Ring entry points + │ Engagement/attention leaders + │ Study material / background favorites + │ Traffic sources / device split + │ + ├─ templates/analytics.j2 (shop-level dashboard) + │ └─ SVG line_chart macro (server-rendered polyline) + │ + └─ templates/analytics_product.j2 (per-product dashboard) + └─ SVG line_chart macro +``` + +## S3 Storage Architecture + +``` +┌─────────────────────────────────────────────────────────┐ +│ MPS Main Bucket │ +│ (DigitalOcean Spaces + CDN) │ +│ │ +│ {shop_id}/products/{product_id}/{filename} │ +│ {shop_id}/products/{product_id}/thumb/{filename} │ +│ {shop_id}/shop/{logo|banner} │ +│ {shop_id}/products/{product_id}/karaoke/{filename} │ +└───────────┬─────────────────────────────────┬────────────┘ + │ │ + Presigned URLs Mirror sync + (15 min TTL) (daemon thread) + │ │ + ▼ ▼ + Browser / CDN ┌────────────────────┐ + │ Shop Mirror Bucket │ + │ (shop.has_s3_mirror)│ + │ │ + │ Same key structure │ + │ Passive copy │ + │ Backfill on setup │ + └─────────────────────┘ + +┌─────────────────────────────────────────────────────────┐ +│ User Artifact Bucket │ +│ (user.has_s3_bucket) │ +│ │ +│ sandbox/{user_id}/{timestamp}-{filename} │ +│ │ +│ Presigned POST from /u/sandbox/upload │ +│ Browser uploads directly (never through MPS server) │ +│ 50MB max per file │ +└─────────────────────────────────────────────────────────┘ +``` + +## Feature Toggle Matrix + +| Feature | Model Column | Form Section | Default | +|---------|-------------|--------------|---------| +| Watch mode | `shop.watch_mode_enabled` | `ribbon-settings` | Off | +| Sandbox mode | `shop.sandbox_mode` | `ribbon-settings` | Off | +| Show dates | `shop.show_dates` | `ribbon-settings` | On | +| Grid lanes | `shop.grid_lanes_enabled` | `ribbon-settings` | Off | +| Color filter | `shop.color_filter` | `ribbon-settings` | 0 (none) | +| Comments | `shop.comments_enabled` | `comment-settings` | On | +| Stripe | `shop.stripe_enabled` | `stripe-settings` | On | +| PayPal | `shop.paypal_*` | `paypal-settings` | Off | +| Crypto | `shop.monero_*` / `shop.dogecoin_*` | `crypto-settings` | Off | +| S3 mirror | `shop.mirror_s3_*` | `mirror-settings` | Off | +| Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed | +| Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off | + +## Ticket Index + +| Ticket | Title | Status | +|--------|-------|--------| +| [MPS-0](tickets/mps-0.md) | AJAX Comment Submission | Complete | +| [MPS-1](tickets/mps-1.md) | YouTube-Style Watch Experience | Complete | +| [MPS-2](tickets/mps-2.md) | Anonymous Signal Gathering & View Count | Complete | +| [MPS-3](tickets/mps-3.md) | Creator Analytics Dashboard | Complete | +| [MPS-4](tickets/mps-4.md) | Eliminate 502s from uWSGI Worker Recycling | Complete | +| [MPS-5](tickets/mps-5.md) | Investigate uWSGI Worker Memory Growth | Open | +| [MPS-6](tickets/mps-6.md) | Referrer Analytics — Domain, Query, Trend Lines | Complete | +| [MPS-7](tickets/mps-7.md) | Sandbox Mode — Creative Filter System | Complete | +| [MPS-8](tickets/mps-8.md) | User S3 Bucket + Artifact Storage | Complete | +| [MPS-9](tickets/mps-9.md) | Shop S3 Mirror Bucket | Complete | diff --git a/docs/sandbox-mode.md b/docs/sandbox-mode.md index 93462cb..90b3ed2 100644 --- a/docs/sandbox-mode.md +++ b/docs/sandbox-mode.md @@ -93,7 +93,7 @@ img.product-main │ ├──> Download (a.download = "sandbox-export.png") │ - └──> Upload (presigned POST to S3) [future] + └──> Upload (presigned POST to user's S3 bucket) ``` ### Video Export diff --git a/docs/tickets/mps-2.md b/docs/tickets/mps-2.md index a8b82fd..2f997dc 100644 --- a/docs/tickets/mps-2.md +++ b/docs/tickets/mps-2.md @@ -68,6 +68,8 @@ Piggyback on existing `timeupdate`, `play`, `pause`, `ended`, `seeked`, | `is_ring_entry` | Boolean: full page load vs SPA transition | Front door vs ring navigation | | `ring_position` | Integer: how many SPA transitions from entry | Depth in the ring | | `referrer_class` | Enum: `direct`, `search`, `social`, `internal`, `unknown` | Where traffic comes from — derived from Referer header domain, NOT the full URL | +| `referrer_domain` | String (128 chars) | Referer hostname (e.g. "www.google.com") — added in MPS-6 | +| `referrer_query` | String (256 chars) | Search engine query parameter — added in MPS-6 | | `device_class` | Enum: `mobile`, `tablet`, `desktop` | Screen size bucket only, not user agent — derived from viewport width sent in beacon | `referrer_class` is computed server-side by matching the Referer header domain @@ -114,6 +116,8 @@ media_percent_played Float is_ring_entry Boolean ring_position SmallInteger referrer_class SmallInteger (enum: 0=direct 1=search 2=social 3=internal 4=unknown) +referrer_domain Unicode(128) — extracted domain (e.g. "www.google.com") +referrer_query Unicode(256) — search engine query (e.g. "lo-fi beats") device_class SmallInteger (enum: 0=mobile 1=tablet 2=desktop) ``` @@ -283,7 +287,7 @@ from accidental double-fires, not tracking users across pages. - Mouse/touch coordinates - Clicked element identifiers - Any form input content -- Referrer URLs (only domain classification) +- Referrer URLs (only domain + search query parameter — see MPS-6) - Cross-page session linking (each page visit is an island) A viewer could visit every product in a shop and there would be no way to diff --git a/docs/tickets/mps-6.md b/docs/tickets/mps-6.md new file mode 100644 index 0000000..09f6dcf --- /dev/null +++ b/docs/tickets/mps-6.md @@ -0,0 +1,111 @@ +# MPS-6: Referrer Analytics — Domain, Query, and Trend Lines + +## Problem + +MPS-2 stored only a numeric `referrer_class` (0-4) on each page session. This +tells the creator "42% of traffic is from search" but not *which* search engine, +*which* social platform, or *what keywords* people searched. Creators need +actionable referrer intelligence to know where to spend their marketing energy. + +The analytics dashboard also lacked trend visualization — all metrics were +point-in-time tables with no temporal context. + +## Solution + +### 1. Store referrer domain and search query + +Add two columns to `mps_page_session`: + +| Column | Type | Example | +|--------|------|---------| +| `referrer_domain` | `Unicode(128)` | `"www.google.com"`, `"twitter.com"` | +| `referrer_query` | `Unicode(256)` | `"lo-fi beats to study to"` | + +These are populated by refactoring `classify_referrer()` from returning a single +integer to returning a `(class, domain, query)` tuple. The domain is extracted +from the Referer header URL. The query is parsed from search engine URL +parameters (`q=` for Google/DuckDuckGo/Bing, `p=` for Yahoo). + +Privacy: the full Referer URL is never stored. Only the domain and the search +query parameter (if present) are kept. Social and unknown referrers store domain +only. + +### 2. Add SVG line chart trend visualization + +Server-rendered inline SVG polyline charts for temporal trends. No JavaScript +graph libraries — the charts are computed server-side and rendered as a Jinja2 +macro (`line_chart`) that produces `` elements with `` paths. + +Five trend lines added to both shop-level and per-product analytics: + +| Chart | Y-axis | Color | +|-------|--------|-------| +| Session duration | Avg `wall_clock_ms` (daily) | Blue | +| Engagement | Avg `active_ms / wall_clock_ms` (daily) | Green | +| Bounce rate | % sessions with `visible_ms < 7s` (daily) | Red | +| External referrers | Daily count of non-direct, non-internal sessions | Orange | + +Each chart spans 28 days with one data point per day. Zero-days are filled so +the polyline is continuous. + +### 3. Add keyword and referrer domain tables + +Two new analytics sections: + +**Top Referrer Domains** — ranked table of external domains driving traffic, +with session count and percentage bar. Excludes direct and internal traffic. + +**Top Search Queries** — ranked table of search engine queries that led to the +shop, extracted from `referrer_query`. Also includes internal shop search +keywords from `mps_shop_search_request`. + +### 4. Bucketing functions + +Six new query functions in `analytics.py`: + +- `_daily_buckets()` — daily view counts (bar chart) +- `_daily_avg_duration()` — daily avg session duration +- `_daily_engagement()` — daily avg engagement ratio +- `_daily_bounce_rate()` — daily bounce rate +- `_daily_referrer_counts()` — daily external referrer sessions +- `_top_referrer_domains()` — top N referrer domains +- `_top_referrer_queries()` — top N search engine queries +- `_top_search_keywords()` — top N internal search keywords + +## classify_referrer() Refactor + +Before (MPS-2): +```python +def classify_referrer(referrer, request_host): + """Return 0-4 integer class.""" + return 1 # search +``` + +After (MPS-6): +```python +def classify_referrer(referrer, request_host): + """Return (class, domain, query) tuple.""" + return (1, "www.google.com", "lo-fi beats") +``` + +Search engine query extraction: +- Google/Bing/DuckDuckGo: `?q=` parameter +- Yahoo: `?p=` parameter +- Other search engines: `?q=` fallback + +## Files Changed + +| File | Change | +|------|--------| +| `models/page_session.py` | Add `referrer_domain`, `referrer_query` columns | +| `views/signals.py` | Refactor `classify_referrer()` to return tuple; store domain + query | +| `views/analytics.py` | 8 new bucketing/query functions; pass trend data to templates | +| `templates/analytics.j2` | SVG `line_chart` macro; referrer domain table; keyword table | +| `templates/analytics_product.j2` | SVG `line_chart` macro; referrer domain table | +| `scripts/alembic/versions/f3086e09b052_*.py` | Migration: add `referrer_domain`, `referrer_query` | +| `tests/test_models.py` | 12 tests for `classify_referrer()` tuple return | +| `tests/test_functional.py` | Referrer trend chart rendering tests | + +## Depends On + +MPS-3 (analytics dashboard infrastructure) diff --git a/docs/tickets/mps-7.md b/docs/tickets/mps-7.md new file mode 100644 index 0000000..1922bd0 --- /dev/null +++ b/docs/tickets/mps-7.md @@ -0,0 +1,63 @@ +# MPS-7: Sandbox Mode — Client-Side Creative Filter System + +## Problem + +Creators want visitors to interact with their media beyond passive viewing. +Existing tools require downloading, editing in external software, and +re-uploading. The friction kills experimentation. Creators need an in-browser +creative toolkit that lets visitors play with filters, export artifacts, and +optionally upload results — all without leaving the shop. + +## Solution + +A client-side filter engine that runs entirely in the browser. The server stores +a single boolean (`shop.sandbox_mode`). When enabled, every page in the shop +renders a floating toolbar with 32 filter presets, 7 adjustment sliders, SVG +filter effects, canvas export (image + video), and optional face detection via +MediaPipe. + +Full architecture and filter reference: [docs/sandbox-mode.md](../sandbox-mode.md) + +### Server-side (minimal) + +- `sandbox_mode` Boolean column on Shop model +- Toggle in ribbon-settings form section (same form as announcement ribbon, + default theme, grid lanes, show_dates, watch_mode, color_filter) +- `base.j2` conditionally renders toolbar HTML, inline SVG filter definitions, + and loads `sandbox.js` only when `shop.sandbox_mode` is true +- Watch mode SPA integration: `watch.js` calls `window.sandboxReapply()` after + DOM swap to re-apply filters to newly loaded media + +### Client-side (sandbox.js, ~700 lines) + +- 32 CSS filter presets (4 basic, 4 warm, 4 cool, 4 dramatic, 5 color, 7 + Instagram-style, 4 SVG) +- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia, + grayscale) +- Image export via canvas `toBlob()` (requires CORS from CDN) +- Video export via `MediaRecorder` + `captureStream(30fps)` +- Face detection via MediaPipe Face Mesh (4MB WASM, on-demand load) +- localStorage persistence of preset, slider values, panel state +- Upload-to-bucket via presigned POST (see MPS-8) + +## Files Changed + +| File | Change | +|------|--------| +| `models/shop.py` | `sandbox_mode` Boolean column | +| `views/shop.py` | Toggle handler in `ribbon-settings` form section | +| `templates/shop_settings.j2` | Enable/disable radio buttons | +| `templates/base.j2` | Conditional toolbar HTML + inline SVG defs | +| `static/js/sandbox.js` | Client-side filter engine (IIFE) | +| `static/js/watch.js` | `sandboxReapply()` hook in `updatePageContent()` | +| `static/css/common.css` | Toolbar layout (CSS Grid), mobile responsive | +| `scripts/alembic/versions/a2c13d3117f2_*.py` | Migration: `sandbox_mode` | +| `tests/test_functional.py` | Sandbox toggle, toolbar rendering, script tag tests | + +## Depends On + +Nothing. Independent feature. + +## Blocks + +MPS-8 (user S3 bucket for artifact upload) diff --git a/docs/tickets/mps-8.md b/docs/tickets/mps-8.md new file mode 100644 index 0000000..e39d24f --- /dev/null +++ b/docs/tickets/mps-8.md @@ -0,0 +1,92 @@ +# MPS-8: User S3 Bucket + Artifact Storage + +## Problem + +Sandbox mode (MPS-7) lets visitors export filtered images and videos, but +exports download to the user's local device. Creators want a way to +automatically upload sandbox artifacts to their own cloud storage — a personal +S3 bucket where filtered content accumulates without manual file management. + +## Solution + +### User-level S3 credentials + +Add S3-compatible storage credentials to the User model (not Shop — a user may +own multiple shops but uses one storage bucket): + +| Column | Type | Example | +|--------|------|---------| +| `s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` | +| `s3_region` | `Unicode(64)` | `"nyc3"` | +| `s3_bucket` | `Unicode(128)` | `"my-sandbox-exports"` | +| `s3_access_key` | `Unicode(128)` | `"DO00..."` | +| `s3_secret_key` | `Unicode(128)` | `"wJalr..."` | + +Property `has_s3_bucket` returns True when endpoint, bucket, access_key, and +secret_key are all non-empty. + +### Storage settings form + +New section on user settings page (`/u/settings`) with an "Artifact Storage" +form that POSTs to `/u/settings/storage`. Fields: endpoint URL, region +(optional), bucket name, access key, secret key (password field). + +### Presigned upload endpoint + +`POST /u/sandbox/upload` — accepts `filename` and `content_type` parameters, +generates a presigned POST using the user's stored S3 credentials. Returns +`{presigned: {url, fields}, key}`. The browser uploads the blob directly to the +user's bucket — the file never touches the MPS server. + +S3 key format: `sandbox/{user_id}/{timestamp}-{filename}` + +Upload limit: 50MB per file (enforced via presigned POST conditions). + +### Toolbar integration + +When a user has `has_s3_bucket = True`, the sandbox toolbar in `base.j2` +renders with `data-has-bucket="1"`. The `sandbox.js` upload-to-bucket button +appears only when this attribute is present. The upload flow: + +1. User exports image/video (canvas toBlob) +2. User clicks "Upload to Bucket" +3. JS POSTs to `/u/sandbox/upload` with filename + content_type +4. Server returns presigned POST URL + fields +5. JS uploads blob directly to user's S3 bucket via FormData POST + +### Supported services + +Any S3-compatible endpoint: DigitalOcean Spaces, AWS S3, MinIO, Backblaze B2, +Cloudflare R2. + +### Security + +Credentials stored as plain Unicode columns (same pattern as Stripe/PayPal/Adyen +keys on Shop model). Secret key field uses `type="password"` in the form. +Credentials are only used server-side to generate presigned URLs — they are +never exposed to the browser. + +## Shop-Level S3 Mirror (MPS-9) + +A separate but related feature: shops can configure their own S3 bucket as a +**mirror** of the MPS main bucket. Every file uploaded to a shop (product files, +thumbnails, shop assets) is automatically copied to the shop's mirror bucket in +a background thread. See [MPS-9](mps-9.md). + +## Files Changed + +| File | Change | +|------|--------| +| `models/user.py` | S3 credential columns + `has_s3_bucket` property | +| `views/user.py` | Storage settings POST handler + S3 fields in settings dict | +| `views/user_sandbox.py` | Presigned upload endpoint | +| `templates/user_settings.j2` | Artifact Storage form | +| `templates/base.j2` | `data-has-bucket` attribute on sandbox toolbar | +| `static/js/sandbox.js` | Upload-to-bucket button + presigned POST flow | +| `routes.py` | `user_storage_settings`, `user_sandbox_upload` routes | +| `scripts/alembic/versions/f898ba460612_*.py` | Migration: user S3 columns | +| `tests/test_functional.py` | S3 bucket save/clear, presigned upload, toolbar attribute tests | + +## Depends On + +MPS-7 (sandbox mode — toolbar must exist to add upload button) diff --git a/docs/tickets/mps-9.md b/docs/tickets/mps-9.md new file mode 100644 index 0000000..2ccd16c --- /dev/null +++ b/docs/tickets/mps-9.md @@ -0,0 +1,92 @@ +# MPS-9: Shop S3 Mirror Bucket + +## Problem + +Creators want a copy of all their shop files in their own S3 bucket — for +backup, CDN flexibility, or migration away from MPS. Currently all files live +exclusively in the MPS DigitalOcean Spaces bucket. + +## Solution + +### Shop-level mirror credentials + +Add S3-compatible mirror credentials to the Shop model: + +| Column | Type | Example | +|--------|------|---------| +| `mirror_s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` | +| `mirror_s3_region` | `Unicode(64)` | `"nyc3"` | +| `mirror_s3_bucket` | `Unicode(128)` | `"my-shop-mirror"` | +| `mirror_s3_access_key` | `Unicode(128)` | `"DO00..."` | +| `mirror_s3_secret_key` | `Unicode(128)` | `"wJalr..."` | + +Property `has_s3_mirror` returns True when all required fields are non-empty. + +### Mirror sync engine (`lib/s3_mirror.py`) + +Fire-and-forget sync: every file written to the MPS bucket is copied to the +shop's mirror bucket in a daemon thread. The MPS bucket remains the +origin/CDN — the mirror is a passive copy. + +Key functions: + +- `mirror_key()` — stream-copy a single key (get_object → put_object) +- `mirror_key_async()` — fire-and-forget single key in daemon thread +- `mirror_keys_async()` — fire-and-forget multiple keys in one thread +- `test_mirror_connection()` — validate credentials by listing bucket +- `backfill_mirror_async()` — double-fork detached process that copies all + existing shop files to the mirror (survives uWSGI worker recycling) + +Thread safety: ORM objects are not accessed from background threads. All +credentials are captured as plain strings before thread creation. Each thread +creates its own boto3 client. + +### Sync hooks + +Mirror sync is triggered from: + +- **Product upload** (`views/product.py`) — product file + thumbnail +- **Shop asset upload** (`views/shop.py`) — logo, banner + +### Mirror settings form + +New "Mirror Bucket" section in shop settings (`mirror-settings` form section): + +- Endpoint URL, Region, Bucket, Access Key, Secret Key +- "Test Connection" — validates credentials on save +- "Backfill" toggle — triggers `backfill_mirror_async()` to copy all existing + files on first setup + +### Backfill architecture + +The backfill process needs to survive uWSGI worker recycling (workers get killed +at 512MB RSS). Solution: double-fork to fully detach from uWSGI: + +``` +Request handler + └─ fork() ─── intermediate child + └─ setsid() + fork() ─── grandchild (fully detached) + └─ fcntl.flock() guard + └─ create own SQLAlchemy engine + └─ list_objects_v2 + mirror_key loop + └─ os._exit(0) +``` + +One backfill per shop at a time (flock on `/tmp/s3_mirror_backfill_{shop_id}.lock`). + +## Files Changed + +| File | Change | +|------|--------| +| `models/shop.py` | Mirror S3 credential columns + `has_s3_mirror` property | +| `lib/s3_mirror.py` | Mirror engine: sync, async, test, backfill | +| `views/shop.py` | `mirror-settings` form handler + sync hooks | +| `views/product.py` | Mirror sync hooks on product upload | +| `templates/shop_settings.j2` | Mirror Bucket settings form | +| `routes.py` | No new routes (uses existing shop settings POST) | +| `scripts/alembic/versions/6b516114c393_*.py` | Migration: shop mirror S3 columns | +| `tests/test_functional.py` | Mirror settings save, connection test, backfill | + +## Depends On + +Nothing. Independent feature (but complements MPS-8 user S3 bucket).