russell@unturf.com 32a19a339c docs: add MPS-6 through MPS-9 tickets, architecture diagram, update JS docs

- MPS-6: referrer analytics (domain, query, trend line charts)
- MPS-7: sandbox mode creative filter system
- MPS-8: user S3 bucket + artifact storage
- MPS-9: shop S3 mirror bucket
- architecture.md: system diagram, request flow, data pipeline, S3 layout
- JAVASCRIPT.md: add sandbox.js, signals.js, MediaPipe SDK entries
- sandbox-mode.md: mark S3 upload as implemented
- mps-2.md: document referrer_domain + referrer_query columns

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-05 06:19:56 -05:00

3.4 KiB

Raw Blame History

MPS-9: Shop S3 Mirror Bucket

Problem

Creators want a copy of all their shop files in their own S3 bucket — for backup, CDN flexibility, or migration away from MPS. Currently all files live exclusively in the MPS DigitalOcean Spaces bucket.

Solution

Shop-level mirror credentials

Add S3-compatible mirror credentials to the Shop model:

Column	Type	Example
`mirror_s3_endpoint`	`Unicode(256)`	`"https://nyc3.digitaloceanspaces.com"`
`mirror_s3_region`	`Unicode(64)`	`"nyc3"`
`mirror_s3_bucket`	`Unicode(128)`	`"my-shop-mirror"`
`mirror_s3_access_key`	`Unicode(128)`	`"DO00..."`
`mirror_s3_secret_key`	`Unicode(128)`	`"wJalr..."`

Property has_s3_mirror returns True when all required fields are non-empty.

Mirror sync engine (`lib/s3_mirror.py`)

Fire-and-forget sync: every file written to the MPS bucket is copied to the shop's mirror bucket in a daemon thread. The MPS bucket remains the origin/CDN — the mirror is a passive copy.

Key functions:

mirror_key() — stream-copy a single key (get_object → put_object)
mirror_key_async() — fire-and-forget single key in daemon thread
mirror_keys_async() — fire-and-forget multiple keys in one thread
test_mirror_connection() — validate credentials by listing bucket
backfill_mirror_async() — double-fork detached process that copies all existing shop files to the mirror (survives uWSGI worker recycling)

Thread safety: ORM objects are not accessed from background threads. All credentials are captured as plain strings before thread creation. Each thread creates its own boto3 client.

Sync hooks

Mirror sync is triggered from:

Product upload (views/product.py) — product file + thumbnail
Shop asset upload (views/shop.py) — logo, banner

Mirror settings form

New "Mirror Bucket" section in shop settings (mirror-settings form section):

Endpoint URL, Region, Bucket, Access Key, Secret Key
"Test Connection" — validates credentials on save
"Backfill" toggle — triggers backfill_mirror_async() to copy all existing files on first setup

Backfill architecture

The backfill process needs to survive uWSGI worker recycling (workers get killed at 512MB RSS). Solution: double-fork to fully detach from uWSGI:

Request handler
  └─ fork() ─── intermediate child
                  └─ setsid() + fork() ─── grandchild (fully detached)
                       └─ fcntl.flock() guard
                       └─ create own SQLAlchemy engine
                       └─ list_objects_v2 + mirror_key loop
                       └─ os._exit(0)

One backfill per shop at a time (flock on /tmp/s3_mirror_backfill_{shop_id}.lock).

Files Changed

File	Change
`models/shop.py`	Mirror S3 credential columns + `has_s3_mirror` property
`lib/s3_mirror.py`	Mirror engine: sync, async, test, backfill
`views/shop.py`	`mirror-settings` form handler + sync hooks
`views/product.py`	Mirror sync hooks on product upload
`templates/shop_settings.j2`	Mirror Bucket settings form
`routes.py`	No new routes (uses existing shop settings POST)
`scripts/alembic/versions/6b516114c393_*.py`	Migration: shop mirror S3 columns
`tests/test_functional.py`	Mirror settings save, connection test, backfill

Depends On

Nothing. Independent feature (but complements MPS-8 user S3 bucket).

3.4 KiB Raw Blame History