make_post_sell/docs/tickets/mps-16.md

MPS-16: Bring Your Own Bucket (BYOB) — Primary S3 Per Shop

Summary

Allow shops to use their own S3-compatible bucket as the primary storage for all media (products, thumbnails, previews, karaoke tracks). Files are uploaded directly to the shop's bucket — MPS never stores them on the platform bucket.

Free trial users must bring their own bucket during onboarding (zero storage cost for MPS during trial). Paid plan users can use BYOB or the MPS bucket.

Current Architecture

• All uploads go to MPS DigitalOcean Spaces bucket (global, configured in INI)
• request.secure_uploads_client is a single global boto3 client
• Presigned URLs always use request.app["bucket.secure_uploads.get_endpoint"]
• Mirror S3 (shop.mirror_s3_*) is an async secondary copy

Target Architecture

• Each shop can optionally specify a primary S3 bucket
• If configured, all uploads, presigned URLs, and thumbnail CDN URLs use the shop's bucket
• MPS bucket is never touched for BYOB shops
• Mirror S3 continues to work as a secondary copy (shop can mirror from their primary to another bucket)

Model Changes

Shop model (models/shop.py)

Reuse existing mirror_s3_* columns but add a new flag to indicate primary vs mirror:

primary_s3_enabled = Column(Boolean, default=False)
# When True: mirror_s3_* columns are used as the PRIMARY bucket
# When False: mirror_s3_* columns are used as mirror (current behavior)

Or add separate columns for clarity:

primary_s3_endpoint = Column(Unicode(256), nullable=True)
primary_s3_region = Column(Unicode(64), nullable=True)
primary_s3_bucket = Column(Unicode(128), nullable=True)
primary_s3_access_key = Column(Unicode(128), nullable=True)
primary_s3_secret_key = Column(Unicode(128), nullable=True)
primary_s3_cdn_endpoint = Column(Unicode(256), nullable=True)  # public CDN URL for thumbnails
primary_s3_enabled = Column(Boolean, default=False)

The CDN endpoint is critical — thumbnails and previews use public CDN URLs, not presigned URLs. The shop owner must configure their bucket's CDN endpoint (e.g., https://mybucket.nyc3.cdn.digitaloceanspaces.com).

Properties

@property
def has_primary_s3(self):
    return bool(
        self.primary_s3_enabled
        and self.primary_s3_endpoint
        and self.primary_s3_bucket
        and self.primary_s3_access_key
        and self.primary_s3_secret_key
        and self.primary_s3_cdn_endpoint
    )

@property
def media_cdn_endpoint(self):
    """Return the CDN endpoint for this shop's media."""
    if self.has_primary_s3:
        return self.primary_s3_cdn_endpoint
    return None  # caller falls back to request.app default

Request Method Changes

request_methods.py

Add a shop-aware S3 client factory:

def add_shop_uploads_client(request):
    """Return S3 client for the current shop (BYOB or MPS default)."""
    shop = request.shop
    if shop and shop.has_primary_s3:
        import boto3
        session = boto3.session.Session()
        return session.client(
            "s3",
            region_name=shop.primary_s3_region,
            endpoint_url=shop.primary_s3_endpoint,
            aws_access_key_id=shop.primary_s3_access_key,
            aws_secret_access_key=shop.primary_s3_secret_key,
        )
    return request.secure_uploads_client  # default MPS bucket

Add request.shop_uploads_client as a reified request method.

Bucket name resolution

def get_shop_bucket_name(request):
    shop = request.shop
    if shop and shop.has_primary_s3:
        return shop.primary_s3_bucket
    return request.app["bucket.secure_uploads"]

Add request.shop_bucket_name as a reified request method.

View Changes

views/product.py

All S3 operations must use request.shop_uploads_client and request.shop_bucket_name instead of request.secure_uploads_client and request.app["bucket.secure_uploads"]:

• Presigned GET (downloads, line 45-83): use shop client + shop bucket
• Presigned POST (uploads, line 429-456): use shop client + shop bucket
• Copy object (line 344): use shop client + shop bucket
• Delete object: use shop client + shop bucket

Template changes

All thumbnail/media URLs must resolve through the shop's CDN endpoint:

{# Before #}
{{ request.app["bucket.secure_uploads.get_endpoint"] }}/{{ product.s3_path }}/thumbnail1

{# After #}
{{ product.shop.media_cdn_endpoint or request.app["bucket.secure_uploads.get_endpoint"] }}/{{ product.s3_path }}/thumbnail1

Affected templates:

• product.j2 (lines 13, 43, 75, 97, 112, 131-132)
• cart.j2 (line 124)
• shop.j2 (thumbnail rendering)
• content.j2
• snippets/related_content.j2

views/watch.py

Watch mode JSON endpoint (line 96-99) must use shop CDN endpoint:

cdn = shop.media_cdn_endpoint or request.app["bucket.secure_uploads.get_endpoint"]
thumbnail_url = f"{cdn}/{product.s3_path}/thumbnail1?ts={product.updated_timestamp}"

lib/karaoke.py

Karaoke downloads/uploads must use shop client + shop bucket.

lib/s3_mirror.py

When primary_s3 is enabled, mirror source becomes the shop's bucket (not MPS). Mirror destination is still the mirror_s3_* config.

Settings UI

New form section: bucket-settings

In shop_settings.j2, add a "Storage" or "Media Bucket" section:

• Endpoint URL (text input)
• Region (text input)
• Bucket name (text input)
• Access key (text input)
• Secret key (password input)
• CDN endpoint (text input, with help text: "Public URL for thumbnails")
• Enable checkbox
• Test connection button (reuse test_mirror_connection pattern)

Validation

• Endpoint must start with https://
• All 6 fields required if any provided
• Connection test: list bucket, attempt a test PUT/GET/DELETE cycle
• CDN endpoint must be reachable (optional HEAD request)

Onboarding for Trial Users

During shop creation (/s/new), after the shop is created and the user is redirected to /s/{shop_id}/settings:

• If trial user (no paid plan), show a prominent "Set Up Storage" step
• Guide them through configuring their S3 bucket
• Trial shops cannot upload files until BYOB is configured
• Provide documentation links for DigitalOcean Spaces, AWS S3, Backblaze B2, MinIO

Migration

• Add primary_s3_* columns (6 columns) to mps_shop
• All nullable, primary_s3_enabled server_default="0"
• Idempotent guards

Tests

• Unit: has_primary_s3 property, media_cdn_endpoint property
• Integration: shop with BYOB config, verify client resolution
• Functional: enable BYOB via settings, verify connection test, verify upload uses shop bucket