docs: add MPS-6 through MPS-9 tickets, architecture diagram, update JS docs
- MPS-6: referrer analytics (domain, query, trend line charts) - MPS-7: sandbox mode creative filter system - MPS-8: user S3 bucket + artifact storage - MPS-9: shop S3 mirror bucket - architecture.md: system diagram, request flow, data pipeline, S3 layout - JAVASCRIPT.md: add sandbox.js, signals.js, MediaPipe SDK entries - sandbox-mode.md: mark S3 upload as implemented - mps-2.md: document referrer_domain + referrer_query columns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
7a6fadeeaa
commit
32a19a339c
8 changed files with 590 additions and 2 deletions
|
|
@ -37,6 +37,26 @@ Located in `make_post_sell/static/js/`:
|
||||||
- Autoplay toggle persisted in localStorage
|
- Autoplay toggle persisted in localStorage
|
||||||
- Ring state (position, direction, history) persisted in localStorage
|
- Ring state (position, direction, history) persisted in localStorage
|
||||||
|
|
||||||
|
### sandbox.js
|
||||||
|
- Client-side creative filter engine (IIFE, ~700 lines)
|
||||||
|
- Used by: `base.j2` when shop has `sandbox_mode` enabled
|
||||||
|
- 32 CSS filter presets across 7 categories (basic, warm, cool, dramatic, color, Instagram-style, SVG)
|
||||||
|
- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia, grayscale)
|
||||||
|
- Image export via canvas `toBlob()` (PNG)
|
||||||
|
- Video export via `MediaRecorder` + `captureStream(30fps)` (WebM)
|
||||||
|
- On-demand face detection via MediaPipe Face Mesh (468 landmarks, eye glow, face mask overlays)
|
||||||
|
- Upload-to-bucket: presigned POST to user's S3 bucket via `/u/sandbox/upload`
|
||||||
|
- localStorage persistence: preset, slider values, panel state, filter mode
|
||||||
|
- Exposes `window.sandboxReapply()` for watch.js SPA integration
|
||||||
|
|
||||||
|
### signals.js
|
||||||
|
- Anonymous page visit signal collector
|
||||||
|
- Used by: all product/content pages
|
||||||
|
- Collects presence (wall_clock, visible, active), scroll (depth, direction changes), and media playback signals
|
||||||
|
- Sends single JSON beacon (~300 bytes) via `navigator.sendBeacon()` on page unload
|
||||||
|
- No cookies, no IPs, no fingerprints — purely behavioral signals
|
||||||
|
- Viewport width sent for device class derivation (server-side)
|
||||||
|
|
||||||
### comments.js
|
### comments.js
|
||||||
- AJAX comment submission (progressive enhancement)
|
- AJAX comment submission (progressive enhancement)
|
||||||
- Intercepts comment form POST, submits via `fetch()` with `X-Requested-With: XMLHttpRequest`
|
- Intercepts comment form POST, submits via `fetch()` with `X-Requested-With: XMLHttpRequest`
|
||||||
|
|
@ -138,6 +158,7 @@ Located in `make_post_sell/static/js/`:
|
||||||
| Stripe | `js.stripe.com/v3/` | Card payment form |
|
| Stripe | `js.stripe.com/v3/` | Card payment form |
|
||||||
| Plausible | Shop-configured domain | Privacy analytics |
|
| Plausible | Shop-configured domain | Privacy analytics |
|
||||||
| Google Analytics | `googletagmanager.com` | Google analytics |
|
| Google Analytics | `googletagmanager.com` | Google analytics |
|
||||||
|
| MediaPipe | `cdn.jsdelivr.net` | Face detection (sandbox mode, on-demand) |
|
||||||
|
|
||||||
## CSRF Protection
|
## CSRF Protection
|
||||||
|
|
||||||
|
|
|
||||||
205
docs/architecture.md
Normal file
205
docs/architecture.md
Normal file
|
|
@ -0,0 +1,205 @@
|
||||||
|
# MPS Architecture Overview
|
||||||
|
|
||||||
|
## System Diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
Internet
|
||||||
|
|
|
||||||
|
┌────────┴────────┐
|
||||||
|
│ Caddy (HTTPS) │
|
||||||
|
│ reverse proxy │
|
||||||
|
│ :443 → :6001 │
|
||||||
|
└────────┬────────┘
|
||||||
|
|
|
||||||
|
┌────────┴────────┐
|
||||||
|
│ uWSGI │
|
||||||
|
│ 2 proc, 8 thr │
|
||||||
|
│ reload@512MB │
|
||||||
|
└────┬───────┬────┘
|
||||||
|
| |
|
||||||
|
┌──────────────┘ └──────────────┐
|
||||||
|
| |
|
||||||
|
┌────────┴────────┐ ┌────────┴────────┐
|
||||||
|
│ Pyramid / WSGI │ │ Background │
|
||||||
|
│ Request Cycle │ │ Threads │
|
||||||
|
│ │ │ │
|
||||||
|
│ views/ │ │ S3 mirror │
|
||||||
|
│ models/ │ │ Karaoke vocal │
|
||||||
|
│ templates/ │ │ isolation │
|
||||||
|
│ lib/ │ │ │
|
||||||
|
└───┬────┬────┬───┘ └────────┬────────┘
|
||||||
|
| | | |
|
||||||
|
┌─────────┘ | └─────────┐ |
|
||||||
|
| | | |
|
||||||
|
┌────┴────┐ ┌─────┴─────┐ ┌────┴────────┐ ┌───────┴───────┐
|
||||||
|
│ SQLite │ │ DO Spaces │ │ Payment │ │ DO Spaces │
|
||||||
|
│ DB │ │ (CDN) │ │ Providers │ │ + User S3 │
|
||||||
|
│ │ │ │ │ │ │ + Mirror S3 │
|
||||||
|
│ models │ │ media │ │ Stripe │ │ │
|
||||||
|
│ sessions│ │ thumbs │ │ PayPal │ │ Presigned │
|
||||||
|
│ signals │ │ assets │ │ Crypto │ │ URLs only │
|
||||||
|
└─────────┘ └────────────┘ └──────────────┘ └───────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Request Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Browser GET /s/{shop_id}/{slug}
|
||||||
|
│
|
||||||
|
├─ Pyramid route dispatch
|
||||||
|
│ └─ views/content.py or views/product.py
|
||||||
|
│ ├─ Query product + shop from SQLite
|
||||||
|
│ ├─ Compute discovery ring related products
|
||||||
|
│ ├─ Generate presigned URLs for media (15 min TTL)
|
||||||
|
│ └─ Render Jinja2 template
|
||||||
|
│
|
||||||
|
├─ Template layers:
|
||||||
|
│ ├─ base.j2 (theme, nav, conditional sandbox toolbar)
|
||||||
|
│ ├─ product.j2 / content.j2 (media, metadata, CTA)
|
||||||
|
│ ├─ snippets/related_content.j2 (ring sidebar)
|
||||||
|
│ ├─ snippets/comments.j2 (comment form + list)
|
||||||
|
│ └─ snippets/analytics.j2 (optional Plausible/GA)
|
||||||
|
│
|
||||||
|
├─ Client JS (progressive enhancement):
|
||||||
|
│ ├─ signals.js → anonymous beacon on unload
|
||||||
|
│ ├─ watch.js → SPA navigation (if watch_mode_enabled)
|
||||||
|
│ ├─ sandbox.js → filter toolbar (if sandbox_mode)
|
||||||
|
│ └─ comments.js → AJAX comment submission
|
||||||
|
│
|
||||||
|
└─ Media served from CDN via presigned URLs (never through uwsgi)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Collection Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
Page visit (browser)
|
||||||
|
│
|
||||||
|
│ signals.js collects:
|
||||||
|
│ - presence (wall_clock, visible, active ms)
|
||||||
|
│ - scroll (depth, direction changes)
|
||||||
|
│ - media (play, pause, seek, speed, completion)
|
||||||
|
│ - viewport width
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
navigator.sendBeacon("/signals/beacon")
|
||||||
|
│ ~300 bytes JSON, one per page visit
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
views/signals.py
|
||||||
|
│
|
||||||
|
├─ classify_referrer(Referer header)
|
||||||
|
│ → (class, domain, query) tuple
|
||||||
|
│ Stores: referrer_class, referrer_domain, referrer_query
|
||||||
|
│
|
||||||
|
├─ classify_device(viewport_width)
|
||||||
|
│ → 0=mobile, 1=tablet, 2=desktop
|
||||||
|
│
|
||||||
|
├─ Insert mps_page_session row
|
||||||
|
│
|
||||||
|
└─ If visible_ms >= 7000:
|
||||||
|
increment product.view_count
|
||||||
|
```
|
||||||
|
|
||||||
|
## Analytics Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
mps_page_session (raw rows)
|
||||||
|
│
|
||||||
|
├─ views/analytics.py
|
||||||
|
│ │
|
||||||
|
│ ├─ Daily bucketing functions (28-day windows):
|
||||||
|
│ │ _daily_buckets() → view counts (bar chart)
|
||||||
|
│ │ _daily_avg_duration() → avg session duration
|
||||||
|
│ │ _daily_engagement() → engagement ratio
|
||||||
|
│ │ _daily_bounce_rate() → bounce rate
|
||||||
|
│ │ _daily_referrer_counts()→ external referrer volume
|
||||||
|
│ │
|
||||||
|
│ ├─ Ranked queries:
|
||||||
|
│ │ _top_referrer_domains() → top external domains
|
||||||
|
│ │ _top_referrer_queries() → top search engine queries
|
||||||
|
│ │ _top_search_keywords() → top internal search terms
|
||||||
|
│ │
|
||||||
|
│ └─ Aggregate queries:
|
||||||
|
│ Overview strip (7d)
|
||||||
|
│ Top products by views (7/14/21d)
|
||||||
|
│ Ring entry points
|
||||||
|
│ Engagement/attention leaders
|
||||||
|
│ Study material / background favorites
|
||||||
|
│ Traffic sources / device split
|
||||||
|
│
|
||||||
|
├─ templates/analytics.j2 (shop-level dashboard)
|
||||||
|
│ └─ SVG line_chart macro (server-rendered polyline)
|
||||||
|
│
|
||||||
|
└─ templates/analytics_product.j2 (per-product dashboard)
|
||||||
|
└─ SVG line_chart macro
|
||||||
|
```
|
||||||
|
|
||||||
|
## S3 Storage Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ MPS Main Bucket │
|
||||||
|
│ (DigitalOcean Spaces + CDN) │
|
||||||
|
│ │
|
||||||
|
│ {shop_id}/products/{product_id}/{filename} │
|
||||||
|
│ {shop_id}/products/{product_id}/thumb/{filename} │
|
||||||
|
│ {shop_id}/shop/{logo|banner} │
|
||||||
|
│ {shop_id}/products/{product_id}/karaoke/{filename} │
|
||||||
|
└───────────┬─────────────────────────────────┬────────────┘
|
||||||
|
│ │
|
||||||
|
Presigned URLs Mirror sync
|
||||||
|
(15 min TTL) (daemon thread)
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
Browser / CDN ┌────────────────────┐
|
||||||
|
│ Shop Mirror Bucket │
|
||||||
|
│ (shop.has_s3_mirror)│
|
||||||
|
│ │
|
||||||
|
│ Same key structure │
|
||||||
|
│ Passive copy │
|
||||||
|
│ Backfill on setup │
|
||||||
|
└─────────────────────┘
|
||||||
|
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ User Artifact Bucket │
|
||||||
|
│ (user.has_s3_bucket) │
|
||||||
|
│ │
|
||||||
|
│ sandbox/{user_id}/{timestamp}-{filename} │
|
||||||
|
│ │
|
||||||
|
│ Presigned POST from /u/sandbox/upload │
|
||||||
|
│ Browser uploads directly (never through MPS server) │
|
||||||
|
│ 50MB max per file │
|
||||||
|
└─────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Feature Toggle Matrix
|
||||||
|
|
||||||
|
| Feature | Model Column | Form Section | Default |
|
||||||
|
|---------|-------------|--------------|---------|
|
||||||
|
| Watch mode | `shop.watch_mode_enabled` | `ribbon-settings` | Off |
|
||||||
|
| Sandbox mode | `shop.sandbox_mode` | `ribbon-settings` | Off |
|
||||||
|
| Show dates | `shop.show_dates` | `ribbon-settings` | On |
|
||||||
|
| Grid lanes | `shop.grid_lanes_enabled` | `ribbon-settings` | Off |
|
||||||
|
| Color filter | `shop.color_filter` | `ribbon-settings` | 0 (none) |
|
||||||
|
| Comments | `shop.comments_enabled` | `comment-settings` | On |
|
||||||
|
| Stripe | `shop.stripe_enabled` | `stripe-settings` | On |
|
||||||
|
| PayPal | `shop.paypal_*` | `paypal-settings` | Off |
|
||||||
|
| Crypto | `shop.monero_*` / `shop.dogecoin_*` | `crypto-settings` | Off |
|
||||||
|
| S3 mirror | `shop.mirror_s3_*` | `mirror-settings` | Off |
|
||||||
|
| Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed |
|
||||||
|
| Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off |
|
||||||
|
|
||||||
|
## Ticket Index
|
||||||
|
|
||||||
|
| Ticket | Title | Status |
|
||||||
|
|--------|-------|--------|
|
||||||
|
| [MPS-0](tickets/mps-0.md) | AJAX Comment Submission | Complete |
|
||||||
|
| [MPS-1](tickets/mps-1.md) | YouTube-Style Watch Experience | Complete |
|
||||||
|
| [MPS-2](tickets/mps-2.md) | Anonymous Signal Gathering & View Count | Complete |
|
||||||
|
| [MPS-3](tickets/mps-3.md) | Creator Analytics Dashboard | Complete |
|
||||||
|
| [MPS-4](tickets/mps-4.md) | Eliminate 502s from uWSGI Worker Recycling | Complete |
|
||||||
|
| [MPS-5](tickets/mps-5.md) | Investigate uWSGI Worker Memory Growth | Open |
|
||||||
|
| [MPS-6](tickets/mps-6.md) | Referrer Analytics — Domain, Query, Trend Lines | Complete |
|
||||||
|
| [MPS-7](tickets/mps-7.md) | Sandbox Mode — Creative Filter System | Complete |
|
||||||
|
| [MPS-8](tickets/mps-8.md) | User S3 Bucket + Artifact Storage | Complete |
|
||||||
|
| [MPS-9](tickets/mps-9.md) | Shop S3 Mirror Bucket | Complete |
|
||||||
|
|
@ -93,7 +93,7 @@ img.product-main
|
||||||
│
|
│
|
||||||
├──> Download (a.download = "sandbox-export.png")
|
├──> Download (a.download = "sandbox-export.png")
|
||||||
│
|
│
|
||||||
└──> Upload (presigned POST to S3) [future]
|
└──> Upload (presigned POST to user's S3 bucket)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Video Export
|
### Video Export
|
||||||
|
|
|
||||||
|
|
@ -68,6 +68,8 @@ Piggyback on existing `timeupdate`, `play`, `pause`, `ended`, `seeked`,
|
||||||
| `is_ring_entry` | Boolean: full page load vs SPA transition | Front door vs ring navigation |
|
| `is_ring_entry` | Boolean: full page load vs SPA transition | Front door vs ring navigation |
|
||||||
| `ring_position` | Integer: how many SPA transitions from entry | Depth in the ring |
|
| `ring_position` | Integer: how many SPA transitions from entry | Depth in the ring |
|
||||||
| `referrer_class` | Enum: `direct`, `search`, `social`, `internal`, `unknown` | Where traffic comes from — derived from Referer header domain, NOT the full URL |
|
| `referrer_class` | Enum: `direct`, `search`, `social`, `internal`, `unknown` | Where traffic comes from — derived from Referer header domain, NOT the full URL |
|
||||||
|
| `referrer_domain` | String (128 chars) | Referer hostname (e.g. "www.google.com") — added in MPS-6 |
|
||||||
|
| `referrer_query` | String (256 chars) | Search engine query parameter — added in MPS-6 |
|
||||||
| `device_class` | Enum: `mobile`, `tablet`, `desktop` | Screen size bucket only, not user agent — derived from viewport width sent in beacon |
|
| `device_class` | Enum: `mobile`, `tablet`, `desktop` | Screen size bucket only, not user agent — derived from viewport width sent in beacon |
|
||||||
|
|
||||||
`referrer_class` is computed server-side by matching the Referer header domain
|
`referrer_class` is computed server-side by matching the Referer header domain
|
||||||
|
|
@ -114,6 +116,8 @@ media_percent_played Float
|
||||||
is_ring_entry Boolean
|
is_ring_entry Boolean
|
||||||
ring_position SmallInteger
|
ring_position SmallInteger
|
||||||
referrer_class SmallInteger (enum: 0=direct 1=search 2=social 3=internal 4=unknown)
|
referrer_class SmallInteger (enum: 0=direct 1=search 2=social 3=internal 4=unknown)
|
||||||
|
referrer_domain Unicode(128) — extracted domain (e.g. "www.google.com")
|
||||||
|
referrer_query Unicode(256) — search engine query (e.g. "lo-fi beats")
|
||||||
device_class SmallInteger (enum: 0=mobile 1=tablet 2=desktop)
|
device_class SmallInteger (enum: 0=mobile 1=tablet 2=desktop)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -283,7 +287,7 @@ from accidental double-fires, not tracking users across pages.
|
||||||
- Mouse/touch coordinates
|
- Mouse/touch coordinates
|
||||||
- Clicked element identifiers
|
- Clicked element identifiers
|
||||||
- Any form input content
|
- Any form input content
|
||||||
- Referrer URLs (only domain classification)
|
- Referrer URLs (only domain + search query parameter — see MPS-6)
|
||||||
- Cross-page session linking (each page visit is an island)
|
- Cross-page session linking (each page visit is an island)
|
||||||
|
|
||||||
A viewer could visit every product in a shop and there would be no way to
|
A viewer could visit every product in a shop and there would be no way to
|
||||||
|
|
|
||||||
111
docs/tickets/mps-6.md
Normal file
111
docs/tickets/mps-6.md
Normal file
|
|
@ -0,0 +1,111 @@
|
||||||
|
# MPS-6: Referrer Analytics — Domain, Query, and Trend Lines
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
MPS-2 stored only a numeric `referrer_class` (0-4) on each page session. This
|
||||||
|
tells the creator "42% of traffic is from search" but not *which* search engine,
|
||||||
|
*which* social platform, or *what keywords* people searched. Creators need
|
||||||
|
actionable referrer intelligence to know where to spend their marketing energy.
|
||||||
|
|
||||||
|
The analytics dashboard also lacked trend visualization — all metrics were
|
||||||
|
point-in-time tables with no temporal context.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
### 1. Store referrer domain and search query
|
||||||
|
|
||||||
|
Add two columns to `mps_page_session`:
|
||||||
|
|
||||||
|
| Column | Type | Example |
|
||||||
|
|--------|------|---------|
|
||||||
|
| `referrer_domain` | `Unicode(128)` | `"www.google.com"`, `"twitter.com"` |
|
||||||
|
| `referrer_query` | `Unicode(256)` | `"lo-fi beats to study to"` |
|
||||||
|
|
||||||
|
These are populated by refactoring `classify_referrer()` from returning a single
|
||||||
|
integer to returning a `(class, domain, query)` tuple. The domain is extracted
|
||||||
|
from the Referer header URL. The query is parsed from search engine URL
|
||||||
|
parameters (`q=` for Google/DuckDuckGo/Bing, `p=` for Yahoo).
|
||||||
|
|
||||||
|
Privacy: the full Referer URL is never stored. Only the domain and the search
|
||||||
|
query parameter (if present) are kept. Social and unknown referrers store domain
|
||||||
|
only.
|
||||||
|
|
||||||
|
### 2. Add SVG line chart trend visualization
|
||||||
|
|
||||||
|
Server-rendered inline SVG polyline charts for temporal trends. No JavaScript
|
||||||
|
graph libraries — the charts are computed server-side and rendered as a Jinja2
|
||||||
|
macro (`line_chart`) that produces `<svg>` elements with `<polyline>` paths.
|
||||||
|
|
||||||
|
Five trend lines added to both shop-level and per-product analytics:
|
||||||
|
|
||||||
|
| Chart | Y-axis | Color |
|
||||||
|
|-------|--------|-------|
|
||||||
|
| Session duration | Avg `wall_clock_ms` (daily) | Blue |
|
||||||
|
| Engagement | Avg `active_ms / wall_clock_ms` (daily) | Green |
|
||||||
|
| Bounce rate | % sessions with `visible_ms < 7s` (daily) | Red |
|
||||||
|
| External referrers | Daily count of non-direct, non-internal sessions | Orange |
|
||||||
|
|
||||||
|
Each chart spans 28 days with one data point per day. Zero-days are filled so
|
||||||
|
the polyline is continuous.
|
||||||
|
|
||||||
|
### 3. Add keyword and referrer domain tables
|
||||||
|
|
||||||
|
Two new analytics sections:
|
||||||
|
|
||||||
|
**Top Referrer Domains** — ranked table of external domains driving traffic,
|
||||||
|
with session count and percentage bar. Excludes direct and internal traffic.
|
||||||
|
|
||||||
|
**Top Search Queries** — ranked table of search engine queries that led to the
|
||||||
|
shop, extracted from `referrer_query`. Also includes internal shop search
|
||||||
|
keywords from `mps_shop_search_request`.
|
||||||
|
|
||||||
|
### 4. Bucketing functions
|
||||||
|
|
||||||
|
Six new query functions in `analytics.py`:
|
||||||
|
|
||||||
|
- `_daily_buckets()` — daily view counts (bar chart)
|
||||||
|
- `_daily_avg_duration()` — daily avg session duration
|
||||||
|
- `_daily_engagement()` — daily avg engagement ratio
|
||||||
|
- `_daily_bounce_rate()` — daily bounce rate
|
||||||
|
- `_daily_referrer_counts()` — daily external referrer sessions
|
||||||
|
- `_top_referrer_domains()` — top N referrer domains
|
||||||
|
- `_top_referrer_queries()` — top N search engine queries
|
||||||
|
- `_top_search_keywords()` — top N internal search keywords
|
||||||
|
|
||||||
|
## classify_referrer() Refactor
|
||||||
|
|
||||||
|
Before (MPS-2):
|
||||||
|
```python
|
||||||
|
def classify_referrer(referrer, request_host):
|
||||||
|
"""Return 0-4 integer class."""
|
||||||
|
return 1 # search
|
||||||
|
```
|
||||||
|
|
||||||
|
After (MPS-6):
|
||||||
|
```python
|
||||||
|
def classify_referrer(referrer, request_host):
|
||||||
|
"""Return (class, domain, query) tuple."""
|
||||||
|
return (1, "www.google.com", "lo-fi beats")
|
||||||
|
```
|
||||||
|
|
||||||
|
Search engine query extraction:
|
||||||
|
- Google/Bing/DuckDuckGo: `?q=` parameter
|
||||||
|
- Yahoo: `?p=` parameter
|
||||||
|
- Other search engines: `?q=` fallback
|
||||||
|
|
||||||
|
## Files Changed
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|------|--------|
|
||||||
|
| `models/page_session.py` | Add `referrer_domain`, `referrer_query` columns |
|
||||||
|
| `views/signals.py` | Refactor `classify_referrer()` to return tuple; store domain + query |
|
||||||
|
| `views/analytics.py` | 8 new bucketing/query functions; pass trend data to templates |
|
||||||
|
| `templates/analytics.j2` | SVG `line_chart` macro; referrer domain table; keyword table |
|
||||||
|
| `templates/analytics_product.j2` | SVG `line_chart` macro; referrer domain table |
|
||||||
|
| `scripts/alembic/versions/f3086e09b052_*.py` | Migration: add `referrer_domain`, `referrer_query` |
|
||||||
|
| `tests/test_models.py` | 12 tests for `classify_referrer()` tuple return |
|
||||||
|
| `tests/test_functional.py` | Referrer trend chart rendering tests |
|
||||||
|
|
||||||
|
## Depends On
|
||||||
|
|
||||||
|
MPS-3 (analytics dashboard infrastructure)
|
||||||
63
docs/tickets/mps-7.md
Normal file
63
docs/tickets/mps-7.md
Normal file
|
|
@ -0,0 +1,63 @@
|
||||||
|
# MPS-7: Sandbox Mode — Client-Side Creative Filter System
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Creators want visitors to interact with their media beyond passive viewing.
|
||||||
|
Existing tools require downloading, editing in external software, and
|
||||||
|
re-uploading. The friction kills experimentation. Creators need an in-browser
|
||||||
|
creative toolkit that lets visitors play with filters, export artifacts, and
|
||||||
|
optionally upload results — all without leaving the shop.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
A client-side filter engine that runs entirely in the browser. The server stores
|
||||||
|
a single boolean (`shop.sandbox_mode`). When enabled, every page in the shop
|
||||||
|
renders a floating toolbar with 32 filter presets, 7 adjustment sliders, SVG
|
||||||
|
filter effects, canvas export (image + video), and optional face detection via
|
||||||
|
MediaPipe.
|
||||||
|
|
||||||
|
Full architecture and filter reference: [docs/sandbox-mode.md](../sandbox-mode.md)
|
||||||
|
|
||||||
|
### Server-side (minimal)
|
||||||
|
|
||||||
|
- `sandbox_mode` Boolean column on Shop model
|
||||||
|
- Toggle in ribbon-settings form section (same form as announcement ribbon,
|
||||||
|
default theme, grid lanes, show_dates, watch_mode, color_filter)
|
||||||
|
- `base.j2` conditionally renders toolbar HTML, inline SVG filter definitions,
|
||||||
|
and loads `sandbox.js` only when `shop.sandbox_mode` is true
|
||||||
|
- Watch mode SPA integration: `watch.js` calls `window.sandboxReapply()` after
|
||||||
|
DOM swap to re-apply filters to newly loaded media
|
||||||
|
|
||||||
|
### Client-side (sandbox.js, ~700 lines)
|
||||||
|
|
||||||
|
- 32 CSS filter presets (4 basic, 4 warm, 4 cool, 4 dramatic, 5 color, 7
|
||||||
|
Instagram-style, 4 SVG)
|
||||||
|
- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia,
|
||||||
|
grayscale)
|
||||||
|
- Image export via canvas `toBlob()` (requires CORS from CDN)
|
||||||
|
- Video export via `MediaRecorder` + `captureStream(30fps)`
|
||||||
|
- Face detection via MediaPipe Face Mesh (4MB WASM, on-demand load)
|
||||||
|
- localStorage persistence of preset, slider values, panel state
|
||||||
|
- Upload-to-bucket via presigned POST (see MPS-8)
|
||||||
|
|
||||||
|
## Files Changed
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|------|--------|
|
||||||
|
| `models/shop.py` | `sandbox_mode` Boolean column |
|
||||||
|
| `views/shop.py` | Toggle handler in `ribbon-settings` form section |
|
||||||
|
| `templates/shop_settings.j2` | Enable/disable radio buttons |
|
||||||
|
| `templates/base.j2` | Conditional toolbar HTML + inline SVG defs |
|
||||||
|
| `static/js/sandbox.js` | Client-side filter engine (IIFE) |
|
||||||
|
| `static/js/watch.js` | `sandboxReapply()` hook in `updatePageContent()` |
|
||||||
|
| `static/css/common.css` | Toolbar layout (CSS Grid), mobile responsive |
|
||||||
|
| `scripts/alembic/versions/a2c13d3117f2_*.py` | Migration: `sandbox_mode` |
|
||||||
|
| `tests/test_functional.py` | Sandbox toggle, toolbar rendering, script tag tests |
|
||||||
|
|
||||||
|
## Depends On
|
||||||
|
|
||||||
|
Nothing. Independent feature.
|
||||||
|
|
||||||
|
## Blocks
|
||||||
|
|
||||||
|
MPS-8 (user S3 bucket for artifact upload)
|
||||||
92
docs/tickets/mps-8.md
Normal file
92
docs/tickets/mps-8.md
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
# MPS-8: User S3 Bucket + Artifact Storage
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Sandbox mode (MPS-7) lets visitors export filtered images and videos, but
|
||||||
|
exports download to the user's local device. Creators want a way to
|
||||||
|
automatically upload sandbox artifacts to their own cloud storage — a personal
|
||||||
|
S3 bucket where filtered content accumulates without manual file management.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
### User-level S3 credentials
|
||||||
|
|
||||||
|
Add S3-compatible storage credentials to the User model (not Shop — a user may
|
||||||
|
own multiple shops but uses one storage bucket):
|
||||||
|
|
||||||
|
| Column | Type | Example |
|
||||||
|
|--------|------|---------|
|
||||||
|
| `s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` |
|
||||||
|
| `s3_region` | `Unicode(64)` | `"nyc3"` |
|
||||||
|
| `s3_bucket` | `Unicode(128)` | `"my-sandbox-exports"` |
|
||||||
|
| `s3_access_key` | `Unicode(128)` | `"DO00..."` |
|
||||||
|
| `s3_secret_key` | `Unicode(128)` | `"wJalr..."` |
|
||||||
|
|
||||||
|
Property `has_s3_bucket` returns True when endpoint, bucket, access_key, and
|
||||||
|
secret_key are all non-empty.
|
||||||
|
|
||||||
|
### Storage settings form
|
||||||
|
|
||||||
|
New section on user settings page (`/u/settings`) with an "Artifact Storage"
|
||||||
|
form that POSTs to `/u/settings/storage`. Fields: endpoint URL, region
|
||||||
|
(optional), bucket name, access key, secret key (password field).
|
||||||
|
|
||||||
|
### Presigned upload endpoint
|
||||||
|
|
||||||
|
`POST /u/sandbox/upload` — accepts `filename` and `content_type` parameters,
|
||||||
|
generates a presigned POST using the user's stored S3 credentials. Returns
|
||||||
|
`{presigned: {url, fields}, key}`. The browser uploads the blob directly to the
|
||||||
|
user's bucket — the file never touches the MPS server.
|
||||||
|
|
||||||
|
S3 key format: `sandbox/{user_id}/{timestamp}-{filename}`
|
||||||
|
|
||||||
|
Upload limit: 50MB per file (enforced via presigned POST conditions).
|
||||||
|
|
||||||
|
### Toolbar integration
|
||||||
|
|
||||||
|
When a user has `has_s3_bucket = True`, the sandbox toolbar in `base.j2`
|
||||||
|
renders with `data-has-bucket="1"`. The `sandbox.js` upload-to-bucket button
|
||||||
|
appears only when this attribute is present. The upload flow:
|
||||||
|
|
||||||
|
1. User exports image/video (canvas toBlob)
|
||||||
|
2. User clicks "Upload to Bucket"
|
||||||
|
3. JS POSTs to `/u/sandbox/upload` with filename + content_type
|
||||||
|
4. Server returns presigned POST URL + fields
|
||||||
|
5. JS uploads blob directly to user's S3 bucket via FormData POST
|
||||||
|
|
||||||
|
### Supported services
|
||||||
|
|
||||||
|
Any S3-compatible endpoint: DigitalOcean Spaces, AWS S3, MinIO, Backblaze B2,
|
||||||
|
Cloudflare R2.
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
Credentials stored as plain Unicode columns (same pattern as Stripe/PayPal/Adyen
|
||||||
|
keys on Shop model). Secret key field uses `type="password"` in the form.
|
||||||
|
Credentials are only used server-side to generate presigned URLs — they are
|
||||||
|
never exposed to the browser.
|
||||||
|
|
||||||
|
## Shop-Level S3 Mirror (MPS-9)
|
||||||
|
|
||||||
|
A separate but related feature: shops can configure their own S3 bucket as a
|
||||||
|
**mirror** of the MPS main bucket. Every file uploaded to a shop (product files,
|
||||||
|
thumbnails, shop assets) is automatically copied to the shop's mirror bucket in
|
||||||
|
a background thread. See [MPS-9](mps-9.md).
|
||||||
|
|
||||||
|
## Files Changed
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|------|--------|
|
||||||
|
| `models/user.py` | S3 credential columns + `has_s3_bucket` property |
|
||||||
|
| `views/user.py` | Storage settings POST handler + S3 fields in settings dict |
|
||||||
|
| `views/user_sandbox.py` | Presigned upload endpoint |
|
||||||
|
| `templates/user_settings.j2` | Artifact Storage form |
|
||||||
|
| `templates/base.j2` | `data-has-bucket` attribute on sandbox toolbar |
|
||||||
|
| `static/js/sandbox.js` | Upload-to-bucket button + presigned POST flow |
|
||||||
|
| `routes.py` | `user_storage_settings`, `user_sandbox_upload` routes |
|
||||||
|
| `scripts/alembic/versions/f898ba460612_*.py` | Migration: user S3 columns |
|
||||||
|
| `tests/test_functional.py` | S3 bucket save/clear, presigned upload, toolbar attribute tests |
|
||||||
|
|
||||||
|
## Depends On
|
||||||
|
|
||||||
|
MPS-7 (sandbox mode — toolbar must exist to add upload button)
|
||||||
92
docs/tickets/mps-9.md
Normal file
92
docs/tickets/mps-9.md
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
# MPS-9: Shop S3 Mirror Bucket
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Creators want a copy of all their shop files in their own S3 bucket — for
|
||||||
|
backup, CDN flexibility, or migration away from MPS. Currently all files live
|
||||||
|
exclusively in the MPS DigitalOcean Spaces bucket.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
### Shop-level mirror credentials
|
||||||
|
|
||||||
|
Add S3-compatible mirror credentials to the Shop model:
|
||||||
|
|
||||||
|
| Column | Type | Example |
|
||||||
|
|--------|------|---------|
|
||||||
|
| `mirror_s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` |
|
||||||
|
| `mirror_s3_region` | `Unicode(64)` | `"nyc3"` |
|
||||||
|
| `mirror_s3_bucket` | `Unicode(128)` | `"my-shop-mirror"` |
|
||||||
|
| `mirror_s3_access_key` | `Unicode(128)` | `"DO00..."` |
|
||||||
|
| `mirror_s3_secret_key` | `Unicode(128)` | `"wJalr..."` |
|
||||||
|
|
||||||
|
Property `has_s3_mirror` returns True when all required fields are non-empty.
|
||||||
|
|
||||||
|
### Mirror sync engine (`lib/s3_mirror.py`)
|
||||||
|
|
||||||
|
Fire-and-forget sync: every file written to the MPS bucket is copied to the
|
||||||
|
shop's mirror bucket in a daemon thread. The MPS bucket remains the
|
||||||
|
origin/CDN — the mirror is a passive copy.
|
||||||
|
|
||||||
|
Key functions:
|
||||||
|
|
||||||
|
- `mirror_key()` — stream-copy a single key (get_object → put_object)
|
||||||
|
- `mirror_key_async()` — fire-and-forget single key in daemon thread
|
||||||
|
- `mirror_keys_async()` — fire-and-forget multiple keys in one thread
|
||||||
|
- `test_mirror_connection()` — validate credentials by listing bucket
|
||||||
|
- `backfill_mirror_async()` — double-fork detached process that copies all
|
||||||
|
existing shop files to the mirror (survives uWSGI worker recycling)
|
||||||
|
|
||||||
|
Thread safety: ORM objects are not accessed from background threads. All
|
||||||
|
credentials are captured as plain strings before thread creation. Each thread
|
||||||
|
creates its own boto3 client.
|
||||||
|
|
||||||
|
### Sync hooks
|
||||||
|
|
||||||
|
Mirror sync is triggered from:
|
||||||
|
|
||||||
|
- **Product upload** (`views/product.py`) — product file + thumbnail
|
||||||
|
- **Shop asset upload** (`views/shop.py`) — logo, banner
|
||||||
|
|
||||||
|
### Mirror settings form
|
||||||
|
|
||||||
|
New "Mirror Bucket" section in shop settings (`mirror-settings` form section):
|
||||||
|
|
||||||
|
- Endpoint URL, Region, Bucket, Access Key, Secret Key
|
||||||
|
- "Test Connection" — validates credentials on save
|
||||||
|
- "Backfill" toggle — triggers `backfill_mirror_async()` to copy all existing
|
||||||
|
files on first setup
|
||||||
|
|
||||||
|
### Backfill architecture
|
||||||
|
|
||||||
|
The backfill process needs to survive uWSGI worker recycling (workers get killed
|
||||||
|
at 512MB RSS). Solution: double-fork to fully detach from uWSGI:
|
||||||
|
|
||||||
|
```
|
||||||
|
Request handler
|
||||||
|
└─ fork() ─── intermediate child
|
||||||
|
└─ setsid() + fork() ─── grandchild (fully detached)
|
||||||
|
└─ fcntl.flock() guard
|
||||||
|
└─ create own SQLAlchemy engine
|
||||||
|
└─ list_objects_v2 + mirror_key loop
|
||||||
|
└─ os._exit(0)
|
||||||
|
```
|
||||||
|
|
||||||
|
One backfill per shop at a time (flock on `/tmp/s3_mirror_backfill_{shop_id}.lock`).
|
||||||
|
|
||||||
|
## Files Changed
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|------|--------|
|
||||||
|
| `models/shop.py` | Mirror S3 credential columns + `has_s3_mirror` property |
|
||||||
|
| `lib/s3_mirror.py` | Mirror engine: sync, async, test, backfill |
|
||||||
|
| `views/shop.py` | `mirror-settings` form handler + sync hooks |
|
||||||
|
| `views/product.py` | Mirror sync hooks on product upload |
|
||||||
|
| `templates/shop_settings.j2` | Mirror Bucket settings form |
|
||||||
|
| `routes.py` | No new routes (uses existing shop settings POST) |
|
||||||
|
| `scripts/alembic/versions/6b516114c393_*.py` | Migration: shop mirror S3 columns |
|
||||||
|
| `tests/test_functional.py` | Mirror settings save, connection test, backfill |
|
||||||
|
|
||||||
|
## Depends On
|
||||||
|
|
||||||
|
Nothing. Independent feature (but complements MPS-8 user S3 bucket).
|
||||||
Loading…
Add table
Add a link
Reference in a new issue