Two paths to stale client state, both closed:
1. HTML cache — content.py and product.py now send Cache-Control:
no-store, must-revalidate on responses. Browsers were holding
onto rendered sidebar HTML from before the pocket fix deployed,
producing phantom 'this didn't work' reports.
2. localStorage cache — shop.json_discovery_ring + GIT_HASH are
hashed into a short cache_version token, stamped on every page
(<meta name='mps-cache-version'>) and every watch_json response.
watch.js stores it in localStorage; on page load and every SPA
nav, mismatch triggers removal of watchRing, watchRingPosition,
watchRingHistory, watchRingLoops, watchQueue before anything
reads them.
watch_json error responses (404 no media, 403 not public) also
carry cache_version so clients can flush even when the target
product can't be played.
Tests:
- test_models.py TestCacheVersion: 6 unit tests (stability, ring
content change, ring order change, empty ring, none shop,
GIT_HASH flip via patch).
- test_functional.py: 3 functional tests (content page sends
no-store + meta tag, watch_json error carries cache_version,
cache_version shifts after reforge).
All 370 model+integration tests + 8 new functional tests green.
Adds validate_discovery_ring(shop) in models/shop.py that returns a
dict diagnosing four ring topology defects:
- duplicates: IDs appearing more than once in ring (greedy-walk bug)
- orphans: public products missing from ring (added after reforge)
- stale: ring IDs no longer public/present (deleted or unlisted
after reforge — the 'pocket' condition we just patched)
- length_mismatch: ring_length != public_count
Wired into reforge_discovery_ring_async — anomalies log a warning
after each background reforge, making silent drift visible.
New route /s/{shop_id}/ring/health.json exposes the validator to
shop mods (403 for anon and non-editor users, 404 for missing shop).
Tests across all three layers:
- Unit (test_models.py, 7 tests): mocked shop.products, each
anomaly class verified in isolation.
- Integration (test_integration.py, 4 tests): real shop + products
+ reforge, simulates visibility changes and late additions,
confirms reforge heals the ring.
- Functional (test_functional.py, 5 tests): auth required, mod
ownership enforced, 404 on unknown shop, real-world stale
detection through the HTTP endpoint.
get_ring_related_products walked exactly backward+forward positions
in the ring and silently dropped entries whose IDs no longer resolved
to visible products. Result: sparse offsets like [-2, -1, 1, 5, 28]
visible in Up Next — a 'pocket' of live items in an otherwise stale
ring slice.
New behavior: fetch every ring product once (bulk query), keep only
visibility==1, then walk further along the ring to collect the
requested backward/forward VALID neighbors. Offsets are renumbered
contiguously (-N..-1, 1..N). Pocket is filled by skipping past
deleted/unlisted entries until we have the requested count or
exhaust the ring.
Ring traversal on the client (ringPosition + direction) uses ring
indices directly and is unaffected — only the rendered Up Next
sidebar slice is densified.
Previous cinema CSS used width:auto which pinned small-resolution videos
to their natural size, leaving huge side-margins on wide displays.
Cinema explicitly wants edge-to-edge video: width:100%, height:auto,
max-height:92vh, object-fit:contain. This trades a small letterbox on
extra-wide viewports for real full-width rendering — intentional
override of the CLAUDE.md 'never combine width:100% with max-height'
rule, which exists to prevent dead whitespace on images. For cinema
the tradeoff is reversed: fox wants big video, accepts edge letterbox.
New Cinema toggle alongside Fresh/Reverse/Autoplay. When on, adds
.cinema-mode class to section.two-column, restructuring the grid:
row 1: video (full viewport width, max 85vh, preserved aspect)
row 2: description (full width)
row 3: product-right (price + download + Up Next) | comments
Uses display:contents on .watch-left so watch-mode children bubble
up as direct grid items. Video sizing follows CLAUDE.md media rule
(width:auto + max-width:100% + max-height:85vh) to avoid letterbox
whitespace.
Preference persisted in localStorage (watchCinemaMode). Toggle
handler attached in rebindToggles; class applied on init and on
every toggle change.
Karaoke toggle now spans full width of ring-header-controls grid,
pushing nav buttons (Prev/Random/Next) to their own 3-column row.
When karaoke is display:none (non-eligible shops), nav buttons
flow naturally into row 2. No template changes.
Every forward path through the ring now uses a single selector:
chooseNextInRing() = freshMode ? getNextUnwatchedItem() : getNextItem()
Next button, autoplay countdown, DJ crossfade target, preload, and
countdown-play-now all route through it. Previously, Next button
hardcoded skipToNextUnwatched() regardless of Fresh toggle — user
would see offset +1 in sidebar but land on a farther unwatched item.
navigateToNext and completeDjFadeout now sync ringPosition via
indexOf(target) instead of blindly advancing by +direction — necessary
when Fresh mode jumps past watched items.
KeyError 'bucket.secure_uploads.region' in the detached karaoke child
on prod. production.ini stores keys with the app. prefix
(app.bucket.secure_uploads.region) and a request hook strips that
prefix into a dict attached as request.app. request.registry.settings
still carries the raw, prefixed keys.
The upload-time and on-demand karaoke call sites were passing
request.registry.settings; capture_karaoke_config expected the
stripped dict. Dev .ini happens to match both layouts which masked
this — prod raised KeyError and the response 502'd.
Switch both call sites to request.app (matches the pattern used by
backfill_karaoke_async and backfill_mirror_async) and document the
expected shape on capture_karaoke_config.
ParamValidationError on copy_object after upload — ContentType=None
hit S3. get_content_type("thumbnail2") returned None for a .flv file
because mimetypes.types_map only holds Python's built-in table (no
.flv, .mkv, .opus, .m4v, etc.).
Switch to mimetypes.guess_type, which initializes from the OS mime
database and covers every common format. Keep None as the return for
truly unknown extensions so downstream callers that chain `or
"video/mp4"` still pick the right context-aware fallback.
Guard the two copy_object / mirror_key_async sites in views/product.py
with `or "application/octet-stream"` so even a genuinely unknown
extension can't crash the upload handler again.
Upload handler called process_karaoke synchronously — for audio/video
products, vocal isolation (download, ffmpeg, voxsplit, reupload) can
take several minutes. The response hung until karaoke finished, so
users saw a black screen after a successful upload while Caddy/uwsgi
timed out. The file itself was already in S3+DB, which is why hitting
Back showed the edit page with the upload present.
Fix: extract the fork+detach logic the on-demand path already used
(views/watch.py:karaoke_process) into a reusable helper,
process_karaoke_detached, in lib/karaoke.py. Both the upload path and
the on-demand path now hand off to the helper and return immediately.
The helper takes a snapshot of shop+S3+mirror config in the parent
(capture_karaoke_config) so the grandchild only touches the DB at the
end for the final metadata write — by then the parent has long since
committed. Mirror writes run synchronously in the grandchild instead
of via mirror_keys_async daemon threads that were dying at os._exit.
digest_sender cron was crashing with transaction.interfaces.NoTransaction.
bootstrap() doesn't start a transaction — pyramid_tm handles that for web
requests but not console scripts. Wrap DB work in transaction.manager context
instead of manual transaction.commit() calls.
Same latent defect fixed in backfill_karaoke.py.
Both _torrent_backfill_async and _checksum_backfill_async were blocking
the request thread by iterating all products before returning — causing
the settings form to hang on "Saving...".
Fix: snapshot shop_id, bucket, S3 creds, and work list while the DB
session is live, then hand off to a single daemon thread and return.
The request redirects instantly.
Paid products seed: preview + thumbnails + index.md + index.html
Free content seeds: content file + thumbnails + index.md + index.html
The paid product file is never included in any torrent.
- lib/torrent.py: redesign generate_torrent for directory bundles (torf multi-file)
- description written as index.md + rendered index.html
- torrent stored as bundle.torrent (was product.torrent)
- build_bundle_files() encapsulates what belongs in each bundle type
- views/product.py: trigger fires on preview upload for paid products,
product file upload for free content
- views/shop.py: backfill eligibility mirrors same rule; backfill_status too
- tests: rewrite TestTorrentLib for new bundle interface, add bundle_files tests
- Add torrent_opt_in boolean to mps_product (default false)
- Gate all torrent seeding (upload trigger, backfill) on torrent_opt_in AND visibility==1
- Suppress magnet/torrent display for all users when product is private or unlisted
- Add permanence warning and explicit consent checkbox on product edit page
- backfill_status endpoint counts only opted-in public products
- Alembic migration: 0ee5654cfe7d
Adds GET /s/{shop_id}/torrent-backfill-status JSON endpoint returning
total/done/remaining counts for products needing magnet link generation.
Shop settings page shows a progress bar + status line when torrent
distribution is enabled. Polls every 4 seconds, stops when done.
Adds a public/private key pair authentication system and REST API endpoints
for programmatic product and content management. Designed for CI/CD pipelines
(permacomputer.com image hosting).
Auth: HMAC-SHA256 signed requests using public/private key pairs.
The secret key never travels over the wire. Replay window: 300 seconds.
Endpoints:
POST /api/v1/products create product (fiat/crypto priced)
POST /api/v1/content create content (free)
GET /api/v1/products/{id} get product
GET /api/v1/content/{id} get content
POST /api/v1/products/{id}/upload-url presigned S3 POST for direct upload
POST /api/v1/content/{id}/upload-url presigned S3 POST for direct upload
POST /api/v1/products/{id}/files/confirm confirm upload, register metadata
POST /api/v1/content/{id}/files/confirm confirm upload, register metadata
Key management UI in shop settings. Secret shown once on generation.
Migration: mps_api_key table (id, shop_id, public_key, secret_key, label,
created_timestamp, last_used_timestamp, is_active)
Tests: 12 MpsApiKey unit tests, 8 REST API functional tests (269 total passing)
The torrent_distribution_support migration was manually created with a
duplicate revision ID (a1b2c3d4e5f6) already in use by an older migration,
and pointed to the wrong down_revision (fd9f7e2f2b78 root instead of
9884324a48e3 current head). This caused a cycle in the revision map and
alembic upgrade head failed silently, leaving prod with new code but old
schema → 502 on all shop endpoints.
Deleted the broken file, regenerated with alembic revision to get a proper
unique ID (c0236e351476), set correct down_revision, preserved idempotent
_column_exists guards.
- backfill: enabling torrent on a shop auto-generates .torrent for all
existing products that have a product file (no manual trigger needed)
- web seed (BEP 19): CDN url embedded in .torrent + magnet link so
clients bootstrap via HTTP then seed to peers (no seeder process needed)
- torrent_file_url: stored on Product, shown as download link on content
and edit pages alongside the magnet link
- migration: idempotent _column_exists guards on all add_column calls
- template: grid layout (not flex) for magnet/torrent buttons on edit page
- tests: 13 passing tests covering all new paths (unit + functional)
including backfill trigger, web seed construction, visibility gating
lib/torrent.py: generate_torrent_async() — daemon thread that fires after
upload, downloads file from S3, builds .torrent via torf, uploads
product.torrent to S3 (public-read), saves magnet link to DB.
views/product.py: trigger generate_torrent_async when file_key=='product'
and shop.torrent_enabled. Removed manual magnet link save — it's automatic.
templates/product_edit.j2: magnet link is read-only + copy button + Open
button when generated; 'upload the product file' hint when not yet generated.
tests: updated to reflect auto-generation model, added content page test.
Queue was sessionStorage (per-tab only). Switch to localStorage so any
tab can add to our shared queue. Storage events fire in all other same-origin
tabs on mutation — renderQueue() wired to keep UI in sync automatically.
Add limit_html_nesting() to sanitize_html.py. Flattens any HTML element
nested beyond depth 20 using html.parser (O(N)) before the content reaches
bleach/html5lib. N=35 attack drops from 12.8s to 0.04s.
Wire into markdown_to_html() in render.py — single enforcement point for
all callers: product descriptions, shop descriptions, privacy policy,
terms of service, markup preview.
No byte cap — books, long-form content, and deep table-of-contents
structures are fully supported. The depth limit (20 levels) prevents the
exponential zone while accommodating any legitimate nesting depth.
markup_editor_preview: reject requests where data exceeds 100KB
before passing to markdown/bleach pipeline (same class as
CWE-407 bleach O(2^N) finding — no input cap on AJAX preview).
add_shop_theme_classes: replace single compound regex with four
anchored non-backtracking patterns to eliminate ReDoS risk on
color validation. Practical risk was low (theme_link_color is
always derived from color_scale()), but the pattern was unsafe.
