Upload handler called process_karaoke synchronously — for audio/video
products, vocal isolation (download, ffmpeg, voxsplit, reupload) can
take several minutes. The response hung until karaoke finished, so
users saw a black screen after a successful upload while Caddy/uwsgi
timed out. The file itself was already in S3+DB, which is why hitting
Back showed the edit page with the upload present.
Fix: extract the fork+detach logic the on-demand path already used
(views/watch.py:karaoke_process) into a reusable helper,
process_karaoke_detached, in lib/karaoke.py. Both the upload path and
the on-demand path now hand off to the helper and return immediately.
The helper takes a snapshot of shop+S3+mirror config in the parent
(capture_karaoke_config) so the grandchild only touches the DB at the
end for the final metadata write — by then the parent has long since
committed. Mirror writes run synchronously in the grandchild instead
of via mirror_keys_async daemon threads that were dying at os._exit.
digest_sender cron was crashing with transaction.interfaces.NoTransaction.
bootstrap() doesn't start a transaction — pyramid_tm handles that for web
requests but not console scripts. Wrap DB work in transaction.manager context
instead of manual transaction.commit() calls.
Same latent defect fixed in backfill_karaoke.py.
Both _torrent_backfill_async and _checksum_backfill_async were blocking
the request thread by iterating all products before returning — causing
the settings form to hang on "Saving...".
Fix: snapshot shop_id, bucket, S3 creds, and work list while the DB
session is live, then hand off to a single daemon thread and return.
The request redirects instantly.
Paid products seed: preview + thumbnails + index.md + index.html
Free content seeds: content file + thumbnails + index.md + index.html
The paid product file is never included in any torrent.
- lib/torrent.py: redesign generate_torrent for directory bundles (torf multi-file)
- description written as index.md + rendered index.html
- torrent stored as bundle.torrent (was product.torrent)
- build_bundle_files() encapsulates what belongs in each bundle type
- views/product.py: trigger fires on preview upload for paid products,
product file upload for free content
- views/shop.py: backfill eligibility mirrors same rule; backfill_status too
- tests: rewrite TestTorrentLib for new bundle interface, add bundle_files tests
- Add torrent_opt_in boolean to mps_product (default false)
- Gate all torrent seeding (upload trigger, backfill) on torrent_opt_in AND visibility==1
- Suppress magnet/torrent display for all users when product is private or unlisted
- Add permanence warning and explicit consent checkbox on product edit page
- backfill_status endpoint counts only opted-in public products
- Alembic migration: 0ee5654cfe7d
Adds GET /s/{shop_id}/torrent-backfill-status JSON endpoint returning
total/done/remaining counts for products needing magnet link generation.
Shop settings page shows a progress bar + status line when torrent
distribution is enabled. Polls every 4 seconds, stops when done.
Adds a public/private key pair authentication system and REST API endpoints
for programmatic product and content management. Designed for CI/CD pipelines
(permacomputer.com image hosting).
Auth: HMAC-SHA256 signed requests using public/private key pairs.
The secret key never travels over the wire. Replay window: 300 seconds.
Endpoints:
POST /api/v1/products create product (fiat/crypto priced)
POST /api/v1/content create content (free)
GET /api/v1/products/{id} get product
GET /api/v1/content/{id} get content
POST /api/v1/products/{id}/upload-url presigned S3 POST for direct upload
POST /api/v1/content/{id}/upload-url presigned S3 POST for direct upload
POST /api/v1/products/{id}/files/confirm confirm upload, register metadata
POST /api/v1/content/{id}/files/confirm confirm upload, register metadata
Key management UI in shop settings. Secret shown once on generation.
Migration: mps_api_key table (id, shop_id, public_key, secret_key, label,
created_timestamp, last_used_timestamp, is_active)
Tests: 12 MpsApiKey unit tests, 8 REST API functional tests (269 total passing)
The torrent_distribution_support migration was manually created with a
duplicate revision ID (a1b2c3d4e5f6) already in use by an older migration,
and pointed to the wrong down_revision (fd9f7e2f2b78 root instead of
9884324a48e3 current head). This caused a cycle in the revision map and
alembic upgrade head failed silently, leaving prod with new code but old
schema → 502 on all shop endpoints.
Deleted the broken file, regenerated with alembic revision to get a proper
unique ID (c0236e351476), set correct down_revision, preserved idempotent
_column_exists guards.
- backfill: enabling torrent on a shop auto-generates .torrent for all
existing products that have a product file (no manual trigger needed)
- web seed (BEP 19): CDN url embedded in .torrent + magnet link so
clients bootstrap via HTTP then seed to peers (no seeder process needed)
- torrent_file_url: stored on Product, shown as download link on content
and edit pages alongside the magnet link
- migration: idempotent _column_exists guards on all add_column calls
- template: grid layout (not flex) for magnet/torrent buttons on edit page
- tests: 13 passing tests covering all new paths (unit + functional)
including backfill trigger, web seed construction, visibility gating
lib/torrent.py: generate_torrent_async() — daemon thread that fires after
upload, downloads file from S3, builds .torrent via torf, uploads
product.torrent to S3 (public-read), saves magnet link to DB.
views/product.py: trigger generate_torrent_async when file_key=='product'
and shop.torrent_enabled. Removed manual magnet link save — it's automatic.
templates/product_edit.j2: magnet link is read-only + copy button + Open
button when generated; 'upload the product file' hint when not yet generated.
tests: updated to reflect auto-generation model, added content page test.
Queue was sessionStorage (per-tab only). Switch to localStorage so any
tab can add to our shared queue. Storage events fire in all other same-origin
tabs on mutation — renderQueue() wired to keep UI in sync automatically.
Add limit_html_nesting() to sanitize_html.py. Flattens any HTML element
nested beyond depth 20 using html.parser (O(N)) before the content reaches
bleach/html5lib. N=35 attack drops from 12.8s to 0.04s.
Wire into markdown_to_html() in render.py — single enforcement point for
all callers: product descriptions, shop descriptions, privacy policy,
terms of service, markup preview.
No byte cap — books, long-form content, and deep table-of-contents
structures are fully supported. The depth limit (20 levels) prevents the
exponential zone while accommodating any legitimate nesting depth.
markup_editor_preview: reject requests where data exceeds 100KB
before passing to markdown/bleach pipeline (same class as
CWE-407 bleach O(2^N) finding — no input cap on AJAX preview).
add_shop_theme_classes: replace single compound regex with four
anchored non-backtracking patterns to eliminate ReDoS risk on
color validation. Practical risk was low (theme_link_color is
always derived from color_scale()), but the pattern was unsafe.
Add Security section to CLAUDE.md documenting both CWE-407 surfaces:
- Search/feed endpoints (fixed, commit f9cbebb)
- Bleach HTML sanitization: O(2^N) on crafted HTML, no input cap in MPS
Add docs/poc-cwe407.py: proof-of-concept timing harness covering
rbox-search, rbox-page, rbox-dump, mps-search, mps-sitemap vectors.
Authorized use only — run against own staging/dev instance.
- shop.py: strip empty tokens, cap keywords to 10 before passing to model
- product.py: add .limit(200) per keyword query — was unbounded .all()
- feeds.py: add .limit(1000) on product/content feed queries — was unbounded
Design doc for ingesting WordPress sites into MPS shops. Covers two
input modes (REST API + WXR XML), 4-phase HTML conversion pipeline,
content/media/comment mapping, CLI interface, competitive analysis,
and future enhancements. Includes rendered dot diagrams for the
architecture overview and HTML conversion detail flow.
Add docs/karaoke-pipeline.md covering the full streaming pipeline from
MPS through unsandbox API to zerotrust container and back. Includes two
Graphviz dot diagrams (rendered to SVG):
- karaoke-pipeline.dot: full system flow across MPS, API, pool, container
- karaoke-ondemand.dot: watch mode on-demand user flow
Update architecture.md feature toggle matrix and related docs table.
Update CLAUDE.md karaoke section with streaming path and on-demand info.
Karaoke button (🎤) now always visible for audio/video products when the
shop has unsandbox API keys configured. Clicking it triggers on-demand
processing via POST /karaoke/{product_id} if tracks don't exist yet.
Server forks a detached child process (survives uWSGI recycling) to run
process_karaoke, then the existing 10s watch_json refresh picks up the
new URLs when processing completes. Button shows hourglass during
processing and auto-switches to instrumentals on completion.
Changes:
- New route + view: POST /karaoke/{product_id} (on-demand processing)
- watch_json + content.py: add karaoke_eligible flag
- content.j2: data-karaoke-eligible attribute on media container
- watch.js: show button when eligible, trigger processing, detect
completion via URL refresh, auto-switch to instrumentals
- related_content.j2: mic emoji button, JS controls visibility
- CSS: disabled state for processing button
- 8 new functional tests covering eligibility, processing, edge cases
1. Crossfade race condition: when a DJ crossfade is active and the user
clicks a new song, cancelDjCrossfade resets djCrossfadeActive but the
old song is still near its end — timeupdate immediately re-triggers
startDjCrossfade, racing with the in-flight fetch. Added
navigationPending flag to block DJ crossfade and ended handler while
a user-initiated navigation is in progress.
2. getNextItem/getNextUnwatchedItem now respect media type filters.
Previously, filtering to "video only" still showed an image in the
countdown because the ring walked by position without checking
data-media-type. Now skips filtered items.
3. Countdown overlay moved from position:absolute inside the video/audio
container (covering native controls) to a flow-positioned element
between the media and title. Removed watch-countdown-static class
since all countdown instances now use the same in-flow layout.
- reforge_discovery_ring_async: add non-production guard matching sync version
- is_trial_active/is_trial_expired: check trial_ended flag
- has_primary_s3: include primary_s3_region in validation
- gift card migration downgrade: add missing json_gift_cards drop + existence guards
