diff --git a/CLAUDE.md b/CLAUDE.md
index e57d55e..118fc82 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -29,15 +29,20 @@ Files are NEVER streamed through uwsgi. The server only generates presigned URLs
### Karaoke Pipeline (lib/karaoke.py)
-Disk-backed vocal isolation pipeline. Downloads media from S3, builds a JSON
-request on disk with streaming base64 encoding, POSTs to unsandbox API, streams
-response to disk, decodes artifacts one at a time and uploads to S3.
+Disk-backed vocal isolation pipeline using spectral mid-side Wiener masking
+(`voxsplit.c`, zero ML deps). Streams media to unsandbox via `POST /upload`
+(64KB chunks, constant memory), executes in zerotrust container, streams
+response back, uploads instrumentals + vocals to S3.
+
+Full architecture doc: `docs/karaoke-pipeline.md` (with dot diagrams).
- **Concurrency**: `ThreadPoolExecutor` sized to account's unsandbox concurrency limit
-- **Memory**: ~64KB per worker on the request side (disk-backed, not in-memory)
+- **Memory**: ~64KB per worker at every stage (disk-backed, not in-memory)
- **Upstream limit**: 3.698GB / 3,698,742,051 bytes per file (unsandbox `@max_upload_bytes`)
- **Retries**: 3 attempts with exponential backoff (5s, 10s)
-- **Callers**: `views/product.py`, `views/shop.py` (backfill), `scripts/backfill_karaoke.py`
+- **Callers**: `views/product.py` (upload), `views/watch.py` (on-demand), `views/shop.py` (backfill), `scripts/backfill_karaoke.py`
+- **On-demand**: `POST /karaoke/{product_id}` — forks detached child, watch.js 10s refresh detects completion, auto-switches to instrumentals
+- **Streaming path**: MPS → `POST /upload` → API encrypts to disk → pool pulls via `GET /internal/upload/{id}` → pipes into container `/root/input/` — zero bytes cross Erlang distribution
## Project Setup
diff --git a/docs/architecture.md b/docs/architecture.md
index 1368f7a..28b0640 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -206,6 +206,7 @@ mps_page_session (raw rows)
| Trial | `shop.trial_started_timestamp` | Auto on creation | 21 days |
| Discovery ring | `shop.discovery_ring` | Automatic | Auto-computed |
| Subscriptions | `shop.subscription_*` | `ribbon-settings` | Off |
+| Karaoke (vocal isolation) | `shop.unsandbox_*_key` | `unsandbox-settings` | Off |
## Ticket Index
@@ -235,5 +236,6 @@ mps_page_session (raw rows)
|-----|---------|
| [Design System](design-system.md) | Design tokens, CSS architecture, component library |
| [JavaScript](JAVASCRIPT.md) | Client-side JS architecture |
+| [Karaoke Pipeline](karaoke-pipeline.md) | Vocal isolation: pipeline, on-demand, streaming architecture |
| [Sandbox Mode](sandbox-mode.md) | Creative filter system |
| [Testing Performance](testing-performance.md) | Test suite optimization |
diff --git a/docs/karaoke-ondemand.dot b/docs/karaoke-ondemand.dot
new file mode 100644
index 0000000..2773a5b
--- /dev/null
+++ b/docs/karaoke-ondemand.dot
@@ -0,0 +1,30 @@
+// On-Demand Karaoke — Watch Mode User Flow
+// Render: dot -Tsvg docs/karaoke-ondemand.dot -o docs/karaoke-ondemand.dot.svg
+digraph karaoke_ondemand {
+ rankdir=TB;
+ node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
+ edge [fontname="monospace", fontsize=9];
+
+ click [label="User clicks 🎤\n(watch.js)", fillcolor="#e0f2f1"];
+ check [label="Has tracks?", shape=diamond, fillcolor="#fff8e1"];
+ cycle [label="cycleKaraoke()\nOriginal → Instrumentals → Vocals", fillcolor="#e8f5e9"];
+ post [label="POST /karaoke/{id}\n(fetch, non-blocking)", fillcolor="#e8eaf6"];
+ fork [label="Server forks\ndetached grandchild", fillcolor="#e8eaf6"];
+ process [label="process_karaoke()\n(~30-120s)", fillcolor="#fce4ec"];
+ hourglass [label="Button shows ⌛\nkaraokeProcessing=true", fillcolor="#e0f2f1"];
+ refresh [label="10s URL refresh\nfetchWatchData()", fillcolor="#e0f2f1"];
+ detect [label="instrumentals_url\nappears in JSON?", shape=diamond, fillcolor="#fff8e1"];
+ autoswitch [label="Auto-switch to\ninstrumentals 🎤", fillcolor="#e8f5e9"];
+
+ click -> check;
+ check -> cycle [label="yes"];
+ check -> post [label="no (eligible)"];
+ post -> fork [label="200 {status: processing}"];
+ post -> hourglass;
+ fork -> process;
+ hourglass -> refresh [label="every 10s"];
+ refresh -> detect;
+ detect -> refresh [label="not yet"];
+ detect -> autoswitch [label="tracks ready"];
+ process -> detect [label="DB updated", style=dashed];
+}
diff --git a/docs/karaoke-ondemand.dot.svg b/docs/karaoke-ondemand.dot.svg
new file mode 100644
index 0000000..53e6023
--- /dev/null
+++ b/docs/karaoke-ondemand.dot.svg
@@ -0,0 +1,155 @@
+
+
+
+
+
diff --git a/docs/karaoke-pipeline.dot b/docs/karaoke-pipeline.dot
new file mode 100644
index 0000000..16b5b7d
--- /dev/null
+++ b/docs/karaoke-pipeline.dot
@@ -0,0 +1,77 @@
+// Karaoke Pipeline — MPS → Unsandbox → Container → S3
+// Render: dot -Tsvg docs/karaoke-pipeline.dot -o docs/karaoke-pipeline.dot.svg
+digraph karaoke_pipeline {
+ rankdir=LR;
+ node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
+ edge [fontname="monospace", fontsize=9];
+
+ subgraph cluster_mps {
+ label="MPS (uWSGI)";
+ style=dashed;
+ color="#5871ad";
+
+ trigger [label="Trigger\n(upload / on-demand / backfill)", fillcolor="#e8eaf6"];
+ s3_down [label="S3 Download\n→ /tmp/karaoke_*/media.bin\n64KB chunks", fillcolor="#e8eaf6"];
+ upload_media [label="POST /upload\n(streaming file body)\n→ upload_id", fillcolor="#e8eaf6"];
+ upload_vox [label="POST /upload\nvoxsplit.c\n→ upload_id", fillcolor="#e8eaf6"];
+ execute [label="POST /execute\n{upload_ids, script}\ntiny JSON, no file bytes", fillcolor="#e8eaf6"];
+ stream_resp [label="Stream response\n→ /tmp/karaoke_*/response.json\n64KB chunks", fillcolor="#e8eaf6"];
+ decode [label="Decode artifacts\n(base64 → tmpfile)\nupload to S3", fillcolor="#e8eaf6"];
+ db_update [label="Update product\nextensions + file_bytes\nset S3 ACLs", fillcolor="#e8eaf6"];
+ }
+
+ subgraph cluster_api {
+ label="api.unsandbox.com";
+ style=dashed;
+ color="#ad5871";
+
+ recv_upload [label="Receive upload\nAES-256-CTR encrypt\n→ /tmp/uploads/{id}.enc\nkey in ETS only", fillcolor="#fce4ec"];
+ dispatch [label="Dispatch execute\nErlang RPC\n~200B metadata only", fillcolor="#fce4ec"];
+ }
+
+ subgraph cluster_pool {
+ label="Pool Node";
+ style=dashed;
+ color="#58ad71";
+
+ pull [label="GET /internal/upload/{id}\nX-Upload-Key auth\nstreaming decrypt", fillcolor="#e8f5e9"];
+ inject [label="lxc exec ... cat >\n/root/input/{filename}\n64KB streaming pipe", fillcolor="#e8f5e9"];
+ }
+
+ subgraph cluster_container {
+ label="Zerotrust Container";
+ style=dashed;
+ color="#ad8f58";
+
+ compile [label="gcc -O2 voxsplit.c -lm", fillcolor="#fff8e1"];
+ extract [label="ffmpeg -i media\n-vn -acodec pcm_s16le\n-ar 44100 -ac 2\naudio.wav", fillcolor="#fff8e1"];
+ split [label="voxsplit audio.wav\n→ split-instrumental.wav\n→ split-vocal.wav", fillcolor="#fff8e1"];
+ remux [label="ffmpeg remux\n(video: copy video +\nisolated audio)\n(audio: copy wav)", fillcolor="#fff8e1"];
+ artifacts [label="/tmp/artifacts/\ninstrumentals.{ext}\nvocals.{ext}", fillcolor="#fff8e1"];
+ }
+
+ s3 [label="S3 / CDN\n(DO Spaces or BYOB)", shape=cylinder, fillcolor="#f3e5f5"];
+ browser [label="Browser\n(watch.js)", shape=ellipse, fillcolor="#e0f2f1"];
+
+ trigger -> s3_down;
+ s3_down -> upload_media [label="file on disk"];
+ s3_down -> upload_vox;
+ upload_media -> recv_upload [label="streaming\noctet-stream"];
+ upload_vox -> recv_upload;
+ recv_upload -> execute [label="upload_id", style=dashed, dir=back];
+ execute -> dispatch [label="JSON\n{upload_ids}"];
+ dispatch -> pull [label="RPC\nmetadata only"];
+ pull -> recv_upload [label="HTTPS GET\nstreaming decrypt", style=dotted];
+ pull -> inject;
+ inject -> compile;
+ compile -> extract;
+ extract -> split;
+ split -> remux;
+ remux -> artifacts;
+ artifacts -> stream_resp [label="response JSON\nbase64 artifacts", style=dotted];
+ stream_resp -> decode;
+ decode -> s3 [label="PUT instrumentals\nPUT vocals"];
+ db_update -> s3 [label="ACL update", style=dashed];
+ decode -> db_update;
+ s3 -> browser [label="presigned URL\n15 min TTL"];
+}
diff --git a/docs/karaoke-pipeline.dot.svg b/docs/karaoke-pipeline.dot.svg
new file mode 100644
index 0000000..8681fd9
--- /dev/null
+++ b/docs/karaoke-pipeline.dot.svg
@@ -0,0 +1,327 @@
+
+
+
+
+
diff --git a/docs/karaoke-pipeline.md b/docs/karaoke-pipeline.md
new file mode 100644
index 0000000..5370856
--- /dev/null
+++ b/docs/karaoke-pipeline.md
@@ -0,0 +1,286 @@
+# Karaoke Pipeline — Vocal Isolation via Unsandbox
+
+## Overview
+
+MPS separates instrumentals and vocals from audio/video uploads using spectral
+mid-side Wiener masking (`voxsplit.c`, zero ML dependencies). Processing runs
+inside unsandbox zerotrust containers. The pipeline is disk-backed with ~64KB
+memory per worker at every stage.
+
+## Architecture Diagram
+
+```dot
+// Render: dot -Tsvg docs/karaoke-pipeline.dot -o docs/karaoke-pipeline.dot.svg
+digraph karaoke_pipeline {
+ rankdir=LR;
+ node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
+ edge [fontname="monospace", fontsize=9];
+
+ subgraph cluster_mps {
+ label="MPS (uWSGI)";
+ style=dashed;
+ color="#5871ad";
+
+ trigger [label="Trigger\n(upload / on-demand / backfill)", fillcolor="#e8eaf6"];
+ s3_down [label="S3 Download\n→ /tmp/karaoke_*/media.bin\n64KB chunks", fillcolor="#e8eaf6"];
+ upload_media [label="POST /upload\n(streaming file body)\n→ upload_id", fillcolor="#e8eaf6"];
+ upload_vox [label="POST /upload\nvoxsplit.c\n→ upload_id", fillcolor="#e8eaf6"];
+ execute [label="POST /execute\n{upload_ids, script}\ntiny JSON, no file bytes", fillcolor="#e8eaf6"];
+ stream_resp [label="Stream response\n→ /tmp/karaoke_*/response.json\n64KB chunks", fillcolor="#e8eaf6"];
+ decode [label="Decode artifacts\n(base64 → tmpfile)\nupload to S3", fillcolor="#e8eaf6"];
+ db_update [label="Update product\nextensions + file_bytes\nset S3 ACLs", fillcolor="#e8eaf6"];
+ }
+
+ subgraph cluster_api {
+ label="api.unsandbox.com";
+ style=dashed;
+ color="#ad5871";
+
+ recv_upload [label="Receive upload\nAES-256-CTR encrypt\n→ /tmp/uploads/{id}.enc\nkey in ETS only", fillcolor="#fce4ec"];
+ dispatch [label="Dispatch execute\nErlang RPC\n~200B metadata only", fillcolor="#fce4ec"];
+ }
+
+ subgraph cluster_pool {
+ label="Pool Node";
+ style=dashed;
+ color="#58ad71";
+
+ pull [label="GET /internal/upload/{id}\nX-Upload-Key auth\nstreaming decrypt", fillcolor="#e8f5e9"];
+ inject [label="lxc exec ... cat >\n/root/input/{filename}\n64KB streaming pipe", fillcolor="#e8f5e9"];
+ }
+
+ subgraph cluster_container {
+ label="Zerotrust Container";
+ style=dashed;
+ color="#ad8f58";
+
+ compile [label="gcc -O2 voxsplit.c -lm", fillcolor="#fff8e1"];
+ extract [label="ffmpeg -i media\n-vn -acodec pcm_s16le\n-ar 44100 -ac 2\naudio.wav", fillcolor="#fff8e1"];
+ split [label="voxsplit audio.wav\n→ split-instrumental.wav\n→ split-vocal.wav", fillcolor="#fff8e1"];
+ remux [label="ffmpeg remux\n(video: copy video +\nisolated audio)\n(audio: copy wav)", fillcolor="#fff8e1"];
+ artifacts [label="/tmp/artifacts/\ninstrumentals.{ext}\nvocals.{ext}", fillcolor="#fff8e1"];
+ }
+
+ s3 [label="S3 / CDN\n(DO Spaces or BYOB)", shape=cylinder, fillcolor="#f3e5f5"];
+ browser [label="Browser\n(watch.js)", shape=ellipse, fillcolor="#e0f2f1"];
+
+ trigger -> s3_down;
+ s3_down -> upload_media [label="file on disk"];
+ s3_down -> upload_vox;
+ upload_media -> recv_upload [label="streaming\noctet-stream"];
+ upload_vox -> recv_upload;
+ recv_upload -> execute [label="upload_id", style=dashed, dir=back];
+ execute -> dispatch [label="JSON\n{upload_ids}"];
+ dispatch -> pull [label="RPC\nmetadata only"];
+ pull -> recv_upload [label="HTTPS GET\nstreaming decrypt", style=dotted];
+ pull -> inject;
+ inject -> compile;
+ compile -> extract;
+ extract -> split;
+ split -> remux;
+ remux -> artifacts;
+ artifacts -> stream_resp [label="response JSON\nbase64 artifacts", style=dotted];
+ stream_resp -> decode;
+ decode -> s3 [label="PUT instrumentals\nPUT vocals"];
+ db_update -> s3 [label="ACL update", style=dashed];
+ decode -> db_update;
+ s3 -> browser [label="presigned URL\n15 min TTL"];
+}
+```
+
+## Pipeline Stages
+
+### Stage 1: Trigger
+
+Three entry points, all converge on `process_karaoke()`:
+
+| Entry Point | File | When |
+|-------------|------|------|
+| Upload | `views/product.py:382` | User uploads audio/video to product |
+| On-demand | `views/watch.py:karaoke_process` | User clicks 🎤 button in watch mode |
+| Backfill | `lib/karaoke.py:backfill_karaoke_async` | Shop owner triggers from settings |
+
+On-demand and backfill fork a detached grandchild process (double-fork +
+`os.setsid()`) that survives uWSGI worker recycling. Per-product lockfile
+(`/tmp/karaoke_{product_id}.lock`) prevents duplicate processing.
+
+### Stage 2: Download from S3
+
+```python
+# karaoke.py:217-226
+resp = s3_client.get_object(Bucket=bucket, Key=s3_key)
+with open(media_path, "wb") as f:
+ while True:
+ chunk = body.read(65536) # 64KB
+ if not chunk: break
+ f.write(chunk)
+```
+
+### Stage 3: Upload to Unsandbox API
+
+```python
+# karaoke.py:45-77
+# HMAC signs empty body — API skips body parsing for /upload
+headers = {"X-Filename": filename, "Content-Type": "application/octet-stream"}
+with open(file_path, "rb") as body:
+ response = requests.post(url, data=body, headers=headers)
+# Returns: {"upload_id": "uuid"}
+```
+
+Two uploads: the media file + `voxsplit.c` source. Both stream from disk,
+constant memory.
+
+### Stage 4: API Receives + Encrypts
+
+`upload_store.ex` writes incoming bytes to `/tmp/uploads/{id}.enc` with
+AES-256-CTR encryption. Unique key+IV per upload. Keys live only in ETS
+(BEAM memory, never written to disk). Files auto-expire after 15 minutes.
+
+### Stage 5: Execute (Metadata Only)
+
+```python
+# karaoke.py:80-125
+payload = {
+ "language": "bash",
+ "code": script,
+ "network_mode": "zerotrust",
+ "input_files": [
+ {"upload_id": media_id, "filename": "media"},
+ {"upload_id": voxsplit_id, "filename": "voxsplit.c"},
+ ],
+}
+```
+
+Tiny JSON body. Zero file content crosses Erlang distribution.
+
+### Stage 6: Pool Pulls Files
+
+Pool node calls `GET /internal/upload/{id}` back to the API. API decrypts
+on-the-fly, streams plaintext. Pool pipes into container:
+
+```
+lxc exec {container} -- sh -c 'cat > /root/input/media'
+```
+
+`/tmp/input` symlinks to `/root/input/` (canonical location).
+
+### Stage 7: Container Processing
+
+```bash
+# Compile voxsplit (spectral mid-side Wiener masking, libc + libm only)
+gcc -O2 -o /tmp/voxsplit /tmp/input/voxsplit.c -lm
+
+# Extract audio from media
+ffmpeg -y -i /tmp/input/media -vn -acodec pcm_s16le -ar 44100 -ac 2 /tmp/audio.wav
+
+# Separate instrumentals and vocals
+/tmp/voxsplit /tmp/audio.wav -o /tmp/split
+
+# For video: remux original video with each isolated audio
+ffmpeg -y -i /tmp/input/media -i /tmp/split-instrumental.wav \
+ -c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/instrumentals.mp4
+ffmpeg -y -i /tmp/input/media -i /tmp/split-vocal.wav \
+ -c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/vocals.mp4
+```
+
+### Stage 8: Response + S3 Upload
+
+Response JSON streams back to MPS temp file. Artifacts are base64-encoded in
+the response. MPS decodes each one to a temp file, uploads to S3, deletes
+temp file. Updates product metadata and S3 ACLs.
+
+## On-Demand Flow (Watch Mode)
+
+```dot
+// Render: dot -Tsvg docs/karaoke-ondemand.dot -o docs/karaoke-ondemand.dot.svg
+digraph karaoke_ondemand {
+ rankdir=TB;
+ node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
+ edge [fontname="monospace", fontsize=9];
+
+ click [label="User clicks 🎤\n(watch.js)", fillcolor="#e0f2f1"];
+ check [label="Has tracks?", shape=diamond, fillcolor="#fff8e1"];
+ cycle [label="cycleKaraoke()\nOriginal → Instrumentals → Vocals", fillcolor="#e8f5e9"];
+ post [label="POST /karaoke/{id}\n(fetch, non-blocking)", fillcolor="#e8eaf6"];
+ fork [label="Server forks\ndetached grandchild", fillcolor="#e8eaf6"];
+ process [label="process_karaoke()\n(~30-120s)", fillcolor="#fce4ec"];
+ hourglass [label="Button shows ⌛\nkaraokeProcessing=true", fillcolor="#e0f2f1"];
+ refresh [label="10s URL refresh\nfetchWatchData()", fillcolor="#e0f2f1"];
+ detect [label="instrumentals_url\nappears in JSON", shape=diamond, fillcolor="#fff8e1"];
+ autoswitch [label="Auto-switch to\ninstrumentals 🎤", fillcolor="#e8f5e9"];
+
+ click -> check;
+ check -> cycle [label="yes"];
+ check -> post [label="no (eligible)"];
+ post -> fork [label="200 {status: processing}"];
+ post -> hourglass;
+ fork -> process;
+ hourglass -> refresh [label="every 10s"];
+ refresh -> detect;
+ detect -> refresh [label="not yet"];
+ detect -> autoswitch [label="tracks ready"];
+ process -> detect [label="DB updated", style=dashed];
+}
+```
+
+## Security Properties
+
+- **Encryption at rest**: AES-256-CTR, unique key+IV per upload, key in ETS only
+- **Single-use**: Upload deleted immediately after pool pulls it
+- **TTL**: 15-minute expiry, swept every 60 seconds
+- **Key loss on restart**: BEAM restart = all encryption keys gone = files undecryptable
+- **Auth**: `X-Upload-Key` header + 128-bit random UUID per upload
+- **Zerotrust network**: Container has no outbound network access
+
+## Limits
+
+| Limit | Value | Source |
+|-------|-------|--------|
+| Max file size | 3.698 GB (3,698,742,051 bytes) | `upload_store.ex @max_upload_bytes` |
+| Memory per worker | ~64 KB | Disk-backed streaming at every stage |
+| Concurrency | Per-key limit from `validate_keys()` | Unsandbox account tier |
+| Retries | 3 attempts, exponential backoff (5s, 10s, 20s) | `karaoke.py` backfill |
+| URL TTL | 15 minutes | Presigned URL expiry |
+| Upload TTL | 15 minutes | API auto-cleanup |
+| Container TTL | 300 seconds | `/execute` payload `ttl` field |
+
+## S3 Key Structure
+
+```
+{shop_id}/products/{product_id}/
+ product # original media file
+ preview # preview (for sellable products)
+ thumbnail1-4 # thumbnails
+ instrumentals # karaoke: isolated instrumentals
+ vocals # karaoke: isolated vocals
+```
+
+## Product Metadata
+
+```python
+product.extensions = {
+ "product": "mp4", # original
+ "instrumentals": "mp4", # same ext for video, "wav" for audio
+ "vocals": "mp4",
+}
+product.file_bytes = {
+ "product": 15000000,
+ "instrumentals": 12000000,
+ "vocals": 8000000,
+}
+```
+
+## Related Files
+
+| File | Role |
+|------|------|
+| `lib/karaoke.py` | Core pipeline: download, upload, execute, process response |
+| `lib/voxsplit.c` | Spectral mid-side vocal isolation (C, libc+libm only) |
+| `lib/un.py` | Unsandbox API client, HMAC signing, key validation |
+| `views/watch.py` | On-demand `POST /karaoke/{id}` endpoint + watch JSON |
+| `views/product.py` | Upload-triggered karaoke processing |
+| `views/content.py` | Content page karaoke URL generation |
+| `static/js/watch.js` | Client: button states, on-demand trigger, auto-switch |
+| `templates/snippets/related_content.j2` | Karaoke button HTML |
+| `templates/content.j2` | `data-karaoke-eligible` attribute |
+| `scripts/backfill_karaoke.py` | Standalone backfill script |