make_post_sell/docs/karaoke-pipeline.md

Karaoke Pipeline — Vocal Isolation via Unsandbox

Overview

MPS separates instrumentals and vocals from audio/video uploads using spectral mid-side Wiener masking (voxsplit.c, zero ML dependencies). Processing runs inside unsandbox zerotrust containers. The pipeline is disk-backed with ~64KB memory per worker at every stage.

Architecture Diagram

// Render: dot -Tsvg docs/karaoke-pipeline.dot -o docs/karaoke-pipeline.dot.svg
digraph karaoke_pipeline {
    rankdir=LR;
    node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
    edge [fontname="monospace", fontsize=9];

    subgraph cluster_mps {
        label="MPS (uWSGI)";
        style=dashed;
        color="#5871ad";

        trigger [label="Trigger\n(upload / on-demand / backfill)", fillcolor="#e8eaf6"];
        s3_down [label="S3 Download\n→ /tmp/karaoke_*/media.bin\n64KB chunks", fillcolor="#e8eaf6"];
        upload_media [label="POST /upload\n(streaming file body)\n→ upload_id", fillcolor="#e8eaf6"];
        upload_vox [label="POST /upload\nvoxsplit.c\n→ upload_id", fillcolor="#e8eaf6"];
        execute [label="POST /execute\n{upload_ids, script}\ntiny JSON, no file bytes", fillcolor="#e8eaf6"];
        stream_resp [label="Stream response\n→ /tmp/karaoke_*/response.json\n64KB chunks", fillcolor="#e8eaf6"];
        decode [label="Decode artifacts\n(base64 → tmpfile)\nupload to S3", fillcolor="#e8eaf6"];
        db_update [label="Update product\nextensions + file_bytes\nset S3 ACLs", fillcolor="#e8eaf6"];
    }

    subgraph cluster_api {
        label="api.unsandbox.com";
        style=dashed;
        color="#ad5871";

        recv_upload [label="Receive upload\nAES-256-CTR encrypt\n→ /tmp/uploads/{id}.enc\nkey in ETS only", fillcolor="#fce4ec"];
        dispatch [label="Dispatch execute\nErlang RPC\n~200B metadata only", fillcolor="#fce4ec"];
    }

    subgraph cluster_pool {
        label="Pool Node";
        style=dashed;
        color="#58ad71";

        pull [label="GET /internal/upload/{id}\nX-Upload-Key auth\nstreaming decrypt", fillcolor="#e8f5e9"];
        inject [label="lxc exec ... cat >\n/root/input/{filename}\n64KB streaming pipe", fillcolor="#e8f5e9"];
    }

    subgraph cluster_container {
        label="Zerotrust Container";
        style=dashed;
        color="#ad8f58";

        compile [label="gcc -O2 voxsplit.c -lm", fillcolor="#fff8e1"];
        extract [label="ffmpeg -i media\n-vn -acodec pcm_s16le\n-ar 44100 -ac 2\naudio.wav", fillcolor="#fff8e1"];
        split [label="voxsplit audio.wav\n→ split-instrumental.wav\n→ split-vocal.wav", fillcolor="#fff8e1"];
        remux [label="ffmpeg remux\n(video: copy video +\nisolated audio)\n(audio: copy wav)", fillcolor="#fff8e1"];
        artifacts [label="/tmp/artifacts/\ninstrumentals.{ext}\nvocals.{ext}", fillcolor="#fff8e1"];
    }

    s3 [label="S3 / CDN\n(DO Spaces or BYOB)", shape=cylinder, fillcolor="#f3e5f5"];
    browser [label="Browser\n(watch.js)", shape=ellipse, fillcolor="#e0f2f1"];

    trigger -> s3_down;
    s3_down -> upload_media [label="file on disk"];
    s3_down -> upload_vox;
    upload_media -> recv_upload [label="streaming\noctet-stream"];
    upload_vox -> recv_upload;
    recv_upload -> execute [label="upload_id", style=dashed, dir=back];
    execute -> dispatch [label="JSON\n{upload_ids}"];
    dispatch -> pull [label="RPC\nmetadata only"];
    pull -> recv_upload [label="HTTPS GET\nstreaming decrypt", style=dotted];
    pull -> inject;
    inject -> compile;
    compile -> extract;
    extract -> split;
    split -> remux;
    remux -> artifacts;
    artifacts -> stream_resp [label="response JSON\nbase64 artifacts", style=dotted];
    stream_resp -> decode;
    decode -> s3 [label="PUT instrumentals\nPUT vocals"];
    db_update -> s3 [label="ACL update", style=dashed];
    decode -> db_update;
    s3 -> browser [label="presigned URL\n15 min TTL"];
}

Pipeline Stages

Stage 1: Trigger

Three entry points, all converge on process_karaoke():

Entry Point	File	When
Upload	views/product.py:382	User uploads audio/video to product
On-demand	views/watch.py:karaoke_process	User clicks 🎤 button in watch mode
Backfill	lib/karaoke.py:backfill_karaoke_async	Shop owner triggers from settings

On-demand and backfill fork a detached grandchild process (double-fork + os.setsid()) that survives uWSGI worker recycling. Per-product lockfile (/tmp/karaoke_{product_id}.lock) prevents duplicate processing.

Stage 2: Download from S3

# karaoke.py:217-226
resp = s3_client.get_object(Bucket=bucket, Key=s3_key)
with open(media_path, "wb") as f:
    while True:
        chunk = body.read(65536)  # 64KB
        if not chunk: break
        f.write(chunk)

Stage 3: Upload to Unsandbox API

# karaoke.py:45-77
# HMAC signs empty body — API skips body parsing for /upload
headers = {"X-Filename": filename, "Content-Type": "application/octet-stream"}
with open(file_path, "rb") as body:
    response = requests.post(url, data=body, headers=headers)
# Returns: {"upload_id": "uuid"}

Two uploads: the media file + voxsplit.c source. Both stream from disk, constant memory.

Stage 4: API Receives + Encrypts

upload_store.ex writes incoming bytes to /tmp/uploads/{id}.enc with AES-256-CTR encryption. Unique key+IV per upload. Keys live only in ETS (BEAM memory, never written to disk). Files auto-expire after 15 minutes.

Stage 5: Execute (Metadata Only)

# karaoke.py:80-125
payload = {
    "language": "bash",
    "code": script,
    "network_mode": "zerotrust",
    "input_files": [
        {"upload_id": media_id, "filename": "media"},
        {"upload_id": voxsplit_id, "filename": "voxsplit.c"},
    ],
}

Tiny JSON body. Zero file content crosses Erlang distribution.

Stage 6: Pool Pulls Files

Pool node calls GET /internal/upload/{id} back to the API. API decrypts on-the-fly, streams plaintext. Pool pipes into container:

lxc exec {container} -- sh -c 'cat > /root/input/media'

/tmp/input symlinks to /root/input/ (canonical location).

Stage 7: Container Processing

# Compile voxsplit (spectral mid-side Wiener masking, libc + libm only)
gcc -O2 -o /tmp/voxsplit /tmp/input/voxsplit.c -lm

# Extract audio from media
ffmpeg -y -i /tmp/input/media -vn -acodec pcm_s16le -ar 44100 -ac 2 /tmp/audio.wav

# Separate instrumentals and vocals
/tmp/voxsplit /tmp/audio.wav -o /tmp/split

# For video: remux original video with each isolated audio
ffmpeg -y -i /tmp/input/media -i /tmp/split-instrumental.wav \
    -c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/instrumentals.mp4
ffmpeg -y -i /tmp/input/media -i /tmp/split-vocal.wav \
    -c:v copy -map 0:v -map 1:a -shortest /tmp/artifacts/vocals.mp4

Stage 8: Response + S3 Upload

Response JSON streams back to MPS temp file. Artifacts are base64-encoded in the response. MPS decodes each one to a temp file, uploads to S3, deletes temp file. Updates product metadata and S3 ACLs.

On-Demand Flow (Watch Mode)

// Render: dot -Tsvg docs/karaoke-ondemand.dot -o docs/karaoke-ondemand.dot.svg
digraph karaoke_ondemand {
    rankdir=TB;
    node [shape=box, style="rounded,filled", fontname="monospace", fontsize=10];
    edge [fontname="monospace", fontsize=9];

    click [label="User clicks 🎤\n(watch.js)", fillcolor="#e0f2f1"];
    check [label="Has tracks?", shape=diamond, fillcolor="#fff8e1"];
    cycle [label="cycleKaraoke()\nOriginal → Instrumentals → Vocals", fillcolor="#e8f5e9"];
    post [label="POST /karaoke/{id}\n(fetch, non-blocking)", fillcolor="#e8eaf6"];
    fork [label="Server forks\ndetached grandchild", fillcolor="#e8eaf6"];
    process [label="process_karaoke()\n(~30-120s)", fillcolor="#fce4ec"];
    hourglass [label="Button shows ⌛\nkaraokeProcessing=true", fillcolor="#e0f2f1"];
    refresh [label="10s URL refresh\nfetchWatchData()", fillcolor="#e0f2f1"];
    detect [label="instrumentals_url\nappears in JSON", shape=diamond, fillcolor="#fff8e1"];
    autoswitch [label="Auto-switch to\ninstrumentals 🎤", fillcolor="#e8f5e9"];

    click -> check;
    check -> cycle [label="yes"];
    check -> post [label="no (eligible)"];
    post -> fork [label="200 {status: processing}"];
    post -> hourglass;
    fork -> process;
    hourglass -> refresh [label="every 10s"];
    refresh -> detect;
    detect -> refresh [label="not yet"];
    detect -> autoswitch [label="tracks ready"];
    process -> detect [label="DB updated", style=dashed];
}

Security Properties

• Encryption at rest: AES-256-CTR, unique key+IV per upload, key in ETS only
• Single-use: Upload deleted immediately after pool pulls it
• TTL: 15-minute expiry, swept every 60 seconds
• Key loss on restart: BEAM restart = all encryption keys gone = files undecryptable
• Auth: X-Upload-Key header + 128-bit random UUID per upload
• Zerotrust network: Container has no outbound network access

Limits

Limit	Value	Source
Max file size	3.698 GB (3,698,742,051 bytes)	upload_store.ex @max_upload_bytes
Memory per worker	~64 KB	Disk-backed streaming at every stage
Concurrency	Per-key limit from validate_keys()	Unsandbox account tier
Retries	3 attempts, exponential backoff (5s, 10s, 20s)	karaoke.py backfill
URL TTL	15 minutes	Presigned URL expiry
Upload TTL	15 minutes	API auto-cleanup
Container TTL	300 seconds	/execute payload ttl field

S3 Key Structure

{shop_id}/products/{product_id}/
    product              # original media file
    preview              # preview (for sellable products)
    thumbnail1-4         # thumbnails
    instrumentals        # karaoke: isolated instrumentals
    vocals               # karaoke: isolated vocals

Product Metadata

product.extensions = {
    "product": "mp4",           # original
    "instrumentals": "mp4",     # same ext for video, "wav" for audio
    "vocals": "mp4",
}
product.file_bytes = {
    "product": 15000000,
    "instrumentals": 12000000,
    "vocals": 8000000,
}

Related Files

File	Role
lib/karaoke.py	Core pipeline: download, upload, execute, process response
lib/voxsplit.c	Spectral mid-side vocal isolation (C, libc+libm only)
lib/un.py	Unsandbox API client, HMAC signing, key validation
views/watch.py	On-demand POST /karaoke/{id} endpoint + watch JSON
views/product.py	Upload-triggered karaoke processing
views/content.py	Content page karaoke URL generation
static/js/watch.js	Client: button states, on-demand trigger, auto-switch
templates/snippets/related_content.j2	Karaoke button HTML
templates/content.j2	data-karaoke-eligible attribute
scripts/backfill_karaoke.py	Standalone backfill script