make_post_sell/docs/tickets/mps-18.md

MPS-18: Karaoke Mode — Diagnose & Fix Vocal Isolation Pipeline

Status

BROKEN in production. Code paths exist (lib/karaoke.py, lib/voxsplit.c, scripts/backfill_karaoke.py, on-demand route POST /karaoke/{product_id}, backfill button in shop settings), but end-to-end runs do not produce playable instrumental + vocal tracks attached to a product. Watch mode toggle never finds populated karaoke URLs, so listeners never hear it.

Problem

Karaoke depends on a long chain — every link must hold:

1. Shop has unsandbox_public_key + unsandbox_secret_key configured
2. Source media reachable via presigned get_object from shop bucket
3. Source streams in 64KB chunks to unsandbox POST /upload (HMAC-signed against empty body because API skips body parsing for /upload)
4. voxsplit.c source uploaded the same way (compiled inside container)
5. POST /execute references both upload_ids, runs voxsplit in zerotrust container, returns instrumentals + vocals as base64 or upload_ids
6. Output streamed back, written to disk, uploaded to S3 at deterministic paths under karaoke/{product_id}/{instrumentals,vocals}.{ext}
7. Product row updated with karaoke URLs
8. Watch JSON + content/watch templates surface URLs to player.js
9. Player.js toggle swaps audio source between original and instrumentals

A failure at any link looks identical to a user: toggle does nothing.

Diagnosis Required

Before the fix lands, run a one-shot reproducible test against staging:

Probe	Expected	Actual
Shop has unsandbox creds set	yes	?
POST /upload with empty-body HMAC succeeds	200 + upload_id	?
POST /execute returns instrumentals + vocals	200 + bytes	?
S3 write to karaoke path succeeds	object exists in bucket	?
product.karaoke_instrumentals_url / _vocals_url columns populated	URL strings	?
Watch JSON includes karaoke_*_url keys	populated when eligible	?
Player toggle swaps audio src	swap occurs	?

Log every link with structured records (karaoke.step=upload_input, step=execute, step=mirror_output, step=db_write). A silent failure upstream is the worst kind — fail-loud at every step.

Proposal

1. Add karaoke health endpoint

GET /s/{shop_id}/karaoke-health — owner-only, returns JSON:

{
  "credentials_present": true,
  "api_reachable": true,
  "last_run_at": "2026-05-09T14:00:00Z",
  "last_run_status": "success|failed|never",
  "last_error": null,
  "products_with_karaoke": 12,
  "products_eligible": 47
}

Backed by a new MpsKaraokeRun model row written for every detached run.

2. Persist run state

New table mps_karaoke_run:

Column	Type	Purpose
id	UUID	row id
shop_id	UUID FK	shop
product_id	UUID FK nullable	product (null for backfill batch)
started_timestamp	int (ms)	run start
finished_timestamp	int (ms) nullable	run end
status	int	0=running 1=success 2=failed 3=cancelled
step	str	last completed step name
error_message	UnicodeText nullable	failure detail
bytes_in	int nullable	source size
bytes_out	int nullable	output size

Replaces fire-and-forget invisibility. Owner UI lists last 10 runs per shop.

3. Fix the most likely broken link first

Grep current lib/karaoke.py for:

• _sign_body called with empty string for /upload — confirm API still signs empty body for streaming endpoints. If API changed, signing is wrong.
• _resolve_credentials(shop, app) — confirm shop creds picked up before app fallback, BYOB-aware
• process_karaoke_detached double-fork — confirm child survives parent request teardown (no leaked DB sessions, no closed file handles)
• Output retrieval — confirm we pull from GET /internal/upload/{id} not from base64 in /execute response (memory blow-up on 3.6 GB inputs)

4. Wire on-demand auto-refresh

watch.js already polls every 10s while karaoke is running. Confirm it flips on completion. If polling is broken, add SSE or a single targeted refresh after MpsKaraokeRun.status flips to success.

5. Backfill button for shop owners

POST /s/{shop_id}/settings form_section=backfill-karaoke exists. Validate:

• Concurrency capped at unsandbox account's max_concurrent_executions
• Per-product idempotency — skip if karaoke_instrumentals_url populated AND mtime newer than source
• Backfill records one MpsKaraokeRun per product, not one per batch

Go-to-Market

Surface	Action
docs/karaoke-pipeline.md	Update with run-state model + health endpoint
docs/architecture.md	Add karaoke run table to feature toggle matrix
/styleguide	Karaoke health card pattern (status + last run + error)
~/git/www.makepostsell.com/index.html	Feature card: "Vocal isolation, on-demand or batched. No ML deps. Spectral mid-side Wiener masking."
~/git/www.makepostsell.com/pricing.html	Mention karaoke as included on all paid plans (depends on unsandbox account)
Shop settings UI	Health card next to backfill button — show current state without surprise

Files

File	Change
make_post_sell/models/karaoke_run.py	New: MpsKaraokeRun
make_post_sell/models/__init__.py	Import MpsKaraokeRun
make_post_sell/models/meta.py	Register in CLASS_TO_TABLE
make_post_sell/lib/karaoke.py	Write MpsKaraokeRun rows at every step; harden empty-body HMAC; verify creds at entry
make_post_sell/views/shop.py	karaoke_health route handler
make_post_sell/routes.py	Add /s/{shop_id}/karaoke-health
make_post_sell/templates/shop_settings.j2	Health card; recent runs list
make_post_sell/static/js/watch.js	Confirm 10s poll triggers reload on completion; fall back to single refresh on MpsKaraokeRun flip
make_post_sell/scripts/alembic/versions/XXXX_mps_karaoke_run_table.py	Migration with _table_exists guard
make_post_sell/tests/test_models.py	MpsKaraokeRun unit tests
make_post_sell/tests/test_integration.py	End-to-end run record write/read; backfill idempotency
make_post_sell/tests/test_functional.py	Health endpoint auth, JSON shape, settings page renders card

Tests

Unit (test_models.py)

• MpsKaraokeRun.is_running / is_success / is_failed properties
• MpsKaraokeRun.duration_ms calculated from start/finish timestamps
• Shop.karaoke_eligible returns True only when both unsandbox keys set
• Shop.karaoke_health_summary returns dict matching health endpoint shape

Integration (test_integration.py)

• Backfill skips products with fresh karaoke URLs (idempotency)
• Failed run leaves status=failed and populated error_message
• Concurrent runs respect shop unsandbox concurrency limit
• BYOB: outputs land in shop's bucket, not MPS default

Functional (test_functional.py)

• GET /s/{shop_id}/karaoke-health — anon 403, non-owner 403, owner 200 JSON
• Health JSON includes all keys above; numbers match seeded fixtures
• Shop settings page renders karaoke health card with correct current status
• POST /karaoke/{product_id} — anon 403, owner 200 + run row created
• Backfill button POST creates run rows for eligible products only

Verification

1. source vars.sh && make test — all pass
2. Local dev: configure unsandbox keys, upload an audio product, click karaoke button, watch run row write, watch URLs populate, watch player toggle work
3. Health endpoint reflects each step (running → success)
4. Force a failure (revoke creds mid-run) — status=failed, error captured
5. Push → CI green → deploy → check https://my.makepostsell.com/version
6. Bump GIT_HASH, push, verify