# MPS-18: Karaoke Mode — Diagnose & Fix Vocal Isolation Pipeline

## Status

**BROKEN in production.** Code paths exist (`lib/karaoke.py`, `lib/voxsplit.c`,
`scripts/backfill_karaoke.py`, on-demand route `POST /karaoke/{product_id}`,
backfill button in shop settings), but end-to-end runs do not produce playable
instrumental + vocal tracks attached to a product. Watch mode toggle never
finds populated karaoke URLs, so listeners never hear it.

## Problem

Karaoke depends on a long chain — every link must hold:

1. Shop has `unsandbox_public_key` + `unsandbox_secret_key` configured
2. Source media reachable via presigned `get_object` from shop bucket
3. Source streams in 64KB chunks to unsandbox `POST /upload` (HMAC-signed
   against **empty body** because API skips body parsing for `/upload`)
4. `voxsplit.c` source uploaded the same way (compiled inside container)
5. `POST /execute` references both `upload_id`s, runs voxsplit in zerotrust
   container, returns instrumentals + vocals as base64 or upload_ids
6. Output streamed back, written to disk, uploaded to S3 at deterministic
   paths under `karaoke/{product_id}/{instrumentals,vocals}.{ext}`
7. Product row updated with karaoke URLs
8. Watch JSON + content/watch templates surface URLs to player.js
9. Player.js toggle swaps audio source between original and instrumentals

A failure at any link looks identical to a user: toggle does nothing.

## Diagnosis Required

Before the fix lands, run a one-shot reproducible test against staging:

| Probe | Expected | Actual |
|-------|----------|--------|
| Shop has unsandbox creds set | yes | ? |
| `POST /upload` with empty-body HMAC succeeds | 200 + `upload_id` | ? |
| `POST /execute` returns instrumentals + vocals | 200 + bytes | ? |
| S3 write to karaoke path succeeds | object exists in bucket | ? |
| `product.karaoke_instrumentals_url` / `_vocals_url` columns populated | URL strings | ? |
| Watch JSON includes `karaoke_*_url` keys | populated when eligible | ? |
| Player toggle swaps audio src | swap occurs | ? |

Log every link with structured records (`karaoke.step=upload_input`,
`step=execute`, `step=mirror_output`, `step=db_write`). A silent failure
upstream is the worst kind — fail-loud at every step.

## Proposal

### 1. Add karaoke health endpoint

`GET /s/{shop_id}/karaoke-health` — owner-only, returns JSON:

```json
{
  "credentials_present": true,
  "api_reachable": true,
  "last_run_at": "2026-05-09T14:00:00Z",
  "last_run_status": "success|failed|never",
  "last_error": null,
  "products_with_karaoke": 12,
  "products_eligible": 47
}
```

Backed by a new `MpsKaraokeRun` model row written for every detached run.

### 2. Persist run state

New table `mps_karaoke_run`:

| Column | Type | Purpose |
|--------|------|---------|
| `id` | UUID | row id |
| `shop_id` | UUID FK | shop |
| `product_id` | UUID FK nullable | product (null for backfill batch) |
| `started_timestamp` | int (ms) | run start |
| `finished_timestamp` | int (ms) nullable | run end |
| `status` | int | 0=running 1=success 2=failed 3=cancelled |
| `step` | str | last completed step name |
| `error_message` | UnicodeText nullable | failure detail |
| `bytes_in` | int nullable | source size |
| `bytes_out` | int nullable | output size |

Replaces fire-and-forget invisibility. Owner UI lists last 10 runs per shop.

### 3. Fix the most likely broken link first

Grep current `lib/karaoke.py` for:

- `_sign_body` called with empty string for `/upload` — confirm API still
  signs empty body for streaming endpoints. If API changed, signing is wrong.
- `_resolve_credentials(shop, app)` — confirm shop creds picked up before
  app fallback, BYOB-aware
- `process_karaoke_detached` double-fork — confirm child survives parent
  request teardown (no leaked DB sessions, no closed file handles)
- Output retrieval — confirm we pull from `GET /internal/upload/{id}` not
  from base64 in `/execute` response (memory blow-up on 3.6 GB inputs)

### 4. Wire on-demand auto-refresh

`watch.js` already polls every 10s while karaoke is running. Confirm it
flips on completion. If polling is broken, add SSE or a single targeted
refresh after `MpsKaraokeRun.status` flips to success.

### 5. Backfill button for shop owners

`POST /s/{shop_id}/settings` `form_section=backfill-karaoke` exists.
Validate:

- Concurrency capped at unsandbox account's `max_concurrent_executions`
- Per-product idempotency — skip if `karaoke_instrumentals_url` populated
  AND mtime newer than source
- Backfill records one `MpsKaraokeRun` per product, not one per batch

## Go-to-Market

| Surface | Action |
|---------|--------|
| `docs/karaoke-pipeline.md` | Update with run-state model + health endpoint |
| `docs/architecture.md` | Add karaoke run table to feature toggle matrix |
| `/styleguide` | Karaoke health card pattern (status + last run + error) |
| `~/git/www.makepostsell.com/index.html` | Feature card: "Vocal isolation, on-demand or batched. No ML deps. Spectral mid-side Wiener masking." |
| `~/git/www.makepostsell.com/pricing.html` | Mention karaoke as included on all paid plans (depends on unsandbox account) |
| Shop settings UI | Health card next to backfill button — show current state without surprise |

## Files

| File | Change |
|------|--------|
| `make_post_sell/models/karaoke_run.py` | New: `MpsKaraokeRun` |
| `make_post_sell/models/__init__.py` | Import `MpsKaraokeRun` |
| `make_post_sell/models/meta.py` | Register in `CLASS_TO_TABLE` |
| `make_post_sell/lib/karaoke.py` | Write `MpsKaraokeRun` rows at every step; harden empty-body HMAC; verify creds at entry |
| `make_post_sell/views/shop.py` | `karaoke_health` route handler |
| `make_post_sell/routes.py` | Add `/s/{shop_id}/karaoke-health` |
| `make_post_sell/templates/shop_settings.j2` | Health card; recent runs list |
| `make_post_sell/static/js/watch.js` | Confirm 10s poll triggers reload on completion; fall back to single refresh on `MpsKaraokeRun` flip |
| `make_post_sell/scripts/alembic/versions/XXXX_mps_karaoke_run_table.py` | Migration with `_table_exists` guard |
| `make_post_sell/tests/test_models.py` | `MpsKaraokeRun` unit tests |
| `make_post_sell/tests/test_integration.py` | End-to-end run record write/read; backfill idempotency |
| `make_post_sell/tests/test_functional.py` | Health endpoint auth, JSON shape, settings page renders card |

## Tests

### Unit (`test_models.py`)

- `MpsKaraokeRun.is_running` / `is_success` / `is_failed` properties
- `MpsKaraokeRun.duration_ms` calculated from start/finish timestamps
- `Shop.karaoke_eligible` returns `True` only when both unsandbox keys set
- `Shop.karaoke_health_summary` returns dict matching health endpoint shape

### Integration (`test_integration.py`)

- Backfill skips products with fresh karaoke URLs (idempotency)
- Failed run leaves `status=failed` and populated `error_message`
- Concurrent runs respect shop unsandbox concurrency limit
- BYOB: outputs land in shop's bucket, not MPS default

### Functional (`test_functional.py`)

- `GET /s/{shop_id}/karaoke-health` — anon 403, non-owner 403, owner 200 JSON
- Health JSON includes all keys above; numbers match seeded fixtures
- Shop settings page renders karaoke health card with correct current status
- `POST /karaoke/{product_id}` — anon 403, owner 200 + run row created
- Backfill button POST creates run rows for eligible products only

## Verification

1. `source vars.sh && make test` — all pass
2. Local dev: configure unsandbox keys, upload an audio product, click karaoke
   button, watch run row write, watch URLs populate, watch player toggle work
3. Health endpoint reflects each step (running → success)
4. Force a failure (revoke creds mid-run) — `status=failed`, error captured
5. Push → CI green → deploy → check `https://my.makepostsell.com/version`
6. Bump GIT_HASH, push, verify
