engineering/make_post_sell
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions

docs: karaoke pipeline architecture with dot diagrams

Browse source 
Add docs/karaoke-pipeline.md covering the full streaming pipeline from
MPS through unsandbox API to zerotrust container and back. Includes two
Graphviz dot diagrams (rendered to SVG):

- karaoke-pipeline.dot: full system flow across MPS, API, pool, container
- karaoke-ondemand.dot: watch mode on-demand user flow

Update architecture.md feature toggle matrix and related docs table.
Update CLAUDE.md karaoke section with streaming path and on-demand info.
This commit is contained in:
russell@unturf.com 2026-03-11 17:49:57 -04:00
parent 6fd6bfd756
commit 8649e6aaae
7 changed files with 887 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

15
CLAUDE.md
View file

@ -29,15 +29,20 @@ Files are NEVER streamed through uwsgi. The server only generates presigned URLs
### Karaoke Pipeline (lib/karaoke.py)
Disk-backed vocal isolation pipeline. Downloads media from S3, builds a JSON
request on disk with streaming base64 encoding, POSTs to unsandbox API, streams
response to disk, decodes artifacts one at a time and uploads to S3.
Disk-backed vocal isolation pipeline using spectral mid-side Wiener masking
(`voxsplit.c`, zero ML deps). Streams media to unsandbox via `POST /upload`
(64KB chunks, constant memory), executes in zerotrust container, streams
response back, uploads instrumentals + vocals to S3.
Full architecture doc: `docs/karaoke-pipeline.md` (with dot diagrams).
- **Concurrency**: `ThreadPoolExecutor` sized to account's unsandbox concurrency limit
- **Memory**: ~64KB per worker on the request side (disk-backed, not in-memory)
- **Memory**: ~64KB per worker at every stage (disk-backed, not in-memory)
- **Upstream limit**: 3.698GB / 3,698,742,051 bytes per file (unsandbox `@max_upload_bytes`)
- **Retries**: 3 attempts with exponential backoff (5s, 10s)
- **Callers**: `views/product.py`, `views/shop.py` (backfill), `scripts/backfill_karaoke.py`
- **Callers**: `views/product.py` (upload), `views/watch.py` (on-demand), `views/shop.py` (backfill), `scripts/backfill_karaoke.py`
- **On-demand**: `POST /karaoke/{product_id}` — forks detached child, watch.js 10s refresh detects completion, auto-switches to instrumentals
- **Streaming path**: MPS → `POST /upload` → API encrypts to disk → pool pulls via `GET /internal/upload/{id}` → pipes into container `/root/input/` — zero bytes cross Erlang distribution
## Project Setup