diff --git a/CLAUDE.md b/CLAUDE.md index 946158e..94b4231 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,45 +39,41 @@ **CRITICAL: NEVER use ssh, scp, or rsync to access remote servers.** Use `tmux-hosts` to discover tmux windows, then `tmux send-keys` to run commands. +Window numbers shift; never hardcode them. Always discover first. ```bash -# Discover available hosts tmux-hosts -# Output: -# 0:0 ai.foxhop.net -# 0:1 3090-ai.foxhop.net +# example output (yours WILL differ — verify every session): +# 0:0 3090-ai.foxhop.net +# 0:2 ai.foxhop.net # 4090, current speech prod -# Run a command on 3090-ai -tmux send-keys -t 0:1 'command here' Enter - -# Read output -tmux capture-pane -t 0:1 -p | tail -20 +tmux send-keys -t 0:2 'command here' Enter +tmux capture-pane -t 0:2 -p | tail -20 ``` -- **NEVER** use `ssh user@host "command"` -- use `tmux send-keys -t 0:1` -- **NEVER** use `rsync` or `scp` -- use `git push` then `tmux send-keys -t 0:1 'git pull' Enter` -- **ALWAYS** discover our correct window with `tmux-hosts` first +- **NEVER** use `ssh user@host "command"` — use `tmux send-keys` +- **NEVER** use `rsync` or `scp` — use `git push` + `tmux send-keys '... git pull ...' Enter` +- **ALWAYS** rerun `tmux-hosts` to confirm window numbers before sending keys ### 3. Git-Based Deployment **We use git, not rsync/scp.** All code syncs via git push/pull. ```bash -# Deploy workflow: -# 1. Commit and push locally +# 1. local: commit and push git add files && git commit -m "message" && git push -# 2. Pull and rebuild on server via tmux -tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker compose up -d --build' Enter +# 2. remote: pull + restart speech.py (see Production Deployment below for the relaunch) +tmux send-keys -t 'cd /mnt/data/git/uncloseai-speech && sudo -u fox git pull' Enter ``` ### 4. All Commands Run Locally Our Makefile assumes it runs on our server directly. No remote execution. -When you need to run make targets on our server, use tmux: +When you need to run a make target on a remote host, use tmux: ```bash -tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && make deploy' Enter +tmux send-keys -t 'cd /mnt/data/git/uncloseai-speech && make ' Enter ``` ### 5. Configuration Management @@ -93,119 +89,113 @@ tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && make deploy' Enter ## Production Deployment -**Production Server:** `3090-ai.foxhop.net` (tmux window `0:1`) -- **URL:** https://speech.ai.unturf.com -- **Repo Location:** `/home/fox/git/uncloseai-speech` -- **Container:** `uncloseai-speech-server-1` -- **Git remote:** `ssh://git@git.unturf.com:2222/engineering/unturf/uncloseai-speech.git` -- **Git user on server:** `timehexon ` +Prod runs as a **bare python process** on a 4090 host (`ai.foxhop.net`) — no Docker. +Caddy on 80/443 reverse-proxies https://speech.ai.unturf.com to local port 8000. -### Deployment Workflow +| | | +|---|---| +| Host | `ai.foxhop.net` (4090). Find current tmux window with `tmux-hosts`. | +| Run user | `fox` | +| Repo | `/mnt/data/git/uncloseai-speech` (symlinked from `/home/fox/git/uncloseai-speech`) | +| Venv | `/mnt/data/f5-sidecar/venv/` | +| Logs | `/mnt/data/f5-sidecar/logs/speech.log` | +| Launch cmd | `python speech.py --workers 1 --port 8000 --log-level INFO` | +| Frontend | Caddy → `https://speech.ai.unturf.com` | +| Git remote | `ssh://git@git.unturf.com:2222/engineering/unturf/uncloseai-speech.git` | +| Git user on server | `fox` | -``` -Local: - /home/fox/git/uncloseai-speech/ +Docker compose files (`docker-compose.yml`, `Dockerfile`) are kept for parity with +self-hosted deploys, but our prod does NOT use them. - git commit && git push - -Remote (3090-ai, tmux 0:1): - /home/fox/git/uncloseai-speech/ - - git pull && sudo docker compose up -d --build - -Container: - /app/ - ├── speech.py - ├── cloned-voices/ (mounted from host) - ├── voices/ (mounted from host) - └── config/ - └── voice_to_speaker.yaml -``` - -### Quick Production Commands +### Deploy a code/config change ```bash -# Discover tmux windows -tmux-hosts - -# Deploy changes +# 1. local git push -tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker compose up -d --build' Enter -# Check container status -tmux send-keys -t 0:1 'sudo docker ps | grep speech' Enter +# 2. remote (find window via tmux-hosts, then): +tmux send-keys -t <4090-window> '\ + cd /mnt/data/git/uncloseai-speech && \ + sudo -u fox git pull && \ + PID=$(sudo ss -tlnp | awk "/:8000 / {print \$NF}" | sed "s/.*pid=\([0-9]*\),.*/\1/") && \ + echo "killing PID $PID" && sudo -u fox kill $PID && \ + until ! sudo ss -tln | grep -q :8000; do sleep 1; done && \ + cd /mnt/data/git/uncloseai-speech && \ + sudo -u fox bash -c "nohup env PYTHONUNBUFFERED=1 /mnt/data/f5-sidecar/venv/bin/python speech.py --workers 1 --port 8000 --log-level INFO > /mnt/data/f5-sidecar/logs/speech.log 2>&1 &" && \ + until sudo ss -tln | grep -q :8000; do sleep 2; done && echo PORT_BOUND \ +' Enter +``` -# View logs -tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech-server-1' Enter +Audio yaml / `cloned-voices/` changes don't need a code change — same restart picks them up +(speech.py loads yaml at startup). -# Restart container -tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && sudo docker compose restart' Enter +### Quick checks -# Read output from tmux -tmux capture-pane -t 0:1 -p | tail -20 +```bash +# is prod up? +curl -sS -o /dev/null -w "%{http_code}\n" https://speech.ai.unturf.com/v1/voices + +# what PID is bound to :8000? +tmux send-keys -t <4090-window> 'sudo ss -tlnp | grep :8000' Enter + +# tail prod log +tmux send-keys -t <4090-window> 'tail -50 /mnt/data/f5-sidecar/logs/speech.log' Enter + +# GPU state +tmux send-keys -t <4090-window> 'nvidia-smi' Enter ``` ## Voice Configuration -21 distinct gendered voices from LibriSpeech test-clean (public domain): -- **Female (11):** aria, clara, elena, grace, hazel, iris, luna, maya, ruby, sage, sofia -- **Male (10):** atlas, caleb, felix, hugo, jasper, kai, leo, marcus, owen, theo +40 distinct gendered voices from LibriSpeech test-clean (public domain). +Roster lives in `cloned-voices/voices_metadata.json`. -Voice WAV files are in `cloned-voices/`, mounted into our container. -Config is in `voice_to_speaker.default.yaml`. +Voice WAV files in `cloned-voices/` are read directly by `speech.py` (no container mount — +prod runs bare). Engine→voice mapping in `voice_to_speaker.default.yaml`. + +F5-TTS `ref_text` values are generated by `make whisper-refs` (whisper-large-v3 over +the actual wav files) — they match what the audio sounds like, not LibriSpeech ground-truth +labels. Regenerate any time `cloned-voices/` changes. ## TTS Engine Status -### Default Model (Qwen3-TTS) -- **Qwen3-TTS (tts-1-qwen)** - DEFAULT - 1.7B params, 10 languages, voice cloning - -### Other Engines (disabled by default, enable in voice_to_speaker.yaml) -- Piper TTS (tts-1) - 55 voices, fast CPU inference -- XTTS v2 (tts-1-hd) - Voice cloning, multilingual -- Silero TTS (tts-1-silero) - 142 voices, 5 languages, auto-downloads -- Kokoro TTS (tts-1-kokoro) - 32 voices, lightweight (82M params) - -See `docs/MODELS.md` for complete roadmap. +`tts-1-f5` (F5-TTS) is what prod serves today. `tts-1-qwen` (Qwen3-TTS) is also wired up. +Other engines (Piper, XTTS, Silero, Kokoro, Chatterbox) live in `voice_to_speaker.default.yaml` +but are disabled by default. See `docs/MODELS.md` for the full roadmap. ## Testing ```bash -# Run on server via tmux -tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && make test' Enter - -# Or test from any machine with curl -curl -X POST http://3090-ai.foxhop.net:8000/v1/audio/speech \ +# end-to-end via the prod URL (always-correct) +curl -sS -X POST https://speech.ai.unturf.com/v1/audio/speech \ -H "Content-Type: application/json" \ - -d '{"model":"tts-1-qwen","voice":"aria","input":"Test"}' \ - -o /tmp/test.mp3 + -d '{"model":"tts-1-f5","voice":"aria","input":"Smoke test."}' \ + -o /tmp/test.mp3 -w "http=%{http_code} size=%{size_download}\n" + +# on the prod host directly (via tmux), bypassing Caddy +tmux send-keys -t <4090-window> 'cd /mnt/data/git/uncloseai-speech && make test-f5' Enter ``` ## When Things Break ```bash -# Check logs -tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech-server-1' Enter +# tail prod log +tmux send-keys -t <4090-window> 'tail -100 /mnt/data/f5-sidecar/logs/speech.log' Enter -# Check GPU memory -tmux send-keys -t 0:1 'nvidia-smi' Enter +# GPU memory +tmux send-keys -t <4090-window> 'nvidia-smi' Enter -# Restart container -tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && sudo docker compose restart' Enter - -# Full rebuild -tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker compose up -d --build' Enter - -# Read tmux output -tmux capture-pane -t 0:1 -p | tail -20 +# kill + relaunch (see "Deploy a code/config change" above for the full sequence) ``` ## Common Mistakes to Avoid -- DON'T use `ssh user@host "command"` -- use `tmux send-keys -t 0:1` -- DON'T use `rsync` or `scp` -- use `git push` + `tmux send-keys -t 0:1 'git pull' Enter` -- DON'T hardcode tmux window numbers -- use `tmux-hosts` to discover them -- DON'T put scripts in docs/ -- put them in scripts/ +- DON'T use `ssh user@host "command"` — use `tmux send-keys` +- DON'T use `rsync` or `scp` — use `git push` + tmux `git pull` +- DON'T hardcode tmux window numbers — rediscover via `tmux-hosts` every session +- DON'T put scripts in docs/ — put them in scripts/ - DON'T forget to push before deploying +- DON'T assume prod uses docker — it's bare python; the compose files are for self-hosted parity only ## AGPL v3 Compliance