engineering/uncloseai-speech
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CLAUDE.md: replace stale 3090-ai+docker prod docs with 4090+bare-python reality

Browse source 
The prior docs claimed prod = 3090-ai running uncloseai-speech-server-1 via
docker compose. Reality (verified today): prod is bare python on the 4090
(ai.foxhop.net) using the f5-sidecar venv at /mnt/data/f5-sidecar/venv,
fronted by Caddy on 80/443. The container on 3090-ai exited 2 months ago and
docker compose v2 isn't installed there.

Cost of the drift today: one wrong-host deploy attempt that bounced off dead
infrastructure. This rewrites Production Deployment, Quick Production
Commands, Testing, When Things Break, and Common Mistakes to match reality.

Also drops hardcoded tmux window numbers (they shift) — every example now
reads <prod-window> with a reminder to rediscover via tmux-hosts.
This commit is contained in:
russell@unturf.com 2026-05-24 16:02:08 -04:00
parent 4ac192585d
commit 72a9553047
No known key found for this signature in database
1 changed files with 86 additions and 96 deletions
Download patch file Download diff file Expand all files Collapse all files

182
CLAUDE.md
View file

@ -39,45 +39,41 @@
**CRITICAL: NEVER use ssh, scp, or rsync to access remote servers.**
Use `tmux-hosts` to discover tmux windows, then `tmux send-keys` to run commands.
Window numbers shift; never hardcode them. Always discover first.
```bash
# Discover available hosts
tmux-hosts
# Output:
# 0:0 ai.foxhop.net
# 0:1 3090-ai.foxhop.net
# example output (yours WILL differ — verify every session):
# 0:0 3090-ai.foxhop.net
# 0:2 ai.foxhop.net # 4090, current speech prod
# Run a command on 3090-ai
tmux send-keys -t 0:1 'command here' Enter
# Read output
tmux capture-pane -t 0:1 -p | tail -20
tmux send-keys -t 0:2 'command here' Enter
tmux capture-pane -t 0:2 -p | tail -20
```
- **NEVER** use `ssh user@host "command"` -- use `tmux send-keys -t 0:1`
- **NEVER** use `rsync` or `scp` -- use `git push` then `tmux send-keys -t 0:1 'git pull' Enter`
- **ALWAYS** discover our correct window with `tmux-hosts` first
- **NEVER** use `ssh user@host "command"` — use `tmux send-keys`
- **NEVER** use `rsync` or `scp` — use `git push` + `tmux send-keys '... git pull ...' Enter`
- **ALWAYS** rerun `tmux-hosts` to confirm window numbers before sending keys
### 3. Git-Based Deployment
**We use git, not rsync/scp.** All code syncs via git push/pull.
```bash
# Deploy workflow:
# 1. Commit and push locally
# 1. local: commit and push
git add files && git commit -m "message" && git push
# 2. Pull and rebuild on server via tmux
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker compose up -d --build' Enter
# 2. remote: pull + restart speech.py (see Production Deployment below for the relaunch)
tmux send-keys -t <prod-window> 'cd /mnt/data/git/uncloseai-speech && sudo -u fox git pull' Enter
```
### 4. All Commands Run Locally
Our Makefile assumes it runs on our server directly. No remote execution.
When you need to run make targets on our server, use tmux:
When you need to run a make target on a remote host, use tmux:
```bash
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && make deploy' Enter
tmux send-keys -t <prod-window> 'cd /mnt/data/git/uncloseai-speech && make <target>' Enter
```
### 5. Configuration Management
@ -93,119 +89,113 @@ tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && make deploy' Enter
## Production Deployment
**Production Server:** `3090-ai.foxhop.net` (tmux window `0:1`)
- **URL:** https://speech.ai.unturf.com
- **Repo Location:** `/home/fox/git/uncloseai-speech`
- **Container:** `uncloseai-speech-server-1`
- **Git remote:** `ssh://git@git.unturf.com:2222/engineering/unturf/uncloseai-speech.git`
- **Git user on server:** `timehexon <timehexon@unturf.com>`
Prod runs as a **bare python process** on a 4090 host (`ai.foxhop.net`) — no Docker.
Caddy on 80/443 reverse-proxies https://speech.ai.unturf.com to local port 8000.
### Deployment Workflow
| | |
|---|---|
| Host | `ai.foxhop.net` (4090). Find current tmux window with `tmux-hosts`. |
| Run user | `fox` |
| Repo | `/mnt/data/git/uncloseai-speech` (symlinked from `/home/fox/git/uncloseai-speech`) |
| Venv | `/mnt/data/f5-sidecar/venv/` |
| Logs | `/mnt/data/f5-sidecar/logs/speech.log` |
| Launch cmd | `python speech.py --workers 1 --port 8000 --log-level INFO` |
| Frontend | Caddy → `https://speech.ai.unturf.com` |
| Git remote | `ssh://git@git.unturf.com:2222/engineering/unturf/uncloseai-speech.git` |
| Git user on server | `fox` |
```
Local:
/home/fox/git/uncloseai-speech/
Docker compose files (`docker-compose.yml`, `Dockerfile`) are kept for parity with
self-hosted deploys, but our prod does NOT use them.
git commit && git push
Remote (3090-ai, tmux 0:1):
/home/fox/git/uncloseai-speech/
git pull && sudo docker compose up -d --build
Container:
/app/
├── speech.py
├── cloned-voices/ (mounted from host)
├── voices/ (mounted from host)
└── config/
└── voice_to_speaker.yaml
```
### Quick Production Commands
### Deploy a code/config change
```bash
# Discover tmux windows
tmux-hosts
# Deploy changes
# 1. local
git push
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker compose up -d --build' Enter
# Check container status
tmux send-keys -t 0:1 'sudo docker ps | grep speech' Enter
# 2. remote (find window via tmux-hosts, then):
tmux send-keys -t <4090-window> '\
cd /mnt/data/git/uncloseai-speech && \
sudo -u fox git pull && \
PID=$(sudo ss -tlnp | awk "/:8000 / {print \$NF}" | sed "s/.*pid=\([0-9]*\),.*/\1/") && \
echo "killing PID $PID" && sudo -u fox kill $PID && \
until ! sudo ss -tln | grep -q :8000; do sleep 1; done && \
cd /mnt/data/git/uncloseai-speech && \
sudo -u fox bash -c "nohup env PYTHONUNBUFFERED=1 /mnt/data/f5-sidecar/venv/bin/python speech.py --workers 1 --port 8000 --log-level INFO > /mnt/data/f5-sidecar/logs/speech.log 2>&1 &" && \
until sudo ss -tln | grep -q :8000; do sleep 2; done && echo PORT_BOUND \
' Enter
```
# View logs
tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech-server-1' Enter
Audio yaml / `cloned-voices/` changes don't need a code change — same restart picks them up
(speech.py loads yaml at startup).
# Restart container
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && sudo docker compose restart' Enter
### Quick checks
# Read output from tmux
tmux capture-pane -t 0:1 -p | tail -20
```bash
# is prod up?
curl -sS -o /dev/null -w "%{http_code}\n" https://speech.ai.unturf.com/v1/voices
# what PID is bound to :8000?
tmux send-keys -t <4090-window> 'sudo ss -tlnp | grep :8000' Enter
# tail prod log
tmux send-keys -t <4090-window> 'tail -50 /mnt/data/f5-sidecar/logs/speech.log' Enter
# GPU state
tmux send-keys -t <4090-window> 'nvidia-smi' Enter
```
## Voice Configuration
21 distinct gendered voices from LibriSpeech test-clean (public domain):
- **Female (11):** aria, clara, elena, grace, hazel, iris, luna, maya, ruby, sage, sofia
- **Male (10):** atlas, caleb, felix, hugo, jasper, kai, leo, marcus, owen, theo
40 distinct gendered voices from LibriSpeech test-clean (public domain).
Roster lives in `cloned-voices/voices_metadata.json`.
Voice WAV files are in `cloned-voices/`, mounted into our container.
Config is in `voice_to_speaker.default.yaml`.
Voice WAV files in `cloned-voices/` are read directly by `speech.py` (no container mount —
prod runs bare). Engine→voice mapping in `voice_to_speaker.default.yaml`.
F5-TTS `ref_text` values are generated by `make whisper-refs` (whisper-large-v3 over
the actual wav files) — they match what the audio sounds like, not LibriSpeech ground-truth
labels. Regenerate any time `cloned-voices/` changes.
## TTS Engine Status
### Default Model (Qwen3-TTS)
- **Qwen3-TTS (tts-1-qwen)** - DEFAULT - 1.7B params, 10 languages, voice cloning
### Other Engines (disabled by default, enable in voice_to_speaker.yaml)
- Piper TTS (tts-1) - 55 voices, fast CPU inference
- XTTS v2 (tts-1-hd) - Voice cloning, multilingual
- Silero TTS (tts-1-silero) - 142 voices, 5 languages, auto-downloads
- Kokoro TTS (tts-1-kokoro) - 32 voices, lightweight (82M params)
See `docs/MODELS.md` for complete roadmap.
`tts-1-f5` (F5-TTS) is what prod serves today. `tts-1-qwen` (Qwen3-TTS) is also wired up.
Other engines (Piper, XTTS, Silero, Kokoro, Chatterbox) live in `voice_to_speaker.default.yaml`
but are disabled by default. See `docs/MODELS.md` for the full roadmap.
## Testing
```bash
# Run on server via tmux
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && make test' Enter
# Or test from any machine with curl
curl -X POST http://3090-ai.foxhop.net:8000/v1/audio/speech \
# end-to-end via the prod URL (always-correct)
curl -sS -X POST https://speech.ai.unturf.com/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model":"tts-1-qwen","voice":"aria","input":"Test"}' \
-o /tmp/test.mp3
-d '{"model":"tts-1-f5","voice":"aria","input":"Smoke test."}' \
-o /tmp/test.mp3 -w "http=%{http_code} size=%{size_download}\n"
# on the prod host directly (via tmux), bypassing Caddy
tmux send-keys -t <4090-window> 'cd /mnt/data/git/uncloseai-speech && make test-f5' Enter
```
## When Things Break
```bash
# Check logs
tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech-server-1' Enter
# tail prod log
tmux send-keys -t <4090-window> 'tail -100 /mnt/data/f5-sidecar/logs/speech.log' Enter
# Check GPU memory
tmux send-keys -t 0:1 'nvidia-smi' Enter
# GPU memory
tmux send-keys -t <4090-window> 'nvidia-smi' Enter
# Restart container
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && sudo docker compose restart' Enter
# Full rebuild
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker compose up -d --build' Enter
# Read tmux output
tmux capture-pane -t 0:1 -p | tail -20
# kill + relaunch (see "Deploy a code/config change" above for the full sequence)
```
## Common Mistakes to Avoid
- DON'T use `ssh user@host "command"` -- use `tmux send-keys -t 0:1`
- DON'T use `rsync` or `scp` -- use `git push` + `tmux send-keys -t 0:1 'git pull' Enter`
- DON'T hardcode tmux window numbers -- use `tmux-hosts` to discover them
- DON'T put scripts in docs/ -- put them in scripts/
- DON'T use `ssh user@host "command"` — use `tmux send-keys`
- DON'T use `rsync` or `scp` — use `git push` + tmux `git pull`
- DON'T hardcode tmux window numbers — rediscover via `tmux-hosts` every session
- DON'T put scripts in docs/ — put them in scripts/
- DON'T forget to push before deploying
- DON'T assume prod uses docker — it's bare python; the compose files are for self-hosted parity only
## AGPL v3 Compliance