diff --git a/CLAUDE.md b/CLAUDE.md index 81bcdc3..7e101be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -107,8 +107,8 @@ When adding features, update ALL relevant docs: # 1. Clean everything make clean -# 2. Deploy (syncs files, creates env, builds container) -make deploy +# 2. Deploy (commit, push, pull on server, rebuild container) +git push && tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker-compose up --build -d' Enter # 3. Download voices (Piper + XTTS samples) make voices @@ -125,7 +125,7 @@ make test-xtts 3. Implement engine wrapper in `speech.py` or `src/engines/` 4. Add test target (e.g., `test-silero`) 5. Update `make voices` to include it -6. Test full cycle: `make clean && make deploy && make voices` +6. Test full cycle: commit, push, pull on server, rebuild, test ### Debugging Issues @@ -195,30 +195,31 @@ See `docs/MODELS.md` for complete roadmap and detailed model documentation. ## Production Deployment -**Production Server:** `ai.foxhop.net` -- **URL:** https://ai.foxhop.net (port 8000 internal) +**Production Server:** `3090-ai.foxhop.net` +- **URL:** https://speech.ai.unturf.com (proxied via ai.foxhop.net) - **Repo Location:** `/home/fox/git/uncloseai-speech` -- **Container:** `uncloseai-speech-server-1` (image: `uncloseai-speech:local`) -- **tmux access:** `tmux send-keys -t 0:1 'command' Enter` (window 1 is AI server) +- **Container:** `uncloseai-speech_server_1` (image: `uncloseai-speech:local`) +- **tmux access:** `tmux send-keys -t 0:1 'command' Enter` (window 0:1 is 3090-ai) ### Deployment Workflow +**We use git, not rsync.** Commit locally, push, pull on server, rebuild. + ``` Local: /home/fox/git/uncloseai-speech/ - ↓ make deploy (rsync) + ↓ git commit && git push -Remote (ai.foxhop.net): +Remote (3090-ai.foxhop.net): /home/fox/git/uncloseai-speech/ - ↓ docker compose up --build + ↓ git pull && sudo docker-compose up --build -d Container: /app/ ├── speech.py - ├── voices/ - │ └── en/en_US/libritts_r/medium/*.onnx + ├── voices/samples/ (cloned-voices mounted) └── config/ └── voice_to_speaker.yaml ``` @@ -226,28 +227,30 @@ Container: ### Quick Production Commands ```bash +# Use tmux-hosts to see available tmux windows +tmux-hosts + +# Push changes and deploy +git add -A && git commit -m "message" && git push +tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker-compose up --build -d' Enter + # Check container status -tmux send-keys -t 0:1 'docker ps' Enter +tmux send-keys -t 0:1 'sudo docker ps | grep speech' Enter # View logs -tmux send-keys -t 0:1 'docker logs -f uncloseai-speech-server-1' Enter +tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech_server_1' Enter # Restart container -tmux send-keys -t 0:1 'cd /home/fox/git/uncloseai-speech && docker compose restart' Enter - -# Rebuild and redeploy -tmux send-keys -t 0:1 'cd /home/fox/git/uncloseai-speech && docker compose up --build -d' Enter +tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && sudo docker-compose restart' Enter ``` ## Testing Philosophy **Full stack testing workflow:** -1. `make clean` - Clean state -2. `make deploy` - Fresh deploy -3. `make voices` - Download voice models -4. `make test` - Basic API test -5. `make hydrate` - Test all 245 voices (sequential, safe) -6. `make load-test` - 100 concurrent requests (stress test) +1. Commit and push changes +2. Pull on server and rebuild: `git pull && sudo docker-compose up --build -d` +3. Wait for model to load (check logs) +4. Test via curl or browser **Never assume** - always test from scratch after changes. @@ -299,8 +302,8 @@ When running uncloseai-speech as a network service, you must provide: ❌ DON'T create directories with raw ssh ✅ DO add Makefile target for deployment -❌ DON'T assume container has changes after rsync -✅ DO rebuild with `make deploy` (runs docker compose up --build) +❌ DON'T use rsync - we use git +✅ DO commit, push, pull on server, then rebuild with docker-compose ❌ DON'T put scripts in docs/ ✅ DO put scripts in scripts/, reference from docs @@ -308,16 +311,16 @@ When running uncloseai-speech as a network service, you must provide: ❌ DON'T hardcode paths/hosts ✅ DO use vars.sh variables -❌ DON'T forget to test from scratch -✅ DO run `make clean && make deploy && make voices` +❌ DON'T forget to commit and push before deploying +✅ DO commit, push, pull on server, rebuild container ## When Things Break -1. Check `make logs` for errors -2. Verify Makefile was updated -3. Test from clean state -4. Check if container was rebuilt (`make deploy` does this) -5. Verify voices downloaded (`ls` in container via `make logs` approach) +1. Check logs: `tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech_server_1' Enter` +2. Check GPU memory: `tmux send-keys -t 0:1 'nvidia-smi' Enter` +3. Restart container: `tmux send-keys -t 0:1 'sudo docker-compose restart' Enter` +4. Rebuild: `git pull && sudo docker-compose up --build -d` +5. Check voice files: `tmux send-keys -t 0:1 'ls voices/samples/' Enter` ## Multiprocess Architecture (uvicorn workers=4) diff --git a/sample.env b/sample.env index 12a70d5..0003f53 100644 --- a/sample.env +++ b/sample.env @@ -1,9 +1,10 @@ TTS_HOME=voices HF_HOME=voices -# Worker processes (default: 4 for concurrent TTS requests) -# Each worker can handle one TTS request at a time -WORKERS=4 +# Worker processes for concurrent TTS requests +# Use WORKERS=1 for GPU models like Qwen3-TTS (each worker loads its own model copy) +# Use WORKERS=4 for CPU models like Piper +WORKERS=1 #PRELOAD_MODEL=xtts #PRELOAD_MODEL=xtts_v2.0.2 diff --git a/speech.py b/speech.py index 63915aa..190e408 100755 --- a/speech.py +++ b/speech.py @@ -1125,8 +1125,20 @@ async def generate_speech(request: GenerateSpeechRequest): ffmpeg_args.extend(["-"]) ffmpeg_proc = subprocess.Popen(ffmpeg_args, stdin=subprocess.PIPE, stdout=subprocess.PIPE) - ffmpeg_proc.stdin.write(audio_data) - ffmpeg_proc.stdin.close() + + # Use threading to avoid deadlock when audio is large + # (pipe buffers can fill causing ffmpeg to block on stdout write, + # which blocks our stdin write, causing deadlock) + def write_audio(): + try: + ffmpeg_proc.stdin.write(audio_data) + except Exception as e: + logger.error(f"Qwen ffmpeg write error: {e}") + finally: + ffmpeg_proc.stdin.close() + + writer_thread = threading.Thread(target=write_audio, daemon=True) + writer_thread.start() return StreamingResponse(content=ffmpeg_proc.stdout, media_type=media_type) else: