Fix Qwen TTS deadlock, reduce workers to 1 for GPU
- Fix subprocess deadlock in Qwen TTS by using threading for stdin write (prevents pipe buffer deadlock on large audio output) - Set WORKERS=1 for GPU models to avoid VRAM duplication (4 workers × 3GB model = OOM, 1 worker works fine) - Update CLAUDE.md: use git push/pull instead of rsync for deployment
This commit is contained in:
parent
cac40d75d3
commit
a148088cb0
3 changed files with 54 additions and 38 deletions
69
CLAUDE.md
69
CLAUDE.md
|
|
@ -107,8 +107,8 @@ When adding features, update ALL relevant docs:
|
|||
# 1. Clean everything
|
||||
make clean
|
||||
|
||||
# 2. Deploy (syncs files, creates env, builds container)
|
||||
make deploy
|
||||
# 2. Deploy (commit, push, pull on server, rebuild container)
|
||||
git push && tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker-compose up --build -d' Enter
|
||||
|
||||
# 3. Download voices (Piper + XTTS samples)
|
||||
make voices
|
||||
|
|
@ -125,7 +125,7 @@ make test-xtts
|
|||
3. Implement engine wrapper in `speech.py` or `src/engines/`
|
||||
4. Add test target (e.g., `test-silero`)
|
||||
5. Update `make voices` to include it
|
||||
6. Test full cycle: `make clean && make deploy && make voices`
|
||||
6. Test full cycle: commit, push, pull on server, rebuild, test
|
||||
|
||||
### Debugging Issues
|
||||
|
||||
|
|
@ -195,30 +195,31 @@ See `docs/MODELS.md` for complete roadmap and detailed model documentation.
|
|||
|
||||
## Production Deployment
|
||||
|
||||
**Production Server:** `ai.foxhop.net`
|
||||
- **URL:** https://ai.foxhop.net (port 8000 internal)
|
||||
**Production Server:** `3090-ai.foxhop.net`
|
||||
- **URL:** https://speech.ai.unturf.com (proxied via ai.foxhop.net)
|
||||
- **Repo Location:** `/home/fox/git/uncloseai-speech`
|
||||
- **Container:** `uncloseai-speech-server-1` (image: `uncloseai-speech:local`)
|
||||
- **tmux access:** `tmux send-keys -t 0:1 'command' Enter` (window 1 is AI server)
|
||||
- **Container:** `uncloseai-speech_server_1` (image: `uncloseai-speech:local`)
|
||||
- **tmux access:** `tmux send-keys -t 0:1 'command' Enter` (window 0:1 is 3090-ai)
|
||||
|
||||
### Deployment Workflow
|
||||
|
||||
**We use git, not rsync.** Commit locally, push, pull on server, rebuild.
|
||||
|
||||
```
|
||||
Local:
|
||||
/home/fox/git/uncloseai-speech/
|
||||
|
||||
↓ make deploy (rsync)
|
||||
↓ git commit && git push
|
||||
|
||||
Remote (ai.foxhop.net):
|
||||
Remote (3090-ai.foxhop.net):
|
||||
/home/fox/git/uncloseai-speech/
|
||||
|
||||
↓ docker compose up --build
|
||||
↓ git pull && sudo docker-compose up --build -d
|
||||
|
||||
Container:
|
||||
/app/
|
||||
├── speech.py
|
||||
├── voices/
|
||||
│ └── en/en_US/libritts_r/medium/*.onnx
|
||||
├── voices/samples/ (cloned-voices mounted)
|
||||
└── config/
|
||||
└── voice_to_speaker.yaml
|
||||
```
|
||||
|
|
@ -226,28 +227,30 @@ Container:
|
|||
### Quick Production Commands
|
||||
|
||||
```bash
|
||||
# Use tmux-hosts to see available tmux windows
|
||||
tmux-hosts
|
||||
|
||||
# Push changes and deploy
|
||||
git add -A && git commit -m "message" && git push
|
||||
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && git pull && sudo docker-compose up --build -d' Enter
|
||||
|
||||
# Check container status
|
||||
tmux send-keys -t 0:1 'docker ps' Enter
|
||||
tmux send-keys -t 0:1 'sudo docker ps | grep speech' Enter
|
||||
|
||||
# View logs
|
||||
tmux send-keys -t 0:1 'docker logs -f uncloseai-speech-server-1' Enter
|
||||
tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech_server_1' Enter
|
||||
|
||||
# Restart container
|
||||
tmux send-keys -t 0:1 'cd /home/fox/git/uncloseai-speech && docker compose restart' Enter
|
||||
|
||||
# Rebuild and redeploy
|
||||
tmux send-keys -t 0:1 'cd /home/fox/git/uncloseai-speech && docker compose up --build -d' Enter
|
||||
tmux send-keys -t 0:1 'cd ~/git/uncloseai-speech && sudo docker-compose restart' Enter
|
||||
```
|
||||
|
||||
## Testing Philosophy
|
||||
|
||||
**Full stack testing workflow:**
|
||||
1. `make clean` - Clean state
|
||||
2. `make deploy` - Fresh deploy
|
||||
3. `make voices` - Download voice models
|
||||
4. `make test` - Basic API test
|
||||
5. `make hydrate` - Test all 245 voices (sequential, safe)
|
||||
6. `make load-test` - 100 concurrent requests (stress test)
|
||||
1. Commit and push changes
|
||||
2. Pull on server and rebuild: `git pull && sudo docker-compose up --build -d`
|
||||
3. Wait for model to load (check logs)
|
||||
4. Test via curl or browser
|
||||
|
||||
**Never assume** - always test from scratch after changes.
|
||||
|
||||
|
|
@ -299,8 +302,8 @@ When running uncloseai-speech as a network service, you must provide:
|
|||
❌ DON'T create directories with raw ssh
|
||||
✅ DO add Makefile target for deployment
|
||||
|
||||
❌ DON'T assume container has changes after rsync
|
||||
✅ DO rebuild with `make deploy` (runs docker compose up --build)
|
||||
❌ DON'T use rsync - we use git
|
||||
✅ DO commit, push, pull on server, then rebuild with docker-compose
|
||||
|
||||
❌ DON'T put scripts in docs/
|
||||
✅ DO put scripts in scripts/, reference from docs
|
||||
|
|
@ -308,16 +311,16 @@ When running uncloseai-speech as a network service, you must provide:
|
|||
❌ DON'T hardcode paths/hosts
|
||||
✅ DO use vars.sh variables
|
||||
|
||||
❌ DON'T forget to test from scratch
|
||||
✅ DO run `make clean && make deploy && make voices`
|
||||
❌ DON'T forget to commit and push before deploying
|
||||
✅ DO commit, push, pull on server, rebuild container
|
||||
|
||||
## When Things Break
|
||||
|
||||
1. Check `make logs` for errors
|
||||
2. Verify Makefile was updated
|
||||
3. Test from clean state
|
||||
4. Check if container was rebuilt (`make deploy` does this)
|
||||
5. Verify voices downloaded (`ls` in container via `make logs` approach)
|
||||
1. Check logs: `tmux send-keys -t 0:1 'sudo docker logs --tail 50 uncloseai-speech_server_1' Enter`
|
||||
2. Check GPU memory: `tmux send-keys -t 0:1 'nvidia-smi' Enter`
|
||||
3. Restart container: `tmux send-keys -t 0:1 'sudo docker-compose restart' Enter`
|
||||
4. Rebuild: `git pull && sudo docker-compose up --build -d`
|
||||
5. Check voice files: `tmux send-keys -t 0:1 'ls voices/samples/' Enter`
|
||||
|
||||
## Multiprocess Architecture (uvicorn workers=4)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,10 @@
|
|||
TTS_HOME=voices
|
||||
HF_HOME=voices
|
||||
|
||||
# Worker processes (default: 4 for concurrent TTS requests)
|
||||
# Each worker can handle one TTS request at a time
|
||||
WORKERS=4
|
||||
# Worker processes for concurrent TTS requests
|
||||
# Use WORKERS=1 for GPU models like Qwen3-TTS (each worker loads its own model copy)
|
||||
# Use WORKERS=4 for CPU models like Piper
|
||||
WORKERS=1
|
||||
|
||||
#PRELOAD_MODEL=xtts
|
||||
#PRELOAD_MODEL=xtts_v2.0.2
|
||||
|
|
|
|||
16
speech.py
16
speech.py
|
|
@ -1125,8 +1125,20 @@ async def generate_speech(request: GenerateSpeechRequest):
|
|||
|
||||
ffmpeg_args.extend(["-"])
|
||||
ffmpeg_proc = subprocess.Popen(ffmpeg_args, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
|
||||
ffmpeg_proc.stdin.write(audio_data)
|
||||
ffmpeg_proc.stdin.close()
|
||||
|
||||
# Use threading to avoid deadlock when audio is large
|
||||
# (pipe buffers can fill causing ffmpeg to block on stdout write,
|
||||
# which blocks our stdin write, causing deadlock)
|
||||
def write_audio():
|
||||
try:
|
||||
ffmpeg_proc.stdin.write(audio_data)
|
||||
except Exception as e:
|
||||
logger.error(f"Qwen ffmpeg write error: {e}")
|
||||
finally:
|
||||
ffmpeg_proc.stdin.close()
|
||||
|
||||
writer_thread = threading.Thread(target=write_audio, daemon=True)
|
||||
writer_thread.start()
|
||||
|
||||
return StreamingResponse(content=ffmpeg_proc.stdout, media_type=media_type)
|
||||
else:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue