Add comprehensive TTS model and mirror documentation
docs/MODELS.md: - Document 10+ abandoned TTS engines to integrate - Piper TTS (integrated, fixed) - Coqui XTTS v2 (integrated, company shut down) - Silero TTS (HIGH PRIORITY - still active, fast) - StyleTTS2 (HIGH PRIORITY - best quality) - Fish Speech (active, good quality) - Kokoro, Bark, Tortoise, MetaVoice (lower priority) - Integration roadmap with time estimates - Performance targets and storage requirements docs/MIRRORS.md: - Multi-tier mirror strategy for resilience - Tier 1: Upstream (HuggingFace, PyPI, GitHub) - Tier 2: Self-hosted MinIO on ai.foxhop.net - Tier 3: Archive.org for public archival - Tier 4: IPFS for decentralization - Complete implementation with scripts and configs - Fallback download logic - Recovery scenarios - Cost: $0-20/month Raccoon mission: Ensure TTS keeps working when upstream dies. Documentation-first approach before implementing features. 🦝 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
e8183b4585
commit
3a9e27e556
2 changed files with 801 additions and 0 deletions
375
docs/MODELS.md
Normal file
375
docs/MODELS.md
Normal file
|
|
@ -0,0 +1,375 @@
|
|||
# TTS Models and Engines
|
||||
|
||||
**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into UncloseAI Speech
|
||||
|
||||
## Currently Integrated
|
||||
|
||||
### 1. Piper TTS ✅
|
||||
|
||||
**Status:** Working with absolute paths
|
||||
**Original Project:** rhasspy/piper (abandoned)
|
||||
**Fork:** OHF-Voice/piper1-gpl v1.3.0
|
||||
**Current Package:** PyPI `piper-tts>=1.2.0`
|
||||
|
||||
**Features:**
|
||||
- Fast CPU-based neural TTS
|
||||
- ~100+ high-quality voices
|
||||
- Multilingual support
|
||||
- ONNX runtime
|
||||
- Low memory footprint (~100MB per voice)
|
||||
|
||||
**Voices Available:**
|
||||
- English (US, GB, multiple accents)
|
||||
- Spanish, French, German, Italian
|
||||
- Russian, Polish, Ukrainian
|
||||
- Chinese, Japanese, Korean
|
||||
- Many more languages
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `rhasspy/piper-voices`
|
||||
- Direct download: `https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/`
|
||||
|
||||
**Integration:**
|
||||
- Used for `tts-1` model (fast, lower quality)
|
||||
- Models stored in `/app/voices/en/en_US/libritts_r/medium/`
|
||||
- Configuration via absolute paths in `voice_to_speaker.yaml`
|
||||
|
||||
**Example Config:**
|
||||
```yaml
|
||||
tts-1:
|
||||
alloy:
|
||||
model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
|
||||
speaker: 79
|
||||
```
|
||||
|
||||
**Raccoon Notes:**
|
||||
- Original rhasspy project abandoned
|
||||
- OHF-Voice fork has no PyPI package
|
||||
- Need to create our own PyPI package or vendor the code
|
||||
- Mirror all voices to prevent HuggingFace dependency
|
||||
|
||||
---
|
||||
|
||||
### 2. Coqui XTTS v2 ✅
|
||||
|
||||
**Status:** Integrated as tts-1-hd
|
||||
**Original Project:** coqui-ai/TTS (company shut down, archived)
|
||||
**Current Package:** PyPI `coqui-tts[languages]`
|
||||
|
||||
**Features:**
|
||||
- High-quality multilingual TTS
|
||||
- Voice cloning from 6-second samples
|
||||
- Emotional prosody control
|
||||
- GPU accelerated (NVIDIA/ROCm)
|
||||
- ~1.8GB model size
|
||||
|
||||
**Languages:**
|
||||
- English, Spanish, French, German, Italian, Portuguese
|
||||
- Polish, Turkish, Russian, Dutch, Czech
|
||||
- Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `coqui/XTTS-v2`
|
||||
- Auto-downloaded on first use
|
||||
|
||||
**Integration:**
|
||||
- Used for `tts-1-hd` model (slow, high quality)
|
||||
- Voice cloning with custom WAV samples
|
||||
- Language auto-detection with `langdetect`
|
||||
|
||||
**Example Config:**
|
||||
```yaml
|
||||
tts-1-hd:
|
||||
alloy:
|
||||
model: xtts
|
||||
speaker: /app/voices/alloy.wav
|
||||
language: en
|
||||
```
|
||||
|
||||
**Raccoon Notes:**
|
||||
- Coqui company shut down in 2024
|
||||
- Repository archived but code still works
|
||||
- Community forks emerging
|
||||
- Must mirror XTTS-v2 weights before they disappear
|
||||
- Consider forking to uncloseai-xtts
|
||||
|
||||
---
|
||||
|
||||
## High Priority Integration Targets
|
||||
|
||||
### 3. Silero TTS 🎯
|
||||
|
||||
**Status:** NOT INTEGRATED - HIGH PRIORITY
|
||||
**Project:** snakers4/silero-models (still active!)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Why Integrate:**
|
||||
- STILL ACTIVELY MAINTAINED
|
||||
- Fast, small models (~50-100MB each)
|
||||
- High quality for size
|
||||
- Easy integration (PyTorch)
|
||||
- Commercial-friendly license
|
||||
|
||||
**Features:**
|
||||
- Multilingual: English, Russian, German, Spanish, French
|
||||
- Multiple speakers per language
|
||||
- Emotion control
|
||||
- CPU friendly
|
||||
- Real-time capable
|
||||
|
||||
**Models:**
|
||||
- English: 4 speakers (en_v4)
|
||||
- Russian: 8+ speakers (ru_v4)
|
||||
- German: 1 speaker (de_v3)
|
||||
- Spanish: 2 speakers (es_v1)
|
||||
- French: 1 speaker (fr_v3)
|
||||
|
||||
**Model Source:**
|
||||
- GitHub Releases: https://github.com/snakers4/silero-models/releases
|
||||
- PyTorch Hub
|
||||
- Direct ONNX models available
|
||||
|
||||
**Estimated Integration Effort:** 2-4 hours
|
||||
- Add to requirements.txt: `silero` or direct PyTorch load
|
||||
- Create `src/engines/silero.py`
|
||||
- Download models to `/app/models/silero/`
|
||||
- Add voice mappings to config
|
||||
|
||||
**Example Usage:**
|
||||
```python
|
||||
import torch
|
||||
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
|
||||
repo_or_dir='snakers4/silero-models',
|
||||
model='silero_tts',
|
||||
language='en',
|
||||
speaker='v4_en'
|
||||
)
|
||||
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)
|
||||
```
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio)
|
||||
|
||||
---
|
||||
|
||||
### 4. StyleTTS2 🎯
|
||||
|
||||
**Status:** NOT INTEGRATED - HIGH PRIORITY
|
||||
**Project:** yl4579/StyleTTS2 (research, somewhat active)
|
||||
**License:** MIT
|
||||
|
||||
**Why Integrate:**
|
||||
- State-of-the-art quality
|
||||
- Best prosody and naturalness
|
||||
- Voice cloning capability
|
||||
- Style/emotion control
|
||||
- Research-grade results
|
||||
|
||||
**Features:**
|
||||
- Human-level prosody
|
||||
- Zero-shot voice cloning
|
||||
- Style transfer
|
||||
- Emotion and speaking style control
|
||||
- LibriTTS trained models
|
||||
|
||||
**Challenges:**
|
||||
- Complex dependencies
|
||||
- Requires phonemizer
|
||||
- Slower than other engines
|
||||
- GPU recommended
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `yl4579/StyleTTS2-LibriTTS`
|
||||
- GitHub releases
|
||||
|
||||
**Estimated Integration Effort:** 6-8 hours
|
||||
- Complex dependency chain
|
||||
- Need phonemizer setup
|
||||
- Create custom engine wrapper
|
||||
- May need model quantization for production
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐⭐ (Best quality, but complex)
|
||||
|
||||
---
|
||||
|
||||
### 5. Fish Speech 🎯
|
||||
|
||||
**Status:** NOT INTEGRATED - MEDIUM PRIORITY
|
||||
**Project:** fishaudio/fish-speech (active)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Why Integrate:**
|
||||
- Fast and efficient
|
||||
- Good multilingual support
|
||||
- Active development
|
||||
- Clean API
|
||||
|
||||
**Features:**
|
||||
- Fast inference
|
||||
- Multilingual (EN, ZH, JA)
|
||||
- Voice cloning
|
||||
- Streaming support
|
||||
- Modern architecture
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `fishaudio/fish-speech-1`
|
||||
- GitHub releases
|
||||
|
||||
**Estimated Integration Effort:** 4-6 hours
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐ (Active, good quality, but newer/less proven)
|
||||
|
||||
---
|
||||
|
||||
## Medium Priority Targets
|
||||
|
||||
### 6. Kokoro TTS
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** hexgrad/kokoro (new, active)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Features:**
|
||||
- Fast, small, quality
|
||||
- Multiple voices
|
||||
- Good English support
|
||||
- Emerging project
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐ (Promising but new)
|
||||
|
||||
---
|
||||
|
||||
### 7. Bark (Suno AI)
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** suno-ai/bark (archived, company pivoted to music)
|
||||
**License:** MIT
|
||||
|
||||
**Why Consider:**
|
||||
- Can generate music and sound effects
|
||||
- Non-verbal sounds (laughs, sighs)
|
||||
- Multiple languages
|
||||
- Background audio
|
||||
|
||||
**Why Low Priority:**
|
||||
- Very slow generation
|
||||
- Large models (~10GB)
|
||||
- Company abandoned it
|
||||
- Quality inconsistent
|
||||
|
||||
**Raccoon Priority:** ⭐⭐ (Unique features, but slow and abandoned)
|
||||
|
||||
---
|
||||
|
||||
## Low Priority / Archived
|
||||
|
||||
### 8. Tortoise TTS
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** neonbjb/tortoise-tts (low activity)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Features:**
|
||||
- Very high quality
|
||||
- Voice cloning
|
||||
|
||||
**Why Low Priority:**
|
||||
- Extremely slow (minutes per sentence)
|
||||
- Not practical for API use
|
||||
- Better alternatives exist now
|
||||
|
||||
**Raccoon Priority:** ⭐ (Too slow for production)
|
||||
|
||||
---
|
||||
|
||||
### 9. MetaVoice
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** metavoiceio/metavoice-src (partially abandoned)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Features:**
|
||||
- Long-form TTS
|
||||
- Emotional control
|
||||
- Voice cloning
|
||||
|
||||
**Why Low Priority:**
|
||||
- Unclear maintenance status
|
||||
- Complex setup
|
||||
- Alternatives are better
|
||||
|
||||
**Raccoon Priority:** ⭐ (Uncertain future)
|
||||
|
||||
---
|
||||
|
||||
### 10. Mozilla TTS
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** mozilla/TTS (archived, became Coqui)
|
||||
**License:** MPL 2.0
|
||||
|
||||
**Why Skip:**
|
||||
- Fully superseded by Coqui
|
||||
- No unique capabilities
|
||||
- Outdated architecture
|
||||
|
||||
**Raccoon Priority:** ⛔ (Skip - use Coqui instead)
|
||||
|
||||
---
|
||||
|
||||
## Integration Roadmap
|
||||
|
||||
### Phase 1: Quick Wins (Next 1-2 weeks)
|
||||
1. ✅ Fix Piper absolute paths
|
||||
2. ✅ Audit repository
|
||||
3. [ ] Integrate Silero TTS (2-4 hours)
|
||||
4. [ ] Set up model mirror on ai.foxhop.net
|
||||
5. [ ] Test Silero with existing API
|
||||
|
||||
### Phase 2: High Quality (2-4 weeks)
|
||||
1. [ ] Integrate StyleTTS2
|
||||
2. [ ] Create engine abstraction layer
|
||||
3. [ ] Refactor speech.py to use engines
|
||||
4. [ ] Add Fish Speech support
|
||||
|
||||
### Phase 3: Resilience (1-2 months)
|
||||
1. [ ] Implement binary mirror system
|
||||
2. [ ] Create fallback download logic
|
||||
3. [ ] Archive critical models to Archive.org
|
||||
4. [ ] Document all model sources
|
||||
|
||||
### Phase 4: Advanced Features (2+ months)
|
||||
1. [ ] Voice cloning API endpoint
|
||||
2. [ ] Emotion/style control
|
||||
3. [ ] Streaming TTS
|
||||
4. [ ] Multi-speaker conversations
|
||||
|
||||
## Model Storage Requirements
|
||||
|
||||
Current:
|
||||
- Piper voices: ~2GB (all languages)
|
||||
- XTTS v2: ~1.8GB
|
||||
|
||||
With all planned engines:
|
||||
- Silero models: ~500MB (all languages)
|
||||
- StyleTTS2: ~2GB (base model)
|
||||
- Fish Speech: ~1.5GB
|
||||
- **Total: ~8GB** for complete coverage
|
||||
|
||||
Mirror storage needed: ~20GB (with redundancy and archives)
|
||||
|
||||
## Performance Targets
|
||||
|
||||
| Engine | Speed (RTF) | Quality | Use Case |
|
||||
|--------|-------------|---------|----------|
|
||||
| Piper | 0.05x | Good | Fast API responses |
|
||||
| Silero | 0.1x | Good | Balanced speed/quality |
|
||||
| XTTS | 0.3x | Excellent | Voice cloning |
|
||||
| StyleTTS2 | 0.5x | Best | Premium quality |
|
||||
| Fish Speech | 0.15x | Very Good | Multilingual |
|
||||
|
||||
RTF = Real-time factor (lower is faster, 1.0 = real-time)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-11-09
|
||||
**Raccoon Status:** 🦝 Actively hunting for TTS models in the dumpsters of abandoned repos
|
||||
Loading…
Add table
Add a link
Reference in a new issue