Russell Ballestrini
7559e56d0c
- Add CHANGELOG.md with full version history (moved from README)
- Update all documentation to use lowercase 'uncloseai-speech' project name
- Update organization references to lowercase 'uncloseai' (not 'UncloseAI')
- Add Brand Identity section to docs/CLAUDE.md with naming guidelines
- Update speech.py argparse description to match branding
- Update README.md headers and sections with consistent naming
- Update all model documentation with consistent branding
Files updated:
- CHANGELOG.md (new file)
- README.md (changelog reference, server options, multilingual section)
- speech.py (--workers argument, branding in argparse)
- Makefile (header comment)
- docs/CLAUDE.md (Brand Identity section)
- docs/MODELS.md
- docs/MIRRORS.md
- docs/AUDIT.md
- docs/models/coqui-tts.md
- docs/research/tts-models-overview.md
Branding standard:
- Project: uncloseai-speech (lowercase, hyphenated)
- Organization: uncloseai (lowercase, one word)
🦝 Generated with Claude Code
512 lines
13 KiB
Markdown
512 lines
13 KiB
Markdown
# TTS Models and Engines
|
|
|
|
**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into uncloseai-speech
|
|
|
|
## Documentation Index
|
|
|
|
### Comprehensive Research
|
|
- 📊 [TTS Models Overview & Research](research/tts-models-overview.md) - Complete comparison matrix, feature analysis, and integration roadmap
|
|
|
|
### Individual Model Documentation
|
|
Each model has detailed documentation covering technical specs, integration status, and Raccoon Mission notes:
|
|
|
|
**Currently Integrated:**
|
|
- 📄 [Coqui TTS (XTTS-v2)](models/coqui-tts.md) - High-quality multilingual TTS with voice cloning
|
|
- 📄 [Piper TTS](models/piper-tts.md) - Fast, lightweight neural TTS with 100+ voices
|
|
- 📄 [Silero TTS](models/silero-tts.md) - CPU-friendly, actively maintained, 5 languages, 148 voices ✨
|
|
- 📄 [Kokoro TTS](models/kokoro-tts.md) - Fast decoder-only architecture, 34 voices, Apache-2.0 ✨
|
|
|
|
**High Priority Candidates:**
|
|
- 📄 [Chatterbox](models/chatterbox.md) - Emotion control, 23 languages, zero-shot cloning
|
|
|
|
**Specialized Models:**
|
|
- 📄 [Mimic 3](models/mimic3.md) - Privacy-focused, offline, lightweight
|
|
- 📄 [eSpeak NG](models/espeak-ng.md) - 100+ languages, accessibility-focused
|
|
- 📄 [Maya1](models/maya1.md) - Indic languages, diverse accents
|
|
- 📄 [Step-Audio-EditX](models/step-audio-editx.md) - LLM-based audio editing (experimental)
|
|
|
|
**Historical/Archived:**
|
|
- 📄 [Mozilla TTS](models/mozilla-tts.md) - Superseded by Coqui TTS
|
|
- 📄 [Tortoise TTS](models/tortoise-tts.md) - Studio-quality but slow (archival)
|
|
|
|
---
|
|
|
|
## Currently Integrated
|
|
|
|
### 1. Piper TTS ✅
|
|
|
|
> 📖 **See [detailed documentation](models/piper-tts.md)** for comprehensive technical specs and integration guide
|
|
|
|
**Status:** Working with absolute paths
|
|
**Original Project:** rhasspy/piper (abandoned)
|
|
**Fork:** OHF-Voice/piper1-gpl v1.3.0
|
|
**Current Package:** PyPI `piper-tts>=1.2.0`
|
|
|
|
**Features:**
|
|
- Fast CPU-based neural TTS
|
|
- ~100+ high-quality voices
|
|
- Multilingual support
|
|
- ONNX runtime
|
|
- Low memory footprint (~100MB per voice)
|
|
|
|
**Voices Available:**
|
|
- English (US, GB, multiple accents)
|
|
- Spanish, French, German, Italian
|
|
- Russian, Polish, Ukrainian
|
|
- Chinese, Japanese, Korean
|
|
- Many more languages
|
|
|
|
**Model Source:**
|
|
- HuggingFace: `rhasspy/piper-voices`
|
|
- Direct download: `https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/`
|
|
|
|
**Integration:**
|
|
- Used for `tts-1` model (fast, lower quality)
|
|
- Models stored in `/app/voices/en/en_US/libritts_r/medium/`
|
|
- Configuration via absolute paths in `voice_to_speaker.yaml`
|
|
|
|
**Example Config:**
|
|
```yaml
|
|
tts-1:
|
|
alloy:
|
|
model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
|
|
speaker: 79
|
|
```
|
|
|
|
**Raccoon Notes:**
|
|
- Original rhasspy project abandoned
|
|
- OHF-Voice fork has no PyPI package
|
|
- Need to create our own PyPI package or vendor the code
|
|
- Mirror all voices to prevent HuggingFace dependency
|
|
|
|
---
|
|
|
|
### 2. Coqui XTTS v2 ✅
|
|
|
|
> 📖 **See [detailed documentation](models/coqui-tts.md)** for comprehensive technical specs and integration guide
|
|
|
|
**Status:** Integrated as tts-1-hd
|
|
**Original Project:** coqui-ai/TTS (company shut down, archived)
|
|
**Current Package:** PyPI `coqui-tts[languages]`
|
|
|
|
**Features:**
|
|
- High-quality multilingual TTS
|
|
- Voice cloning from 6-second samples
|
|
- Emotional prosody control
|
|
- GPU accelerated (NVIDIA/ROCm)
|
|
- ~1.8GB model size
|
|
|
|
**Languages:**
|
|
- English, Spanish, French, German, Italian, Portuguese
|
|
- Polish, Turkish, Russian, Dutch, Czech
|
|
- Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi
|
|
|
|
**Model Source:**
|
|
- HuggingFace: `coqui/XTTS-v2`
|
|
- Auto-downloaded on first use
|
|
|
|
**Integration:**
|
|
- Used for `tts-1-hd` model (slow, high quality)
|
|
- Voice cloning with custom WAV samples
|
|
- Language auto-detection with `langdetect`
|
|
|
|
**Example Config:**
|
|
```yaml
|
|
tts-1-hd:
|
|
alloy:
|
|
model: xtts
|
|
speaker: /app/voices/alloy.wav
|
|
language: en
|
|
```
|
|
|
|
**Raccoon Notes:**
|
|
- Coqui company shut down in 2024
|
|
- Repository archived but code still works
|
|
- Community forks emerging
|
|
- Must mirror XTTS-v2 weights before they disappear
|
|
- Consider forking to uncloseai-xtts
|
|
|
|
---
|
|
|
|
## High Priority Integration Targets
|
|
|
|
### 3. Silero TTS ✅
|
|
|
|
> 📖 **See [detailed documentation](models/silero-tts.md)** for comprehensive technical specs (documentation pending)
|
|
|
|
**Status:** INTEGRATED as tts-1-silero
|
|
**Project:** snakers4/silero-models (actively maintained!)
|
|
**License:** Apache 2.0
|
|
|
|
**Integration Benefits:**
|
|
- ACTIVELY MAINTAINED - no abandonment risk!
|
|
- Fast, small models (~50-100MB each)
|
|
- High quality for size
|
|
- Easy integration via PyTorch Hub
|
|
- Commercial-friendly license
|
|
- CPU friendly - no GPU required
|
|
|
|
**Features:**
|
|
- Multilingual: English, Russian, German, Spanish, French
|
|
- Multiple speakers per language (English: 117 speakers!)
|
|
- Emotion control
|
|
- Real-time capable on CPU
|
|
- 48kHz sample rate
|
|
|
|
**Models:**
|
|
- English: 117 speakers (v4_en)
|
|
- Russian: 8+ speakers (v4_ru)
|
|
- German: 1 speaker (v3_de)
|
|
- Spanish: 2 speakers (v1_es)
|
|
- French: 1 speaker (v3_fr)
|
|
|
|
**Model Source:**
|
|
- PyTorch Hub: `torch.hub.load('snakers4/silero-models')`
|
|
- Models downloaded on first use
|
|
- Cached in `/app/voices/` directory
|
|
|
|
**Integration:**
|
|
- Used for `tts-1-silero` model (fast, CPU-friendly)
|
|
- Loaded via torch.hub on demand
|
|
- 6 OpenAI-compatible voices mapped to Silero speakers
|
|
|
|
**Example Config:**
|
|
```yaml
|
|
tts-1-silero:
|
|
alloy:
|
|
language: en
|
|
speaker: en_0
|
|
silero_speaker: v4_en
|
|
```
|
|
|
|
**Makefile Targets:**
|
|
```bash
|
|
make voices-silero # Download Silero models (en, ru, de, es, fr)
|
|
make test-silero # Test Silero TTS endpoint
|
|
```
|
|
|
|
**Example Usage:**
|
|
```python
|
|
import torch
|
|
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
|
|
repo_or_dir='snakers4/silero-models',
|
|
model='silero_tts',
|
|
language='en',
|
|
speaker='v4_en'
|
|
)
|
|
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)
|
|
```
|
|
|
|
**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio)
|
|
|
|
---
|
|
|
|
### 4. StyleTTS2 🎯
|
|
|
|
**Status:** NOT INTEGRATED - HIGH PRIORITY
|
|
**Project:** yl4579/StyleTTS2 (research, somewhat active)
|
|
**License:** MIT
|
|
|
|
**Why Integrate:**
|
|
- State-of-the-art quality
|
|
- Best prosody and naturalness
|
|
- Voice cloning capability
|
|
- Style/emotion control
|
|
- Research-grade results
|
|
|
|
**Features:**
|
|
- Human-level prosody
|
|
- Zero-shot voice cloning
|
|
- Style transfer
|
|
- Emotion and speaking style control
|
|
- LibriTTS trained models
|
|
|
|
**Challenges:**
|
|
- Complex dependencies
|
|
- Requires phonemizer
|
|
- Slower than other engines
|
|
- GPU recommended
|
|
|
|
**Model Source:**
|
|
- HuggingFace: `yl4579/StyleTTS2-LibriTTS`
|
|
- GitHub releases
|
|
|
|
**Estimated Integration Effort:** 6-8 hours
|
|
- Complex dependency chain
|
|
- Need phonemizer setup
|
|
- Create custom engine wrapper
|
|
- May need model quantization for production
|
|
|
|
**Raccoon Priority:** ⭐⭐⭐⭐ (Best quality, but complex)
|
|
|
|
---
|
|
|
|
### 5. Fish Speech 🎯
|
|
|
|
**Status:** NOT INTEGRATED - MEDIUM PRIORITY
|
|
**Project:** fishaudio/fish-speech (active)
|
|
**License:** Apache 2.0
|
|
|
|
**Why Integrate:**
|
|
- Fast and efficient
|
|
- Good multilingual support
|
|
- Active development
|
|
- Clean API
|
|
|
|
**Features:**
|
|
- Fast inference
|
|
- Multilingual (EN, ZH, JA)
|
|
- Voice cloning
|
|
- Streaming support
|
|
- Modern architecture
|
|
|
|
**Model Source:**
|
|
- HuggingFace: `fishaudio/fish-speech-1`
|
|
- GitHub releases
|
|
|
|
**Estimated Integration Effort:** 4-6 hours
|
|
|
|
**Raccoon Priority:** ⭐⭐⭐ (Active, good quality, but newer/less proven)
|
|
|
|
---
|
|
|
|
## Medium Priority Targets
|
|
|
|
### 6. Kokoro TTS ✅
|
|
|
|
> 📖 **See [detailed documentation](models/kokoro-tts.md)** for comprehensive technical specs
|
|
|
|
**Status:** INTEGRATED as tts-1-kokoro
|
|
**Project:** hexgrad/kokoro (new, active)
|
|
**License:** Apache 2.0
|
|
|
|
**Integration Benefits:**
|
|
- Fast decoder-only architecture (82M params)
|
|
- 34 voices (American and British English)
|
|
- 24kHz sample rate
|
|
- Apache-2.0 license
|
|
- Lightweight and efficient
|
|
|
|
**Features:**
|
|
- American English: 20 voices (11 female, 9 male)
|
|
- British English: 14 voices (4 female, 4 male + variations)
|
|
- Speed control
|
|
- Real-time capable
|
|
|
|
**Model Source:**
|
|
- HuggingFace: `hexgrad/kokoro-82m`
|
|
- Downloaded via huggingface-cli
|
|
|
|
**Integration:**
|
|
- Used for `tts-1-kokoro` model (fast, quality)
|
|
- Loaded via kokoro Python package
|
|
- OpenAI-compatible voice aliases
|
|
|
|
**Example Config:**
|
|
```yaml
|
|
tts-1-kokoro:
|
|
alloy:
|
|
lang_code: a
|
|
kokoro_voice: af_alloy
|
|
```
|
|
|
|
**Makefile Targets:**
|
|
```bash
|
|
make voices-kokoro # Download Kokoro models
|
|
make test-kokoro # Test Kokoro TTS endpoint
|
|
```
|
|
|
|
**Raccoon Priority:** ⭐⭐⭐⭐ (Successfully integrated!)
|
|
|
|
---
|
|
|
|
### 7. Bark (Suno AI)
|
|
|
|
**Status:** NOT INTEGRATED
|
|
**Project:** suno-ai/bark (archived, company pivoted to music)
|
|
**License:** MIT
|
|
|
|
**Why Consider:**
|
|
- Can generate music and sound effects
|
|
- Non-verbal sounds (laughs, sighs)
|
|
- Multiple languages
|
|
- Background audio
|
|
|
|
**Why Low Priority:**
|
|
- Very slow generation
|
|
- Large models (~10GB)
|
|
- Company abandoned it
|
|
- Quality inconsistent
|
|
|
|
**Raccoon Priority:** ⭐⭐ (Unique features, but slow and abandoned)
|
|
|
|
---
|
|
|
|
## Low Priority / Archived
|
|
|
|
### 8. Tortoise TTS
|
|
|
|
> 📖 **See [detailed documentation](models/tortoise-tts.md)** for comprehensive technical specs
|
|
|
|
**Status:** NOT INTEGRATED
|
|
**Project:** neonbjb/tortoise-tts (low activity)
|
|
**License:** Apache 2.0
|
|
|
|
**Features:**
|
|
- Very high quality
|
|
- Voice cloning
|
|
|
|
**Why Low Priority:**
|
|
- Extremely slow (minutes per sentence)
|
|
- Not practical for API use
|
|
- Better alternatives exist now
|
|
|
|
**Raccoon Priority:** ⭐ (Too slow for production)
|
|
|
|
---
|
|
|
|
### 9. MetaVoice
|
|
|
|
**Status:** NOT INTEGRATED
|
|
**Project:** metavoiceio/metavoice-src (partially abandoned)
|
|
**License:** Apache 2.0
|
|
|
|
**Features:**
|
|
- Long-form TTS
|
|
- Emotional control
|
|
- Voice cloning
|
|
|
|
**Why Low Priority:**
|
|
- Unclear maintenance status
|
|
- Complex setup
|
|
- Alternatives are better
|
|
|
|
**Raccoon Priority:** ⭐ (Uncertain future)
|
|
|
|
---
|
|
|
|
### 10. Mozilla TTS
|
|
|
|
> 📖 **See [detailed documentation](models/mozilla-tts.md)** for historical context and relationship to Coqui
|
|
|
|
**Status:** NOT INTEGRATED
|
|
**Project:** mozilla/TTS (archived, became Coqui)
|
|
**License:** MPL 2.0
|
|
|
|
**Why Skip:**
|
|
- Fully superseded by Coqui
|
|
- No unique capabilities
|
|
- Outdated architecture
|
|
|
|
**Raccoon Priority:** ⛔ (Skip - use Coqui instead)
|
|
|
|
---
|
|
|
|
## Integration Roadmap
|
|
|
|
### Phase 1: Quick Wins (Next 1-2 weeks)
|
|
1. ✅ Fix Piper absolute paths
|
|
2. ✅ Audit repository
|
|
3. ✅ Integrate Silero TTS (COMPLETED!)
|
|
4. ✅ Integrate Kokoro (fast decoder) (COMPLETED!)
|
|
5. ✅ Add /v1/models API endpoint for voice discovery
|
|
6. [ ] Set up model mirror on ai.foxhop.net
|
|
7. [ ] Integrate Chatterbox (emotion control)
|
|
|
|
### Phase 2: High Quality (2-4 weeks)
|
|
1. [ ] Integrate StyleTTS2
|
|
2. [ ] Create engine abstraction layer
|
|
3. [ ] Refactor speech.py to use engines
|
|
4. [ ] Add Fish Speech support
|
|
|
|
### Phase 3: Resilience (1-2 months)
|
|
1. [ ] Implement binary mirror system
|
|
2. [ ] Create fallback download logic
|
|
3. [ ] Archive critical models to Archive.org
|
|
4. [ ] Document all model sources
|
|
|
|
### Phase 4: Advanced Features (2+ months)
|
|
1. [ ] Voice cloning API endpoint
|
|
2. [ ] Emotion/style control
|
|
3. [ ] Streaming TTS
|
|
4. [ ] Multi-speaker conversations
|
|
|
|
## Model Storage Requirements
|
|
|
|
Current:
|
|
- Piper voices: ~2GB (all languages)
|
|
- XTTS v2: ~1.8GB
|
|
|
|
With all planned engines:
|
|
- Silero models: ~500MB (all languages)
|
|
- StyleTTS2: ~2GB (base model)
|
|
- Fish Speech: ~1.5GB
|
|
- **Total: ~8GB** for complete coverage
|
|
|
|
Mirror storage needed: ~20GB (with redundancy and archives)
|
|
|
|
## Performance Targets
|
|
|
|
| Engine | Speed (RTF) | Quality | Use Case |
|
|
|--------|-------------|---------|----------|
|
|
| Piper | 0.05x | Good | Fast API responses |
|
|
| Silero | 0.1x | Good | Balanced speed/quality |
|
|
| XTTS | 0.3x | Excellent | Voice cloning |
|
|
| StyleTTS2 | 0.5x | Best | Premium quality |
|
|
| Fish Speech | 0.15x | Very Good | Multilingual |
|
|
|
|
RTF = Real-time factor (lower is faster, 1.0 = real-time)
|
|
|
|
---
|
|
|
|
## Additional Models Under Research
|
|
|
|
The following models have detailed documentation but are not yet integrated or prioritized:
|
|
|
|
### Chatterbox
|
|
**Priority:** High - Emotion control features
|
|
📄 [Full Documentation](models/chatterbox.md)
|
|
- Multilingual zero-shot TTS from Resemble AI
|
|
- 23 languages with emotion exaggeration control
|
|
- Production-grade, actively maintained
|
|
- License: Apache-2.0
|
|
|
|
### Mimic 3
|
|
**Priority:** Medium - Privacy/embedded use cases
|
|
📄 [Full Documentation](models/mimic3.md)
|
|
- Lightweight offline TTS from Mycroft AI
|
|
- 20-50MB models, SSML support
|
|
- Privacy-focused, embeddable
|
|
- License: Apache-2.0
|
|
|
|
### eSpeak NG
|
|
**Priority:** Low - Niche accessibility use
|
|
📄 [Full Documentation](models/espeak-ng.md)
|
|
- Formant-based synthesis for 100+ languages
|
|
- Extremely portable (<10MB)
|
|
- Actively maintained by accessibility community
|
|
- License: GPL-3.0
|
|
|
|
### Step-Audio-EditX
|
|
**Priority:** Research - Experimental
|
|
📄 [Full Documentation](models/step-audio-editx.md)
|
|
- New LLM-based audio editing (November 2025)
|
|
- Post-generation emotion/style editing
|
|
- Cutting-edge but experimental
|
|
- License: Apache-2.0
|
|
|
|
### Maya1
|
|
**Priority:** Research - Emerging
|
|
📄 [Full Documentation](models/maya1.md)
|
|
- India-based multilingual voice model
|
|
- Strong Indic language support (Hindi, Tamil, etc.)
|
|
- High benchmark rankings
|
|
- License: MIT
|
|
|
|
---
|
|
|
|
**Last Updated:** 2025-11-09
|
|
**Raccoon Status:** 🦝 4 models rescued! Silero and Kokoro TTS integrated successfully
|
|
**Integration Status:** ✅ Piper (55 voices), XTTS (8 voices), Silero (148 voices), Kokoro (34 voices) | 🎯 Next: Chatterbox, StyleTTS2
|
|
**API Endpoints:** tts-1, tts-1-hd, tts-1-silero, tts-1-kokoro | /v1/models for discovery
|
|
**Documentation Status:** 📚 10 models fully documented, 1 comprehensive research overview
|