Russell Ballestrini
7559e56d0c
- Add CHANGELOG.md with full version history (moved from README)
- Update all documentation to use lowercase 'uncloseai-speech' project name
- Update organization references to lowercase 'uncloseai' (not 'UncloseAI')
- Add Brand Identity section to docs/CLAUDE.md with naming guidelines
- Update speech.py argparse description to match branding
- Update README.md headers and sections with consistent naming
- Update all model documentation with consistent branding
Files updated:
- CHANGELOG.md (new file)
- README.md (changelog reference, server options, multilingual section)
- speech.py (--workers argument, branding in argparse)
- Makefile (header comment)
- docs/CLAUDE.md (Brand Identity section)
- docs/MODELS.md
- docs/MIRRORS.md
- docs/AUDIT.md
- docs/models/coqui-tts.md
- docs/research/tts-models-overview.md
Branding standard:
- Project: uncloseai-speech (lowercase, hyphenated)
- Organization: uncloseai (lowercase, one word)
🦝 Generated with Claude Code
370 lines
13 KiB
Markdown
370 lines
13 KiB
Markdown
# TTS Models Research Overview
|
|
|
|
**Last Updated:** 2025-11-09
|
|
**Raccoon Mission Status:** 🦝 Active Rescue Operations
|
|
|
|
## Executive Summary
|
|
|
|
This document provides a comprehensive overview of open-source Text-to-Speech (TTS) models researched for integration into uncloseai-speech. Our "Raccoon Mission" aims to rescue abandoned and at-risk TTS projects, ensuring their long-term preservation and availability.
|
|
|
|
## Model Inventory
|
|
|
|
### Currently Integrated ✅
|
|
|
|
| Model | Status | Quality | Speed | Use Case |
|
|
|-------|--------|---------|-------|----------|
|
|
| [Piper TTS](../models/piper-tts.md) | Production | Good | Fast (0.05x RTF) | tts-1 (fast responses) |
|
|
| [Coqui XTTS-v2](../models/coqui-tts.md) | Production | Excellent | Medium (0.3x RTF) | tts-1-hd (high quality) |
|
|
|
|
### High Priority Candidates 🎯
|
|
|
|
| Model | Priority | Key Strengths | Integration Effort |
|
|
|-------|----------|---------------|-------------------|
|
|
| [Chatterbox](../models/chatterbox.md) | High | Emotion control, 23 languages | Medium |
|
|
| [Kokoro TTS](../models/kokoro-tts.md) | Medium | Fast, Apache-2.0 licensed | Medium |
|
|
| Silero TTS | High | Active maintenance, small models | Low |
|
|
| StyleTTS2 | High | Best quality/prosody | High |
|
|
|
|
### Specialized Models 🔬
|
|
|
|
| Model | Specialization | Integration Status |
|
|
|-------|----------------|-------------------|
|
|
| [Mimic 3](../models/mimic3.md) | Privacy-focused, offline | Candidate |
|
|
| [eSpeak NG](../models/espeak-ng.md) | 100+ languages, accessibility | Niche |
|
|
| [Maya1](../models/maya1.md) | Indic languages, diverse accents | Research |
|
|
| [Step-Audio-EditX](../models/step-audio-editx.md) | Post-generation editing | Experimental |
|
|
|
|
### Low Priority / Archived 📦
|
|
|
|
| Model | Reason | Status |
|
|
|-------|--------|--------|
|
|
| [Mozilla TTS](../models/mozilla-tts.md) | Superseded by Coqui | Archived |
|
|
| [Tortoise TTS](../models/tortoise-tts.md) | Too slow for production | Low priority |
|
|
|
|
## Model Comparison Matrix
|
|
|
|
### Performance Characteristics
|
|
|
|
| Model | RTF | Quality (MOS) | Languages | License | Model Size |
|
|
|-------|-----|---------------|-----------|---------|------------|
|
|
| Piper TTS | 0.05x | 3.5-4.0 | 50+ | MIT | ~100MB |
|
|
| Coqui XTTS-v2 | 0.3x | 4.2-4.5 | 20+ | Apache-2.0 | ~1.8GB |
|
|
| Chatterbox | 0.2x | 4.0-4.3 | 23 | Apache-2.0 | ~1.2GB |
|
|
| Kokoro TTS | 0.1-0.3x | 4.0-4.2 | Limited | Apache-2.0 | ~500MB |
|
|
| Mimic 3 | 0.1x | 3.8-4.0 | Multiple | Apache-2.0 | 20-50MB |
|
|
| eSpeak NG | <0.01x | 2.5-3.0 | 100+ | GPL-3.0 | <10MB |
|
|
| Maya1 | Unknown | 4.0+ | 10+ Indic | MIT | ~1.5GB |
|
|
| Tortoise TTS | 10-30x | 4.5-4.8 | English | Apache-2.0 | ~2GB |
|
|
| Step-Audio-EditX | Variable | N/A | Multiple | Apache-2.0 | Unknown |
|
|
| Mozilla TTS | 0.5x | 3.8-4.0 | Limited | MPL-2.0 | ~500MB |
|
|
|
|
**RTF = Real-Time Factor** (lower is faster, 1.0 = real-time)
|
|
**MOS = Mean Opinion Score** (1-5 scale, higher is better)
|
|
|
|
### Feature Matrix
|
|
|
|
| Model | Voice Cloning | Emotion Control | Multilingual | Offline | GPU Required |
|
|
|-------|---------------|----------------|--------------|---------|--------------|
|
|
| Piper TTS | ❌ | ❌ | ✅ | ✅ | ❌ |
|
|
| Coqui XTTS-v2 | ✅ | ✅ | ✅ | ✅ | Recommended |
|
|
| Chatterbox | ✅ | ✅ (unique) | ✅ | ✅ | Recommended |
|
|
| Kokoro TTS | ✅ | ✅ | Limited | ✅ | Optional |
|
|
| Mimic 3 | ❌ | Limited | ✅ | ✅ | ❌ |
|
|
| eSpeak NG | ❌ | ❌ | ✅ | ✅ | ❌ |
|
|
| Maya1 | ✅ | Unknown | ✅ | ✅ | Yes |
|
|
| Tortoise TTS | ✅ | ✅ | Limited | ✅ | Yes |
|
|
| Step-Audio-EditX | N/A | ✅ (editing) | ✅ | ✅ | Yes |
|
|
| Mozilla TTS | Limited | ❌ | Limited | ✅ | Recommended |
|
|
|
|
## Research Findings by Category
|
|
|
|
### 1. Production-Ready Models
|
|
|
|
#### Coqui XTTS-v2 (Currently Integrated)
|
|
- **Status:** Company shut down 2024, community-maintained
|
|
- **Quality:** Excellent (4.2-4.5 MOS)
|
|
- **Key Feature:** Zero-shot voice cloning from 6-second samples
|
|
- **Risk:** Upstream archived, needs mirroring
|
|
- **Recommendation:** Continue use, establish mirrors
|
|
- **Documentation:** [docs/models/coqui-tts.md](../models/coqui-tts.md)
|
|
|
|
#### Piper TTS (Currently Integrated)
|
|
- **Status:** Original project abandoned, OHF-Voice fork
|
|
- **Quality:** Good (3.5-4.0 MOS)
|
|
- **Key Feature:** Fastest inference, 100+ voices
|
|
- **Risk:** Fork has no PyPI package
|
|
- **Recommendation:** Vendor code or create PyPI package
|
|
- **Documentation:** [docs/models/piper-tts.md](../models/piper-tts.md)
|
|
|
|
#### Chatterbox (High Priority)
|
|
- **Status:** Active development by Resemble AI
|
|
- **Quality:** Very Good (4.0-4.3 MOS)
|
|
- **Key Feature:** Unique emotion exaggeration control
|
|
- **Risk:** Low - actively maintained
|
|
- **Recommendation:** Integrate for emotion control features
|
|
- **Documentation:** [docs/models/chatterbox.md](../models/chatterbox.md)
|
|
|
|
### 2. Fast & Lightweight Models
|
|
|
|
#### Kokoro TTS
|
|
- **Architecture:** Decoder-only for speed
|
|
- **Performance:** 0.1-0.3x RTF
|
|
- **Best For:** Low-latency applications
|
|
- **Limitation:** Fewer expressive options
|
|
- **Documentation:** [docs/models/kokoro-tts.md](../models/kokoro-tts.md)
|
|
|
|
#### Mimic 3
|
|
- **Size:** 20-50MB per voice
|
|
- **Performance:** 50-100ms latency
|
|
- **Best For:** Privacy-focused, embedded systems
|
|
- **Limitation:** Some robotic elements
|
|
- **Documentation:** [docs/models/mimic3.md](../models/mimic3.md)
|
|
|
|
#### eSpeak NG
|
|
- **Technology:** Formant synthesis (not neural)
|
|
- **Performance:** Extremely fast (<10ms)
|
|
- **Best For:** Accessibility, 100+ languages
|
|
- **Limitation:** Less natural than neural models
|
|
- **Documentation:** [docs/models/espeak-ng.md](../models/espeak-ng.md)
|
|
|
|
### 3. High-Quality Studio Models
|
|
|
|
#### Tortoise TTS
|
|
- **Technology:** Diffusion-based
|
|
- **Quality:** Studio-grade (4.5-4.8 MOS)
|
|
- **Performance:** 2-5 minutes per sentence
|
|
- **Best For:** Offline content creation
|
|
- **Not Suitable For:** Real-time API
|
|
- **Documentation:** [docs/models/tortoise-tts.md](../models/tortoise-tts.md)
|
|
|
|
### 4. Multilingual & Accent Diversity
|
|
|
|
#### Maya1
|
|
- **Origin:** India-based research
|
|
- **Specialization:** Indic languages (Hindi, Tamil, etc.)
|
|
- **Status:** Emerging, early documentation
|
|
- **Best For:** Non-English markets
|
|
- **Documentation:** [docs/models/maya1.md](../models/maya1.md)
|
|
|
|
### 5. Experimental & Cutting Edge
|
|
|
|
#### Step-Audio-EditX (November 2025)
|
|
- **Innovation:** LLM-based audio editing
|
|
- **Approach:** Post-generation refinement
|
|
- **Status:** Very new, experimental
|
|
- **Best For:** Creative audio workflows
|
|
- **Documentation:** [docs/models/step-audio-editx.md](../models/step-audio-editx.md)
|
|
|
|
### 6. Historical / Archived
|
|
|
|
#### Mozilla TTS
|
|
- **Status:** Archived, became Coqui TTS
|
|
- **Historical Significance:** Pioneer in open-source TTS
|
|
- **Current Recommendation:** Use Coqui instead
|
|
- **Documentation:** [docs/models/mozilla-tts.md](../models/mozilla-tts.md)
|
|
|
|
## License Compatibility Analysis
|
|
|
|
### Commercial-Friendly Licenses ✅
|
|
- **Apache-2.0:** Coqui XTTS-v2, Chatterbox, Kokoro, Mimic 3, Tortoise, Step-Audio-EditX
|
|
- **MIT:** Piper TTS, Maya1
|
|
- **MPL-2.0:** Mozilla TTS (permissive with copyleft for modifications)
|
|
|
|
### Restricted Licenses ⚠️
|
|
- **GPL-3.0:** eSpeak NG (copyleft, requires derivative works to be GPL)
|
|
|
|
### Recommendation
|
|
For commercial deployment, prioritize Apache-2.0 and MIT licensed models. eSpeak NG can be used as a service but requires careful licensing consideration for code modifications.
|
|
|
|
## Integration Roadmap
|
|
|
|
### Phase 1: Stabilization (Weeks 1-2)
|
|
- ✅ Fix Piper absolute paths
|
|
- ✅ Create model documentation
|
|
- ✅ Audit repository
|
|
- [ ] Set up model mirror infrastructure
|
|
- [ ] Document all model sources
|
|
|
|
### Phase 2: Quick Wins (Weeks 3-4)
|
|
- [ ] Integrate Silero TTS (actively maintained)
|
|
- [ ] Integrate Chatterbox (emotion control)
|
|
- [ ] Test all models with existing API
|
|
- [ ] Create engine abstraction layer
|
|
|
|
### Phase 3: Advanced Features (Months 2-3)
|
|
- [ ] Integrate StyleTTS2 (best quality)
|
|
- [ ] Add Fish Speech support
|
|
- [ ] Implement voice cloning API endpoint
|
|
- [ ] Add emotion/style control API
|
|
|
|
### Phase 4: Resilience (Months 3-4)
|
|
- [ ] Complete model mirroring to ai.foxhop.net
|
|
- [ ] Archive critical models to Archive.org
|
|
- [ ] Create fallback download logic
|
|
- [ ] Implement automatic mirror selection
|
|
|
|
### Phase 5: Experimental (Months 4+)
|
|
- [ ] Evaluate Maya1 for production
|
|
- [ ] Test Step-Audio-EditX integration
|
|
- [ ] Research Kokoro TTS integration
|
|
- [ ] Implement streaming TTS
|
|
|
|
## Raccoon Mission Priorities
|
|
|
|
### Critical Rescue Operations 🚨
|
|
1. **Coqui XTTS-v2** - Company shut down, repository archived
|
|
- Action: Mirror all weights (1.8GB)
|
|
- Action: Fork repository to uncloseai-xtts
|
|
- Timeline: Immediate
|
|
|
|
2. **Piper TTS** - Original project abandoned
|
|
- Action: Mirror all voices (2GB)
|
|
- Action: Vendor code or create PyPI package
|
|
- Timeline: Week 1-2
|
|
|
|
### High-Value Acquisitions ⭐
|
|
1. **Chatterbox** - Active but could be abandoned
|
|
- Action: Monitor development status
|
|
- Action: Mirror models (1.2GB)
|
|
- Timeline: Month 1
|
|
|
|
2. **Silero TTS** - Active but should be backed up
|
|
- Action: Mirror all language models (500MB)
|
|
- Timeline: Week 2
|
|
|
|
### Research & Watch 👀
|
|
1. **Maya1** - Emerging, evaluate stability
|
|
2. **Kokoro TTS** - New project, monitor adoption
|
|
3. **Step-Audio-EditX** - Experimental, track development
|
|
|
|
### Low Priority 📋
|
|
1. **Tortoise TTS** - Too slow, but preserve for quality
|
|
2. **eSpeak NG** - Actively maintained, not at risk
|
|
3. **Mozilla TTS** - Historical archive only
|
|
|
|
## Storage Requirements
|
|
|
|
### Current Infrastructure
|
|
- Piper voices: ~2GB
|
|
- XTTS v2: ~1.8GB
|
|
- **Total:** ~4GB
|
|
|
|
### Planned Integration
|
|
- Silero models: ~500MB
|
|
- Chatterbox: ~1.2GB
|
|
- StyleTTS2: ~2GB
|
|
- Fish Speech: ~1.5GB
|
|
- Kokoro: ~500MB
|
|
- Maya1: ~1.5GB
|
|
- **Total New:** ~7.2GB
|
|
|
|
### Complete Mirror Strategy
|
|
- Production models: ~4GB
|
|
- Integration candidates: ~7.2GB
|
|
- Archive/backup: ~8GB (duplicates + older versions)
|
|
- **Total Required:** ~20GB
|
|
|
|
## Risk Assessment
|
|
|
|
### High Risk - Immediate Action Required
|
|
| Model | Risk Factor | Mitigation |
|
|
|-------|-------------|------------|
|
|
| Coqui XTTS-v2 | Company defunct, repo archived | Mirror weights, fork code |
|
|
| Piper TTS | Original abandoned, fork unstable | Vendor code, mirror voices |
|
|
|
|
### Medium Risk - Monitor Closely
|
|
| Model | Risk Factor | Mitigation |
|
|
|-------|-------------|------------|
|
|
| Chatterbox | Company-backed, could pivot | Regular backups, monitor status |
|
|
| Tortoise TTS | Low activity | Mirror weights |
|
|
|
|
### Low Risk
|
|
| Model | Status |
|
|
|-------|--------|
|
|
| Silero TTS | Actively maintained |
|
|
| eSpeak NG | Active community |
|
|
| StyleTTS2 | Active research |
|
|
|
|
## Technical Architecture Recommendations
|
|
|
|
### Engine Abstraction Layer
|
|
```python
|
|
class TTSEngine:
|
|
def synthesize(text: str, voice: str, **kwargs) -> bytes
|
|
def get_voices() -> List[Voice]
|
|
def clone_voice(audio_sample: bytes) -> Voice
|
|
def supports_emotion() -> bool
|
|
def supports_streaming() -> bool
|
|
```
|
|
|
|
### Model Selection Strategy
|
|
1. **Fast responses (tts-1):** Piper TTS, Silero
|
|
2. **High quality (tts-1-hd):** Coqui XTTS-v2, StyleTTS2
|
|
3. **Voice cloning:** Coqui XTTS-v2, Chatterbox, Tortoise
|
|
4. **Emotion control:** Chatterbox, Coqui XTTS-v2
|
|
5. **Multilingual:** Coqui XTTS-v2, Maya1, Piper
|
|
6. **Privacy/offline:** Mimic 3, Piper, eSpeak NG
|
|
|
|
## Research Methodology
|
|
|
|
### Evaluation Criteria
|
|
1. **Quality:** MOS scores, naturalness, prosody
|
|
2. **Performance:** RTF, latency, resource usage
|
|
3. **Features:** Voice cloning, emotion control, multilingual
|
|
4. **Maintenance:** Active development, community support
|
|
5. **License:** Commercial compatibility
|
|
6. **Risk:** Project abandonment probability
|
|
7. **Integration:** Ease of deployment, dependencies
|
|
|
|
### Testing Protocol
|
|
1. Install and run basic synthesis
|
|
2. Evaluate audio quality (subjective MOS)
|
|
3. Measure performance (RTF, latency)
|
|
4. Test advanced features (cloning, emotion)
|
|
5. Assess resource requirements (CPU, GPU, RAM)
|
|
6. Review code quality and documentation
|
|
7. Check license compatibility
|
|
|
|
## Community & Ecosystem
|
|
|
|
### Active Communities
|
|
- **Coqui/XTTS:** Large community, multiple forks
|
|
- **Silero:** Active GitHub, regular updates
|
|
- **eSpeak NG:** Accessibility-focused community
|
|
- **Piper:** Rhasspy ecosystem, home automation
|
|
|
|
### At-Risk Projects
|
|
- **Mozilla TTS:** Archived, historical only
|
|
- **Tortoise TTS:** Low activity, mostly complete
|
|
- **Mimic 3:** Mycroft AI restructuring
|
|
|
|
### Emerging Projects
|
|
- **Kokoro TTS:** New, gaining traction
|
|
- **Maya1:** Research project, early stage
|
|
- **Step-Audio-EditX:** Cutting edge, experimental
|
|
|
|
## Conclusion
|
|
|
|
The TTS landscape is rapidly evolving with several high-quality open-source options. However, many projects face abandonment risk, making the Raccoon Mission critical for long-term viability.
|
|
|
|
### Key Takeaways
|
|
1. **Immediate Focus:** Secure Coqui XTTS-v2 and Piper TTS through mirroring
|
|
2. **Quick Wins:** Integrate Chatterbox and Silero for feature diversity
|
|
3. **Quality Goal:** StyleTTS2 for best-in-class naturalness
|
|
4. **Diversity:** Maya1 for non-English markets
|
|
5. **Innovation:** Monitor Step-Audio-EditX for future capabilities
|
|
|
|
### Success Metrics
|
|
- ✅ All critical models mirrored (0/2 complete)
|
|
- 🎯 3+ production engines integrated (2/3 complete)
|
|
- 🎯 Voice cloning API functional (1/1 complete with XTTS)
|
|
- 🎯 Emotion control available (0/1 complete)
|
|
- 🎯 <100ms latency option (1/1 complete with Piper)
|
|
- 🎯 20GB mirror infrastructure (0% complete)
|
|
|
|
---
|
|
|
|
**Raccoon Mission Status:** 🦝 2/10 models rescued and integrated
|
|
**Next Action:** Set up mirror infrastructure and integrate Chatterbox
|
|
**Documentation Maintained By:** uncloseai
|
|
**Last Updated:** 2025-11-09
|