# TTS Models Research Overview

**Last Updated:** 2025-11-09
**Raccoon Mission Status:** 🦝 Active Rescue Operations

## Executive Summary

This document provides a comprehensive overview of open-source Text-to-Speech (TTS) models researched for integration into uncloseai-speech. Our "Raccoon Mission" aims to rescue abandoned and at-risk TTS projects, ensuring their long-term preservation and availability.

## Model Inventory

### Currently Integrated ✅

| Model | Status | Quality | Speed | Use Case |
|-------|--------|---------|-------|----------|
| [Piper TTS](../models/piper-tts.md) | Production | Good | Fast (0.05x RTF) | tts-1 (fast responses) |
| [Coqui XTTS-v2](../models/coqui-tts.md) | Production | Excellent | Medium (0.3x RTF) | tts-1-hd (high quality) |

### High Priority Candidates 🎯

| Model | Priority | Key Strengths | Integration Effort |
|-------|----------|---------------|-------------------|
| [Chatterbox](../models/chatterbox.md) | High | Emotion control, 23 languages | Medium |
| [Kokoro TTS](../models/kokoro-tts.md) | Medium | Fast, Apache-2.0 licensed | Medium |
| Silero TTS | High | Active maintenance, small models | Low |
| StyleTTS2 | High | Best quality/prosody | High |

### Specialized Models 🔬

| Model | Specialization | Integration Status |
|-------|----------------|-------------------|
| [Mimic 3](../models/mimic3.md) | Privacy-focused, offline | Candidate |
| [eSpeak NG](../models/espeak-ng.md) | 100+ languages, accessibility | Niche |
| [Maya1](../models/maya1.md) | Indic languages, diverse accents | Research |
| [Step-Audio-EditX](../models/step-audio-editx.md) | Post-generation editing | Experimental |

### Low Priority / Archived 📦

| Model | Reason | Status |
|-------|--------|--------|
| [Mozilla TTS](../models/mozilla-tts.md) | Superseded by Coqui | Archived |
| [Tortoise TTS](../models/tortoise-tts.md) | Too slow for production | Low priority |

## Model Comparison Matrix

### Performance Characteristics

| Model | RTF | Quality (MOS) | Languages | License | Model Size |
|-------|-----|---------------|-----------|---------|------------|
| Piper TTS | 0.05x | 3.5-4.0 | 50+ | MIT | ~100MB |
| Coqui XTTS-v2 | 0.3x | 4.2-4.5 | 20+ | Apache-2.0 | ~1.8GB |
| Chatterbox | 0.2x | 4.0-4.3 | 23 | Apache-2.0 | ~1.2GB |
| Kokoro TTS | 0.1-0.3x | 4.0-4.2 | Limited | Apache-2.0 | ~500MB |
| Mimic 3 | 0.1x | 3.8-4.0 | Multiple | Apache-2.0 | 20-50MB |
| eSpeak NG | <0.01x | 2.5-3.0 | 100+ | GPL-3.0 | <10MB |
| Maya1 | Unknown | 4.0+ | 10+ Indic | MIT | ~1.5GB |
| Tortoise TTS | 10-30x | 4.5-4.8 | English | Apache-2.0 | ~2GB |
| Step-Audio-EditX | Variable | N/A | Multiple | Apache-2.0 | Unknown |
| Mozilla TTS | 0.5x | 3.8-4.0 | Limited | MPL-2.0 | ~500MB |

**RTF = Real-Time Factor** (lower is faster, 1.0 = real-time)
**MOS = Mean Opinion Score** (1-5 scale, higher is better)

### Feature Matrix

| Model | Voice Cloning | Emotion Control | Multilingual | Offline | GPU Required |
|-------|---------------|----------------|--------------|---------|--------------|
| Piper TTS | ❌ | ❌ | ✅ | ✅ | ❌ |
| Coqui XTTS-v2 | ✅ | ✅ | ✅ | ✅ | Recommended |
| Chatterbox | ✅ | ✅ (unique) | ✅ | ✅ | Recommended |
| Kokoro TTS | ✅ | ✅ | Limited | ✅ | Optional |
| Mimic 3 | ❌ | Limited | ✅ | ✅ | ❌ |
| eSpeak NG | ❌ | ❌ | ✅ | ✅ | ❌ |
| Maya1 | ✅ | Unknown | ✅ | ✅ | Yes |
| Tortoise TTS | ✅ | ✅ | Limited | ✅ | Yes |
| Step-Audio-EditX | N/A | ✅ (editing) | ✅ | ✅ | Yes |
| Mozilla TTS | Limited | ❌ | Limited | ✅ | Recommended |

## Research Findings by Category

### 1. Production-Ready Models

#### Coqui XTTS-v2 (Currently Integrated)
- **Status:** Company shut down 2024, community-maintained
- **Quality:** Excellent (4.2-4.5 MOS)
- **Key Feature:** Zero-shot voice cloning from 6-second samples
- **Risk:** Upstream archived, needs mirroring
- **Recommendation:** Continue use, establish mirrors
- **Documentation:** [docs/models/coqui-tts.md](../models/coqui-tts.md)

#### Piper TTS (Currently Integrated)
- **Status:** Original project abandoned, OHF-Voice fork
- **Quality:** Good (3.5-4.0 MOS)
- **Key Feature:** Fastest inference, 100+ voices
- **Risk:** Fork has no PyPI package
- **Recommendation:** Vendor code or create PyPI package
- **Documentation:** [docs/models/piper-tts.md](../models/piper-tts.md)

#### Chatterbox (High Priority)
- **Status:** Active development by Resemble AI
- **Quality:** Very Good (4.0-4.3 MOS)
- **Key Feature:** Unique emotion exaggeration control
- **Risk:** Low - actively maintained
- **Recommendation:** Integrate for emotion control features
- **Documentation:** [docs/models/chatterbox.md](../models/chatterbox.md)

### 2. Fast & Lightweight Models

#### Kokoro TTS
- **Architecture:** Decoder-only for speed
- **Performance:** 0.1-0.3x RTF
- **Best For:** Low-latency applications
- **Limitation:** Fewer expressive options
- **Documentation:** [docs/models/kokoro-tts.md](../models/kokoro-tts.md)

#### Mimic 3
- **Size:** 20-50MB per voice
- **Performance:** 50-100ms latency
- **Best For:** Privacy-focused, embedded systems
- **Limitation:** Some robotic elements
- **Documentation:** [docs/models/mimic3.md](../models/mimic3.md)

#### eSpeak NG
- **Technology:** Formant synthesis (not neural)
- **Performance:** Extremely fast (<10ms)
- **Best For:** Accessibility, 100+ languages
- **Limitation:** Less natural than neural models
- **Documentation:** [docs/models/espeak-ng.md](../models/espeak-ng.md)

### 3. High-Quality Studio Models

#### Tortoise TTS
- **Technology:** Diffusion-based
- **Quality:** Studio-grade (4.5-4.8 MOS)
- **Performance:** 2-5 minutes per sentence
- **Best For:** Offline content creation
- **Not Suitable For:** Real-time API
- **Documentation:** [docs/models/tortoise-tts.md](../models/tortoise-tts.md)

### 4. Multilingual & Accent Diversity

#### Maya1
- **Origin:** India-based research
- **Specialization:** Indic languages (Hindi, Tamil, etc.)
- **Status:** Emerging, early documentation
- **Best For:** Non-English markets
- **Documentation:** [docs/models/maya1.md](../models/maya1.md)

### 5. Experimental & Cutting Edge

#### Step-Audio-EditX (November 2025)
- **Innovation:** LLM-based audio editing
- **Approach:** Post-generation refinement
- **Status:** Very new, experimental
- **Best For:** Creative audio workflows
- **Documentation:** [docs/models/step-audio-editx.md](../models/step-audio-editx.md)

### 6. Historical / Archived

#### Mozilla TTS
- **Status:** Archived, became Coqui TTS
- **Historical Significance:** Pioneer in open-source TTS
- **Current Recommendation:** Use Coqui instead
- **Documentation:** [docs/models/mozilla-tts.md](../models/mozilla-tts.md)

## License Compatibility Analysis

### Commercial-Friendly Licenses ✅
- **Apache-2.0:** Coqui XTTS-v2, Chatterbox, Kokoro, Mimic 3, Tortoise, Step-Audio-EditX
- **MIT:** Piper TTS, Maya1
- **MPL-2.0:** Mozilla TTS (permissive with copyleft for modifications)

### Restricted Licenses ⚠️
- **GPL-3.0:** eSpeak NG (copyleft, requires derivative works to be GPL)

### Recommendation
For commercial deployment, prioritize Apache-2.0 and MIT licensed models. eSpeak NG can be used as a service but requires careful licensing consideration for code modifications.

## Integration Roadmap

### Phase 1: Stabilization (Weeks 1-2)
- ✅ Fix Piper absolute paths
- ✅ Create model documentation
- ✅ Audit repository
- [ ] Set up model mirror infrastructure
- [ ] Document all model sources

### Phase 2: Quick Wins (Weeks 3-4)
- [ ] Integrate Silero TTS (actively maintained)
- [ ] Integrate Chatterbox (emotion control)
- [ ] Test all models with existing API
- [ ] Create engine abstraction layer

### Phase 3: Advanced Features (Months 2-3)
- [ ] Integrate StyleTTS2 (best quality)
- [ ] Add Fish Speech support
- [ ] Implement voice cloning API endpoint
- [ ] Add emotion/style control API

### Phase 4: Resilience (Months 3-4)
- [ ] Complete model mirroring to ai.foxhop.net
- [ ] Archive critical models to Archive.org
- [ ] Create fallback download logic
- [ ] Implement automatic mirror selection

### Phase 5: Experimental (Months 4+)
- [ ] Evaluate Maya1 for production
- [ ] Test Step-Audio-EditX integration
- [ ] Research Kokoro TTS integration
- [ ] Implement streaming TTS

## Raccoon Mission Priorities

### Critical Rescue Operations 🚨
1. **Coqui XTTS-v2** - Company shut down, repository archived
   - Action: Mirror all weights (1.8GB)
   - Action: Fork repository to uncloseai-xtts
   - Timeline: Immediate

2. **Piper TTS** - Original project abandoned
   - Action: Mirror all voices (2GB)
   - Action: Vendor code or create PyPI package
   - Timeline: Week 1-2

### High-Value Acquisitions ⭐
1. **Chatterbox** - Active but could be abandoned
   - Action: Monitor development status
   - Action: Mirror models (1.2GB)
   - Timeline: Month 1

2. **Silero TTS** - Active but should be backed up
   - Action: Mirror all language models (500MB)
   - Timeline: Week 2

### Research & Watch 👀
1. **Maya1** - Emerging, evaluate stability
2. **Kokoro TTS** - New project, monitor adoption
3. **Step-Audio-EditX** - Experimental, track development

### Low Priority 📋
1. **Tortoise TTS** - Too slow, but preserve for quality
2. **eSpeak NG** - Actively maintained, not at risk
3. **Mozilla TTS** - Historical archive only

## Storage Requirements

### Current Infrastructure
- Piper voices: ~2GB
- XTTS v2: ~1.8GB
- **Total:** ~4GB

### Planned Integration
- Silero models: ~500MB
- Chatterbox: ~1.2GB
- StyleTTS2: ~2GB
- Fish Speech: ~1.5GB
- Kokoro: ~500MB
- Maya1: ~1.5GB
- **Total New:** ~7.2GB

### Complete Mirror Strategy
- Production models: ~4GB
- Integration candidates: ~7.2GB
- Archive/backup: ~8GB (duplicates + older versions)
- **Total Required:** ~20GB

## Risk Assessment

### High Risk - Immediate Action Required
| Model | Risk Factor | Mitigation |
|-------|-------------|------------|
| Coqui XTTS-v2 | Company defunct, repo archived | Mirror weights, fork code |
| Piper TTS | Original abandoned, fork unstable | Vendor code, mirror voices |

### Medium Risk - Monitor Closely
| Model | Risk Factor | Mitigation |
|-------|-------------|------------|
| Chatterbox | Company-backed, could pivot | Regular backups, monitor status |
| Tortoise TTS | Low activity | Mirror weights |

### Low Risk
| Model | Status |
|-------|--------|
| Silero TTS | Actively maintained |
| eSpeak NG | Active community |
| StyleTTS2 | Active research |

## Technical Architecture Recommendations

### Engine Abstraction Layer
```python
class TTSEngine:
    def synthesize(text: str, voice: str, **kwargs) -> bytes
    def get_voices() -> List[Voice]
    def clone_voice(audio_sample: bytes) -> Voice
    def supports_emotion() -> bool
    def supports_streaming() -> bool
```

### Model Selection Strategy
1. **Fast responses (tts-1):** Piper TTS, Silero
2. **High quality (tts-1-hd):** Coqui XTTS-v2, StyleTTS2
3. **Voice cloning:** Coqui XTTS-v2, Chatterbox, Tortoise
4. **Emotion control:** Chatterbox, Coqui XTTS-v2
5. **Multilingual:** Coqui XTTS-v2, Maya1, Piper
6. **Privacy/offline:** Mimic 3, Piper, eSpeak NG

## Research Methodology

### Evaluation Criteria
1. **Quality:** MOS scores, naturalness, prosody
2. **Performance:** RTF, latency, resource usage
3. **Features:** Voice cloning, emotion control, multilingual
4. **Maintenance:** Active development, community support
5. **License:** Commercial compatibility
6. **Risk:** Project abandonment probability
7. **Integration:** Ease of deployment, dependencies

### Testing Protocol
1. Install and run basic synthesis
2. Evaluate audio quality (subjective MOS)
3. Measure performance (RTF, latency)
4. Test advanced features (cloning, emotion)
5. Assess resource requirements (CPU, GPU, RAM)
6. Review code quality and documentation
7. Check license compatibility

## Community & Ecosystem

### Active Communities
- **Coqui/XTTS:** Large community, multiple forks
- **Silero:** Active GitHub, regular updates
- **eSpeak NG:** Accessibility-focused community
- **Piper:** Rhasspy ecosystem, home automation

### At-Risk Projects
- **Mozilla TTS:** Archived, historical only
- **Tortoise TTS:** Low activity, mostly complete
- **Mimic 3:** Mycroft AI restructuring

### Emerging Projects
- **Kokoro TTS:** New, gaining traction
- **Maya1:** Research project, early stage
- **Step-Audio-EditX:** Cutting edge, experimental

## Conclusion

The TTS landscape is rapidly evolving with several high-quality open-source options. However, many projects face abandonment risk, making the Raccoon Mission critical for long-term viability.

### Key Takeaways
1. **Immediate Focus:** Secure Coqui XTTS-v2 and Piper TTS through mirroring
2. **Quick Wins:** Integrate Chatterbox and Silero for feature diversity
3. **Quality Goal:** StyleTTS2 for best-in-class naturalness
4. **Diversity:** Maya1 for non-English markets
5. **Innovation:** Monitor Step-Audio-EditX for future capabilities

### Success Metrics
- ✅ All critical models mirrored (0/2 complete)
- 🎯 3+ production engines integrated (2/3 complete)
- 🎯 Voice cloning API functional (1/1 complete with XTTS)
- 🎯 Emotion control available (0/1 complete)
- 🎯 <100ms latency option (1/1 complete with Piper)
- 🎯 20GB mirror infrastructure (0% complete)

---

**Raccoon Mission Status:** 🦝 2/10 models rescued and integrated
**Next Action:** Set up mirror infrastructure and integrate Chatterbox
**Documentation Maintained By:** uncloseai
**Last Updated:** 2025-11-09