uncloseai-speech/docs/research/tts-models-overview.md

TTS Models Research Overview

Last Updated: 2025-11-09 Raccoon Mission Status: 🦝 Active Rescue Operations

Executive Summary

This document provides a comprehensive overview of open-source Text-to-Speech (TTS) models researched for integration into uncloseai-speech. Our "Raccoon Mission" aims to rescue abandoned and at-risk TTS projects, ensuring their long-term preservation and availability.

Model Inventory

Currently Integrated ✅

Model	Status	Quality	Speed	Use Case
Piper TTS	Production	Good	Fast (0.05x RTF)	tts-1 (fast responses)
Coqui XTTS-v2	Production	Excellent	Medium (0.3x RTF)	tts-1-hd (high quality)

High Priority Candidates 🎯

Model	Priority	Key Strengths	Integration Effort
Chatterbox	High	Emotion control, 23 languages	Medium
Kokoro TTS	Medium	Fast, Apache-2.0 licensed	Medium
Silero TTS	High	Active maintenance, small models	Low
StyleTTS2	High	Best quality/prosody	High

Specialized Models 🔬

Model	Specialization	Integration Status
Mimic 3	Privacy-focused, offline	Candidate
eSpeak NG	100+ languages, accessibility	Niche
Maya1	Indic languages, diverse accents	Research
Step-Audio-EditX	Post-generation editing	Experimental

Low Priority / Archived 📦

Model	Reason	Status
Mozilla TTS	Superseded by Coqui	Archived
Tortoise TTS	Too slow for production	Low priority

Model Comparison Matrix

Performance Characteristics

Model	RTF	Quality (MOS)	Languages	License	Model Size
Piper TTS	0.05x	3.5-4.0	50+	MIT	~100MB
Coqui XTTS-v2	0.3x	4.2-4.5	20+	Apache-2.0	~1.8GB
Chatterbox	0.2x	4.0-4.3	23	Apache-2.0	~1.2GB
Kokoro TTS	0.1-0.3x	4.0-4.2	Limited	Apache-2.0	~500MB
Mimic 3	0.1x	3.8-4.0	Multiple	Apache-2.0	20-50MB
eSpeak NG	<0.01x	2.5-3.0	100+	GPL-3.0	<10MB
Maya1	Unknown	4.0+	10+ Indic	MIT	~1.5GB
Tortoise TTS	10-30x	4.5-4.8	English	Apache-2.0	~2GB
Step-Audio-EditX	Variable	N/A	Multiple	Apache-2.0	Unknown
Mozilla TTS	0.5x	3.8-4.0	Limited	MPL-2.0	~500MB

RTF = Real-Time Factor (lower is faster, 1.0 = real-time) MOS = Mean Opinion Score (1-5 scale, higher is better)

Feature Matrix

Model	Voice Cloning	Emotion Control	Multilingual	Offline	GPU Required
Piper TTS	❌	❌	✅	✅	❌
Coqui XTTS-v2	✅	✅	✅	✅	Recommended
Chatterbox	✅	✅ (unique)	✅	✅	Recommended
Kokoro TTS	✅	✅	Limited	✅	Optional
Mimic 3	❌	Limited	✅	✅	❌
eSpeak NG	❌	❌	✅	✅	❌
Maya1	✅	Unknown	✅	✅	Yes
Tortoise TTS	✅	✅	Limited	✅	Yes
Step-Audio-EditX	N/A	✅ (editing)	✅	✅	Yes
Mozilla TTS	Limited	❌	Limited	✅	Recommended

Research Findings by Category

1. Production-Ready Models

Coqui XTTS-v2 (Currently Integrated)

• Status: Company shut down 2024, community-maintained
• Quality: Excellent (4.2-4.5 MOS)
• Key Feature: Zero-shot voice cloning from 6-second samples
• Risk: Upstream archived, needs mirroring
• Recommendation: Continue use, establish mirrors
• Documentation: docs/models/coqui-tts.md

Piper TTS (Currently Integrated)

• Status: Original project abandoned, OHF-Voice fork
• Quality: Good (3.5-4.0 MOS)
• Key Feature: Fastest inference, 100+ voices
• Risk: Fork has no PyPI package
• Recommendation: Vendor code or create PyPI package
• Documentation: docs/models/piper-tts.md

Chatterbox (High Priority)

• Status: Active development by Resemble AI
• Quality: Very Good (4.0-4.3 MOS)
• Key Feature: Unique emotion exaggeration control
• Risk: Low - actively maintained
• Recommendation: Integrate for emotion control features
• Documentation: docs/models/chatterbox.md

2. Fast & Lightweight Models

Kokoro TTS

• Architecture: Decoder-only for speed
• Performance: 0.1-0.3x RTF
• Best For: Low-latency applications
• Limitation: Fewer expressive options
• Documentation: docs/models/kokoro-tts.md

Mimic 3

• Size: 20-50MB per voice
• Performance: 50-100ms latency
• Best For: Privacy-focused, embedded systems
• Limitation: Some robotic elements
• Documentation: docs/models/mimic3.md

eSpeak NG

• Technology: Formant synthesis (not neural)
• Performance: Extremely fast (<10ms)
• Best For: Accessibility, 100+ languages
• Limitation: Less natural than neural models
• Documentation: docs/models/espeak-ng.md

3. High-Quality Studio Models

Tortoise TTS

• Technology: Diffusion-based
• Quality: Studio-grade (4.5-4.8 MOS)
• Performance: 2-5 minutes per sentence
• Best For: Offline content creation
• Not Suitable For: Real-time API
• Documentation: docs/models/tortoise-tts.md

4. Multilingual & Accent Diversity

Maya1

• Origin: India-based research
• Specialization: Indic languages (Hindi, Tamil, etc.)
• Status: Emerging, early documentation
• Best For: Non-English markets
• Documentation: docs/models/maya1.md

5. Experimental & Cutting Edge

Step-Audio-EditX (November 2025)

• Innovation: LLM-based audio editing
• Approach: Post-generation refinement
• Status: Very new, experimental
• Best For: Creative audio workflows
• Documentation: docs/models/step-audio-editx.md

6. Historical / Archived

Mozilla TTS

• Status: Archived, became Coqui TTS
• Historical Significance: Pioneer in open-source TTS
• Current Recommendation: Use Coqui instead
• Documentation: docs/models/mozilla-tts.md

License Compatibility Analysis

Commercial-Friendly Licenses ✅

• Apache-2.0: Coqui XTTS-v2, Chatterbox, Kokoro, Mimic 3, Tortoise, Step-Audio-EditX
• MIT: Piper TTS, Maya1
• MPL-2.0: Mozilla TTS (permissive with copyleft for modifications)

Restricted Licenses ⚠️

• GPL-3.0: eSpeak NG (copyleft, requires derivative works to be GPL)

Recommendation

For commercial deployment, prioritize Apache-2.0 and MIT licensed models. eSpeak NG can be used as a service but requires careful licensing consideration for code modifications.

Integration Roadmap

Phase 1: Stabilization (Weeks 1-2)

• ✅ Fix Piper absolute paths
• ✅ Create model documentation
• ✅ Audit repository
• Set up model mirror infrastructure
• Document all model sources

Phase 2: Quick Wins (Weeks 3-4)

• Integrate Silero TTS (actively maintained)
• Integrate Chatterbox (emotion control)
• Test all models with existing API
• Create engine abstraction layer

Phase 3: Advanced Features (Months 2-3)

• Integrate StyleTTS2 (best quality)
• Add Fish Speech support
• Implement voice cloning API endpoint
• Add emotion/style control API

Phase 4: Resilience (Months 3-4)

• Complete model mirroring to ai.foxhop.net
• Archive critical models to Archive.org
• Create fallback download logic
• Implement automatic mirror selection

Phase 5: Experimental (Months 4+)

• Evaluate Maya1 for production
• Test Step-Audio-EditX integration
• Research Kokoro TTS integration
• Implement streaming TTS

Raccoon Mission Priorities

Critical Rescue Operations 🚨

1. Coqui XTTS-v2 - Company shut down, repository archived

   • Action: Mirror all weights (1.8GB)
   • Action: Fork repository to uncloseai-xtts
   • Timeline: Immediate

2. Piper TTS - Original project abandoned

   • Action: Mirror all voices (2GB)
   • Action: Vendor code or create PyPI package
   • Timeline: Week 1-2

High-Value Acquisitions ⭐

1. Chatterbox - Active but could be abandoned

   • Action: Monitor development status
   • Action: Mirror models (1.2GB)
   • Timeline: Month 1

2. Silero TTS - Active but should be backed up

   • Action: Mirror all language models (500MB)
   • Timeline: Week 2

Research & Watch 👀

1. Maya1 - Emerging, evaluate stability
2. Kokoro TTS - New project, monitor adoption
3. Step-Audio-EditX - Experimental, track development

Low Priority 📋

1. Tortoise TTS - Too slow, but preserve for quality
2. eSpeak NG - Actively maintained, not at risk
3. Mozilla TTS - Historical archive only

Storage Requirements

Current Infrastructure

• Piper voices: ~2GB
• XTTS v2: ~1.8GB
• Total: ~4GB

Planned Integration

• Silero models: ~500MB
• Chatterbox: ~1.2GB
• StyleTTS2: ~2GB
• Fish Speech: ~1.5GB
• Kokoro: ~500MB
• Maya1: ~1.5GB
• Total New: ~7.2GB

Complete Mirror Strategy

• Production models: ~4GB
• Integration candidates: ~7.2GB
• Archive/backup: ~8GB (duplicates + older versions)
• Total Required: ~20GB

Risk Assessment

High Risk - Immediate Action Required

Model	Risk Factor	Mitigation
Coqui XTTS-v2	Company defunct, repo archived	Mirror weights, fork code
Piper TTS	Original abandoned, fork unstable	Vendor code, mirror voices

Medium Risk - Monitor Closely

Model	Risk Factor	Mitigation
Chatterbox	Company-backed, could pivot	Regular backups, monitor status
Tortoise TTS	Low activity	Mirror weights

Low Risk

Model	Status
Silero TTS	Actively maintained
eSpeak NG	Active community
StyleTTS2	Active research

Technical Architecture Recommendations

Engine Abstraction Layer

class TTSEngine:
    def synthesize(text: str, voice: str, **kwargs) -> bytes
    def get_voices() -> List[Voice]
    def clone_voice(audio_sample: bytes) -> Voice
    def supports_emotion() -> bool
    def supports_streaming() -> bool

Model Selection Strategy

1. Fast responses (tts-1): Piper TTS, Silero
2. High quality (tts-1-hd): Coqui XTTS-v2, StyleTTS2
3. Voice cloning: Coqui XTTS-v2, Chatterbox, Tortoise
4. Emotion control: Chatterbox, Coqui XTTS-v2
5. Multilingual: Coqui XTTS-v2, Maya1, Piper
6. Privacy/offline: Mimic 3, Piper, eSpeak NG

Research Methodology

Evaluation Criteria

1. Quality: MOS scores, naturalness, prosody
2. Performance: RTF, latency, resource usage
3. Features: Voice cloning, emotion control, multilingual
4. Maintenance: Active development, community support
5. License: Commercial compatibility
6. Risk: Project abandonment probability
7. Integration: Ease of deployment, dependencies

Testing Protocol

1. Install and run basic synthesis
2. Evaluate audio quality (subjective MOS)
3. Measure performance (RTF, latency)
4. Test advanced features (cloning, emotion)
5. Assess resource requirements (CPU, GPU, RAM)
6. Review code quality and documentation
7. Check license compatibility

Community & Ecosystem

Active Communities

• Coqui/XTTS: Large community, multiple forks
• Silero: Active GitHub, regular updates
• eSpeak NG: Accessibility-focused community
• Piper: Rhasspy ecosystem, home automation

At-Risk Projects

• Mozilla TTS: Archived, historical only
• Tortoise TTS: Low activity, mostly complete
• Mimic 3: Mycroft AI restructuring

Emerging Projects

• Kokoro TTS: New, gaining traction
• Maya1: Research project, early stage
• Step-Audio-EditX: Cutting edge, experimental

Conclusion

The TTS landscape is rapidly evolving with several high-quality open-source options. However, many projects face abandonment risk, making the Raccoon Mission critical for long-term viability.

Key Takeaways

1. Immediate Focus: Secure Coqui XTTS-v2 and Piper TTS through mirroring
2. Quick Wins: Integrate Chatterbox and Silero for feature diversity
3. Quality Goal: StyleTTS2 for best-in-class naturalness
4. Diversity: Maya1 for non-English markets
5. Innovation: Monitor Step-Audio-EditX for future capabilities

Success Metrics

• ✅ All critical models mirrored (0/2 complete)
• 🎯 3+ production engines integrated (2/3 complete)
• 🎯 Voice cloning API functional (1/1 complete with XTTS)
• 🎯 Emotion control available (0/1 complete)
• 🎯 <100ms latency option (1/1 complete with Piper)
• 🎯 20GB mirror infrastructure (0% complete)

---

Raccoon Mission Status: 🦝 2/10 models rescued and integrated Next Action: Set up mirror infrastructure and integrate Chatterbox Documentation Maintained By: uncloseai Last Updated: 2025-11-09