uncloseai-speech/docs/MODELS.md

# TTS Models and Engines

**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into UncloseAI Speech

This document tracks all TTS engines under consideration for integration. Each engine is evaluated for:
- License compatibility (AGPL-friendly)
- Quality and speed
- Maintenance status (active or abandoned)
- Integration effort

## Currently Integrated

### 1. Piper TTS ✅

**Status:** Working with absolute paths
**License:** MIT
**Original Project:** rhasspy/piper (abandoned)
**Fork:** OHF-Voice/piper1-gpl v1.3.0
**Current Package:** PyPI `piper-tts>=1.2.0`
**Repository:** https://github.com/rhasspy/piper
**Model Hub:** https://huggingface.co/rhasspy/piper-voices

**Description:**
Fast, local neural text-to-speech engine using ONNX runtime. Originally created by Rhasspy for voice assistants, now community-maintained. One of the most widely-deployed open-source TTS engines.

**Key Features:**
- Fast CPU-based neural TTS
- ~100+ high-quality voices across 40+ languages
- Multilingual support (English, Spanish, French, German, Italian, Russian, Polish, Ukrainian, Chinese, Japanese, Korean, and many more)
- ONNX runtime for efficient inference
- Low memory footprint (~100MB per voice)
- No GPU required
- Production-ready quality

**Model Source:**
- HuggingFace: `rhasspy/piper-voices`
- Direct download: `https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/`
- Over 100 voice models available
- Multiple quality levels (low/medium/high)

**Integration:**
- Used for `tts-1` model (fast, good quality)
- Models stored in `/app/voices/en/en_US/libritts_r/medium/`
- Configuration via absolute paths in `voice_to_speaker.yaml`
- Download with: `make voices-piper`

**Example Config:**
```yaml
tts-1:
  alloy:
    model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
    speaker: 79
```

**Performance:**
- Speed: ~0.05x RTF (real-time factor)
- Memory: 100-200MB per model
- Latency: <100ms for short sentences

**Raccoon Notes:**
- Original rhasspy project abandoned by creator
- OHF-Voice fork has no PyPI package
- Community maintaining model repository on HuggingFace
- Mirror all voices to prevent HuggingFace dependency
- Consider creating uncloseai-piper fork for long-term stability

**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Production-ready, widely used)

---

### 2. Coqui TTS (XTTS v2) ✅

**Status:** Integrated as tts-1-hd
**License:** MPL-2.0 / Apache-2.0 (model-dependent)
**Original Project:** coqui-ai/TTS (company shut down, archived)
**Current Package:** PyPI `coqui-tts[languages]`
**Repository:** https://github.com/coqui-ai/TTS
**Model Hub:** https://huggingface.co/coqui/XTTS-v2

**Description:**
Professional-grade multilingual TTS with voice cloning capabilities. Originally developed by Coqui AI (a commercial venture spun out of Mozilla TTS), now community-maintained after company shutdown in 2024. XTTS v2 is the flagship model.

**Key Features:**
- High-quality multilingual TTS (16+ languages)
- Voice cloning from 6+ second audio samples
- Zero-shot voice conversion
- Emotional prosody control
- Streaming TTS support
- GPU accelerated (NVIDIA/ROCm)
- Fine-tuning capabilities
- ~1.8GB model size

**Languages:**
English, Spanish, French, German, Italian, Portuguese, Polish, Turkish, Russian, Dutch, Czech, Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi

**Model Source:**
- HuggingFace: `coqui/XTTS-v2`
- Auto-downloaded on first use
- Pre-trained model: ~1.8GB
- Speaker embeddings: user-provided WAV files

**Integration:**
- Used for `tts-1-hd` model (slower, high quality)
- Voice cloning with custom WAV samples
- Language auto-detection with `langdetect`
- Download speaker samples with: `make voices-xtts`

**Example Config:**
```yaml
tts-1-hd:
  alloy:
    model: xtts
    speaker: /app/voices/alloy.wav
    language: en
```

**Performance:**
- Speed: ~0.3x RTF (GPU), ~1.5x RTF (CPU)
- Memory: 2GB GPU VRAM / 4GB RAM (CPU)
- Latency: 1-5 seconds for first chunk
- Quality: Excellent, human-like prosody

**Raccoon Notes:**
- Coqui company shut down in 2024, repository archived
- Repository still works perfectly, code is stable
- Community forks emerging (XTTS-v2 continuation projects)
- Must mirror XTTS-v2 weights before they disappear from HuggingFace
- High priority to fork as uncloseai-xtts for long-term maintenance
- Large, active community still using it

**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Best quality voice cloning, critical to preserve)

---

## High Priority Integration Targets

### 3. Mozilla TTS 🎯

**Status:** NOT INTEGRATED - HISTORICAL REFERENCE
**License:** Mozilla Public License 2.0
**Original Project:** mozilla/TTS (archived, became Coqui)
**Repository:** https://github.com/mozilla/TTS

**Description:**
Mozilla's original text-to-speech engine, launched as part of Project Common Voice initiative. Archived in 2021 when team spun out to form Coqui AI. Historical predecessor to Coqui TTS.

**Key Features:**
- Multiple TTS architectures (Tacotron, Glow-TTS, etc.)
- Multi-speaker capabilities
- Voice conversion
- Attention mechanisms for alignment
- Neural vocoder support (WaveGrad, MelGAN, etc.)

**Raccoon Notes:**
- Fully superseded by Coqui TTS (XTTS v2)
- No unique capabilities beyond what Coqui offers
- Outdated architecture compared to modern engines
- Historical importance: pioneered open-source neural TTS at Mozilla
- Code still available for research purposes

**Integration Decision:** Skip in favor of Coqui TTS, which is the direct successor with better quality and features.

**Raccoon Priority:** ⛔ (Skip - use Coqui XTTS v2 instead)

---

### 4. Chatterbox 🎯

**Status:** NOT INTEGRATED - HIGH PRIORITY
**License:** Apache-2.0
**Project:** chatterbox-ai/chatterbox (community project)
**Repository:** https://github.com/chatterbox-ai/chatterbox

**Description:**
Community-driven voice assistant TTS framework focused on privacy and offline operation. Designed as a Mycroft alternative with modern architecture.

**Key Features:**
- Privacy-first, fully offline
- Plugin architecture for multiple TTS backends
- Wake word detection integration
- Voice assistant optimized (low latency)
- Multiple voice options
- Lightweight deployment

**Raccoon Notes:**
- Active community development
- Could integrate as backend engine provider
- Focuses on voice assistant use case (similar to our API goals)
- May provide additional voice models
- Needs investigation for model availability

**Integration Effort:** 4-6 hours (needs research)

**Raccoon Priority:** ⭐⭐⭐ (Interesting for voice assistant features)

---

### 5. Mimic 3 🎯

**Status:** NOT INTEGRATED - MEDIUM PRIORITY
**License:** Apache-2.0
**Project:** MycroftAI/mimic3 (Mycroft discontinued)
**Repository:** https://github.com/MycroftAI/mimic3
**Model Hub:** https://huggingface.co/mycroftai

**Description:**
Mycroft AI's third-generation TTS engine, based on VITS architecture. Developed before Mycroft's shutdown in 2023. Uses neural TTS with high-quality voices.

**Key Features:**
- VITS-based neural TTS
- Multiple languages (English, German, French, Spanish, Italian, Dutch, Russian, etc.)
- ONNX runtime for fast inference
- Offline-capable
- Multiple voices per language
- Low resource requirements

**Model Source:**
- HuggingFace: `mycroftai/mimic3`
- Pre-built ONNX models
- Voice models still available

**Raccoon Notes:**
- Mycroft company shut down in 2023
- Models still hosted on HuggingFace
- VITS architecture is proven and efficient
- Similar to Piper but different model training
- Could offer additional voice variety
- Risk: HuggingFace models may disappear

**Integration Effort:** 3-5 hours

**Raccoon Priority:** ⭐⭐⭐⭐ (Good quality, at-risk from Mycroft shutdown)

---

### 6. eSpeak NG 🎯

**Status:** NOT INTEGRATED - LEGACY REFERENCE
**License:** GPL-3.0
**Project:** espeak-ng/espeak-ng (actively maintained)
**Repository:** https://github.com/espeak-ng/espeak-ng

**Description:**
Classic formant synthesis TTS engine. Not neural, but incredibly lightweight and supports 100+ languages. The "eSpeak Next Generation" fork is actively maintained. Used in accessibility tools worldwide.

**Key Features:**
- 100+ languages supported
- Tiny footprint (<10MB)
- No model files needed (rule-based)
- Real-time synthesis
- Highly portable (embedded devices)
- SSML support
- IPA phoneme output

**Raccoon Notes:**
- NOT neural TTS - uses formant synthesis (robotic sound)
- Quality much lower than neural models
- Historical importance: accessibility standard
- Useful fallback for unsupported languages
- GPL-3.0 license compatible with AGPL
- Could serve as pronunciation engine for neural TTS

**Integration Decision:** Low priority for main TTS, but could use for phoneme generation or ultra-low-resource fallback.

**Raccoon Priority:** ⭐⭐ (Useful as fallback, not primary TTS)

---

### 7. Kokoro TTS 🎯

**Status:** NOT INTEGRATED - HIGH PRIORITY
**License:** Apache-2.0
**Project:** hexgrad/kokoro (new, actively developed)
**Repository:** https://github.com/hexgrad/kokoro
**Model Hub:** https://huggingface.co/hexgrad/Kokoro-82M

**Description:**
Fast, efficient neural TTS with StyleTTS2-based architecture. Released in 2024 as an optimized, production-ready alternative to larger models. Focuses on quality-to-speed ratio.

**Key Features:**
- Fast inference (optimized StyleTTS2)
- Small model size (82M parameters)
- High-quality English voices
- Multiple speaker support
- Good prosody and naturalness
- CPU-friendly

**Model Source:**
- HuggingFace: `hexgrad/Kokoro-82M`
- Pre-trained models available
- Active model updates

**Raccoon Notes:**
- New project (2024) but very promising
- Developer actively improving it
- Good balance of quality and speed
- Could be excellent middle ground between Piper and XTTS
- Still maturing, but worth watching

**Integration Effort:** 4-6 hours

**Raccoon Priority:** ⭐⭐⭐⭐ (Promising new engine, active development)

---

### 8. Silero TTS 🎯

**Status:** NOT INTEGRATED - HIGHEST PRIORITY
**License:** Apache-2.0
**Project:** snakers4/silero-models (ACTIVELY MAINTAINED)
**Repository:** https://github.com/snakers4/silero-models
**Model Hub:** https://models.silero.ai/

**Description:**
Enterprise-grade TTS models from Silero AI team. One of the few actively maintained open-source TTS projects. Offers excellent quality-to-size ratio with production-ready stability.

**Key Features:**
- ACTIVELY MAINTAINED (critical for raccoon mission)
- Fast, small models (~50-100MB each)
- High quality for size
- Multiple languages: English, Russian, German, Spanish, French, Ukrainian
- Multiple speakers per language
- Emotion/speed control
- PyTorch and ONNX formats
- CPU-friendly, real-time capable
- Commercial-friendly license

**Languages & Speakers:**
- English: 4+ speakers (en_v4)
- Russian: 8+ speakers (ru_v4) - best quality
- German: 2 speakers (de_v3)
- Spanish: 2 speakers (es_v1)
- French: 1 speaker (fr_v3)
- Ukrainian: 1 speaker (ua_v3)

**Model Source:**
- Official site: https://models.silero.ai/
- GitHub Releases: https://github.com/snakers4/silero-models/releases
- PyTorch Hub integration
- Direct ONNX models available

**Integration Plan:**
1. Add to requirements.txt: `torch` (already have) or load via PyTorch Hub
2. Create `src/engines/silero.py`
3. Download models to `/app/voices/silero/`
4. Add `make voices-silero` target
5. Map OpenAI voice names to Silero speakers

**Example Usage:**
```python
import torch
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
    repo_or_dir='snakers4/silero-models',
    model='silero_tts',
    language='en',
    speaker='v4_en'
)
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)
```

**Performance:**
- Speed: ~0.1x RTF (very fast)
- Memory: 50-100MB per model
- Latency: <200ms
- Quality: Excellent for size

**Raccoon Notes:**
- STILL ACTIVELY MAINTAINED - rare in TTS landscape!
- Silero AI team responds to issues and updates models
- Best quality-to-size ratio available
- Production-ready and widely deployed
- Russian TTS quality is exceptional
- Low risk of abandonment

**Integration Effort:** 2-4 hours (straightforward PyTorch integration)

**Raccoon Priority:** ⭐⭐⭐⭐⭐ (HIGHEST - active maintenance, excellent quality, easy integration)

---

### 4. StyleTTS2 🎯

**Status:** NOT INTEGRATED - HIGH PRIORITY
**Project:** yl4579/StyleTTS2 (research, somewhat active)
**License:** MIT

**Why Integrate:**
- State-of-the-art quality
- Best prosody and naturalness
- Voice cloning capability
- Style/emotion control
- Research-grade results

**Features:**
- Human-level prosody
- Zero-shot voice cloning
- Style transfer
- Emotion and speaking style control
- LibriTTS trained models

**Challenges:**
- Complex dependencies
- Requires phonemizer
- Slower than other engines
- GPU recommended

**Model Source:**
- HuggingFace: `yl4579/StyleTTS2-LibriTTS`
- GitHub releases

**Estimated Integration Effort:** 6-8 hours
- Complex dependency chain
- Need phonemizer setup
- Create custom engine wrapper
- May need model quantization for production

**Raccoon Priority:** ⭐⭐⭐⭐ (Best quality, but complex)

---

### 5. Fish Speech 🎯

**Status:** NOT INTEGRATED - MEDIUM PRIORITY
**Project:** fishaudio/fish-speech (active)
**License:** Apache 2.0

**Why Integrate:**
- Fast and efficient
- Good multilingual support
- Active development
- Clean API

**Features:**
- Fast inference
- Multilingual (EN, ZH, JA)
- Voice cloning
- Streaming support
- Modern architecture

**Model Source:**
- HuggingFace: `fishaudio/fish-speech-1`
- GitHub releases

**Estimated Integration Effort:** 4-6 hours

**Raccoon Priority:** ⭐⭐⭐ (Active, good quality, but newer/less proven)

---

## Medium Priority Targets

### 6. Kokoro TTS

**Status:** NOT INTEGRATED
**Project:** hexgrad/kokoro (new, active)
**License:** Apache 2.0

**Features:**
- Fast, small, quality
- Multiple voices
- Good English support
- Emerging project

**Raccoon Priority:** ⭐⭐⭐ (Promising but new)

---

### 7. Bark (Suno AI)

**Status:** NOT INTEGRATED
**Project:** suno-ai/bark (archived, company pivoted to music)
**License:** MIT

**Why Consider:**
- Can generate music and sound effects
- Non-verbal sounds (laughs, sighs)
- Multiple languages
- Background audio

**Why Low Priority:**
- Very slow generation
- Large models (~10GB)
- Company abandoned it
- Quality inconsistent

**Raccoon Priority:** ⭐⭐ (Unique features, but slow and abandoned)

---

## Low Priority / Archived

### 8. Tortoise TTS

**Status:** NOT INTEGRATED
**Project:** neonbjb/tortoise-tts (low activity)
**License:** Apache 2.0

**Features:**
- Very high quality
- Voice cloning

**Why Low Priority:**
- Extremely slow (minutes per sentence)
- Not practical for API use
- Better alternatives exist now

**Raccoon Priority:** ⭐ (Too slow for production)

---

### 9. MetaVoice

**Status:** NOT INTEGRATED
**Project:** metavoiceio/metavoice-src (partially abandoned)
**License:** Apache 2.0

**Features:**
- Long-form TTS
- Emotional control
- Voice cloning

**Why Low Priority:**
- Unclear maintenance status
- Complex setup
- Alternatives are better

**Raccoon Priority:** ⭐ (Uncertain future)

---

### 10. Mozilla TTS

**Status:** NOT INTEGRATED
**Project:** mozilla/TTS (archived, became Coqui)
**License:** MPL 2.0

**Why Skip:**
- Fully superseded by Coqui
- No unique capabilities
- Outdated architecture

**Raccoon Priority:** ⛔ (Skip - use Coqui instead)

---

## Integration Roadmap

### Phase 1: Quick Wins (Next 1-2 weeks)
1. ✅ Fix Piper absolute paths
2. ✅ Audit repository
3. [ ] Integrate Silero TTS (2-4 hours)
4. [ ] Set up model mirror on ai.foxhop.net
5. [ ] Test Silero with existing API

### Phase 2: High Quality (2-4 weeks)
1. [ ] Integrate StyleTTS2
2. [ ] Create engine abstraction layer
3. [ ] Refactor speech.py to use engines
4. [ ] Add Fish Speech support

### Phase 3: Resilience (1-2 months)
1. [ ] Implement binary mirror system
2. [ ] Create fallback download logic
3. [ ] Archive critical models to Archive.org
4. [ ] Document all model sources

### Phase 4: Advanced Features (2+ months)
1. [ ] Voice cloning API endpoint
2. [ ] Emotion/style control
3. [ ] Streaming TTS
4. [ ] Multi-speaker conversations

## Model Storage Requirements

Current:
- Piper voices: ~2GB (all languages)
- XTTS v2: ~1.8GB

With all planned engines:
- Silero models: ~500MB (all languages)
- StyleTTS2: ~2GB (base model)
- Fish Speech: ~1.5GB
- **Total: ~8GB** for complete coverage

Mirror storage needed: ~20GB (with redundancy and archives)

## Performance Targets

| Engine | Speed (RTF) | Quality | Use Case |
|--------|-------------|---------|----------|
| Piper | 0.05x | Good | Fast API responses |
| Silero | 0.1x | Good | Balanced speed/quality |
| XTTS | 0.3x | Excellent | Voice cloning |
| StyleTTS2 | 0.5x | Best | Premium quality |
| Fish Speech | 0.15x | Very Good | Multilingual |

RTF = Real-time factor (lower is faster, 1.0 = real-time)

---

**Last Updated:** 2025-11-09
**Raccoon Status:** 🦝 Actively hunting for TTS models in the dumpsters of abandoned repos