uncloseai-speech/docs/MODELS.md

# TTS Models and Engines

**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into uncloseai-speech

## Default Model: Qwen3-TTS

**🎯 Qwen3-TTS is now the default and only enabled model.**

All other models (Piper, XTTS, Silero, Kokoro) are disabled by default. To enable them, uncomment their sections in `voice_to_speaker.yaml`.

### Quick Start
```bash
# Test Qwen3-TTS (default)
make test

# The model downloads automatically on first use (~3.4GB)
```

---

## Documentation Index

### Comprehensive Research
- 📊 [TTS Models Overview & Research](research/tts-models-overview.md) - Complete comparison matrix, feature analysis, and integration roadmap

### Individual Model Documentation
Each model has detailed documentation covering technical specs, integration status, and Raccoon Mission notes:

**Currently Integrated:**
- 📄 [Coqui TTS (XTTS-v2)](models/coqui-tts.md) - High-quality multilingual TTS with voice cloning
- 📄 [Piper TTS](models/piper-tts.md) - Fast, lightweight neural TTS with 100+ voices
- 📄 [Silero TTS](models/silero-tts.md) - CPU-friendly, actively maintained, 5 languages, 148 voices ✨
- 📄 [Kokoro TTS](models/kokoro-tts.md) - Fast decoder-only architecture, 34 voices, Apache-2.0 ✨

**High Priority Candidates:**
- 📄 [Chatterbox](models/chatterbox.md) - Emotion control, 23 languages, zero-shot cloning

**Specialized Models:**
- 📄 [Mimic 3](models/mimic3.md) - Privacy-focused, offline, lightweight
- 📄 [eSpeak NG](models/espeak-ng.md) - 100+ languages, accessibility-focused
- 📄 [Maya1](models/maya1.md) - Indic languages, diverse accents
- 📄 [Step-Audio-EditX](models/step-audio-editx.md) - LLM-based audio editing (experimental)

**Historical/Archived:**
- 📄 [Mozilla TTS](models/mozilla-tts.md) - Superseded by Coqui TTS
- 📄 [Tortoise TTS](models/tortoise-tts.md) - Studio-quality but slow (archival)

---

## Currently Integrated

### 0. Qwen3-TTS ✅ (DEFAULT)

**Status:** INTEGRATED as tts-1-qwen (DEFAULT MODEL)
**Project:** Qwen/Qwen3-TTS (Alibaba, actively maintained)
**License:** Apache 2.0
**Model:** Qwen3-TTS-12Hz-1.7B-Base

**Why Default:**
- State-of-the-art quality with 1.7B parameters
- Extremely low latency (97ms first packet)
- Voice cloning from 3-second samples
- 10 languages: Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
- Apache 2.0 license (commercial-friendly)
- Actively maintained by Alibaba

**Features:**
- Universal end-to-end architecture (no cascading errors)
- 12Hz acoustic tokenizer for efficient compression
- Dual-track streaming/non-streaming generation
- High-fidelity speech reconstruction
- Natural language instruction control
- Supports both GPU and CPU inference

**Model Specs:**
- Parameters: 1.7B
- Sample Rate: ~24kHz
- Input: Text + Reference Audio (3+ seconds)
- Languages: 10 (zh, en, ja, ko, de, fr, ru, pt, es, it)
- Size: ~3.4GB

**Model Source:**
- HuggingFace: `Qwen/Qwen3-TTS-12Hz-1.7B-Base`
- Auto-downloaded on first use via huggingface-hub
- Cached in `/app/voices/hub/`

**Integration:**
- Used for `tts-1-qwen` model (default)
- Voice cloning with reference audio + transcript
- Pre-configured with Qwen's demo voice

**Example Config:**
```yaml
tts-1-qwen:
  alloy:
    ref_audio: https://example.com/reference.wav
    ref_text: "The exact text spoken in the reference audio"
    language: English
```

**Custom Voice Setup:**
1. Record 3+ seconds of clear speech
2. Transcribe the audio exactly
3. Add to `voice_to_speaker.yaml`:
```yaml
tts-1-qwen:
  my_voice:
    ref_audio: voices/my_voice_sample.wav
    ref_text: "Hello, this is my voice sample for cloning."
    language: English
```

**Makefile Targets:**
```bash
make test        # Test Qwen3-TTS (default)
make test-qwen   # Test Qwen3-TTS explicitly
```

**Hardware Requirements:**
- GPU: NVIDIA with 8GB+ VRAM (recommended)
- CPU: Works but slower (~10x)
- FlashAttention 2 recommended for lower memory

**Raccoon Priority:** ⭐⭐⭐⭐⭐ (State-of-the-art, actively maintained, Apache 2.0)

---

### F5-TTS ✅ (also enabled by default)

**Status:** INTEGRATED as tts-1-f5 (additive, alongside tts-1-qwen)
**Project:** SWivid/F5-TTS (community-maintained, public HuggingFace checkpoint)
**License:** MIT (model + code)
**Model:** F5-TTS_v1

**Why Integrated:**
- Empirical benchmark (Richard, 2026-05-23): faster inference + better voice clones than Qwen3-TTS on the same reference clips
- Smaller model (~336M params vs Qwen3-TTS 1.7B) — lower VRAM, fits comfortably on modest GPUs
- Flow-matching architecture, zero-shot cloning, no fine-tuning needed
- 24kHz output, matches Qwen3-TTS sample rate (drop-in voice swap for clients)

**Model Specs:**
- Parameters: ~336M
- Sample Rate: 24kHz
- Input: Text + Reference Audio (3+ seconds) + Reference Transcript
- Languages: English (primary); community fine-tunes available for others
- Size: ~1.5GB (F5-TTS_v1 + Vocos vocoder)

**Model Source:**
- HuggingFace: `SWivid/F5-TTS` (public, no license accept required)
- Auto-downloaded on first use; `HF_TOKEN` optional (only for higher rate limits)

**Integration:**
- Used for `tts-1-f5` model (additive default)
- Voice cloning with `ref_audio` + `ref_text` (same shape as tts-1-qwen)
- Reuses the same 40 LibriSpeech voices as tts-1-qwen

**Example Config:**
```yaml
tts-1-f5:
  aria:
    ref_audio: cloned-voices/aria.wav
    ref_text: "BUT THE WINDOWS ARE PATCHED WITH WOODEN PANES AND THE DOOR I THINK IS LIKE THE GATE IT IS NEVER OPENED"
    language: English
```

**Tuning Knobs (engine-specific, not OpenAI-compatible):**
- `nfe_step` (default 32) — ODE step count; higher = better quality, slower
- `cfg_strength` (default 2.0) — classifier-free guidance strength
- `speed` (default 1.0) — pitch-preserving speed multiplier
- Does NOT support `temperature` / `top_p` / `top_k` (flow-matching, not autoregressive)

**Makefile Targets:**
```bash
make test-f5     # Test F5-TTS voice cloning
```

**Hardware Requirements:**
- GPU: NVIDIA with 4GB+ VRAM (lighter than Qwen3-TTS)
- CPU: Works but many× realtime
- MPS (Apple Silicon): supported, ~1.5–2× realtime per VoiceClone benchmarks

**Source of Inspiration:** [MonumentalSystems/VoiceClone](https://github.com/MonumentalSystems/VoiceClone) — a single-file F5-TTS web app that proved the engine on our reference workload. Our wrapper mirrors their `F5TTS.infer()` call pattern.

**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Empirically beats current default, MIT, lower VRAM)

---

### 1. Piper TTS (disabled by default) ✅

> 📖 **See [detailed documentation](models/piper-tts.md)** for comprehensive technical specs and integration guide

**Status:** Working with absolute paths
**Original Project:** rhasspy/piper (abandoned)
**Fork:** OHF-Voice/piper1-gpl v1.3.0
**Current Package:** PyPI `piper-tts>=1.2.0`

**Features:**
- Fast CPU-based neural TTS
- ~100+ high-quality voices
- Multilingual support
- ONNX runtime
- Low memory footprint (~100MB per voice)

**Voices Available:**
- English (US, GB, multiple accents)
- Spanish, French, German, Italian
- Russian, Polish, Ukrainian
- Chinese, Japanese, Korean
- Many more languages

**Model Source:**
- HuggingFace: `rhasspy/piper-voices`
- Direct download: `https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/`

**Integration:**
- Used for `tts-1` model (fast, lower quality)
- Models stored in `/app/voices/en/en_US/libritts_r/medium/`
- Configuration via absolute paths in `voice_to_speaker.yaml`

**Example Config:**
```yaml
tts-1:
  alloy:
    model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
    speaker: 79
```

**Raccoon Notes:**
- Original rhasspy project abandoned
- OHF-Voice fork has no PyPI package
- Need to create our own PyPI package or vendor the code
- Mirror all voices to prevent HuggingFace dependency

---

### 2. Coqui XTTS v2 ✅

> 📖 **See [detailed documentation](models/coqui-tts.md)** for comprehensive technical specs and integration guide

**Status:** Integrated as tts-1-hd
**Original Project:** coqui-ai/TTS (company shut down, archived)
**Current Package:** PyPI `coqui-tts[languages]`

**Features:**
- High-quality multilingual TTS
- Voice cloning from 6-second samples
- Emotional prosody control
- GPU accelerated (NVIDIA/ROCm)
- ~1.8GB model size

**Languages:**
- English, Spanish, French, German, Italian, Portuguese
- Polish, Turkish, Russian, Dutch, Czech
- Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi

**Model Source:**
- HuggingFace: `coqui/XTTS-v2`
- Auto-downloaded on first use

**Integration:**
- Used for `tts-1-hd` model (slow, high quality)
- Voice cloning with custom WAV samples
- Language auto-detection with `langdetect`

**Example Config:**
```yaml
tts-1-hd:
  alloy:
    model: xtts
    speaker: /app/voices/alloy.wav
    language: en
```

**Raccoon Notes:**
- Coqui company shut down in 2024
- Repository archived but code still works
- Community forks emerging
- Must mirror XTTS-v2 weights before they disappear
- Consider forking to uncloseai-xtts

---

## High Priority Integration Targets

### 3. Silero TTS ✅

> 📖 **See [detailed documentation](models/silero-tts.md)** for comprehensive technical specs (documentation pending)

**Status:** INTEGRATED as tts-1-silero
**Project:** snakers4/silero-models (actively maintained!)
**License:** Apache 2.0

**Integration Benefits:**
- ACTIVELY MAINTAINED - no abandonment risk!
- Fast, small models (~50-100MB each)
- High quality for size
- Easy integration via PyTorch Hub
- Commercial-friendly license
- CPU friendly - no GPU required

**Features:**
- Multilingual: English, Russian, German, Spanish, French
- Multiple speakers per language (English: 117 speakers!)
- Emotion control
- Real-time capable on CPU
- 48kHz sample rate

**Models:**
- English: 117 speakers (v4_en)
- Russian: 8+ speakers (v4_ru)
- German: 1 speaker (v3_de)
- Spanish: 2 speakers (v1_es)
- French: 1 speaker (v3_fr)

**Model Source:**
- PyTorch Hub: `torch.hub.load('snakers4/silero-models')`
- Models downloaded on first use
- Cached in `/app/voices/` directory

**Integration:**
- Used for `tts-1-silero` model (fast, CPU-friendly)
- Loaded via torch.hub on demand
- 6 OpenAI-compatible voices mapped to Silero speakers

**Example Config:**
```yaml
tts-1-silero:
  alloy:
    language: en
    speaker: en_0
    silero_speaker: v4_en
```

**Makefile Targets:**
```bash
make voices-silero  # Download Silero models (en, ru, de, es, fr)
make test-silero    # Test Silero TTS endpoint
```

**Example Usage:**
```python
import torch
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
    repo_or_dir='snakers4/silero-models',
    model='silero_tts',
    language='en',
    speaker='v4_en'
)
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)
```

**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio)

---

### 4. StyleTTS2 🎯

**Status:** NOT INTEGRATED - HIGH PRIORITY
**Project:** yl4579/StyleTTS2 (research, somewhat active)
**License:** MIT

**Why Integrate:**
- State-of-the-art quality
- Best prosody and naturalness
- Voice cloning capability
- Style/emotion control
- Research-grade results

**Features:**
- Human-level prosody
- Zero-shot voice cloning
- Style transfer
- Emotion and speaking style control
- LibriTTS trained models

**Challenges:**
- Complex dependencies
- Requires phonemizer
- Slower than other engines
- GPU recommended

**Model Source:**
- HuggingFace: `yl4579/StyleTTS2-LibriTTS`
- GitHub releases

**Estimated Integration Effort:** 6-8 hours
- Complex dependency chain
- Need phonemizer setup
- Create custom engine wrapper
- May need model quantization for production

**Raccoon Priority:** ⭐⭐⭐⭐ (Best quality, but complex)

---

### 5. Fish Speech 🎯

**Status:** NOT INTEGRATED - MEDIUM PRIORITY
**Project:** fishaudio/fish-speech (active)
**License:** Apache 2.0

**Why Integrate:**
- Fast and efficient
- Good multilingual support
- Active development
- Clean API

**Features:**
- Fast inference
- Multilingual (EN, ZH, JA)
- Voice cloning
- Streaming support
- Modern architecture

**Model Source:**
- HuggingFace: `fishaudio/fish-speech-1`
- GitHub releases

**Estimated Integration Effort:** 4-6 hours

**Raccoon Priority:** ⭐⭐⭐ (Active, good quality, but newer/less proven)

---

## Medium Priority Targets

### 6. Kokoro TTS ✅

> 📖 **See [detailed documentation](models/kokoro-tts.md)** for comprehensive technical specs

**Status:** INTEGRATED as tts-1-kokoro
**Project:** hexgrad/kokoro (new, active)
**License:** Apache 2.0

**Integration Benefits:**
- Fast decoder-only architecture (82M params)
- 34 voices (American and British English)
- 24kHz sample rate
- Apache-2.0 license
- Lightweight and efficient

**Features:**
- American English: 20 voices (11 female, 9 male)
- British English: 14 voices (4 female, 4 male + variations)
- Speed control
- Real-time capable

**Model Source:**
- HuggingFace: `hexgrad/kokoro-82m`
- Downloaded via huggingface-cli

**Integration:**
- Used for `tts-1-kokoro` model (fast, quality)
- Loaded via kokoro Python package
- OpenAI-compatible voice aliases

**Example Config:**
```yaml
tts-1-kokoro:
  alloy:
    lang_code: a
    kokoro_voice: af_alloy
```

**Makefile Targets:**
```bash
make voices-kokoro  # Download Kokoro models
make test-kokoro    # Test Kokoro TTS endpoint
```

**Raccoon Priority:** ⭐⭐⭐⭐ (Successfully integrated!)

---

### 7. Bark (Suno AI)

**Status:** NOT INTEGRATED
**Project:** suno-ai/bark (archived, company pivoted to music)
**License:** MIT

**Why Consider:**
- Can generate music and sound effects
- Non-verbal sounds (laughs, sighs)
- Multiple languages
- Background audio

**Why Low Priority:**
- Very slow generation
- Large models (~10GB)
- Company abandoned it
- Quality inconsistent

**Raccoon Priority:** ⭐⭐ (Unique features, but slow and abandoned)

---

## Low Priority / Archived

### 8. Tortoise TTS

> 📖 **See [detailed documentation](models/tortoise-tts.md)** for comprehensive technical specs

**Status:** NOT INTEGRATED
**Project:** neonbjb/tortoise-tts (low activity)
**License:** Apache 2.0

**Features:**
- Very high quality
- Voice cloning

**Why Low Priority:**
- Extremely slow (minutes per sentence)
- Not practical for API use
- Better alternatives exist now

**Raccoon Priority:** ⭐ (Too slow for production)

---

### 9. MetaVoice

**Status:** NOT INTEGRATED
**Project:** metavoiceio/metavoice-src (partially abandoned)
**License:** Apache 2.0

**Features:**
- Long-form TTS
- Emotional control
- Voice cloning

**Why Low Priority:**
- Unclear maintenance status
- Complex setup
- Alternatives are better

**Raccoon Priority:** ⭐ (Uncertain future)

---

### 10. Mozilla TTS

> 📖 **See [detailed documentation](models/mozilla-tts.md)** for historical context and relationship to Coqui

**Status:** NOT INTEGRATED
**Project:** mozilla/TTS (archived, became Coqui)
**License:** MPL 2.0

**Why Skip:**
- Fully superseded by Coqui
- No unique capabilities
- Outdated architecture

**Raccoon Priority:** ⛔ (Skip - use Coqui instead)

---

## Integration Roadmap

### Phase 1: Quick Wins (Next 1-2 weeks)
1. ✅ Fix Piper absolute paths
2. ✅ Audit repository
3. ✅ Integrate Silero TTS (COMPLETED!)
4. ✅ Integrate Kokoro (fast decoder) (COMPLETED!)
5. ✅ Add /v1/models API endpoint for voice discovery
6. [ ] Set up model mirror on ai.foxhop.net
7. [ ] Integrate Chatterbox (emotion control)

### Phase 2: High Quality (2-4 weeks)
1. [ ] Integrate StyleTTS2
2. [ ] Create engine abstraction layer
3. [ ] Refactor speech.py to use engines
4. [ ] Add Fish Speech support

### Phase 3: Resilience (1-2 months)
1. [ ] Implement binary mirror system
2. [ ] Create fallback download logic
3. [ ] Archive critical models to Archive.org
4. [ ] Document all model sources

### Phase 4: Advanced Features (2+ months)
1. [ ] Voice cloning API endpoint
2. [ ] Emotion/style control
3. [ ] Streaming TTS
4. [ ] Multi-speaker conversations

## Model Storage Requirements

Current:
- Piper voices: ~2GB (all languages)
- XTTS v2: ~1.8GB

With all planned engines:
- Silero models: ~500MB (all languages)
- StyleTTS2: ~2GB (base model)
- Fish Speech: ~1.5GB
- **Total: ~8GB** for complete coverage

Mirror storage needed: ~20GB (with redundancy and archives)

## Performance Targets

| Engine | Speed (RTF) | Quality | Use Case |
|--------|-------------|---------|----------|
| Piper | 0.05x | Good | Fast API responses |
| Silero | 0.1x | Good | Balanced speed/quality |
| XTTS | 0.3x | Excellent | Voice cloning |
| StyleTTS2 | 0.5x | Best | Premium quality |
| Fish Speech | 0.15x | Very Good | Multilingual |

RTF = Real-time factor (lower is faster, 1.0 = real-time)

---

## Additional Models Under Research

The following models have detailed documentation but are not yet integrated or prioritized:

### Chatterbox
**Priority:** High - Emotion control features
📄 [Full Documentation](models/chatterbox.md)
- Multilingual zero-shot TTS from Resemble AI
- 23 languages with emotion exaggeration control
- Production-grade, actively maintained
- License: Apache-2.0

### Mimic 3
**Priority:** Medium - Privacy/embedded use cases
📄 [Full Documentation](models/mimic3.md)
- Lightweight offline TTS from Mycroft AI
- 20-50MB models, SSML support
- Privacy-focused, embeddable
- License: Apache-2.0

### eSpeak NG
**Priority:** Low - Niche accessibility use
📄 [Full Documentation](models/espeak-ng.md)
- Formant-based synthesis for 100+ languages
- Extremely portable (<10MB)
- Actively maintained by accessibility community
- License: GPL-3.0

### Step-Audio-EditX
**Priority:** Research - Experimental
📄 [Full Documentation](models/step-audio-editx.md)
- New LLM-based audio editing (November 2025)
- Post-generation emotion/style editing
- Cutting-edge but experimental
- License: Apache-2.0

### Maya1
**Priority:** Research - Emerging
📄 [Full Documentation](models/maya1.md)
- India-based multilingual voice model
- Strong Indic language support (Hindi, Tamil, etc.)
- High benchmark rankings
- License: MIT

---

**Last Updated:** 2026-01-26
**Raccoon Status:** 🦝 5 models rescued! Qwen3-TTS is now the default model
**Integration Status:** ✅ Qwen3-TTS (default, unlimited voices via cloning) | Disabled: Piper (55), XTTS (8), Silero (148), Kokoro (34)
**API Endpoints:** tts-1-qwen (default) | Others available: tts-1, tts-1-hd, tts-1-silero, tts-1-kokoro
**Documentation Status:** 📚 11 models fully documented, 1 comprehensive research overview