uncloseai-speech/docs/MODELS.md

TTS Models and Engines

Raccoon Mission: Rescue abandoned open-source TTS models and integrate them into UncloseAI Speech

Currently Integrated

1. Piper TTS ✅

Status: Working with absolute paths Original Project: rhasspy/piper (abandoned) Fork: OHF-Voice/piper1-gpl v1.3.0 Current Package: PyPI piper-tts>=1.2.0

Features:

• Fast CPU-based neural TTS
• ~100+ high-quality voices
• Multilingual support
• ONNX runtime
• Low memory footprint (~100MB per voice)

Voices Available:

• English (US, GB, multiple accents)
• Spanish, French, German, Italian
• Russian, Polish, Ukrainian
• Chinese, Japanese, Korean
• Many more languages

Model Source:

• HuggingFace: rhasspy/piper-voices
• Direct download: https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/

Integration:

• Used for tts-1 model (fast, lower quality)
• Models stored in /app/voices/en/en_US/libritts_r/medium/
• Configuration via absolute paths in voice_to_speaker.yaml

Example Config:

tts-1:
  alloy:
    model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
    speaker: 79

Raccoon Notes:

• Original rhasspy project abandoned
• OHF-Voice fork has no PyPI package
• Need to create our own PyPI package or vendor the code
• Mirror all voices to prevent HuggingFace dependency

---

2. Coqui XTTS v2 ✅

Status: Integrated as tts-1-hd Original Project: coqui-ai/TTS (company shut down, archived) Current Package: PyPI coqui-tts[languages]

Features:

• High-quality multilingual TTS
• Voice cloning from 6-second samples
• Emotional prosody control
• GPU accelerated (NVIDIA/ROCm)
• ~1.8GB model size

Languages:

• English, Spanish, French, German, Italian, Portuguese
• Polish, Turkish, Russian, Dutch, Czech
• Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi

Model Source:

• HuggingFace: coqui/XTTS-v2
• Auto-downloaded on first use

Integration:

• Used for tts-1-hd model (slow, high quality)
• Voice cloning with custom WAV samples
• Language auto-detection with langdetect

Example Config:

tts-1-hd:
  alloy:
    model: xtts
    speaker: /app/voices/alloy.wav
    language: en

Raccoon Notes:

• Coqui company shut down in 2024
• Repository archived but code still works
• Community forks emerging
• Must mirror XTTS-v2 weights before they disappear
• Consider forking to uncloseai-xtts

---

High Priority Integration Targets

3. Silero TTS 🎯

Status: NOT INTEGRATED - HIGH PRIORITY Project: snakers4/silero-models (still active!) License: Apache 2.0

Why Integrate:

• STILL ACTIVELY MAINTAINED
• Fast, small models (~50-100MB each)
• High quality for size
• Easy integration (PyTorch)
• Commercial-friendly license

Features:

• Multilingual: English, Russian, German, Spanish, French
• Multiple speakers per language
• Emotion control
• CPU friendly
• Real-time capable

Models:

• English: 4 speakers (en_v4)
• Russian: 8+ speakers (ru_v4)
• German: 1 speaker (de_v3)
• Spanish: 2 speakers (es_v1)
• French: 1 speaker (fr_v3)

Model Source:

• GitHub Releases: https://github.com/snakers4/silero-models/releases
• PyTorch Hub
• Direct ONNX models available

Estimated Integration Effort: 2-4 hours

• Add to requirements.txt: silero or direct PyTorch load
• Create src/engines/silero.py
• Download models to /app/models/silero/
• Add voice mappings to config

Example Usage:

import torch
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
    repo_or_dir='snakers4/silero-models',
    model='silero_tts',
    language='en',
    speaker='v4_en'
)
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)

Raccoon Priority: ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio)

---

4. StyleTTS2 🎯

Status: NOT INTEGRATED - HIGH PRIORITY Project: yl4579/StyleTTS2 (research, somewhat active) License: MIT

Why Integrate:

• State-of-the-art quality
• Best prosody and naturalness
• Voice cloning capability
• Style/emotion control
• Research-grade results

Features:

• Human-level prosody
• Zero-shot voice cloning
• Style transfer
• Emotion and speaking style control
• LibriTTS trained models

Challenges:

• Complex dependencies
• Requires phonemizer
• Slower than other engines
• GPU recommended

Model Source:

• HuggingFace: yl4579/StyleTTS2-LibriTTS
• GitHub releases

Estimated Integration Effort: 6-8 hours

• Complex dependency chain
• Need phonemizer setup
• Create custom engine wrapper
• May need model quantization for production

Raccoon Priority: ⭐⭐⭐⭐ (Best quality, but complex)

---

5. Fish Speech 🎯

Status: NOT INTEGRATED - MEDIUM PRIORITY Project: fishaudio/fish-speech (active) License: Apache 2.0

Why Integrate:

• Fast and efficient
• Good multilingual support
• Active development
• Clean API

Features:

• Fast inference
• Multilingual (EN, ZH, JA)
• Voice cloning
• Streaming support
• Modern architecture

Model Source:

• HuggingFace: fishaudio/fish-speech-1
• GitHub releases

Estimated Integration Effort: 4-6 hours

Raccoon Priority: ⭐⭐⭐ (Active, good quality, but newer/less proven)

---

Medium Priority Targets

6. Kokoro TTS

Status: NOT INTEGRATED Project: hexgrad/kokoro (new, active) License: Apache 2.0

Features:

• Fast, small, quality
• Multiple voices
• Good English support
• Emerging project

Raccoon Priority: ⭐⭐⭐ (Promising but new)

---

7. Bark (Suno AI)

Status: NOT INTEGRATED Project: suno-ai/bark (archived, company pivoted to music) License: MIT

Why Consider:

• Can generate music and sound effects
• Non-verbal sounds (laughs, sighs)
• Multiple languages
• Background audio

Why Low Priority:

• Very slow generation
• Large models (~10GB)
• Company abandoned it
• Quality inconsistent

Raccoon Priority: ⭐⭐ (Unique features, but slow and abandoned)

---

Low Priority / Archived

8. Tortoise TTS

Status: NOT INTEGRATED Project: neonbjb/tortoise-tts (low activity) License: Apache 2.0

Features:

• Very high quality
• Voice cloning

Why Low Priority:

• Extremely slow (minutes per sentence)
• Not practical for API use
• Better alternatives exist now

Raccoon Priority: ⭐ (Too slow for production)

---

9. MetaVoice

Status: NOT INTEGRATED Project: metavoiceio/metavoice-src (partially abandoned) License: Apache 2.0

Features:

• Long-form TTS
• Emotional control
• Voice cloning

Why Low Priority:

• Unclear maintenance status
• Complex setup
• Alternatives are better

Raccoon Priority: ⭐ (Uncertain future)

---

10. Mozilla TTS

Status: NOT INTEGRATED Project: mozilla/TTS (archived, became Coqui) License: MPL 2.0

Why Skip:

• Fully superseded by Coqui
• No unique capabilities
• Outdated architecture

Raccoon Priority: ⛔ (Skip - use Coqui instead)

---

Integration Roadmap

Phase 1: Quick Wins (Next 1-2 weeks)

1. ✅ Fix Piper absolute paths
2. ✅ Audit repository
3. Integrate Silero TTS (2-4 hours)
4. Set up model mirror on ai.foxhop.net
5. Test Silero with existing API

Phase 2: High Quality (2-4 weeks)

1. Integrate StyleTTS2
2. Create engine abstraction layer
3. Refactor speech.py to use engines
4. Add Fish Speech support

Phase 3: Resilience (1-2 months)

1. Implement binary mirror system
2. Create fallback download logic
3. Archive critical models to Archive.org
4. Document all model sources

Phase 4: Advanced Features (2+ months)

1. Voice cloning API endpoint
2. Emotion/style control
3. Streaming TTS
4. Multi-speaker conversations

Model Storage Requirements

Current:

• Piper voices: ~2GB (all languages)
• XTTS v2: ~1.8GB

With all planned engines:

• Silero models: ~500MB (all languages)
• StyleTTS2: ~2GB (base model)
• Fish Speech: ~1.5GB
• Total: ~8GB for complete coverage

Mirror storage needed: ~20GB (with redundancy and archives)

Performance Targets

Engine	Speed (RTF)	Quality	Use Case
Piper	0.05x	Good	Fast API responses
Silero	0.1x	Good	Balanced speed/quality
XTTS	0.3x	Excellent	Voice cloning
StyleTTS2	0.5x	Best	Premium quality
Fish Speech	0.15x	Very Good	Multilingual

RTF = Real-time factor (lower is faster, 1.0 = real-time)

---

Last Updated: 2025-11-09 Raccoon Status: 🦝 Actively hunting for TTS models in the dumpsters of abandoned repos