engineering/uncloseai-speech

Fork 0

russell@unturf.com 2df34f85cc

Add F5-TTS as tts-1-f5 engine (additive, alongside tts-1-qwen)

2026-05-23 13:24:25 -04:00

18 KiB

Raw Blame History

TTS Models and Engines

Raccoon Mission: Rescue abandoned open-source TTS models and integrate them into uncloseai-speech

Default Model: Qwen3-TTS

🎯 Qwen3-TTS is now the default and only enabled model.

All other models (Piper, XTTS, Silero, Kokoro) are disabled by default. To enable them, uncomment their sections in voice_to_speaker.yaml.

Quick Start

# Test Qwen3-TTS (default)
make test

# The model downloads automatically on first use (~3.4GB)

Documentation Index

Comprehensive Research

📊 TTS Models Overview & Research - Complete comparison matrix, feature analysis, and integration roadmap

Individual Model Documentation

Each model has detailed documentation covering technical specs, integration status, and Raccoon Mission notes:

Currently Integrated:

📄 Coqui TTS (XTTS-v2) - High-quality multilingual TTS with voice cloning
📄 Piper TTS - Fast, lightweight neural TTS with 100+ voices
📄 Silero TTS - CPU-friendly, actively maintained, 5 languages, 148 voices ✨
📄 Kokoro TTS - Fast decoder-only architecture, 34 voices, Apache-2.0 ✨

High Priority Candidates:

📄 Chatterbox - Emotion control, 23 languages, zero-shot cloning

Specialized Models:

📄 Mimic 3 - Privacy-focused, offline, lightweight
📄 eSpeak NG - 100+ languages, accessibility-focused
📄 Maya1 - Indic languages, diverse accents
📄 Step-Audio-EditX - LLM-based audio editing (experimental)

Historical/Archived:

📄 Mozilla TTS - Superseded by Coqui TTS
📄 Tortoise TTS - Studio-quality but slow (archival)

Currently Integrated

0. Qwen3-TTS ✅ (DEFAULT)

Status: INTEGRATED as tts-1-qwen (DEFAULT MODEL) Project: Qwen/Qwen3-TTS (Alibaba, actively maintained) License: Apache 2.0 Model: Qwen3-TTS-12Hz-1.7B-Base

Why Default:

State-of-the-art quality with 1.7B parameters
Extremely low latency (97ms first packet)
Voice cloning from 3-second samples
10 languages: Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
Apache 2.0 license (commercial-friendly)
Actively maintained by Alibaba

Features:

Universal end-to-end architecture (no cascading errors)
12Hz acoustic tokenizer for efficient compression
Dual-track streaming/non-streaming generation
High-fidelity speech reconstruction
Natural language instruction control
Supports both GPU and CPU inference

Model Specs:

Parameters: 1.7B
Sample Rate: ~24kHz
Input: Text + Reference Audio (3+ seconds)
Languages: 10 (zh, en, ja, ko, de, fr, ru, pt, es, it)
Size: ~3.4GB

Model Source:

HuggingFace: Qwen/Qwen3-TTS-12Hz-1.7B-Base
Auto-downloaded on first use via huggingface-hub
Cached in /app/voices/hub/

Integration:

Used for tts-1-qwen model (default)
Voice cloning with reference audio + transcript
Pre-configured with Qwen's demo voice

Example Config:

tts-1-qwen:
  alloy:
    ref_audio: https://example.com/reference.wav
    ref_text: "The exact text spoken in the reference audio"
    language: English

Custom Voice Setup:

Record 3+ seconds of clear speech
Transcribe the audio exactly
Add to voice_to_speaker.yaml:

tts-1-qwen:
  my_voice:
    ref_audio: voices/my_voice_sample.wav
    ref_text: "Hello, this is my voice sample for cloning."
    language: English

Makefile Targets:

make test        # Test Qwen3-TTS (default)
make test-qwen   # Test Qwen3-TTS explicitly

Hardware Requirements:

GPU: NVIDIA with 8GB+ VRAM (recommended)
CPU: Works but slower (~10x)
FlashAttention 2 recommended for lower memory

Raccoon Priority: ⭐⭐⭐⭐⭐ (State-of-the-art, actively maintained, Apache 2.0)

F5-TTS ✅ (also enabled by default)

Status: INTEGRATED as tts-1-f5 (additive, alongside tts-1-qwen) Project: SWivid/F5-TTS (community-maintained, gated HuggingFace checkpoint) License: MIT (model + code) Model: F5-TTS_v1

Why Integrated:

Empirical benchmark (Richard, 2026-05-23): faster inference + better voice clones than Qwen3-TTS on the same reference clips
Smaller model (~336M params vs Qwen3-TTS 1.7B) — lower VRAM, fits comfortably on modest GPUs
Flow-matching architecture, zero-shot cloning, no fine-tuning needed
24kHz output, matches Qwen3-TTS sample rate (drop-in voice swap for clients)

Model Specs:

Parameters: ~336M
Sample Rate: 24kHz
Input: Text + Reference Audio (3+ seconds) + Reference Transcript
Languages: English (primary); community fine-tunes available for others
Size: ~1.5GB (F5-TTS_v1 + Vocos vocoder)

Model Source:

HuggingFace: SWivid/F5-TTS (gated — accept license at https://huggingface.co/SWivid/F5-TTS)
Auto-downloaded on first use; requires HF_TOKEN env var

Integration:

Used for tts-1-f5 model (additive default)
Voice cloning with ref_audio + ref_text (same shape as tts-1-qwen)
Reuses the same 40 LibriSpeech voices as tts-1-qwen

Example Config:

tts-1-f5:
  aria:
    ref_audio: cloned-voices/aria.wav
    ref_text: "BUT THE WINDOWS ARE PATCHED WITH WOODEN PANES AND THE DOOR I THINK IS LIKE THE GATE IT IS NEVER OPENED"
    language: English

Tuning Knobs (engine-specific, not OpenAI-compatible):

nfe_step (default 32) — ODE step count; higher = better quality, slower
cfg_strength (default 2.0) — classifier-free guidance strength
speed (default 1.0) — pitch-preserving speed multiplier
Does NOT support temperature / top_p / top_k (flow-matching, not autoregressive)

Makefile Targets:

make test-f5     # Test F5-TTS voice cloning

Hardware Requirements:

GPU: NVIDIA with 4GB+ VRAM (lighter than Qwen3-TTS)
CPU: Works but many× realtime
MPS (Apple Silicon): supported, ~1.5–2× realtime per VoiceClone benchmarks

Source of Inspiration: MonumentalSystems/VoiceClone — a single-file F5-TTS web app that proved the engine on our reference workload. Our wrapper mirrors their F5TTS.infer() call pattern.

Raccoon Priority: ⭐⭐⭐⭐⭐ (Empirically beats current default, MIT, lower VRAM)

1. Piper TTS (disabled by default) ✅

📖 See detailed documentation for comprehensive technical specs and integration guide

Status: Working with absolute paths Original Project: rhasspy/piper (abandoned) Fork: OHF-Voice/piper1-gpl v1.3.0 Current Package: PyPI piper-tts>=1.2.0

Features:

Fast CPU-based neural TTS
~100+ high-quality voices
Multilingual support
ONNX runtime
Low memory footprint (~100MB per voice)

Voices Available:

English (US, GB, multiple accents)
Spanish, French, German, Italian
Russian, Polish, Ukrainian
Chinese, Japanese, Korean
Many more languages

Model Source:

HuggingFace: rhasspy/piper-voices
Direct download: https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/

Integration:

Used for tts-1 model (fast, lower quality)
Models stored in /app/voices/en/en_US/libritts_r/medium/
Configuration via absolute paths in voice_to_speaker.yaml

Example Config:

tts-1:
  alloy:
    model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
    speaker: 79

Raccoon Notes:

Original rhasspy project abandoned
OHF-Voice fork has no PyPI package
Need to create our own PyPI package or vendor the code
Mirror all voices to prevent HuggingFace dependency

2. Coqui XTTS v2 ✅

📖 See detailed documentation for comprehensive technical specs and integration guide

Status: Integrated as tts-1-hd Original Project: coqui-ai/TTS (company shut down, archived) Current Package: PyPI coqui-tts[languages]

Features:

High-quality multilingual TTS
Voice cloning from 6-second samples
Emotional prosody control
GPU accelerated (NVIDIA/ROCm)
~1.8GB model size

Languages:

English, Spanish, French, German, Italian, Portuguese
Polish, Turkish, Russian, Dutch, Czech
Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi

Model Source:

HuggingFace: coqui/XTTS-v2
Auto-downloaded on first use

Integration:

Used for tts-1-hd model (slow, high quality)
Voice cloning with custom WAV samples
Language auto-detection with langdetect

Example Config:

tts-1-hd:
  alloy:
    model: xtts
    speaker: /app/voices/alloy.wav
    language: en

Raccoon Notes:

Coqui company shut down in 2024
Repository archived but code still works
Community forks emerging
Must mirror XTTS-v2 weights before they disappear
Consider forking to uncloseai-xtts

High Priority Integration Targets

3. Silero TTS ✅

📖 See detailed documentation for comprehensive technical specs (documentation pending)

Status: INTEGRATED as tts-1-silero Project: snakers4/silero-models (actively maintained!) License: Apache 2.0

Integration Benefits:

ACTIVELY MAINTAINED - no abandonment risk!
Fast, small models (~50-100MB each)
High quality for size
Easy integration via PyTorch Hub
Commercial-friendly license
CPU friendly - no GPU required

Features:

Multilingual: English, Russian, German, Spanish, French
Multiple speakers per language (English: 117 speakers!)
Emotion control
Real-time capable on CPU
48kHz sample rate

Models:

English: 117 speakers (v4_en)
Russian: 8+ speakers (v4_ru)
German: 1 speaker (v3_de)
Spanish: 2 speakers (v1_es)
French: 1 speaker (v3_fr)

Model Source:

PyTorch Hub: torch.hub.load('snakers4/silero-models')
Models downloaded on first use
Cached in /app/voices/ directory

Integration:

Used for tts-1-silero model (fast, CPU-friendly)
Loaded via torch.hub on demand
6 OpenAI-compatible voices mapped to Silero speakers

Example Config:

tts-1-silero:
  alloy:
    language: en
    speaker: en_0
    silero_speaker: v4_en

Makefile Targets:

make voices-silero  # Download Silero models (en, ru, de, es, fr)
make test-silero    # Test Silero TTS endpoint

Example Usage:

import torch
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
    repo_or_dir='snakers4/silero-models',
    model='silero_tts',
    language='en',
    speaker='v4_en'
)
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)

Raccoon Priority: ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio)

4. StyleTTS2 🎯

Status: NOT INTEGRATED - HIGH PRIORITY Project: yl4579/StyleTTS2 (research, somewhat active) License: MIT

Why Integrate:

State-of-the-art quality
Best prosody and naturalness
Voice cloning capability
Style/emotion control
Research-grade results

Features:

Human-level prosody
Zero-shot voice cloning
Style transfer
Emotion and speaking style control
LibriTTS trained models

Challenges:

Complex dependencies
Requires phonemizer
Slower than other engines
GPU recommended

Model Source:

HuggingFace: yl4579/StyleTTS2-LibriTTS
GitHub releases

Estimated Integration Effort: 6-8 hours

Complex dependency chain
Need phonemizer setup
Create custom engine wrapper
May need model quantization for production

Raccoon Priority: ⭐⭐⭐⭐ (Best quality, but complex)

5. Fish Speech 🎯

Status: NOT INTEGRATED - MEDIUM PRIORITY Project: fishaudio/fish-speech (active) License: Apache 2.0

Why Integrate:

Fast and efficient
Good multilingual support
Active development
Clean API

Features:

Fast inference
Multilingual (EN, ZH, JA)
Voice cloning
Streaming support
Modern architecture

Model Source:

HuggingFace: fishaudio/fish-speech-1
GitHub releases

Estimated Integration Effort: 4-6 hours

Raccoon Priority: ⭐⭐⭐ (Active, good quality, but newer/less proven)

Medium Priority Targets

6. Kokoro TTS ✅

📖 See detailed documentation for comprehensive technical specs

Status: INTEGRATED as tts-1-kokoro Project: hexgrad/kokoro (new, active) License: Apache 2.0

Integration Benefits:

Fast decoder-only architecture (82M params)
34 voices (American and British English)
24kHz sample rate
Apache-2.0 license
Lightweight and efficient

Features:

American English: 20 voices (11 female, 9 male)
British English: 14 voices (4 female, 4 male + variations)
Speed control
Real-time capable

Model Source:

HuggingFace: hexgrad/kokoro-82m
Downloaded via huggingface-cli

Integration:

Used for tts-1-kokoro model (fast, quality)
Loaded via kokoro Python package
OpenAI-compatible voice aliases

Example Config:

tts-1-kokoro:
  alloy:
    lang_code: a
    kokoro_voice: af_alloy

Makefile Targets:

make voices-kokoro  # Download Kokoro models
make test-kokoro    # Test Kokoro TTS endpoint

Raccoon Priority: ⭐⭐⭐⭐ (Successfully integrated!)

7. Bark (Suno AI)

Status: NOT INTEGRATED Project: suno-ai/bark (archived, company pivoted to music) License: MIT

Why Consider:

Can generate music and sound effects
Non-verbal sounds (laughs, sighs)
Multiple languages
Background audio

Why Low Priority:

Very slow generation
Large models (~10GB)
Company abandoned it
Quality inconsistent

Raccoon Priority: ⭐⭐ (Unique features, but slow and abandoned)

Low Priority / Archived

8. Tortoise TTS

📖 See detailed documentation for comprehensive technical specs

Status: NOT INTEGRATED Project: neonbjb/tortoise-tts (low activity) License: Apache 2.0

Features:

Very high quality
Voice cloning

Why Low Priority:

Extremely slow (minutes per sentence)
Not practical for API use
Better alternatives exist now

Raccoon Priority: ⭐ (Too slow for production)

9. MetaVoice

Status: NOT INTEGRATED Project: metavoiceio/metavoice-src (partially abandoned) License: Apache 2.0

Features:

Long-form TTS
Emotional control
Voice cloning

Why Low Priority:

Unclear maintenance status
Complex setup
Alternatives are better

Raccoon Priority: ⭐ (Uncertain future)

10. Mozilla TTS

📖 See detailed documentation for historical context and relationship to Coqui

Status: NOT INTEGRATED Project: mozilla/TTS (archived, became Coqui) License: MPL 2.0

Why Skip:

Fully superseded by Coqui
No unique capabilities
Outdated architecture

Raccoon Priority: ⛔ (Skip - use Coqui instead)

Integration Roadmap

Phase 1: Quick Wins (Next 1-2 weeks)

✅ Fix Piper absolute paths
✅ Audit repository
✅ Integrate Silero TTS (COMPLETED!)
✅ Integrate Kokoro (fast decoder) (COMPLETED!)
✅ Add /v1/models API endpoint for voice discovery
Set up model mirror on ai.foxhop.net
Integrate Chatterbox (emotion control)

Phase 2: High Quality (2-4 weeks)

Integrate StyleTTS2
Create engine abstraction layer
Refactor speech.py to use engines
Add Fish Speech support

Phase 3: Resilience (1-2 months)

Implement binary mirror system
Create fallback download logic
Archive critical models to Archive.org
Document all model sources

Phase 4: Advanced Features (2+ months)

Voice cloning API endpoint
Emotion/style control
Streaming TTS
Multi-speaker conversations

Model Storage Requirements

Current:

Piper voices: ~2GB (all languages)
XTTS v2: ~1.8GB

With all planned engines:

Silero models: ~500MB (all languages)
StyleTTS2: ~2GB (base model)
Fish Speech: ~1.5GB
Total: ~8GB for complete coverage

Mirror storage needed: ~20GB (with redundancy and archives)

Performance Targets

Engine	Speed (RTF)	Quality	Use Case
Piper	0.05x	Good	Fast API responses
Silero	0.1x	Good	Balanced speed/quality
XTTS	0.3x	Excellent	Voice cloning
StyleTTS2	0.5x	Best	Premium quality
Fish Speech	0.15x	Very Good	Multilingual

RTF = Real-time factor (lower is faster, 1.0 = real-time)

Additional Models Under Research

The following models have detailed documentation but are not yet integrated or prioritized:

Chatterbox

Priority: High - Emotion control features 📄 Full Documentation

Multilingual zero-shot TTS from Resemble AI
23 languages with emotion exaggeration control
Production-grade, actively maintained
License: Apache-2.0

Mimic 3

Priority: Medium - Privacy/embedded use cases 📄 Full Documentation

Lightweight offline TTS from Mycroft AI
20-50MB models, SSML support
Privacy-focused, embeddable
License: Apache-2.0

eSpeak NG

Priority: Low - Niche accessibility use 📄 Full Documentation

Formant-based synthesis for 100+ languages
Extremely portable (<10MB)
Actively maintained by accessibility community
License: GPL-3.0

Step-Audio-EditX

Priority: Research - Experimental 📄 Full Documentation

New LLM-based audio editing (November 2025)
Post-generation emotion/style editing
Cutting-edge but experimental
License: Apache-2.0

Maya1

Priority: Research - Emerging 📄 Full Documentation

India-based multilingual voice model
Strong Indic language support (Hindi, Tamil, etc.)
High benchmark rankings
License: MIT

Last Updated: 2026-01-26 Raccoon Status: 🦝 5 models rescued! Qwen3-TTS is now the default model Integration Status: ✅ Qwen3-TTS (default, unlimited voices via cloning) | Disabled: Piper (55), XTTS (8), Silero (148), Kokoro (34) API Endpoints: tts-1-qwen (default) | Others available: tts-1, tts-1-hd, tts-1-silero, tts-1-kokoro Documentation Status: 📚 11 models fully documented, 1 comprehensive research overview

18 KiB Raw Blame History Unescape Escape

TTS Models and Engines

Default Model: Qwen3-TTS

Quick Start

Documentation Index

Comprehensive Research

Individual Model Documentation

Currently Integrated

0. Qwen3-TTS ✅ (DEFAULT)

F5-TTS ✅ (also enabled by default)

1. Piper TTS (disabled by default) ✅

2. Coqui XTTS v2 ✅

High Priority Integration Targets

3. Silero TTS ✅

4. StyleTTS2 🎯

5. Fish Speech 🎯

Medium Priority Targets

6. Kokoro TTS ✅

7. Bark (Suno AI)

Low Priority / Archived

8. Tortoise TTS

9. MetaVoice

10. Mozilla TTS

Integration Roadmap

Phase 1: Quick Wins (Next 1-2 weeks)

Phase 2: High Quality (2-4 weeks)

Phase 3: Resilience (1-2 months)

Phase 4: Advanced Features (2+ months)

Model Storage Requirements

Performance Targets

Additional Models Under Research

Chatterbox

Mimic 3

eSpeak NG

Step-Audio-EditX

Maya1

18 KiB

Raw Blame History