Add comprehensive TTS model and mirror documentation
docs/MODELS.md: - Document 10+ abandoned TTS engines to integrate - Piper TTS (integrated, fixed) - Coqui XTTS v2 (integrated, company shut down) - Silero TTS (HIGH PRIORITY - still active, fast) - StyleTTS2 (HIGH PRIORITY - best quality) - Fish Speech (active, good quality) - Kokoro, Bark, Tortoise, MetaVoice (lower priority) - Integration roadmap with time estimates - Performance targets and storage requirements docs/MIRRORS.md: - Multi-tier mirror strategy for resilience - Tier 1: Upstream (HuggingFace, PyPI, GitHub) - Tier 2: Self-hosted MinIO on ai.foxhop.net - Tier 3: Archive.org for public archival - Tier 4: IPFS for decentralization - Complete implementation with scripts and configs - Fallback download logic - Recovery scenarios - Cost: $0-20/month Raccoon mission: Ensure TTS keeps working when upstream dies. Documentation-first approach before implementing features. 🦝 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
e8183b4585
commit
3a9e27e556
2 changed files with 801 additions and 0 deletions
426
docs/MIRRORS.md
Normal file
426
docs/MIRRORS.md
Normal file
|
|
@ -0,0 +1,426 @@
|
|||
# Binary Mirror Strategy
|
||||
|
||||
**Purpose:** Ensure UncloseAI Speech keeps working even if upstream model sources disappear
|
||||
|
||||
## The Problem
|
||||
|
||||
### Upstream Fragility
|
||||
- HuggingFace repos can be deleted
|
||||
- GitHub releases disappear when repos are archived
|
||||
- PyPI packages can be yanked
|
||||
- Companies shut down and take their models offline
|
||||
- Rate limiting breaks automated deployments
|
||||
|
||||
### Real Examples
|
||||
- ✅ Coqui AI: Company shut down 2024, repo archived
|
||||
- ✅ Rhasspy Piper: Original project abandoned
|
||||
- ✅ Suno Bark: Company pivoted, model archived
|
||||
- ⚠️ XTTS-v2: Depends on archived Coqui repo
|
||||
|
||||
## Multi-Tier Mirror Architecture
|
||||
|
||||
### Tier 1: Upstream Sources (Primary)
|
||||
Always try upstream first - they're fastest and most up-to-date.
|
||||
|
||||
**Sources:**
|
||||
- HuggingFace Hub (huggingface.co)
|
||||
- PyPI (pypi.org)
|
||||
- GitHub Releases
|
||||
- Official project websites
|
||||
|
||||
**Advantages:**
|
||||
- Latest versions
|
||||
- Fast CDN delivery
|
||||
- Community validation
|
||||
|
||||
**Disadvantages:**
|
||||
- Can disappear
|
||||
- Rate limits
|
||||
- Requires internet
|
||||
|
||||
---
|
||||
|
||||
### Tier 2: UncloseAI Mirror (Secondary)
|
||||
Self-hosted mirror under our control.
|
||||
|
||||
**Location:** ai.foxhop.net
|
||||
**Storage:** MinIO S3-compatible object storage
|
||||
**Capacity:** 100GB allocated for models
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
# Install MinIO on ai.foxhop.net
|
||||
docker run -d \
|
||||
-p 9000:9000 \
|
||||
-p 9001:9001 \
|
||||
--name minio \
|
||||
-v /data/minio:/data \
|
||||
-e "MINIO_ROOT_USER=admin" \
|
||||
-e "MINIO_ROOT_PASSWORD=<secure_password>" \
|
||||
minio/minio server /data --console-address ":9001"
|
||||
|
||||
# Create bucket for models
|
||||
mc alias set unclose http://ai.foxhop.net:9000 admin <password>
|
||||
mc mb unclose/tts-models
|
||||
mc policy set download unclose/tts-models
|
||||
```
|
||||
|
||||
**Directory Structure:**
|
||||
```
|
||||
tts-models/
|
||||
├── piper/
|
||||
│ ├── v1.0.0/
|
||||
│ │ ├── en/
|
||||
│ │ │ ├── en_US/
|
||||
│ │ │ │ └── libritts_r/
|
||||
│ │ │ │ └── medium/
|
||||
│ │ │ │ ├── en_US-libritts_r-medium.onnx
|
||||
│ │ │ │ └── en_US-libritts_r-medium.onnx.json
|
||||
│ │ └── voices.json (metadata)
|
||||
├── xtts/
|
||||
│ └── v2.0.3/
|
||||
│ ├── model.pth
|
||||
│ ├── config.json
|
||||
│ ├── vocab.json
|
||||
│ └── README.md
|
||||
├── silero/
|
||||
│ └── v4/
|
||||
│ ├── en_v4.pt
|
||||
│ ├── ru_v4.pt
|
||||
│ └── models.json
|
||||
├── styletts2/
|
||||
│ └── libritts/
|
||||
│ ├── checkpoint.pt
|
||||
│ └── config.yml
|
||||
└── metadata.json (master index)
|
||||
```
|
||||
|
||||
**Sync Script:**
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/sync_models_to_mirror.sh
|
||||
# Sync upstream models to UncloseAI mirror
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
MIRROR_URL="http://ai.foxhop.net:9000/tts-models"
|
||||
TEMP_DIR="/tmp/model_sync"
|
||||
|
||||
# Sync Piper voices
|
||||
sync_piper() {
|
||||
echo "Syncing Piper models..."
|
||||
for voice in en_US-libritts_r-medium en_GB-northern_english_male-medium; do
|
||||
wget -P "$TEMP_DIR/piper/" \
|
||||
"https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx" \
|
||||
"https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx.json"
|
||||
done
|
||||
mc cp --recursive "$TEMP_DIR/piper/" unclose/tts-models/piper/v1.0.0/
|
||||
}
|
||||
|
||||
# Sync XTTS
|
||||
sync_xtts() {
|
||||
echo "Syncing XTTS v2..."
|
||||
# Use huggingface-cli or git lfs
|
||||
git clone https://huggingface.co/coqui/XTTS-v2 "$TEMP_DIR/xtts"
|
||||
mc cp --recursive "$TEMP_DIR/xtts/" unclose/tts-models/xtts/v2.0.3/
|
||||
}
|
||||
|
||||
# Sync Silero
|
||||
sync_silero() {
|
||||
echo "Syncing Silero models..."
|
||||
wget -P "$TEMP_DIR/silero/" \
|
||||
"https://models.silero.ai/models/tts/en/v4_en.pt" \
|
||||
"https://models.silero.ai/models/tts/ru/v4_ru.pt"
|
||||
mc cp --recursive "$TEMP_DIR/silero/" unclose/tts-models/silero/v4/
|
||||
}
|
||||
|
||||
sync_piper
|
||||
sync_xtts
|
||||
sync_silero
|
||||
|
||||
echo "✅ Mirror sync complete"
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
- Under our control
|
||||
- No rate limits
|
||||
- Fast local access
|
||||
- Can modify models
|
||||
|
||||
**Disadvantages:**
|
||||
- Maintenance overhead
|
||||
- Storage costs
|
||||
- Single point of failure (us)
|
||||
|
||||
---
|
||||
|
||||
### Tier 3: Archive.org (Tertiary)
|
||||
Public archive for critical models.
|
||||
|
||||
**Purpose:** Long-term preservation, public good
|
||||
|
||||
**What to Archive:**
|
||||
- Piper voice pack (full 2GB)
|
||||
- XTTS-v2 weights
|
||||
- Key Silero models
|
||||
- StyleTTS2 checkpoints
|
||||
|
||||
**Upload Process:**
|
||||
```bash
|
||||
# Install internet archive CLI
|
||||
pip install internetarchive
|
||||
|
||||
# Configure
|
||||
ia configure
|
||||
|
||||
# Upload critical model
|
||||
ia upload uncloseai-piper-voices-v1.0.0 \
|
||||
piper_voices.tar.gz \
|
||||
--metadata="title:Piper TTS Voices v1.0.0" \
|
||||
--metadata="description:Complete Piper TTS voice collection from rhasspy/piper-voices" \
|
||||
--metadata="subject:text-to-speech;tts;piper;neural-tts" \
|
||||
--metadata="creator:UncloseAI Speech Raccoon Mission" \
|
||||
--metadata="date:2025-11-09"
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
- Permanent storage
|
||||
- Public access
|
||||
- Free
|
||||
- Trusted platform
|
||||
|
||||
**Disadvantages:**
|
||||
- Slow downloads
|
||||
- No control over availability
|
||||
- Upload limits
|
||||
|
||||
---
|
||||
|
||||
### Tier 4: IPFS (Experimental)
|
||||
Decentralized storage for the future.
|
||||
|
||||
**Purpose:** Censorship-resistant, distributed
|
||||
|
||||
**Implementation:**
|
||||
```bash
|
||||
# Pin critical models to IPFS
|
||||
ipfs add -r piper_voices/
|
||||
# Output: QmXXXXXXXXXXXXXXXX
|
||||
|
||||
# Pin via Pinata or other service
|
||||
curl -X POST "https://api.pinata.cloud/pinning/pinByHash" \
|
||||
-H "pinata_api_key: YOUR_KEY" \
|
||||
-d '{"hashToPin":"QmXXXXXXXXXXXXXXXX"}'
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
- Decentralized
|
||||
- Censorship resistant
|
||||
- Content-addressed
|
||||
|
||||
**Disadvantages:**
|
||||
- Slow
|
||||
- Requires pinning service
|
||||
- Less reliable
|
||||
|
||||
---
|
||||
|
||||
## Download Strategy with Fallbacks
|
||||
|
||||
### Smart Downloader
|
||||
|
||||
```python
|
||||
# src/utils/model_downloader.py
|
||||
from typing import List, Optional
|
||||
import requests
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class ModelDownloader:
|
||||
"""Download models with automatic fallback to mirrors"""
|
||||
|
||||
def __init__(self):
|
||||
self.mirrors = [
|
||||
"https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/",
|
||||
"http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/",
|
||||
"https://archive.org/download/uncloseai-piper-voices-v1.0.0/",
|
||||
]
|
||||
|
||||
def download(self, model_path: str, output_path: str) -> bool:
|
||||
"""Try each mirror until successful"""
|
||||
for mirror_url in self.mirrors:
|
||||
full_url = f"{mirror_url}{model_path}"
|
||||
logger.info(f"Trying {full_url}...")
|
||||
|
||||
try:
|
||||
response = requests.get(full_url, stream=True, timeout=30)
|
||||
if response.status_code == 200:
|
||||
with open(output_path, 'wb') as f:
|
||||
for chunk in response.iter_content(chunk_size=8192):
|
||||
f.write(chunk)
|
||||
logger.info(f"✅ Downloaded from {mirror_url}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.warning(f"❌ Failed {mirror_url}: {e}")
|
||||
continue
|
||||
|
||||
logger.error(f"All mirrors failed for {model_path}")
|
||||
return False
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```yaml
|
||||
# config/mirrors.yaml
|
||||
mirrors:
|
||||
piper:
|
||||
- https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/
|
||||
- http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/
|
||||
- https://archive.org/download/uncloseai-piper-voices-v1.0.0/
|
||||
- ipfs://QmXXXXXXXXXXXXXXXX/
|
||||
|
||||
xtts:
|
||||
- https://huggingface.co/coqui/XTTS-v2/resolve/main/
|
||||
- http://ai.foxhop.net:9000/tts-models/xtts/v2.0.3/
|
||||
- https://archive.org/download/uncloseai-xtts-v2/
|
||||
|
||||
silero:
|
||||
- https://models.silero.ai/models/tts/
|
||||
- http://ai.foxhop.net:9000/tts-models/silero/v4/
|
||||
- https://github.com/snakers4/silero-models/releases/download/
|
||||
|
||||
retry:
|
||||
max_attempts: 3
|
||||
timeout_seconds: 30
|
||||
backoff_multiplier: 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
### Phase 1: Setup Mirror Infrastructure
|
||||
- [ ] Deploy MinIO on ai.foxhop.net
|
||||
- [ ] Create `tts-models` bucket
|
||||
- [ ] Set up public read access
|
||||
- [ ] Configure DNS/CDN (optional)
|
||||
|
||||
### Phase 2: Initial Sync
|
||||
- [ ] Download all Piper voices (2GB)
|
||||
- [ ] Download XTTS-v2 (1.8GB)
|
||||
- [ ] Upload to MinIO mirror
|
||||
- [ ] Test download from mirror
|
||||
|
||||
### Phase 3: Implement Fallback Logic
|
||||
- [ ] Create `ModelDownloader` class
|
||||
- [ ] Add mirror config to `config/mirrors.yaml`
|
||||
- [ ] Update download scripts to use fallbacks
|
||||
- [ ] Add mirror health checks
|
||||
|
||||
### Phase 4: Archive Critical Models
|
||||
- [ ] Upload Piper voices to Archive.org
|
||||
- [ ] Upload XTTS-v2 to Archive.org
|
||||
- [ ] Document archive locations
|
||||
- [ ] Test restoration from archive
|
||||
|
||||
### Phase 5: Automation
|
||||
- [ ] Create sync script (`scripts/sync_models.sh`)
|
||||
- [ ] Set up cron job for weekly sync
|
||||
- [ ] Monitor mirror disk usage
|
||||
- [ ] Alert on upstream changes
|
||||
|
||||
### Phase 6: Future Engines
|
||||
- [ ] Add Silero to mirror
|
||||
- [ ] Add StyleTTS2 to mirror
|
||||
- [ ] Add Fish Speech to mirror
|
||||
|
||||
---
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/check_mirrors.sh
|
||||
# Verify all mirrors are accessible
|
||||
|
||||
MODELS=(
|
||||
"piper/v1.0.0/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx"
|
||||
"xtts/v2.0.3/model.pth"
|
||||
)
|
||||
|
||||
for model in "${MODELS[@]}"; do
|
||||
echo "Checking $model..."
|
||||
|
||||
# Check HuggingFace
|
||||
curl -I "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/$model" | head -n 1
|
||||
|
||||
# Check our mirror
|
||||
curl -I "http://ai.foxhop.net:9000/tts-models/$model" | head -n 1
|
||||
|
||||
echo "---"
|
||||
done
|
||||
```
|
||||
|
||||
### Storage Usage
|
||||
|
||||
```bash
|
||||
# Monitor MinIO usage
|
||||
mc du unclose/tts-models
|
||||
|
||||
# Expected:
|
||||
# piper/: 2GB
|
||||
# xtts/: 1.8GB
|
||||
# silero/: 500MB
|
||||
# Total: ~5GB
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost Estimation
|
||||
|
||||
### Storage (100GB allocated)
|
||||
- MinIO on existing server: **$0** (using spare disk)
|
||||
- Bandwidth: **$0** (self-hosted, unlimited)
|
||||
|
||||
### Archive.org
|
||||
- Storage: **$0** (free)
|
||||
- Bandwidth: **$0** (free)
|
||||
|
||||
### IPFS Pinning (Optional)
|
||||
- Pinata: **$20/month** for 100GB
|
||||
- Or self-host: **$0**
|
||||
|
||||
**Total Cost: $0-20/month**
|
||||
|
||||
---
|
||||
|
||||
## Recovery Scenarios
|
||||
|
||||
### Scenario 1: HuggingFace is down
|
||||
1. Downloader tries HF, gets timeout
|
||||
2. Falls back to ai.foxhop.net mirror ✅
|
||||
3. Download succeeds in 30 seconds
|
||||
|
||||
### Scenario 2: Our mirror is down
|
||||
1. Downloader tries ai.foxhop.net, fails
|
||||
2. Falls back to Archive.org ✅
|
||||
3. Download succeeds in 2 minutes (slower)
|
||||
|
||||
### Scenario 3: Total internet failure
|
||||
1. Models already cached in `/app/voices/`
|
||||
2. Service continues with cached models ✅
|
||||
3. No downloads needed for operation
|
||||
|
||||
### Scenario 4: Apocalypse (all servers gone)
|
||||
1. Restore from Archive.org archive
|
||||
2. Restore from IPFS if configured
|
||||
3. Restore from torrents if distributed
|
||||
4. Rebuild from source if absolutely necessary
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-11-09
|
||||
**Raccoon Status:** 🦝 Building resilient caches like storing nuts for winter
|
||||
375
docs/MODELS.md
Normal file
375
docs/MODELS.md
Normal file
|
|
@ -0,0 +1,375 @@
|
|||
# TTS Models and Engines
|
||||
|
||||
**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into UncloseAI Speech
|
||||
|
||||
## Currently Integrated
|
||||
|
||||
### 1. Piper TTS ✅
|
||||
|
||||
**Status:** Working with absolute paths
|
||||
**Original Project:** rhasspy/piper (abandoned)
|
||||
**Fork:** OHF-Voice/piper1-gpl v1.3.0
|
||||
**Current Package:** PyPI `piper-tts>=1.2.0`
|
||||
|
||||
**Features:**
|
||||
- Fast CPU-based neural TTS
|
||||
- ~100+ high-quality voices
|
||||
- Multilingual support
|
||||
- ONNX runtime
|
||||
- Low memory footprint (~100MB per voice)
|
||||
|
||||
**Voices Available:**
|
||||
- English (US, GB, multiple accents)
|
||||
- Spanish, French, German, Italian
|
||||
- Russian, Polish, Ukrainian
|
||||
- Chinese, Japanese, Korean
|
||||
- Many more languages
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `rhasspy/piper-voices`
|
||||
- Direct download: `https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/`
|
||||
|
||||
**Integration:**
|
||||
- Used for `tts-1` model (fast, lower quality)
|
||||
- Models stored in `/app/voices/en/en_US/libritts_r/medium/`
|
||||
- Configuration via absolute paths in `voice_to_speaker.yaml`
|
||||
|
||||
**Example Config:**
|
||||
```yaml
|
||||
tts-1:
|
||||
alloy:
|
||||
model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
|
||||
speaker: 79
|
||||
```
|
||||
|
||||
**Raccoon Notes:**
|
||||
- Original rhasspy project abandoned
|
||||
- OHF-Voice fork has no PyPI package
|
||||
- Need to create our own PyPI package or vendor the code
|
||||
- Mirror all voices to prevent HuggingFace dependency
|
||||
|
||||
---
|
||||
|
||||
### 2. Coqui XTTS v2 ✅
|
||||
|
||||
**Status:** Integrated as tts-1-hd
|
||||
**Original Project:** coqui-ai/TTS (company shut down, archived)
|
||||
**Current Package:** PyPI `coqui-tts[languages]`
|
||||
|
||||
**Features:**
|
||||
- High-quality multilingual TTS
|
||||
- Voice cloning from 6-second samples
|
||||
- Emotional prosody control
|
||||
- GPU accelerated (NVIDIA/ROCm)
|
||||
- ~1.8GB model size
|
||||
|
||||
**Languages:**
|
||||
- English, Spanish, French, German, Italian, Portuguese
|
||||
- Polish, Turkish, Russian, Dutch, Czech
|
||||
- Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `coqui/XTTS-v2`
|
||||
- Auto-downloaded on first use
|
||||
|
||||
**Integration:**
|
||||
- Used for `tts-1-hd` model (slow, high quality)
|
||||
- Voice cloning with custom WAV samples
|
||||
- Language auto-detection with `langdetect`
|
||||
|
||||
**Example Config:**
|
||||
```yaml
|
||||
tts-1-hd:
|
||||
alloy:
|
||||
model: xtts
|
||||
speaker: /app/voices/alloy.wav
|
||||
language: en
|
||||
```
|
||||
|
||||
**Raccoon Notes:**
|
||||
- Coqui company shut down in 2024
|
||||
- Repository archived but code still works
|
||||
- Community forks emerging
|
||||
- Must mirror XTTS-v2 weights before they disappear
|
||||
- Consider forking to uncloseai-xtts
|
||||
|
||||
---
|
||||
|
||||
## High Priority Integration Targets
|
||||
|
||||
### 3. Silero TTS 🎯
|
||||
|
||||
**Status:** NOT INTEGRATED - HIGH PRIORITY
|
||||
**Project:** snakers4/silero-models (still active!)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Why Integrate:**
|
||||
- STILL ACTIVELY MAINTAINED
|
||||
- Fast, small models (~50-100MB each)
|
||||
- High quality for size
|
||||
- Easy integration (PyTorch)
|
||||
- Commercial-friendly license
|
||||
|
||||
**Features:**
|
||||
- Multilingual: English, Russian, German, Spanish, French
|
||||
- Multiple speakers per language
|
||||
- Emotion control
|
||||
- CPU friendly
|
||||
- Real-time capable
|
||||
|
||||
**Models:**
|
||||
- English: 4 speakers (en_v4)
|
||||
- Russian: 8+ speakers (ru_v4)
|
||||
- German: 1 speaker (de_v3)
|
||||
- Spanish: 2 speakers (es_v1)
|
||||
- French: 1 speaker (fr_v3)
|
||||
|
||||
**Model Source:**
|
||||
- GitHub Releases: https://github.com/snakers4/silero-models/releases
|
||||
- PyTorch Hub
|
||||
- Direct ONNX models available
|
||||
|
||||
**Estimated Integration Effort:** 2-4 hours
|
||||
- Add to requirements.txt: `silero` or direct PyTorch load
|
||||
- Create `src/engines/silero.py`
|
||||
- Download models to `/app/models/silero/`
|
||||
- Add voice mappings to config
|
||||
|
||||
**Example Usage:**
|
||||
```python
|
||||
import torch
|
||||
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
|
||||
repo_or_dir='snakers4/silero-models',
|
||||
model='silero_tts',
|
||||
language='en',
|
||||
speaker='v4_en'
|
||||
)
|
||||
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)
|
||||
```
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio)
|
||||
|
||||
---
|
||||
|
||||
### 4. StyleTTS2 🎯
|
||||
|
||||
**Status:** NOT INTEGRATED - HIGH PRIORITY
|
||||
**Project:** yl4579/StyleTTS2 (research, somewhat active)
|
||||
**License:** MIT
|
||||
|
||||
**Why Integrate:**
|
||||
- State-of-the-art quality
|
||||
- Best prosody and naturalness
|
||||
- Voice cloning capability
|
||||
- Style/emotion control
|
||||
- Research-grade results
|
||||
|
||||
**Features:**
|
||||
- Human-level prosody
|
||||
- Zero-shot voice cloning
|
||||
- Style transfer
|
||||
- Emotion and speaking style control
|
||||
- LibriTTS trained models
|
||||
|
||||
**Challenges:**
|
||||
- Complex dependencies
|
||||
- Requires phonemizer
|
||||
- Slower than other engines
|
||||
- GPU recommended
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `yl4579/StyleTTS2-LibriTTS`
|
||||
- GitHub releases
|
||||
|
||||
**Estimated Integration Effort:** 6-8 hours
|
||||
- Complex dependency chain
|
||||
- Need phonemizer setup
|
||||
- Create custom engine wrapper
|
||||
- May need model quantization for production
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐⭐ (Best quality, but complex)
|
||||
|
||||
---
|
||||
|
||||
### 5. Fish Speech 🎯
|
||||
|
||||
**Status:** NOT INTEGRATED - MEDIUM PRIORITY
|
||||
**Project:** fishaudio/fish-speech (active)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Why Integrate:**
|
||||
- Fast and efficient
|
||||
- Good multilingual support
|
||||
- Active development
|
||||
- Clean API
|
||||
|
||||
**Features:**
|
||||
- Fast inference
|
||||
- Multilingual (EN, ZH, JA)
|
||||
- Voice cloning
|
||||
- Streaming support
|
||||
- Modern architecture
|
||||
|
||||
**Model Source:**
|
||||
- HuggingFace: `fishaudio/fish-speech-1`
|
||||
- GitHub releases
|
||||
|
||||
**Estimated Integration Effort:** 4-6 hours
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐ (Active, good quality, but newer/less proven)
|
||||
|
||||
---
|
||||
|
||||
## Medium Priority Targets
|
||||
|
||||
### 6. Kokoro TTS
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** hexgrad/kokoro (new, active)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Features:**
|
||||
- Fast, small, quality
|
||||
- Multiple voices
|
||||
- Good English support
|
||||
- Emerging project
|
||||
|
||||
**Raccoon Priority:** ⭐⭐⭐ (Promising but new)
|
||||
|
||||
---
|
||||
|
||||
### 7. Bark (Suno AI)
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** suno-ai/bark (archived, company pivoted to music)
|
||||
**License:** MIT
|
||||
|
||||
**Why Consider:**
|
||||
- Can generate music and sound effects
|
||||
- Non-verbal sounds (laughs, sighs)
|
||||
- Multiple languages
|
||||
- Background audio
|
||||
|
||||
**Why Low Priority:**
|
||||
- Very slow generation
|
||||
- Large models (~10GB)
|
||||
- Company abandoned it
|
||||
- Quality inconsistent
|
||||
|
||||
**Raccoon Priority:** ⭐⭐ (Unique features, but slow and abandoned)
|
||||
|
||||
---
|
||||
|
||||
## Low Priority / Archived
|
||||
|
||||
### 8. Tortoise TTS
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** neonbjb/tortoise-tts (low activity)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Features:**
|
||||
- Very high quality
|
||||
- Voice cloning
|
||||
|
||||
**Why Low Priority:**
|
||||
- Extremely slow (minutes per sentence)
|
||||
- Not practical for API use
|
||||
- Better alternatives exist now
|
||||
|
||||
**Raccoon Priority:** ⭐ (Too slow for production)
|
||||
|
||||
---
|
||||
|
||||
### 9. MetaVoice
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** metavoiceio/metavoice-src (partially abandoned)
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Features:**
|
||||
- Long-form TTS
|
||||
- Emotional control
|
||||
- Voice cloning
|
||||
|
||||
**Why Low Priority:**
|
||||
- Unclear maintenance status
|
||||
- Complex setup
|
||||
- Alternatives are better
|
||||
|
||||
**Raccoon Priority:** ⭐ (Uncertain future)
|
||||
|
||||
---
|
||||
|
||||
### 10. Mozilla TTS
|
||||
|
||||
**Status:** NOT INTEGRATED
|
||||
**Project:** mozilla/TTS (archived, became Coqui)
|
||||
**License:** MPL 2.0
|
||||
|
||||
**Why Skip:**
|
||||
- Fully superseded by Coqui
|
||||
- No unique capabilities
|
||||
- Outdated architecture
|
||||
|
||||
**Raccoon Priority:** ⛔ (Skip - use Coqui instead)
|
||||
|
||||
---
|
||||
|
||||
## Integration Roadmap
|
||||
|
||||
### Phase 1: Quick Wins (Next 1-2 weeks)
|
||||
1. ✅ Fix Piper absolute paths
|
||||
2. ✅ Audit repository
|
||||
3. [ ] Integrate Silero TTS (2-4 hours)
|
||||
4. [ ] Set up model mirror on ai.foxhop.net
|
||||
5. [ ] Test Silero with existing API
|
||||
|
||||
### Phase 2: High Quality (2-4 weeks)
|
||||
1. [ ] Integrate StyleTTS2
|
||||
2. [ ] Create engine abstraction layer
|
||||
3. [ ] Refactor speech.py to use engines
|
||||
4. [ ] Add Fish Speech support
|
||||
|
||||
### Phase 3: Resilience (1-2 months)
|
||||
1. [ ] Implement binary mirror system
|
||||
2. [ ] Create fallback download logic
|
||||
3. [ ] Archive critical models to Archive.org
|
||||
4. [ ] Document all model sources
|
||||
|
||||
### Phase 4: Advanced Features (2+ months)
|
||||
1. [ ] Voice cloning API endpoint
|
||||
2. [ ] Emotion/style control
|
||||
3. [ ] Streaming TTS
|
||||
4. [ ] Multi-speaker conversations
|
||||
|
||||
## Model Storage Requirements
|
||||
|
||||
Current:
|
||||
- Piper voices: ~2GB (all languages)
|
||||
- XTTS v2: ~1.8GB
|
||||
|
||||
With all planned engines:
|
||||
- Silero models: ~500MB (all languages)
|
||||
- StyleTTS2: ~2GB (base model)
|
||||
- Fish Speech: ~1.5GB
|
||||
- **Total: ~8GB** for complete coverage
|
||||
|
||||
Mirror storage needed: ~20GB (with redundancy and archives)
|
||||
|
||||
## Performance Targets
|
||||
|
||||
| Engine | Speed (RTF) | Quality | Use Case |
|
||||
|--------|-------------|---------|----------|
|
||||
| Piper | 0.05x | Good | Fast API responses |
|
||||
| Silero | 0.1x | Good | Balanced speed/quality |
|
||||
| XTTS | 0.3x | Excellent | Voice cloning |
|
||||
| StyleTTS2 | 0.5x | Best | Premium quality |
|
||||
| Fish Speech | 0.15x | Very Good | Multilingual |
|
||||
|
||||
RTF = Real-time factor (lower is faster, 1.0 = real-time)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-11-09
|
||||
**Raccoon Status:** 🦝 Actively hunting for TTS models in the dumpsters of abandoned repos
|
||||
Loading…
Add table
Add a link
Reference in a new issue