Add comprehensive TTS model and mirror documentation

docs/MODELS.md:
- Document 10+ abandoned TTS engines to integrate
- Piper TTS (integrated, fixed)
- Coqui XTTS v2 (integrated, company shut down)
- Silero TTS (HIGH PRIORITY - still active, fast)
- StyleTTS2 (HIGH PRIORITY - best quality)
- Fish Speech (active, good quality)
- Kokoro, Bark, Tortoise, MetaVoice (lower priority)
- Integration roadmap with time estimates
- Performance targets and storage requirements

docs/MIRRORS.md:
- Multi-tier mirror strategy for resilience
- Tier 1: Upstream (HuggingFace, PyPI, GitHub)
- Tier 2: Self-hosted MinIO on ai.foxhop.net
- Tier 3: Archive.org for public archival
- Tier 4: IPFS for decentralization
- Complete implementation with scripts and configs
- Fallback download logic
- Recovery scenarios
- Cost: $0-20/month

Raccoon mission: Ensure TTS keeps working when upstream dies.
Documentation-first approach before implementing features.

🦝 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Russell Ballestrini 2025-11-09 09:28:02 -05:00
parent e8183b4585
commit 3a9e27e556
2 changed files with 801 additions and 0 deletions

426
docs/MIRRORS.md Normal file
View file

@ -0,0 +1,426 @@
# Binary Mirror Strategy
**Purpose:** Ensure UncloseAI Speech keeps working even if upstream model sources disappear
## The Problem
### Upstream Fragility
- HuggingFace repos can be deleted
- GitHub releases disappear when repos are archived
- PyPI packages can be yanked
- Companies shut down and take their models offline
- Rate limiting breaks automated deployments
### Real Examples
- ✅ Coqui AI: Company shut down 2024, repo archived
- ✅ Rhasspy Piper: Original project abandoned
- ✅ Suno Bark: Company pivoted, model archived
- ⚠️ XTTS-v2: Depends on archived Coqui repo
## Multi-Tier Mirror Architecture
### Tier 1: Upstream Sources (Primary)
Always try upstream first - they're fastest and most up-to-date.
**Sources:**
- HuggingFace Hub (huggingface.co)
- PyPI (pypi.org)
- GitHub Releases
- Official project websites
**Advantages:**
- Latest versions
- Fast CDN delivery
- Community validation
**Disadvantages:**
- Can disappear
- Rate limits
- Requires internet
---
### Tier 2: UncloseAI Mirror (Secondary)
Self-hosted mirror under our control.
**Location:** ai.foxhop.net
**Storage:** MinIO S3-compatible object storage
**Capacity:** 100GB allocated for models
**Setup:**
```bash
# Install MinIO on ai.foxhop.net
docker run -d \
-p 9000:9000 \
-p 9001:9001 \
--name minio \
-v /data/minio:/data \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=<secure_password>" \
minio/minio server /data --console-address ":9001"
# Create bucket for models
mc alias set unclose http://ai.foxhop.net:9000 admin <password>
mc mb unclose/tts-models
mc policy set download unclose/tts-models
```
**Directory Structure:**
```
tts-models/
├── piper/
│ ├── v1.0.0/
│ │ ├── en/
│ │ │ ├── en_US/
│ │ │ │ └── libritts_r/
│ │ │ │ └── medium/
│ │ │ │ ├── en_US-libritts_r-medium.onnx
│ │ │ │ └── en_US-libritts_r-medium.onnx.json
│ │ └── voices.json (metadata)
├── xtts/
│ └── v2.0.3/
│ ├── model.pth
│ ├── config.json
│ ├── vocab.json
│ └── README.md
├── silero/
│ └── v4/
│ ├── en_v4.pt
│ ├── ru_v4.pt
│ └── models.json
├── styletts2/
│ └── libritts/
│ ├── checkpoint.pt
│ └── config.yml
└── metadata.json (master index)
```
**Sync Script:**
```bash
#!/bin/bash
# scripts/sync_models_to_mirror.sh
# Sync upstream models to UncloseAI mirror
set -euo pipefail
MIRROR_URL="http://ai.foxhop.net:9000/tts-models"
TEMP_DIR="/tmp/model_sync"
# Sync Piper voices
sync_piper() {
echo "Syncing Piper models..."
for voice in en_US-libritts_r-medium en_GB-northern_english_male-medium; do
wget -P "$TEMP_DIR/piper/" \
"https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx" \
"https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx.json"
done
mc cp --recursive "$TEMP_DIR/piper/" unclose/tts-models/piper/v1.0.0/
}
# Sync XTTS
sync_xtts() {
echo "Syncing XTTS v2..."
# Use huggingface-cli or git lfs
git clone https://huggingface.co/coqui/XTTS-v2 "$TEMP_DIR/xtts"
mc cp --recursive "$TEMP_DIR/xtts/" unclose/tts-models/xtts/v2.0.3/
}
# Sync Silero
sync_silero() {
echo "Syncing Silero models..."
wget -P "$TEMP_DIR/silero/" \
"https://models.silero.ai/models/tts/en/v4_en.pt" \
"https://models.silero.ai/models/tts/ru/v4_ru.pt"
mc cp --recursive "$TEMP_DIR/silero/" unclose/tts-models/silero/v4/
}
sync_piper
sync_xtts
sync_silero
echo "✅ Mirror sync complete"
```
**Advantages:**
- Under our control
- No rate limits
- Fast local access
- Can modify models
**Disadvantages:**
- Maintenance overhead
- Storage costs
- Single point of failure (us)
---
### Tier 3: Archive.org (Tertiary)
Public archive for critical models.
**Purpose:** Long-term preservation, public good
**What to Archive:**
- Piper voice pack (full 2GB)
- XTTS-v2 weights
- Key Silero models
- StyleTTS2 checkpoints
**Upload Process:**
```bash
# Install internet archive CLI
pip install internetarchive
# Configure
ia configure
# Upload critical model
ia upload uncloseai-piper-voices-v1.0.0 \
piper_voices.tar.gz \
--metadata="title:Piper TTS Voices v1.0.0" \
--metadata="description:Complete Piper TTS voice collection from rhasspy/piper-voices" \
--metadata="subject:text-to-speech;tts;piper;neural-tts" \
--metadata="creator:UncloseAI Speech Raccoon Mission" \
--metadata="date:2025-11-09"
```
**Advantages:**
- Permanent storage
- Public access
- Free
- Trusted platform
**Disadvantages:**
- Slow downloads
- No control over availability
- Upload limits
---
### Tier 4: IPFS (Experimental)
Decentralized storage for the future.
**Purpose:** Censorship-resistant, distributed
**Implementation:**
```bash
# Pin critical models to IPFS
ipfs add -r piper_voices/
# Output: QmXXXXXXXXXXXXXXXX
# Pin via Pinata or other service
curl -X POST "https://api.pinata.cloud/pinning/pinByHash" \
-H "pinata_api_key: YOUR_KEY" \
-d '{"hashToPin":"QmXXXXXXXXXXXXXXXX"}'
```
**Advantages:**
- Decentralized
- Censorship resistant
- Content-addressed
**Disadvantages:**
- Slow
- Requires pinning service
- Less reliable
---
## Download Strategy with Fallbacks
### Smart Downloader
```python
# src/utils/model_downloader.py
from typing import List, Optional
import requests
import logging
logger = logging.getLogger(__name__)
class ModelDownloader:
"""Download models with automatic fallback to mirrors"""
def __init__(self):
self.mirrors = [
"https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/",
"http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/",
"https://archive.org/download/uncloseai-piper-voices-v1.0.0/",
]
def download(self, model_path: str, output_path: str) -> bool:
"""Try each mirror until successful"""
for mirror_url in self.mirrors:
full_url = f"{mirror_url}{model_path}"
logger.info(f"Trying {full_url}...")
try:
response = requests.get(full_url, stream=True, timeout=30)
if response.status_code == 200:
with open(output_path, 'wb') as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
logger.info(f"✅ Downloaded from {mirror_url}")
return True
except Exception as e:
logger.warning(f"❌ Failed {mirror_url}: {e}")
continue
logger.error(f"All mirrors failed for {model_path}")
return False
```
### Configuration
```yaml
# config/mirrors.yaml
mirrors:
piper:
- https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/
- http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/
- https://archive.org/download/uncloseai-piper-voices-v1.0.0/
- ipfs://QmXXXXXXXXXXXXXXXX/
xtts:
- https://huggingface.co/coqui/XTTS-v2/resolve/main/
- http://ai.foxhop.net:9000/tts-models/xtts/v2.0.3/
- https://archive.org/download/uncloseai-xtts-v2/
silero:
- https://models.silero.ai/models/tts/
- http://ai.foxhop.net:9000/tts-models/silero/v4/
- https://github.com/snakers4/silero-models/releases/download/
retry:
max_attempts: 3
timeout_seconds: 30
backoff_multiplier: 2
```
---
## Implementation Checklist
### Phase 1: Setup Mirror Infrastructure
- [ ] Deploy MinIO on ai.foxhop.net
- [ ] Create `tts-models` bucket
- [ ] Set up public read access
- [ ] Configure DNS/CDN (optional)
### Phase 2: Initial Sync
- [ ] Download all Piper voices (2GB)
- [ ] Download XTTS-v2 (1.8GB)
- [ ] Upload to MinIO mirror
- [ ] Test download from mirror
### Phase 3: Implement Fallback Logic
- [ ] Create `ModelDownloader` class
- [ ] Add mirror config to `config/mirrors.yaml`
- [ ] Update download scripts to use fallbacks
- [ ] Add mirror health checks
### Phase 4: Archive Critical Models
- [ ] Upload Piper voices to Archive.org
- [ ] Upload XTTS-v2 to Archive.org
- [ ] Document archive locations
- [ ] Test restoration from archive
### Phase 5: Automation
- [ ] Create sync script (`scripts/sync_models.sh`)
- [ ] Set up cron job for weekly sync
- [ ] Monitor mirror disk usage
- [ ] Alert on upstream changes
### Phase 6: Future Engines
- [ ] Add Silero to mirror
- [ ] Add StyleTTS2 to mirror
- [ ] Add Fish Speech to mirror
---
## Monitoring and Maintenance
### Health Checks
```bash
#!/bin/bash
# scripts/check_mirrors.sh
# Verify all mirrors are accessible
MODELS=(
"piper/v1.0.0/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx"
"xtts/v2.0.3/model.pth"
)
for model in "${MODELS[@]}"; do
echo "Checking $model..."
# Check HuggingFace
curl -I "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/$model" | head -n 1
# Check our mirror
curl -I "http://ai.foxhop.net:9000/tts-models/$model" | head -n 1
echo "---"
done
```
### Storage Usage
```bash
# Monitor MinIO usage
mc du unclose/tts-models
# Expected:
# piper/: 2GB
# xtts/: 1.8GB
# silero/: 500MB
# Total: ~5GB
```
---
## Cost Estimation
### Storage (100GB allocated)
- MinIO on existing server: **$0** (using spare disk)
- Bandwidth: **$0** (self-hosted, unlimited)
### Archive.org
- Storage: **$0** (free)
- Bandwidth: **$0** (free)
### IPFS Pinning (Optional)
- Pinata: **$20/month** for 100GB
- Or self-host: **$0**
**Total Cost: $0-20/month**
---
## Recovery Scenarios
### Scenario 1: HuggingFace is down
1. Downloader tries HF, gets timeout
2. Falls back to ai.foxhop.net mirror ✅
3. Download succeeds in 30 seconds
### Scenario 2: Our mirror is down
1. Downloader tries ai.foxhop.net, fails
2. Falls back to Archive.org ✅
3. Download succeeds in 2 minutes (slower)
### Scenario 3: Total internet failure
1. Models already cached in `/app/voices/`
2. Service continues with cached models ✅
3. No downloads needed for operation
### Scenario 4: Apocalypse (all servers gone)
1. Restore from Archive.org archive
2. Restore from IPFS if configured
3. Restore from torrents if distributed
4. Rebuild from source if absolutely necessary
---
**Last Updated:** 2025-11-09
**Raccoon Status:** 🦝 Building resilient caches like storing nuts for winter

375
docs/MODELS.md Normal file
View file

@ -0,0 +1,375 @@
# TTS Models and Engines
**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into UncloseAI Speech
## Currently Integrated
### 1. Piper TTS ✅
**Status:** Working with absolute paths
**Original Project:** rhasspy/piper (abandoned)
**Fork:** OHF-Voice/piper1-gpl v1.3.0
**Current Package:** PyPI `piper-tts>=1.2.0`
**Features:**
- Fast CPU-based neural TTS
- ~100+ high-quality voices
- Multilingual support
- ONNX runtime
- Low memory footprint (~100MB per voice)
**Voices Available:**
- English (US, GB, multiple accents)
- Spanish, French, German, Italian
- Russian, Polish, Ukrainian
- Chinese, Japanese, Korean
- Many more languages
**Model Source:**
- HuggingFace: `rhasspy/piper-voices`
- Direct download: `https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/`
**Integration:**
- Used for `tts-1` model (fast, lower quality)
- Models stored in `/app/voices/en/en_US/libritts_r/medium/`
- Configuration via absolute paths in `voice_to_speaker.yaml`
**Example Config:**
```yaml
tts-1:
alloy:
model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
speaker: 79
```
**Raccoon Notes:**
- Original rhasspy project abandoned
- OHF-Voice fork has no PyPI package
- Need to create our own PyPI package or vendor the code
- Mirror all voices to prevent HuggingFace dependency
---
### 2. Coqui XTTS v2 ✅
**Status:** Integrated as tts-1-hd
**Original Project:** coqui-ai/TTS (company shut down, archived)
**Current Package:** PyPI `coqui-tts[languages]`
**Features:**
- High-quality multilingual TTS
- Voice cloning from 6-second samples
- Emotional prosody control
- GPU accelerated (NVIDIA/ROCm)
- ~1.8GB model size
**Languages:**
- English, Spanish, French, German, Italian, Portuguese
- Polish, Turkish, Russian, Dutch, Czech
- Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi
**Model Source:**
- HuggingFace: `coqui/XTTS-v2`
- Auto-downloaded on first use
**Integration:**
- Used for `tts-1-hd` model (slow, high quality)
- Voice cloning with custom WAV samples
- Language auto-detection with `langdetect`
**Example Config:**
```yaml
tts-1-hd:
alloy:
model: xtts
speaker: /app/voices/alloy.wav
language: en
```
**Raccoon Notes:**
- Coqui company shut down in 2024
- Repository archived but code still works
- Community forks emerging
- Must mirror XTTS-v2 weights before they disappear
- Consider forking to uncloseai-xtts
---
## High Priority Integration Targets
### 3. Silero TTS 🎯
**Status:** NOT INTEGRATED - HIGH PRIORITY
**Project:** snakers4/silero-models (still active!)
**License:** Apache 2.0
**Why Integrate:**
- STILL ACTIVELY MAINTAINED
- Fast, small models (~50-100MB each)
- High quality for size
- Easy integration (PyTorch)
- Commercial-friendly license
**Features:**
- Multilingual: English, Russian, German, Spanish, French
- Multiple speakers per language
- Emotion control
- CPU friendly
- Real-time capable
**Models:**
- English: 4 speakers (en_v4)
- Russian: 8+ speakers (ru_v4)
- German: 1 speaker (de_v3)
- Spanish: 2 speakers (es_v1)
- French: 1 speaker (fr_v3)
**Model Source:**
- GitHub Releases: https://github.com/snakers4/silero-models/releases
- PyTorch Hub
- Direct ONNX models available
**Estimated Integration Effort:** 2-4 hours
- Add to requirements.txt: `silero` or direct PyTorch load
- Create `src/engines/silero.py`
- Download models to `/app/models/silero/`
- Add voice mappings to config
**Example Usage:**
```python
import torch
model, symbols, sample_rate, example_text, apply_tts = torch.hub.load(
repo_or_dir='snakers4/silero-models',
model='silero_tts',
language='en',
speaker='v4_en'
)
audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate)
```
**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio)
---
### 4. StyleTTS2 🎯
**Status:** NOT INTEGRATED - HIGH PRIORITY
**Project:** yl4579/StyleTTS2 (research, somewhat active)
**License:** MIT
**Why Integrate:**
- State-of-the-art quality
- Best prosody and naturalness
- Voice cloning capability
- Style/emotion control
- Research-grade results
**Features:**
- Human-level prosody
- Zero-shot voice cloning
- Style transfer
- Emotion and speaking style control
- LibriTTS trained models
**Challenges:**
- Complex dependencies
- Requires phonemizer
- Slower than other engines
- GPU recommended
**Model Source:**
- HuggingFace: `yl4579/StyleTTS2-LibriTTS`
- GitHub releases
**Estimated Integration Effort:** 6-8 hours
- Complex dependency chain
- Need phonemizer setup
- Create custom engine wrapper
- May need model quantization for production
**Raccoon Priority:** ⭐⭐⭐⭐ (Best quality, but complex)
---
### 5. Fish Speech 🎯
**Status:** NOT INTEGRATED - MEDIUM PRIORITY
**Project:** fishaudio/fish-speech (active)
**License:** Apache 2.0
**Why Integrate:**
- Fast and efficient
- Good multilingual support
- Active development
- Clean API
**Features:**
- Fast inference
- Multilingual (EN, ZH, JA)
- Voice cloning
- Streaming support
- Modern architecture
**Model Source:**
- HuggingFace: `fishaudio/fish-speech-1`
- GitHub releases
**Estimated Integration Effort:** 4-6 hours
**Raccoon Priority:** ⭐⭐⭐ (Active, good quality, but newer/less proven)
---
## Medium Priority Targets
### 6. Kokoro TTS
**Status:** NOT INTEGRATED
**Project:** hexgrad/kokoro (new, active)
**License:** Apache 2.0
**Features:**
- Fast, small, quality
- Multiple voices
- Good English support
- Emerging project
**Raccoon Priority:** ⭐⭐⭐ (Promising but new)
---
### 7. Bark (Suno AI)
**Status:** NOT INTEGRATED
**Project:** suno-ai/bark (archived, company pivoted to music)
**License:** MIT
**Why Consider:**
- Can generate music and sound effects
- Non-verbal sounds (laughs, sighs)
- Multiple languages
- Background audio
**Why Low Priority:**
- Very slow generation
- Large models (~10GB)
- Company abandoned it
- Quality inconsistent
**Raccoon Priority:** ⭐⭐ (Unique features, but slow and abandoned)
---
## Low Priority / Archived
### 8. Tortoise TTS
**Status:** NOT INTEGRATED
**Project:** neonbjb/tortoise-tts (low activity)
**License:** Apache 2.0
**Features:**
- Very high quality
- Voice cloning
**Why Low Priority:**
- Extremely slow (minutes per sentence)
- Not practical for API use
- Better alternatives exist now
**Raccoon Priority:** ⭐ (Too slow for production)
---
### 9. MetaVoice
**Status:** NOT INTEGRATED
**Project:** metavoiceio/metavoice-src (partially abandoned)
**License:** Apache 2.0
**Features:**
- Long-form TTS
- Emotional control
- Voice cloning
**Why Low Priority:**
- Unclear maintenance status
- Complex setup
- Alternatives are better
**Raccoon Priority:** ⭐ (Uncertain future)
---
### 10. Mozilla TTS
**Status:** NOT INTEGRATED
**Project:** mozilla/TTS (archived, became Coqui)
**License:** MPL 2.0
**Why Skip:**
- Fully superseded by Coqui
- No unique capabilities
- Outdated architecture
**Raccoon Priority:** ⛔ (Skip - use Coqui instead)
---
## Integration Roadmap
### Phase 1: Quick Wins (Next 1-2 weeks)
1. ✅ Fix Piper absolute paths
2. ✅ Audit repository
3. [ ] Integrate Silero TTS (2-4 hours)
4. [ ] Set up model mirror on ai.foxhop.net
5. [ ] Test Silero with existing API
### Phase 2: High Quality (2-4 weeks)
1. [ ] Integrate StyleTTS2
2. [ ] Create engine abstraction layer
3. [ ] Refactor speech.py to use engines
4. [ ] Add Fish Speech support
### Phase 3: Resilience (1-2 months)
1. [ ] Implement binary mirror system
2. [ ] Create fallback download logic
3. [ ] Archive critical models to Archive.org
4. [ ] Document all model sources
### Phase 4: Advanced Features (2+ months)
1. [ ] Voice cloning API endpoint
2. [ ] Emotion/style control
3. [ ] Streaming TTS
4. [ ] Multi-speaker conversations
## Model Storage Requirements
Current:
- Piper voices: ~2GB (all languages)
- XTTS v2: ~1.8GB
With all planned engines:
- Silero models: ~500MB (all languages)
- StyleTTS2: ~2GB (base model)
- Fish Speech: ~1.5GB
- **Total: ~8GB** for complete coverage
Mirror storage needed: ~20GB (with redundancy and archives)
## Performance Targets
| Engine | Speed (RTF) | Quality | Use Case |
|--------|-------------|---------|----------|
| Piper | 0.05x | Good | Fast API responses |
| Silero | 0.1x | Good | Balanced speed/quality |
| XTTS | 0.3x | Excellent | Voice cloning |
| StyleTTS2 | 0.5x | Best | Premium quality |
| Fish Speech | 0.15x | Very Good | Multilingual |
RTF = Real-time factor (lower is faster, 1.0 = real-time)
---
**Last Updated:** 2025-11-09
**Raccoon Status:** 🦝 Actively hunting for TTS models in the dumpsters of abandoned repos