diff --git a/docs/MIRRORS.md b/docs/MIRRORS.md new file mode 100644 index 0000000..096f75a --- /dev/null +++ b/docs/MIRRORS.md @@ -0,0 +1,426 @@ +# Binary Mirror Strategy + +**Purpose:** Ensure UncloseAI Speech keeps working even if upstream model sources disappear + +## The Problem + +### Upstream Fragility +- HuggingFace repos can be deleted +- GitHub releases disappear when repos are archived +- PyPI packages can be yanked +- Companies shut down and take their models offline +- Rate limiting breaks automated deployments + +### Real Examples +- ✅ Coqui AI: Company shut down 2024, repo archived +- ✅ Rhasspy Piper: Original project abandoned +- ✅ Suno Bark: Company pivoted, model archived +- ⚠️ XTTS-v2: Depends on archived Coqui repo + +## Multi-Tier Mirror Architecture + +### Tier 1: Upstream Sources (Primary) +Always try upstream first - they're fastest and most up-to-date. + +**Sources:** +- HuggingFace Hub (huggingface.co) +- PyPI (pypi.org) +- GitHub Releases +- Official project websites + +**Advantages:** +- Latest versions +- Fast CDN delivery +- Community validation + +**Disadvantages:** +- Can disappear +- Rate limits +- Requires internet + +--- + +### Tier 2: UncloseAI Mirror (Secondary) +Self-hosted mirror under our control. + +**Location:** ai.foxhop.net +**Storage:** MinIO S3-compatible object storage +**Capacity:** 100GB allocated for models + +**Setup:** +```bash +# Install MinIO on ai.foxhop.net +docker run -d \ + -p 9000:9000 \ + -p 9001:9001 \ + --name minio \ + -v /data/minio:/data \ + -e "MINIO_ROOT_USER=admin" \ + -e "MINIO_ROOT_PASSWORD=" \ + minio/minio server /data --console-address ":9001" + +# Create bucket for models +mc alias set unclose http://ai.foxhop.net:9000 admin +mc mb unclose/tts-models +mc policy set download unclose/tts-models +``` + +**Directory Structure:** +``` +tts-models/ +├── piper/ +│ ├── v1.0.0/ +│ │ ├── en/ +│ │ │ ├── en_US/ +│ │ │ │ └── libritts_r/ +│ │ │ │ └── medium/ +│ │ │ │ ├── en_US-libritts_r-medium.onnx +│ │ │ │ └── en_US-libritts_r-medium.onnx.json +│ │ └── voices.json (metadata) +├── xtts/ +│ └── v2.0.3/ +│ ├── model.pth +│ ├── config.json +│ ├── vocab.json +│ └── README.md +├── silero/ +│ └── v4/ +│ ├── en_v4.pt +│ ├── ru_v4.pt +│ └── models.json +├── styletts2/ +│ └── libritts/ +│ ├── checkpoint.pt +│ └── config.yml +└── metadata.json (master index) +``` + +**Sync Script:** +```bash +#!/bin/bash +# scripts/sync_models_to_mirror.sh +# Sync upstream models to UncloseAI mirror + +set -euo pipefail + +MIRROR_URL="http://ai.foxhop.net:9000/tts-models" +TEMP_DIR="/tmp/model_sync" + +# Sync Piper voices +sync_piper() { + echo "Syncing Piper models..." + for voice in en_US-libritts_r-medium en_GB-northern_english_male-medium; do + wget -P "$TEMP_DIR/piper/" \ + "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx" \ + "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx.json" + done + mc cp --recursive "$TEMP_DIR/piper/" unclose/tts-models/piper/v1.0.0/ +} + +# Sync XTTS +sync_xtts() { + echo "Syncing XTTS v2..." + # Use huggingface-cli or git lfs + git clone https://huggingface.co/coqui/XTTS-v2 "$TEMP_DIR/xtts" + mc cp --recursive "$TEMP_DIR/xtts/" unclose/tts-models/xtts/v2.0.3/ +} + +# Sync Silero +sync_silero() { + echo "Syncing Silero models..." + wget -P "$TEMP_DIR/silero/" \ + "https://models.silero.ai/models/tts/en/v4_en.pt" \ + "https://models.silero.ai/models/tts/ru/v4_ru.pt" + mc cp --recursive "$TEMP_DIR/silero/" unclose/tts-models/silero/v4/ +} + +sync_piper +sync_xtts +sync_silero + +echo "✅ Mirror sync complete" +``` + +**Advantages:** +- Under our control +- No rate limits +- Fast local access +- Can modify models + +**Disadvantages:** +- Maintenance overhead +- Storage costs +- Single point of failure (us) + +--- + +### Tier 3: Archive.org (Tertiary) +Public archive for critical models. + +**Purpose:** Long-term preservation, public good + +**What to Archive:** +- Piper voice pack (full 2GB) +- XTTS-v2 weights +- Key Silero models +- StyleTTS2 checkpoints + +**Upload Process:** +```bash +# Install internet archive CLI +pip install internetarchive + +# Configure +ia configure + +# Upload critical model +ia upload uncloseai-piper-voices-v1.0.0 \ + piper_voices.tar.gz \ + --metadata="title:Piper TTS Voices v1.0.0" \ + --metadata="description:Complete Piper TTS voice collection from rhasspy/piper-voices" \ + --metadata="subject:text-to-speech;tts;piper;neural-tts" \ + --metadata="creator:UncloseAI Speech Raccoon Mission" \ + --metadata="date:2025-11-09" +``` + +**Advantages:** +- Permanent storage +- Public access +- Free +- Trusted platform + +**Disadvantages:** +- Slow downloads +- No control over availability +- Upload limits + +--- + +### Tier 4: IPFS (Experimental) +Decentralized storage for the future. + +**Purpose:** Censorship-resistant, distributed + +**Implementation:** +```bash +# Pin critical models to IPFS +ipfs add -r piper_voices/ +# Output: QmXXXXXXXXXXXXXXXX + +# Pin via Pinata or other service +curl -X POST "https://api.pinata.cloud/pinning/pinByHash" \ + -H "pinata_api_key: YOUR_KEY" \ + -d '{"hashToPin":"QmXXXXXXXXXXXXXXXX"}' +``` + +**Advantages:** +- Decentralized +- Censorship resistant +- Content-addressed + +**Disadvantages:** +- Slow +- Requires pinning service +- Less reliable + +--- + +## Download Strategy with Fallbacks + +### Smart Downloader + +```python +# src/utils/model_downloader.py +from typing import List, Optional +import requests +import logging + +logger = logging.getLogger(__name__) + +class ModelDownloader: + """Download models with automatic fallback to mirrors""" + + def __init__(self): + self.mirrors = [ + "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/", + "http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/", + "https://archive.org/download/uncloseai-piper-voices-v1.0.0/", + ] + + def download(self, model_path: str, output_path: str) -> bool: + """Try each mirror until successful""" + for mirror_url in self.mirrors: + full_url = f"{mirror_url}{model_path}" + logger.info(f"Trying {full_url}...") + + try: + response = requests.get(full_url, stream=True, timeout=30) + if response.status_code == 200: + with open(output_path, 'wb') as f: + for chunk in response.iter_content(chunk_size=8192): + f.write(chunk) + logger.info(f"✅ Downloaded from {mirror_url}") + return True + except Exception as e: + logger.warning(f"❌ Failed {mirror_url}: {e}") + continue + + logger.error(f"All mirrors failed for {model_path}") + return False +``` + +### Configuration + +```yaml +# config/mirrors.yaml +mirrors: + piper: + - https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/ + - http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/ + - https://archive.org/download/uncloseai-piper-voices-v1.0.0/ + - ipfs://QmXXXXXXXXXXXXXXXX/ + + xtts: + - https://huggingface.co/coqui/XTTS-v2/resolve/main/ + - http://ai.foxhop.net:9000/tts-models/xtts/v2.0.3/ + - https://archive.org/download/uncloseai-xtts-v2/ + + silero: + - https://models.silero.ai/models/tts/ + - http://ai.foxhop.net:9000/tts-models/silero/v4/ + - https://github.com/snakers4/silero-models/releases/download/ + +retry: + max_attempts: 3 + timeout_seconds: 30 + backoff_multiplier: 2 +``` + +--- + +## Implementation Checklist + +### Phase 1: Setup Mirror Infrastructure +- [ ] Deploy MinIO on ai.foxhop.net +- [ ] Create `tts-models` bucket +- [ ] Set up public read access +- [ ] Configure DNS/CDN (optional) + +### Phase 2: Initial Sync +- [ ] Download all Piper voices (2GB) +- [ ] Download XTTS-v2 (1.8GB) +- [ ] Upload to MinIO mirror +- [ ] Test download from mirror + +### Phase 3: Implement Fallback Logic +- [ ] Create `ModelDownloader` class +- [ ] Add mirror config to `config/mirrors.yaml` +- [ ] Update download scripts to use fallbacks +- [ ] Add mirror health checks + +### Phase 4: Archive Critical Models +- [ ] Upload Piper voices to Archive.org +- [ ] Upload XTTS-v2 to Archive.org +- [ ] Document archive locations +- [ ] Test restoration from archive + +### Phase 5: Automation +- [ ] Create sync script (`scripts/sync_models.sh`) +- [ ] Set up cron job for weekly sync +- [ ] Monitor mirror disk usage +- [ ] Alert on upstream changes + +### Phase 6: Future Engines +- [ ] Add Silero to mirror +- [ ] Add StyleTTS2 to mirror +- [ ] Add Fish Speech to mirror + +--- + +## Monitoring and Maintenance + +### Health Checks + +```bash +#!/bin/bash +# scripts/check_mirrors.sh +# Verify all mirrors are accessible + +MODELS=( + "piper/v1.0.0/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx" + "xtts/v2.0.3/model.pth" +) + +for model in "${MODELS[@]}"; do + echo "Checking $model..." + + # Check HuggingFace + curl -I "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/$model" | head -n 1 + + # Check our mirror + curl -I "http://ai.foxhop.net:9000/tts-models/$model" | head -n 1 + + echo "---" +done +``` + +### Storage Usage + +```bash +# Monitor MinIO usage +mc du unclose/tts-models + +# Expected: +# piper/: 2GB +# xtts/: 1.8GB +# silero/: 500MB +# Total: ~5GB +``` + +--- + +## Cost Estimation + +### Storage (100GB allocated) +- MinIO on existing server: **$0** (using spare disk) +- Bandwidth: **$0** (self-hosted, unlimited) + +### Archive.org +- Storage: **$0** (free) +- Bandwidth: **$0** (free) + +### IPFS Pinning (Optional) +- Pinata: **$20/month** for 100GB +- Or self-host: **$0** + +**Total Cost: $0-20/month** + +--- + +## Recovery Scenarios + +### Scenario 1: HuggingFace is down +1. Downloader tries HF, gets timeout +2. Falls back to ai.foxhop.net mirror ✅ +3. Download succeeds in 30 seconds + +### Scenario 2: Our mirror is down +1. Downloader tries ai.foxhop.net, fails +2. Falls back to Archive.org ✅ +3. Download succeeds in 2 minutes (slower) + +### Scenario 3: Total internet failure +1. Models already cached in `/app/voices/` +2. Service continues with cached models ✅ +3. No downloads needed for operation + +### Scenario 4: Apocalypse (all servers gone) +1. Restore from Archive.org archive +2. Restore from IPFS if configured +3. Restore from torrents if distributed +4. Rebuild from source if absolutely necessary + +--- + +**Last Updated:** 2025-11-09 +**Raccoon Status:** 🦝 Building resilient caches like storing nuts for winter diff --git a/docs/MODELS.md b/docs/MODELS.md new file mode 100644 index 0000000..c92e42e --- /dev/null +++ b/docs/MODELS.md @@ -0,0 +1,375 @@ +# TTS Models and Engines + +**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into UncloseAI Speech + +## Currently Integrated + +### 1. Piper TTS ✅ + +**Status:** Working with absolute paths +**Original Project:** rhasspy/piper (abandoned) +**Fork:** OHF-Voice/piper1-gpl v1.3.0 +**Current Package:** PyPI `piper-tts>=1.2.0` + +**Features:** +- Fast CPU-based neural TTS +- ~100+ high-quality voices +- Multilingual support +- ONNX runtime +- Low memory footprint (~100MB per voice) + +**Voices Available:** +- English (US, GB, multiple accents) +- Spanish, French, German, Italian +- Russian, Polish, Ukrainian +- Chinese, Japanese, Korean +- Many more languages + +**Model Source:** +- HuggingFace: `rhasspy/piper-voices` +- Direct download: `https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/` + +**Integration:** +- Used for `tts-1` model (fast, lower quality) +- Models stored in `/app/voices/en/en_US/libritts_r/medium/` +- Configuration via absolute paths in `voice_to_speaker.yaml` + +**Example Config:** +```yaml +tts-1: + alloy: + model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx + speaker: 79 +``` + +**Raccoon Notes:** +- Original rhasspy project abandoned +- OHF-Voice fork has no PyPI package +- Need to create our own PyPI package or vendor the code +- Mirror all voices to prevent HuggingFace dependency + +--- + +### 2. Coqui XTTS v2 ✅ + +**Status:** Integrated as tts-1-hd +**Original Project:** coqui-ai/TTS (company shut down, archived) +**Current Package:** PyPI `coqui-tts[languages]` + +**Features:** +- High-quality multilingual TTS +- Voice cloning from 6-second samples +- Emotional prosody control +- GPU accelerated (NVIDIA/ROCm) +- ~1.8GB model size + +**Languages:** +- English, Spanish, French, German, Italian, Portuguese +- Polish, Turkish, Russian, Dutch, Czech +- Arabic, Chinese (Mandarin), Japanese, Hungarian, Korean, Hindi + +**Model Source:** +- HuggingFace: `coqui/XTTS-v2` +- Auto-downloaded on first use + +**Integration:** +- Used for `tts-1-hd` model (slow, high quality) +- Voice cloning with custom WAV samples +- Language auto-detection with `langdetect` + +**Example Config:** +```yaml +tts-1-hd: + alloy: + model: xtts + speaker: /app/voices/alloy.wav + language: en +``` + +**Raccoon Notes:** +- Coqui company shut down in 2024 +- Repository archived but code still works +- Community forks emerging +- Must mirror XTTS-v2 weights before they disappear +- Consider forking to uncloseai-xtts + +--- + +## High Priority Integration Targets + +### 3. Silero TTS 🎯 + +**Status:** NOT INTEGRATED - HIGH PRIORITY +**Project:** snakers4/silero-models (still active!) +**License:** Apache 2.0 + +**Why Integrate:** +- STILL ACTIVELY MAINTAINED +- Fast, small models (~50-100MB each) +- High quality for size +- Easy integration (PyTorch) +- Commercial-friendly license + +**Features:** +- Multilingual: English, Russian, German, Spanish, French +- Multiple speakers per language +- Emotion control +- CPU friendly +- Real-time capable + +**Models:** +- English: 4 speakers (en_v4) +- Russian: 8+ speakers (ru_v4) +- German: 1 speaker (de_v3) +- Spanish: 2 speakers (es_v1) +- French: 1 speaker (fr_v3) + +**Model Source:** +- GitHub Releases: https://github.com/snakers4/silero-models/releases +- PyTorch Hub +- Direct ONNX models available + +**Estimated Integration Effort:** 2-4 hours +- Add to requirements.txt: `silero` or direct PyTorch load +- Create `src/engines/silero.py` +- Download models to `/app/models/silero/` +- Add voice mappings to config + +**Example Usage:** +```python +import torch +model, symbols, sample_rate, example_text, apply_tts = torch.hub.load( + repo_or_dir='snakers4/silero-models', + model='silero_tts', + language='en', + speaker='v4_en' +) +audio = apply_tts(text=text, speaker='en_0', sample_rate=sample_rate) +``` + +**Raccoon Priority:** ⭐⭐⭐⭐⭐ (Active project, great quality/size ratio) + +--- + +### 4. StyleTTS2 🎯 + +**Status:** NOT INTEGRATED - HIGH PRIORITY +**Project:** yl4579/StyleTTS2 (research, somewhat active) +**License:** MIT + +**Why Integrate:** +- State-of-the-art quality +- Best prosody and naturalness +- Voice cloning capability +- Style/emotion control +- Research-grade results + +**Features:** +- Human-level prosody +- Zero-shot voice cloning +- Style transfer +- Emotion and speaking style control +- LibriTTS trained models + +**Challenges:** +- Complex dependencies +- Requires phonemizer +- Slower than other engines +- GPU recommended + +**Model Source:** +- HuggingFace: `yl4579/StyleTTS2-LibriTTS` +- GitHub releases + +**Estimated Integration Effort:** 6-8 hours +- Complex dependency chain +- Need phonemizer setup +- Create custom engine wrapper +- May need model quantization for production + +**Raccoon Priority:** ⭐⭐⭐⭐ (Best quality, but complex) + +--- + +### 5. Fish Speech 🎯 + +**Status:** NOT INTEGRATED - MEDIUM PRIORITY +**Project:** fishaudio/fish-speech (active) +**License:** Apache 2.0 + +**Why Integrate:** +- Fast and efficient +- Good multilingual support +- Active development +- Clean API + +**Features:** +- Fast inference +- Multilingual (EN, ZH, JA) +- Voice cloning +- Streaming support +- Modern architecture + +**Model Source:** +- HuggingFace: `fishaudio/fish-speech-1` +- GitHub releases + +**Estimated Integration Effort:** 4-6 hours + +**Raccoon Priority:** ⭐⭐⭐ (Active, good quality, but newer/less proven) + +--- + +## Medium Priority Targets + +### 6. Kokoro TTS + +**Status:** NOT INTEGRATED +**Project:** hexgrad/kokoro (new, active) +**License:** Apache 2.0 + +**Features:** +- Fast, small, quality +- Multiple voices +- Good English support +- Emerging project + +**Raccoon Priority:** ⭐⭐⭐ (Promising but new) + +--- + +### 7. Bark (Suno AI) + +**Status:** NOT INTEGRATED +**Project:** suno-ai/bark (archived, company pivoted to music) +**License:** MIT + +**Why Consider:** +- Can generate music and sound effects +- Non-verbal sounds (laughs, sighs) +- Multiple languages +- Background audio + +**Why Low Priority:** +- Very slow generation +- Large models (~10GB) +- Company abandoned it +- Quality inconsistent + +**Raccoon Priority:** ⭐⭐ (Unique features, but slow and abandoned) + +--- + +## Low Priority / Archived + +### 8. Tortoise TTS + +**Status:** NOT INTEGRATED +**Project:** neonbjb/tortoise-tts (low activity) +**License:** Apache 2.0 + +**Features:** +- Very high quality +- Voice cloning + +**Why Low Priority:** +- Extremely slow (minutes per sentence) +- Not practical for API use +- Better alternatives exist now + +**Raccoon Priority:** ⭐ (Too slow for production) + +--- + +### 9. MetaVoice + +**Status:** NOT INTEGRATED +**Project:** metavoiceio/metavoice-src (partially abandoned) +**License:** Apache 2.0 + +**Features:** +- Long-form TTS +- Emotional control +- Voice cloning + +**Why Low Priority:** +- Unclear maintenance status +- Complex setup +- Alternatives are better + +**Raccoon Priority:** ⭐ (Uncertain future) + +--- + +### 10. Mozilla TTS + +**Status:** NOT INTEGRATED +**Project:** mozilla/TTS (archived, became Coqui) +**License:** MPL 2.0 + +**Why Skip:** +- Fully superseded by Coqui +- No unique capabilities +- Outdated architecture + +**Raccoon Priority:** ⛔ (Skip - use Coqui instead) + +--- + +## Integration Roadmap + +### Phase 1: Quick Wins (Next 1-2 weeks) +1. ✅ Fix Piper absolute paths +2. ✅ Audit repository +3. [ ] Integrate Silero TTS (2-4 hours) +4. [ ] Set up model mirror on ai.foxhop.net +5. [ ] Test Silero with existing API + +### Phase 2: High Quality (2-4 weeks) +1. [ ] Integrate StyleTTS2 +2. [ ] Create engine abstraction layer +3. [ ] Refactor speech.py to use engines +4. [ ] Add Fish Speech support + +### Phase 3: Resilience (1-2 months) +1. [ ] Implement binary mirror system +2. [ ] Create fallback download logic +3. [ ] Archive critical models to Archive.org +4. [ ] Document all model sources + +### Phase 4: Advanced Features (2+ months) +1. [ ] Voice cloning API endpoint +2. [ ] Emotion/style control +3. [ ] Streaming TTS +4. [ ] Multi-speaker conversations + +## Model Storage Requirements + +Current: +- Piper voices: ~2GB (all languages) +- XTTS v2: ~1.8GB + +With all planned engines: +- Silero models: ~500MB (all languages) +- StyleTTS2: ~2GB (base model) +- Fish Speech: ~1.5GB +- **Total: ~8GB** for complete coverage + +Mirror storage needed: ~20GB (with redundancy and archives) + +## Performance Targets + +| Engine | Speed (RTF) | Quality | Use Case | +|--------|-------------|---------|----------| +| Piper | 0.05x | Good | Fast API responses | +| Silero | 0.1x | Good | Balanced speed/quality | +| XTTS | 0.3x | Excellent | Voice cloning | +| StyleTTS2 | 0.5x | Best | Premium quality | +| Fish Speech | 0.15x | Very Good | Multilingual | + +RTF = Real-time factor (lower is faster, 1.0 = real-time) + +--- + +**Last Updated:** 2025-11-09 +**Raccoon Status:** 🦝 Actively hunting for TTS models in the dumpsters of abandoned repos