# Binary Mirror Strategy **Purpose:** Ensure uncloseai-speech keeps working even if upstream model sources disappear ## The Problem ### Upstream Fragility - HuggingFace repos can be deleted - GitHub releases disappear when repos are archived - PyPI packages can be yanked - Companies shut down and take their models offline - Rate limiting breaks automated deployments ### Real Examples - ✅ Coqui AI: Company shut down 2024, repo archived - ✅ Rhasspy Piper: Original project abandoned - ✅ Suno Bark: Company pivoted, model archived - ⚠️ XTTS-v2: Depends on archived Coqui repo ## Multi-Tier Mirror Architecture ### Tier 1: Upstream Sources (Primary) Always try upstream first - they're fastest and most up-to-date. **Sources:** - HuggingFace Hub (huggingface.co) - PyPI (pypi.org) - GitHub Releases - Official project websites **Advantages:** - Latest versions - Fast CDN delivery - Community validation **Disadvantages:** - Can disappear - Rate limits - Requires internet --- ### Tier 2: UncloseAI Mirror (Secondary) Self-hosted mirror under our control. **Location:** ai.foxhop.net **Storage:** MinIO S3-compatible object storage **Capacity:** 100GB allocated for models **Setup:** ```bash # Install MinIO on ai.foxhop.net docker run -d \ -p 9000:9000 \ -p 9001:9001 \ --name minio \ -v /data/minio:/data \ -e "MINIO_ROOT_USER=admin" \ -e "MINIO_ROOT_PASSWORD=" \ minio/minio server /data --console-address ":9001" # Create bucket for models mc alias set unclose http://ai.foxhop.net:9000 admin mc mb unclose/tts-models mc policy set download unclose/tts-models ``` **Directory Structure:** ``` tts-models/ ├── piper/ │ ├── v1.0.0/ │ │ ├── en/ │ │ │ ├── en_US/ │ │ │ │ └── libritts_r/ │ │ │ │ └── medium/ │ │ │ │ ├── en_US-libritts_r-medium.onnx │ │ │ │ └── en_US-libritts_r-medium.onnx.json │ │ └── voices.json (metadata) ├── xtts/ │ └── v2.0.3/ │ ├── model.pth │ ├── config.json │ ├── vocab.json │ └── README.md ├── silero/ │ └── v4/ │ ├── en_v4.pt │ ├── ru_v4.pt │ └── models.json ├── styletts2/ │ └── libritts/ │ ├── checkpoint.pt │ └── config.yml └── metadata.json (master index) ``` **Sync Script:** ```bash #!/bin/bash # scripts/sync_models_to_mirror.sh # Sync upstream models to UncloseAI mirror set -euo pipefail MIRROR_URL="http://ai.foxhop.net:9000/tts-models" TEMP_DIR="/tmp/model_sync" # Sync Piper voices sync_piper() { echo "Syncing Piper models..." for voice in en_US-libritts_r-medium en_GB-northern_english_male-medium; do wget -P "$TEMP_DIR/piper/" \ "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx" \ "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/libritts_r/medium/${voice}.onnx.json" done mc cp --recursive "$TEMP_DIR/piper/" unclose/tts-models/piper/v1.0.0/ } # Sync XTTS sync_xtts() { echo "Syncing XTTS v2..." # Use huggingface-cli or git lfs git clone https://huggingface.co/coqui/XTTS-v2 "$TEMP_DIR/xtts" mc cp --recursive "$TEMP_DIR/xtts/" unclose/tts-models/xtts/v2.0.3/ } # Sync Silero sync_silero() { echo "Syncing Silero models..." wget -P "$TEMP_DIR/silero/" \ "https://models.silero.ai/models/tts/en/v4_en.pt" \ "https://models.silero.ai/models/tts/ru/v4_ru.pt" mc cp --recursive "$TEMP_DIR/silero/" unclose/tts-models/silero/v4/ } sync_piper sync_xtts sync_silero echo "✅ Mirror sync complete" ``` **Advantages:** - Under our control - No rate limits - Fast local access - Can modify models **Disadvantages:** - Maintenance overhead - Storage costs - Single point of failure (us) --- ### Tier 3: Archive.org (Tertiary) Public archive for critical models. **Purpose:** Long-term preservation, public good **What to Archive:** - Piper voice pack (full 2GB) - XTTS-v2 weights - Key Silero models - StyleTTS2 checkpoints **Upload Process:** ```bash # Install internet archive CLI pip install internetarchive # Configure ia configure # Upload critical model ia upload uncloseai-piper-voices-v1.0.0 \ piper_voices.tar.gz \ --metadata="title:Piper TTS Voices v1.0.0" \ --metadata="description:Complete Piper TTS voice collection from rhasspy/piper-voices" \ --metadata="subject:text-to-speech;tts;piper;neural-tts" \ --metadata="creator:uncloseai Raccoon Mission" \ --metadata="date:2025-11-09" ``` **Advantages:** - Permanent storage - Public access - Free - Trusted platform **Disadvantages:** - Slow downloads - No control over availability - Upload limits --- ### Tier 4: IPFS (Experimental) Decentralized storage for the future. **Purpose:** Censorship-resistant, distributed **Implementation:** ```bash # Pin critical models to IPFS ipfs add -r piper_voices/ # Output: QmXXXXXXXXXXXXXXXX # Pin via Pinata or other service curl -X POST "https://api.pinata.cloud/pinning/pinByHash" \ -H "pinata_api_key: YOUR_KEY" \ -d '{"hashToPin":"QmXXXXXXXXXXXXXXXX"}' ``` **Advantages:** - Decentralized - Censorship resistant - Content-addressed **Disadvantages:** - Slow - Requires pinning service - Less reliable --- ## Download Strategy with Fallbacks ### Smart Downloader ```python # src/utils/model_downloader.py from typing import List, Optional import requests import logging logger = logging.getLogger(__name__) class ModelDownloader: """Download models with automatic fallback to mirrors""" def __init__(self): self.mirrors = [ "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/", "http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/", "https://archive.org/download/uncloseai-piper-voices-v1.0.0/", ] def download(self, model_path: str, output_path: str) -> bool: """Try each mirror until successful""" for mirror_url in self.mirrors: full_url = f"{mirror_url}{model_path}" logger.info(f"Trying {full_url}...") try: response = requests.get(full_url, stream=True, timeout=30) if response.status_code == 200: with open(output_path, 'wb') as f: for chunk in response.iter_content(chunk_size=8192): f.write(chunk) logger.info(f"✅ Downloaded from {mirror_url}") return True except Exception as e: logger.warning(f"❌ Failed {mirror_url}: {e}") continue logger.error(f"All mirrors failed for {model_path}") return False ``` ### Configuration ```yaml # config/mirrors.yaml mirrors: piper: - https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/ - http://ai.foxhop.net:9000/tts-models/piper/v1.0.0/ - https://archive.org/download/uncloseai-piper-voices-v1.0.0/ - ipfs://QmXXXXXXXXXXXXXXXX/ xtts: - https://huggingface.co/coqui/XTTS-v2/resolve/main/ - http://ai.foxhop.net:9000/tts-models/xtts/v2.0.3/ - https://archive.org/download/uncloseai-xtts-v2/ silero: - https://models.silero.ai/models/tts/ - http://ai.foxhop.net:9000/tts-models/silero/v4/ - https://github.com/snakers4/silero-models/releases/download/ retry: max_attempts: 3 timeout_seconds: 30 backoff_multiplier: 2 ``` --- ## Implementation Checklist ### Phase 1: Setup Mirror Infrastructure - [ ] Deploy MinIO on ai.foxhop.net - [ ] Create `tts-models` bucket - [ ] Set up public read access - [ ] Configure DNS/CDN (optional) ### Phase 2: Initial Sync - [ ] Download all Piper voices (2GB) - [ ] Download XTTS-v2 (1.8GB) - [ ] Upload to MinIO mirror - [ ] Test download from mirror ### Phase 3: Implement Fallback Logic - [ ] Create `ModelDownloader` class - [ ] Add mirror config to `config/mirrors.yaml` - [ ] Update download scripts to use fallbacks - [ ] Add mirror health checks ### Phase 4: Archive Critical Models - [ ] Upload Piper voices to Archive.org - [ ] Upload XTTS-v2 to Archive.org - [ ] Document archive locations - [ ] Test restoration from archive ### Phase 5: Automation - [ ] Create sync script (`scripts/sync_models.sh`) - [ ] Set up cron job for weekly sync - [ ] Monitor mirror disk usage - [ ] Alert on upstream changes ### Phase 6: Future Engines - [ ] Add Silero to mirror - [ ] Add StyleTTS2 to mirror - [ ] Add Fish Speech to mirror --- ## Monitoring and Maintenance ### Health Checks ```bash #!/bin/bash # scripts/check_mirrors.sh # Verify all mirrors are accessible MODELS=( "piper/v1.0.0/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx" "xtts/v2.0.3/model.pth" ) for model in "${MODELS[@]}"; do echo "Checking $model..." # Check HuggingFace curl -I "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/$model" | head -n 1 # Check our mirror curl -I "http://ai.foxhop.net:9000/tts-models/$model" | head -n 1 echo "---" done ``` ### Storage Usage ```bash # Monitor MinIO usage mc du unclose/tts-models # Expected: # piper/: 2GB # xtts/: 1.8GB # silero/: 500MB # Total: ~5GB ``` --- ## Cost Estimation ### Storage (100GB allocated) - MinIO on existing server: **$0** (using spare disk) - Bandwidth: **$0** (self-hosted, unlimited) ### Archive.org - Storage: **$0** (free) - Bandwidth: **$0** (free) ### IPFS Pinning (Optional) - Pinata: **$20/month** for 100GB - Or self-host: **$0** **Total Cost: $0-20/month** --- ## Recovery Scenarios ### Scenario 1: HuggingFace is down 1. Downloader tries HF, gets timeout 2. Falls back to ai.foxhop.net mirror ✅ 3. Download succeeds in 30 seconds ### Scenario 2: Our mirror is down 1. Downloader tries ai.foxhop.net, fails 2. Falls back to Archive.org ✅ 3. Download succeeds in 2 minutes (slower) ### Scenario 3: Total internet failure 1. Models already cached in `/app/voices/` 2. Service continues with cached models ✅ 3. No downloads needed for operation ### Scenario 4: Apocalypse (all servers gone) 1. Restore from Archive.org archive 2. Restore from IPFS if configured 3. Restore from torrents if distributed 4. Rebuild from source if absolutely necessary --- **Last Updated:** 2025-11-09 **Raccoon Status:** 🦝 Building resilient caches like storing nuts for winter