uncloseai-speech/docs/AUDIT.md

UncloseAI Speech Repository Audit

Date: 2025-11-09 Mission: Raccoon TTS - Build a unified, resilient TTS system from abandoned open source projects

Current Repository Structure

Core Service Files (KEEP - Essential)

• speech.py (18K) - Main FastAPI server, handles OpenAI-compatible TTS API
• openedai.py (5.8K) - OpenAI API models and data structures
• audio_reader.py (3.7K) - Audio processing utilities
• voice_to_speaker.default.yaml - Default voice configuration mapping
• pre_process_map.default.yaml - Text preprocessing rules

Docker & Deployment (KEEP - Essential)

• Dockerfile - Main docker build (Piper + XTTS with NVIDIA GPU)
• Dockerfile.min - Minimal build (Piper only, no GPU)
• docker-compose.yml - Standard deployment
• docker-compose.min.yml - Minimal deployment
• docker-compose.rocm.yml - AMD ROCm GPU support
• Makefile - Raccoon deployment system (recently added)
• vars.sh.example - Deployment configuration template

Requirements Files (KEEP - Essential)

• requirements.txt - Main Python dependencies
• requirements-min.txt - Minimal dependencies (Piper only)
• requirements-rocm.txt - AMD ROCm dependencies

Startup Scripts (KEEP - Essential)

• startup.sh - Main startup with voice downloads
• startup.min.sh - Minimal startup
• download_voices_tts-1.sh - Download Piper voices
• download_voices_tts-1-hd.sh - Download XTTS HD voices

CLI Utilities (KEEP - Useful)

• say.py (3.5K) - Command-line TTS client using OpenAI API
• add_voice.py (2.8K) - Tool to add custom voices to config
• test_voices.sh (2.1K) - Test script for all voice models

Windows Batch Files (EVALUATE - Questionable)

• startup.bat (185B) - Windows startup
• download_samples.bat (230B) - Download sample WAVs
• download_voices_tts-1.bat (323B) - Download voices on Windows
• download_voices_tts-1-hd.bat (199B) - Download HD voices on Windows

Assessment: These are only useful if we support Windows deployment. Most raccoon missions run on Linux servers. Could move to docs/windows/ or remove entirely.

Sample Scripts (EVALUATE - Non-Essential)

• download_samples.sh (205B) - Downloads OpenAI sample WAV files
• boot_docker_compose_ubuntu.sh (623B) - One-time NVIDIA Docker setup for Ubuntu

Assessment: download_samples.sh is only for comparing against OpenAI's original voices - move to docs/tools/. The boot script is one-time setup - move to docs/setup/.

Documentation (KEEP - Update)

• README.md (16.6K) - Main documentation
• LICENSE - MIT license
• sample.env - Environment variable examples

CI/CD (KEEP)

• .github/workflows/build-docker.yml - GitHub Actions for building Docker images

Configuration (KEEP)

• .gitignore - Git ignore patterns
• config/config_files_will_go_here.txt - Placeholder

Files to Remove or Relocate

REMOVE (Windows-specific, low value):

• startup.bat
• download_samples.bat
• download_voices_tts-1.bat
• download_voices_tts-1-hd.bat

MOVE to docs/tools/:

• download_samples.sh (useful for voice comparison)
• test_voices.sh (useful for testing)

MOVE to docs/setup/:

• boot_docker_compose_ubuntu.sh (one-time setup instructions)

Abandoned TTS Models to Integrate

Based on the raccoon mission to rescue abandoned TTS projects:

1. Piper TTS (Currently Integrated)

• Status: ✅ Working with absolute paths
• Original: rhasspy/piper (abandoned)
• Fork: OHF-Voice/piper1-gpl v1.3.0 (no PyPI package)
• Current: Using PyPI piper-tts v1.2.0
• Models: ~100+ voices from HuggingFace rhasspy/piper-voices
• Action: Consider creating our own PyPI package from OHF-Voice fork

2. Coqui XTTS (Currently Integrated)

• Status: ✅ Integrated as tts-1-hd
• Original: coqui-ai/TTS (company shut down, repo archived)
• Package: coqui-tts (still on PyPI)
• Models: XTTS v2 from HuggingFace
• Action: Mirror XTTS v2 models, explore community forks

3. Mozilla TTS (Predecessor to Coqui)

• Status: ❌ Not integrated, abandoned
• Original: mozilla/TTS (archived, became Coqui)
• Action: Evaluate if any unique models worth preserving

4. Silero TTS

• Status: ❌ Not integrated
• Repo: snakers4/silero-models (still active!)
• Models: Fast, multilingual, small footprint
• Languages: Russian, English, German, Spanish, French
• Action: HIGH PRIORITY - Add Silero support

5. Bark (Suno AI)

• Status: ❌ Not integrated
• Repo: suno-ai/bark (archived, company pivoted)
• Features: Music, sound effects, non-speech audio
• Action: Evaluate for unique use cases

6. Tortoise TTS

• Status: ❌ Not integrated
• Repo: neonbjb/tortoise-tts (low activity)
• Features: High quality, very slow generation
• Action: Low priority - too slow for production

7. StyleTTS2

• Status: ❌ Not integrated
• Repo: yl4579/StyleTTS2 (active research project)
• Features: State-of-art quality, prosody
• Action: HIGH PRIORITY - Best quality available

8. MetaVoice

• Status: ❌ Not integrated
• Repo: metavoiceio/metavoice-src (partially abandoned)
• Features: Long-form TTS, emotional control
• Action: Evaluate community forks

9. Fish Speech

• Status: ❌ Not integrated
• Repo: fishaudio/fish-speech (active)
• Features: Fast, multilingual
• Action: MEDIUM PRIORITY - Active project

10. Kokoro TTS

• Status: ❌ Not integrated
• Repo: hexgrad/kokoro (new, active)
• Features: Fast, small, quality
• Action: MEDIUM PRIORITY - Promising new project

Binary Mirror Strategy

Problem

• HuggingFace models can disappear
• Upstream repositories get archived/deleted
• Need resilience for production systems

Solution: Multi-Tier Mirror System

Tier 1: Primary Sources (Current)

• HuggingFace Hub
• PyPI packages
• GitHub releases

Tier 2: UncloseAI Mirror (Proposed)

• Self-hosted model repository
• Options:
  • MinIO/S3-compatible storage
  • LFS-backed Git repository
  • Dedicated file server with rsync

Tier 3: Fallback (Proposed)

• Archive.org uploads for critical models
• IPFS pinning for decentralization
• Torrent magnets for popular models

Priority Models to Mirror

1. Piper voices - ~2GB from rhasspy/piper-voices
2. XTTS v2 - ~1.8GB model + config
3. Silero models - ~100MB per language
4. StyleTTS2 weights - TBD size

Implementation Plan

• Set up MinIO on ai.foxhop.net
• Create download script with fallback URLs
• Document mirror usage in docs
• Automated sync from upstream sources

Recommended Refactoring

1. Directory Structure

uncloseai-speech/
├── docs/
│   ├── AUDIT.md (this file)
│   ├── MODELS.md (TTS engine documentation)
│   ├── MIRRORS.md (Binary mirror setup)
│   ├── API.md (API documentation)
│   └── setup/
│       └── nvidia-docker-ubuntu.md
├── scripts/
│   ├── add_voice.py (add custom voices to config)
│   ├── say.py (CLI TTS client)
│   ├── test_voices.sh (test all voices)
│   ├── download_samples.sh (download OpenAI sample WAVs)
│   ├── download_voices_tts-1.sh (Piper voice downloader)
│   └── download_voices_tts-1-hd.sh (XTTS voice downloader)
├── src/  (future refactor)
│   ├── engines/
│   │   ├── piper.py
│   │   ├── xtts.py
│   │   ├── silero.py (NEW)
│   │   └── base.py (NEW)
│   ├── api/
│   │   ├── openedai.py
│   │   └── server.py
│   └── utils/
│       └── audio_reader.py
├── config/
│   ├── voice_to_speaker.default.yaml
│   └── pre_process_map.default.yaml
├── Dockerfile
├── Dockerfile.min
├── docker-compose.yml
├── docker-compose.min.yml
├── docker-compose.rocm.yml
├── startup.sh
├── startup.min.sh
├── Makefile
├── speech.py (current monolith - will refactor to src/)
├── openedai.py (will move to src/api/)
├── audio_reader.py (will move to src/utils/)
├── requirements.txt
├── requirements-min.txt
├── requirements-rocm.txt
└── README.md

2. Engine Abstraction

Create a base engine class that all TTS engines implement:

# src/engines/base.py
from abc import ABC, abstractmethod
from typing import BinaryIO, Optional

class TTSEngine(ABC):
    @abstractmethod
    def synthesize(self, text: str, voice: str,
                   speed: float = 1.0,
                   language: Optional[str] = None) -> BinaryIO:
        """Synthesize speech from text"""
        pass

    @abstractmethod
    def list_voices(self) -> list[str]:
        """Return available voices for this engine"""
        pass

    @abstractmethod
    def download_models(self, mirror_urls: list[str] = None):
        """Download required models with fallback mirrors"""
        pass

3. Configuration Schema

Extend voice config to support multiple engines:

# Future voice_to_speaker.yaml format
tts-1:
  alloy:
    engine: piper  # or xtts, silero, styletts2
    model: /app/voices/en/en_US/libritts_r/medium/en_US-libritts_r-medium.onnx
    speaker: 79
  nova:
    engine: silero
    model: v3_1_ru
    speaker: aidar

tts-1-hd:
  alloy:
    engine: xtts
    model: xtts
    speaker: /app/voices/alloy.wav
  custom-voice:
    engine: styletts2
    model: /app/models/styletts2-base
    speaker: /app/voices/custom.wav
    language: en

Next Steps

Immediate (This Session)

1. ✅ Create docs directory
2. ✅ Write audit document
3. Remove Windows .bat files
4. Relocate tools to docs/tools/
5. Create MODELS.md documenting each TTS engine

Short Term (Next Few Sessions)

1. Implement engine abstraction layer
2. Add Silero TTS support
3. Set up model mirror on ai.foxhop.net
4. Test all existing functionality still works

Medium Term

1. Add StyleTTS2 support
2. Add Fish Speech support
3. Implement fallback mirror system
4. Create automated model sync

Long Term

1. Add Bark for sound effects
2. Implement voice cloning
3. Multi-language auto-detection
4. Performance benchmarking dashboard

Raccoon Mission Status

🦝 Mission: Rescue abandoned TTS models and build a resilient unified system

Current Status:

• ✅ Piper TTS working with absolute paths
• ✅ XTTS integrated
• ✅ Deployment system (Makefile + vars.sh)
• ✅ Renamed to UncloseAI Speech
• 📝 Repository audit complete
• 🔄 Documentation in progress

Key Risks:

• Upstream model URLs can break (need mirrors)
• PyPI packages could disappear (need vendoring strategy)
• HuggingFace rate limits (need caching)

Mitigation:

• Build multi-tier mirror system
• Document all model sources
• Create offline installation option