engineering/uncloseai-speech
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Standardize project naming to uncloseai-speech across all files

Browse source 
- Add CHANGELOG.md with full version history (moved from README)
- Update all documentation to use lowercase 'uncloseai-speech' project name
- Update organization references to lowercase 'uncloseai' (not 'UncloseAI')
- Add Brand Identity section to docs/CLAUDE.md with naming guidelines
- Update speech.py argparse description to match branding
- Update README.md headers and sections with consistent naming
- Update all model documentation with consistent branding

Files updated:
- CHANGELOG.md (new file)
- README.md (changelog reference, server options, multilingual section)
- speech.py (--workers argument, branding in argparse)
- Makefile (header comment)
- docs/CLAUDE.md (Brand Identity section)
- docs/MODELS.md
- docs/MIRRORS.md
- docs/AUDIT.md
- docs/models/coqui-tts.md
- docs/research/tts-models-overview.md

Branding standard:
- Project: uncloseai-speech (lowercase, hyphenated)
- Organization: uncloseai (lowercase, one word)

🦝 Generated with Claude Code
This commit is contained in:
Russell Ballestrini 2025-11-10 05:23:34 -05:00
parent da8e960b2d
commit 7559e56d0c
10 changed files with 313 additions and 165 deletions
Download patch file Download diff file Expand all files Collapse all files

127
CHANGELOG.md Normal file
View file

@ -0,0 +1,127 @@
# uncloseai-speech - Changelog
## Recent Changes
**Raccoon Mission Updates, 2025-11-09**
* 🦝 **Production-ready multiprocess architecture** - 4 uvicorn workers for true concurrency, bypassing Python's GIL
* 🦝 **Voice auto-detection** - `model` parameter now optional, automatically selects correct engine from voice name
* 🦝 **Voice cache initialization fix** - All worker processes now properly initialize voice-to-model lookup cache
* 🦝 **Args initialization fix** - Worker processes now have access to server configuration via DefaultArgs class
* 🦝 **235/245 voices working** (95.9% hydration success rate):
- Piper: 55/55 voices (100%)
- XTTS: 6/8 voices (75%)
- Silero: 142/148 voices (95.9%)
- Kokoro: 32/34 voices (94.1%)
* 🦝 **Extended `/v1/voices` endpoint** - Returns all available voices with engine metadata
* 🦝 **Makefile targets** - `make hydrate` (test all voices), `make load-test` (concurrent stress test)
* 🦝 **Comprehensive docs** - See `docs/CLAUDE.md`, `docs/MODELS.md`, `docs/MIRRORS.md`, `docs/AUDIT.md`
Version 0.18.2, 2024-08-16
* Fix docker building for amd64, refactor github actions again, free up more disk space
Version 0.18.1, 2024-08-15
* refactor github actions
Version 0.18.0, 2024-08-15
* Allow folders of wav samples in xtts. Samples will be combined, allowing for mixed voices and collections of small samples. Still limited to 30 seconds total. Thanks @nathanhere.
* Fix missing yaml requirement in -min image
* fix fr_FR-tom-medium and other 44khz piper voices (detect non-default sample rates)
* minor updates
Version 0.17.2, 2024-07-01
* fix -min image (re: langdetect)
Version 0.17.1, 2024-07-01
* fix ROCm (add langdetect to requirements-rocm.txt)
* Fix zh-cn for xtts
Version 0.17.0, 2024-07-01
* Automatic language detection, thanks [@RodolfoCastanheira](https://github.com/RodolfoCastanheira)
Version 0.16.0, 2024-06-29
* Multi-client safe version. Audio generation is synchronized in a single process. The estimated 'realtime' factor of XTTS on a GPU is roughly 1/3, this means that multiple streams simultaneously, or `speed` over 2, may experience audio underrun (delays or pauses in playback). This makes multiple clients possible and safe, but in practice 2 or 3 simultaneous streams is the maximum without audio underrun.
Version 0.15.1, 2024-06-27
* Remove deepspeed from requirements.txt, it's too complex for typical users. A more detailed deepspeed install document will be required.
Version 0.15.0, 2024-06-26
* Switch to [coqui-tts](https://github.com/idiap/coqui-ai-TTS) (updated fork), updated simpler dependencies, torch 2.3, etc.
* Resolve cuda threading issues
Version 0.14.1, 2024-06-26
* Make deepspeed possible (`--use-deepspeed`), but not enabled in pre-built docker images (too large). Requires the cuda-toolkit installed, see the Dockerfile comment for details
Version 0.14.0, 2024-06-26
* Added `response_format`: `wav` and `pcm` support
* Output streaming (while generating) for `tts-1` and `tts-1-hd`
* Enhanced [generation parameters](#generation-parameters) for xtts models (temperature, top_p, etc.)
* Idle unload timer (optional) - doesn't work perfectly yet
* Improved error handling
Version 0.13.0, 2024-06-25
* Added [Custom fine-tuned XTTS model support](#custom-fine-tuned-model-support)
* Initial prebuilt arm64 image support (Apple M-series, Raspberry Pi - MPS is not supported in XTTS/torch), thanks [@JakeStevenson](https://github.com/JakeStevenson), [@hchasens](https://github.com/hchasens)
* Initial attempt at AMD GPU (ROCm 5.7) support
* Parler-tts support removed
* Move the *.default.yaml to the root folder
* Run the docker as a service by default (`restart: unless-stopped`)
* Added `audio_reader.py` for streaming text input and reading long texts
Version 0.12.3, 2024-06-17
* Additional logging details for BadRequests (400)
Version 0.12.2, 2024-06-16
* Fix :min image requirements (numpy<2?)
Version 0.12.0, 2024-06-16
* Improved error handling and logging
* Restore the original alloy tts-1-hd voice by default, use alloy-alt for the old voice.
Version 0.11.0, 2024-05-29
* 🌐 [Multilingual](#multilingual) support (16 languages) with XTTS
* Remove high Unicode filtering from the default `config/pre_process_map.yaml`
* Update Docker build & app startup. thanks @justinh-rahb
* Fix: "Plan failed with a cudnnException"
* Remove piper cuda support
Version: 0.10.1, 2024-05-05
* Remove `runtime: nvidia` from docker-compose.yml, this assumes nvidia/cuda compatible runtime is available by default. thanks [@jmtatsch](https://github.com/jmtatsch)
Version: 0.10.0, 2024-04-27
* Pre-built & tested docker images, smaller docker images (8GB or 860MB)
* Better upgrades: reorganize config files under `config/`, voice models under `voices/`
* **Compatibility!** If you customized your `voice_to_speaker.yaml` or `pre_process_map.yaml` you need to move them to the `config/` folder.
* default listen host to 0.0.0.0
Version: 0.9.0, 2024-04-23
* Fix bug with yaml and loading UTF-8
* New sample text-to-speech application `say.py`
* Smaller docker base image
* Add beta [parler-tts](https://huggingface.co/parler-tts/parler_tts_mini_v0.1) support (you can describe very basic features of the speaker voice), See: (https://www.text-description-to-speech.com/) for some examples of how to describe voices. Voices can be defined in the `voice_to_speaker.default.yaml`. Two example [parler-tts](https://huggingface.co/parler-tts/parler_tts_mini_v0.1) voices are included in the `voice_to_speaker.default.yaml` file. `parler-tts` is experimental software and is kind of slow. The exact voice will be slightly different each generation but should be similar to the basic description.
...
Version: 0.7.3, 2024-03-20
* Allow different xtts versions per voice in `voice_to_speaker.yaml`, ex. xtts_v2.0.2
* Quality: Fix xtts sample rate (24000 vs. 22050 for piper) and pops

2
Makefile
View file

@ -1,4 +1,4 @@
# Raccoon Mission: UncloseAI Speech Development Makefile # Raccoon Mission: uncloseai-speech Development Makefile
# Deploy to remote server with ease # Deploy to remote server with ease
# Configuration is loaded from vars.sh (copy vars.sh.example to vars.sh) # Configuration is loaded from vars.sh (copy vars.sh.example to vars.sh)

163
README.md
View file

@ -1,4 +1,4 @@
# UncloseAI Speech # uncloseai-speech
🦝 **Raccoon Mission Fork:** Rescuing abandoned TTS models and building a unified, resilient text-to-speech system. 🦝 **Raccoon Mission Fork:** Rescuing abandoned TTS models and building a unified, resilient text-to-speech system.
@ -79,130 +79,7 @@ If you find a better voice match for `tts-1` or `tts-1-hd`, please let me know s
## Recent Changes ## Recent Changes
**Raccoon Mission Updates, 2025-11-09** See [CHANGELOG.md](CHANGELOG.md) for full version history.
* 🦝 **Production-ready multiprocess architecture** - 4 uvicorn workers for true concurrency, bypassing Python's GIL
* 🦝 **Voice auto-detection** - `model` parameter now optional, automatically selects correct engine from voice name
* 🦝 **Voice cache initialization fix** - All worker processes now properly initialize voice-to-model lookup cache
* 🦝 **Args initialization fix** - Worker processes now have access to server configuration via DefaultArgs class
* 🦝 **235/245 voices working** (95.9% hydration success rate):
- Piper: 55/55 voices (100%)
- XTTS: 6/8 voices (75%)
- Silero: 142/148 voices (95.9%)
- Kokoro: 32/34 voices (94.1%)
* 🦝 **Extended `/v1/voices` endpoint** - Returns all available voices with engine metadata
* 🦝 **Makefile targets** - `make hydrate` (test all voices), `make load-test` (concurrent stress test)
* 🦝 **Comprehensive docs** - See `docs/CLAUDE.md`, `docs/MODELS.md`, `docs/MIRRORS.md`, `docs/AUDIT.md`
Version 0.18.2, 2024-08-16
* Fix docker building for amd64, refactor github actions again, free up more disk space
Version 0.18.1, 2024-08-15
* refactor github actions
Version 0.18.0, 2024-08-15
* Allow folders of wav samples in xtts. Samples will be combined, allowing for mixed voices and collections of small samples. Still limited to 30 seconds total. Thanks @nathanhere.
* Fix missing yaml requirement in -min image
* fix fr_FR-tom-medium and other 44khz piper voices (detect non-default sample rates)
* minor updates
Version 0.17.2, 2024-07-01
* fix -min image (re: langdetect)
Version 0.17.1, 2024-07-01
* fix ROCm (add langdetect to requirements-rocm.txt)
* Fix zh-cn for xtts
Version 0.17.0, 2024-07-01
* Automatic language detection, thanks [@RodolfoCastanheira](https://github.com/RodolfoCastanheira)
Version 0.16.0, 2024-06-29
* Multi-client safe version. Audio generation is synchronized in a single process. The estimated 'realtime' factor of XTTS on a GPU is roughly 1/3, this means that multiple streams simultaneously, or `speed` over 2, may experience audio underrun (delays or pauses in playback). This makes multiple clients possible and safe, but in practice 2 or 3 simultaneous streams is the maximum without audio underrun.
Version 0.15.1, 2024-06-27
* Remove deepspeed from requirements.txt, it's too complex for typical users. A more detailed deepspeed install document will be required.
Version 0.15.0, 2024-06-26
* Switch to [coqui-tts](https://github.com/idiap/coqui-ai-TTS) (updated fork), updated simpler dependencies, torch 2.3, etc.
* Resolve cuda threading issues
Version 0.14.1, 2024-06-26
* Make deepspeed possible (`--use-deepspeed`), but not enabled in pre-built docker images (too large). Requires the cuda-toolkit installed, see the Dockerfile comment for details
Version 0.14.0, 2024-06-26
* Added `response_format`: `wav` and `pcm` support
* Output streaming (while generating) for `tts-1` and `tts-1-hd`
* Enhanced [generation parameters](#generation-parameters) for xtts models (temperature, top_p, etc.)
* Idle unload timer (optional) - doesn't work perfectly yet
* Improved error handling
Version 0.13.0, 2024-06-25
* Added [Custom fine-tuned XTTS model support](#custom-fine-tuned-model-support)
* Initial prebuilt arm64 image support (Apple M-series, Raspberry Pi - MPS is not supported in XTTS/torch), thanks [@JakeStevenson](https://github.com/JakeStevenson), [@hchasens](https://github.com/hchasens)
* Initial attempt at AMD GPU (ROCm 5.7) support
* Parler-tts support removed
* Move the *.default.yaml to the root folder
* Run the docker as a service by default (`restart: unless-stopped`)
* Added `audio_reader.py` for streaming text input and reading long texts
Version 0.12.3, 2024-06-17
* Additional logging details for BadRequests (400)
Version 0.12.2, 2024-06-16
* Fix :min image requirements (numpy<2?)
Version 0.12.0, 2024-06-16
* Improved error handling and logging
* Restore the original alloy tts-1-hd voice by default, use alloy-alt for the old voice.
Version 0.11.0, 2024-05-29
* 🌐 [Multilingual](#multilingual) support (16 languages) with XTTS
* Remove high Unicode filtering from the default `config/pre_process_map.yaml`
* Update Docker build & app startup. thanks @justinh-rahb
* Fix: "Plan failed with a cudnnException"
* Remove piper cuda support
Version: 0.10.1, 2024-05-05
* Remove `runtime: nvidia` from docker-compose.yml, this assumes nvidia/cuda compatible runtime is available by default. thanks [@jmtatsch](https://github.com/jmtatsch)
Version: 0.10.0, 2024-04-27
* Pre-built & tested docker images, smaller docker images (8GB or 860MB)
* Better upgrades: reorganize config files under `config/`, voice models under `voices/`
* **Compatibility!** If you customized your `voice_to_speaker.yaml` or `pre_process_map.yaml` you need to move them to the `config/` folder.
* default listen host to 0.0.0.0
Version: 0.9.0, 2024-04-23
* Fix bug with yaml and loading UTF-8
* New sample text-to-speech application `say.py`
* Smaller docker base image
* Add beta [parler-tts](https://huggingface.co/parler-tts/parler_tts_mini_v0.1) support (you can describe very basic features of the speaker voice), See: (https://www.text-description-to-speech.com/) for some examples of how to describe voices. Voices can be defined in the `voice_to_speaker.default.yaml`. Two example [parler-tts](https://huggingface.co/parler-tts/parler_tts_mini_v0.1) voices are included in the `voice_to_speaker.default.yaml` file. `parler-tts` is experimental software and is kind of slow. The exact voice will be slightly different each generation but should be similar to the basic description.
...
Version: 0.7.3, 2024-03-20
* Allow different xtts versions per voice in `voice_to_speaker.yaml`, ex. xtts_v2.0.2
* Quality: Fix xtts sample rate (24000 vs. 22050 for piper) and pops
## Installation instructions ## Installation instructions
@ -298,10 +175,11 @@ bash startup.sh
## Server Options ## Server Options
```shell ```shell
usage: speech.py [-h] [--xtts_device XTTS_DEVICE] [--preload PRELOAD] [--unload-timer UNLOAD_TIMER] [--use-deepspeed] [--no-cache-speaker] [-P PORT] [-H HOST] usage: speech.py [-h] [--xtts_device XTTS_DEVICE] [--preload PRELOAD] [--unload-timer UNLOAD_TIMER]
[--use-deepspeed] [--no-cache-speaker] [-W WORKERS] [-P PORT] [-H HOST]
[-L {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-L {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
UncloseAI Speech API Server uncloseai-speech API Server
options: options:
-h, --help show this help message and exit -h, --help show this help message and exit
@ -312,6 +190,8 @@ options:
Idle unload timer for the XTTS model in seconds, Ex. 900 for 15 minutes (default: None) Idle unload timer for the XTTS model in seconds, Ex. 900 for 15 minutes (default: None)
--use-deepspeed Use deepspeed with xtts (this option is unsupported) (default: False) --use-deepspeed Use deepspeed with xtts (this option is unsupported) (default: False)
--no-cache-speaker Don't use the speaker wav embeddings cache (default: False) --no-cache-speaker Don't use the speaker wav embeddings cache (default: False)
-W WORKERS, --workers WORKERS
Number of uvicorn worker processes for concurrent request handling (default: 4)
-P PORT, --port PORT Server tcp port (default: 8000) -P PORT, --port PORT Server tcp port (default: 8000)
-H HOST, --host HOST Host to listen on, Ex. 0.0.0.0 (default: 0.0.0.0) -H HOST, --host HOST Host to listen on, Ex. 0.0.0.0 (default: 0.0.0.0)
-L {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} -L {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
@ -452,11 +332,13 @@ Where the `voices/mixed/` folder contains multiple wav files. The total audio le
## Multilingual ## Multilingual
Multilingual cloning support was added in version 0.11.0 and is available only with the XTTS v2 model. To use multilingual voices with piper simply download a language specific voice. uncloseai-speech supports multiple languages across different TTS engines:
Coqui XTTSv2 has support for multiple languages: English (`en`), Spanish (`es`), French (`fr`), German (`de`), Italian (`it`), Portuguese (`pt`), Polish (`pl`), Turkish (`tr`), Russian (`ru`), Dutch (`nl`), Czech (`cs`), Arabic (`ar`), Chinese (`zh-cn`), Hungarian (`hu`), Korean (`ko`), Japanese (`ja`), and Hindi (`hi`). When not set, an attempt will be made to automatically detect the language, falling back to English (`en`). ### XTTS (tts-1-hd) - 17 Languages
Unfortunately the OpenAI API does not support language, but you can create your own custom speaker voice and set the language for that. Multilingual cloning support was added in version 0.11.0. Coqui XTTSv2 has support for multiple languages: English (`en`), Spanish (`es`), French (`fr`), German (`de`), Italian (`it`), Portuguese (`pt`), Polish (`pl`), Turkish (`tr`), Russian (`ru`), Dutch (`nl`), Czech (`cs`), Arabic (`ar`), Chinese (`zh-cn`), Hungarian (`hu`), Korean (`ko`), Japanese (`ja`), and Hindi (`hi`). When not set, an attempt will be made to automatically detect the language, falling back to English (`en`).
Unfortunately the OpenAI API does not support language parameters, but you can create your own custom speaker voice and set the language for that.
1) Create the WAV file for your speaker, as in [Custom Voices Howto](#custom-voices-howto) 1) Create the WAV file for your speaker, as in [Custom Voices Howto](#custom-voices-howto)
2) Add the voice to `config/voice_to_speaker.yaml` and include the correct Coqui `language` code for the speaker. For example: 2) Add the voice to `config/voice_to_speaker.yaml` and include the correct Coqui `language` code for the speaker. For example:
@ -476,10 +358,29 @@ Remove:
- '' - ''
``` ```
These lines were added to the `config/pre_process_map.yaml` config file by default before version 0.11.0: These lines were added to the `config/pre_process_map.yaml` config file by default before version 0.11.0.
4) Your new multi-lingual speaker voice is ready to use! 4) Your new multi-lingual speaker voice is ready to use!
### Silero (tts-1-silero) - 5 Languages
Silero TTS provides native multilingual support with 148 voices across 5 languages:
- **English (en)** - 117 voices
- **Russian (ru)** - 10 voices
- **German (de)** - 5 voices
- **Spanish (es)** - 3 voices
- **French (fr)** - 5 voices
Each language has multiple speaker variations. Voices are automatically configured in `config/voice_to_speaker.yaml`. Simply select the appropriate voice (e.g., `en_0`, `ru_0`, `de_0`) and the correct language model will be loaded automatically.
### Kokoro (tts-1-kokoro) - English Only
Kokoro TTS currently supports only American and British English with 34 high-quality voices. The model uses a lightweight decoder-only architecture (82M parameters) optimized for English speech synthesis.
### Piper (tts-1) - 50+ Languages
For Piper TTS, simply download language-specific voice models from [piper samples](https://rhasspy.github.io/piper-samples/). Piper supports 50+ languages with hundreds of voice options. Add the voice to `config/voice_to_speaker.yaml` as shown in [Custom Voices Howto](#custom-voices-howto).
## Custom Fine-Tuned Model Support ## Custom Fine-Tuned Model Support

4
docs/AUDIT.md
View file

@ -1,4 +1,4 @@
# UncloseAI Speech Repository Audit # uncloseai-speech Repository Audit
**Date:** 2025-11-09 **Date:** 2025-11-09
**Mission:** Raccoon TTS - Build a unified, resilient TTS system from abandoned open source projects **Mission:** Raccoon TTS - Build a unified, resilient TTS system from abandoned open source projects
@ -328,7 +328,7 @@ tts-1-hd:
- ✅ Piper TTS working with absolute paths - ✅ Piper TTS working with absolute paths
- ✅ XTTS integrated - ✅ XTTS integrated
- ✅ Deployment system (Makefile + vars.sh) - ✅ Deployment system (Makefile + vars.sh)
- ✅ Renamed to UncloseAI Speech - ✅ Renamed to uncloseai-speech
- 📝 Repository audit complete - 📝 Repository audit complete
- 🔄 Documentation in progress - 🔄 Documentation in progress

45
docs/CLAUDE.md
View file

@ -1,8 +1,51 @@
# Instructions for Claude Code # Instructions for Claude Code
**Project:** UncloseAI Speech - Raccoon Mission TTS System **Project:** uncloseai-speech - Raccoon Mission TTS System
**License:** AGPL v3 (must provide source code to network service users) **License:** AGPL v3 (must provide source code to network service users)
## Brand Identity
**CRITICAL: Always use consistent naming across all files.**
### Project Name
- ✅ **Correct:** `uncloseai-speech` (lowercase, hyphenated)
- ❌ **Wrong:** "UncloseAI Speech", "Uncloseai Speech", "UncloseAI-Speech"
### Organization Name
- ✅ **Correct:** `uncloseai` (lowercase, one word)
- ❌ **Wrong:** "UncloseAI", "Unclose AI", "UnClose AI"
### Usage Guidelines
- **In code:** Use `uncloseai-speech` for project references
- **In documentation:** Use `uncloseai-speech` for project name
- **In comments:** Use `uncloseai-speech` consistently
- **Repository URLs:** `uncloseai-speech` (lowercase, hyphenated)
- **Docker images:** `uncloseai-speech` (lowercase, hyphenated)
- **API responses:** Use `"owned_by": "uncloseai"` (lowercase, one word)
### Examples
```python
# Correct
description='uncloseai-speech API Server'
owned_by = "uncloseai"
# Wrong
description='UncloseAI Speech API Server'
owned_by = "UncloseAI"
```
```markdown
# Correct
# uncloseai-speech
**Raccoon Mission:** Rescue abandoned TTS models and integrate them into uncloseai-speech
# Wrong
# UncloseAI Speech
**Raccoon Mission:** Rescue abandoned TTS models and integrate them into UncloseAI Speech
```
## Core Principles ## Core Principles
### 1. Makefile-First Development ### 1. Makefile-First Development

4
docs/MIRRORS.md
View file

@ -1,6 +1,6 @@
# Binary Mirror Strategy # Binary Mirror Strategy
**Purpose:** Ensure UncloseAI Speech keeps working even if upstream model sources disappear **Purpose:** Ensure uncloseai-speech keeps working even if upstream model sources disappear
## The Problem ## The Problem
@ -179,7 +179,7 @@ ia upload uncloseai-piper-voices-v1.0.0 \
--metadata="title:Piper TTS Voices v1.0.0" \ --metadata="title:Piper TTS Voices v1.0.0" \
--metadata="description:Complete Piper TTS voice collection from rhasspy/piper-voices" \ --metadata="description:Complete Piper TTS voice collection from rhasspy/piper-voices" \
--metadata="subject:text-to-speech;tts;piper;neural-tts" \ --metadata="subject:text-to-speech;tts;piper;neural-tts" \
--metadata="creator:UncloseAI Speech Raccoon Mission" \ --metadata="creator:uncloseai Raccoon Mission" \
--metadata="date:2025-11-09" --metadata="date:2025-11-09"
``` ```

2
docs/MODELS.md
View file

@ -1,6 +1,6 @@
# TTS Models and Engines # TTS Models and Engines
**Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into UncloseAI Speech **Raccoon Mission:** Rescue abandoned open-source TTS models and integrate them into uncloseai-speech
## Documentation Index ## Documentation Index

10
docs/models/coqui-tts.md
View file

@ -170,7 +170,7 @@ pip install TTS==14.5.0
## Integration Status ## Integration Status
### Current Implementation ### Current Implementation
- **UncloseAI Model Name:** `tts-1-hd` - **uncloseai-speech Model Name:** `tts-1-hd`
- **Status:** ✅ Fully Integrated - **Status:** ✅ Fully Integrated
- **Integration Date:** Active (as of 2025-11-09) - **Integration Date:** Active (as of 2025-11-09)
- **Container Path:** Model auto-downloaded to `/root/.local/share/tts/` on first use - **Container Path:** Model auto-downloaded to `/root/.local/share/tts/` on first use
@ -316,7 +316,7 @@ docker run --gpus all \
### OpenAI-Compatible API Integration ### OpenAI-Compatible API Integration
```python ```python
# Direct integration with UncloseAI Speech # Direct integration with uncloseai-speech
import requests import requests
import json import json
@ -490,7 +490,7 @@ Priority 4: Fallback inference
5. **Document alternatives** - Create comparison guides with other TTS systems 5. **Document alternatives** - Create comparison guides with other TTS systems
6. **Support community implementations** - Fund AllTalk TTS development 6. **Support community implementations** - Fund AllTalk TTS development
### Integration with UncloseAI Speech ### Integration with uncloseai-speech
**Current Role:** **Current Role:**
- Primary high-quality TTS engine - Primary high-quality TTS engine
@ -636,7 +636,7 @@ print(get_supported_languages())
- **Last Updated:** 2025-11-09 - **Last Updated:** 2025-11-09
- **Status:** Complete and current - **Status:** Complete and current
- **Maintained By:** Raccoon Mission (UncloseAI Speech) - **Maintained By:** Raccoon Mission (uncloseai)
- **Related Files:** `/home/user/uncloseai-speech/docs/MODELS.md`, `/home/user/uncloseai-speech/docs/AUDIT.md` - **Related Files:** `/home/user/uncloseai-speech/docs/MODELS.md`, `/home/user/uncloseai-speech/docs/AUDIT.md`
- **Integration Level:** Production-ready - **Integration Level:** Production-ready
- **Community Status:** ✅ Actively maintained by fork community - **Community Status:** ✅ Actively maintained by fork community
@ -645,4 +645,4 @@ print(get_supported_languages())
**Raccoon Mission:** 🦝 Preserving abandoned TTS systems for a free and open future. **Raccoon Mission:** 🦝 Preserving abandoned TTS systems for a free and open future.
*This document is part of the UncloseAI Speech project - rescuing open-source TTS models from abandonment and unifying them under one API.* *This document is part of the uncloseai-speech project - rescuing open-source TTS models from abandonment and unifying them under one API.*

4
docs/research/tts-models-overview.md
View file

@ -5,7 +5,7 @@
## Executive Summary ## Executive Summary
This document provides a comprehensive overview of open-source Text-to-Speech (TTS) models researched for integration into UncloseAI Speech. Our "Raccoon Mission" aims to rescue abandoned and at-risk TTS projects, ensuring their long-term preservation and availability. This document provides a comprehensive overview of open-source Text-to-Speech (TTS) models researched for integration into uncloseai-speech. Our "Raccoon Mission" aims to rescue abandoned and at-risk TTS projects, ensuring their long-term preservation and availability.
## Model Inventory ## Model Inventory
@ -366,5 +366,5 @@ The TTS landscape is rapidly evolving with several high-quality open-source opti
**Raccoon Mission Status:** 🦝 2/10 models rescued and integrated **Raccoon Mission Status:** 🦝 2/10 models rescued and integrated
**Next Action:** Set up mirror infrastructure and integrate Chatterbox **Next Action:** Set up mirror infrastructure and integrate Chatterbox
**Documentation Maintained By:** UncloseAI Speech Team **Documentation Maintained By:** uncloseai
**Last Updated:** 2025-11-09 **Last Updated:** 2025-11-09

117
speech.py
View file

@ -110,6 +110,15 @@ async def lifespan(app):
except: except:
pass pass
# We return 'mps' but currently XTTS will not work with mps devices as the cuda support is incomplete
def auto_torch_device():
try:
import torch
return 'cuda' if torch.cuda.is_available() else 'mps' if ( torch.backends.mps.is_available() and torch.backends.mps.is_built() ) else 'cpu'
except:
return 'none'
app = OpenAIStub(lifespan=lifespan) app = OpenAIStub(lifespan=lifespan)
xtts = None xtts = None
silero_model = None silero_model = None
@ -134,8 +143,11 @@ args = DefaultArgs()
# Main process will override this in __main__ with argparse # Main process will override this in __main__ with argparse
if args.xtts_device is None: if args.xtts_device is None:
try: try:
args.xtts_device = auto_torch_device() detected_device = auto_torch_device()
except: args.xtts_device = detected_device
logger.debug(f"Worker process initialized with device: {detected_device}")
except Exception as e:
logger.warning(f"Failed to detect torch device: {e}, falling back to CPU")
args.xtts_device = 'cpu' args.xtts_device = 'cpu'
# Voice-to-model lookup cache (loaded at startup) # Voice-to-model lookup cache (loaded at startup)
@ -316,13 +328,18 @@ class kokoro_wrapper():
# Collect all audio chunks # Collect all audio chunks
audio_chunks = [] audio_chunks = []
chunk_count = 0
for _, _, audio in generator: for _, _, audio in generator:
if audio is not None and len(audio) > 0: if audio is not None and len(audio) > 0:
audio_chunks.append(audio) audio_chunks.append(audio)
chunk_count += 1
if chunk_count % 10 == 0:
logger.debug(f"Kokoro generated {chunk_count} chunks so far...")
# Concatenate all chunks # Concatenate all chunks
if len(audio_chunks) > 0: if len(audio_chunks) > 0:
full_audio = np.concatenate(audio_chunks) full_audio = np.concatenate(audio_chunks)
logger.info(f"Kokoro generation complete: {chunk_count} chunks, {len(full_audio)} samples")
# Convert float32 numpy array to bytes # Convert float32 numpy array to bytes
return full_audio.astype(np.float32).tobytes() return full_audio.astype(np.float32).tobytes()
else: else:
@ -353,11 +370,45 @@ def preprocess(raw_input):
pre_process_map = yaml.safe_load(file) pre_process_map = yaml.safe_load(file)
for a, b in pre_process_map: for a, b in pre_process_map:
raw_input = re.sub(a, b, raw_input) raw_input = re.sub(a, b, raw_input)
raw_input = raw_input.strip() raw_input = raw_input.strip()
#logger.debug(f"preprocess: after: {[raw_input]}") #logger.debug(f"preprocess: after: {[raw_input]}")
return raw_input return raw_input
def simple_sentence_split(text: str, max_length: int = 500) -> list[str]:
"""Split text into sentences for better TTS processing.
Simple sentence splitter that breaks on sentence boundaries (.!?)
and ensures no sentence exceeds max_length characters.
"""
# Split on sentence boundaries
sentences = re.split(r'([.!?]+[\s\n]+)', text)
# Recombine sentences with their punctuation
result = []
current = ""
for i in range(0, len(sentences), 2):
sentence = sentences[i]
punct = sentences[i+1] if i+1 < len(sentences) else ""
# If adding this sentence would exceed max_length, save current and start new
if current and len(current) + len(sentence) + len(punct) > max_length:
result.append(current.strip())
current = sentence + punct
else:
current += sentence + punct
# Add remaining text
if current.strip():
result.append(current.strip())
# If we got nothing (no sentence boundaries), split on max_length
if not result and text:
result = [text[i:i+max_length] for i in range(0, len(text), max_length)]
return result if result else [text]
# Auto-detect which model a voice belongs to (uses cached mapping) # Auto-detect which model a voice belongs to (uses cached mapping)
def detect_model_from_voice(voice: str) -> str: def detect_model_from_voice(voice: str) -> str:
"""Find which model supports a given voice name. """Find which model supports a given voice name.
@ -815,34 +866,59 @@ async def generate_speech(request: GenerateSpeechRequest):
kokoro_pipeline = await asyncio.to_thread(kokoro_wrapper, lang_code=lang_code, device=device) kokoro_pipeline = await asyncio.to_thread(kokoro_wrapper, lang_code=lang_code, device=device)
kokoro_lang = lang_code kokoro_lang = lang_code
# Generate audio (also blocking, so run in thread pool) # Split long text into sentences for streaming
audio_data = await asyncio.to_thread(kokoro_pipeline.tts, input_text, voice=kokoro_voice, speed=speed) sentences = simple_sentence_split(input_text, max_length=500)
logger.info(f"Split text into {len(sentences)} sentences for Kokoro streaming")
# Kokoro outputs float32 PCM at 24000 Hz # Kokoro outputs float32 PCM at 24000 Hz
ffmpeg_args = build_ffmpeg_args(response_format, input_format="f32le", sample_rate="24000") ffmpeg_args = build_ffmpeg_args(response_format, input_format="f32le", sample_rate="24000")
ffmpeg_args.extend(["-"]) ffmpeg_args.extend(["-"])
ffmpeg_proc = subprocess.Popen(ffmpeg_args, stdin=subprocess.PIPE, stdout=subprocess.PIPE) ffmpeg_proc = subprocess.Popen(ffmpeg_args, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
ffmpeg_proc.stdin.write(audio_data)
ffmpeg_proc.stdin.close() # Use threading approach like XTTS to ensure proper sequential processing
in_q = queue.Queue() # audio chunks
def generator():
"""Process sentences sequentially and feed to queue"""
try:
for idx, sentence in enumerate(sentences):
logger.debug(f"Processing sentence {idx+1}/{len(sentences)}: {len(sentence)} chars")
audio_bytes = kokoro_pipeline.tts(sentence, voice=kokoro_voice, speed=speed)
in_q.put(audio_bytes)
logger.debug(f"Kokoro: queued sentence {idx+1}/{len(sentences)}")
except Exception as e:
logger.error(f"Kokoro streaming error: {e}")
finally:
in_q.put(None) # sentinel
logger.info(f"Kokoro streaming complete: {len(sentences)} sentences processed")
def out_writer():
"""Write audio from queue to ffmpeg stdin"""
try:
while True:
chunk = in_q.get()
if chunk is None: # sentinel
break
ffmpeg_proc.stdin.write(chunk)
except Exception as e:
logger.error(f"Kokoro ffmpeg write error: {e}")
ffmpeg_proc.kill()
finally:
ffmpeg_proc.stdin.close()
generator_worker = threading.Thread(target=generator, daemon=True)
generator_worker.start()
out_writer_worker = threading.Thread(target=out_writer, daemon=True)
out_writer_worker.start()
return StreamingResponse(content=ffmpeg_proc.stdout, media_type=media_type) return StreamingResponse(content=ffmpeg_proc.stdout, media_type=media_type)
else: else:
raise BadRequestError("No such model, must be tts-1, tts-1-hd, tts-1-silero, or tts-1-kokoro.", param='model') raise BadRequestError("No such model, must be tts-1, tts-1-hd, tts-1-silero, or tts-1-kokoro.", param='model')
# We return 'mps' but currently XTTS will not work with mps devices as the cuda support is incomplete
def auto_torch_device():
try:
import torch
return 'cuda' if torch.cuda.is_available() else 'mps' if ( torch.backends.mps.is_available() and torch.backends.mps.is_built() ) else 'cpu'
except:
return 'none'
if __name__ == "__main__": if __name__ == "__main__":
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
description='UncloseAI Speech API Server', description='uncloseai-speech API Server',
formatter_class=argparse.ArgumentDefaultsHelpFormatter) formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--xtts_device', action='store', default=auto_torch_device(), help="Set the device for the xtts model. The special value of 'none' will use piper for all models.") parser.add_argument('--xtts_device', action='store', default=auto_torch_device(), help="Set the device for the xtts model. The special value of 'none' will use piper for all models.")
@ -850,6 +926,7 @@ if __name__ == "__main__":
parser.add_argument('--unload-timer', action='store', default=None, type=int, help="Idle unload timer for the XTTS model in seconds, Ex. 900 for 15 minutes") parser.add_argument('--unload-timer', action='store', default=None, type=int, help="Idle unload timer for the XTTS model in seconds, Ex. 900 for 15 minutes")
parser.add_argument('--use-deepspeed', action='store_true', default=False, help="Use deepspeed with xtts (this option is unsupported)") parser.add_argument('--use-deepspeed', action='store_true', default=False, help="Use deepspeed with xtts (this option is unsupported)")
parser.add_argument('--no-cache-speaker', action='store_true', default=False, help="Don't use the speaker wav embeddings cache") parser.add_argument('--no-cache-speaker', action='store_true', default=False, help="Don't use the speaker wav embeddings cache")
parser.add_argument('-W', '--workers', action='store', default=4, type=int, help="Number of uvicorn worker processes for concurrent request handling")
parser.add_argument('-P', '--port', action='store', default=8000, type=int, help="Server tcp port") parser.add_argument('-P', '--port', action='store', default=8000, type=int, help="Server tcp port")
parser.add_argument('-H', '--host', action='store', default='0.0.0.0', help="Host to listen on, Ex. 0.0.0.0") parser.add_argument('-H', '--host', action='store', default='0.0.0.0', help="Host to listen on, Ex. 0.0.0.0")
parser.add_argument('-L', '--log-level', default="INFO", choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"], help="Set the log level") parser.add_argument('-L', '--log-level', default="INFO", choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"], help="Set the log level")
@ -872,4 +949,4 @@ if __name__ == "__main__":
# Use multiple workers for true concurrency (each worker = separate process with own GIL) # Use multiple workers for true concurrency (each worker = separate process with own GIL)
# This prevents thread pool exhaustion and allows concurrent model loading # This prevents thread pool exhaustion and allows concurrent model loading
# Must use import string format for workers to function # Must use import string format for workers to function
uvicorn.run("speech:app", host=args.host, port=args.port, workers=4, timeout_keep_alive=300) uvicorn.run("speech:app", host=args.host, port=args.port, workers=args.workers, timeout_keep_alive=300)