Add detailed AGPL v3 license obligations documentation
Explains source code requirements for network service operators, practical compliance methods, and Raccoon Mission rationale. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
99bc6bf014
commit
4a019cf897
2 changed files with 338 additions and 346 deletions
72
CLAUDE.md
72
CLAUDE.md
|
|
@ -163,31 +163,22 @@ uncloseai-speech/
|
|||
|
||||
## TTS Engine Status
|
||||
|
||||
### Production Ready (95.9% success rate across 245 voices)
|
||||
- ✅ Piper TTS (tts-1) - 55 voices, fast CPU inference
|
||||
- ✅ XTTS v2 (tts-1-hd) - Voice cloning, multilingual
|
||||
- ✅ Silero TTS (tts-1-silero) - 142 voices, 5 languages, auto-downloads
|
||||
- ✅ Kokoro TTS (tts-1-kokoro) - 32 voices, lightweight (82M params)
|
||||
### Default Model (Qwen3-TTS)
|
||||
- ✅ **Qwen3-TTS (tts-1-qwen)** - DEFAULT - 1.7B params, 10 languages, voice cloning, 97ms latency
|
||||
|
||||
### High Priority Integration
|
||||
### Other Engines (disabled by default, enable in voice_to_speaker.yaml)
|
||||
- Piper TTS (tts-1) - 55 voices, fast CPU inference
|
||||
- XTTS v2 (tts-1-hd) - Voice cloning, multilingual
|
||||
- Silero TTS (tts-1-silero) - 142 voices, 5 languages, auto-downloads
|
||||
- Kokoro TTS (tts-1-kokoro) - 32 voices, lightweight (82M params)
|
||||
|
||||
1. **StyleTTS2** ⭐⭐⭐⭐⭐
|
||||
- Why: State-of-the-art quality, best prosody and naturalness
|
||||
- License: MIT (permissive)
|
||||
- Challenge: Complex dependencies (phonemizer), slower inference
|
||||
- Priority: HIGH - Best quality available
|
||||
|
||||
2. **Fish Speech** ⭐⭐⭐⭐
|
||||
- Why: Fast, modern, active development, good multilingual support
|
||||
- License: Apache 2.0
|
||||
- Challenge: Newer/less proven
|
||||
- Priority: MEDIUM-HIGH - Good balance of quality and speed
|
||||
|
||||
3. **Chatterbox** ⭐⭐⭐⭐
|
||||
- Why: Emotion control, 23 languages, zero-shot cloning
|
||||
- License: Apache 2.0
|
||||
- Challenge: Production complexity
|
||||
- Priority: MEDIUM-HIGH - Unique emotion features
|
||||
### Why Qwen3-TTS is Default
|
||||
- State-of-the-art quality with voice cloning
|
||||
- Actively maintained by Alibaba
|
||||
- Apache 2.0 license (commercial-friendly)
|
||||
- 10 languages: Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
|
||||
- Fast first-packet latency (97ms)
|
||||
- Easy voice cloning from 3-second samples
|
||||
|
||||
See `docs/MODELS.md` for complete roadmap and detailed model documentation.
|
||||
|
||||
|
|
@ -257,6 +248,41 @@ tmux send-keys -t 0:1 'cd /home/fox/git/uncloseai-speech && docker compose up --
|
|||
4. **Liberation** - Keep TTS libre (AGPL v3)
|
||||
5. **Unification** - All TTS engines, one API
|
||||
|
||||
## AGPL v3 Compliance
|
||||
|
||||
**This project is AGPL v3 licensed.** The key obligation: anyone who uses this TTS service over a network must be able to access the source code.
|
||||
|
||||
### What This Means
|
||||
|
||||
Unlike regular GPL, AGPL closes the "SaaS loophole". If you run uncloseai-speech as a service (even without distributing binaries), users have the right to request source code.
|
||||
|
||||
### Requirements for Operators
|
||||
|
||||
When running uncloseai-speech as a network service, you must provide:
|
||||
- Complete source code of the running version
|
||||
- Any modifications you've made
|
||||
- Build instructions
|
||||
|
||||
### How to Comply
|
||||
|
||||
1. **Link in API response** - Add source URL to `/v1/models` or root endpoint
|
||||
2. **Link in documentation** - Include repository URL in service docs
|
||||
3. **Host source code** - Keep your fork in a public git repository
|
||||
|
||||
### Example Implementation
|
||||
|
||||
```python
|
||||
# In API responses
|
||||
"source_code": "https://github.com/uncloseai/uncloseai-speech"
|
||||
```
|
||||
|
||||
### Why AGPL?
|
||||
|
||||
- Ensures forks remain open source
|
||||
- Community improvements flow back to the project
|
||||
- Prevents proprietary TTS services from using our work without sharing back
|
||||
- Aligns with Raccoon Mission: **Keep TTS libre**
|
||||
|
||||
## Common Mistakes to Avoid
|
||||
|
||||
❌ DON'T create directories with raw ssh
|
||||
|
|
|
|||
608
README.md
608
README.md
|
|
@ -1,421 +1,387 @@
|
|||
# uncloseai-speech
|
||||
|
||||
🦝 **Raccoon Mission Fork:** Rescuing abandoned TTS models and building a unified, resilient text-to-speech system.
|
||||
OpenAI-compatible text-to-speech API server with state-of-the-art voice cloning.
|
||||
|
||||
**Repository:** https://git.unturf.com/engineering/unturf/uncloseai-speech (GitLab)
|
||||
**Default Engine:** [Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) - 1.7B parameters, 10 languages, 97ms latency
|
||||
|
||||
**Original Notice:** The original `openedai-speech` project (GitHub) was archived and no longer maintained. This is the active fork.
|
||||
|
||||
**Raccoon Mission:** We're bringing it back to life with:
|
||||
- ✅ Working Piper TTS (tts-1) - 55 voices, fast CPU inference
|
||||
- ✅ Working XTTS v2 (tts-1-hd) - 8 voices with cloning capability
|
||||
- ✅ Working Silero TTS (tts-1-silero) - 148 voices, 5 languages, CPU-friendly
|
||||
- ✅ Working Kokoro TTS (tts-1-kokoro) - 34 voices, lightweight decoder
|
||||
- 🎯 Next integrations: StyleTTS2 (best quality), Fish Speech (fast multilingual)
|
||||
- 📚 Comprehensive documentation in `docs/`
|
||||
- 🛠️ Makefile-driven deployment workflow
|
||||
- 🔒 AGPL v3 - keeps TTS libre forever
|
||||
|
||||
See `docs/MODELS.md` for the complete roadmap and `docs/CLAUDE.md` for contribution guidelines.
|
||||
|
||||
----
|
||||
|
||||
An OpenAI API compatible text to speech server.
|
||||
|
||||
* Compatible with the OpenAI audio/speech API
|
||||
* Serves the [/v1/audio/speech endpoint](https://platform.openai.com/docs/api-reference/audio/createSpeech)
|
||||
* Not affiliated with OpenAI in any way, does not require an OpenAI API Key
|
||||
* A free, private, text-to-speech server with custom voice cloning
|
||||
|
||||
Full Compatibility:
|
||||
* `tts-1`: `alloy`, `echo`, `fable`, `onyx`, `nova`, and `shimmer` (configurable, 100+ Piper voices available)
|
||||
* `tts-1-hd`: `alloy`, `echo`, `fable`, `onyx`, `nova`, and `shimmer` (configurable, uses OpenAI samples by default)
|
||||
* `tts-1-silero`: 148 voices with native names (`en_0`, `en_1`, etc.) across 5 languages (English, Russian, German, Spanish, French)
|
||||
* `tts-1-kokoro`: `alloy`, `echo`, `fable`, `onyx`, `nova`, `shimmer` (OpenAI-themed voices) + 34 native voices
|
||||
* `model` parameter is optional - voice auto-detection automatically selects the correct engine
|
||||
* response_format: `mp3`, `opus`, `aac`, `flac`, `wav` and `pcm`
|
||||
* speed 0.25-4.0 (and more)
|
||||
|
||||
Available TTS Engines:
|
||||
* Model `tts-1` via [piper tts](https://github.com/rhasspy/piper) (very fast, runs on cpu)
|
||||
* You can map your own [piper voices](https://rhasspy.github.io/piper-samples/) via the `voice_to_speaker.yaml` configuration file
|
||||
* Model `tts-1-hd` via [coqui-ai/TTS](https://github.com/coqui-ai/TTS) xtts_v2 voice cloning (fast, but requires around 4GB GPU VRAM)
|
||||
* Custom cloned voices can be used for tts-1-hd, See: [Custom Voices Howto](#custom-voices-howto)
|
||||
* 🌐 [Multilingual](#multilingual) support with XTTS voices, the language is automatically detected if not set
|
||||
* [Custom fine-tuned XTTS model support](#custom-fine-tuned-model-support)
|
||||
* Configurable [generation parameters](#generation-parameters)
|
||||
* Streamed output while generating
|
||||
* Model `tts-1-silero` via [Silero TTS](https://github.com/snakers4/silero-models) (fast CPU inference, actively maintained)
|
||||
* 148 voices across 5 languages (English, Russian, German, Spanish, French)
|
||||
* 48kHz sample rate, excellent quality/speed ratio
|
||||
* No GPU required, real-time capable on CPU
|
||||
* Model `tts-1-kokoro` via [Kokoro TTS](https://github.com/hexgrad/kokoro) (lightweight decoder-only architecture)
|
||||
* 34 voices (American and British English)
|
||||
* 82M parameters, fast inference
|
||||
* 24kHz sample rate, Apache 2.0 license
|
||||
* Occasionally, certain words or symbols may sound incorrect, you can fix them with regex via `pre_process_map.yaml`
|
||||
* Tested with python 3.9-3.11, piper does not install on python 3.12 yet
|
||||
|
||||
## High Priority Integration Targets
|
||||
|
||||
We're actively working on integrating these state-of-the-art TTS engines:
|
||||
|
||||
1. **StyleTTS2** ⭐⭐⭐⭐⭐
|
||||
- Why: State-of-the-art quality, best prosody and naturalness
|
||||
- License: MIT (permissive)
|
||||
- Challenge: Complex dependencies (phonemizer), slower inference
|
||||
- Priority: HIGH - Best quality available
|
||||
|
||||
2. **Fish Speech** ⭐⭐⭐⭐
|
||||
- Why: Fast, modern, active development, good multilingual support
|
||||
- License: Apache 2.0
|
||||
- Challenge: Newer/less proven
|
||||
- Priority: MEDIUM-HIGH - Good balance of quality and speed
|
||||
|
||||
See [docs/MODELS.md](docs/MODELS.md) for the complete integration roadmap and detailed documentation on all supported and planned TTS engines.
|
||||
|
||||
|
||||
If you find a better voice match for `tts-1` or `tts-1-hd`, please let me know so I can update the defaults.
|
||||
|
||||
## Recent Changes
|
||||
|
||||
See [CHANGELOG.md](CHANGELOG.md) for full version history.
|
||||
|
||||
## Installation instructions
|
||||
|
||||
### Recommended: Makefile-based workflow
|
||||
|
||||
The project includes a comprehensive Makefile for deployment and development. See available commands:
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
make help
|
||||
git clone https://github.com/uncloseai/uncloseai-speech.git
|
||||
cd uncloseai-speech
|
||||
|
||||
# Option 1: Docker with GPU (recommended)
|
||||
make local
|
||||
|
||||
# Option 2: Docker CPU only
|
||||
make local-cpu
|
||||
|
||||
# Option 3: Python venv (no Docker)
|
||||
make venv && make venv-run
|
||||
```
|
||||
|
||||
#### Quick Start with Makefile
|
||||
|
||||
1. **Create deployment configuration** (if deploying to remote server):
|
||||
Test the API:
|
||||
```bash
|
||||
cp vars.sh.example vars.sh
|
||||
# Edit vars.sh with your server details
|
||||
curl http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"input":"Hello from Qwen TTS!","voice":"alloy"}' \
|
||||
-o test.mp3
|
||||
```
|
||||
|
||||
2. **Deploy to remote server**:
|
||||
## Requirements
|
||||
|
||||
| Setup | GPU | RAM | Disk | Notes |
|
||||
|-------|-----|-----|------|-------|
|
||||
| Docker + GPU | NVIDIA 8GB+ VRAM | 8GB | 5GB | Recommended, fastest |
|
||||
| Docker + CPU | None | 16GB | 5GB | ~10x slower |
|
||||
| Python venv | Optional | 16GB | 5GB | Direct install |
|
||||
|
||||
### GPU Setup (NVIDIA)
|
||||
|
||||
Install [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html):
|
||||
|
||||
```bash
|
||||
make deploy # Sync files, rebuild container, restart
|
||||
make voices # Download Piper + XTTS voice models
|
||||
make test # Test the API
|
||||
make logs # View live logs
|
||||
# Ubuntu/Debian
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
||||
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
|
||||
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
|
||||
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
||||
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
|
||||
sudo nvidia-ctk runtime configure --runtime=docker
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
3. **Local development**:
|
||||
Verify GPU access:
|
||||
```bash
|
||||
make local-deploy # Deploy locally with docker compose
|
||||
docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi
|
||||
```
|
||||
|
||||
See `docs/CLAUDE.md` for detailed Makefile usage and development workflow.
|
||||
## Installation
|
||||
|
||||
### Alternative: Manual Docker setup
|
||||
### Docker with GPU
|
||||
|
||||
#### Create a `speech.env` environment file
|
||||
|
||||
Copy the `sample.env` to `speech.env` (customize if needed)
|
||||
```bash
|
||||
cp sample.env speech.env
|
||||
make local
|
||||
# Or: docker compose up -d --build
|
||||
```
|
||||
|
||||
**Defaults:**
|
||||
### Docker CPU Only
|
||||
|
||||
```bash
|
||||
TTS_HOME=voices
|
||||
HF_HOME=voices
|
||||
#PRELOAD_MODEL=xtts
|
||||
#PRELOAD_MODEL=xtts_v2.0.2
|
||||
#EXTRA_ARGS=--log-level DEBUG --unload-timer 300
|
||||
#USE_ROCM=1
|
||||
cp sample.env speech.env
|
||||
make local-cpu
|
||||
# Or: docker compose -f docker-compose.cpu.yml up -d --build
|
||||
```
|
||||
|
||||
#### Docker Images
|
||||
### Python Virtual Environment
|
||||
|
||||
**Nvidia GPU (cuda)**
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
```bash
|
||||
# Create and activate venv
|
||||
make venv
|
||||
|
||||
**AMD GPU (ROCm support)**
|
||||
```shell
|
||||
docker compose -f docker-compose.rocm.yml up
|
||||
```
|
||||
# Run the server
|
||||
make venv-run
|
||||
|
||||
**ARM64 (Apple M-series, Raspberry Pi)**
|
||||
> XTTS only has CPU support here and will be very slow, you can use the Nvidia image for XTTS with CPU (slow), or use the piper only image (recommended)
|
||||
|
||||
**CPU only, No GPU (piper only)**
|
||||
> For a minimal docker image with only piper support (<1GB vs. 8GB).
|
||||
```shell
|
||||
docker compose -f docker-compose.min.yml up
|
||||
```
|
||||
|
||||
### Alternative: Manual Python installation
|
||||
|
||||
```shell
|
||||
# install curl and ffmpeg
|
||||
sudo apt install curl ffmpeg
|
||||
# Create & activate a new virtual environment (optional but recommended)
|
||||
python -m venv .venv
|
||||
# Or manually:
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
# Install the Python requirements
|
||||
# - use requirements-rocm.txt for AMD GPU (ROCm support)
|
||||
# - use requirements-min.txt for piper only (CPU only)
|
||||
pip install -U -r requirements.txt
|
||||
# run the server
|
||||
bash startup.sh
|
||||
pip install -r requirements.txt
|
||||
python speech.py
|
||||
```
|
||||
|
||||
> On first run, the voice models will be downloaded automatically. This might take a while depending on your network connection.
|
||||
### AMD GPU (ROCm)
|
||||
|
||||
## Server Options
|
||||
|
||||
```shell
|
||||
usage: speech.py [-h] [--xtts_device XTTS_DEVICE] [--preload PRELOAD] [--unload-timer UNLOAD_TIMER]
|
||||
[--use-deepspeed] [--no-cache-speaker] [-W WORKERS] [-P PORT] [-H HOST]
|
||||
[-L {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
|
||||
|
||||
uncloseai-speech API Server
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
--xtts_device XTTS_DEVICE
|
||||
Set the device for the xtts model. The special value of 'none' will use piper for all models. (default: cuda)
|
||||
--preload PRELOAD Preload a model (Ex. 'xtts' or 'xtts_v2.0.2'). By default it's loaded on first use. (default: None)
|
||||
--unload-timer UNLOAD_TIMER
|
||||
Idle unload timer for the XTTS model in seconds, Ex. 900 for 15 minutes (default: None)
|
||||
--use-deepspeed Use deepspeed with xtts (this option is unsupported) (default: False)
|
||||
--no-cache-speaker Don't use the speaker wav embeddings cache (default: False)
|
||||
-W WORKERS, --workers WORKERS
|
||||
Number of uvicorn worker processes for concurrent request handling (default: 4)
|
||||
-P PORT, --port PORT Server tcp port (default: 8000)
|
||||
-H HOST, --host HOST Host to listen on, Ex. 0.0.0.0 (default: 0.0.0.0)
|
||||
-L {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
|
||||
Set the log level (default: INFO)
|
||||
```bash
|
||||
docker compose -f docker-compose.rocm.yml up -d --build
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
## Sample Usage
|
||||
### Generate Speech
|
||||
|
||||
You can use it like this:
|
||||
```bash
|
||||
POST /v1/audio/speech
|
||||
```
|
||||
|
||||
```shell
|
||||
curl http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" -d '{
|
||||
"model": "tts-1",
|
||||
"input": "The quick brown fox jumped over the lazy dog.",
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `input` | string | required | Text to synthesize |
|
||||
| `voice` | string | `alloy` | Voice name |
|
||||
| `model` | string | `tts-1-qwen` | Model ID |
|
||||
| `response_format` | string | `mp3` | `mp3`, `opus`, `aac`, `flac`, `wav`, `pcm` |
|
||||
| `speed` | float | `1.0` | Speed multiplier (0.25-4.0) |
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
curl http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "tts-1-qwen",
|
||||
"voice": "alloy",
|
||||
"input": "The quick brown fox jumped over the lazy dog.",
|
||||
"response_format": "mp3",
|
||||
"speed": 1.0
|
||||
}' > speech.mp3
|
||||
}' -o speech.mp3
|
||||
```
|
||||
|
||||
Or just like this:
|
||||
### List Models
|
||||
|
||||
```shell
|
||||
curl -s http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" -d '{
|
||||
"input": "The quick brown fox jumped over the lazy dog."}' > speech.mp3
|
||||
```bash
|
||||
GET /v1/models
|
||||
```
|
||||
|
||||
Or like this example from the [OpenAI Text to speech guide](https://platform.openai.com/docs/guides/text-to-speech):
|
||||
### List Voices
|
||||
|
||||
```bash
|
||||
GET /v1/voices
|
||||
```
|
||||
|
||||
Returns all voices with metadata including engine, sample rate, and language support.
|
||||
|
||||
## Python SDK Usage
|
||||
|
||||
```python
|
||||
import openai
|
||||
|
||||
client = openai.OpenAI(
|
||||
# This part is not needed if you set these environment variables before import openai
|
||||
# export OPENAI_API_KEY=sk-11111111111
|
||||
# export OPENAI_BASE_URL=http://localhost:8000/v1
|
||||
api_key = "sk-111111111",
|
||||
base_url = "http://localhost:8000/v1",
|
||||
api_key="not-needed",
|
||||
base_url="http://localhost:8000/v1",
|
||||
)
|
||||
|
||||
# Basic usage
|
||||
with client.audio.speech.with_streaming_response.create(
|
||||
model="tts-1",
|
||||
model="tts-1-qwen",
|
||||
voice="alloy",
|
||||
input="Today is a wonderful day to build something people love!"
|
||||
input="Hello world!"
|
||||
) as response:
|
||||
response.stream_to_file("speech.mp3")
|
||||
|
||||
# With options
|
||||
with client.audio.speech.with_streaming_response.create(
|
||||
model="tts-1-qwen",
|
||||
voice="nova",
|
||||
input="This is faster speech.",
|
||||
response_format="opus",
|
||||
speed=1.2
|
||||
) as response:
|
||||
response.stream_to_file("speech.opus")
|
||||
```
|
||||
|
||||
Also see the `say.py` sample application for an example of how to use the openai-python API.
|
||||
## Voice Cloning
|
||||
|
||||
```shell
|
||||
# play the audio, requires 'pip install playsound'
|
||||
python say.py -t "The quick brown fox jumped over the lazy dog." -p
|
||||
# save to a file in flac format
|
||||
python say.py -t "The quick brown fox jumped over the lazy dog." -m tts-1-hd -v onyx -f flac -o fox.flac
|
||||
Qwen3-TTS clones any voice from a 3+ second audio sample.
|
||||
|
||||
### 1. Prepare Reference Audio
|
||||
|
||||
- **Length:** 3-30 seconds (6-10 optimal)
|
||||
- **Quality:** Clear speech, minimal noise
|
||||
- **Format:** WAV, MP3, or URL
|
||||
|
||||
### 2. Configure Voice
|
||||
|
||||
Edit `config/voice_to_speaker.yaml`:
|
||||
|
||||
```yaml
|
||||
tts-1-qwen:
|
||||
my_voice:
|
||||
ref_audio: voices/my_sample.wav # Local file or URL
|
||||
ref_text: "Exact transcript of the audio."
|
||||
language: English
|
||||
```
|
||||
|
||||
You can also try the included `audio_reader.py` for listening to longer text and streamed input.
|
||||
### 3. Use the Voice
|
||||
|
||||
Example usage:
|
||||
```bash
|
||||
python audio_reader.py -s 2 < LICENSE # read the software license - fast
|
||||
curl http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"voice":"my_voice","input":"Hello in my cloned voice!"}' \
|
||||
-o output.mp3
|
||||
```
|
||||
|
||||
## OpenAI API Documentation and Guide
|
||||
### Supported Languages
|
||||
|
||||
* [OpenAI Text to speech guide](https://platform.openai.com/docs/guides/text-to-speech)
|
||||
* [OpenAI API Reference](https://platform.openai.com/docs/api-reference/audio/createSpeech)
|
||||
Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
|
||||
|
||||
## Default Voices
|
||||
|
||||
## Custom Voices Howto
|
||||
| Voice | Description |
|
||||
|-------|-------------|
|
||||
| `alloy` | Neutral, balanced |
|
||||
| `echo` | Warm, conversational |
|
||||
| `fable` | Expressive, storytelling |
|
||||
| `onyx` | Deep, authoritative |
|
||||
| `nova` | Friendly, upbeat |
|
||||
| `shimmer` | Soft, gentle |
|
||||
|
||||
### Piper
|
||||
All voices use Qwen3-TTS voice cloning with pre-configured reference audio.
|
||||
|
||||
1. Select the piper voice and model from the [piper samples](https://rhasspy.github.io/piper-samples/)
|
||||
2. Update the `config/voice_to_speaker.yaml` with a new section for the voice, for example:
|
||||
```yaml
|
||||
...
|
||||
tts-1:
|
||||
ryan:
|
||||
model: voices/en_US-ryan-high.onnx
|
||||
speaker: # default speaker
|
||||
```
|
||||
3. New models will be downloaded as needed, of you can download them in advance with `download_voices_tts-1.sh`. For example:
|
||||
```shell
|
||||
bash download_voices_tts-1.sh en_US-ryan-high
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Edit `speech.env`:
|
||||
|
||||
```bash
|
||||
TTS_HOME=voices # Model cache directory
|
||||
HF_HOME=voices # HuggingFace cache
|
||||
EXTRA_ARGS=--log-level INFO # Additional server args
|
||||
```
|
||||
|
||||
### Coqui XTTS v2
|
||||
### Server Arguments
|
||||
|
||||
Coqui XTTS v2 voice cloning can work with as little as 6 seconds of clear audio. To create a custom voice clone, you must prepare a WAV file sample of the voice.
|
||||
|
||||
#### Guidelines for preparing good sample files for Coqui XTTS v2
|
||||
* Mono (single channel) 22050 Hz WAV file
|
||||
* 6-30 seconds long - longer isn't always better (I've had some good results with as little as 4 seconds)
|
||||
* low noise (no hiss or hum)
|
||||
* No partial words, breathing, laughing, music or backgrounds sounds
|
||||
* An even speaking pace with a variety of words is best, like in interviews or audiobooks.
|
||||
* Audio longer than 30 seconds will be silently truncated.
|
||||
|
||||
You can use FFmpeg to prepare your audio files, here are some examples:
|
||||
|
||||
```shell
|
||||
# convert a multi-channel audio file to mono, set sample rate to 22050 hz, trim to 6 seconds, and output as WAV file.
|
||||
ffmpeg -i input.mp3 -ac 1 -ar 22050 -t 6 -y me.wav
|
||||
# use a simple noise filter to clean up audio, and select a start time start for sampling.
|
||||
ffmpeg -i input.wav -af "highpass=f=200, lowpass=f=3000" -ac 1 -ar 22050 -ss 00:13:26.2 -t 6 -y me.wav
|
||||
# A more complex noise reduction setup, including volume adjustment
|
||||
ffmpeg -i input.mkv -af "highpass=f=200, lowpass=f=3000, volume=5, afftdn=nf=25" -ac 1 -ar 22050 -ss 00:13:26.2 -t 6 -y me.wav
|
||||
```
|
||||
--xtts_device DEVICE Device: cuda, cpu, none (default: auto-detect)
|
||||
--workers N Worker processes (default: 4)
|
||||
--port PORT Listen port (default: 8000)
|
||||
--host HOST Bind address (default: 0.0.0.0)
|
||||
--log-level LEVEL DEBUG, INFO, WARNING, ERROR, CRITICAL
|
||||
```
|
||||
|
||||
Once your WAV file is prepared, save it in the `/voices/` directory and update the `config/voice_to_speaker.yaml` file with the new file name.
|
||||
## Makefile Commands
|
||||
|
||||
For example:
|
||||
```bash
|
||||
make help # Show all commands
|
||||
|
||||
```yaml
|
||||
...
|
||||
tts-1-hd:
|
||||
me:
|
||||
model: xtts
|
||||
speaker: voices/me.wav # this could be you
|
||||
# Local Development
|
||||
make local # Docker with GPU
|
||||
make local-cpu # Docker CPU only
|
||||
make venv # Create Python venv
|
||||
make venv-run # Run in venv
|
||||
|
||||
# Testing
|
||||
make test # Test API
|
||||
make logs # View logs
|
||||
|
||||
# Remote Deployment
|
||||
make deploy # Sync + restart remote
|
||||
make sync # Sync files only
|
||||
make restart # Restart container
|
||||
|
||||
# Container
|
||||
make start # Start container
|
||||
make stop # Stop container
|
||||
make clean # Remove container
|
||||
```
|
||||
|
||||
You can also use a sub folder for multiple audio samples to combine small samples or to mix different samples together.
|
||||
## Other TTS Engines
|
||||
|
||||
For example:
|
||||
These engines are disabled by default. Enable by uncommenting in `config/voice_to_speaker.yaml` and `requirements.txt`.
|
||||
|
||||
```yaml
|
||||
...
|
||||
tts-1-hd:
|
||||
mixed:
|
||||
model: xtts
|
||||
speaker: voices/mixed
|
||||
| Model | Engine | Voices | Speed | Notes |
|
||||
|-------|--------|--------|-------|-------|
|
||||
| `tts-1` | Piper | 55 | Fast | CPU optimized |
|
||||
| `tts-1-hd` | XTTS v2 | 8 | Medium | Voice cloning |
|
||||
| `tts-1-silero` | Silero | 148 | Fast | 5 languages |
|
||||
| `tts-1-kokoro` | Kokoro | 34 | Fast | 82M params |
|
||||
|
||||
See [docs/MODELS.md](docs/MODELS.md) for details.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Model Download Fails
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
docker logs uncloseai-speech-server-1
|
||||
|
||||
# Manual download
|
||||
docker exec -it uncloseai-speech-server-1 \
|
||||
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base
|
||||
```
|
||||
|
||||
Where the `voices/mixed/` folder contains multiple wav files. The total audio length is still limited to 30 seconds.
|
||||
### Out of GPU Memory
|
||||
|
||||
## Multilingual
|
||||
Qwen3-TTS needs ~6GB VRAM. Options:
|
||||
|
||||
uncloseai-speech supports multiple languages across different TTS engines:
|
||||
1. Add to `speech.env`: `EXTRA_ARGS=--xtts_device cpu`
|
||||
2. Reduce workers: `EXTRA_ARGS=--workers 1`
|
||||
3. Use CPU-only: `make local-cpu`
|
||||
|
||||
### XTTS (tts-1-hd) - 17 Languages
|
||||
### Slow Generation
|
||||
|
||||
Multilingual cloning support was added in version 0.11.0. Coqui XTTSv2 has support for multiple languages: English (`en`), Spanish (`es`), French (`fr`), German (`de`), Italian (`it`), Portuguese (`pt`), Polish (`pl`), Turkish (`tr`), Russian (`ru`), Dutch (`nl`), Czech (`cs`), Arabic (`ar`), Chinese (`zh-cn`), Hungarian (`hu`), Korean (`ko`), Japanese (`ja`), and Hindi (`hi`). When not set, an attempt will be made to automatically detect the language, falling back to English (`en`).
|
||||
- GPU: ~1-2 seconds per sentence
|
||||
- CPU: ~10-20 seconds per sentence
|
||||
|
||||
Unfortunately the OpenAI API does not support language parameters, but you can create your own custom speaker voice and set the language for that.
|
||||
For faster CPU inference, enable Piper or Silero engines.
|
||||
|
||||
1) Create the WAV file for your speaker, as in [Custom Voices Howto](#custom-voices-howto)
|
||||
2) Add the voice to `config/voice_to_speaker.yaml` and include the correct Coqui `language` code for the speaker. For example:
|
||||
### Voice Quality Issues
|
||||
|
||||
```yaml
|
||||
xunjiang:
|
||||
model: xtts
|
||||
speaker: voices/xunjiang.wav
|
||||
language: zh-cn
|
||||
- Use 6-10 seconds of clear reference audio
|
||||
- Ensure transcript exactly matches audio
|
||||
- Avoid background noise
|
||||
- Match language setting to audio language
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Client │
|
||||
│ (OpenAI SDK / curl) │
|
||||
└─────────────────┬───────────────────────────┘
|
||||
│ HTTP POST /v1/audio/speech
|
||||
▼
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ FastAPI Server │
|
||||
│ (speech.py, port 8000) │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────┐ Voice Config │
|
||||
│ │ Qwen3-TTS │◄─────────────────────────┐ │
|
||||
│ │ (default) │ config/voice_to_speaker │ │
|
||||
│ └──────┬──────┘ │ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ ┌─────────────┐ │ │
|
||||
│ │ FFmpeg │ Audio encoding │ │
|
||||
│ │ (mp3/opus) │ │ │
|
||||
│ └──────┬──────┘ │ │
|
||||
│ │ │ │
|
||||
└─────────┼──────────────────────────────────┘
|
||||
│
|
||||
▼ Audio stream
|
||||
Client
|
||||
```
|
||||
|
||||
3) Don't remove high unicode characters in your `config/pre_process_map.yaml`! If you have these lines, you will need to remove them. For example:
|
||||
## License
|
||||
|
||||
Remove:
|
||||
```yaml
|
||||
- - '[\U0001F600-\U0001F64F\U0001F300-\U0001F5FF\U0001F680-\U0001F6FF\U0001F700-\U0001F77F\U0001F780-\U0001F7FF\U0001F800-\U0001F8FF\U0001F900-\U0001F9FF\U0001FA00-\U0001FA6F\U0001FA70-\U0001FAFF\U00002702-\U000027B0\U000024C2-\U0001F251]+'
|
||||
- ''
|
||||
**AGPL v3** - This software is licensed under the [GNU Affero General Public License v3](https://www.gnu.org/licenses/agpl-3.0.html).
|
||||
|
||||
### Key AGPL v3 Obligations
|
||||
|
||||
1. **Network use triggers copyleft** - Unlike regular GPL, AGPL closes the "SaaS loophole". If you run uncloseai-speech as a service (even without distributing binaries), users have the right to request source code.
|
||||
|
||||
2. **What you must provide:**
|
||||
- Complete source code of the running version
|
||||
- Any modifications you've made
|
||||
- Build instructions
|
||||
|
||||
3. **How to comply:**
|
||||
- Link to your source repository in API responses or docs
|
||||
- Offer source code download from the same server
|
||||
- Keep your modifications in a public git repo
|
||||
|
||||
### Practical Implementation
|
||||
|
||||
For a service at `ai.foxhop.net`, you could:
|
||||
|
||||
```python
|
||||
# Add to API response headers or /source endpoint
|
||||
"source_code": "https://github.com/uncloseai/uncloseai-speech"
|
||||
```
|
||||
|
||||
These lines were added to the `config/pre_process_map.yaml` config file by default before version 0.11.0.
|
||||
Or include it in your API's `/models` or root endpoint response.
|
||||
|
||||
4) Your new multi-lingual speaker voice is ready to use!
|
||||
### Why AGPL for TTS?
|
||||
|
||||
### Silero (tts-1-silero) - 5 Languages
|
||||
From the Raccoon Mission values:
|
||||
- **Liberation** - Keeps TTS libre
|
||||
- **Resilience** - Ensures forks remain open
|
||||
- **Unification** - Community improvements flow back
|
||||
|
||||
Silero TTS provides native multilingual support with 148 voices across 5 languages:
|
||||
- **English (en)** - 117 voices
|
||||
- **Russian (ru)** - 10 voices
|
||||
- **German (de)** - 5 voices
|
||||
- **Spanish (es)** - 3 voices
|
||||
- **French (fr)** - 5 voices
|
||||
## Links
|
||||
|
||||
Each language has multiple speaker variations. Voices are automatically configured in `config/voice_to_speaker.yaml`. Simply select the appropriate voice (e.g., `en_0`, `ru_0`, `de_0`) and the correct language model will be loaded automatically.
|
||||
|
||||
### Kokoro (tts-1-kokoro) - English Only
|
||||
|
||||
Kokoro TTS currently supports only American and British English with 34 high-quality voices. The model uses a lightweight decoder-only architecture (82M parameters) optimized for English speech synthesis.
|
||||
|
||||
### Piper (tts-1) - 50+ Languages
|
||||
|
||||
For Piper TTS, simply download language-specific voice models from [piper samples](https://rhasspy.github.io/piper-samples/). Piper supports 50+ languages with hundreds of voice options. Add the voice to `config/voice_to_speaker.yaml` as shown in [Custom Voices Howto](#custom-voices-howto).
|
||||
|
||||
|
||||
## Custom Fine-Tuned Model Support
|
||||
|
||||
Adding a custom xtts model is simple. Here is an example of how to add a custom fine-tuned 'halo' XTTS model.
|
||||
|
||||
1) Save the model folder under `voices/` (all 4 files are required, including the vocab.json from the model)
|
||||
```
|
||||
uncloseai-speech$ ls voices/halo/
|
||||
config.json vocab.json model.pth sample.wav
|
||||
```
|
||||
2) Add the custom voice entry under the `tts-1-hd` section of `config/voice_to_speaker.yaml`:
|
||||
```yaml
|
||||
tts-1-hd:
|
||||
...
|
||||
halo:
|
||||
model: halo # This name is required to be unique
|
||||
speaker: voices/halo/sample.wav # voice sample is required
|
||||
model_path: voices/halo
|
||||
```
|
||||
3) The model will be loaded when you access the voice for the first time (`--preload` doesn't work with custom models yet)
|
||||
|
||||
## Generation Parameters
|
||||
|
||||
The generation of XTTSv2 voices can be fine tuned with the following options (defaults included below):
|
||||
|
||||
```yaml
|
||||
tts-1-hd:
|
||||
alloy:
|
||||
model: xtts
|
||||
speaker: voices/alloy.wav
|
||||
enable_text_splitting: True
|
||||
length_penalty: 1.0
|
||||
repetition_penalty: 10
|
||||
speed: 1.0
|
||||
temperature: 0.75
|
||||
top_k: 50
|
||||
top_p: 0.85
|
||||
```
|
||||
- [Documentation](docs/MODELS.md)
|
||||
- [Contributing](CLAUDE.md)
|
||||
- [Qwen3-TTS Model](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base)
|
||||
- [OpenAI TTS API Reference](https://platform.openai.com/docs/api-reference/audio/createSpeech)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue