diff --git a/CHANGELOG.md b/CHANGELOG.md index b380ad2..8c1d272 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,17 @@ ## Recent Changes +**F5-TTS engine added, 2026-05-24** + +* 🦝 **F5-TTS integrated as `tts-1-f5`** (additive, enabled by default alongside `tts-1-qwen`) + - Flow-matching zero-shot voice cloning ([SWivid/F5-TTS](https://github.com/SWivid/F5-TTS), MIT license) + - ~336M params (vs Qwen3-TTS 1.7B), lower VRAM footprint + - 24kHz output, matches Qwen3-TTS sample rate for drop-in voice swap + - Reuses the same 40 LibriSpeech cloned voices as `tts-1-qwen` (shared `cloned-voices/` references) + - Empirical benchmark: faster + better clones than Qwen3-TTS on identical reference clips + - No `temperature` / `top_p` / `top_k` (flow-matching): uses `cfg_strength` + `nfe_step` instead + - Inspiration: [MonumentalSystems/VoiceClone](https://github.com/MonumentalSystems/VoiceClone) β€” our wrapper mirrors their `F5TTS.infer()` call pattern + **Raccoon Mission Updates, 2025-11-09** * 🦝 **Production-ready multiprocess architecture** - 4 uvicorn workers for true concurrency, bypassing Python's GIL diff --git a/Makefile b/Makefile index cb5b5be..38cee76 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ CONTAINER_NAME ?= uncloseai-speech-server-1 -.PHONY: help deploy restart logs test clean stop start voices voices-qwen voices-piper voices-xtts voices-kokoro test-kokoro voices-silero test-silero voices-chatterbox test-chatterbox push-all hydrate load-test test-qwen test-f5 venv venv-run local local-cpu +.PHONY: help deploy restart logs test clean stop start voices voices-qwen voices-f5 voices-piper voices-xtts voices-kokoro test-kokoro voices-silero test-silero voices-chatterbox test-chatterbox push-all hydrate load-test test-qwen test-f5 venv venv-run local local-cpu help: @echo "Raccoon TTS Mission - Development Commands" @@ -21,7 +21,7 @@ help: @echo " make venv - Create Python virtual environment" @echo " make venv-run - Run server in virtual environment" @echo "" - @echo "Testing:" + @echo "Testing (default engines):" @echo " make test - Test TTS endpoint (Qwen3-TTS)" @echo " make test-qwen - Test Qwen3-TTS voice cloning" @echo " make test-f5 - Test F5-TTS voice cloning" @@ -30,6 +30,7 @@ help: @echo "" @echo "Voices:" @echo " make voices-qwen - Download Qwen3-TTS cloned voice samples" + @echo " make voices-f5 - Prepare F5-TTS voices (reuses Qwen samples)" @echo " make voices-piper - Download Piper voices" @echo " make voices-xtts - Download XTTS voices" @echo " make voices-kokoro - Download Kokoro models" @@ -181,7 +182,11 @@ voices-qwen: @echo "Qwen3-TTS voice samples ready in cloned-voices/" @echo " Model (~3.4GB) downloads automatically on first use" -voices-all: voices-qwen voices-piper voices-xtts voices-silero +voices-f5: voices-qwen + @echo "F5-TTS reuses the same cloned-voices/ samples as Qwen3-TTS" + @echo " Model (~1.5GB: F5-TTS_v1 + Vocos) downloads automatically on first use" + +voices-all: voices-qwen voices-f5 voices-piper voices-xtts voices-silero @echo "All voices downloaded!" voices-piper: diff --git a/README.md b/README.md index a82e4c4..815c865 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,9 @@ OpenAI-compatible text-to-speech API server with state-of-the-art voice cloning. -**Default Engine:** [Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) - 1.7B parameters, 10 languages, 97ms latency +**Default Engines:** +- [Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) - 1.7B parameters, 10 languages, 97ms latency (`tts-1-qwen`) +- [F5-TTS](https://huggingface.co/SWivid/F5-TTS) - 336M parameters, flow-matching zero-shot voice cloning, lower VRAM (`tts-1-f5`) ## Quick Start @@ -264,9 +266,16 @@ make stop # Stop container make clean # Remove container ``` -## Other TTS Engines +## Engines -These engines are disabled by default. Enable by uncommenting in `config/voice_to_speaker.yaml` and `requirements.txt`. +**Enabled by default:** + +| Model | Engine | Voices | Speed | Notes | +|-------|--------|--------|-------|-------| +| `tts-1-qwen` | Qwen3-TTS | 40 | Fast | Voice cloning, 10 languages, 1.7B params | +| `tts-1-f5` | F5-TTS | 40 | Faster | Voice cloning, flow-matching, 336M params, lower VRAM | + +**Disabled by default** β€” enable by uncommenting in `config/voice_to_speaker.yaml` and `requirements.txt`: | Model | Engine | Voices | Speed | Notes | |-------|--------|--------|-------|-------| diff --git a/docs/MODELS.md b/docs/MODELS.md index 21cf541..80a5022 100644 --- a/docs/MODELS.md +++ b/docs/MODELS.md @@ -128,7 +128,7 @@ make test-qwen # Test Qwen3-TTS explicitly ### F5-TTS βœ… (also enabled by default) **Status:** INTEGRATED as tts-1-f5 (additive, alongside tts-1-qwen) -**Project:** SWivid/F5-TTS (community-maintained, gated HuggingFace checkpoint) +**Project:** SWivid/F5-TTS (community-maintained, public HuggingFace checkpoint) **License:** MIT (model + code) **Model:** F5-TTS_v1 @@ -146,8 +146,8 @@ make test-qwen # Test Qwen3-TTS explicitly - Size: ~1.5GB (F5-TTS_v1 + Vocos vocoder) **Model Source:** -- HuggingFace: `SWivid/F5-TTS` (**gated** β€” accept license at https://huggingface.co/SWivid/F5-TTS) -- Auto-downloaded on first use; requires `HF_TOKEN` env var +- HuggingFace: `SWivid/F5-TTS` (public, no license accept required) +- Auto-downloaded on first use; `HF_TOKEN` optional (only for higher rate limits) **Integration:** - Used for `tts-1-f5` model (additive default) diff --git a/docs/models/f5-tts.md b/docs/models/f5-tts.md new file mode 100644 index 0000000..21d85e5 --- /dev/null +++ b/docs/models/f5-tts.md @@ -0,0 +1,176 @@ +# F5-TTS + +## Name + +**F5-TTS** β€” A flow-matching zero-shot voice cloning TTS by SWivid. + +--- + +## Overview + +F5-TTS is a flow-matching text-to-speech model that performs zero-shot voice cloning from a single reference clip plus its transcript. It is smaller and faster than autoregressive alternatives in this stack (~336M params vs Qwen3-TTS's 1.7B) while producing cleaner clones on identical reference audio. Released under MIT license, model weights are openly downloadable from HuggingFace (`SWivid/F5-TTS`). + +--- + +## Integration Status + +- **Model ID:** `tts-1-f5` +- **Status:** Enabled by default (additive, alongside `tts-1-qwen`) +- **Engine:** `f5-tts` (PyPI package `f5-tts==1.1.20`) +- **License:** MIT +- **Added:** 2026-05-24 + +### Supported Languages + +- **English (primary)** β€” what our voice registry currently exercises +- Community fine-tunes exist for Chinese and other languages (not wired up in our voice map yet) + +--- + +## Technical Specifications + +| Field | Value | +|-------|-------| +| Parameters | ~336M | +| Sample rate | 24 kHz (matches `tts-1-qwen`) | +| Vocoder | Vocos (bundled, downloaded on first use) | +| Model download | ~1.5 GB (F5-TTS_v1 + Vocos) | +| Reference format | WAV (3+ seconds preferred) + transcript | +| Inference style | Flow-matching ODE, non-autoregressive | +| Tuning knobs | `nfe_step` (ODE steps), `cfg_strength` (CFG), `speed` | + +### Model Loading + +The model is pre-downloaded by `startup.sh` on container boot. First boot fetches ~1.5 GB into the HuggingFace cache mounted at `/app/voices/hub/`. Subsequent boots hit cache. + +Loaded lazily on first `tts-1-f5` request via `f5_load_semaphore` (single concurrent load). After the first request, the model lives in GPU memory until the process exits. + +--- + +## Integration Details + +### Voice Configuration + +F5-TTS uses the same reference-audio + reference-transcript shape as `tts-1-qwen`, so our voice map literally duplicates the Qwen block under a `tts-1-f5:` heading. All 40 LibriSpeech voices work for both engines from the same WAV files in `cloned-voices/`. + +```yaml +tts-1-f5: + aria: + ref_audio: cloned-voices/aria.wav + ref_text: "BUT THE WINDOWS ARE PATCHED WITH WOODEN PANES AND THE DOOR I THINK IS LIKE THE GATE IT IS NEVER OPENED" + language: English +``` + +To add a new voice, drop a 3+ second clean WAV into `cloned-voices/`, transcribe it accurately, and append a stanza to both the `tts-1-qwen:` and `tts-1-f5:` blocks (or just one, depending which engine you want it on). + +### Makefile Targets + +```bash +make test-f5 # Quick smoke test against tts-1-f5 +make voices-f5 # No-op (alias) β€” reuses voices-qwen LibriSpeech samples +make voices-all # Includes voices-f5 +``` + +### API Usage + +```bash +curl -X POST http://localhost:8000/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{"model":"tts-1-f5","voice":"aria","input":"Raccoon mission TTS test with F5 flow matching"}' \ + -o /tmp/f5_test.mp3 +``` + +`tts-1-f5` accepts all standard OpenAI-compatible request fields (`input`, `voice`, `response_format`, `speed`). Engine-specific knobs (`nfe_step`, `cfg_strength`) are NOT exposed at the HTTP API layer today β€” they default to 32 and 2.0 respectively inside the wrapper. Add request fields only if a benchmark shows they need to be tunable per-call. + +--- + +## Wrapper Implementation + +`speech.py:f5_wrapper` mirrors the `qwen3_wrapper` pattern: + +```python +class f5_wrapper(): + def __init__(self, device='cuda'): + from f5_tts.api import F5TTS + self.model = F5TTS(device=device) + self.sample_rate = 24000 + + def tts(self, text, ref_audio, ref_text, speed=1.0, nfe_step=32, cfg_strength=2.0): + wav, sr, _spec = self.model.infer( + ref_file=ref_audio, + ref_text=ref_text, + gen_text=text, + nfe_step=nfe_step, + cfg_strength=cfg_strength, + speed=speed, + show_info=lambda *a, **k: None, + progress=None, + ) + if hasattr(wav, 'detach'): + wav = wav.detach().to('cpu', dtype=torch.float32).numpy() + return np.asarray(wav, dtype=np.float32).flatten().tobytes() +``` + +The dispatch in `generate_speech()` splits text by sentence (shared `simple_sentence_split`), runs the model on each sentence in a generator thread, pipes float32 PCM into ffmpeg, and streams back the encoded result. Same scaffold as Qwen and Kokoro. + +--- + +## Performance Characteristics + +Per Richard's empirical benchmark on identical reference clips (2026-05-23): +- **Faster** than Qwen3-TTS on the same hardware +- **Better clone fidelity** than Qwen3-TTS on the same reference audio + +Quantitative numbers (latency, RTF, VRAM) pending a fresh `make hydrate` run after first deploy on `3090-ai.foxhop.net`. + +### Reference (from VoiceClone, MonumentalSystems) +- NVIDIA CUDA: well under realtime on a modern GPU +- Apple Silicon MPS: ~1.5–2Γ— realtime +- CPU: manyΓ— realtime (last-resort fallback) + +--- + +## Voice Quality + +Subjective verdict (Richard, fox): clones sound closer to the source speaker than Qwen3-TTS on the same 9-second LibriSpeech clips. Less prosody drift, fewer artifacts on long-form output. + +--- + +## Known Issues + +- **No `temperature` / `top_p` / `top_k` support.** Flow-matching is deterministic given the reference + seed; sampling knobs from autoregressive models do not apply. Requests containing them are accepted at the FastAPI layer (Pydantic ignores extra fields) but silently dropped in the F5 path. +- **First-request latency:** ~1.5 GB model download on first launch if cache is cold. `startup.sh` pre-downloads on boot to avoid hitting end users with this. +- **Non-English fine-tunes not wired up.** Only English voices in our current map. To add Chinese, German, etc., point a voice stanza at a non-English reference clip + transcript and run `make test-f5` against it. + +--- + +## Raccoon Mission Notes + +F5-TTS is a textbook dumpster-dive find: MIT-licensed, smaller than the incumbent, empirically better, and the upstream community already wrote a high-quality single-file web app ([MonumentalSystems/VoiceClone](https://github.com/MonumentalSystems/VoiceClone)) we can reference and learn from. Our wrapper deliberately mirrors their `F5TTS.infer()` call pattern so future upstream changes are easy to follow. + +The VoiceClone wrapper itself ships features we have NOT yet pulled in (chunk inspector, regen queue, session export, server-side audio pipeline). Those are engine-agnostic UX wins; consider porting them separately once F5-TTS proves itself in production here. + +--- + +## Future Enhancements + +- Expose `nfe_step` and `cfg_strength` as request fields after a benchmark shows them load-bearing +- Add non-English voices to the registry (Chinese fine-tunes exist) +- Port VoiceClone's chunk-inspector frontend as a layer atop our existing endpoint +- Consider making `tts-1-f5` the default (replacing `tts-1-qwen`) once production VRAM/latency data confirms Richard's bench + +--- + +## Resources + +- Upstream model + code: https://github.com/SWivid/F5-TTS +- Paper: "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching" +- HuggingFace weights: https://huggingface.co/SWivid/F5-TTS +- PyPI: https://pypi.org/project/f5-tts/ +- Reference web app: https://github.com/MonumentalSystems/VoiceClone + +--- + +## License + +MIT (model + code, SWivid/F5-TTS). diff --git a/requirements.txt b/requirements.txt index a257339..67b1801 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,7 +7,7 @@ qwen-tts>=0.0.5 # F5-TTS - flow-matching zero-shot voice cloning (MIT, SWivid/F5-TTS) # 336M params, 24kHz output, no fine-tuning needed # Checked 2026-05-23: 1.1.20 is latest stable -# Note: gated checkpoint on HuggingFace; requires HF_TOKEN env var + license accept +# Public HuggingFace checkpoint β€” HF_TOKEN optional (only for rate-limit relief) f5-tts==1.1.20 # OHF-Voice fork doesn't have installable Python package yet # Stick with PyPI piper-tts but use absolute paths in config diff --git a/sample.env b/sample.env index a39d1ef..9007301 100644 --- a/sample.env +++ b/sample.env @@ -11,7 +11,8 @@ WORKERS=1 #EXTRA_ARGS=--log-level DEBUG --unload-timer 300 #USE_ROCM=1 -# F5-TTS gated checkpoint requires a HuggingFace token + license accept. -# Visit https://huggingface.co/SWivid/F5-TTS, accept the license, then set -# this in your local speech.env (NOT here β€” sample.env is committed). +# Optional HuggingFace token for higher download rate limits. +# Not required β€” F5-TTS / Qwen3-TTS checkpoints are public and downloadable +# anonymously. Set in your local speech.env (NOT here β€” sample.env is committed) +# if you hit rate limits during model pull. #HF_TOKEN= \ No newline at end of file diff --git a/startup.sh b/startup.sh index db3d6b4..98bfe45 100755 --- a/startup.sh +++ b/startup.sh @@ -5,11 +5,14 @@ # Default to 1 worker for GPU models (Qwen3-TTS) WORKERS=${WORKERS:-1} -echo "First startup may download ~3GB of Qwen3-TTS model. Please wait." +echo "First startup may download ~3GB of Qwen3-TTS model + ~1.5GB of F5-TTS model. Please wait." # Pre-download Qwen3-TTS model (default engine) python -c "from qwen_tts import Qwen3TTSModel; Qwen3TTSModel.from_pretrained('Qwen/Qwen3-TTS-12Hz-1.7B-Base')" 2>/dev/null || echo "Qwen3-TTS will download on first request" +# Pre-download F5-TTS model (default engine, additive) +python -c "from f5_tts.api import F5TTS; F5TTS()" 2>/dev/null || echo "F5-TTS will download on first request" + # Optional: download legacy engines if enabled # bash download_voices_tts-1.sh # bash download_voices_tts-1-hd.sh $PRELOAD_MODEL