F5-TTS engine consistency sweep (Makefile, startup, README, docs)

2026-05-24 08:24:38 -04:00 · 2026-05-24 08:24:38 -04:00 · 482b17960c
commit 482b17960c
parent 2df34f85cc
8 changed files with 219 additions and 14 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -2,6 +2,17 @@

 ## Recent Changes

+**F5-TTS engine added, 2026-05-24**
+
+* 🦝 **F5-TTS integrated as `tts-1-f5`** (additive, enabled by default alongside `tts-1-qwen`)
+  - Flow-matching zero-shot voice cloning ([SWivid/F5-TTS](https://github.com/SWivid/F5-TTS), MIT license)
+  - ~336M params (vs Qwen3-TTS 1.7B), lower VRAM footprint
+  - 24kHz output, matches Qwen3-TTS sample rate for drop-in voice swap
+  - Reuses the same 40 LibriSpeech cloned voices as `tts-1-qwen` (shared `cloned-voices/` references)
+  - Empirical benchmark: faster + better clones than Qwen3-TTS on identical reference clips
+  - No `temperature` / `top_p` / `top_k` (flow-matching): uses `cfg_strength` + `nfe_step` instead
+  - Inspiration: [MonumentalSystems/VoiceClone](https://github.com/MonumentalSystems/VoiceClone) — our wrapper mirrors their `F5TTS.infer()` call pattern
+
 **Raccoon Mission Updates, 2025-11-09**

 * 🦝 **Production-ready multiprocess architecture** - 4 uvicorn workers for true concurrency, bypassing Python's GIL
--- a/11
+++ b/11
@ -3,7 +3,7 @@

 CONTAINER_NAME ?= uncloseai-speech-server-1

-.PHONY: help deploy restart logs test clean stop start voices voices-qwen voices-piper voices-xtts voices-kokoro test-kokoro voices-silero test-silero voices-chatterbox test-chatterbox push-all hydrate load-test test-qwen test-f5 venv venv-run local local-cpu
+.PHONY: help deploy restart logs test clean stop start voices voices-qwen voices-f5 voices-piper voices-xtts voices-kokoro test-kokoro voices-silero test-silero voices-chatterbox test-chatterbox push-all hydrate load-test test-qwen test-f5 venv venv-run local local-cpu

 help:
 	@echo "Raccoon TTS Mission - Development Commands"
@ -21,7 +21,7 @@ help:
 	@echo "  make venv        - Create Python virtual environment"
 	@echo "  make venv-run    - Run server in virtual environment"
 	@echo ""
-	@echo "Testing:"
+	@echo "Testing (default engines):"
 	@echo "  make test        - Test TTS endpoint (Qwen3-TTS)"
 	@echo "  make test-qwen   - Test Qwen3-TTS voice cloning"
 	@echo "  make test-f5     - Test F5-TTS voice cloning"
@ -30,6 +30,7 @@ help:
 	@echo ""
 	@echo "Voices:"
 	@echo "  make voices-qwen   - Download Qwen3-TTS cloned voice samples"
+	@echo "  make voices-f5     - Prepare F5-TTS voices (reuses Qwen samples)"
 	@echo "  make voices-piper  - Download Piper voices"
 	@echo "  make voices-xtts   - Download XTTS voices"
 	@echo "  make voices-kokoro - Download Kokoro models"
@ -181,7 +182,11 @@ voices-qwen:
 	@echo "Qwen3-TTS voice samples ready in cloned-voices/"
 	@echo "  Model (~3.4GB) downloads automatically on first use"

-voices-all: voices-qwen voices-piper voices-xtts voices-silero
+voices-f5: voices-qwen
+	@echo "F5-TTS reuses the same cloned-voices/ samples as Qwen3-TTS"
+	@echo "  Model (~1.5GB: F5-TTS_v1 + Vocos) downloads automatically on first use"
+
+voices-all: voices-qwen voices-f5 voices-piper voices-xtts voices-silero
 	@echo "All voices downloaded!"

 voices-piper:
--- a/README.md
+++ b/README.md
@ -2,7 +2,9 @@

 OpenAI-compatible text-to-speech API server with state-of-the-art voice cloning.

-**Default Engine:** [Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) - 1.7B parameters, 10 languages, 97ms latency
+**Default Engines:**
+- [Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) - 1.7B parameters, 10 languages, 97ms latency (`tts-1-qwen`)
+- [F5-TTS](https://huggingface.co/SWivid/F5-TTS) - 336M parameters, flow-matching zero-shot voice cloning, lower VRAM (`tts-1-f5`)

 ## Quick Start

@ -264,9 +266,16 @@ make stop          # Stop container
 make clean         # Remove container
 ```

-## Other TTS Engines
+## Engines

-These engines are disabled by default. Enable by uncommenting in `config/voice_to_speaker.yaml` and `requirements.txt`.
+**Enabled by default:**
+
+| Model | Engine | Voices | Speed | Notes |
+|-------|--------|--------|-------|-------|
+| `tts-1-qwen` | Qwen3-TTS | 40 | Fast | Voice cloning, 10 languages, 1.7B params |
+| `tts-1-f5` | F5-TTS | 40 | Faster | Voice cloning, flow-matching, 336M params, lower VRAM |
+
+**Disabled by default** — enable by uncommenting in `config/voice_to_speaker.yaml` and `requirements.txt`:

 | Model | Engine | Voices | Speed | Notes |
 |-------|--------|--------|-------|-------|
--- a/docs/MODELS.md
+++ b/docs/MODELS.md
@ -128,7 +128,7 @@ make test-qwen   # Test Qwen3-TTS explicitly
 ### F5-TTS ✅ (also enabled by default)

 **Status:** INTEGRATED as tts-1-f5 (additive, alongside tts-1-qwen)
-**Project:** SWivid/F5-TTS (community-maintained, gated HuggingFace checkpoint)
+**Project:** SWivid/F5-TTS (community-maintained, public HuggingFace checkpoint)
 **License:** MIT (model + code)
 **Model:** F5-TTS_v1

@ -146,8 +146,8 @@ make test-qwen   # Test Qwen3-TTS explicitly
 - Size: ~1.5GB (F5-TTS_v1 + Vocos vocoder)

 **Model Source:**
- HuggingFace: `SWivid/F5-TTS` (**gated** — accept license at https://huggingface.co/SWivid/F5-TTS)
- Auto-downloaded on first use; requires `HF_TOKEN` env var
+- HuggingFace: `SWivid/F5-TTS` (public, no license accept required)
+- Auto-downloaded on first use; `HF_TOKEN` optional (only for higher rate limits)

 **Integration:**
 - Used for `tts-1-f5` model (additive default)
--- a/docs/models/f5-tts.md
+++ b/docs/models/f5-tts.md
@ -0,0 +1,176 @@
+# F5-TTS
+
+## Name
+
+**F5-TTS** — A flow-matching zero-shot voice cloning TTS by SWivid.
+
+---
+
+## Overview
+
+F5-TTS is a flow-matching text-to-speech model that performs zero-shot voice cloning from a single reference clip plus its transcript. It is smaller and faster than autoregressive alternatives in this stack (~336M params vs Qwen3-TTS's 1.7B) while producing cleaner clones on identical reference audio. Released under MIT license, model weights are openly downloadable from HuggingFace (`SWivid/F5-TTS`).
+
+---
+
+## Integration Status
+
+- **Model ID:** `tts-1-f5`
+- **Status:** Enabled by default (additive, alongside `tts-1-qwen`)
+- **Engine:** `f5-tts` (PyPI package `f5-tts==1.1.20`)
+- **License:** MIT
+- **Added:** 2026-05-24
+
+### Supported Languages
+
+- **English (primary)** — what our voice registry currently exercises
+- Community fine-tunes exist for Chinese and other languages (not wired up in our voice map yet)
+
+---
+
+## Technical Specifications
+
+| Field | Value |
+|-------|-------|
+| Parameters | ~336M |
+| Sample rate | 24 kHz (matches `tts-1-qwen`) |
+| Vocoder | Vocos (bundled, downloaded on first use) |
+| Model download | ~1.5 GB (F5-TTS_v1 + Vocos) |
+| Reference format | WAV (3+ seconds preferred) + transcript |
+| Inference style | Flow-matching ODE, non-autoregressive |
+| Tuning knobs | `nfe_step` (ODE steps), `cfg_strength` (CFG), `speed` |
+
+### Model Loading
+
+The model is pre-downloaded by `startup.sh` on container boot. First boot fetches ~1.5 GB into the HuggingFace cache mounted at `/app/voices/hub/`. Subsequent boots hit cache.
+
+Loaded lazily on first `tts-1-f5` request via `f5_load_semaphore` (single concurrent load). After the first request, the model lives in GPU memory until the process exits.
+
+---
+
+## Integration Details
+
+### Voice Configuration
+
+F5-TTS uses the same reference-audio + reference-transcript shape as `tts-1-qwen`, so our voice map literally duplicates the Qwen block under a `tts-1-f5:` heading. All 40 LibriSpeech voices work for both engines from the same WAV files in `cloned-voices/`.
+
+```yaml
+tts-1-f5:
+  aria:
+    ref_audio: cloned-voices/aria.wav
+    ref_text: "BUT THE WINDOWS ARE PATCHED WITH WOODEN PANES AND THE DOOR I THINK IS LIKE THE GATE IT IS NEVER OPENED"
+    language: English
+```
+
+To add a new voice, drop a 3+ second clean WAV into `cloned-voices/`, transcribe it accurately, and append a stanza to both the `tts-1-qwen:` and `tts-1-f5:` blocks (or just one, depending which engine you want it on).
+
+### Makefile Targets
+
+```bash
+make test-f5      # Quick smoke test against tts-1-f5
+make voices-f5    # No-op (alias) — reuses voices-qwen LibriSpeech samples
+make voices-all   # Includes voices-f5
+```
+
+### API Usage
+
+```bash
+curl -X POST http://localhost:8000/v1/audio/speech \
+  -H "Content-Type: application/json" \
+  -d '{"model":"tts-1-f5","voice":"aria","input":"Raccoon mission TTS test with F5 flow matching"}' \
+  -o /tmp/f5_test.mp3
+```
+
+`tts-1-f5` accepts all standard OpenAI-compatible request fields (`input`, `voice`, `response_format`, `speed`). Engine-specific knobs (`nfe_step`, `cfg_strength`) are NOT exposed at the HTTP API layer today — they default to 32 and 2.0 respectively inside the wrapper. Add request fields only if a benchmark shows they need to be tunable per-call.
+
+---
+
+## Wrapper Implementation
+
+`speech.py:f5_wrapper` mirrors the `qwen3_wrapper` pattern:
+
+```python
+class f5_wrapper():
+    def __init__(self, device='cuda'):
+        from f5_tts.api import F5TTS
+        self.model = F5TTS(device=device)
+        self.sample_rate = 24000
+
+    def tts(self, text, ref_audio, ref_text, speed=1.0, nfe_step=32, cfg_strength=2.0):
+        wav, sr, _spec = self.model.infer(
+            ref_file=ref_audio,
+            ref_text=ref_text,
+            gen_text=text,
+            nfe_step=nfe_step,
+            cfg_strength=cfg_strength,
+            speed=speed,
+            show_info=lambda *a, **k: None,
+            progress=None,
+        )
+        if hasattr(wav, 'detach'):
+            wav = wav.detach().to('cpu', dtype=torch.float32).numpy()
+        return np.asarray(wav, dtype=np.float32).flatten().tobytes()
+```
+
+The dispatch in `generate_speech()` splits text by sentence (shared `simple_sentence_split`), runs the model on each sentence in a generator thread, pipes float32 PCM into ffmpeg, and streams back the encoded result. Same scaffold as Qwen and Kokoro.
+
+---
+
+## Performance Characteristics
+
+Per Richard's empirical benchmark on identical reference clips (2026-05-23):
+- **Faster** than Qwen3-TTS on the same hardware
+- **Better clone fidelity** than Qwen3-TTS on the same reference audio
+
+Quantitative numbers (latency, RTF, VRAM) pending a fresh `make hydrate` run after first deploy on `3090-ai.foxhop.net`.
+
+### Reference (from VoiceClone, MonumentalSystems)
+- NVIDIA CUDA: well under realtime on a modern GPU
+- Apple Silicon MPS: ~1.5–2× realtime
+- CPU: many× realtime (last-resort fallback)
+
+---
+
+## Voice Quality
+
+Subjective verdict (Richard, fox): clones sound closer to the source speaker than Qwen3-TTS on the same 9-second LibriSpeech clips. Less prosody drift, fewer artifacts on long-form output.
+
+---
+
+## Known Issues
+
+- **No `temperature` / `top_p` / `top_k` support.** Flow-matching is deterministic given the reference + seed; sampling knobs from autoregressive models do not apply. Requests containing them are accepted at the FastAPI layer (Pydantic ignores extra fields) but silently dropped in the F5 path.
+- **First-request latency:** ~1.5 GB model download on first launch if cache is cold. `startup.sh` pre-downloads on boot to avoid hitting end users with this.
+- **Non-English fine-tunes not wired up.** Only English voices in our current map. To add Chinese, German, etc., point a voice stanza at a non-English reference clip + transcript and run `make test-f5` against it.
+
+---
+
+## Raccoon Mission Notes
+
+F5-TTS is a textbook dumpster-dive find: MIT-licensed, smaller than the incumbent, empirically better, and the upstream community already wrote a high-quality single-file web app ([MonumentalSystems/VoiceClone](https://github.com/MonumentalSystems/VoiceClone)) we can reference and learn from. Our wrapper deliberately mirrors their `F5TTS.infer()` call pattern so future upstream changes are easy to follow.
+
+The VoiceClone wrapper itself ships features we have NOT yet pulled in (chunk inspector, regen queue, session export, server-side audio pipeline). Those are engine-agnostic UX wins; consider porting them separately once F5-TTS proves itself in production here.
+
+---
+
+## Future Enhancements
+
+- Expose `nfe_step` and `cfg_strength` as request fields after a benchmark shows them load-bearing
+- Add non-English voices to the registry (Chinese fine-tunes exist)
+- Port VoiceClone's chunk-inspector frontend as a layer atop our existing endpoint
+- Consider making `tts-1-f5` the default (replacing `tts-1-qwen`) once production VRAM/latency data confirms Richard's bench
+
+---
+
+## Resources
+
+- Upstream model + code: https://github.com/SWivid/F5-TTS
+- Paper: "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"
+- HuggingFace weights: https://huggingface.co/SWivid/F5-TTS
+- PyPI: https://pypi.org/project/f5-tts/
+- Reference web app: https://github.com/MonumentalSystems/VoiceClone
+
+---
+
+## License
+
+MIT (model + code, SWivid/F5-TTS).
--- a/requirements.txt
+++ b/requirements.txt
@ -7,7 +7,7 @@ qwen-tts>=0.0.5
 # F5-TTS - flow-matching zero-shot voice cloning (MIT, SWivid/F5-TTS)
 # 336M params, 24kHz output, no fine-tuning needed
 # Checked 2026-05-23: 1.1.20 is latest stable
-# Note: gated checkpoint on HuggingFace; requires HF_TOKEN env var + license accept
+# Public HuggingFace checkpoint — HF_TOKEN optional (only for rate-limit relief)
 f5-tts==1.1.20
 # OHF-Voice fork doesn't have installable Python package yet
 # Stick with PyPI piper-tts but use absolute paths in config
--- a/sample.env
+++ b/sample.env
@ -11,7 +11,8 @@ WORKERS=1
 #EXTRA_ARGS=--log-level DEBUG --unload-timer 300
 #USE_ROCM=1

-# F5-TTS gated checkpoint requires a HuggingFace token + license accept.
-# Visit https://huggingface.co/SWivid/F5-TTS, accept the license, then set
-# this in your local speech.env (NOT here — sample.env is committed).
+# Optional HuggingFace token for higher download rate limits.
+# Not required — F5-TTS / Qwen3-TTS checkpoints are public and downloadable
+# anonymously. Set in your local speech.env (NOT here — sample.env is committed)
+# if you hit rate limits during model pull.
 #HF_TOKEN=
--- a/startup.sh
+++ b/startup.sh
@ -5,11 +5,14 @@
 # Default to 1 worker for GPU models (Qwen3-TTS)
 WORKERS=${WORKERS:-1}

-echo "First startup may download ~3GB of Qwen3-TTS model. Please wait."
+echo "First startup may download ~3GB of Qwen3-TTS model + ~1.5GB of F5-TTS model. Please wait."

 # Pre-download Qwen3-TTS model (default engine)
 python -c "from qwen_tts import Qwen3TTSModel; Qwen3TTSModel.from_pretrained('Qwen/Qwen3-TTS-12Hz-1.7B-Base')" 2>/dev/null || echo "Qwen3-TTS will download on first request"

+# Pre-download F5-TTS model (default engine, additive)
+python -c "from f5_tts.api import F5TTS; F5TTS()" 2>/dev/null || echo "F5-TTS will download on first request"
+
 # Optional: download legacy engines if enabled
 # bash download_voices_tts-1.sh
 # bash download_voices_tts-1-hd.sh $PRELOAD_MODEL