# F5-TTS

## Name

**F5-TTS** — A flow-matching zero-shot voice cloning TTS by SWivid.

---

## Overview

F5-TTS is a flow-matching text-to-speech model that performs zero-shot voice cloning from a single reference clip plus its transcript. It is smaller and faster than autoregressive alternatives in this stack (~336M params vs Qwen3-TTS's 1.7B) while producing cleaner clones on identical reference audio. Released under MIT license, model weights are openly downloadable from HuggingFace (`SWivid/F5-TTS`).

---

## Integration Status

- **Model ID:** `tts-1-f5`
- **Status:** Enabled by default (additive, alongside `tts-1-qwen`)
- **Engine:** `f5-tts` (PyPI package `f5-tts==1.1.20`)
- **License:** MIT
- **Added:** 2026-05-24

### Supported Languages

- **English (primary)** — what our voice registry currently exercises
- Community fine-tunes exist for Chinese and other languages (not wired up in our voice map yet)

---

## Technical Specifications

| Field | Value |
|-------|-------|
| Parameters | ~336M |
| Sample rate | 24 kHz (matches `tts-1-qwen`) |
| Vocoder | Vocos (bundled, downloaded on first use) |
| Model download | ~1.5 GB (F5-TTS_v1 + Vocos) |
| Reference format | WAV (3+ seconds preferred) + transcript |
| Inference style | Flow-matching ODE, non-autoregressive |
| Tuning knobs | `nfe_step` (ODE steps), `cfg_strength` (CFG), `speed` |

### Model Loading

The model is pre-downloaded by `startup.sh` on container boot. First boot fetches ~1.5 GB into the HuggingFace cache mounted at `/app/voices/hub/`. Subsequent boots hit cache.

Loaded lazily on first `tts-1-f5` request via `f5_load_semaphore` (single concurrent load). After the first request, the model lives in GPU memory until the process exits.

---

## Integration Details

### Voice Configuration

F5-TTS uses the same reference-audio + reference-transcript shape as `tts-1-qwen`, so our voice map literally duplicates the Qwen block under a `tts-1-f5:` heading. All 40 LibriSpeech voices work for both engines from the same WAV files in `cloned-voices/`.

```yaml
tts-1-f5:
  aria:
    ref_audio: cloned-voices/aria.wav
    ref_text: "BUT THE WINDOWS ARE PATCHED WITH WOODEN PANES AND THE DOOR I THINK IS LIKE THE GATE IT IS NEVER OPENED"
    language: English
```

To add a new voice, drop a 3+ second clean WAV into `cloned-voices/`, transcribe it accurately, and append a stanza to both the `tts-1-qwen:` and `tts-1-f5:` blocks (or just one, depending which engine you want it on).

### Makefile Targets

```bash
make test-f5      # Quick smoke test against tts-1-f5
make voices-f5    # No-op (alias) — reuses voices-qwen LibriSpeech samples
make voices-all   # Includes voices-f5
```

### API Usage

```bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1-f5","voice":"aria","input":"Raccoon mission TTS test with F5 flow matching"}' \
  -o /tmp/f5_test.mp3
```

`tts-1-f5` accepts all standard OpenAI-compatible request fields (`input`, `voice`, `response_format`, `speed`). Engine-specific knobs (`nfe_step`, `cfg_strength`) are NOT exposed at the HTTP API layer today — they default to 32 and 2.0 respectively inside the wrapper. Add request fields only if a benchmark shows they need to be tunable per-call.

---

## Wrapper Implementation

`speech.py:f5_wrapper` mirrors the `qwen3_wrapper` pattern:

```python
class f5_wrapper():
    def __init__(self, device='cuda'):
        from f5_tts.api import F5TTS
        self.model = F5TTS(device=device)
        self.sample_rate = 24000

    def tts(self, text, ref_audio, ref_text, speed=1.0, nfe_step=32, cfg_strength=2.0):
        wav, sr, _spec = self.model.infer(
            ref_file=ref_audio,
            ref_text=ref_text,
            gen_text=text,
            nfe_step=nfe_step,
            cfg_strength=cfg_strength,
            speed=speed,
            show_info=lambda *a, **k: None,
            progress=None,
        )
        if hasattr(wav, 'detach'):
            wav = wav.detach().to('cpu', dtype=torch.float32).numpy()
        return np.asarray(wav, dtype=np.float32).flatten().tobytes()
```

The dispatch in `generate_speech()` splits text by sentence (shared `simple_sentence_split`), runs the model on each sentence in a generator thread, pipes float32 PCM into ffmpeg, and streams back the encoded result. Same scaffold as Qwen and Kokoro.

---

## Performance Characteristics

Per Richard's empirical benchmark on identical reference clips (2026-05-23):
- **Faster** than Qwen3-TTS on the same hardware
- **Better clone fidelity** than Qwen3-TTS on the same reference audio

Quantitative numbers (latency, RTF, VRAM) pending a fresh `make hydrate` run after first deploy on `3090-ai.foxhop.net`.

### Reference (from VoiceClone, MonumentalSystems)
- NVIDIA CUDA: well under realtime on a modern GPU
- Apple Silicon MPS: ~1.5–2× realtime
- CPU: many× realtime (last-resort fallback)

---

## Voice Quality

Subjective verdict (Richard, fox): clones sound closer to the source speaker than Qwen3-TTS on the same 9-second LibriSpeech clips. Less prosody drift, fewer artifacts on long-form output.

---

## Known Issues

- **No `temperature` / `top_p` / `top_k` support.** Flow-matching is deterministic given the reference + seed; sampling knobs from autoregressive models do not apply. Requests containing them are accepted at the FastAPI layer (Pydantic ignores extra fields) but silently dropped in the F5 path.
- **First-request latency:** ~1.5 GB model download on first launch if cache is cold. `startup.sh` pre-downloads on boot to avoid hitting end users with this.
- **Non-English fine-tunes not wired up.** Only English voices in our current map. To add Chinese, German, etc., point a voice stanza at a non-English reference clip + transcript and run `make test-f5` against it.

---

## Raccoon Mission Notes

F5-TTS is a textbook dumpster-dive find: MIT-licensed, smaller than the incumbent, empirically better, and the upstream community already wrote a high-quality single-file web app ([MonumentalSystems/VoiceClone](https://github.com/MonumentalSystems/VoiceClone)) we can reference and learn from. Our wrapper deliberately mirrors their `F5TTS.infer()` call pattern so future upstream changes are easy to follow.

The VoiceClone wrapper itself ships features we have NOT yet pulled in (chunk inspector, regen queue, session export, server-side audio pipeline). Those are engine-agnostic UX wins; consider porting them separately once F5-TTS proves itself in production here.

---

## Future Enhancements

- Expose `nfe_step` and `cfg_strength` as request fields after a benchmark shows them load-bearing
- Add non-English voices to the registry (Chinese fine-tunes exist)
- Port VoiceClone's chunk-inspector frontend as a layer atop our existing endpoint
- Consider making `tts-1-f5` the default (replacing `tts-1-qwen`) once production VRAM/latency data confirms Richard's bench

---

## Resources

- Upstream model + code: https://github.com/SWivid/F5-TTS
- Paper: "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"
- HuggingFace weights: https://huggingface.co/SWivid/F5-TTS
- PyPI: https://pypi.org/project/f5-tts/
- Reference web app: https://github.com/MonumentalSystems/VoiceClone

---

## License

MIT (model + code, SWivid/F5-TTS).