engineering/uncloseai-speech

Fork 0

russell@unturf.com 482b17960c

F5-TTS engine consistency sweep (Makefile, startup, README, docs)

2026-05-24 08:24:38 -04:00

7.2 KiB

Raw Blame History

F5-TTS

Name

F5-TTS — A flow-matching zero-shot voice cloning TTS by SWivid.

Overview

F5-TTS is a flow-matching text-to-speech model that performs zero-shot voice cloning from a single reference clip plus its transcript. It is smaller and faster than autoregressive alternatives in this stack (~336M params vs Qwen3-TTS's 1.7B) while producing cleaner clones on identical reference audio. Released under MIT license, model weights are openly downloadable from HuggingFace (SWivid/F5-TTS).

Integration Status

Model ID: tts-1-f5
Status: Enabled by default (additive, alongside tts-1-qwen)
Engine: f5-tts (PyPI package f5-tts==1.1.20)
License: MIT
Added: 2026-05-24

Supported Languages

English (primary) — what our voice registry currently exercises
Community fine-tunes exist for Chinese and other languages (not wired up in our voice map yet)

Technical Specifications

Field	Value
Parameters	~336M
Sample rate	24 kHz (matches `tts-1-qwen`)
Vocoder	Vocos (bundled, downloaded on first use)
Model download	~1.5 GB (F5-TTS_v1 + Vocos)
Reference format	WAV (3+ seconds preferred) + transcript
Inference style	Flow-matching ODE, non-autoregressive
Tuning knobs	`nfe_step` (ODE steps), `cfg_strength` (CFG), `speed`

Model Loading

The model is pre-downloaded by startup.sh on container boot. First boot fetches ~1.5 GB into the HuggingFace cache mounted at /app/voices/hub/. Subsequent boots hit cache.

Loaded lazily on first tts-1-f5 request via f5_load_semaphore (single concurrent load). After the first request, the model lives in GPU memory until the process exits.

Integration Details

Voice Configuration

F5-TTS uses the same reference-audio + reference-transcript shape as tts-1-qwen, so our voice map literally duplicates the Qwen block under a tts-1-f5: heading. All 40 LibriSpeech voices work for both engines from the same WAV files in cloned-voices/.

tts-1-f5:
  aria:
    ref_audio: cloned-voices/aria.wav
    ref_text: "BUT THE WINDOWS ARE PATCHED WITH WOODEN PANES AND THE DOOR I THINK IS LIKE THE GATE IT IS NEVER OPENED"
    language: English

To add a new voice, drop a 3+ second clean WAV into cloned-voices/, transcribe it accurately, and append a stanza to both the tts-1-qwen: and tts-1-f5: blocks (or just one, depending which engine you want it on).

Makefile Targets

make test-f5      # Quick smoke test against tts-1-f5
make voices-f5    # No-op (alias) — reuses voices-qwen LibriSpeech samples
make voices-all   # Includes voices-f5

API Usage

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1-f5","voice":"aria","input":"Raccoon mission TTS test with F5 flow matching"}' \
  -o /tmp/f5_test.mp3

tts-1-f5 accepts all standard OpenAI-compatible request fields (input, voice, response_format, speed). Engine-specific knobs (nfe_step, cfg_strength) are NOT exposed at the HTTP API layer today — they default to 32 and 2.0 respectively inside the wrapper. Add request fields only if a benchmark shows they need to be tunable per-call.

Wrapper Implementation

speech.py:f5_wrapper mirrors the qwen3_wrapper pattern:

class f5_wrapper():
    def __init__(self, device='cuda'):
        from f5_tts.api import F5TTS
        self.model = F5TTS(device=device)
        self.sample_rate = 24000

    def tts(self, text, ref_audio, ref_text, speed=1.0, nfe_step=32, cfg_strength=2.0):
        wav, sr, _spec = self.model.infer(
            ref_file=ref_audio,
            ref_text=ref_text,
            gen_text=text,
            nfe_step=nfe_step,
            cfg_strength=cfg_strength,
            speed=speed,
            show_info=lambda *a, **k: None,
            progress=None,
        )
        if hasattr(wav, 'detach'):
            wav = wav.detach().to('cpu', dtype=torch.float32).numpy()
        return np.asarray(wav, dtype=np.float32).flatten().tobytes()

The dispatch in generate_speech() splits text by sentence (shared simple_sentence_split), runs the model on each sentence in a generator thread, pipes float32 PCM into ffmpeg, and streams back the encoded result. Same scaffold as Qwen and Kokoro.

Performance Characteristics

Per Richard's empirical benchmark on identical reference clips (2026-05-23):

Faster than Qwen3-TTS on the same hardware
Better clone fidelity than Qwen3-TTS on the same reference audio

Quantitative numbers (latency, RTF, VRAM) pending a fresh make hydrate run after first deploy on 3090-ai.foxhop.net.

Reference (from VoiceClone, MonumentalSystems)

NVIDIA CUDA: well under realtime on a modern GPU
Apple Silicon MPS: ~1.5–2× realtime
CPU: many× realtime (last-resort fallback)

Voice Quality

Subjective verdict (Richard, fox): clones sound closer to the source speaker than Qwen3-TTS on the same 9-second LibriSpeech clips. Less prosody drift, fewer artifacts on long-form output.

Known Issues

No temperature / top_p / top_k support. Flow-matching is deterministic given the reference + seed; sampling knobs from autoregressive models do not apply. Requests containing them are accepted at the FastAPI layer (Pydantic ignores extra fields) but silently dropped in the F5 path.
First-request latency: ~1.5 GB model download on first launch if cache is cold. startup.sh pre-downloads on boot to avoid hitting end users with this.
Non-English fine-tunes not wired up. Only English voices in our current map. To add Chinese, German, etc., point a voice stanza at a non-English reference clip + transcript and run make test-f5 against it.

Raccoon Mission Notes

F5-TTS is a textbook dumpster-dive find: MIT-licensed, smaller than the incumbent, empirically better, and the upstream community already wrote a high-quality single-file web app (MonumentalSystems/VoiceClone) we can reference and learn from. Our wrapper deliberately mirrors their F5TTS.infer() call pattern so future upstream changes are easy to follow.

The VoiceClone wrapper itself ships features we have NOT yet pulled in (chunk inspector, regen queue, session export, server-side audio pipeline). Those are engine-agnostic UX wins; consider porting them separately once F5-TTS proves itself in production here.

Future Enhancements

Expose nfe_step and cfg_strength as request fields after a benchmark shows them load-bearing
Add non-English voices to the registry (Chinese fine-tunes exist)
Port VoiceClone's chunk-inspector frontend as a layer atop our existing endpoint
Consider making tts-1-f5 the default (replacing tts-1-qwen) once production VRAM/latency data confirms Richard's bench

Resources

Upstream model + code: https://github.com/SWivid/F5-TTS
Paper: "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"
HuggingFace weights: https://huggingface.co/SWivid/F5-TTS
PyPI: https://pypi.org/project/f5-tts/
Reference web app: https://github.com/MonumentalSystems/VoiceClone

License

MIT (model + code, SWivid/F5-TTS).

7.2 KiB Raw Blame History Unescape Escape