2d1e1b344f
Added detailed documentation for 10 TTS models: - Coqui TTS (XTTS-v2): High-quality multilingual with voice cloning - Mozilla TTS: Historical context, superseded by Coqui - Piper TTS: Fast, lightweight, 100+ voices - Chatterbox: Emotion control, 23 languages - Mimic 3: Privacy-focused, offline capable - eSpeak NG: 100+ languages, accessibility-focused - Kokoro TTS: Fast decoder-only architecture - Tortoise TTS: Studio-quality but slow - Step-Audio-EditX: Experimental LLM-based audio editing - Maya1: Indic languages specialist Created comprehensive research overview: - Complete model comparison matrix - Performance characteristics and feature analysis - License compatibility analysis - Integration roadmap and priorities - Raccoon Mission risk assessment Updated MODELS.md: - Added documentation index with links to all model docs - Added research overview reference - Added detailed doc references in existing sections - Added "Additional Models Under Research" section All documentation follows Raccoon Mission theme of rescuing abandoned open-source TTS models for long-term preservation.
354 lines
12 KiB
Markdown
354 lines
12 KiB
Markdown
# Mozilla TTS
|
|
|
|
## Name
|
|
|
|
**Mozilla TTS** (now **TTS from Hugging Face** / **Coqui TTS**)
|
|
|
|
The project was originally developed and maintained by Mozilla, subsequently evolved into Coqui TTS, and is now hosted under the broader TTS ecosystem on Hugging Face.
|
|
|
|
---
|
|
|
|
## Description
|
|
|
|
Mozilla TTS is an end-to-end neural text-to-speech (TTS) engine that combines the **Tacotron 2** architecture for mel-spectrogram generation with advanced **vocoder** technology such as **HiFi-GAN** for high-quality waveform synthesis. The system generates realistic, natural-sounding speech from text input with strong prosody modeling and accent control.
|
|
|
|
The engine is designed with a modular architecture that separates:
|
|
- **Acoustic modeling** (text → mel-spectrogram)
|
|
- **Vocoding** (mel-spectrogram → waveform)
|
|
|
|
This separation allows for flexible combinations of models and vocoders, enabling researchers and practitioners to experiment with different architectures and configurations.
|
|
|
|
---
|
|
|
|
## Key Features
|
|
|
|
### Strengths
|
|
|
|
- **High-quality voice synthesis**: Produces natural and intelligible speech across multiple languages
|
|
- **Modular architecture**: Separates text processing, acoustic modeling, and vocoding for flexibility
|
|
- **Multiple vocoder options**: Supports HiFi-GAN, MelGAN, and other state-of-the-art vocoders
|
|
- **Fine-tuning on custom datasets**: Allows training on domain-specific or custom voice datasets
|
|
- **Strong prosody modeling**: Handles stress, intonation, and speech variation effectively
|
|
- **Open-source**: Code available on GitHub with Mozilla Public License
|
|
|
|
### Limitations
|
|
|
|
- **Limited out-of-the-box language support**: While multilingual models exist, default pretrained models cover fewer languages compared to commercial solutions
|
|
- **Longer inference time**: CPU inference is slower compared to some lightweight TTS engines
|
|
- **Resource requirements**: GPU recommended for real-time synthesis; requires significant memory for training
|
|
- **Maintenance**: Project transitioned to Coqui and subsequently to community-maintained versions; may have reduced official support
|
|
- **Documentation inconsistency**: Some documentation became outdated after the transition to Coqui
|
|
|
|
---
|
|
|
|
## License
|
|
|
|
**Mozilla Public License 2.0 (MPL 2.0)**
|
|
|
|
This is a weak copyleft license that allows:
|
|
- Commercial use
|
|
- Distribution
|
|
- Modification
|
|
- Private use
|
|
|
|
With the requirement that:
|
|
- Source code must be disclosed
|
|
- The same license applies to modified code
|
|
|
|
---
|
|
|
|
## Links
|
|
|
|
- **Original Mozilla TTS GitHub**: [https://github.com/mozilla/TTS](https://github.com/mozilla/TTS)
|
|
- **Coqui TTS (Current Continuation)**: [https://github.com/coqui-ai/TTS](https://github.com/coqui-ai/TTS)
|
|
- **Hugging Face Model Hub**: [https://huggingface.co/models?search=mozilla](https://huggingface.co/models?search=mozilla)
|
|
- **Documentation**: [https://tts.readthedocs.io/](https://tts.readthedocs.io/)
|
|
- **Paper (Glow-TTS)**: [https://arxiv.org/abs/2005.05957](https://arxiv.org/abs/2005.05957)
|
|
|
|
---
|
|
|
|
## Integration Status
|
|
|
|
**Status**: Not integrated (superseded by Coqui)
|
|
|
|
While Mozilla TTS is not currently integrated into uncloseai-speech, the codebase and models remain highly relevant. The project has been superseded by **Coqui TTS**, which represents the actively maintained continuation of Mozilla TTS development.
|
|
|
|
### Reasons for Non-Integration
|
|
|
|
1. **Maintenance transition**: Development moved from Mozilla to Coqui AI
|
|
2. **Coqui TTS focus**: The successor project (Coqui TTS) is more actively developed with additional features
|
|
3. **Community fork landscape**: Multiple community forks and variants exist, making standardization difficult
|
|
|
|
### Migration Path
|
|
|
|
If Mozilla TTS integration is desired:
|
|
- Consider using **Coqui TTS** instead as the actively maintained fork
|
|
- Alternatively, use legacy Mozilla TTS models via the archived repository for historical/research purposes
|
|
- Hugging Face hosts pretrained checkpoints that can be used directly
|
|
|
|
---
|
|
|
|
## Technical Details
|
|
|
|
### Architecture
|
|
|
|
#### Text Processing Pipeline
|
|
```
|
|
Text → Grapheme/Phoneme Conversion → Text Encoding → Encoder LSTM/Transformer
|
|
```
|
|
|
|
#### Acoustic Model (Tacotron 2)
|
|
- **Encoder**: LSTM-based sequence encoder with attention
|
|
- **Decoder**: Autoregressive mel-spectrogram decoder
|
|
- **Attention mechanism**: Location-sensitive attention for robust alignment
|
|
- **Post-net**: Residual network to refine mel-spectrograms
|
|
|
|
#### Mel-Spectrogram to Waveform (Vocoder)
|
|
- **HiFi-GAN**: Generative adversarial network producing high-quality waveforms
|
|
- **MelGAN**: Lightweight alternative for faster inference
|
|
- **Glow-TTS**: Fast, non-autoregressive alternative to Tacotron 2
|
|
|
|
### Available Models
|
|
|
|
#### Pretrained Checkpoints
|
|
- **glow-tts**: Fast, non-autoregressive model (recommended for inference)
|
|
- **tacotron2**: Full Tacotron 2 implementation (research/baseline)
|
|
- **glow-tts-bn**: Batch-normalized variant for improved stability
|
|
- **speedy-speech**: Ultra-fast lightweight model
|
|
|
|
#### Language Support
|
|
- English (en-US, en-GB)
|
|
- German (de-de)
|
|
- French (fr-fr)
|
|
- Spanish (es-es)
|
|
- Italian (it-it)
|
|
- Portuguese (pt-pt)
|
|
- Turkish (tr-tr)
|
|
- Russian (ru-ru)
|
|
- Polish (pl-pl)
|
|
- Dutch (nl)
|
|
- And others (varies by model)
|
|
|
|
### Vocoder Options
|
|
|
|
| Vocoder | Quality | Speed | Memory | Notes |
|
|
|---------|---------|-------|--------|-------|
|
|
| **HiFi-GAN** | Excellent | Medium | High | Default, highest quality |
|
|
| **MelGAN** | Good | Fast | Medium | Lightweight alternative |
|
|
| **Univnet** | Excellent | Medium | Medium | Recent addition, good balance |
|
|
| **WaveRNN** | Good | Slow | Low | Legacy, rarely used |
|
|
|
|
### Key Hyperparameters
|
|
|
|
```yaml
|
|
# Audio processing
|
|
sample_rate: 22050 # Hz
|
|
fft_size: 1024
|
|
hop_length: 256
|
|
win_length: 1024
|
|
mel_fmin: 55
|
|
mel_fmax: 7600
|
|
|
|
# Model architecture
|
|
encoder_hidden_size: 384
|
|
encoder_num_layers: 4
|
|
decoder_hidden_size: 384
|
|
attention_hidden_size: 128
|
|
attention_num_heads: 2
|
|
|
|
# Training
|
|
batch_size: 32
|
|
learning_rate: 0.001
|
|
gradient_clip_val: 1.0
|
|
num_epochs: 1000
|
|
```
|
|
|
|
### Supported Input Formats
|
|
|
|
- **Text encodings**: UTF-8
|
|
- **Phoneme sets**: IPA (International Phonetic Alphabet)
|
|
- **Language codes**: ISO 639-1 (en, de, fr, es, etc.)
|
|
- **Phoneme-based input**: Direct phoneme sequences for advanced use cases
|
|
|
|
### Output Formats
|
|
|
|
- **Waveform**: PCM float32, WAV format
|
|
- **Sample rate**: 22.05 kHz (standard)
|
|
- **Bit depth**: 16-bit or 32-bit float
|
|
- **Mono output**: Single-channel audio
|
|
|
|
---
|
|
|
|
## Relationship to Coqui
|
|
|
|
### Historical Context
|
|
|
|
Mozilla TTS was the pioneering open-source neural TTS project, released around 2017-2018. It gained significant traction in the open-source community and served as a reference implementation for modern TTS systems.
|
|
|
|
### The Transition
|
|
|
|
1. **Phase 1 (2018-2021)**: Mozilla maintained active development
|
|
- Regular releases
|
|
- Community contributions
|
|
- Active issue resolution
|
|
|
|
2. **Phase 2 (2021-2023)**: Mozilla reduced maintenance
|
|
- Slower release cycle
|
|
- Focus shifted internally at Mozilla
|
|
- Community took over some maintenance tasks
|
|
|
|
3. **Phase 3 (2022-Present)**: Coqui AI fork and continuation
|
|
- **Coqui TTS** became the primary maintained fork
|
|
- Added features: Streaming TTS, better multilinguality, improved models
|
|
- Active development and community support
|
|
|
|
### Key Improvements in Coqui
|
|
|
|
Coqui TTS builds upon Mozilla TTS with:
|
|
- **Real-time streaming synthesis**
|
|
- **Improved multilingual support** (40+ languages)
|
|
- **Newer model architectures** (Glow-TTS variants, FastSpeech)
|
|
- **Better documentation** and tutorials
|
|
- **Hugging Face integration** for model management
|
|
- **Active maintenance** and bug fixes
|
|
|
|
### Compatibility
|
|
|
|
- Coqui TTS is largely backward compatible with Mozilla TTS models
|
|
- Many Mozilla TTS checkpoints can be used directly in Coqui
|
|
- Vocabulary and phoneme sets are compatible
|
|
- Some API changes exist due to improvements
|
|
|
|
### For uncloseai-speech
|
|
|
|
If integration is desired:
|
|
- **Use Coqui TTS** for new development (actively maintained)
|
|
- **Archive Mozilla TTS** for historical documentation and reference
|
|
- **Maintain compatibility layer** if supporting both ecosystems
|
|
|
|
---
|
|
|
|
## Raccoon Mission Notes
|
|
|
|
### Historical Significance
|
|
|
|
Mozilla TTS represents a milestone in open-source speech synthesis:
|
|
|
|
1. **Pioneer in neural TTS**: One of the first production-quality open-source neural TTS systems
|
|
2. **Community catalyst**: Inspired numerous TTS projects and research implementations
|
|
3. **Research benchmark**: Widely used as a baseline in academic papers and research
|
|
4. **Industry adoption**: Influenced commercial TTS solutions and corporate implementations
|
|
|
|
### Archive Status
|
|
|
|
Mozilla TTS is now primarily an **archived reference** for the following reasons:
|
|
|
|
1. **Superseded by Coqui**: The actively maintained fork provides all features plus improvements
|
|
2. **Historical documentation**: Serves as documentation of TTS architecture evolution
|
|
3. **Reference implementation**: Useful for understanding Tacotron 2 and vocoder concepts
|
|
4. **Research reproducibility**: Original implementation for verifying published results
|
|
|
|
### Why It's Preserved
|
|
|
|
Maintaining documentation of Mozilla TTS supports:
|
|
|
|
- **Educational value**: Learning TTS fundamentals from the original implementation
|
|
- **Research reproducibility**: Ability to reproduce papers using Mozilla TTS
|
|
- **Comparative analysis**: Benchmarking improvements in Coqui and other projects
|
|
- **Architectural understanding**: Reference for modular TTS design patterns
|
|
- **Community history**: Recognition of Mozilla's contributions to open-source speech tech
|
|
|
|
### Current Usage Recommendations
|
|
|
|
For uncloseai-speech:
|
|
|
|
- **New implementations**: Use **Coqui TTS** (actively maintained)
|
|
- **Legacy support**: Keep Mozilla TTS archived for compatibility with existing systems
|
|
- **Research purposes**: Reference Mozilla TTS for understanding baseline architectures
|
|
- **Model evaluation**: Compare Mozilla TTS baseline models with newer approaches
|
|
- **Documentation**: Maintain this archive entry as historical record
|
|
|
|
### Key Milestones
|
|
|
|
| Date | Milestone | Status |
|
|
|------|-----------|--------|
|
|
| 2017-2018 | Initial Mozilla TTS release | Historical |
|
|
| 2019 | Tacotron 2 implementation | Historical |
|
|
| 2020-2021 | HiFi-GAN vocoder integration | Historical |
|
|
| 2021 | Glow-TTS addition | Historical |
|
|
| 2022 | Coqui fork established | Active |
|
|
| 2023-2024 | Mozilla TTS archived | Archived |
|
|
|
|
---
|
|
|
|
## Getting Started (For Reference)
|
|
|
|
### Installation (Legacy)
|
|
|
|
```bash
|
|
# Clone the original Mozilla TTS repository
|
|
git clone https://github.com/mozilla/TTS.git
|
|
cd TTS
|
|
pip install -e .
|
|
```
|
|
|
|
### Basic Usage (Historical Reference)
|
|
|
|
```python
|
|
from TTS.api import TTS
|
|
|
|
# Initialize TTS model
|
|
tts = TTS(model_name="glow-tts", gpu=True)
|
|
|
|
# Synthesize speech
|
|
tts.tts_to_file(
|
|
text="Hello, this is Mozilla TTS.",
|
|
file_path="output.wav"
|
|
)
|
|
```
|
|
|
|
### Alternative: Using Coqui TTS (Recommended)
|
|
|
|
```bash
|
|
# Install Coqui TTS
|
|
pip install TTS
|
|
```
|
|
|
|
```python
|
|
from TTS.api import TTS
|
|
|
|
# Initialize Coqui TTS
|
|
tts = TTS(model_name="tts_models/en/ljspeech/glow-tts", gpu=True)
|
|
|
|
# Synthesize speech
|
|
tts.tts_to_file(
|
|
text="Hello, this is Coqui TTS.",
|
|
file_path="output.wav"
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- **Coqui TTS**: See `/docs/models/coqui-tts.md` for the actively maintained successor
|
|
- **Tacotron 2**: Reference paper and architecture details
|
|
- **HiFi-GAN**: Vocoder architecture documentation
|
|
- **TTS Fundamentals**: General TTS concepts and architectures
|
|
- **Multilingual TTS**: Language support and multilingual synthesis
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
1. **Tacotron 2**: Wang, Y., Skerry-Ryan, R., Stanton, D., et al. (2017). "Natural TTS Synthesis by Conditioning Wavenet on Mel Spectrogram Predictions"
|
|
2. **HiFi-GAN**: Kong, Z., Ping, W., Huang, J., et al. (2020). "HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis"
|
|
3. **Glow-TTS**: Kim, J., Kim, S., Kong, J., et al. (2020). "Glow-TTS: A Generative Flow for Text-to-Speech based on Generative Flow for Raw Audio"
|
|
4. **Mozilla TTS Documentation**: https://tts.readthedocs.io/
|
|
5. **Coqui TTS Repository**: https://github.com/coqui-ai/TTS
|
|
|
|
---
|
|
|
|
*Last Updated: November 2024*
|
|
*Status: Archived Reference*
|
|
*Maintenance: Historical Archive (See Coqui TTS for active development)*
|