pnnbao97 / sea-g2p

🦭 SEA-G2P Fast multilingual text-to-phoneme converter for South East Asian languages. >**Author**: Pham Nguyen Ngoc Bao 🚀 Used By SEA-G2P is the core phonemization engine powering: • **VieNeu-TTS**: An advanced on-device Vietnamese Text-to-Speech model with instant voice cloning. By using SEA-G2P, VieNeu-TTS achieves high-fidelity pronunciation and seamless Vietnamese-English code-switching. Installation Usage Simple Pipeline Individual Modules Features • **Blazing Fast**: Core engine rewritten in Rust with binary mmap lookup. • **Multithreading**: Automatic parallel processing using Rayon/Rust for batch inputs. • **Zero Dependency**: Pre-compiled wheels for Windows, Linux, and macOS. • **Smart Normalization**: Specialized for Vietnamese (numbers, dates, technical terms). • **Bilingual Support**: Handles mixed Vietnamese/English text seamlessly. 📊 Performance The following benchmarks were conducted on a dataset of **1,000,000 sentences**: | Module | Implementation | Throughput | | :--- | :--- | :--- | | **Normalizer** | Rust Core (Parallel) | **~41,000 sentences/s** | | **G2P** | Rust Core (Parallel) | **~415,000 sentences/s** | **Total Pipeline Throughput**: **~37,000 sentences/s** *(Tested on CPython 3.12, Windows 11, Multithreaded)* Technical Architecture SEA-G2P is designed for maximum performance in production environments: • **Memory Mapping (mmap)**: Instead of loading a huge JSON/SQLite into RAM, we use a custom binary format ( ) mapped directly into memory. This allows near-instant startup and extremely low memory overhead. • **String Pooling**: To minimize file size, all unique strings (words and phonemes) are stored once in a global string pool and referenced by 4-byte IDs. • **Binary Search**: Words are pre-sorted during the build process, allowing lookup speeds directly on the memory-mapped data. For full details on the specification, see src/g2p/mod.rs. Development To install for development purposes: • Clone the repository: • Install in editable mode: