livekit-wakeword uses a hybrid ONNX + PyTorch architecture. Two frozen ONNX models handle feature extraction (mel spectrogram and speech embeddings), while a lightweight PyTorch classifier head is trained per wake word.
Training Pipeline: Synthetic speech is generated via VITS TTS with SLERP speaker blending, augmented with noise/reverb, then passed through frozen ONNX models (mel spectrogram → speech embedding) to produce .npy feature files. These features train a lightweight classifier head, which is saved as a .pt model.
Inference Pipeline: Raw 16kHz audio flows through the same frozen ONNX feature extractors (32-band mel spectrogram → 96-dim speech embedding), then through the trained classifier (exported to ONNX) to produce a detection score between 0 and 1.
The mel-spectrogram and speech-embedding models are pre-trained and frozen — they never change between wake words. Running them through ONNX Runtime provides:
- Fast numpy-based inference without loading PyTorch at detection time
- Shared feature extractors across all wake words
- Minimal training data needed since only the small classifier head is trained
The classifier head is trained in PyTorch for flexibility (focal loss, embedding mixup, learning rate schedules), then exported to ONNX for deployment.
| Stage | Output Shape | Description |
|---|---|---|
| Raw audio | (samples,) |
16kHz float32 mono |
| MelSpectrogramFrontend | (batch, time_frames, 32) |
32 mel bands, ~5 frames per 80ms |
| SpeechEmbedding | (batch, n_windows, 96) |
76-frame sliding window, stride 8 |
| Timestep selection | (batch, 16, 96) |
Last 16 windows (or left-padded) |
| Classifier | (batch, 1) |
Sigmoid confidence score |
| Parameter | Value |
|---|---|
| Sample rate | 16000 Hz |
| FFT size | 512 |
| Hop length | 160 (10ms) |
| Window length | 400 (25ms) |
| Mel bands | 32 |
| Frequency range | 60–3800 Hz |
| Centering | Disabled |
| Normalization | power_to_db / 10 + 2 |
Google's speech_embedding CNN (~330k parameters), originally from TensorFlow Hub. Input: (batch, 76, 32, 1) channels-last mel windows. Output: (batch, 96) embedding vectors.
The embedding model uses a 5-block CNN with separable convolutions (1x3 + 3x1), BatchNorm, LeakyReLU, and MaxPool.
src/livekit/wakeword/
├── config.py Pydantic config models + YAML loading
├── cli.py Typer CLI (setup, generate, augment, train, export, eval, run)
├── models/
│ ├── feature_extractor.py MelSpectrogramFrontend + SpeechEmbedding (ONNX)
│ ├── classifier.py DNNClassifier, RNNClassifier, ConvAttentionClassifier
│ └── pipeline.py WakeWordClassifier (training wrapper for classifier head)
├── data/
│ ├── generate.py TTS orchestration (default Piper VITS) + adversarial negatives
│ ├── piper/ Piper VITS + SLERP synthesis stack
│ ├── tts/ Pluggable SpeechSynthesizer backends
│ ├── augment.py AudioAugmentor + clip alignment
│ ├── dataset.py WakeWordDataset (memory-mapped .npy batch generator)
│ └── features.py ONNX feature extraction → .npy files
├── training/
│ ├── trainer.py WakeWordTrainer (3-phase training + checkpoint averaging)
│ └── metrics.py FPPH, recall, balanced accuracy
├── eval/
│ └── evaluate.py DET curve, AUT, FPPH evaluation + plotting
├── export/
│ └── onnx.py ONNX export + INT8 quantization
└── inference/
├── model.py WakeWordModel class (simple prediction API)
└── listener.py WakeWordListener class (async microphone detection)
- Data Generation — Synthesize positive clips via VITS TTS with SLERP speaker blending + adversarial negatives + background noise clips
- Augmentation — Apply EQ, distortion, RIR convolution, background mixing to all splits; align clips to training windows
- Feature Extraction — Extract mel spectrograms and speech embeddings through frozen ONNX models →
.npyfiles - Training — 3-phase adaptive training with focal loss, embedding mixup, and checkpoint averaging
- Export & Inference — Export classifier to ONNX and run real-time streaming detection
- Evaluation — DET curves, AUT, FPPH, and model comparison