feat: add pocket backend (Kyutai pocket-tts, pure Rust CPU) and make it the default#68
Merged
Merged
Conversation
…it the default Integrate the pocket-tts crate (Rust/candle port of Kyutai's 100M-parameter FlowLM + Mimi TTS) as a new cross-platform backend: - Warm model cache in a global Mutex (daemon-compatible, listed as heavy) - 8 predefined voices via public HuggingFace embeddings, zero setup - Voice cloning from a reference WAV when HF_TOKEN is set (gated kyutai/pocket-tts weights); clear error message otherwise - Vendored model config: the crate resolves configs through a compile-time cargo registry path that does not exist on end-user machines, so the YAML is staged under the vox config dir and loaded via an absolute-path variant - pocket becomes DEFAULT_BACKEND on all platforms (was say/piper) Verified end-to-end on Linux (i7-1065G7, CPU only): English and French generation, alternate voices, list-voices, default dispatch. Generation is faster than real-time on CPU; 5-14s end-to-end including model load and full audio playback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The b6369a24 checkpoint bundled with the pocket-tts crate is the English model; all 8 predefined voices are English speakers. Kyutai's per-language checkpoints (french_24l etc.) require config fields the crate does not implement yet (insert_bos_before_voice). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…wise) The pocket checkpoint is English-only, so route non-English languages to piper (per-language voices) when no backend is explicitly selected via CLI flag or stored preference. Applies to both the CLI and the MCP server. Also fix the vox init templates: they hardcoded 'vox -b say', which only exists on macOS — use 'vox -l fr' so the generated CLAUDE.md and Stop hook work on all platforms. Verified live: 'vox -l fr' routes to piper (proper French voice), 'vox' with no lang routes to pocket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Build aarch64-unknown-linux-gnu natively on ubuntu-24.04-arm runners - Package .deb (cargo-deb) and .rpm (cargo-generate-rpm) for both Linux architectures; verified locally with dpkg-deb - Homebrew formula gains a Linux ARM block - install.sh accepts aarch64 Linux - Fix the Linux checksum grep to not match the -cuda archive Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # Cargo.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new
pocketbackend wrapping the pocket-tts crate — the Rust/candle port of Kyutai's Pocket TTS (100M-parameter FlowLM + Mimi codec, 24 kHz, CPU-first) — and makes it the default backend on all platforms (wassayon macOS,piperelsewhere).Why
piper(fast but robotic) andqwen-native(excellent but heavy)alba,marius,javert,jean,fantine,cosette,eponine,azelma)generate_streamAPI available in the crate → natural candidate for the streaming TTS phase of the conversational roadmapLanguage support (important caveat)
The
b6369a24checkpoint bundled with the crate is English-only, and all 8 predefined voices are English speakers. French text synthesizes intelligibly but with a strong English accent. Kyutai publishes per-language checkpoints (french_24l,german_24l, …) with dedicated voices (e.g.estellefor French), but they use config fields the Rust crate does not implement yet (insert_bos_before_voice,remove_semicolons) — proper French needs an upstream contribution to the crate first.Implementation notes
src/backend/pocket_config_b6369a24.yaml): the crate resolves its config through a compile-timeCARGO_MANIFEST_DIRregistry path that does not exist on end-user machines. We stage the YAML under the vox config dir and pass its absolute path as the variant (Path::joinwith an absolute path replaces the base — this is what makes the override work).HF_TOKENset → gatedkyutai/pocket-ttsweights (WAV voice cloning enabled). No token → publickyutai/pocket-tts-without-voice-cloningweights (predefined voices only), with a clear error if a WAV clone is requested.Mutex(same pattern asqwen-native), registered in the daemon heavy-backend list and the clone-capable auto-switch list.Verified end-to-end (Linux, i7-1065G7, CPU only, release build)
-v fantine)--list-voices-bflag)tests/pocket_test.rs)Generation runs faster than real-time on this laptop CPU; most of the wall time is audio playback plus ~2s model load (daemon keeps it warm).
Follow-ups (not in this PR)
insert_bos_before_voice+ per-language configs to the pocket-tts crate, then wire-l fr→french_24l+estellegenerate_stream(speak while generating)metalfeature forwarding + Metal device selection for Apple Silicon (~2x)🤖 Generated with Claude Code