feat: Qwen3-TTS — multilingual AR codec-LM as a 2nd on-device TTS family - #408
Merged
Conversation
Net-new text->ids encoder for the Qwen3-TTS text frontend (HfTokenizer is
decode-only). Reproduces the HF tokenizers pipeline: NFC pass-through,
atomic longest-match added-token splitting, the Qwen2 Split regex (Dart
RegExp supports the inline (?i:...) group and \p{L}/\p{N} directly),
GPT-2 byte<->unicode mapping, and rank-ordered BPE merges.
Verified against Task 0's HF-tokenizers golden fixtures: the templated
prompt (16 ids), 8 diverse bpe_cases (contractions, German/French
accents, CJK, whitespace runs, digits, a literal special-token string),
and atomic special-token ids -- all pass without any NFC normalization.
Qwen3Tables loads codec_embedding/mtp_embeddings whole (small) and keeps text_embedding (622 MB) as an open RandomAccessFile with row-lookup via Task 1.1's npyRowF16AsF32, never loading it fully. embedText() looks up each id's row and runs it through the 2048->1024 SiLU projection MLP ported from Qwen3TtsPipeline._project_text/_embed_text.
Tests 2 and 3 fired-and-forgot the load future, so assertions could run after the test body returned once load() gains real async I/O. Match test 1's await pattern.
Move TtsCore's private _loadGraph/_runGraph/_createF32TensorBuffer into a new litert_graph.dart as public loadLiteRtGraph/runLiteRtGraph/ createF32TensorBuffer, generalized to mixed F32/I32 inputs (GraphInput) and a parameterized signature index, so the upcoming Qwen3 talker/codec graphs can reuse them. Matcha call sites now wrap inputs as F32Input and pass signature index 0 — behavior-preserving, verified byte-exact against the shipped Matcha golden PCM on macOS before and after.
TalkerLayout pins the C1-spike-resolved talker graph layout (3 signatures by embeddings time-dim, 59-in/57-or-56-out) and assertLayout re-verifies what the bound LiteRT C API allows via getInputTensorLayout. First real load of the talker graph (talker_int4.tflite) through our Dart FFI.
Loads the talker/mtp/codec LiteRT graphs + host tables + tokenizer, asserts the frozen talker layout, introspects mtpCacheLen/mtpKvShape/ codecChunk, and ports _run_prefill (qwen3_tts_pipeline.py:315-328) to initialize the talker's 56-tensor KV cache from a prompt embedding.
Ports _run_decode (mask + decode signature + logit split); the golden gate reproduces the PyTorch reference's frame-0 cb0 exactly (1342) on talker_fp32. Qwen3TtsCore.load gains an optional talkerFileName param (defaults to int4) so the golden test can load fp32 without disturbing the runtime default.
Ports Qwen3TtsPipeline._run_mtp (qwen3_tts_pipeline.py:342-380): a 16-step inner loop over a 17-slot KV cache that predicts a frame's 15 residual codebooks from the talker's hidden state + codebook-0 token. Introspects the MTP graph's output_0 (residual-head logits) shape at load time via the bulk getOutputTensorLayouts accessor rather than hardcoding it. Gate: runMtp fed the golden frame-0 hidden/cb0 reproduces the PyTorch reference's 15-code residual exactly.
Ports Qwen3TtsPipeline._decode_codes (qwen3_tts_pipeline.py:382-399): sliding-window codec decode with 25-frame left context, transposing [T,16] codes into the codec's [1,16,chunk] int32 input and cropping context samples out of each window's [1,1,chunk*1920] PCM output. Golden gate against frames.json -> waveform_f32.bin: corr=1.0, maxAbsDiff=0.0 on the real fp32 codec graph.
Qwen3TtsCore.synthesize/synthesizePcm16 tie the talker + MTP + codec into the host-orchestrated text->waveform loop (qwen3_tts_pipeline.py :173-266). runMtp gains a sampled residual-picking path. Golden gate (fp32, greedy): corr=1.0, frame agreement=1.0 vs the PyTorch reference.
Adds the qwen3 enum value + its 9-file install manifest (plain basenames matching Qwen3TtsCore.load's artifactPaths keys exactly), extends the TTS size-floor to .npy/.npz, and teaches TtsInstallationBuilder to fetch the 4 table files + demo voice from their tables/voices subdirectory on HF without letting that leak into the installed identity (I5: plain-basename manifest, not nested or double-underscore-flattened — see tts_model_spec.dart doc comments).
…or wiring Review fix (round 1): the prior "flat<->url round trip" test in tts_model_spec_test.dart re-implemented the urlSuffixFor wiring in its own sourceFor closure instead of exercising TtsInstallationBuilder .install() itself, so a revert of the joinUrl change would pass every test while fresh qwen3 network installs 404 on the 5 subdir-hosted files. Adds a builder-level test (mocked ServiceRegistry + a recording DownloadService, mirroring install_identity_namespacing_test.dart's pattern) that calls install() for TtsModelType.qwen3 and asserts the resulting NetworkSource URLs carry the tables/voices subdirectory while every prefsKey stays a plain basename. Verified this test fails if the joinUrl wiring is reverted, then confirmed it passes against the real code. Also adds a small matcha spot-check.
Add TtsPipelineKind.qwen3ArCodec and const TtsModelProfile.qwen3() (subwordTokens + no G2P; pipeline is the only load-bearing field, Qwen3TtsCore hardcodes its own manifest basenames). forType(qwen3) resolves it; supertonic/kokoro still fail-loud. Add fail-loud arms in TtsTextFrontend.load and clarify TtsCore.load's guard message so qwen3ArCodec never silently runs the Matcha pipeline.
_workerEntry now branches on profile.pipeline: matchaCfm keeps the existing TtsTextFrontend+TtsCore path (clause split + CFM seed + inter-clause silence) unchanged in _runMatchaWorker; qwen3ArCodec loads Qwen3TtsCore + the demo x-vector once and serves each request with a single synthesizePcm16 call (no clause-splitting, no CFM seed, no inter-clause silence — Qwen3 is one AR pass over its own KV cache). Load failures surface to the caller, never fall back to Matcha. TtsWorker.spawn/LiteRtSpeechSynthesizer.create gain an optional language param (default 'english') threaded to the qwen3 arm; Matcha ignores it.
LiteRtSpeechSynthesizer.create now validates language against a new public qwen3SupportedLanguages list (languageIds keys + 'auto', single source of truth also used by Qwen3Prompt.build) before spawning the worker, fail-fast ahead of the ~1.9 GB model load. voice is wired through to the worker as a forward-compat x-vector override, defaulting to the bundle's demo voice. RuntimeConfig/createTtsModel/getActiveTts gain an optional language param so the example can actually select one. tts_screen.dart adds a model dropdown (previously hardcoded to matcha) and a language dropdown shown for qwen3; switching either closes the old synthesizer and reinstalls/reactivates with the new selection. Adds the qwen3 catalog entry (isSupported: true, CPU-only/RTF≈3/ 6GB-RAM note).
createTtsModel's same-model reuse branch (mobile + desktop shells) compared only the active model name, so a second getActiveTts() call with a different language silently returned the stale synthesizer — wrong-language audio with no error. Now stores the language the singleton was built with and throws StateError on a mismatch instead of reusing it; unaffected when language is unset (matcha). Also: assertQwen3LanguageSupported validated case-insensitively but Qwen3Prompt.build's 'auto' compare was case-sensitive, so 'Auto'/ 'AUTO' passed create() but threw ArgumentError after the ~1.9 GB model load, at the first synthesize call. LiteRtSpeechSynthesizer. create now normalizes the language once, right after validation, via the new normalizeQwen3Language helper.
Adds qwen3_tts_test.dart: installs Qwen3-TTS from the real public HF URL, synthesizes english, closes, switches to german, and asserts the output actually differs — the on-device install->getModelFilePaths-> Qwen3TtsCore.load path the artifact-gated unit tests can't cover since they build their own artifactPaths. Device run found a real bug: TtsInstallationBuilder's legacy-file adoption blindly renamed an unrelated leftover STT tokenizer.json into the qwen3 slot (its uniqueness check only scans the TTS catalog, not STT/embedding/inference catalogs sharing the same generic basename), corrupting the BPE vocab. Not fixed here (needs a design call); local test-machine state cleaned up to get a real signal on the rest of the pipeline, which now passes: english/german PCM differ, RMS/duration plausible, 24 kHz confirmed both ways. CHANGELOG: qwen3-tts entries in speech (0.5.0) and core (1.5.2).
TtsInstallationBuilder.install() adopted an on-disk plain (pre-1.5.1-namespacing) file whenever its basename was unique within the TTS catalog, with zero content check. Qwen3's tokenizer.json is a cross-catalog-generic basename, so a leftover STT tokenizer.json could be silently adopted -> wrong tokenizer. Adoption at install time is removed; a not-yet-installed manifest file is now always downloaded fresh. The restore-time migration (MobileModelManager._migrateLegacyCompanionForRestore) is untouched -- it stays safe because it operates on a single known active model. Sibling builders (stt_/embedding_/inference_installation_builder.dart) never had this pattern -- no changes needed there. Bump core CHANGELOG header 1.5.2 -> 1.6.0: this branch adds public API (TtsModelType.qwen3 + RuntimeConfig.language + facade language params), and a new enum value can break downstream exhaustive switches.
DenisovAV
force-pushed
the
feat/qwen3-tts
branch
from
August 4, 2026 07:11
3ba79ba to
a6ee64a
Compare
…uards - pin flutter_gemma_speech's core dep to ^1.5.2 (TtsModelType.qwen3 + RuntimeConfig.language need it) - warn when Qwen3 TTS hits maxFrames before EOS (truncated output signal, no behavior change) - throw on a GraphInput payload length mismatch instead of silently zero-padding the tensor - fail loud at load time if codecChunk doesn't exceed the codec decode loop's left-context - test that every wired TTS manifest uses only plain basenames; de-duplicate urlSuffixFor's table set - normalize the TTS language reuse-guard (default + lowercase) so null/'english'/'English' don't spuriously collide
Add a T=135 fp32 golden (frames_long.json/waveform_long_f32.bin) and a codec test gate on it — the existing T=26 golden is under codecChunk (64), so decodeCodes's sliding-window + left-context-carry path was never exercised by any test.
Strip SDD/process scaffolding (Task/Phase refs, docs/superpowers path) from shipped dartdoc across the qwen3 TTS port and touched TTS files, fix two dangling doc-links and an overstated install-note. Vendor the litert-samples Qwen3-TTS recipe (Apache-2.0) under tool/qwen3/ so the ~78 recipe citations resolve; soften them to function-name references since the vendored copy's added provenance line shifts its raw line numbers vs. upstream.
…r leak Qwen3TtsCore.synthesize now throws a StateError when the AR loop hits maxFrames without EOS instead of a gemmaLog-only warning, which is compile-time stripped in release and silently truncated audio. litert_graph.dart's _createTensorBuffer now validates the payload length before any native allocation, so a mismatch no longer leaks the aligned buffer + type struct that used to be allocated ahead of the check.
…3-TTS - CLAUDE.md Current Version: flutter_gemma 1.5.2, flutter_gemma_speech 0.4.1 - website pubspec snippets bumped (speech.md, genkit.md, migration.md) - speech.md: document Qwen3-TTS (2nd selectable TTS family, language: param) - speech.md: mark Whisper/Parakeet STT shipped (were stale 'follow-on' since 0.4.0)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Qwen3-TTS-12Hz-0.6B as a second, selectable on-device TTS family in
flutter_gemma_speech, alongside Matcha — an autoregressive codec-LM (Qwen3 talker + MTP + codec decoder) driven by a host loop over the shared LiteRT FFI.Highlights
cb0, MTP residual codebooks, and codec all match their goldens). Goldens are generated from the reference recipe (tool/qwen3/gen_qwen3_goldens.py), so correctness flows Python → fixtures → Dart, never Dart → Dart.native-v0.14.0unchanged, no new dylib. The talker's KV cache is carried host-side as ordinary in/out tensors (the STT encoder-hidden pattern generalized); a shared graph-helper was extracted fromtts_core.dartand given mixed-dtype (f32/i32) input support.getActiveTts(language:); one bundled demo x-vector (voice selection is a follow-on — the published repo ships a single voice and no speaker encoder). CPU-only in v1 (RTF ≈ 3; ~6 GB-RAM-class device — flagged in the catalog).TtsModelType.qwen3+ a plain-basename install manifest (with HF subdir URL routing) and threadslanguagethrough the facade /RuntimeConfig.Install-path hardening
Removes an unsafe install-time legacy-file adoption in
TtsInstallationBuilder: it adopted an on-disk plain file by basename uniqueness within the TTS catalog only, with no content check.tokenizer.jsonis TTS-unique but a name STT models also ship, so a leftover STT tokenizer could be silently renamed into a Qwen3 install → corrupt/crashing model. This is a pre-existing hole that Qwen3 newly exposes; install-time adoption is dropped (unprovable ownership by bare name), and the documented-safe restore-time migration path is left unchanged.Versions
flutter_gemma1.5.1 → 1.6.0 (additive public API — new enum value +RuntimeConfig.language),flutter_gemma_speech0.4.0 → 0.5.0.pubspec.yamlbumps happen at release.