feat: LiteRT engine consolidation + on-device STT (flutter_gemma_speech, moonshine)#389
Open
DenisovAV wants to merge 20 commits into
Open
feat: LiteRT engine consolidation + on-device STT (flutter_gemma_speech, moonshine)#389DenisovAV wants to merge 20 commits into
DenisovAV wants to merge 20 commits into
Conversation
…FFI + engine web JS
Expose LiteRtBindings via a public native-only litertlm/lib/litert_bindings.dart (avoids an implementation_imports lint on embeddings' native-only file), plus trivial prefer_initializing_formals / unnecessary_library_name fixes so flutter_gemma_embeddings analyzes clean again.
…tch + active-model persistence
…ate worker + tokenizer); moonshine profile
…r-path leak, NaN decode diagnostic, hiddenBufPtr leak, unlock status checks
… moonshine pipeline
…core to 1.4.0 (+CHANGELOG); speech requires flutter_gemma ^1.4.0
…repo.ModelType.stt (was silently bucketed into embedding in getInstalledModels/isAnyModelInstalled)
…dow (unconditional in-progress completer guard, mirrors desktop)
…r awaits in stt_screen recording paths
… the consolidated single-hook engine + wired moonshine STT
…port-leak guard, drop dead crypto dep, fix path-import shadow, RuntimeConfig.tokenizerPath doc, stt_screen reentrancy guards
…twork install + bundled clip via public API)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates the shared LiteRT runtime into
flutter_gemma_litertlm(now the full LiteRT engine) and adds on-device Speech-to-Text via a newflutter_gemma_speechpackage. First STT model: moonshine-tiny (raw PCM, no mel frontend). The model is selectable like inference models (SttModelTypeon the spec) — one generic backend + recognizer driven by a per-model profile.Phase 1 — engine consolidation + core STT contracts
flutter_gemma_litertlmabsorbs the LiteRt interpreter FFI (LiteRtBindings, moved in from embeddings) and the weblitert.js/tensorflow.jsruntime (moved in from core) → it is now the sole owner oflibLiteRtLm; the stale-owner coordination marker is retired.flutter_gemma_embeddingsbecomes thin logic depending onlitertlm(drops its own hook + FFI + engine web JS). Public API (LiteRtEmbeddingBackend) unchanged.flutter_gemma(core) loses the engine web JS → truly engine-free.ModelManagementType.stt,SttModelSpec+SttModelType,SttBackendProvider/SttRegistry,SpeechRecognizer,FlutterGemma.initialize(sttBackends:),installStt()/getActiveStt()+createSttModeldispatch (mobile/web/desktop) + active-model persistence.Phase 2 —
flutter_gemma_speech(STT, moonshine)litertlm(sharedlibLiteRtLm, no own hook).LiteRtSttBackend(sole.tfliteSTT backend;canHandle => true; returns the coreSpeechRecognizerinterface). Concrete pipeline (LiteRtSpeechRecognizer,SttModelProfile,SttWorker,SttCore,HfTokenizer) is internal tolib/src/, not exported.SttModelProfile.forType(sttModelType). moonshine profile filled; whisper/parakeet (log-mel) arms throwUnimplementedError(follow-on).Validation (on-device, macOS)
desktop_embedding_testgreen — embeddings still produce a 768-dim vector through the moved FFI onlibLiteRtLm0.14.0.installStt→getActiveStt→transcribe) transcribes a known clip →"Ever tried ever failed, no matter."Follow-ups (not in this PR)
libLiteRtLmafter a version bump).