Add VibeVoice streaming ASR export - #723
Open
justinchuby wants to merge 8 commits into
Open
Conversation
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
2 times, most recently
from
September 3, 2026 22:57
87138b0 to
e6c4950
Compare
Implement an architecture-discriminated, staged ONNX export for the pinned VibeVoice ASR checkpoint, with explicit audio convolution and decoder cache contracts. Preserve the existing TTS path while recording unsupported ORT GenAI orchestration boundaries and validating source parity, processor behavior, and complete checkpoint routing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Apply the repository import and expression formatting rules after restacking the streaming ASR export on the finalized VibeVoice TTS base. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Record the streaming ASR L4/L5 runner limitation and mark its deliberate checkpoint-name translation as incompatible with the generic identity fixture. The pinned source-index audit remains the exact weight-routing proof. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Install the pinned VibeVoice executable source and its compatible Transformers release in a dedicated L3 job. Verify the installed source commit through pip direct-url metadata so source parity cannot silently skip outside a checkout. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
from
September 4, 2026 16:00
e6c4950 to
bd6a812
Compare
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
from
September 4, 2026 16:00
e6c4950 to
bd6a812
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A few robustness/compatibility issues in the new exception handling import path and pinned-source verification logic should be fixed to avoid avoidable CI/runtime failures.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds first-class support for exporting VibeVoice streaming ASR (microsoft/VibeVoice-ASR-Streaming-7B) as a three-stage ONNX pipeline (audio encoder + embedding + decoder), integrated into the existing Mobius model registry/build flow and onnx-genai export tooling.
Changes:
- Introduces
VibeVoiceASRConfig,VibeVoiceASRForConditionalGeneration, andVibeVoiceASRStreamingTaskto export the staged streaming-ASR pipeline with explicit KV + convolution cache I/O. - Extends Transformers config/module resolution to discriminate VibeVoice TTS vs ASR by
architectures, and updates model/test coverage registration for the new architecture key. - Adds targeted tests (including pinned source-parity CI workflow) and onnx-genai advisory export support for the ASR topology.
File summaries
| File | Description |
|---|---|
| tests/weight_alignment_test.py | Adds ASR checkpoint-index routing audit and marks ASR architecture as filtering preprocess. |
| tests/model_coverage_test.py | Documents why generic L4/L5 runner can’t drive this streaming ASR pipeline yet. |
| tests/build_graph/speech_test.py | Declares expected staged outputs for vibevoice-asr-streaming. |
| tests/arch_validation_test.py | Extends HF config loading/unwrapping to handle ASR’s nested decoder config. |
| tests/_test_configs.py | Adds tiny pinned config entry for VibeVoiceForASRStreamingTraining. |
| src/mobius/tasks/_vibevoice.py | Refactors convolution-cache helpers into shared utility module. |
| src/mobius/tasks/_vibevoice_asr.py | New staged task defining ASR graph contracts and explicit cache I/O. |
| src/mobius/tasks/_streaming_convolution.py | New helper module for causal convolution cache inputs/outputs. |
| src/mobius/tasks/init.py | Registers vibevoice-asr-streaming task and exports it. |
| src/mobius/models/vibevoice.py | Adds ASR provenance constants, final-chunk stride padding, and ASR model modules + weight routing. |
| src/mobius/models/vibevoice_asr_test.py | New comprehensive contract/routing and pinned source-parity tests. |
| src/mobius/models/init.py | Exports VibeVoiceASRForConditionalGeneration. |
| src/mobius/integrations/transformers/_builder.py | Adds architecture-discriminated VibeVoice dispatch and pins ASR revision. |
| src/mobius/integrations/onnx_genai/auto_export.py | Detects ASR 3-stage topology and emits advisory metadata + assets. |
| src/mobius/integrations/onnx_genai/auto_export_test.py | Adds coverage for ASR advisory export path and revision forwarding. |
| src/mobius/_registry.py | Registers VibeVoiceForASRStreamingTraining with pinned model+revision. |
| src/mobius/_configs/vibevoice.py | Expands config extraction to include ASR tokenizer constraints + VibeVoiceASRConfig. |
| src/mobius/_configs/init.py | Exports VibeVoiceASRConfig. |
| requirements/ci/vibevoice-asr.txt | Pins the exact upstream VibeVoice ASR source for synthetic parity tests. |
| docs/model-catalog.md | Lists the new ASR architecture in the catalog. |
| .github/workflows/main.yml | Adds dedicated CI job to run ASR pinned source-parity integration tests. |
Review details
- Files reviewed: 21/21 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Generalize the pinned ASR checkpoint set, validate both official variants, and preserve explicit tied LM-head tensors during checkpoint routing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Force-install the Transformers 4.51 reference dependency versions that the source-parity job requires, preventing the public Transformers 5 environment from supplying incompatible Hugging Face Hub and tokenizers releases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Make config fallback compatible with older Hub releases and validate the pinned reference environment before importing its modules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Use the composite ASR configuration dtype for the tokenizer and connector stages instead of the nested Qwen checkpoint storage dtype. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation
VibeVoiceForASRStreamingTrainingsupport for both official Microsoft streaming-ASR checkpoints, without changing the existing VibeVoice TTS route:microsoft/VibeVoice-ASR-Streaming-1.5B@4262d23d8a539a6530cf64fbd0b1751ef9a30853microsoft/VibeVoice-ASR-Streaming-7B@60d858b518b4e19d404af3737f848fc185b30177audio_encoder(acoustic and semantic causal tokenizers plus connectors),embedding(speech embedding replacement), anddecoder(Qwen2 with explicit KV caches).<speech_start>,<speech_end>, forced<|text_chunk_end|>), chunk loop/sampling, hotword prompt assembly, and speaker JSON parsing remain host-owned.mobius.attention.requires_arbitrary_maskcontract, preventing incorrect GQA fusion for left-padded ASR batches.acoustic_tokenizer.decoder.*training-VAE tensors that are not on the ASR inference path. The 1.5B tied embedding/head checkpoint preserves its explicitlm_head.weight.Provenance and runtime boundary
microsoft/VibeVoice@505653d3873b065a488aea551c6ee3dc51d3062fwithtransformers==4.51.3; ASR-critical blobs were checked unchanged through upstream1541f590c7099820f10ea012f48d2399282df69f.huggingface-hub==0.31.4,tokenizers==0.21.4) after the public Transformers 5 environment.onnxruntime-ep-mlx 0.29.5/ ORT 1.29 cannot execute the complete VibeVoice decoder due toCumSum cannot infer output shapes; this blocks local MLX real-runtime evidence.Validation
4730 passed, 56 skipped, 1 subtests passed.lintrunner f --output oneline --all-filesand diff checks.