Add EasyMagpie vLLM-Omni serving#15931
Conversation
376d038 to
df4f340
Compare
Signed-off-by: Viacheslav Klimkov <vklimkov@nvidia.com>
df4f340 to
8f85db5
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Viacheslav Klimkov <vklimkov@nvidia.com>
vklimkov-nvidia
changed the title
|
/ok to test 0734cee |
blisc
left a comment
blisc
left a comment
There was a problem hiding this comment.
Overall the PR looks good from my end. The immediate concerns are
- How we package this component? It looks like it is currently constructed to be completely standalone from the rest of NeMo Speech which can be done.
- On packaging, despite that decision, we need tests to run which means that for testing, the requirements need to be added to NeMo's pyproject.toml and tests need to be moved to the NeMo tests folder, and we need to make sure that they run.
- The current implementation is slightly hard-coded which is fine for initial version, but we need much more strict config validation functions that explain what parameters will break the current implementation, and how we can update the implementation to allow future parameters
| # pipeline: easymagpie_talker | ||
| # | ||
| # scripts/benchmark_model.py uses this deploy config by default. | ||
| pipeline: easymagpie_talker |
There was a problem hiding this comment.
Let's make a better naming convention. Not sure that easymagpie_talker should be used for the pipeline without the codec. @paarthneekhara, what should we refer to the backbone+LT as?
| @@ -0,0 +1,18 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
Unrelated to file, but we need to standard where we keep these implementations. Looks like you are putting it inside of examples, but speechlm2 keeps it inside of collections. Collections is probably where we want to put it since the pip packages ships with collections but not examples.
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| """NeMo-free EasyMagpie spectral codec implementation. |
There was a problem hiding this comment.
Perhaps instead of keeping a nemo-free implementation here, we ought to have a nn.Module base class that both will inherit.
| class _SiluActivation(nn.Module): | ||
| """Module wrapper for ``F.silu``.""" | ||
|
|
||
| def forward(self, x): | ||
| return F.silu(x) |
There was a problem hiding this comment.
Is this a difference between nemo/collections/tts/modules/nemotron_h_decoder.py and automodel? Should we read this from the config instead of auto patching?
| if decoder_config.get("_target_") != expected_target: | ||
| raise ValueError(f"expected {expected_target}, got {decoder_config.get('_target_')}") | ||
| if not decoder_config.get("is_causal"): | ||
| raise ValueError("only the causal spectral codec decoder is supported") |
There was a problem hiding this comment.
I feel like these checks should be more strict.
|
[🤖]: Hi @vklimkov-nvidia 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully. So it might be time to merge this PR or get some approvals. |
8 participants
Summary
POST /v1/audio/speech, and incremental text/audio streaming throughWS /v1/audio/speech/stream.Why
EasyMagpie generates stacked acoustic codes autoregressively and then decodes them into audio. Running both stages in vLLM-Omni provides batched, high-throughput inference while retaining progressive audio output for streaming and voice-agent use cases.
The conversion tool turns the training-time EasyMagpie and causal codec
.nemocheckpoints into a self-contained model directory. It converts the model weights, precomputes the text-embedding lookup, bundles the native codec, saves the tokenizer, and optionally stores speaker embeddings. After conversion, serving requires vLLM/vLLM-Omni and this plugin package, but does not require NeMo at runtime.The native codec keeps its causal history in vLLM-managed state pages, allowing each invocation to process only newly generated acoustic frames and emit asynchronous PCM chunks without replaying the full context.
Validation
pytest -q examples/tts/easymagpie_vllm_omni/tests: 70 passed, 1 skipped.add_special_tokens,NemotronHConfig,ModelLoadConfig, andload_easy_magpie_modelresolve from the Speech checkout.