Skip to content

Implement Multiturn support for EasyMagpie#15764

Open
Edresson wants to merge 123 commits into
NVIDIA-NeMo:mainfrom
Edresson:nemotron_tts_multiturn_user_audio_latest
Open

Implement Multiturn support for EasyMagpie#15764
Edresson wants to merge 123 commits into
NVIDIA-NeMo:mainfrom
Edresson:nemotron_tts_multiturn_user_audio_latest

Conversation

@Edresson

@Edresson Edresson commented Jun 8, 2026

Copy link
Copy Markdown
Member

What does this PR do ?

Implements multiturn support for EasyMagpieTTS, including multiturn Lhotse training/evaluation data handling, user-audio-conditioned inference, multiturn metrics/export support, and related tests.

Collection: TTS

Changelog

  • Added a new EasyMagpie multiturn Lhotse config: examples/tts/conf/magpietts/easy_magpietts_lhotse_multiturn.yaml.
  • Added support for restoring EasyMagpie models from a pretrained checkpoint before normal initialization.
  • Added multiturn Lhotse TTS dataset support for duplex / multi-turn conversational samples.
  • Added EasyMagpie multiturn user-audio inference mode via --easy_magpie_inference_mode multiturn_user_audio.
  • Added support for user-audio conditioning, user silence handling, user turn end-token handling, and max evaluation turns during multiturn inference.
  • Added distributed / multi-GPU inference support for multiturn EasyMagpie evaluation by sharding generation across ranks and merging rank-local outputs for evaluation.
  • Added grouped turn-level and sample-level metrics export for multiturn evaluation, including JSON/CSV outputs.
  • Added phoneme prediction export fields to generated .json / .csv evaluation outputs.
  • Added optional predicted-phoneme inference controls, including phoneme input type, phoneme sampling method, tokenizer override, and text-input dropout.
  • Added additional evaluation options for ASR / EOU batch sizes, emotion cosine similarity, emotion match rate, and text-annotation stripping for metrics.
  • Added Lhotse dataloader speaker filtering support.
  • Added / updated Nemotron-H decoder support needed for prefill in EasyMagpie multiturn inference.
  • Added unit tests for Lhotse TTS filters, MagpieTTS Lhotse dataset behavior, and EasyMagpieTTS model behavior.

Usage

Example EasyMagpie multiturn inference with a .nemo checkpoint:

# Run from the NeMo repository root.
# Replace paths with your local model, codec, evalset config, and output directory.

python examples/tts/magpietts_inference.py \
  --model_type easy_magpie \
  --easy_magpie_inference_mode multiturn_user_audio \
  --nemo_files /path/to/easy_magpie_model.nemo \
  --datasets_json_path /path/to/evalset_config.json \
  --out_dir /path/to/output \
  --codecmodel_path /path/to/codec_model.nemo \
  --batch_size 1 \
  --max_eval_turns 6 \
  --run_evaluation

@copy-pr-bot

copy-pr-bot Bot commented Jun 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Comment thread examples/tts/easy_magpietts_inference_multiturn.py Fixed
Comment thread examples/tts/easy_magpietts_inference_multiturn.py Fixed
Comment thread examples/tts/easy_magpietts_inference_multiturn.py Fixed
Comment thread nemo/collections/tts/models/easy_magpietts_inference.py Fixed
Comment thread nemo/collections/tts/models/easy_magpietts.py Fixed
Comment thread nemo/collections/common/data/lhotse/cutset.py Fixed
Comment thread examples/tts/easy_magpietts_inference_multiturn_multigpu.py Fixed
Comment thread examples/tts/easy_magpietts_inference_multiturn.py Fixed
Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed
Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed
Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed
@Edresson
Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 2600fb8 to 9cb8c01 Compare June 8, 2026 19:08
Comment thread examples/tts/magpietts_inference.py Fixed
Comment thread examples/tts/magpietts_inference.py Fixed
Comment thread examples/tts/magpietts_inference.py Fixed
@Edresson
Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 9cb8c01 to a73132d Compare June 8, 2026 19:44
@github-actions github-actions Bot removed the common label Jun 9, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, these changes was done to support prefill in one step on nemotron_h class.

Comment thread nemo/collections/tts/metrics/emotion_encoder.py Fixed
Comment thread examples/tts/magpietts_inference.py Outdated
@Edresson
Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch 2 times, most recently from c564930 to 1a43737 Compare June 23, 2026 15:36
Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed
Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed
Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed
@Edresson
Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch 3 times, most recently from ba6c288 to 5978897 Compare June 23, 2026 19:06
@Edresson

Copy link
Copy Markdown
Member Author

/ok to test da88283

@Edresson
Edresson marked this pull request as ready for review June 23, 2026 21:14
@Edresson
Edresson requested a review from a team as a code owner June 23, 2026 21:14
@Edresson
Edresson requested review from blisc and paarthneekhara June 23, 2026 21:15
Edresson and others added 28 commits July 25, 2026 17:47
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>
Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>
Signed-off-by: Shehzeen Hussain <shehzeensh@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Shehzeen Hussain <shehzeensh@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson Casanova <edresson1@gmail.com>
@shehzeen
shehzeen force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 3276aa3 to 4156fc2 Compare July 26, 2026 00:54
@shehzeen

Copy link
Copy Markdown
Collaborator

/ok to test 4156fc2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants