Add MiniMax Music 3 by apolinario · Pull Request #14456 · huggingface/diffusers

apolinario · 2026-08-12T15:37:05Z

Adds MiniMax Music 3 (lyrics + music description → complete songs up to 5 minutes, 44.1 kHz stereo). An 8B Qwen3 autoregressive stage predicts per-frame audio codes; its hidden states condition a 2.4B flow-matching transformer that produces Flow-VAE latents in overlapping chunks; a DAC-style decoder renders the waveform. Structurally close to Ace-Step (DiT + pipeline-local submodels) and AudioLDM2 (language model as a pipeline component).

New: MiniMaxMusic3Transformer1DModel, MiniMaxMusic3ConditionEncoder (in models/condition_embedders/), MiniMaxMusic3RVQDepthDecoder, MiniMaxMusic3Vocoder, and MiniMaxMusic3ModularPipeline (modular blocks: text encode → AR semantic generation → chunked flow-matching with guider-abstracted CFG → vocode/stitch), plus conversion script, docs, and model- and modular-pipeline tests. The scheduler is the existing FlowMatchEulerDiscreteScheduler(num_train_timesteps=1, shift=1.0, invert_sigmas=True), which reproduces the reference Euler loop exactly.

Usage:

import soundfile as sf
import torch
from diffusers import ModularPipeline

pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-Music3")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

lyrics = """[verse]
Morning light filtering through the pine
Every quiet street is yours and mine
[chorus]
Softly the world begins to breathe"""

prompt = (
    "Genre: acoustic pop. BPM: 96. Key: C major. Warm and intimate, building gently into the chorus. "
    "Vocals: soft female lead, close and breathy, light stacked harmonies in the chorus. "
    "Arrangement: fingerpicked guitar and soft piano; brushed drums and upright bass enter in the chorus."
)

audio = pipe(
    prompt=prompt,
    lyrics=lyrics,
    audio_duration=60.0,
    generator=torch.Generator("cuda").manual_seed(7),
    output="audios",
)[0]

sf.write("song.wav", audio.T.float().cpu().numpy(), pipe.sampling_rate)

Parity vs the reference implementation: every converted component matches bitwise on CPU/fp32; full 30-step chunked generation matches at 57.8 dB SNR on GPU (residual comes from the fused→split QKV projections).

Weights: final home will be MiniMaxAI/MiniMax-Music3 when the model is out

Some choices I made that would be great to double check:

One generator drives the whole pipeline - the reference derives a separate seed per stage and per chunk, imo it's fine to keep the diffusers convention here.
Output is the vocoder's native 44.1 kHz; the reference server post-resamples to 32 kHz - I don't see a reason to downsample something higher quality, but let me know if following the original would be fundamental here
Generation is single-sample (prompt/lyrics are strings; the AR stage is internally batch-2 for CFG). If we want to optimize batched generation I think it may need an AR-loop redesign, which I would defer to a follow-up if someone would be interested?

HuggingFaceDocBuilderDev · 2026-08-12T15:49:50Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

…script

…I/MiniMax-Music-3

sayakpaul

Thanks!

sayakpaul · 2026-08-13T08:05:50Z

+@dataclass
+class MiniMaxMusic3TransformerOutput(BaseOutput):
+    sample: torch.Tensor


Is it not possible to use the Transformer2DModelOutput here?

Done — now returns Transformer2DModelOutput.

sayakpaul · 2026-08-13T08:06:28Z

+        return freqs.cos().contiguous(), freqs.sin().contiguous()
+
+    def forward(self, seq_len: int, device: torch.device) -> Tuple[torch.Tensor, torch.Tensor]:
+        return self._build(seq_len, device)


Consider folding into forward() so that things stay explicitly inline.

Done — folded into forward() with the cache decorator on it directly.

sayakpaul · 2026-08-13T08:07:43Z

+        self.to_k = nn.Linear(dim, self.inner_dim, bias=False)
+        self.to_v = nn.Linear(dim, self.inner_dim, bias=False)
+        self.to_out = nn.ModuleList([nn.Linear(self.inner_dim, dim, bias=False), nn.Dropout(0.0)])
+        self.set_processor(processor or MiniMaxMusic3AttnProcessor())


Suggested change

self.set_processor(processor or MiniMaxMusic3AttnProcessor())

if processor is None:

processor = self._default_processor_cls()

self.set_processor(processor)

Done (also applied to the depth decoder's attention).

sayakpaul · 2026-08-13T08:17:17Z

+        previous_latent = None
+        previous_condition = None
+        global_step = 0
+        with self.progress_bar(total=self._num_timesteps) as progress_bar:


Why not do self.num_inference_steps or put num_inference_steps * len(chunk_starts) in a separate variable and use it here? This feels like an antipattern to me.

Gone with the modular conversion — this file is removed; step accounting now lives in the loop blocks.

Per review: the conditioner moves to models/condition_embedders, the RVQ depth decoder and vocoder to models/transformers and models/autoencoders, and the standard pipeline is replaced by MiniMaxMusic3ModularPipeline (anima-style blocks, helios-style chunk loop, guider-abstracted CFG with a zeros unconditional branch). Index configs now reference all components under the diffusers library. Also applies the transformer review suggestions (Transformer2DModelOutput, inlined rotary forward, explicit processor default).

apolinario · 2026-08-13T08:40:03Z

Converted to a modular pipeline per @yiyixuxu's suggestion. Also applied Sayak's comments.

sergereview

🤗 Serge says:

Overall this is a well-structured modular integration: the block decomposition (text encoder → AR semantic generation → chunk bookkeeping → per-chunk loop with condition/prepare/set-timesteps/denoise/update sub-blocks → vocoder decode) follows the established LoopSequentialPipelineBlocks pattern, the guider is wired the same way as other modular pipelines, and I verified the chunk geometry is self-consistent (200-frame windows at ~3.445 latents/frame → 689-latent windows, ~344.5-latent hop; the 86/258 crop constants tile the song exactly and the [L-344, L-172) carry aligns with the next window's first 172 latents).

Correctness / clarity

The chunking comments in before_denoise.py, denoise.py, and decoders.py all state that neighboring windows "share 172 latent frames", but at ~3.445 latents per frame a 100-frame hop on 200-frame windows gives ~344.5 shared latent frames; 172 is only the blended/carried prefix. The code is right, the comments are wrong — worth fixing since future readers will re-derive this.
MiniMaxMusic3SemanticGenerationStep raises a misleading "the prompt ended generation immediately" error when audio_duration is shorter than one AR frame (< 0.04 s), since max_frames becomes 0.

Style / docs

MiniMaxMusic3ChunkDenoiseInner uses raw tqdm (one new bar per chunk) instead of routing through the pipeline's progress_bar/set_progress_bar_config machinery; helios has the same pattern, but a single bar or config-respecting bar would be nicer for a 14-chunk song.
Minor import-ordering slip in src/diffusers/models/__init__.py and a wrong autodoc path for Transformer2DModelOutput in the transformer docstring.

Description vs. diff

The PR description's usage example (from diffusers import MiniMaxMusic3Pipeline, .audios[0]) does not match the diff: only MiniMaxMusic3ModularPipeline / MiniMaxMusic3Blocks exist; there is no standard MiniMaxMusic3Pipeline. The in-repo docs correctly use ModularPipeline.from_pretrained, but the description should be updated to avoid confusion.

Tests

pretrained_model_name_or_path = "diffusers-internal-dev/tiny-minimax-music3" is flagged as a placeholder (TODO to move to hf-internal-testing/); please make sure the tiny repo exists and the CI tests actually run before merge. Also note the fast tests only ever exercise a single chunk (audio_duration=0.2 → 5 frames), so the multi-chunk overlap/crop path — the trickiest part of the modular loop — has no automated coverage. A small multi-chunk shape test (e.g. enough frames for 2–3 windows with tiny configs) would be valuable.

serge v0.1.0 · model: claude-fable-5 · 18 LLM turns · 33 tool calls · 1708.7s · 1520108 in / 129503 out tokens

…e-level progress bar, zero-frame duration guard, import ordering, docstring cross-reference

…ubmodule calls (minimax_h3 workaround), clear error when the AR models cannot co-reside

apolinario · 2026-08-13T14:18:03Z

dafe373 is a bug-fix for offloading. it fixes ComponentsManager CPU offloading: the AR step calls LM/depth-decoder submodules, which was bypassing the offload hook on the top-level forward. Applied the same manual pre_forward trigger minimax_h3 uses. Measured on A100: auto offload runs a generation in ~21 GB free (peak 17.3 GB); with apply_group_offloading on the language model it fits in 8 GB (peak 5.5 GB).

…g numbers

yiyixuxu

thanks @apolinario i left some small comments

yiyixuxu · 2026-08-13T19:41:01Z

+def _generate_depth_codes(
+    components: MiniMaxMusic3ModularPipeline,


Suggested change

def _generate_depth_codes(

components: MiniMaxMusic3ModularPipeline,

def _generate_depth_codes(

language_model,

rvq_depth_decoder,

it's better to take the individual components as inputs

yiyixuxu · 2026-08-13T19:41:41Z

+    return torch.multinomial(probs.to(sample_device), 1, generator=generator).squeeze(-1).to(probs.device)
+
+
+def _embed_audio_frame(components: MiniMaxMusic3ModularPipeline, frame_codes: torch.Tensor) -> torch.Tensor:


Suggested change

def _embed_audio_frame(components: MiniMaxMusic3ModularPipeline, frame_codes: torch.Tensor) -> torch.Tensor:

def _embed_audio_frame(

language_model: Qwen3ForCausalLM,

rvq_depth_decoder: MiniMaxMusic3RVQDepthDecoder

frame_codes: torch.Tensor) -> torch.Tensor:

yiyixuxu · 2026-08-13T19:42:06Z

+    return torch.stack(codes, dim=1), torch.cat(hidden_parts, dim=-1)
+
+
+class MiniMaxMusic3TextEncoderStep(ModularPipelineBlocks):


ohh it's actually an tokenize step, can we rename and then in the final assembled MiniMaxMusic3Block, merge it into the MiniMaxMusic3SemanticGenerationStep

Take the individual models rather than `components` in `_generate_depth_codes` and `_embed_audio_frame`; `num_codebooks` / `audio_vocab_size` come off the depth decoder's config, so the two pipeline properties that existed only for them are gone. Assemble the blockset as three top-level children, each owning models and runnable on its own — generate the semantics once and denoise it with different settings: semantic_generator MiniMaxMusic3SemanticGenerationStep [tokenize, generate] denoise MiniMaxMusic3CoreDenoiseStep [prepare_chunks, denoise] decode MiniMaxMusic3VocoderDecodeStep `MiniMaxMusic3TextEncoderStep` is renamed `MiniMaxMusic3TokenizeStep` (it only tokenizes) and the autoregressive leaf `MiniMaxMusic3AutoregressiveStep`, freeing its old name for the sequential. Point the tests at `hf-internal-testing/tiny-minimax-music3-modular-pipe`; the previous fixture repo's component specs pointed at a local path, so nothing loaded off that machine. Drop the PR-branch install tip from the docs. 20s and 60s generations are bit-identical to before these changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

yiyixuxu

thanks a lot @apolinario !!

github-actions Bot added size/L PR with diff > 200 LOC documentation Improvements or additions to documentation models tests utils pipelines and removed size/L PR with diff > 200 LOC labels Aug 12, 2026

apolinario force-pushed the minimax-music3-integration branch from 672f410 to 5698322 Compare August 12, 2026 15:41

github-actions Bot added the size/L PR with diff > 200 LOC label Aug 12, 2026

apolinario changed the title ~~Add MiniMax Music 3 pipeline for text-and-lyrics-to-music generation~~ Add MiniMax Music 3 Aug 12, 2026

apolinario added 7 commits August 13, 2026 07:47

Add MiniMax Music 3 transformer, pipeline sub-models, and conversion …

c19f3bc

…script

Add MiniMax Music 3 pipeline, tests, and docs

23b4ece

Wire example docstring, point diffusers-format references at MiniMaxA…

c765905

…I/MiniMax-Music-3

Use MiniMaxAI/MiniMax-Music3 as the official diffusers weights repo id

04262d7

Apply doc-builder style

820a2ef

Use dtype in examples, demonstrate the structured prompt format

4a3c539

Use soundfile in examples like the other audio pipelines

86b2bc2

sayakpaul reviewed Aug 13, 2026

View reviewed changes

apolinario force-pushed the minimax-music3-integration branch from 46564b9 to bcc6bbf Compare August 13, 2026 08:39

github-actions Bot added modular-pipelines and removed pipelines labels Aug 13, 2026

apolinario added 2 commits August 13, 2026 08:50

Point modular index loading specs at the Hub repo in the converter

8231914

Apply doc-builder style to the modular blocks docstrings

7cd51fa

sergereview Bot reviewed Aug 13, 2026

View reviewed changes

apolinario added 2 commits August 13, 2026 09:48

Address review-bot comments: overlap comment accuracy, single pipelin…

1a258e3

…e-level progress bar, zero-frame duration guard, import ordering, docstring cross-reference

Support CPU offloading in the AR step: trigger offload hooks before s…

dafe373

…ubmodule calls (minimax_h3 workaround), clear error when the AR models cannot co-reside

Docs: PR-install tip and memory-usage section with measured offloadin…

c6da993

…g numbers

yiyixuxu reviewed Aug 13, 2026

View reviewed changes

yiyixuxu mentioned this pull request Aug 13, 2026

Compare full pipeline outputs in the modular tester mixin #14468

Open

yiyixuxu approved these changes Aug 13, 2026

View reviewed changes

yiyixuxu merged commit 2da7040 into main Aug 13, 2026
19 of 20 checks passed

yiyixuxu deleted the minimax-music3-integration branch August 13, 2026 23:48

-        self.set_processor(processor or MiniMaxMusic3AttnProcessor())
+        if processor is None:
+            processor = self._default_processor_cls()
+        self.set_processor(processor)

		def _generate_depth_codes(
		components: MiniMaxMusic3ModularPipeline,

		return torch.multinomial(probs.to(sample_device), 1, generator=generator).squeeze(-1).to(probs.device)


		def _embed_audio_frame(components: MiniMaxMusic3ModularPipeline, frame_codes: torch.Tensor) -> torch.Tensor:

		return torch.stack(codes, dim=1), torch.cat(hidden_parts, dim=-1)


		class MiniMaxMusic3TextEncoderStep(ModularPipelineBlocks):

Uh oh!

Conversation

apolinario commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Aug 12, 2026

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apolinario commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergereview Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

apolinario commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

apolinario commented Aug 12, 2026 •

edited

Loading

apolinario commented Aug 13, 2026 •

edited

Loading

apolinario commented Aug 13, 2026 •

edited

Loading