diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index 8466cebe2a8c..2d0bf5707ad9 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -375,6 +375,8 @@ title: Lumina2Transformer2DModel - local: api/models/lumina_nextdit2d title: LuminaNextDiT2DModel + - local: api/models/minimax_h3_transformer3d + title: MiniMaxH3Transformer3DModel - local: api/models/mochi_transformer3d title: MochiTransformer3DModel - local: api/models/motif_video_transformer_3d @@ -459,6 +461,10 @@ title: AutoencoderKLLTXVideo - local: api/models/autoencoderkl_magvit title: AutoencoderKLMagvit + - local: api/models/autoencoderkl_minimax_h3 + title: AutoencoderKLMiniMaxH3 + - local: api/models/autoencoderkl_minimax_h3_audio + title: AutoencoderKLMiniMaxH3Audio - local: api/models/autoencoderkl_mochi title: AutoencoderKLMochi - local: api/models/autoencoderkl_qwenimage @@ -691,6 +697,8 @@ title: LTX-2 - local: api/pipelines/ltx_video title: LTXVideo + - local: api/pipelines/minimax_h3 + title: MiniMax-H3 - local: api/pipelines/mochi title: Mochi - local: api/pipelines/motif_video @@ -770,6 +778,8 @@ title: LCMScheduler - local: api/schedulers/lms_discrete title: LMSDiscreteScheduler + - local: api/schedulers/minimax_h3 + title: MiniMaxH3Scheduler - local: api/schedulers/pndm title: PNDMScheduler - local: api/schedulers/repaint diff --git a/docs/source/en/api/models/autoencoderkl_minimax_h3.md b/docs/source/en/api/models/autoencoderkl_minimax_h3.md new file mode 100644 index 000000000000..2fc7f4a07a69 --- /dev/null +++ b/docs/source/en/api/models/autoencoderkl_minimax_h3.md @@ -0,0 +1,38 @@ + + +# AutoencoderKLMiniMaxH3 + +The video variational autoencoder (VAE) model with KL loss used in [MiniMax-H3](https://huggingface.co/MiniMaxAI) by MiniMax. It pairs a causal 3D CNN encoder with a non-causal ViT decoder and compresses 16x spatially and 4x temporally. + +Three things set it apart from most autoencoders in the library: + +- **Latents are normalized per channel.** There is no `scaling_factor`: a pipeline encodes with `(latent - latents_mean) / latents_std` and decodes with `latent * latents_std + latents_mean`. +- **The pixel convention is ImageNet-normalized RGB over a `[0, 1]` base range**, not the usual `[-1, 1]`. `encode` expects `(pixel - imagenet_mean) / imagenet_std` and `decode` returns values in that same space, so a pipeline applies `sample * imagenet_std + imagenet_mean` and clamps to `[0, 1]` before postprocessing. +- **Spatial tiling is on by default.** MiniMax-H3 was released with tiling enabled for both encoding and decoding and the released frames are the blended-tile ones, so turning it off changes the output. Use `enable_tiling` to change the tile geometry and `disable_tiling` to switch it off. + +The temporal geometry is fixed by `clip_length` (17 pixel frames per encoder chunk) and `token_drop` (3 trailing latent frames dropped per encode), so `17 * n + 5` pixel frames map to `5 * n + 2` latent frames. + +```python +import torch +from diffusers import AutoencoderKLMiniMaxH3 + +vae = AutoencoderKLMiniMaxH3.from_pretrained( + "MiniMaxAI/MiniMax-H3", subfolder="vae", dtype=torch.float32 +).to("cuda") +``` + +## AutoencoderKLMiniMaxH3 + +[[autodoc]] AutoencoderKLMiniMaxH3 + - encode + - decode + - all diff --git a/docs/source/en/api/models/autoencoderkl_minimax_h3_audio.md b/docs/source/en/api/models/autoencoderkl_minimax_h3_audio.md new file mode 100644 index 000000000000..ab78da3f5e32 --- /dev/null +++ b/docs/source/en/api/models/autoencoderkl_minimax_h3_audio.md @@ -0,0 +1,36 @@ + + +# AutoencoderKLMiniMaxH3Audio + +The audio autoencoder used in [MiniMax-H3](https://huggingface.co/MiniMaxAI) by MiniMax. It is waveform in and waveform out, with no mel front-end and no separate vocoder: a DAC-lineage strided convolutional encoder, a causal-attention projection onto the diffusion latent width, and a BigVGAN decoder. + +The encoder hops 800 samples at 32 kHz, i.e. 40 latents per second, so a waveform of `800 * n` samples encodes to `n` latents. Waveforms that are not a whole number of hops are right-padded. + +The causal-attention projection goes through the attention dispatcher, so `set_attention_backend` applies to it; its mask is `is_causal=True`, which every backend honours except `_native_npu`, whose kernel takes no causal flag. + +The autoencoder is **mono**, and it normalizes latents per channel with `latents_mean` / `latents_std` rather than a scalar `scaling_factor`. MiniMax-H3 carries stereo as two *batch* items, and it always consumes the posterior mean (`latent_dist.mode()`), never a sample. + +```python +import torch +from diffusers import AutoencoderKLMiniMaxH3Audio + +audio_vae = AutoencoderKLMiniMaxH3Audio.from_pretrained( + "MiniMaxAI/MiniMax-H3", subfolder="audio_vae", dtype=torch.float32 +).to("cuda") +``` + +## AutoencoderKLMiniMaxH3Audio + +[[autodoc]] AutoencoderKLMiniMaxH3Audio + - encode + - decode + - all diff --git a/docs/source/en/api/models/minimax_h3_transformer3d.md b/docs/source/en/api/models/minimax_h3_transformer3d.md new file mode 100644 index 000000000000..b027a6ed0f0c --- /dev/null +++ b/docs/source/en/api/models/minimax_h3_transformer3d.md @@ -0,0 +1,41 @@ + + +# MiniMaxH3Transformer3DModel + +A Diffusion Transformer model for joint video and audio generation, introduced in [MiniMax-H3](https://huggingface.co/MiniMaxAI) by MiniMax. + +MiniMax-H3 runs a single stack of blocks over **one packed 1-D sequence** that holds the text conditioning, the conditioning image and video rows, the audio rows and the target video rows at once. Attention is full self-attention over that sequence, so there is no cross-attention and no per-modality block weights. Modality-specific behaviour comes only from the two input patch projections, the per-row modality tag that selects the AdaLN modulation parameters, and the two output heads. + +Building the packed layout is the caller's job, which is why the forward signature takes the layout apart from the latents: the `(t, h, w)` position grid, the per-row modality tags, the per-row timestep indices and the three index tensors that address the video, audio and text rows. [`MiniMaxH3Blocks`] and [`MiniMaxH3Ref2VABlocks`] build all of it. + +A layout that carries padding rows (tag `-1`) needs a masked attention backend, since those rows are kept in their own attention document by a boolean mask; a padless sequence needs no mask and keeps every backend available. + +One repository holds both released checkpoint partitions, so the subfolder is what selects the task: `transformer/` for the text and keyframe tasks, `transformer_ref/` for the omni-reference task. + +```python +import torch +from diffusers import MiniMaxH3Transformer3DModel + +transformer = MiniMaxH3Transformer3DModel.from_pretrained( + "MiniMaxAI/MiniMax-H3", subfolder="transformer", dtype=torch.bfloat16 +).to("cuda") +``` + +The checkpoint is mixed precision: the two input patch projections, the timestep MLP and the two output heads are float32 while the block stack is bfloat16. `from_pretrained` keeps that layout through `_keep_in_fp32_modules`, so pass `dtype=torch.bfloat16` and let it place the float32 modules rather than casting the model with `.to(torch.bfloat16)` afterwards. + +## MiniMaxH3Transformer3DModel + +[[autodoc]] MiniMaxH3Transformer3DModel + +## MiniMaxH3TransformerOutput + +[[autodoc]] models.transformers.transformer_minimax_h3.MiniMaxH3TransformerOutput diff --git a/docs/source/en/api/pipelines/minimax_h3.md b/docs/source/en/api/pipelines/minimax_h3.md new file mode 100644 index 000000000000..cccf2d712b64 --- /dev/null +++ b/docs/source/en/api/pipelines/minimax_h3.md @@ -0,0 +1,270 @@ + + +# MiniMax-H3 + + +> [!TIP] +> MiniMax-H3 is not part of a diffusers release yet. Install diffusers from the pull request to use it: +> `pip install git+https://github.com/huggingface/diffusers.git@refs/pull/14355/head` + + +MiniMax-H3 generates video and its soundtrack **jointly**. One transformer denoises a single packed sequence that holds the text conditioning, the conditioning image, video and audio rows, the target audio rows and the target video rows at once, with full self-attention over all of it. There is no separate vocoder and no audio post-hoc pass: video and audio come out of the same denoising loop. + +You can find the original MiniMax-H3 checkpoints under the [MiniMaxAI](https://huggingface.co/MiniMaxAI) organization. + +MiniMax-H3 is integrated as [Modular Diffusers](../../modular_diffusers/overview) blocks only, the way [Anima](./anima) is: the blocks and their [`MiniMaxH3ModularPipeline`] are the whole integration, and there is no `DiffusionPipeline` half. + +## Checkpoint layout + +MiniMax-H3 was released as two checkpoint partitions that share every component except the transformer, so the diffusers conversion puts both in **one repository**: + +| Subfolder | Blocks | Tasks | +|---|---|---| +| `transformer/` | [`MiniMaxH3Blocks`] | `t2va` (text only) and `fl2va` (first and/or last keyframe) | +| `transformer_ref/` | [`MiniMaxH3Ref2VABlocks`] | `ref2va` (an ordered mix of image, video and audio references) | + +Everything but the transformer, i.e. the video VAE, the audio VAE, the Qwen3-VL conditioner, its tokenizer and processor, and the two schedulers, is shared and stored once. + +The repository carries one `modular_model_index.json`, which names every component of both halves with its own loading spec. Each blockset declares only the components it runs, and `load_components` fetches exactly those subfolders: loading the `t2va` / `fl2va` half never touches `transformer_ref/`, and loading the `ref2va` half never touches `transformer/`. Nothing else in the repository is fetched either, which is what lets one repository carry the two partitions, and the original checkpoint folders next to the converted ones. + +`modular_model_index.json` names the `t2va` / `fl2va` half as its own class, so [`~ModularPipeline.from_pretrained`] resolves that half. The `ref2va` half reads the very same file through its own blocks: + +```py +import torch +from diffusers import ModularPipeline +from diffusers.modular_pipelines import MiniMaxH3Ref2VABlocks + +# `t2va` / `fl2va`: loads `transformer/`, and never `transformer_ref/`. +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3") + +# `ref2va`: loads `transformer_ref/`, and never `transformer/`, out of the same repository. +pipe = MiniMaxH3Ref2VABlocks().init_pipeline("MiniMaxAI/MiniMax-H3") + +pipe.load_components(dtype=torch.bfloat16) +``` + +Each blockset also carries the workflows it serves — `t2va`, `fl2va` and `fl2va_last_frame` for [`MiniMaxH3Blocks`], `ref2va` for [`MiniMaxH3Ref2VABlocks`] — which name the inputs each task requires. + +The conditioner is a `Qwen3VLForConditionalGeneration`, and MiniMax-H3 reads the *unnormalized* hidden state after its 50th decoder layer rather than the last one, so the full released checkpoint is used with its language-model head unused. + +## Two schedulers + +Video and audio latents step down two different schedules inside a single transformer call per step, which is why both blocksets expect two [`MiniMaxH3Scheduler`] instances: `scheduler` for the video latents (`shift=12.0` in the released checkpoints) and `audio_scheduler` for the audio latents (`shift=3.0`). + +The checkpoint is guidance-distilled: guidance is baked into the weights, so there is no guider, no `negative_prompt` and no `guidance_scale`, and every step runs exactly one forward pass. + +## Generation constraints + +- **24 fps, 5 to 15 seconds.** `num_frames` is snapped up to the next `17 * n + 5` the video VAE can decode, and the resulting duration has to stay in that window. +- **A 768 pixel short edge.** `height` and `width` default to MiniMax-H3's own canvas for the aspect ratio of the first keyframe (or 16:9 without one) and must be multiples of 32. +- **One generator, three draws.** A request draws the keyframe or reference conditioning noise first, then the video noise, then the audio noise, all from the `generator` it is passed, so two runs from the same generator state return the same video and soundtrack. Passing `latents` or `audio_latents` replaces the corresponding draw. +- **`num_inference_steps` counts sigma grid points**, the terminal `0` included, so it drives one model evaluation less. + +## Memory + +The transformer alone is 61.7 GB in bfloat16 and the Qwen3-VL conditioner is another 62.1 GB, so the loading recipe depends on the hardware. Smaller canvases are the biggest speed lever on every setup: `height` and `width` only have to be multiples of 32, and 960x544 runs about 2.3x faster per step than the trained 1344x768. + +On one 80 GB card, register the components in a [`ComponentsManager`] and let it move them on and off the accelerator: + +```py +import torch +from diffusers import ComponentsManager, ModularPipeline + +manager = ComponentsManager() +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", components_manager=manager) +pipe.load_components(dtype=torch.bfloat16) +manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="12GB") +pipe.transformer.set_attention_backend("_flash_3_hub") # Hopper, roughly 3x faster; kernels fetched from the Hub +``` + +On a consumer card (24 to 32 GB), quantize the two large components to int8 as they load and stream the transformer's blocks from CPU RAM. Everything below uses supported loaders only, no patches, and works straight from the bfloat16 checkpoint: + +```py +import torch +from diffusers import MiniMaxH3Transformer3DModel, ModularPipeline, TorchAoConfig +from diffusers.hooks import apply_group_offloading +from transformers import Qwen3VLForConditionalGeneration +from transformers import TorchAoConfig as TransformersTorchAoConfig +from torchao.quantization import Int8WeightOnlyConfig + +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3") +pipe.update_components( + transformer=MiniMaxH3Transformer3DModel.from_pretrained( + "MiniMaxAI/MiniMax-H3", subfolder="transformer", dtype=torch.bfloat16, + quantization_config=TorchAoConfig( + Int8WeightOnlyConfig(version=2), + modules_to_not_convert=[ + "proj_in", "audio_proj_in", "context_embedder", "time_embedder", "time_proj", + "token_refiner", "norm_out", "proj_out", "audio_proj_out", + ], + ), + low_cpu_mem_usage=False, + ), + text_encoder=Qwen3VLForConditionalGeneration.from_pretrained( + "MiniMaxAI/MiniMax-H3", subfolder="text_encoder", dtype=torch.bfloat16, + quantization_config=TransformersTorchAoConfig( + Int8WeightOnlyConfig(version=2), + modules_to_not_convert=["model.visual", "model.language_model.embed_tokens", "model.language_model.norm", "lm_head"], + ), + ), +) +pipe.load_components(dtype=torch.bfloat16) + +# version=2 int8 tensors are pinnable, which streamed offload needs, and freezing removes the one autograd +# path the quantized tensors cannot serve. +pipe.transformer.requires_grad_(False) +pipe.text_encoder.requires_grad_(False) + +offload = dict(onload_device=torch.device("cuda"), offload_device=torch.device("cpu"), use_stream=True) +pipe.transformer.enable_group_offload(offload_type="block_level", num_blocks_per_group=1, **offload) +apply_group_offloading(pipe.text_encoder.model, offload_type="leaf_level", **offload) +pipe.vae.to("cuda") +pipe.audio_vae.to("cuda") +``` + +On 12 to 16 GB the same recipe works with the video VAE group offloaded too (`offload_type="leaf_level"`, no stream) and a small canvas such as 960x544. Expect the weights to live in host RAM: around 75 GB of it at int8. + +With two cards nothing has to be offloaded: the conditioner takes the second card through a `device_map` and the denoiser keeps the first. + +```py +import torch +from diffusers import ComponentsManager, ModularPipeline +from transformers import Qwen3VLForConditionalGeneration + +manager = ComponentsManager() +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", components_manager=manager) +pipe.update_components( + text_encoder=Qwen3VLForConditionalGeneration.from_pretrained( + "MiniMaxAI/MiniMax-H3", subfolder="text_encoder", dtype=torch.bfloat16, device_map={"": "cuda:1"} + ), +) +pipe.load_components(dtype=torch.bfloat16) +pipe.transformer.to("cuda:0") +pipe.vae.to("cuda:0") +pipe.audio_vae.to("cuda:0") +``` + +Two 80 GB cards run full bfloat16 this way with nothing streaming; two 48 GB cards do the same with the int8 loading above on both components. + +## Text and keyframes + +[`MiniMaxH3Blocks`] covers text-to-video-and-audio and keyframe conditioning. A keyframe can be the frame the video starts from (`image`), the frame it ends on (`last_image`), or both. + +```py +import torch +from diffusers import ModularPipeline +from diffusers.utils import load_image +from diffusers.utils.export_utils import encode_video + +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3") +pipe.load_components(dtype=torch.bfloat16) +pipe.to("cuda") + +prompt = "A red fox trotting through a snowy pine forest, snow crunching underfoot" + +# Text to video + audio. +state = pipe(prompt=prompt, generator=torch.Generator().manual_seed(42)) + +# First frame (and optionally last frame) to video + audio. The canvas follows the first keyframe. +image = load_image( + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg" +) +state = pipe(prompt=prompt, image=image, generator=torch.Generator().manual_seed(42)) + +encode_video( + state.get("videos")[0], + fps=24, + output_path="minimax_h3_fl2va.mp4", + audio=state.get("audio")[0], + audio_sample_rate=state.get("sampling_rate"), +) +``` + +Video and audio are generated jointly and come out of the call as separate outputs, `videos` and `audio`, next to the `sampling_rate` the soundtrack carries; muxing them into one file is left to the caller, e.g. with [`~utils.export_utils.encode_video`]. + +## Omni-references + +[`MiniMaxH3Ref2VABlocks`] conditions on an ordered list of references: up to 9 images, 3 videos and 3 audio clips, 12 in total. The order is semantic. It labels the references in the prompt presentation (`""`, `"