Add MAGI-1: Autoregressive Video Generation at Scale

[lavinal712]

To fix #11519

Thanks to @tolgacangoz and everyone who contributed to or reviewed the earlier work.

This PR adds initial MAGI-1 support to Diffusers, with the non-quantized MAGI-1 4.5B base checkpoint as the primary validated configuration.

It includes:

• AutoencoderKLMagi for causal video encoding and tiled decoding.
• MagiTransformer3DModel with MAGI attention, rotary embeddings, timestep conditioning, and KV-cache support.
• MagiTextConditioningModel for the learned null-caption, quality, and duration features.
• MagiEulerScheduler and the MAGI three-way classifier-free guider.
• Modular text-to-video, image-to-video, and video-to-video workflows.
• Chunked autoregressive generation and clean-prefix KV-cache management.
• A conversion script for converting the official MAGI-1 checkpoint into Diffusers format.
• Documentation, inference examples, and unit tests.

Validation

The implementation was checked using the converted official 4.5B base weights.

This PR is still under active development and is expected to require a few more weeks of work before it is ready for final review. The implementation and validation are progressing steadily, and the remaining issues are being addressed.

Thank you for your patience and for any feedback during this process.

@yiyixuxu @dg845 and @asomoza

[lavinal712]

Pipeline usage

import torch

from diffusers import ComponentsManager, MagiModularPipeline
from diffusers.utils import export_to_video

manager = ComponentsManager()
manager.enable_auto_cpu_offload(device="cuda:0")

pipe = MagiModularPipeline.from_pretrained(
    "/path/to/MAGI-1-diffusers",
    components_manager=manager,
)
pipe.load_components(
    dtype={
        "default": torch.float32,
        "transformer": torch.bfloat16,
        "vae": torch.bfloat16,
    }
)

pipe.transformer.set_attention_backend("flash_varlen")
pipe.vae.set_attention_backend("flash")
pipe.vae.enable_tiling(tile_sample_min_length=12)

video = pipe(
    prompt="A golden retriever runs through a sunlit meadow.",
    height=512,
    width=512,
    num_frames=96,
    num_inference_steps=64,
    cache_device="cpu",
    generator=torch.Generator("cuda:0").manual_seed(42),
    output_type="np",
    output="videos",
)

export_to_video(video[0], "magi.mp4", fps=24)

[lavinal712]

magi_output.mp4

[github-actions]

Hi @lavinal712, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.