Merge branch 'main' into add-h3-tp-support

Author: JingyaHuang

.ai/references/models.md

@@ -87,13 +87,18 @@ if num_key_value_groups > 1:
 
 `dim=2` because tensors are `(batch_size, seq_len, num_heads, head_dim)` here. Must be `repeat_interleave`, not `repeat` — the groups are contiguous, and `repeat` gives a silently wrong pairing no shape check catches.
 
-Both compute the same thing, so weigh the two on compatibility and performance and recommend whichever fits the model better.
+Both compute the same thing. What differs is which kernel runs. Under the default backend (`native`, a plain `F.scaled_dot_product_attention`), torch has four kernels:
 
-- **Compatibility.** Most backends do not implement `enable_gqa` yet — flash, FA3, sage, cuDNN and the hub kernels raise on it, as does the context-parallel path. Grep `enable_gqa` in `attention_dispatch.py` for the current list rather than trusting this one; it changes as support lands. The flag limits the model to whichever backends still accept it, while repeating works on all of them.
+| kernel | mask | mismatched q/kv heads (`enable_gqa`) |
+|---|---|---|
+| flash | ✗ | ✓ |
+| efficient | ✓ | ✗ |
+| math | ✓ | ✓ — materializes the full `[batch_size, num_heads, seq_len_q, seq_len_kv]` score matrix |
+| cuDNN | ✓ | ✓ |
 
-- **Performance.** Depends on whether the model passes a mask. With a mask, no fused kernel takes a mask *and* mismatched head counts, so SDPA falls back to math and materializes the full `[batch_size, num_heads, seq_len_q, seq_len_kv]` score matrix — no error, no warning, only memory. Without a mask, flash broadcasts inside the kernel and the flag saves the key/value copy. Both effects scale with sequence length and head count, so measure at the model's real shape; `torch.backends.cuda.can_use_flash_attention(params, debug=True)` and `can_use_efficient_attention` print why a kernel was rejected, which is the fastest way to see which one you actually got.
+It tries them in a priority order and takes the first that accepts the call; the order changes across torch versions and GPUs.
 
-- **Recommendation.** Repeat by default — it is portable and never pathological. Reach for `enable_gqa=True` only when the model never passes a mask *and* the measured saving justifies the narrower backend support. For scale: on Krea 2 at 1024×1024, masked, the flag cost 9.02 GiB and 26.7 ms per call against 0.16 GiB and 4.1 ms repeated; unmasked at the same shape it saved 0.11 GiB and 0.1 ms. `transformer_cosmos3.py` is the in-repo case where it is defensible — causal, never masked.
+If the model passes a mask, repeat the key/value heads: (1) `enable_gqa` is rejected by the context-parallel path, and (2) with a mask it can only land on math or cuDNN — if math comes first, it silently materializes the full score matrix.
 
 ## Model class attributes
 

.ai/references/testing.md

@@ -9,7 +9,7 @@ Two test layers must be added for any new pipeline: pipeline-level tests, and (i
 - Keep component sizes tiny so the suite runs fast — small `num_layers`, small hidden/attention dims, low resolution, few frames. Reference `tests/pipelines/wan/test_wan.py` (`get_dummy_components` and `get_dummy_inputs`) for the size scale to target.
 - Build dummy components from the **real classes** at tiny config — a real VAE with tiny dims, a real tokenizer from an `hf-internal-testing/tiny-random-*` repo. Don't substitute a hand-rolled mock (a bare `nn.Module` with a `SimpleNamespace` config, a fake tokenizer) without a good reason: a mock is written by copying whatever the pipeline reads from the component today, so it can only confirm the pipeline against itself — the test stays green when the component renames a config field or the pipeline starts reading one the component doesn't have, and catching exactly that pipeline↔component contract is what a pipeline test is for. A good reason to stub: the component is impractical to instantiate and only its I/O matters to the pipeline (e.g. `DummyCosmosSafetyChecker` standing in for the huge Cosmos guardrail) — then make it a shared, purpose-built class honoring the real interface.
 - The same applies to test doubles at the call level: don't monkeypatch a component method (e.g. the scheduler's `set_timesteps`) just to capture what the code under test passed to it — that only verifies the caller against itself, not against the real method's contract. Call the real component and assert on its resulting state.
-- No LoRA tests in the initial PR (no `LoraTesterMixin`, no `tests/lora/test_lora_layers_<model>.py`).
+- No LoRA tests in the initial PR — don't compose the LoRA tester mixins into the pipeline or model test file (see [LoRA tests](#lora-tests)), and don't add a `tests/lora/test_lora_layers_<model>.py`.
 - No integration / slow tests in the initial PR — don't add anything gated on `@slow` / `RUN_SLOW=1` yet.
 
 ## Pipeline-level tests 
@@ -30,8 +30,28 @@ Follow the style introduced in [#14113](https://github.com/huggingface/diffusers
   - `MemoryTesterMixin` — CPU offload, group offload, layerwise casting.
   - Cache mixins — `PyramidAttentionBroadcastTesterMixin`, `FasterCacheTesterMixin`, `FirstBlockCacheTesterMixin`, `TaylorSeerCacheTesterMixin`, `MagCacheTesterMixin`. Guidance-distilled models override the cache config (e.g. `FASTER_CACHE_CONFIG = {... "is_guidance_distilled": True}`). Don't introduce caching related tests in the first iteration. These tests are added on a case-by-case basis.
   - In the first pass, just add tests related to `PipelineTesterMixin` and `MemoryTesterMixin`.
+- **Declare a component that can't be offloaded — don't hand-write a skip.** Leaf-level offloading hooks only the supported leaf types (`nn.Linear`, `nn.Conv*`, `nn.Embedding` — see `_GO_LC_SUPPORTED_PYTORCH_LAYERS` in `src/diffusers/hooks/_common.py`) and onloads each on its own `forward`, so any code that reads a leaf's `.weight` instead of calling the leaf bypasses that leaf's hook and computes against offloaded weights. Which fix applies depends on who owns the component. For a diffusers model, set `_supports_group_offloading = False` on the `ModelMixin` subclass (as `HunyuanDiT2DModel` does) — both offload mixins honor the flag and skip themselves, so the gap is declared on the model instead of buried in a test file. For a third-party component you can't annotate, such as a `transformers` encoder, list it in `group_offloading_leaf_level_exclude_modules` on the config class; `enable_group_offload` keeps excluded components on the accelerator, so every other component stays covered — including the VAE, which the component-scoped `test_group_offloading_inference` leaves out. Block-level offloading is usually unaffected, hence the level in the name — a component that fails at both levels does need a skip.
+  - `torch.nn.MultiheadAttention` is the common instance: it passes `self.out_proj.weight` straight to `torch.nn.functional.multi_head_attention_forward` instead of calling `self.out_proj`, so the hook on `out_proj` never fires. `SiglipVisionModel`'s attention pooling head wraps one — see `tests/pipelines/hunyuan_video/test_hunyuan_video_framepack.py`, whose `image_encoder` is excluded for this reason.
+  - `HunyuanDiTAttentionPool` (`src/diffusers/models/embeddings.py`) shows the same failure without an MHA module: a plain `nn.Module` that hands its `q_proj` / `k_proj` / `v_proj` / `c_proj` weights to `torch.nn.functional.multi_head_attention_forward`, so all four projections stay offloaded rather than just one. `HunyuanDiT2DModel` opts out of group offloading entirely with `_supports_group_offloading = False`.
+  - Before adding a skip or an exclusion, confirm the failure still reproduces — several existing skips are stale, having outlived the upstream cause.
 - **IP-Adapter tests** live in their own class decorated with `@is_ip_adapter`, subclassing only the config (not `PipelineTesterMixin`).
 
+#### LoRA tests
+
+Since [#14268](https://github.com/huggingface/diffusers/pull/14268), a standard pipeline's LoRA tests live **next to its pipeline tests** — another mixin composed with the same `<Pipeline>PipelineTesterConfig` — not in `tests/lora/test_lora_layers_<model>.py`. Reference: `TestFluxPipelineLoRA` / `TestFluxPipelineLoRAMemory` in `tests/pipelines/flux/test_pipeline_flux.py`.
+
+- Mixins live in `tests/pipelines/testing_utils/lora.py` and are exported from `..testing_utils`. One test class each, named `Test<Pipeline>LoRA...`:
+  - `LoraTesterMixin` — adapter attach/detach, LoRA scale and attention-kwargs, fuse/unfuse, multi-adapter (set/delete/weight), save/load round-trips, adapter metadata. Runs on CPU.
+  - `LoraMemoryTesterMixin` — LoRA × memory optimizations (group offload, model CPU offload, deleting adapters while offloaded). Accelerator-only.
+  - `UNetLoraTesterMixin` — per-block scale tests; UNet pipelines only.
+- **Give each mixin its own test class.** They are marked `@is_lora`, and a mark applies to every test in the class that inherits it — mixing one into `Test<Pipeline>` would mark those tests as LoRA tests too.
+- Run them with `pytest tests/pipelines/ -m "lora"` (what CI does); `pytest -m "not lora"` skips them.
+- **Nothing LoRA-specific goes on the config class.** The mixins read the same contract as every other mixin — `pipeline_class`, `get_dummy_components()`, `get_dummy_inputs()` with `output_type="pt"` — and self-skip when `pipeline_class` isn't a `LoraBaseMixin` subclass.
+- Components to adapt are derived from `pipeline_class._lora_loadable_modules`. Override `denoiser_target_modules` on the test class only when the denoiser's attention modules aren't named `to_q` / `to_k` / `to_v` / `to_out.0`. A text encoder architecture that isn't registered yet needs an entry in `TEXT_ENCODER_TARGET_MODULES` in `tests/pipelines/testing_utils/lora.py` — not a per-class override.
+- **Pipeline-specific LoRA tests are methods on the `Test<Pipeline>LoRA` class**, written against the shared helpers: `self.get_pipeline()`, `self.add_adapters_to_pipeline(pipe, components=[...], **lora_config_kwargs)`, `self.run_pipe(pipe)`, and the class-scoped `base_pipe_output` fixture (baseline output of the un-adapted pipeline). `run_pipe` produces `base_pipe_output`, so the two are directly comparable — don't hand-roll a forward pass to compare against it. See `test_with_alpha_in_state_dict` and `test_lora_expansion_works_for_{absent,extra}_keys` on `TestFluxPipelineLoRA`.
+- Load and save through the public API (`pipe.load_lora_weights`, `pipeline_class.save_lora_weights`, `pipe.set_adapters`, `pipe.unload_lora_weights`), and assert the adapter landed with `check_if_lora_correctly_set` from `...models.testing_utils.lora`.
+- Nightly LoRA-checkpoint integration tests (loading real Hub LoRAs) go in the same file, in their own `@nightly @require_big_accelerator @require_peft_backend` class — see `TestFluxLoRAIntegration`. Still not part of an initial PR.
+
 ### Modular pipelines
 
 - Location: `tests/modular_pipelines/<model>/test_modular_pipeline_<model>.py` (one config class + set of test classes per blockset / pipeline variant).
@@ -60,5 +80,5 @@ python utils/generate_model_tests.py src/diffusers/models/transformers/transform
 - Run with **no `--include` flags** initially. The generator auto-detects mixins/attributes and emits the always-on testers (`ModelTesterMixin`, `MemoryTesterMixin`, `TorchCompileTesterMixin`, plus `AttentionTesterMixin` / `ContextParallelTesterMixin` / `TrainingTesterMixin` as applicable). Optional testers (quantization, caching, single-file, IP adapter, etc.) are added later, after maintainer discussion.
 - The generator writes to `tests/models/transformers/test_models_transformer_<model>.py` (or the matching `unets/` / `autoencoders/` subdir).
 - Fill in the `TODO`s in the generated `<Model>TesterConfig`: `pretrained_model_name_or_path`, `get_init_dict()` (tiny config), `get_dummy_inputs()`, `input_shape`, `output_shape`. Keep init dims small for speed.
-- Do **not** add `LoraTesterMixin` at the start, even if the model subclasses `PeftAdapterMixin` — strip it from the generated file for the initial PR.
+- Do **not** add the model-level `LoraTesterMixin` (from `tests/models/testing_utils/lora.py`, distinct from the pipeline-level one) at the start, even if the model subclasses `PeftAdapterMixin` — strip it from the generated file for the initial PR.
 - Reference: `tests/models/transformers/test_models_transformer_flux.py`.

.github/workflows/pr_link_issue_reminder.yml

@@ -7,8 +7,11 @@ on:
 
 jobs:
   remind:
-    # Reminds external contributors to link an issue. PRs from maintainers, users
-    # with write/admin access, and collaborators are skipped by the script.
+    # Reminds external contributors to link an issue, reports still-unlinked PRs to
+    # Slack (one message per PR, sent once) 5 days after the reminder, and auto-closes
+    # them 10 days after the reminder unless rescued (issue linked or `no-issue-needed`
+    # label added). PRs from maintainers, users with write/admin access, and collaborators
+    # are skipped by the script.
     name: Remind external contributors to link an issue
     if: github.repository == 'huggingface/diffusers'
     runs-on: ubuntu-22.04
@@ -18,6 +21,10 @@ jobs:
       issues: write
     env:
       GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL_PR_LINK_ISSUE }}
+      # Comma-separated Slack member IDs pinged in the Slack rescue messages
+      # (Sayak Paul, YiYi Xu, Dhruv Nair, Daniel Gu).
+      SLACK_MENTION_IDS: ${{ secrets.SLACK_PR_LINK_ISSUE_MENTION_IDS }}
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
 

docker/diffusers-pytorch-cuda/Dockerfile

@@ -10,11 +10,14 @@ RUN apt-get -y update \
     && add-apt-repository ppa:deadsnakes/ppa && \
     apt-get update
 
-# cuda-cudart-dev provides the CUDA headers (cuda.h etc.) AOT-inductor compilation needs;
-# the runtime base image only ships the libraries.
+# The runtime base image only ships the CUDA libraries, but AOT-inductor compiles a C++
+# wrapper against the CUDA headers. cuda-cudart-dev provides cuda.h/cuda_runtime_api.h, and
+# cuda-nvcc pulls in the crt/ headers (crt/host_defines.h etc.) that cuda_runtime_api.h
+# includes in turn -- both are needed or the AOT tests fail with a CppCompileError.
 RUN apt install -y bash \
     build-essential \
     cuda-cudart-dev-12-9 \
+    cuda-nvcc-12-9 \
     git \
     git-lfs \
     curl \

docs/source/en/advanced_inference/outpaint.md

@@ -117,7 +117,7 @@ controlnets = [
         "diffusers/controlnet-zoe-depth-sdxl-1.0", dtype=torch.float16
     ),
 ]
-vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16).to("cuda")
+vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16).to("cuda")  # or "mps", "xpu", "cpu"
 pipeline = StableDiffusionXLControlNetPipeline.from_pretrained(
     "SG161222/RealVisXL_V4.0", dtype=torch.float16, variant="fp16", controlnet=controlnets, vae=vae
 ).to("cuda")
@@ -176,7 +176,7 @@ pipeline = StableDiffusionXLInpaintPipeline.from_pretrained(
     dtype=torch.float16,
     variant="fp16",
     vae=vae,
-).to("cuda")
+).to("cuda")  # or "mps", "xpu", "cpu"
 ```
 
 Prepare a mask for the final outpainted image. To create a more natural transition between the original image and the outpainted background, blur the mask to help it blend better.

docs/source/en/api/models/allegro_transformer3d.md

@@ -18,7 +18,7 @@ The model can be loaded with the following code snippet.
 ```python
 from diffusers import AllegroTransformer3DModel
 
-transformer = AllegroTransformer3DModel.from_pretrained("rhymes-ai/Allegro", subfolder="transformer", dtype=torch.bfloat16).to("cuda")
+transformer = AllegroTransformer3DModel.from_pretrained("rhymes-ai/Allegro", subfolder="transformer", dtype=torch.bfloat16).to("cuda")  # or "mps", "xpu", "cpu"
 ```
 
 ## AllegroTransformer3DModel

docs/source/en/api/models/asymmetricautoencoderkl.md

@@ -41,7 +41,7 @@ mask_image = load_image(mask_url).resize((512, 512))
 
 pipe = StableDiffusionInpaintPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-inpainting")
 pipe.vae = AsymmetricAutoencoderKL.from_pretrained("cross-attention/asymmetric-autoencoder-kl-x-1-5")
-pipe.to("cuda")
+pipe.to("cuda")  # or "mps", "xpu", "cpu"
 
 image = pipe(prompt=prompt, image=original_image, mask_image=mask_image).images[0]
 make_image_grid([original_image, mask_image, image], rows=1, cols=3)

docs/source/en/api/models/autoencoder_dc.md

@@ -36,7 +36,7 @@ Load a model in Diffusers format with [`~ModelMixin.from_pretrained`].
 ```python
 from diffusers import AutoencoderDC
 
-ae = AutoencoderDC.from_pretrained("mit-han-lab/dc-ae-f32c32-sana-1.0-diffusers", dtype=torch.float32).to("cuda")
+ae = AutoencoderDC.from_pretrained("mit-han-lab/dc-ae-f32c32-sana-1.0-diffusers", dtype=torch.float32).to("cuda")  # or "mps", "xpu", "cpu"
 ```
 
 ## Load a model in Diffusers via `from_single_file`

docs/source/en/api/models/autoencoder_rae.md

@@ -34,7 +34,7 @@ from diffusers import AutoencoderRAE
 
 model = AutoencoderRAE.from_pretrained(
     "nyu-visionx/RAE-dinov2-wReg-base-ViTXL-n08"
-).to("cuda").eval()
+).to("cuda").eval()  # or "mps", "xpu", "cpu"
 ```
 
 ## Encoding and decoding a real image
@@ -47,7 +47,7 @@ from torchvision.transforms.functional import to_tensor, to_pil_image
 
 model = AutoencoderRAE.from_pretrained(
     "nyu-visionx/RAE-dinov2-wReg-base-ViTXL-n08"
-).to("cuda").eval()
+).to("cuda").eval()  # or "mps", "xpu", "cpu"
 
 image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
 image = image.convert("RGB").resize((224, 224))
@@ -68,7 +68,7 @@ Some pretrained checkpoints include per-channel `latents_mean` and `latents_std`
 ```python
 model = AutoencoderRAE.from_pretrained(
     "nyu-visionx/RAE-dinov2-wReg-base-ViTXL-n08"
-).to("cuda").eval()
+).to("cuda").eval()  # or "mps", "xpu", "cpu"
 
 # Latent normalization is handled automatically inside encode/decode
 # when the checkpoint config includes latents_mean/latents_std.

docs/source/en/api/models/autoencoder_tiny.md

@@ -24,7 +24,7 @@ pipe = DiffusionPipeline.from_pretrained(
     "stabilityai/stable-diffusion-2-1-base", dtype=torch.float16
 )
 pipe.vae = AutoencoderTiny.from_pretrained("madebyollin/taesd", dtype=torch.float16)
-pipe = pipe.to("cuda")
+pipe = pipe.to("cuda")  # or "mps", "xpu", "cpu"
 
 prompt = "slice of delicious New York-style berry cheesecake"
 image = pipe(prompt, num_inference_steps=25).images[0]
@@ -41,7 +41,7 @@ pipe = DiffusionPipeline.from_pretrained(
     "stabilityai/stable-diffusion-xl-base-1.0", dtype=torch.float16
 )
 pipe.vae = AutoencoderTiny.from_pretrained("madebyollin/taesdxl", dtype=torch.float16)
-pipe = pipe.to("cuda")
+pipe = pipe.to("cuda")  # or "mps", "xpu", "cpu"
 
 prompt = "slice of delicious New York-style berry cheesecake"
 image = pipe(prompt, num_inference_steps=25).images[0]