Merge branch 'main' into lora-tests-modular

Author: sayakpaul

.ai/references/testing.md

@@ -30,8 +30,8 @@ Follow the style introduced in [#14113](https://github.com/huggingface/diffusers
   - `MemoryTesterMixin` — CPU offload, group offload, layerwise casting.
   - Cache mixins — `PyramidAttentionBroadcastTesterMixin`, `FasterCacheTesterMixin`, `FirstBlockCacheTesterMixin`, `TaylorSeerCacheTesterMixin`, `MagCacheTesterMixin`. Guidance-distilled models override the cache config (e.g. `FASTER_CACHE_CONFIG = {... "is_guidance_distilled": True}`). Don't introduce caching related tests in the first iteration. These tests are added on a case-by-case basis.
   - In the first pass, just add tests related to `PipelineTesterMixin` and `MemoryTesterMixin`.
-- **Declare a component that can't be offloaded — don't hand-write a skip.** Leaf-level offloading hooks only the supported leaf types (`nn.Linear`, `nn.Conv*`, `nn.Embedding` — see `_GO_LC_SUPPORTED_PYTORCH_LAYERS` in `src/diffusers/hooks/_common.py`) and onloads each on its own `forward`, so any code that reads a leaf's `.weight` instead of calling the leaf bypasses that leaf's hook and computes against offloaded weights. Which fix applies depends on who owns the component. For a diffusers model, set `_supports_group_offloading = False` on the `ModelMixin` subclass (as `HunyuanDiT2DModel` does) — both offload mixins honor the flag and skip themselves, so the gap is declared on the model instead of buried in a test file. For a third-party component you can't annotate, such as a `transformers` encoder, list it in `group_offloading_leaf_level_exclude_modules` on the config class; `enable_group_offload` keeps excluded components on the accelerator, so every other component stays covered — including the VAE, which the component-scoped `test_group_offloading_inference` leaves out. Block-level offloading is usually unaffected, hence the level in the name — a component that fails at both levels does need a skip.
-  - `torch.nn.MultiheadAttention` is the common instance: it passes `self.out_proj.weight` straight to `torch.nn.functional.multi_head_attention_forward` instead of calling `self.out_proj`, so the hook on `out_proj` never fires. `SiglipVisionModel`'s attention pooling head wraps one — see `tests/pipelines/hunyuan_video/test_hunyuan_video_framepack.py`, whose `image_encoder` is excluded for this reason.
+- **Declare a component that can't be offloaded — don't hand-write a skip.** For a diffusers model, set `_supports_group_offloading = False` on the `ModelMixin` subclass (as `HunyuanDiT2DModel` does); for a third-party component you can't annotate, such as a `transformers` encoder, name it in `group_offloading_leaf_level_exclude_modules` or `group_offloading_block_level_exclude_modules` on the config class. **Every `nn.Module` component is group offloaded unless the list for that level names it**, and the two levels fail on opposite hazards: leaf-level when compute reads a leaf's `.weight` instead of calling the leaf, block-level when a component re-enters submodules without going through the group leader's `forward` (VAE decode paths, hence the `vae` / `image_encoder` defaults). A component that fails at both goes in both, and a name matching no component fails the test as a typo. **Each level's test runs twice, with and without `use_stream`** — a subclass overriding one must re-declare `@MemoryTesterMixin._USE_STREAM`, or the override collapses to a single un-parametrized test that errors on the missing argument and reports as a green `xfail`.
+  - `torch.nn.MultiheadAttention` is the common leaf-level instance: it passes `self.out_proj.weight` straight to `torch.nn.functional.multi_head_attention_forward` instead of calling `self.out_proj`, so the hook on `out_proj` never fires. `SiglipVisionModel`'s attention pooling head wraps one — see `tests/pipelines/hunyuan_video/test_hunyuan_video_framepack.py`, whose `image_encoder` is excluded for this reason.
   - `HunyuanDiTAttentionPool` (`src/diffusers/models/embeddings.py`) shows the same failure without an MHA module: a plain `nn.Module` that hands its `q_proj` / `k_proj` / `v_proj` / `c_proj` weights to `torch.nn.functional.multi_head_attention_forward`, so all four projections stay offloaded rather than just one. `HunyuanDiT2DModel` opts out of group offloading entirely with `_supports_group_offloading = False`.
   - Before adding a skip or an exclusion, confirm the failure still reproduces — several existing skips are stale, having outlived the upstream cause.
 - **A migration that surfaces a `src/` gap marks the test `xfail`, it does not patch the pipeline.** Give the marker a module-level name and a `reason` naming the exact gap (`PNDM_*` in `tests/pipelines/pndm/test_pndm.py` is the worked example), and prefer `strict=True` so the marker reports XPASS — and gets deleted — the day the pipeline is fixed. Use `strict=False` only when one mark covers a group whose members do not all fail. Marking a whole test class keeps the mixin's own marks (`@is_memory`, `@require_accelerator`) intact; overriding individual inherited tests drops the decorators they were declared with, so re-declare those too.

.github/workflows/nightly_tests.yml

@@ -461,7 +461,6 @@ jobs:
       run_big_gpu_torch_tests,
       run_nightly_quantization_tests,
       run_nightly_pipeline_level_quantization_tests,
-      # run_nightly_onnx_tests,
       torch_minimum_version_cuda_tests,
     ]
     if: always()

src/diffusers/models/transformers/transformer_prx.py

@@ -161,7 +161,10 @@ def __call__(
             ones_img = torch.ones((bs, l_img), dtype=torch.bool, device=device)
             attention_mask = attention_mask.to(device=device, dtype=torch.bool)
             joint_mask = torch.cat([attention_mask, ones_img], dim=-1)
-            attn_mask_tensor = joint_mask[:, None, None, :].expand(-1, attn.heads, l_img, -1)
+            # Every attention backend broadcasts the mask itself, so materialising
+            # [B, heads, L_img, L_all] only costs bandwidth: at batch 32, 1024 image tokens
+            # and 28 heads that is a 1.1 GiB mask read per block. Keep it broadcastable.
+            attn_mask_tensor = joint_mask[:, None, None, :]
 
         # Apply attention using dispatch_attention_fn for backend support
         # Reshape to match dispatch_attention_fn expectations: [B, L, H, D]

src/diffusers/pipelines/chroma/pipeline_chroma.py

@@ -628,14 +628,14 @@ def __call__(
                 The height in pixels of the generated image. This is set to 1024 by default for the best results.
             width (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor):
                 The width in pixels of the generated image. This is set to 1024 by default for the best results.
-            num_inference_steps (`int`, *optional*, defaults to 50):
+            num_inference_steps (`int`, *optional*, defaults to 35):
                 The number of denoising steps. More denoising steps usually lead to a higher quality image at the
                 expense of slower inference.
             sigmas (`list[float]`, *optional*):
                 Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in
                 their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed
                 will be used.
-            guidance_scale (`float`, *optional*, defaults to 3.5):
+            guidance_scale (`float`, *optional*, defaults to 5.0):
                 Guidance scale as defined in [Classifier-Free Diffusion
                 Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2.
                 of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting

src/diffusers/pipelines/chroma/pipeline_chroma_img2img.py

@@ -698,7 +698,7 @@ def __call__(
                 Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in
                 their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed
                 will be used.
-            guidance_scale (`float`, *optional*, defaults to 3.5):
+            guidance_scale (`float`, *optional*, defaults to 5.0):
                 Guidance scale as defined in [Classifier-Free Diffusion
                 Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2.
                 of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting

src/diffusers/pipelines/chroma/pipeline_chroma_inpainting.py

@@ -828,20 +828,20 @@ def __call__(
                 with the same aspect ratio of the image and contains all masked area, and then expand that area based
                 on `padding_mask_crop`. The image and mask_image will then be cropped based on the expanded area before
                 resizing to the original image size for inpainting.
-            num_inference_steps (`int`, *optional*, defaults to 35):
+            num_inference_steps (`int`, *optional*, defaults to 28):
                 The number of denoising steps. More denoising steps usually lead to a higher quality image at the
                 expense of slower inference.
             sigmas (`List[float]`, *optional*):
                 Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in
                 their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed
                 will be used.
-            guidance_scale (`float`, *optional*, defaults to 3.5):
+            guidance_scale (`float`, *optional*, defaults to 7.0):
                 Guidance scale as defined in [Classifier-Free Diffusion
                 Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2.
                 of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting
                 `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to
                 the text `prompt`, usually at the expense of lower image quality.
-            strength (`float, *optional*, defaults to 0.9):
+            strength (`float, *optional*, defaults to 0.6):
                 Conceptually, indicates how much to transform the reference image. Must be between 0 and 1. image will
                 be used as a starting point, adding more noise to it the larger the strength. The number of denoising
                 steps depends on the amount of noise initially added. When strength is 1, added noise will be maximum
@@ -900,7 +900,7 @@ def __call__(
                 The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list
                 will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the
                 `._callback_tensor_inputs` attribute of your pipeline class.
-            max_sequence_length (`int` defaults to 512): Maximum sequence length to use with the `prompt`.
+            max_sequence_length (`int` defaults to 256): Maximum sequence length to use with the `prompt`.
 
         Examples:
 

src/diffusers/pipelines/deprecated/stable_diffusion_variants/pipeline_onnx_stable_diffusion_inpaint_legacy.py

@@ -10,7 +10,7 @@
 from ....schedulers import DDIMScheduler, LMSDiscreteScheduler, PNDMScheduler
 from ....utils import deprecate, logging
 from ...onnx_utils import ORT_TO_NP_TYPE, OnnxRuntimeModel
-from ...pipeline_utils import DiffusionPipeline
+from ...pipeline_utils import DeprecatedPipelineMixin, DiffusionPipeline
 from ...stable_diffusion.pipeline_output import StableDiffusionPipelineOutput
 
 
@@ -38,7 +38,9 @@ def preprocess_mask(mask, scale_factor=8):
     return mask
 
 
-class OnnxStableDiffusionInpaintPipelineLegacy(DiffusionPipeline):
+class OnnxStableDiffusionInpaintPipelineLegacy(DeprecatedPipelineMixin, DiffusionPipeline):
+    _last_supported_version = "0.43.0"
+
     r"""
     Pipeline for text-guided image inpainting using Stable Diffusion. This is a *legacy feature* for Onnx pipelines to
     provide compatibility with StableDiffusionInpaintPipelineLegacy and may be removed in the future.

src/diffusers/pipelines/onnx_utils.py

@@ -24,7 +24,7 @@
 from huggingface_hub import hf_hub_download
 from huggingface_hub.utils import validate_hf_hub_args
 
-from ..utils import ONNX_EXTERNAL_WEIGHTS_NAME, ONNX_WEIGHTS_NAME, is_onnx_available, logging
+from ..utils import ONNX_EXTERNAL_WEIGHTS_NAME, ONNX_WEIGHTS_NAME, deprecate, is_onnx_available, logging
 
 
 if is_onnx_available():
@@ -50,8 +50,14 @@
 
 
 class OnnxRuntimeModel:
+    _last_supported_version = "0.43.0"
+
     def __init__(self, model=None, **kwargs):
-        logger.info("`diffusers.OnnxRuntimeModel` is experimental and might change in the future.")
+        deprecate(
+            "OnnxRuntimeModel",
+            "0.43.0",
+            "Please use Optimum for ONNX Runtime support.",
+        )
         self.model = model
         self.model_save_dir = kwargs.get("model_save_dir", None)
         self.latest_model_name = kwargs.get("latest_model_name", ONNX_WEIGHTS_NAME)

src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion.py

@@ -23,14 +23,16 @@
 from ...schedulers import DDIMScheduler, LMSDiscreteScheduler, PNDMScheduler
 from ...utils import deprecate, logging
 from ..onnx_utils import ORT_TO_NP_TYPE, OnnxRuntimeModel
-from ..pipeline_utils import DiffusionPipeline
+from ..pipeline_utils import DeprecatedPipelineMixin, DiffusionPipeline
 from . import StableDiffusionPipelineOutput
 
 
 logger = logging.get_logger(__name__)
 
 
-class OnnxStableDiffusionPipeline(DiffusionPipeline):
+class OnnxStableDiffusionPipeline(DeprecatedPipelineMixin, DiffusionPipeline):
+    _last_supported_version = "0.43.0"
+
     vae_encoder: OnnxRuntimeModel
     vae_decoder: OnnxRuntimeModel
     text_encoder: OnnxRuntimeModel

src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_img2img.py

@@ -24,7 +24,7 @@
 from ...schedulers import DDIMScheduler, LMSDiscreteScheduler, PNDMScheduler
 from ...utils import PIL_INTERPOLATION, deprecate, logging
 from ..onnx_utils import ORT_TO_NP_TYPE, OnnxRuntimeModel
-from ..pipeline_utils import DiffusionPipeline
+from ..pipeline_utils import DeprecatedPipelineMixin, DiffusionPipeline
 from . import StableDiffusionPipelineOutput
 
 
@@ -55,7 +55,9 @@ def preprocess(image):
     return image
 
 
-class OnnxStableDiffusionImg2ImgPipeline(DiffusionPipeline):
+class OnnxStableDiffusionImg2ImgPipeline(DeprecatedPipelineMixin, DiffusionPipeline):
+    _last_supported_version = "0.43.0"
+
     r"""
     Pipeline for text-guided image to image generation using Stable Diffusion.