Merge branch 'minimax-h3' into minimax-h3-refactor

Author: yiyixuxu

.ai/modular.md

@@ -235,6 +235,19 @@ OutputParam(
 
 If a template's predefined description doesn't fit (e.g. the `"latents"` output template means "Denoised latents", which is wrong for the noisy latents out of a prepare-latents step) — drop the template and declare the field directly with an accurate description. See gotcha #5.
 
+**Declare defaults in the `InputParam`, not inside `__call__`.**
+
+```python
+# yes
+InputParam(name="num_frames", type_hint=int, default=189)
+
+# no — works, but the assembled pipeline is not aware of it
+if block_state.num_frames is None:
+    block_state.num_frames = 189
+```
+
+A declared default is part of the block's contract, so the assembled pipeline is aware of it: the generated docstring shows it and `default_call_parameters` reports it. Resolved inside the body instead, the input renders as `*optional*` with no default, and nothing at the pipeline level can report what the block will actually do. Don't worry about branches of a conditional blockset declaring different defaults for the same input — each branch resolves its own at runtime. Resolve inside `__call__` only when the default is *computed* — derived from other inputs or component config (`height = components.default_sample_size * components.vae_scale_factor`). And when several blocks in a sequence share an input, declare the same default on each (or only on the first block that reads it): in a sequence the input is one shared value, so disagreeing declarations are silently resolved first-block-wins.
+
 ## ComponentSpec patterns
 
 ```python

.github/workflows/new_model_request_reply.yml

@@ -0,0 +1,43 @@
+name: New Model Request Reply
+
+on:
+  issues:
+    types: [opened]
+
+jobs:
+  reply:
+    name: Point new model requests at Modular Diffusers
+    # Match the heading the issue form renders for its first field rather than a label: template
+    # labels are applied after the issue is created, so `github.event.issue.labels` is empty here.
+    # Keep this string in sync with .github/ISSUE_TEMPLATE/new-model-addition.yml.
+    if: >-
+      github.repository == 'huggingface/diffusers' &&
+      contains(github.event.issue.body, '### Model/Pipeline/Scheduler description')
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+    steps:
+      - name: Post Modular Diffusers guidance
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_REPO: ${{ github.repository }}
+          ISSUE_NUMBER: ${{ github.event.issue.number }}
+          BODY: |
+            Thanks for the request!
+
+            **How new model support works in Diffusers**
+
+            We're a small team, and our review queue shouldn't be what decides whether a model is usable in Diffusers. With [Modular Diffusers](https://huggingface.co/docs/diffusers/modular_diffusers/overview), a pipeline can live as remote code in any Hub repo and load straight from there with `from_pretrained`.
+
+            🛠️ **Want to bring this model to Diffusers?**
+
+            Please start with a Hub repo — you don't need anything from us to do that, and people can use it immediately. From there we decide how to support it: we might work with the authors, upstream an existing community version, or just point people at the one on the Hub. The pipelines we integrate are usually the ones people are already running.
+
+            Tag `@asomoza` when you have something to share — we'll give feedback on the implementation, help get it in front of people, and add the ones we like to our hand-picked [Modular Pipelines](https://huggingface.co/collections/diffusers/modular-pipelines) collection. Tell us where you hit friction along the way, too: confusing APIs, missing docs, bugs. That feedback is worth as much to us as the pipeline.
+
+            👋 **Are you an author of the model?** We'd love to hear from you — comment here and we'll help you pick the path that fits.
+
+            📚 [Quickstart](https://huggingface.co/docs/diffusers/modular_diffusers/quickstart) · [Building custom blocks](https://huggingface.co/docs/diffusers/modular_diffusers/custom_blocks) — template repo, and how to publish to the Hub · [Modular Pipelines](https://huggingface.co/collections/diffusers/modular-pipelines) and [Custom Blocks](https://huggingface.co/collections/diffusers/modular-diffusers-custom-blocks) — examples to crib from
+
+            *This is an automated message.*
+        run: gh issue comment "$ISSUE_NUMBER" --body "$BODY"

docs/source/en/conceptual/contribution.md

@@ -332,6 +332,11 @@ Good second issues are usually more difficult to get merged compared to good fir
 
 ### 9. Adding pipelines, models, schedulers
 
+> [!TIP]
+> If you are the model's author, please get in touch so we can coordinate the integration with you: open a feature request, or drop a comment if one is already open.
+>
+> If you are a community contributor, please also let us know you're interested under the feature request, and start with a Hub repo at the same time. See the [Modular Diffusers](../modular_diffusers/overview) guide to get started, and [custom blocks](../modular_diffusers/custom_blocks) or [custom models](../using-diffusers/automodel) for publishing as remote code on the Hub.
+
 Pipelines, models, and schedulers are the most important pieces of the Diffusers library.
 They provide easy access to state-of-the-art diffusion technologies and thus allow the community to
 build powerful generative AI applications.
@@ -605,6 +610,4 @@ AI-assisted contributions are welcome, but they must be coordinated, scoped, and
   - The **test commands you ran** and their results (paste relevant output, not just "tests pass").
   - Your **self-review notes** (or a link to the PR comment containing them), as described above.
 
-If you are a model author or part of a team that officially maintains a model, we encourage you to use agents for a new model integration. Follow the repository's [recommended setup](https://github.com/huggingface/diffusers/blob/main/.ai/AGENTS.md) and use the [`model-integration`](https://github.com/huggingface/diffusers/blob/main/.ai/skills/model-integration/SKILL.md) skill. Coordinate the scope with maintainers before opening a PR.
-
-If you are contributing a model to Diffusers for the first time as a community contributor, we generally recommend starting with a custom implementation that loads code from the Hub. This gives users access to the model while its integration into the core library is evaluated. See the [custom models](../using-diffusers/automodel) and [custom modular blocks](../modular_diffusers/custom_blocks) guides for supported patterns.
+If you are a model author or part of a team that officially maintains a model, we encourage you to use agents for a new model integration. Follow the repository's [recommended setup](https://github.com/huggingface/diffusers/blob/main/.ai/AGENTS.md) and use the [`model-integration`](https://github.com/huggingface/diffusers/blob/main/.ai/skills/model-integration/SKILL.md) skill. Coordinate the scope with maintainers before opening a PR — see [Adding pipelines, models, schedulers](#9-adding-pipelines-models-schedulers).

docs/source/en/optimization/attention_backends.md

@@ -82,6 +82,20 @@ with attention_backend("_flash_3_hub"):
 > [!TIP]
 > Most attention backends support `torch.compile` without graph breaks and can be used to further speed up inference.
 
+## Trusting remote kernels
+
+Hub backends and other kernel-backed features (such as [GGUF](../quantization/gguf) and [Nunchaku Lite](../quantization/nunchaku)) download compute kernels from the Hub with [`kernels`](https://github.com/huggingface/kernels) and execute their code locally.
+
+By default, `kernels` only loads a kernel when its publisher is a trusted kernel publisher on the Hub. Kernels published under the [`kernels-community`](https://huggingface.co/kernels-community) organization are trusted, so Diffusers loads them without any additional configuration. The `_flash_3_hub`, `flash_hub`, `sage_hub`, and the other Hub attention backends all resolve to `kernels-community` repositories.
+
+Kernels from any other publisher are not vetted. Loading one downloads and runs code that Diffusers cannot vouch for, so Diffusers keeps it disabled unless you explicitly opt in with the `DIFFUSERS_TRUST_REMOTE_KERNELS` environment variable. When set, Diffusers forwards `trust_remote_code=True` to `kernels` so it loads kernels from untrusted publishers too.
+
+```bash
+export DIFFUSERS_TRUST_REMOTE_KERNELS=true
+```
+
+Only enable this after inspecting the kernel repository, since it grants the downloaded code the ability to run on your machine. Without it, loading a kernel from an untrusted publisher raises an error. Diffusers performs this check itself, so it also applies to `kernels<0.14.0`, which predates the `trust_remote_code` argument. Setting `DIFFUSERS_DISABLE_REMOTE_CODE=true` disables remote code globally and takes precedence over `DIFFUSERS_TRUST_REMOTE_KERNELS`.
+
 ## Checks
 
 The attention dispatcher includes debugging checks that catch common errors before they cause problems.

docs/source/en/quantization/gguf.md

@@ -63,6 +63,8 @@ pip install -U kernels
 
 Once installed, set `DIFFUSERS_GGUF_CUDA_KERNELS=true`  to use optimized kernels when available. Note that CUDA kernels may introduce minor numerical differences compared to the original GGUF implementation, potentially causing subtle visual variations in generated images. To disable CUDA kernel usage, set the environment variable `DIFFUSERS_GGUF_CUDA_KERNELS=false`.
 
+The GGUF kernels are downloaded from the [`Isotr0py/ggml`](https://huggingface.co/Isotr0py/ggml) repository, whose publisher is not a trusted kernel publisher on the Hub. Loading it downloads and executes code from the Hub, so Diffusers requires you to explicitly opt in by setting `DIFFUSERS_TRUST_REMOTE_KERNELS=true`. See [Trusting remote kernels](../optimization/attention_backends#trusting-remote-kernels) for details.
+
 ## Supported Quantization Types
 
 - BF16

docs/source/en/quantization/nunchaku.md

@@ -27,6 +27,8 @@ The kernels package supplies the optimized CUDA kernels, which load automaticall
 pip install -U kernels
 ```
 
+Nunchaku Lite loads its kernels from the [`rootonchair/nunchaku-lite-kernels`](https://huggingface.co/rootonchair/nunchaku-lite-kernels) repository, whose publisher is not a trusted kernel publisher on the Hub. Loading it downloads and executes code from the Hub, so Diffusers requires you to explicitly opt in by setting `DIFFUSERS_TRUST_REMOTE_KERNELS=true`. See [Trusting remote kernels](../optimization/attention_backends#trusting-remote-kernels) for details.
+
 ## Load a quantized pipeline
 
 Load the prequantized pipeline with [`~DiffusionPipeline.from_pretrained`], which reads the quantization

src/diffusers/modular_pipelines/components_manager.py

@@ -23,6 +23,7 @@
 import torch
 
 from ..hooks import ModelHook
+from ..hooks.group_offloading import _is_group_offload_enabled
 from ..utils import (
     is_accelerate_available,
     logging,
@@ -76,12 +77,20 @@ def add_other_hook(self, hook: "UserCustomOffloadHook"):
         self.other_hooks.append(hook)
 
     def init_hook(self, module):
+        # A group offloaded module holds one group at a time and refuses `.to()`. Moving it here would be a
+        # silent no-op that leaves this hook recording an offload that never happened.
+        if _is_group_offload_enabled(module):
+            return module
         return module.to("cpu")
 
     def pre_forward(self, module, *args, **kwargs):
         if module.device != self.execution_device:
             if self.other_hooks is not None:
-                hooks_to_offload = [hook for hook in self.other_hooks if hook.model.device == self.execution_device]
+                hooks_to_offload = [
+                    hook
+                    for hook in self.other_hooks
+                    if hook.model.device == self.execution_device and not _is_group_offload_enabled(hook.model)
+                ]
                 # offload all other hooks
                 start_time = time.perf_counter()
                 if self.offload_strategy is not None:
@@ -104,7 +113,10 @@ def pre_forward(self, module, *args, **kwargs):
 
                 if hooks_to_offload:
                     clear_device_cache()
-            module.to(self.execution_device)
+            # The strategy still runs above, so a group offloaded model can make room for itself by moving other
+            # models — it just places itself.
+            if not _is_group_offload_enabled(module):
+                module.to(self.execution_device)
         return send_to_device(args, self.execution_device), send_to_device(kwargs, self.execution_device)
 
 
@@ -336,6 +348,7 @@ def __init__(self):
         self.collections = OrderedDict()  # collection_name -> set of component_names
         self.model_hooks = None
         self._auto_offload_enabled = False
+        self._offload_strategy = None
 
     def _lookup_ids(
         self,
@@ -692,7 +705,12 @@ def matches_pattern(component_id, pattern, exact_match=False):
 
         return get_return_dict(matches, return_dict_with_names)
 
-    def enable_auto_cpu_offload(self, device: str | int | torch.device = None, memory_reserve_margin="3GB"):
+    def enable_auto_cpu_offload(
+        self,
+        device: str | int | torch.device = None,
+        memory_reserve_margin="3GB",
+        offload_strategy=None,
+    ):
         """
         Enable automatic CPU offloading for all components.
 
@@ -703,11 +721,19 @@ def enable_auto_cpu_offload(self, device: str | int | torch.device = None, memor
         4. The system tries to offload the smallest combination of models that frees enough memory
         5. Models stay on the execution device until another model needs memory and forces them off
 
+        A group offloaded model takes part in this but places itself: it can still make room by moving other models
+        aside, and is never moved to make room for them. Either order works — group offload before or after enabling
+        this. `AutoOffloadStrategy` sizes its decisions from model memory footprints, which do not describe a model
+        holding one group at a time, so pass an `offload_strategy` that decides from the workflow instead.
+
         Args:
             device (str | int | torch.device): The execution device where models are moved for forward passes
             memory_reserve_margin (str): The memory reserve margin to use, default is 3GB. This is the amount of
                                         memory to keep free on the device to avoid running out of memory during model
                                         execution (e.g., for intermediate activations, gradients, etc.)
+            offload_strategy: Any callable with the signature `(hooks, model_id, model, execution_device) -> hooks`,
+                              returning which resident models to offload before the incoming one loads. Defaults to
+                              `AutoOffloadStrategy`, which frees the smallest sufficient combination.
         """
         if not is_accelerate_available():
             raise ImportError("Make sure to install accelerate to use auto_cpu_offload")
@@ -732,7 +758,17 @@ def enable_auto_cpu_offload(self, device: str | int | torch.device = None, memor
                 remove_hook_from_module(component, recurse=True)
 
         self.disable_auto_cpu_offload()
-        offload_strategy = AutoOffloadStrategy(memory_reserve_margin=memory_reserve_margin)
+        if offload_strategy is None:
+            offload_strategy = AutoOffloadStrategy(memory_reserve_margin=memory_reserve_margin)
+            if any(
+                isinstance(component, torch.nn.Module) and _is_group_offload_enabled(component)
+                for component in self.components.values()
+            ):
+                logger.warning(
+                    "`AutoOffloadStrategy` decides what to move from model memory footprints, which do not "
+                    "describe a group offloaded model: it holds one group at a time, not its whole weight. Pass "
+                    "an `offload_strategy` that decides from the workflow instead."
+                )
 
         all_hooks = []
         for name, component in self.components.items():
@@ -749,6 +785,23 @@ def enable_auto_cpu_offload(self, device: str | int | torch.device = None, memor
         self.model_hooks = all_hooks
         self._auto_offload_enabled = True
         self._auto_offload_device = device
+        self._offload_strategy = offload_strategy
+
+    def set_offload_strategy(self, offload_strategy):
+        """
+        Replace the offload strategy on all managed models. Only valid while auto CPU offloading is enabled.
+
+        Args:
+            offload_strategy:
+                Any callable with the signature `(hooks, model_id, model, execution_device) -> hooks`: it receives the
+                hooks of the models currently on the device and returns the ones to offload before the incoming model
+                loads. The default is `AutoOffloadStrategy`, which frees the smallest sufficient combination.
+        """
+        if not self._auto_offload_enabled:
+            raise ValueError("Auto CPU offloading is not enabled. Call `enable_auto_cpu_offload` first.")
+        for user_hook in self.model_hooks:
+            user_hook.hook.offload_strategy = offload_strategy
+        self._offload_strategy = offload_strategy
 
     def disable_auto_cpu_offload(self):
         """
@@ -765,6 +818,7 @@ def disable_auto_cpu_offload(self):
             clear_device_cache()
         self.model_hooks = None
         self._auto_offload_enabled = False
+        self._offload_strategy = None
 
     def get_model_info(
         self,

src/diffusers/modular_pipelines/cosmos/modular_blocks_cosmos3.py

@@ -143,7 +143,7 @@ class Cosmos3AutoTextEncoderStep(AutoPipelineBlocks):
               The text prompt that guides Cosmos3 generation.
           negative_prompt (`str`, *optional*):
               The negative text prompt used for classifier-free guidance.
-          use_system_prompt (`bool`, *optional*, defaults to True):
+          use_system_prompt (`bool`, *optional*, defaults to True or None, depending on the workflow):
               Whether to prepend the Cosmos3 transfer system prompt.
           action (`CosmosActionCondition`, *optional*):
               Action-conditioning metadata and its reference visual input.
@@ -1187,7 +1187,7 @@ class Cosmos3OmniBlocks(SequentialPipelineBlocks):
               The text prompt that guides Cosmos3 generation.
           negative_prompt (`str`, *optional*):
               The negative text prompt used for classifier-free guidance.
-          use_system_prompt (`bool`, *optional*, defaults to True):
+          use_system_prompt (`bool`, *optional*, defaults to True or None, depending on the workflow):
               Whether to prepend the Cosmos3 transfer system prompt.
           action (`CosmosActionCondition`, *optional*):
               Action-conditioning metadata and its reference visual input.

src/diffusers/modular_pipelines/mellon_node_utils.py

@@ -1058,6 +1058,7 @@ def from_custom_block(
         inputs = []
         model_inputs = []
         outputs = []
+        required_inputs = []
 
         # Process block inputs
         for input_param in block.inputs:
@@ -1066,7 +1067,8 @@ def from_custom_block(
             if input_param.name in input_types:
                 input_param = copy.copy(input_param)
                 input_param.metadata = {"mellon": input_types[input_param.name]}
-            print(f" processing input: {input_param.name}, metadata: {input_param.metadata}")
+            if input_param.required:
+                required_inputs.append(input_param.name)
             inputs.append(input_param_to_mellon_param(input_param))
 
         # Process block outputs
@@ -1090,7 +1092,7 @@ def from_custom_block(
             "inputs": inputs,
             "model_inputs": model_inputs,
             "outputs": outputs,
-            "required_inputs": [],
+            "required_inputs": required_inputs,
             "required_model_inputs": [],
             "block_name": "custom",
         }