[kernels] download kernels when users request for it. (#14298)

* download kernels when users request for it.

* address review feedback

Author: sayakpaul

docs/source/en/optimization/attention_backends.md

@@ -82,6 +82,20 @@ with attention_backend("_flash_3_hub"):
 > [!TIP]
 > Most attention backends support `torch.compile` without graph breaks and can be used to further speed up inference.
 
+## Trusting remote kernels
+
+Hub backends and other kernel-backed features (such as [GGUF](../quantization/gguf) and [Nunchaku Lite](../quantization/nunchaku)) download compute kernels from the Hub with [`kernels`](https://github.com/huggingface/kernels) and execute their code locally.
+
+By default, `kernels` only loads a kernel when its publisher is a trusted kernel publisher on the Hub. Kernels published under the [`kernels-community`](https://huggingface.co/kernels-community) organization are trusted, so Diffusers loads them without any additional configuration. The `_flash_3_hub`, `flash_hub`, `sage_hub`, and the other Hub attention backends all resolve to `kernels-community` repositories.
+
+Kernels from any other publisher are not vetted. Loading one downloads and runs code that Diffusers cannot vouch for, so Diffusers keeps it disabled unless you explicitly opt in with the `DIFFUSERS_TRUST_REMOTE_KERNELS` environment variable. When set, Diffusers forwards `trust_remote_code=True` to `kernels` so it loads kernels from untrusted publishers too.
+
+```bash
+export DIFFUSERS_TRUST_REMOTE_KERNELS=true
+```
+
+Only enable this after inspecting the kernel repository, since it grants the downloaded code the ability to run on your machine. Without it, loading a kernel from an untrusted publisher raises an error. Diffusers performs this check itself, so it also applies to `kernels<0.14.0`, which predates the `trust_remote_code` argument. Setting `DIFFUSERS_DISABLE_REMOTE_CODE=true` disables remote code globally and takes precedence over `DIFFUSERS_TRUST_REMOTE_KERNELS`.
+
 ## Checks
 
 The attention dispatcher includes debugging checks that catch common errors before they cause problems.

docs/source/en/quantization/gguf.md

@@ -63,6 +63,8 @@ pip install -U kernels
 
 Once installed, set `DIFFUSERS_GGUF_CUDA_KERNELS=true`  to use optimized kernels when available. Note that CUDA kernels may introduce minor numerical differences compared to the original GGUF implementation, potentially causing subtle visual variations in generated images. To disable CUDA kernel usage, set the environment variable `DIFFUSERS_GGUF_CUDA_KERNELS=false`.
 
+The GGUF kernels are downloaded from the [`Isotr0py/ggml`](https://huggingface.co/Isotr0py/ggml) repository, whose publisher is not a trusted kernel publisher on the Hub. Loading it downloads and executes code from the Hub, so Diffusers requires you to explicitly opt in by setting `DIFFUSERS_TRUST_REMOTE_KERNELS=true`. See [Trusting remote kernels](../optimization/attention_backends#trusting-remote-kernels) for details.
+
 ## Supported Quantization Types
 
 - BF16

docs/source/en/quantization/nunchaku.md

@@ -27,6 +27,8 @@ The kernels package supplies the optimized CUDA kernels, which load automaticall
 pip install -U kernels
 ```
 
+Nunchaku Lite loads its kernels from the [`rootonchair/nunchaku-lite-kernels`](https://huggingface.co/rootonchair/nunchaku-lite-kernels) repository, whose publisher is not a trusted kernel publisher on the Hub. Loading it downloads and executes code from the Hub, so Diffusers requires you to explicitly opt in by setting `DIFFUSERS_TRUST_REMOTE_KERNELS=true`. See [Trusting remote kernels](../optimization/attention_backends#trusting-remote-kernels) for details.
+
 ## Load a quantized pipeline
 
 Load the prequantized pipeline with [`~DiffusionPipeline.from_pretrained`], which reads the quantization

src/diffusers/quantizers/gguf/utils.py

@@ -20,7 +20,8 @@
 import torch
 import torch.nn as nn
 
-from ...utils import is_accelerate_available, is_kernels_available
+from ...utils import is_accelerate_available, is_kernels_available, is_kernels_version
+from ...utils.constants import DIFFUSERS_TRUST_REMOTE_KERNELS
 
 
 if is_accelerate_available():
@@ -37,7 +38,15 @@
 if can_use_cuda_kernels and is_kernels_available():
     from kernels import get_kernel
 
-    ops = get_kernel("Isotr0py/ggml")
+    if not DIFFUSERS_TRUST_REMOTE_KERNELS:
+        raise ValueError(
+            "`Isotr0py/ggml` is not published by a trusted kernel publisher on the Hub, so loading it downloads "
+            "and executes remote code. Set `DIFFUSERS_TRUST_REMOTE_KERNELS=true` to allow it, or set "
+            "`DIFFUSERS_GGUF_CUDA_KERNELS=false` to run without the CUDA kernels."
+        )
+    # `kernels<0.14.0` has no `trust_remote_code` argument and executes the downloaded code unconditionally.
+    trust_kwargs = {"trust_remote_code": True} if is_kernels_version(">=", "0.14.0") else {}
+    ops = get_kernel("Isotr0py/ggml", **trust_kwargs)
 else:
     ops = None
 

src/diffusers/quantizers/nunchaku/utils.py

@@ -8,7 +8,8 @@
 import torch
 import torch.nn as nn
 
-from ...utils import is_accelerate_available, is_kernels_available
+from ...utils import is_accelerate_available, is_kernels_available, is_kernels_version
+from ...utils.constants import DIFFUSERS_TRUST_REMOTE_KERNELS
 
 
 if is_accelerate_available():
@@ -22,7 +23,14 @@
 if is_kernels_available():
     from kernels import get_kernel
 
-    ops = get_kernel(_HF_KERNEL_REPO, version=_HF_KERNEL_VERSION, trust_remote_code=True).ops
+    if not DIFFUSERS_TRUST_REMOTE_KERNELS:
+        raise ValueError(
+            f"`{_HF_KERNEL_REPO}` is not published by a trusted kernel publisher on the Hub, so loading it "
+            "downloads and executes remote code. Set `DIFFUSERS_TRUST_REMOTE_KERNELS=true` to allow it."
+        )
+    # `kernels<0.14.0` has no `trust_remote_code` argument and executes the downloaded code unconditionally.
+    trust_kwargs = {"trust_remote_code": True} if is_kernels_version(">=", "0.14.0") else {}
+    ops = get_kernel(_HF_KERNEL_REPO, version=_HF_KERNEL_VERSION, **trust_kwargs).ops
 else:
     raise ImportError(
         "Loading Nunchaku checkpoints requires the Hugging Face `kernels` package. "

src/diffusers/utils/constants.py

@@ -48,6 +48,11 @@
 HF_ENABLE_PARALLEL_LOADING = os.environ.get("HF_ENABLE_PARALLEL_LOADING", "").upper() in ENV_VARS_TRUE_VALUES
 DIFFUSERS_DISABLE_REMOTE_CODE = os.getenv("DIFFUSERS_DISABLE_REMOTE_CODE", "false").upper() in ENV_VARS_TRUE_VALUES
 DIFFUSERS_SDNQ_TRANSFORMERS = os.getenv("DIFFUSERS_SDNQ_TRANSFORMERS", "false").upper() in ENV_VARS_TRUE_VALUES
+# Kernels published by untrusted publishers execute remote code, so a globally disabled remote code wins over the opt-in.
+DIFFUSERS_TRUST_REMOTE_KERNELS = (
+    os.getenv("DIFFUSERS_TRUST_REMOTE_KERNELS", "false").upper() in ENV_VARS_TRUE_VALUES
+    and not DIFFUSERS_DISABLE_REMOTE_CODE
+)
 
 # Below should be `True` if the current version of `peft` and `transformers` are compatible with
 # PEFT backend. Will automatically fall back to PEFT backend if the correct versions of the libraries are