[Looking for community contribution] support DiffSynth Controlnet in diffusers

[yiyixuxu]

Model/Pipeline/Scheduler description

Hi!
We want to add first party support for DiffSynth controlnet in diffusers, and we are looking for some help from the community!

Let me know if you're interested!

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://huggingface.co/SahilCarterr/Qwen-Image-Blockwise-ControlNet-Canny
https://huggingface.co/SahilCarterr/Qwen-Image-Blockwise-ControlNet-Depth

[tanuj-rai]

Hey @yiyixuxu! I am interested in this issue.

[yiyixuxu]

@tanuj-rai look forward to your PR!

[Aishwarya0811]

Hi @yiyixuxu! 👋 I'm very interested in contributing the DiffSynth ControlNet support for Qwen-Image.

What I understand from the existing models:

• DiffSynth-Studio has already trained Qwen-Image ControlNet models (Canny, Depth, Inpaint) using a "Blockwise-ControlNet" architecture
• These models use blockwise_controlnet_inputs=[ControlNetInput(image=controlnet_image)] in their pipeline
• The current implementation requires DiffSynth-Studio's custom QwenImagePipeline

What needs to be implemented in diffusers:

1. QwenImageControlNet model class - following the pattern of FluxControlNet, SD3ControlNet
2. QwenImageControlNetPipeline - similar to existing ControlNet pipelines in diffusers
3. Support for the "blockwise" architecture used by DiffSynth models

Questions before I start:

• Should this follow the same API pattern as FluxControlNetPipeline and StableDiffusion3ControlNetPipeline?
• Are there any specific requirements for the blockwise ControlNet implementation?
• Should I start with one model type (e.g., Canny) and expand from there?

I'd be happy to take this on!

[MengAiDev]

LGTM! I love to work on this.

[akiseakusa]

Hi @ALL,

I’ve opened a draft PR for this: #12317.
I wasn’t able to run full GPU tests yet due to resource limitations, but I’ve aligned the pipeline changes with the DiffSynth Blockwise ControlNet implementation.

Any feedback, suggestions, or improvements would be greatly appreciated!

[aniket2405]

Hi @yiyixuxu
I see that the PR associated with this hasn't been worked on for some time now.
If this is still open for contribution, I'd like to take a look at this. Please let me know.

[aniket2405]

Please could you assign this to me, if this is open for contribution.
Thanks!

[aryabyte21]

Hi @yiyixuxu, I've opened #13268 to add DiffSynth blockwise ControlNet support. Based on your review feedback on #12317, I integrated it into the existing QwenImageControlNetModel with a controlnet_block_type config flag instead of creating separate classes. No pipeline changes needed and all existing tests pass. Would love your review when you get a chance!

[LeonxLJX]

I'd like to take this one ([Looking for community contribution] support DiffSynth Contr). I'll look into the root cause and follow up with a PR. (claiming via @LeonxLJX)