Feature/qwen eligen support

[nolan4]

Pull Request: Add Entity-Level Image Generation (EliGen) for Qwen Image

Summary

This update implements Entity-Level Image Generation (EliGen) for the Qwen Image model, allowing region-specific prompts through spatial masks. The feature provides fine-grained control over image generation by applying separate attention masks for each entity.

Key Features
• Spatial attention masking with isolated entity prompts
• Automatic mask resizing to match latent dimensions
• RoPE embedding implementation aligned with DiffSynth Studio
• Support for batch_size > 1
• Backward compatible with standard Qwen Image workflows

[krigeta]

Hey @nolan4 thank you so much for this amazing work! I am trying to test this out but not able to run it, I am using this PR and using the Eligen Lora provided by Diffsynth Studio and using this workflow, please check.

[krigeta]

hey @nolan4 may you please reply?

[nolan4]

Hi @krigeta — here’s a screenshot of my workflow, which is based on the Qwen text-to-image template. I’m also using the same EliGen LoRA from Diffsynth Studio that you linked. Hope this helps you get it running!

[krigeta]

Hey @nolan4 thank you so much for this, what if this branch is not merged then is it possible to create a custom node of this?

and yeah i will test this and share the results for sure.

[krigeta]

hey @nolan4 it is not working in my case, please check.

[nolan4]

Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications:

[Kosinkadink]

test locally and works, comfy will do a code review to see if anything else needs to be changed!

[krigeta]

Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications:

Will gonna test it asap and this time I am gonna test the official example as shown in the official Eligen lora implementation and one more thing I want to ask:

In the Diffsynth studio repo, they said it is important to make those colored masks with text overlays. Is that true?

As in their official example, the masks are overlaid to achieve smooth results.

[Amazon90]

How can I get this node?

[krigeta]

How can I get this node?

you have to install this PR and not the actual comfyUI setup to use this node, as of now it is not the part of the main repo.

[krigeta]

Hey @nolan4, I guess this implementation is missing the colour-coded masks that help the lora to differentiate between the regions when they overlap. Please look into it.

[geroldmeisinger]

btw something similar was implemented in the Inspire Pack called "regional conditioning by color masks" in case you need inspiration or code
https://github.com/ltdrdata/ComfyUI-extension-tutorials/blob/Main/ComfyUI-Inspire-Pack/workflow/RegionalCFG-RegionalConditioning.png

https://github.com/ltdrdata/ComfyUI-Inspire-Pack

[geroldmeisinger]

How can I get this node?

comfy-cli --workspace ./ComfyUI_eligen install --pr "#10473"
comfy-cli --workspace ./ComfyUI_eligen launch

[geroldmeisinger]

Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications

I can confirm it works: 8step lora -> eligen lora -> ksampler

I guess this implementation is missing the colour-coded masks that help the lora to differentiate between the regions when they overlap. Please look into it.

I can confirm it works

eligen_colorspheres.json

note i use euler+beta, cfg=1, 1328x1328 but that shouldn't make much difference

[krigeta]

Hey @geroldmeisinger , thank you so much for the share, is there any other social media where we can chat?

[geroldmeisinger]

here are the original masks https://www.modelscope.cn/datasets/DiffSynth-Studio/examples_in_diffsynth/files data/examples/eligen/entity_control
of the eligen demo page https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen

[krigeta]

here are the original masks https://www.modelscope.cn/datasets/DiffSynth-Studio/examples_in_diffsynth/files data/examples/eligen/entity_control of the eligen demo page https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen

These are the ones I got to know about Eligen. In my case, the overlapped masking is not working properly, as I want to create two characters in front of each other, and the view is from the back. Or I would say I use one entity for a character and another for another character, and with ControlNet? Does it work?

And if possible, may you share your Discord or other social app?