Support for controlnet lora

[adhikjoshi]

There's new control lora release,

https://huggingface.co/stabilityai/control-lora?s=09

How can we use it with diffusers?

[sayakpaul]

You cannot yet.

[haofanwang]

Any timeline for supporting it?

[MaxTran96]

any timeline for supporting this?

[yanchaoguo]

+1

[patrickvonplaten]

Working on it!

[patrickvonplaten]

@sayakpaul do you have bandwitdh for it? Otherwise I can look into picking it up :-)

[HighCWu]

I implemented a control-lora training code based on diffusers' controlnet. The idea is basically the same as stabilityai's. I found diffusers clear the lora layer of attention layers when xformers is enabled, so I did a little hack. At present, I have only tested the training and inference on sd v1.5. In theory, it can be compatible with sdxl's control lora by making some modifies.
Here is my repo HighCWu/control-lora-v2.

[patrickvonplaten]

It's being worked on: #4899

[HighCWu]

It's being worked on: #4899

Great work.

[yanchaoguo]

It's being worked on: #4899

Please make it merge into the main branch as soon as possible

[sayakpaul]

@HighCWu thanks for bringing your implementation to our attention. From what I can see from here, we're doing exactly the same in #4899. If you have any comments there let me know :)

[yanchaoguo]

It's being worked on: #4899

No effect! @sayakpaul @patrickvonplaten @HighCWu

please，when this problem can be solved:
【Results don't seem to be following the conditioning image at all. I will play with the hyperparameters a bit to see how the results get affected】

[MaxTran96]

Any eta on when this will be done?