A new pipeline for inpainting(object removal) is needed!!

[lawsonxwl]

Is your feature request related to a problem? Please describe.
Thanks for your great work!!
Right now I'm using StableDiffusionInpaintPipeline with fixed prompt(such as "the empty background") to do object removal task, but one thing i've found is that when the inpaint mask doesn't cover the shadow or hairline, the model will randomly generate objects else in the mask rather than generating pure background, which is frustrating. I wonder if the inpaint pipeline is trained for removing object ? If not, when will the inpaint model trained for object removal be released(works like LDM inpainting or LaMa)? As this feature can be really useful in practice!

Describe the solution you'd like
Plz release the inpaint pipeline specialized for object removal!

Describe alternatives you've considered
Give some advice about getting rid of unwanted object generation

unwanted object generation:

[GitHub1712]

Would be cool if we could train tasks like that ourself. I am missing inpaint training code example still very much.

[lawsonxwl]

Would be cool if we could train tasks like that ourself. I am missing inpaint training code example still very much.

By modifying txt2img training code, inpainting training code can be easy to get.. But controlling the object generated by the model can be really hard!

[GitHub1712]

How could we do that? How did the inpaint models learned their abilty to continue structures against txt2img models?

[lawsonxwl]

How could we do that? How did the inpaint models learned their abilty to continue structures against txt2img models?

By finetuning on txt2img models and using random masking strategy, Stability AI trained and released the stable-diffusion inpainting model.. You can check it out on diffusers

[GitHub1712]

Thank you, I want to train own if possible based on inpainting sdxl, is there any documentation how this was exactly trained and created, training example code?
I am missing that for all inpaint models...

[donggoing]

longing for the example code for finetuning, too.

[sayakpaul]

@asomoza do we have any examples/resources for this now?

[asomoza]

we have now object removal training-free guides and models but the discussion was also about training code examples for inpainting which we don't have yet.