What's the implementation difference between highres fix and SD upscale?

[nothings]

Reddit posts on this question are not at all clear or informative. According to the docs they both split into tiles and then use SD on the upscaled tiles, but they behave very differently.

SD Upscale can leave obvious areas where it's crossfading between the tiles, which I never really notice in Highres fix.

Highres fix tends to duplicate faces across the whole image, whereas SD Upscale seems to, at a minimum, generate different faces in each tile.

So presumably they must be doing very different things under the hood to be able to have such different effects.

[PaperOrb]

Did you ever find an explanation for this?

[nothings]

No, I still have no idea.

I played with using ultimate SD upscale, but I settled on using hires fix and then using the Batch Face Swap extension to make faces distinct.

[noisefloordev]

I was curious about this too. I'm pretty sure "SD upscale" is just what it looks like: img2img each tile, then blend them back together, so it can work with arbitrary image sizes (just more independant tiles) but denoising strength has to be very low or the tiles won't line up. You could do this manually (tediously) by upscaling the image, breaking it into slightly overlapping tiles, running img2img on each tile, then blending the tiles back together.

"Hires Fix" is less obvious. It seems similar to taking the low-res image, upscaling it (at least for the non-latent upscale modes), and putting the result in img2img, but if I do that manually the results are very different, and hires fix gives better results (it seems less likely to mutate the image incorrectly). Hires fix also says it's operating on tiles on the console, which a simple high-res img2img doesn't, though hires fix still OOMs quickly for me so I'm not sure what's actually being processed in tiles.

I guess I should spend a bit of time poking at the code and seeing what's actually happening, but at least as far as initial upscales go, hires fix is definitely better. "SD upscale" does have some benefits: it isn't VRAM-limited like hires fix (where I OOM if I go beyond around 768x768 x2), and it works on existing images where hires fix only works during image generation.

[nothings]

I did skim through the code, but with no familiarity of the core of Stable Diffusion, I had no idea what it was doing. I didn't see it actually doing anything tile-ish, though, but I could easily have missed it. It sure seemed straightforward in its interaction with SD.

As to the possiblity it's not tiled, R-ESRGAN dumps the list of tiles that it's upscaling, but I thought maybe it was possible that was just part of R-ESRGRAN upscaling internally (since none of the other upscalers I use dump that info), and not the actual hires fix processing. However, the hires scale SD iteration count in the console is seems like it's always a multiple of the specified step count, so I assume the iteration count is the number of tiles times the raw step count? But I don't really know, I gave up on figuring it out and haven't thought about it in a while.

[git4ai]

I've always had that question too, and you've asked my doubts and longed for an answer.

[catboxanon]

OP's answer is mostly correct. If you enable the live preview in settings it's more obvious that the SD upscaler script upscales in tiles.

Really the only point in using SD upscale or Ultimate SD upscaler is if you're VRAM limited or upscaling something huge (also due to VRAM), because it performs worse at a higher denoise value since the prompt then no longer directly correlates to a full image but instead individual tiles.

Technically all the GAN upscalers should be directly upscaling the full image to 2x/4x as well (this is why the filenames usually contain these), but they're done as tiles by default instead to also save VRAM. This can be changed in settings under Upscaling.

[iSupremacy]

Really the only point in using SD upscale or Ultimate SD upscaler is if you're VRAM limited or upscaling something huge (also due to VRAM), because it performs worse at a higher denoise value since the prompt then no longer directly correlates to a full image but instead individual tiles.

But even it if truly is HiRes sadly isn't usuable after the fact, only then and their. I like to just do a shit load of gens then do evaluations later. That's why USDU is main tool

[davedfx]

From what I got. And I could be wrong. Txt2Img Highrez fix performs the Upscaling in latent space without converting to pixel space. Img2Img converts pixels to latent.
GAN upscalers works in pixel space.

[catboxanon]

Txt2Img Highrez fix performs the Upscaling in latent space without converting to pixel space

This is only true if you choose a latent upscaler. Latent upscalers actually perform worse because upscaling in the latent space is incredibly noisy and requires a higher denoise to clean up, which eventually diverges off from the original image, so it's better to use anything else, such as a GAN-based upscaler.