Stability of `tiled_inference`

[lnicola]

I'm probably missing something, but tiled_inference doesn't seem to be very stable when changing the tile size, which is visible as checkerboard artifacts. sen1floods11_v1.1/data/S1GRDHand/USA_348639_S1Hand.tif seems to work fine. However, when using another, larger (7845x4968), image, the output looks like:

(this is LULC computed from an S1GRD with crop=256, stride=192, batch_size=16 from the example notebook).

Bumping crop to 512 produces very different results:

Increasing stride to 284 brings back the checkerboard pattern:

Is there a way to get a stable inference over a larger image?

PS: I also tried to use tiled_inference over an entire GRD product and, while it was working fine in the beginning, the speed dropped to ~0 about halfway. It might be related to the memory usage, but to write out some of the data, I'd have to implement tiling myself anyway, so the question of how to pick the tile sizes and stride remains.

PPS: here's the class 1 probability for Santiago.tif, this time with the default crop and stride. The tiling artifacts are still visible.

[blumenstiel]

Hi @lnicola, I would always use the size that a model was trained on. If you do LULC generations, this means 224x224. If I run the generation with crop=224, stride=192 on Santiago I get the following result:

If I increase it to your setting crop=512, stride=284 , I also see patchy generations. However, the model was never trained for this setting. If you finetune a model on Sen1Floods11 with 512x512 images, then this setting should work better. Transformer models sometimes produce artefacts along the edge. If that is the case, you can increase the default delta= 8 to 16 to cut off more of the edges.

About the probabilities: Did you visualised the logits (output of the model) or the probabilities (after softmax)? The logits can have very different scales for each chip, which results in the checkerboard pattern.

E.g. class idx 1 (water) for logits:

After applying pred = pred.softmax(dim=0), we get smoother probabilities:

fyi: I run the tests with the current main branch form terratorch.

[lnicola]

I would always use the size that a model was trained on. If you do LULC generations, this means 224x224.

That's what I assumed, however, on large inputs, it still results in checkerboarding like in my first screenshot (or used to, I'll have to try it again). Note that's a larger image, probably around 4000 pixels in width.

If you finetune a model on Sen1Floods11 with 512x512 images, then this setting should work better.

I'll give it a try after your fix. By the way, I think Sen1Floods11 is actually a bit larger than the archive linked in the notebook. Am I right to assume that the notebook is training only on the flood data, and not on the permanent water datasets? Sounds like combining the LULC and a fine-tuned model would not be a bad idea, in this case.

Did you visualised the logits (output of the model) or the probabilities (after softmax)?

I applied softmax. I'm not sure why yours looks smoother:

Can you zoom in a little?

Anyway, I'll have to retest, I think I'm still on terratorch 1.2.2 (not that I expect the terratorch version to make a difference).

[blumenstiel]

You are right, if I zoom in I also see some pattern.

I think that is hard to fix from the model output directly, as the probably is influenced by the context. E..g in this crop, the model is more uncertain if pixel belong to water or flooded vegetation (light green) as you can see in this screenshot:

I don't think you can fix that if you use TerraMind directly. It's better to finetune it for the task (Inference is also going to be faster because the decoder would be much smaller).

About Sen1Floods11: We use a the annotated subset mainly for demo purposes as it is pretty easy to use and quite small. You can of course use more data.

Don't think that there are big changes from 1.2.2 onwards.

[lnicola]

As a small update, I tried to run the inference multiple times, in a staggered pattern (roughly Golden ratio from https://www.graphics.rwth-aachen.de/media/papers/jgt.pdf), then keep the median for each pixel in the stack:

Single run:

5 runs:

I can't say for sure that it works well, but it's probably not terrible.

[blumenstiel]

Cool, thanks for sharing!