Continual pre-training for TerraMind

[faurefrancois]

Hi!

We already fine-tuned TerraMind backbone + UPerNet for geomorphological objects segmentation on satellite images (and it gives the best results among other models! A paper is in preparation about the potential of TerraMind for geomorphology ^^). The next idea we have is to do continual pre-training to make TerraMind more expert on our task and data. If I'm not wrong, ReconstructionTask is not adapted to TerraMind. This way, is there a task or a framework that we could use to do a continual pre-training. For now, we tried a naive approach by using the RegressionTask to reconstruct some satellite modalities from other ones. Of course, this is not very efficient but that was the way directly using the classic TerraMind fine-tuning framework (in this case used for continual pre-training, with backbone unfreeze).

Thank you!

[blumenstiel]

Hi @faurefrancois, great to hear that TerraMind works well for you task, thanks for sharing.
The pretraining code is not released. Mainly because it is a lot of work due to the two stage approach with tokenization, very custom to our setup, and because the pretraining code is still changing a lot.

Doing continuous pretraining with a reconstruction objective is probably a good way to boost performance while limiting the required effort. If you want to use something closer to TM pretraining and you have a subset of the pretraining modalities, you can create tokens with the tokenizer and write a simple training pipeline yourself (should be simple with the current LLM tools). But the resulting model might generalize less than after pretraining, depending on how big and diverse your data is.