Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar

[brunosan]

https://sustainability.fb.com/blog/2024/04/22/using-artificial-intelligence-to-map-the-earths-forests/
https://github.com/facebookresearch/HighResCanopyHeight/blob/main/README.md

This is an icnredibly interesting paper, that seems to include making a version of Clay, and also applies it to a very specific usecase.

Vegetation structure mapping is critical for understanding the global carbon cycle and monitoring nature-based
approaches to climate adaptation and mitigation. Repeated measurements of these data allow for the observation
of deforestation or degradation of existing forests, natural forest regeneration, and the implementation of sustainable agricultural practices like agroforestry. Assessments of tree canopy height and crown projected area at a
high spatial resolution are also important for monitoring carbon fluxes and assessing tree-based land uses, since
forest structures can be highly spatially heterogeneous, especially in agroforestry systems. Very high resolution
satellite imagery (less than one meter (1 m) Ground Sample Distance) makes it possible to extract information at
the tree level while allowing monitoring at a very large scale. This paper presents the first high-resolution canopy
height map concurrently produced for multiple sub-national jurisdictions. Specifically, we produce very high
resolution canopy height maps for the states of California and S˜
ao Paulo, a significant improvement in resolution
over the ten meter (10 m) resolution of previous Sentinel / GEDI based worldwide maps of canopy height. The
maps are generated by the extraction of features from a self-supervised model trained on Maxar imagery from
2017 to 2020, and the training of a dense prediction decoder against aerial lidar maps. We also introduce a postprocessing step using a convolutional network trained on GEDI observations. We evaluate the proposed maps
with set-aside validation lidar data as well as by comparing with other remotely sensed maps and field-collected
data, and find our model produces an average Mean Absolute Error (MAE) of 2.8 m and Mean Error (ME) of 0.6
m

[brunosan]

Claude summary:

This paper presents a novel approach for generating high-resolution (< 1 m) canopy height maps (CHMs) from RGB satellite imagery using deep learning. The method leverages self-supervised pre-training on a large dataset of unlabeled images, followed by supervised fine-tuning on a smaller dataset of aerial LiDAR measurements.

Datasets:

1. Self-supervised pre-training: 18 million 256x256 pixel patches from Maxar satellite imagery (WorldView-2, WorldView-3, and QuickBird) at 0.5 m resolution, covering global locations from 2017-2020. The dataset is used without any labels.

2. Supervised fine-tuning: ~5,800 aerial LiDAR canopy height models (CHMs) from the National Ecological Observatory Network (NEON) across the United States. Each CHM covers a 1 km × 1 km area at 1 m resolution. The LiDAR data is split into 80% training, 10% validation, and 10% test sets, with no spatial overlap between sets.

3. GEDI calibration: Spaceborne LiDAR data from the Global Ecosystem Dynamics Investigation (GEDI) mission, providing near-global coverage of canopy height estimates at a 25 m footprint. The GEDI data is used to train a separate CNN model for calibrating the high-res CHM predictions.

Model Architecture and Training:

1. Self-supervised pre-training: A Vision Transformer (ViT) encoder is pre-trained on the Maxar dataset using the DINOv2 approach. The model learns to match the output distributions of a student and teacher network fed with differently augmented views of the same image. The pre-training takes ~3 days on 16 GPUs, resulting in a model with 303M (ViT-Large) or 606M (ViT-Huge) parameters.

2. Supervised fine-tuning: A Dense Prediction Transformer (DPT) decoder is attached to the pre-trained ViT encoder and fine-tuned on the NEON LiDAR data to predict canopy height. The encoder weights are frozen, and only the decoder (34.2M parameters) is trained for 140k steps using a combination of regression and classification losses. Fine-tuning takes 8-9 hours on 8 GPUs.

3. GEDI calibration: A separate CNN model is trained on GEDI data to predict canopy height at a lower resolution (76 m). The CNN takes the Maxar images, GEDI height estimates, and additional metadata (latitude, longitude, view angle, sun angle, terrain slope) as input. The GEDI model is used to calibrate the high-res CHM predictions by rescaling them based on the lower-res height estimates.

Evaluation and Results:

• The method is evaluated on held-out test sets from NEON and two other LiDAR datasets covering different geographies and forest types. The proposed approach achieves state-of-the-art performance, with an average Mean Absolute Error (MAE) of 2.8 m and a Mean Error (ME) of 0.6 m across all test sets.
• Ablation studies show that both the self-supervised pre-training and the GEDI calibration step significantly improve the model's accuracy and generalization to new areas.
• The model also performs well on a tree segmentation task, achieving an average Intersection-over-Union (IoU) score of 0.77 on a globally-distributed, human-annotated dataset.
• The resulting CHMs are generated at a 0.5 m resolution for the entire states of California and São Paulo, covering a diverse range of forest types and topographies.

In summary, this work demonstrates the effectiveness of combining self-supervised learning, transformers, and multi-modal data for high-resolution canopy height mapping. The proposed approach leverages a large volume of unlabeled satellite imagery to learn a generic visual representation, which is then adapted to the specific task of canopy height estimation using a smaller dataset of LiDAR measurements. The results show state-of-the-art performance and good generalization to new geographies, highlighting the potential of this method for large-scale, high-resolution forest monitoring applications.