1CYENS CoE, Cyprus, 2Concordia University
[Project Webpage] [Paper] [Video]
- 05.06.2025 - Pretrained checkpoints for Pix2Poly on the various datasets used in the paper are released. See pretrained checkpoints.
- 21.05.2025 - As reported by the authors of the $P^3$ dataset, Pix2Poly achieves state-of-the-art results for multimodal building vectorization from image and LiDAR data sources.
Extraction of building footprint polygons from remotely sensed data is essential for several urban understanding tasks such as reconstruction, navigation, and mapping. Despite significant progress in the area, extracting accurate polygonal building footprints remains an open problem. In this paper, we introduce Pix2Poly, an attention-based end-to-end trainable and differentiable deep neural network capable of directly generating explicit high-quality building footprints in a ring graph format. Pix2Poly employs a generative encoder-decoder transformer to produce a sequence of graph vertex tokens whose connectivity information is learned by an optimal matching network. Compared to previous graph learning methods, ours is a truly end-to-end trainable approach that extracts high-quality building footprints and road networks without requiring complicated, computationally intensive raster loss functions and intricate training pipelines. Upon evaluating Pix2Poly on several complex and challenging datasets, we report that Pix2Poly outperforms state-of-the-art methods in several vector shape quality metrics while being an entirely explicit method.
Overview of the Pix2Poly architecture: The Pix2Poly architecture consists of three major components: (i) The Discrete Sequence Tokenizer, (ii) the Vertex Sequence Detector, (iii) and the Optimal Matching Network. The Discrete Sequence Tokenizer is used to convert the continuous building corner coordinates into discrete building corner coordinate tokens which form the ground truth for training Pix2Poly. The Vertex Sequence Detector is an encoder-decoder transformer that predicts a sequence of sequence of discrete building corner coordinate tokens. The Optimal Matching Network takes the predicted corner coordinate tokens and the per-corner features from the vertex sequence detector and predicts a N X N permutation matrix which contains the connectivity information between every possible corner pair. Together, the predicted building corners and permutation matrix are used to recover the final building polygons.
Pix2Poly was developed with python=3.11, pytorch=2.1.2, pytorch-cuda=11.8, timm=0.9.12, transformers=4.32.1
Create a conda environment with the following specification:
Conda requirements:
channels:
- defaults
dependencies:
- torchvision=0.16.2
- pytorch=2.1.2
- pytorch-cuda=11.8
- torchaudio=2.1.2
- timm=0.9.12
- transformers=4.32.1
- pycocotools=2.0.6
- torchmetrics=1.2.1
- tensorboard=2.15.1
- pip:
- albumentations==1.3.1
- imageio==2.33.1
- matplotlib-inline==0.1.6
- opencv-python-headless==4.8.1.78
- scikit-image==0.22.0
- scikit-learn==1.3.2
- scipy==1.11.4
- shapely==2.0.4
See datasets preprocessing for instructions on preparing the various datasets for training/inference.
Pretrained checkpoints for the various datasets used in the paper are available for download at the following links: Google Drive | MEGA
Download the zip file, extract and place the individual runs folder in the runs directory at the root of the project.
Start training with the following command:
torchrun --nproc_per_node=<num GPUs> train_ddp.py
To generate predictions for the INRIA dataset, run the following:
python predict_inria_coco_val_set.py -d inria_coco_224_negAug \
-e <path_to_exp_folder> \
-c <checkpoint_name> \
-o <Name of output subdirectory>
python postprocess_coco_parts.py # change input and output paths in L006 to L010.To generate predictions for the Spacenet 2 dataset, run the following:
python predict_spacenet_coco_val_set.py -d spacenet_coco \
-e <path_to_exp_folder> \
-c <checkpoint_name> \
-o <Name of output subdirectory>
python postprocess_coco_parts.py # change input and output paths in L006 to L010.To generate predictions for the WHU Buildings dataset, run the following:
python predict_whu_buildings_coco_test_set.py -d whu_buildings_224_coco \
-e <path_to_exp_folder> \
-c <checkpoint_name> \
-o <Name of output subdirectory>
python postprocess_coco_parts.py # change input and output paths in L006 to L010.To generate predictions for the Massachusetts Roads dataset, run the following:
python predict_mass_roads_test_set.py -e <path_to_exp_folder> \
-c <checkpoint_name> \
-s <split> \ # 'test' or 'val'
--img_size 224 \
--input_size 224 \
--batch_size 24 \ # modify according to resourcesOnce predictions are made, metrics can be computed for the predicted files as follows:
python evaluation.py --gt-file path/to/ground/truth/annotation.json --dt-file path/to/prediction.json --eval-type <metric_type>where metric_type can be one of the following: ciou, angle, polis, topdig.
Once raster predictions are made for the Massachusetts Roads dataset, metrics can be computed for the predicted files as follows:
python evaluate_mass_roads_predictions.py --gt-dir data/mass_roads_1500/test/map --dt-dir path/to/predicted/raster/masks/folderIf you find our work useful, please consider citing:
@misc{adimoolam2024pix2poly,
title={Pix2Poly: A Sequence Prediction Method for End-to-end Polygonal Building Footprint Extraction from Remote Sensing Imagery},
author={Yeshwanth Kumar Adimoolam and Charalambos Poullis and Melinos Averkiou},
year={2024},
eprint={2412.07899},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.07899},
}This repository benefits from the following open-source work. We thank the authors for their great work.
Pix2Poly provides a Docker setup for easy deployment and inference. The Docker container includes a FastAPI server for REST API inference and supports command-line inference. The API request and response format are suitable for running as a AWS Sagemaker inference endpoint running on a ml.g4dn.xlarge and the inference AMI version al2-ami-sagemaker-inference-gpu-3-1, where it is able to infer at a rate of 5-10 seconds per tile.
docker buildx build --platform linux/amd64 -t pix2poly .The Docker container automatically starts a FastAPI server on port 8080. You can run it with:
docker run -p 8080:8080 pix2polyThe API server will automatically download the pretrained model files on first startup and provide the following endpoints:
POST /invocations- Main inference endpoint for processing imagesGET /ping- Health check endpoint
The /invocations endpoint accepts images in multiple formats:
- File Upload (multipart/form-data):
curl -X POST "http://localhost:8080/invocations" \
-H "Content-Type: multipart/form-data" \
-F "file=@your_image.jpg"- Base64 Encoded Image:
curl -X POST "http://localhost:8080/invocations" \
-H "Content-Type: application/json" \
-d '{"image": "base64_encoded_image_data"}'- Raw Image Data:
curl -X POST "http://localhost:8080/invocations" \
-H "Content-Type: image/jpeg" \
--data-binary @your_image.jpgThe API returns JSON with the detected polygons:
{
"polygons": [
[[x1, y1], [x2, y2], ...],
...
]
}
You can customize the Docker container behavior with these environment variables:
MODEL_URL: URL to download the pretrained model files (default:https://github.com/safelease/Pix2Poly/releases/download/main/runs_share.zip)API_KEY: Optional API key for authentication (if not set, authentication is disabled)
Example with custom configuration:
docker run -p 8080:8080 \
-e MODEL_URL=https://github.com/safelease/Pix2Poly/releases/download/main/runs_share.zip \
-e API_KEY=your_secret_key \
pix2poly