SUE

This is the official PyTorch implementation of SUE from the paper "The Hidden Power of Unpaired Data for Multimodal Learning".

Installation

To run the project, clone this repo and then create a conda environment via:

conda env create -f environment.yml

Subsequently, activate this environment:

conda activate sue

Running

To run an example of the project on the retrieval task, follow these steps:

1. Download the model checkpoints and data encodings from here.

2. Unzip the downloaded files.

3. Locate:

   • The model checkpoint file: checkpoints_flickr30.pth (inside the checkpoints folder).
   • The data encodings: found under data/flickr30.

4. Run the following command:

python retrieval.py --test flickr30

5. If you want to train the model from scratch, use the following command:

python retrieval.py --train flickr30