Skip to content

Latest commit

 

History

History
 
 

T2T_ViT

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet, arxiv

PaddlePaddle training/validation code and pretrained models for T2T-ViT.

The official pytorch implementation is here.

This implementation is developed by PaddleViT.

drawing

T2T-ViT Model Overview

Update

  • Update (2021-09-27): Model FLOPs and # params are uploaded.
  • Update (2021-08-18): Code is released and ported weights are uploaded.

Models Zoo

Model Acc@1 Acc@5 #Params FLOPs Image Size Crop_pct Interpolation Link
t2t_vit_7 71.68 90.89 4.3M 1.0G 224 0.9 bicubic google/baidu(1hpa)
t2t_vit_10 75.15 92.80 5.8M 1.3G 224 0.9 bicubic google/baidu(ixug)
t2t_vit_12 76.48 93.49 6.9M 1.5G 224 0.9 bicubic google/baidu(qpbb)
t2t_vit_14 81.50 95.67 21.5M 4.4G 224 0.9 bicubic google/baidu(c2u8)
t2t_vit_19 81.93 95.74 39.1M 7.8G 224 0.9 bicubic google/baidu(4in3)
t2t_vit_24 82.28 95.89 64.0M 12.8G 224 0.9 bicubic google/baidu(4in3)
t2t_vit_t_14 81.69 95.85 21.5M 4.4G 224 0.9 bicubic google/baidu(4in3)
t2t_vit_t_19 82.44 96.08 39.1M 7.9G 224 0.9 bicubic google/baidu(mier)
t2t_vit_t_24 82.55 96.07 64.0M 12.9G 224 0.9 bicubic google/baidu(6vxc)
t2t_vit_14_384 83.34 96.50 21.5M 13.0G 384 1.0 bicubic google/baidu(r685)

*The results are evaluated on ImageNet2012 validation set.

Notebooks

We provide a few notebooks in aistudio to help you get started:

*(coming soon)*

Requirements

Data

ImageNet2012 dataset is used in the following folder structure:

│imagenet/
├──train/
│  ├── n01440764
│  │   ├── n01440764_10026.JPEG
│  │   ├── n01440764_10027.JPEG
│  │   ├── ......
│  ├── ......
├──val/
│  ├── n01440764
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── ......

Usage

To use the model with pretrained weights, download the .pdparam weight file and change related file paths in the following python scripts. The model config files are located in ./configs/.

For example, assume the downloaded weight file is stored in ./t2t_vit_7.pdparams, to use the t2t_vit_7 model in python:

from config import get_config
from t2t_vit import build_t2t_vit as build_model
# config files in ./configs/
config = get_config('./configs/t2t_vit_7.yaml')
# build model
model = build_model(config)
# load pretrained weights, .pdparams is NOT needed
model_state_dict = paddle.load('./t2t_vit_7')
model.set_dict(model_state_dict)

Evaluation

To evaluate T2T-ViT model performance on ImageNet2012 with a single GPU, run the following script using command line:

sh run_eval.sh

or

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
    -cfg='./configs/t2t_vit_7.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./t2t_vit_7'
Run evaluation using multi-GPUs:
sh run_eval_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg='./configs/t2t_vit_7.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./t2t_vit_7'

Training

To train the T2T-ViT Transformer model on ImageNet2012 with single GPU, run the following script using command line:

sh run_train.sh

or

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
  -cfg='./configs/t2t_vit_7.yaml' \
  -dataset='imagenet2012' \
  -batch_size=16 \
  -data_path='/dataset/imagenet' \
Run training using multi-GPUs:
sh run_train_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python main_multi_gpu.py \
    -cfg='./configs/t2t_vit_7.yaml' \
    -dataset='imagenet2012' \
    -batch_size=32 \
    -data_path='/dataset/imagenet' \

Visualization Attention Map

(coming soon)

Reference

@article{yuan2021tokens,
  title={Tokens-to-token vit: Training vision transformers from scratch on imagenet},
  author={Yuan, Li and Chen, Yunpeng and Wang, Tao and Yu, Weihao and Shi, Yujun and Jiang, Zihang and Tay, Francis EH and Feng, Jiashi and Yan, Shuicheng},
  journal={arXiv preprint arXiv:2101.11986},
  year={2021}
}