Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet, arxiv
PaddlePaddle training/validation code and pretrained models for T2T-ViT.
The official pytorch implementation is here.
This implementation is developed by PaddleViT.
- Update (2021-09-27): Model FLOPs and # params are uploaded.
- Update (2021-08-18): Code is released and ported weights are uploaded.
Model | Acc@1 | Acc@5 | #Params | FLOPs | Image Size | Crop_pct | Interpolation | Link |
---|---|---|---|---|---|---|---|---|
t2t_vit_7 | 71.68 | 90.89 | 4.3M | 1.0G | 224 | 0.9 | bicubic | google/baidu(1hpa) |
t2t_vit_10 | 75.15 | 92.80 | 5.8M | 1.3G | 224 | 0.9 | bicubic | google/baidu(ixug) |
t2t_vit_12 | 76.48 | 93.49 | 6.9M | 1.5G | 224 | 0.9 | bicubic | google/baidu(qpbb) |
t2t_vit_14 | 81.50 | 95.67 | 21.5M | 4.4G | 224 | 0.9 | bicubic | google/baidu(c2u8) |
t2t_vit_19 | 81.93 | 95.74 | 39.1M | 7.8G | 224 | 0.9 | bicubic | google/baidu(4in3) |
t2t_vit_24 | 82.28 | 95.89 | 64.0M | 12.8G | 224 | 0.9 | bicubic | google/baidu(4in3) |
t2t_vit_t_14 | 81.69 | 95.85 | 21.5M | 4.4G | 224 | 0.9 | bicubic | google/baidu(4in3) |
t2t_vit_t_19 | 82.44 | 96.08 | 39.1M | 7.9G | 224 | 0.9 | bicubic | google/baidu(mier) |
t2t_vit_t_24 | 82.55 | 96.07 | 64.0M | 12.9G | 224 | 0.9 | bicubic | google/baidu(6vxc) |
t2t_vit_14_384 | 83.34 | 96.50 | 21.5M | 13.0G | 384 | 1.0 | bicubic | google/baidu(r685) |
*The results are evaluated on ImageNet2012 validation set.
We provide a few notebooks in aistudio to help you get started:
*(coming soon)*
- Python>=3.6
- yaml>=0.2.5
- PaddlePaddle>=2.1.0
- yacs>=0.1.8
ImageNet2012 dataset is used in the following folder structure:
│imagenet/
├──train/
│ ├── n01440764
│ │ ├── n01440764_10026.JPEG
│ │ ├── n01440764_10027.JPEG
│ │ ├── ......
│ ├── ......
├──val/
│ ├── n01440764
│ │ ├── ILSVRC2012_val_00000293.JPEG
│ │ ├── ILSVRC2012_val_00002138.JPEG
│ │ ├── ......
│ ├── ......
To use the model with pretrained weights, download the .pdparam
weight file and change related file paths in the following python scripts. The model config files are located in ./configs/
.
For example, assume the downloaded weight file is stored in ./t2t_vit_7.pdparams
, to use the t2t_vit_7
model in python:
from config import get_config
from t2t_vit import build_t2t_vit as build_model
# config files in ./configs/
config = get_config('./configs/t2t_vit_7.yaml')
# build model
model = build_model(config)
# load pretrained weights, .pdparams is NOT needed
model_state_dict = paddle.load('./t2t_vit_7')
model.set_dict(model_state_dict)
To evaluate T2T-ViT model performance on ImageNet2012 with a single GPU, run the following script using command line:
sh run_eval.sh
or
CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
-cfg='./configs/t2t_vit_7.yaml' \
-dataset='imagenet2012' \
-batch_size=16 \
-data_path='/dataset/imagenet' \
-eval \
-pretrained='./t2t_vit_7'
Run evaluation using multi-GPUs:
sh run_eval_multi.sh
or
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
-cfg='./configs/t2t_vit_7.yaml' \
-dataset='imagenet2012' \
-batch_size=16 \
-data_path='/dataset/imagenet' \
-eval \
-pretrained='./t2t_vit_7'
To train the T2T-ViT Transformer model on ImageNet2012 with single GPU, run the following script using command line:
sh run_train.sh
or
CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
-cfg='./configs/t2t_vit_7.yaml' \
-dataset='imagenet2012' \
-batch_size=16 \
-data_path='/dataset/imagenet' \
Run training using multi-GPUs:
sh run_train_multi.sh
or
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python main_multi_gpu.py \
-cfg='./configs/t2t_vit_7.yaml' \
-dataset='imagenet2012' \
-batch_size=32 \
-data_path='/dataset/imagenet' \
(coming soon)
@article{yuan2021tokens,
title={Tokens-to-token vit: Training vision transformers from scratch on imagenet},
author={Yuan, Li and Chen, Yunpeng and Wang, Tao and Yu, Weihao and Shi, Yujun and Jiang, Zihang and Tay, Francis EH and Feng, Jiashi and Yan, Shuicheng},
journal={arXiv preprint arXiv:2101.11986},
year={2021}
}