troubles when reproducing

[HanGuangXin]

Hi, thanks for the wonderful work.

But I had some troubles when trying to reproduce the results with command:

python3 projects/IDOL/train_net.py --config-file projects/IDOL/configs/ytvis19_r50.yaml --num-gpus 8 MODEL.WEIGHTS projects/IDOL/weights/cocopretrain_R50.pth SOLVER.IMS_PER_BATCH 16

The results is 46.96, which is lower than provided results 49.5. I'm using Torch 1.9.0, and batchsize was set to 16 instead of 32.

Is there something I missed? Looking forward to your reply.

[NingYuanxiang]

Hi, thanks for the wonderful work.

But I had some troubles when trying to reproduce the results with command:

python3 projects/IDOL/train_net.py --config-file projects/IDOL/configs/ytvis19_r50.yaml --num-gpus 8 MODEL.WEIGHTS projects/IDOL/weights/cocopretrain_R50.pth SOLVER.IMS_PER_BATCH 16

The results is 46.96, which is lower than provided results 49.5. I'm using Torch 1.9.0, and batchsize was set to 16 instead of 32.

Is there something I missed? Looking forward to your reply.

跑的这么快，几张卡啊大哥。STEPS、 MAX_ITER增大了吗？

[HanGuangXin]

@NingYuanxiang I then doubled MAX_ITER as I cut IMS_PER_BATCH by half. The results is 0.4710, which is still much lower than provided results 49.5. I'll take a look at STEPS.

[wjf5203]

Thanks for your attention!

After seeing your issue, I conducted more experiments to provide some reference results, hoping it can help.
First of all, in order to achieve the results reported in the paper, we need to infer at 480p resolution according to the description in Table 3, so please set 'MIN_SIZE_TEST' to 480.
Directly reducing batchsize will result in insufficient usage of the dataset, so I have made three settings for IMS_PER_BATCH and MAX_ITER, and each setting is repeated five times.
The settings and results are as follows:

Original:
IMS_PER_BATCH: 32
STEPS: (8000,)
MAX_ITER: 12000
Results (mAP): [50.5, 49.4, 49.8, 49.7, 48.8 ]

Setting1:
IMS_PER_BATCH: 16
STEPS: (8000,)
MAX_ITER: 12000
Results (mAP): [ 47.5, 47.1, 47.9, 47.4, 48.1]

Setting2:
IMS_PER_BATCH: 16
STEPS: (16000,)
MAX_ITER: 24000
Results (mAP): [49.0, 47.9, 48.5, 49.1, 48.6]

So, the reported results can be stably reproduced by default setting.
If you reduce the batchsize from 32 to 16, you may need to adjust the number of iterations appropriately.
Welcome to continue exploring the impact of different settings on training results and discuss here.

[HanGuangXin]

@wjf5203 Thanks for the result log. It seems that reducing batchsize from 2 image/GPU to 1 image/GPU will lose 1.0% mAP (48.62 v.s. 49.64), which is consistent with my results below.

Is it has something to do with BN layer when using a smaller batchsize? Will SyncBN or other methods be help?

Thanks in advance!

[wjf5203]

Yes, I think SyncBN will help. In addition, regarding STEPS and MAX_ITER, direct doubling may not be the best solution. They should be increased but increasing too much may lead to overfitting, considering the small scale of YTVIS2019.

[HanGuangXin]

Thanks for the insights, truly!

[9p15p]

Hi, Dr.Wu, Does 46.4 in Table 3. means randomly initialize the model in cocopretrain and inference it in 360P.

[aylinaydincs]

How can we see the results of a training, they are not in log.txt?