some problem in training

[leileihl]

I encountered an anomaly during training. At a certain batch size, P@1 dropped from 0.9 to 0.1.

[kongds]

Maybe gradient has exploration during training, what dataset did you use ?

[leileihl]

AmazonCat13k, i used apex with O1. When i used O0, model has not gradient explosion.