Performance on GenImage lower than reported

[Chuchad]

Hi, thank you for sharing this great work!
I was trying to reproduce the results on the GenImage dataset following the training and evaluation instructions in the repository, but I found that the reproduced performance is significantly lower than what was reported in the paper.
Following settings are chosen by myself, while others remain the same with the original paper.

Settings:

• CLIP-ConvNeXt pretained model: open_clip_pytorch_model.bin under laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg-soup
• resnet-50 pretained model: resnet50-0676ba61.pth under pytorch
• hardware: 1 * RTX4090

Results:

Subset	Acc
adm	0.7106
biggan	0.5447
glide	0.9565
midjourney	0.7913
sdv4	0.9993
sdv5	0.9989
vqdm	0.8103
wukong	0.9963
AVG	0.8510

As you can see, the Performance is lower than reported.
I set batchsize as 32 and choose the best ckpt in the 5 epochs, but I'did use 8 * A100 as reported in the paper. Since Learning rate is not changed, I am not sure if I need to scale lr according to my hardware setting. I wonder if you have seen such behavior in your experiment. Thank you.

[Johann356]

I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.

[Asyou-GD]

Isn't this pretty much the same?

[Johann356]

I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.

forgot to use pretrained resnet. Maybe the results will be the same if pretrained ones are used.

[lovegd-asyou]

I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.

CNNDetection: How much accuracy did you achieve after training for 20 epochs? The mean accuracy? I got a very low accuracy rate. I don't know why.

[Johann356]

I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.

CNNDetection: How much accuracy did you achieve after training for 20 epochs? The mean accuracy? I got a very low accuracy rate. I don't know why.

trained 20 epochs on CNNDetection (resnets forgot to use pretrained weight),epoch12:

CNNDetection 测试集

测试集	Accuracy	Avg Precision
biggan	0.72025	0.8000844691830253
crn	0.5365872767157631	0.6421978263673008
cyclegan	0.9072672218016654	0.9829853367473628
deepfake	0.5398704902867715	0.6892271403795274
gaugan	0.6069	0.7286883767469993
imle	0.5373707301786274	0.7834663542279399
progan	0.996375	0.9998983927237407
san	0.6917808219178082	0.8000423166181219
stargan	0.9942471235617809	0.999944842431401
stylegan	0.9831413787347688	0.998229921425742
stylegan2	0.9767776664997496	0.997156066106713
whichfaceisreal	0.889	0.96896608318801

GenImage 测试集

测试集	Accuracy	Avg Precision
adm	0.8910833333333333	0.9688910822097034
biggan	0.8599166666666667	0.9415706366912934
glide	0.9134166666666667	0.9832866998232219
midjourney	0.7631666666666667	0.8437958577052245
sd14	0.9064166666666666	0.9795961927299013
sd15	0.896556034752172	0.9721477597462465
vqdm	0.8668333333333333	0.9440023234856975
wukong	0.9003333333333333	0.9703778911169451

[lovegd-asyou]

{"train_lr": 5.07786304123261e-05, "train_loss": 0.23767979066172626, "train_class_acc": 0.9891468965043055, "train_min_lr": 5.07786304123261e-05, "test_loss": 0.5558900009078456, "test_acc1": 75.90385891815451, "test_acc5": 100.0, "epoch": 0, "n_parameters": 54432466}
{"train_lr": 5.076291203383308e-05, "train_loss": 0.21532221244741248, "train_class_acc": 0.9963892483607938, "train_min_lr": 5.076291203383308e-05, "test_loss": 0.3683272960285346, "test_acc1": 86.2042039547151, "test_acc5": 100.0, "epoch": 1, "n_parameters": 54432466}
{"train_lr": 5.073148378575147e-05, "train_loss": 0.21160123621401622, "train_class_acc": 0.9970669595100622, "train_min_lr": 5.073148378575147e-05, "test_loss": 0.35336238008435117, "test_acc1": 86.86634627273635, "test_acc5": 100.0, "epoch": 2, "n_parameters": 54432466}
{"train_lr": 4.709174377446268e-05, "train_loss": 0.20996936735510288, "train_class_acc": 0.997344709966432, "train_min_lr": 4.709174377446268e-05, "test_loss": 0.38198807171973165, "test_acc1": 85.49190874479763, "test_acc5": 100.0, "epoch": 3, "n_parameters": 54432466}
{"train_lr": 4.479838119291914e-05, "train_loss": 0.20844324520938898, "train_class_acc": 0.9976557902985046, "train_min_lr": 4.479838119291914e-05, "test_loss": 0.34143052139802044, "test_acc1": 87.17995910371359, "test_acc5": 100.0, "epoch": 4, "n_parameters": 54432466}
{"train_lr": 4.203944690266106e-05, "train_loss": 0.2073439945434702, "train_class_acc": 0.9978696579514783, "train_min_lr": 4.203944690266106e-05, "test_loss": 0.3502461508979941, "test_acc1": 87.3558110505868, "test_acc5": 100.0, "epoch": 5, "n_parameters": 54432466}。我正确配置了权重，貌似也无法到达作者的精度

[fucheng2004]

请问单卡4090训练20epoch要多久时间啊

[fucheng2004]

I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.

I would like to ask approximately how long it takes to train for 20 epochs on a single RTX 5090 in your setup.Thank you for your time and assistance.

[Johann356]

I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.

I would like to ask approximately how long it takes to train for 20 epochs on a single RTX 5090 in your setup.Thank you for your time and assistance.

I remember it took about a day or more