Trying to reproduce paper results.

[rabitwhte]

Do you perhaps have a script that invokes run_estimation.py multiple times with the right parameters to reproduce the main table with the results in the paper?

I'm trying to at least reproduce single number from that table (for CIFAR-10, for one of the methods, e.g. AC or COT itself) but Im getting different values than you get.

What parameters do you have to specify to get natural shift results for CIFAR-10? Is it enough to set --dirichlet_alpha to 0? Or in that case OOD will be just sampled from IID?

Out of curiosity - why didn't you include models in the repository? Too heavy? It would make things easier for people who want to reproduce (faster + no randomness from the stochastic process of fitting).

[luyuzhe111]

Hi @rabitwhte, thanks for your interest in our work! To your questions:

1. All scripts are under https://github.com/luyuzhe111/COT/tree/main/scripts. The parameters should correspond to the main table.
2. It's perfectly normal to get different values when a different model is used.
3. Yep it's sufficient to simply make it 0.
4. We didn't include it because we don't think the model checkpoints are the core piece of our work, as we believe the proposed method should work on any classifier. Previous works such as ATC did not share the classifier weights either so we trained our own. We provided all the hyperparameter settings we deemed relevant in Appendix D of our paper for reproduction purposes .

[rabitwhte]

Great, thanks for quick reply. I didn't notice the scripts out there!

[rabitwhte]

@luyuzhe111 I'm working on COT again. My goal is to reproduce table 1 results first. For CIFAR10 and COT for now.

Here is what I did:

1. Ran setup/setup_CIFAR10.sh
2. Ran scripts/train_CIFAR10.sh
3. Ran scripts/estimate_CIFAR10.sh (changed metrics to COT)

This produced ./results/CIFAR-10/scratch_ts/da_50/resnet18_1-300/COT_-1. There are all the corruptions results:

brightness.json  defocus_blur.json       frost.json           glass_blur.json        motion_blur.json
clean.json ...

Now, is clean the result that should correspond to Table 1, row CIFAR, row shift==Natural, column==COT? The result in the paper is 5.41 with very small std (in Table 2, it is 0.09). Here's what I get in clean.json:

{"corruption": "clean", "corruption level": 0, "metric": 0.8924537897109985, "ref": "COT", "acc": 0.08480120450258255, "error": 0.9151987954974174, "subpopulation": "same", "pretrained": false}

If I get it right this this estimation abs error |0.8924537897109985 - 0.9151987954974174|=0.0227. The table shows 5.41 (so probably 0.0541.

The only difference in the whole pipeline is that I trained CIFAR10 with resnet18 on batches of 100 instead of 200. I did not update the learning rate (maybe I should). At the end of training I got:

Epoch 300 Validation Acc: 0.9281
base model saved to ./checkpoints/CIFAR-10/resnet18/scratch/base_model_1-300.pt

What am I doing wrong?

[rabitwhte]

@luyuzhe111 A reminder on this :)