Imagenet-C Scores dependent on Python Version?

[ArneNx]

Hello,

I've been struggling to reproduce the ImageNet-C scores that you report in the README with a newer setup that I have.
I finally managed to reproduce it and found that the difference in performance is dependent on the environment, specifically changing the Python version von 3.8 to 3.10.6 seems to make a difference.

I uploaded the evaluation script (an updated and extended version of your script here), the two minimal Singularity-files for which it works and breaks, and the corresponding outputs here: https://gist.github.com/ArneNx/fd91d60cef787a7909c4e8fab2755d25
The outputs are for running python3 imagenet_c_test.py -m resnet50 in each container (i.e. evaluating the ResNet50 baseline).
The results are 76.70 for the old Python version and 76.43 for the new one.

My guess would be that the improved performance when switching to a newer version of Python has something to do with the image processing that might have changed and maybe removes some of the corruptions when loading the images?

[stsavian]

Thanks for replicating the results, I am also having troubles with this.