Beta-VAE FID GIVT

[adarshm-nd]

Does the beta-VAE for beta 5e-5 checkpointed in the GIVT readme file give the reconstruction FID of approx 4 as plotted in figure 5 in GIVT paper ?

[adarshm-nd]

Yes.

[mitscha]

Yes, correct. It's also used for the models in the colab in case you want to have a quick look at the reconstruction.

[adarshm-nd]

Thanks . Tested and got 4.26 FID for Beta-VAE with value 5e-5 and 50000 validation images

[mitscha]

That sounds plausible. In the paper we use the ADM evaluation suite throughout, which uses the train set to compute the FID reference statistics.

[adarshm-nd]

Thank you for the clarification

[adarshm-nd]

Actually, I wasn't able to get the VAE FID numbers in Fig 5 on the ADM suite. I used the weights on the GIVT github page for beta-VAE 5e-5, the pp_eval in the configuration file givt_imagenet2012, and reconstructed 1 pass of the ImageNet validation set with the VAE. To use pp_eval, I modified the Input_pipline.training function to use pp_eval instead of pp. I saved all reconstructed images as .png, uint8, with value range [0, 255]. Is there anything that I am missing? I have kept the rest of the VAE configuration on the givt_imagenet2012 file as is and Train=False in the vae.apply function. Any help would be much appreciated.
I would also like to note that I successfully reproduced FID scores reported in Table 1 of the paper for the transformers models, by using the ADM suite. I used the weights for GIVT-Causal (t=0.95, DB-CFG=0.4) and GIVT-MaskGIT (t_C=35, DB-CFG=0.1) from github. So, I only face an issue with the VAE when using the ADM suite. Thanks

[mitscha]

I believe the issue is the train flag: If train=False the VAE call won't run the reparametrization:

big_vision/big_vision/models/proj/givt/vae.py

Line 89 in 01edb81

z = self.reparametrize(mu, logvar)

Sorry for that, this behavior is not ideal.

Great to hear that you could reproduce Table 1.

[adarshm-nd]

Thank you so much for the quick response. I was able to get the results in the paper . Got an FID of ~3.5 for beta-vae 5e-5 and ~5.1 for beta-vae 1e-4 on the ADM suite. Thanks again for the clarification. Really appreciate the help