performance on Stanford2D3D is worse than reported

[apple2373]

I used the pretrained model ( https://drive.google.com/drive/folders/1b6mZJhF3j914AZ6TOGXrqgtGcHzHUAOc ) and do
$ CUDA_VISIBLE_DEVICES=0 python test.py
, which gave me

miniconda3/envs/omnifusion/lib/python3.7/site-packages/mmcv/__init__.py:21: UserWarning: On January 1, 2023, MMCV will release v2.0.0
, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names
 mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  'On January 1, 2023, MMCV will release v2.0.0, in which it will remove '
## patch size: (256, 256)
## fov: 80
## Number of first model parameters: 42466730
0it [00:00, ?it/s]/home/localstorage/miniconda3/envs/omnifusion/lib/python3.7/site-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling
 behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the do
cumentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode)
186it [04:29,  1.45s/it]
  Avg. Abs. Rel. Error: 0.0954
  Avg. Sq. Rel. Error: 0.0573
  Avg. Lin. RMS Error: 0.3718
  Avg. Log RMS Error: 0.1689
  Inlier D1: 0.8998
  Inlier D2: 0.9726
  Inlier D3: 0.9902

This is worse than the reported performance (Table 1 in the paper), which is,

• Avg. Abs. Rel. Error (lower better): 0.0950
• Avg. Sq. Rel. Error(lower better): 0.0491
• Avg. Lin. RMS Error(lower better): 0.3474
• Avg. Log RMS Error(lower better): 0.1599
• Inlier D1(higher better): 0.8988
• Inlier D2(higher better): 0.9769
• Inlier D3(higher better): 0.9924

Am I missing anything? Did anybody else get the same issue?

[apple2373]

Not only using the provided pretrained models, I also try training the model myself using train_erp_depth_iterative.py. The performance is even worse. This is what I get at its default settings.

  Avg. Abs. Rel. Error: 0.0981
  Avg. Sq. Rel. Error: 0.0596
  Avg. Lin. RMS Error: 0.3610
  Avg. Log RMS Error: 0.1688
  Inlier D1: 0.8930
  Inlier D2: 0.9706
  Inlier D3: 0.9906

After changing the learning rate schedule, weight decay, and the number of epochs, the best I got so fat is this following.

  Avg. Abs. Rel. Error: 0.0943
  Avg. Sq. Rel. Error: 0.0563
  Avg. Lin. RMS Error: 0.3726
  Avg. Log RMS Error: 0.1694
  Inlier D1: 0.8937
  Inlier D2: 0.9675
  Inlier D3: 0.9911

But this is still worse than what is reported except Avg. Abs. Rel. Error.

[Dharnidharka]

@apple2373 Did you need to change the spherical_fusion model parameters? I am trying to run the test for Stanford dataset using the pretrained model provided in the link(https://drive.google.com/drive/folders/1b6mZJhF3j914AZ6TOGXrqgtGcHzHUAOc) but am getting this error:

File "/home/ubuntu/OmniFusion/test.py", line 122, in
network.load_state_dict(ckpt)
File "/home/ubuntu/OmniFusion/omni-venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DataParallel:
size mismatch for module.down1.weight: copying a param with shape torch.Size([8, 512, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 512, 1, 1, 1]).
size mismatch for module.down1.bias: copying a param with shape torch.Size([8]) from checkpoint, the shape in current model is torch.Size([32]).

[apple2373]

yes, I describe it here: #3

[Dharnidharka]

@apple2373 Thank you so much, that worked