S3DIS performance

[TXH-mercury]

Hi zeliu
Thanks for the awesome work and open-source code!
I have encontered some trouble running the code. I changed nothing with the code and only use the default configs to train on S3DIS.
But I get the result(adaptive weights : 57.5mIOU ,pospool-xyz: 57.6mIOU ) which harshly lower than your report (66.5 , 66.5)
I can't figure out the performance gap , is there something I am supposed to change ? \

Another question is : In adaptive-weights method , the weight is a vector and not a matrix . Even if this can reduce the param nums and FLOPs, but how about the performance influence?

Thank you!

[TXH-mercury]

trained using pytorch , 4 gpu ,batch_size=8 (the same as the instructions)

[zeliu98]

Hi @TXH-mercury Thanks for your interest in our work.
The performance drop may due to the different reproduction details in PyTorch, like some hyper parameters and the masks in downsampling & neighborhoods. We will figure it out and update the code as soon as possible.
As for the weights, we use a depth-wise convolution and we found little influence compared with the normal convolution.

[alitabet]

Hi @zeliu98 thank you for your code. Have you managed to look into the issue of hyperparameters in the PyTorch version? If not, do you have some guidance as to which parameters we need to tune and some range of values for them? I'm trying to reproduce your results and get similar performance as @TXH-mercury, I can do a sweep of hyperparameters given the appropriate guidance.

Thanks again for your great work!

[zhulf0804]

Hi @zeliu98 , i ran the code you provided above trained on s3dis datasets with pytorch/cfgs/s3dis/pospool_sin_cos_avg.yaml.
I also can't get the higher iou as your paper. The result can be seen bellow:

Can you figure out what causes the low iou ?

Hi @TXH-mercury , have you solved the the lower metric problem ?

[zeliu98]

Hi all, we have released the pytorch models of S3DIS. Please let me you know if you have other questions.

[guochengqian]

@zeliu98 i am wondering if you are using the default parameter provided in the cfgs? all experiments are conducted using 4 GPUs? I have run S3DIS using pospool with 4 V100s. I get mIoU 64.5, which is fairly good, but still 1 point lower than your reported value. Is that because of the randomness or I used incorrect experiment setting (like not 4GPUs).

[tangbohu]

How you get the final performance. Evalute the models at 600 epochs, or the model with the best validation accuracy?