Discrepancies between paper method description and code

[lericson]

Hello, there are some discrepancies between the published work and the actual implementation you've published here. I'm curious to know what these differences are, exactly, and how I would go about obtaining an identical setup to the one presented in your work.

Masked convolutions The paper presents a method where an 80x80 mask is used, but that is not the case in this implementation.

Inflation It looks like you did not use inflation in the published results, however, the default is indeed using inflation. Is the user expected to turn this off?

Splitting test and training sets How did you split the KTH dataset to prevent overlap and memorization effects? There are a lot of duplicate floor plans in the dataset, so a sophisticated splitting technique is necessary.

Computation of evaluation metrics I cannot see how you computed your evaluation metrics. Does the calculation only include the predicted pixels, and is it the entire predicted image (i.e. the 256x256 image) or just the center 80x80, or is it the masked region? It is not specified in the published work.

Network predictive performance It would be helpful if you published your trained network; it could well be that the data collection phase is too difficult to reproduce faithfully. In that case, your work could still serve as a baseline if the predictor was "easily accessible" with pretrained weights.

I'll add more to this issue if I come across more issues.

[lericson]

Masked convolutions The paper presents a method where an 80x80 mask is used, but that is not the case in this implementation.

I have found how to change this in utils.py.

[rakeshshrestha31]

Hi,

I somehow missed this issue along with most others that were created on this repo until recently. I'll try to answer some questions very belatedly now (attn: #8).

Masked convolutions The setup in this repo indeed does not perfectly match the paper. The paper presented setup that gave us the best results while this repo contains our last setup, which was a mistake on my part.. To enable masked convolutions set MASK_SIZE_FROM_FRONTIER to False in utils/constants.py

Inflation The obstacles in the map are slightly inflated even if not explicitly mentioned in the paper (a point in the laser scan is mapped as 4x4 pixels)

Splitting test and training sets This was done manually. Unfortunately I have lost the original split due to hard drive failure

Computation of evaluation metrics The calculation only include the predicted pixels. Refer evaluation script

Pretrained weights Unfortunately I have lost the pretrained weights as well due to damaged hard drive

Best,
Rakesh