How to prepare datasets for other objects?

[Tariq-Abuhashim]

Hi;

I have two questions related to extending this framework to other objects:
(1) How to prepare the data and generate the weights and labels files for other objects (trucks, signs, cyclists, pedestrians) ?
(2) What would be the most efficient way to run multiple detections in this case? DSP-SLAM source is built around networks that are trained to work with cars, how efficient would it be to load weights based on detected object to allow it to work with multiple object classes?

[JingwenWang95]

Hi Tariq-Abuhashim,

Thanks for your interest! This is a very good question. Generally speaking, if you want to run on other object category you will need to train s DeepSDF model for that category (like what we've done with Redwood chairs). Among the 4 classes you mentioned, cyclists and pedestrains are deformable objects, so DeepSDF might not work for them. But for those two I guess you can try some simpler models, such as ellipsoids or cuboids.

Trucks are very similar to cars (as far as I know both trucks and cars are classified as 'cars' in ShapeNet), so it might have already been able to work with trucks, and you just need to enable the 'truck' class in 2D detector.

For signs I'm not sure if they exist in ShapeNet dataset, but I've got the feeling that using lines and planes might be a better representation.

So to run on multiple object categories you will need list of DeepSDF models and select the corresponding DeepSDF models for each detected class at run-time. Performance might drop, but I guess you could do something like only reconstruct 1 class at each key-frame or using multiple GPUs. Anyway, this requires major changes to the code and there might be a lot of engineering stuff involved.

To be honest I don't have a very satisfied answer to your questions at the moment, but I'm happy to have a further discussion with you along this direction. You can reach out to me via my email.

Best
Jingwen

[Tariq-Abuhashim]

Thanks for the answer, I will check.
Is it possible to integrate NeRF as well using your pipeline, in place of DSDF? It would be different in some aspects, but would that require major changes?

[JingwenWang95]

I guess you mean NeRF conditioned on latent code similar to DeepSDF? There is a paper from our group called CodeNeRF. I guess it is possible to optimize the object pose, shape code and texture code jointly using similar approach as in my code, but I'm not sure if Gauss Newton would work in this case. If not, it will be too slow for a SLAM system.

[Tariq-Abuhashim]

awesome thank you, good work, I will close this and contact you via email if I have more question.

[Tariq-Abuhashim]

One more question, for 2D detections, I managed to enable the detector to detect trucks. What about the 3D detector? do you have any ideas on how to do that? Thank you.

[JingwenWang95]

Hi! For the 3D detector it is trained on KITTI 3D which has three lables cars, pedestrains and cyclists, and in my code it should return all the detected cars. But I'm not sure if the car labels include trucks? If not, I've got the feeling that other newer dataset (such as Lyft, Waymo and Nuscenes) should have truck in their labels. You can try other pretrained models in mmdetection3d. They've got models trained on other datasets. But you will need to setup your own config files and change the code in 3D detector. It should be fairly easy to follow the tutorial and instructions from mmdetection3d.

Feel free to send me an email if you have further questions: [email protected]

[Tariq-Abuhashim]

Thank you, for all the ideas.