Awesome Multi-Modal 3D Detection

This is a collecttion of papers for LiDAR-camera fusion 3D detection methods.

We classify papers into three groups based on their fusion types: Early Fusion、Intermediate Fusion、Late Fusion.

If you find some overlooked papers, please open issues or pull requests. ⭐⭐⭐

Contents

• Early Fusion
• Intermediate Fusion
  • 2024
  • 2023
  • 2022
  • 2021
  • 2020
  • 2019
  • 2018
• Late Fusion

Early Fusion

PAI3D: Painting Adaptive Instance-Prior for 3D Object Detection
ECCVW 2022. [paper]

Multimodal Virtual Point 3D Detection
NeurIPS 2021. [paper] [code]

PointAugmenting: Cross-Modal Augmentation for 3D Object Detection
CVPR 2021. [paper] [code]

Frustum-PointPillars: A Multi-Stage Approach for 3D Object Detection using RGB Camera and LiDAR
ICCVW 2021. [paper] [code]

FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection
ITSC 2021. [paper]

PointPainting: Sequential Fusion for 3D Object Detection
CVPR 2020. [paper] [code]

Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds
CVPRW 2019. [paper] [code]

Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection
IROS 2019. [paper] [code]

RoarNet: A Robust 3D Object Detection based on RegiOn Approximation Refinement
IV 2019. [paper] [code]

Frustum PointNets for 3D Object Detection from RGB-D Data
CVPR 2018. [paper] [code]

Intermediate Fusion

2024

IS-FUSION: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
CVPR 2024. [paper] [code]

Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detection
ECCV 2024. [paper] [code]

Lift-Attend-Splat: Bird’s-eye-view camera-lidar fusion using transformers
CVPRW 2024. [paper] [code]

2023

MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detection
CVPR 2023. [paper] [code]

UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representation
ICCV 2023. [paper] [code]

Cross Modal Transformer: Towards Fast and Robust 3D Object Detection
ICCV 2023. [paper] [code]

SemanticBEVFusion: Rethinking LiDAR-Camera Fusion in Unified Bird's-Eye View Representation for 3D Object Detection
IROS 2023. [paper]

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation
ICRA 2023. [paper] [code]

Center Feature Fusion: Selective Multi-Sensor Fusion of Center-based Objects
ICRA 2023. [paper]

2022

TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
CVPR 2022. [paper] [code]

DeepInteraction: 3D Object Detection via Modality Interaction
NeurIPS 2022. [paper] [code]

Unifying Voxel-based Representation with Transformer for 3D Object Detection
NeurIPS 2022. [paper] [code]

AutoAlignV2: Deformable Feature Aggregation for Dynamic Multi-Modal 3D Object Detection
ECCV 2022. [paper] [code]

AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection
IJCAI 2022. [paper]

2021

Multi-Stage Fusion for Multi-Class 3D Lidar Detection
ICCVW 2021. [paper]

2020

3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection
ECCV 2020. [paper] [code]

EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection
ECCV 2020. [paper] [code]

2019

Multi-Task Multi-Sensor Fusion for 3D Object Detection
CVPR 2019. [paper] [code]

MVX-Net: Multimodal VoxelNet for 3D Object Detection
ICRA 2019. [paper]

2018

PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
CVPR 2018. [paper] [code]

Fusing Bird’s Eye View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection
IV 2018. [paper] [code]

Deep Continuous Fusion for Multi-Sensor 3D Object Detection
ECCV 2018. [paper]

Late Fusion

SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection
ICCV 2023. [paper] [code]

Fast-CLOCs: Fast Camera-LiDAR Object Candidates Fusion for 3D Object Detection
WACV 2022. [paper]

Cross-Modality 3D Object Detection
WACV 2021. [paper] [code]

CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection
IROS 2020. [paper] [code]

Multi-View 3D Object Detection Network for Autonomous Driving
CVPR 2017. [paper] [code]