Awesome Failure Detection / Reliable Prediction

Failure Detection is a machine learning problem, which aims to detect out-of-distribution (OOD) and misclassified samples based on reliable confidence estimation. This topic is important for risk-sensitive applications (e.g., autonomous driving, clinical decision making), and is gathering much attention in the research community.

Here, we provide a list of papers that studies OOD detection and misclassification detection (MisD).

Misclassification Detection / Selective Classification / Failure Prediction

• Unified Out-Of-Distribution Detection: A Model-Specific Perspective (ICCV 2023) [paper]
• OpenMix: Exploring Outlier Samples for Misclassification Detection (CVPR 2023) [paper] [code]
• Towards More Reliable Confidence Estimation (TPAMI 2023) [paper]
• Failure Detection for Motion Prediction of Autonomous Driving: An Uncertainty Perspective (ICRA 2023) [paper]
• A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification (ICLR 2023) [paper] [code]
• What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers (ICLR 2023) [paper] [code]
• Towards Better Selective Classification (ICLR 2023) [paper] [code]
• AUC-based Selective Classification (AISTATS 2023) [paper]
• Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed (TMLR 2022) [paper] [code]
• Rethinking Confidence Calibration for Failure Prediction (ECCV 2022) [paper] [code]
• Improving the Reliability for Confidence Estimation (ECCV 2022) [paper]
• Trust, but Verify: Using Self-Supervised Probing to Improve Trustworthiness (ECCV 2022) [paper]
• Confidence estimation via auxiliary models (TPAMI 2021) [paper]
• Learning to predict trustworthiness with steep slope loss (NeurIPS 2021) [paper] [code]
• Confidence-Aware Learning for Deep Neural Networks (ICML 2020) [paper] [code]
• Self-Adaptive Training: beyond Empirical Risk Minimization (NeurIPS 2020) [paper] [code]
• Selectivenet: A deep neural network with an integrated reject option (ICML 2019) [paper]
• Addressing Failure Prediction by Learning Model Confidence (NeurIPS 2019) [paper] [code]
• Deep Gamblers: Learning to Abstain with Portfolio Theory (NeurIPS 2019) [paper] [code]
• To Trust Or Not To Trust A Classifier (NeurIPS 2018) [paper] [code]
• Selective classification for deep neural networks (NeurIPS 2017) [paper]
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (ICLR 2017) [paper]
• An Optimum Character Recognition System Using Decision Functions (IRE Transactions on Electronic Computers 1957) [paper]

OOD Detection

2023

• Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detection (Arxiv 2023) [paper]
• WDiscOOD: Out-of-Distribution Detection via Whitened Linear Discriminant Analysis (ICCV 2023) [paper]
• DIFFGUARD: Semantic Mismatch-Guided Out-of-Distribution Detection Using Pre-Trained Diffusion Models (ICCV 2023) [paper] [code]
• Understanding the Feature Norm for Out-of-Distribution Detection (ICCV 2023) [paper]
• Nearest Neighbor Guidance for Out-of-Distribution Detection (ICCV 2023) [paper]
• Residual Pattern Learning for Pixel-Wise Out-of-Distribution Detection in Semantic Segmentation (ICCV 2023) [paper] [code]
• Unified Out-Of-Distribution Detection: A Model-Specific Perspective (ICCV 2023) [paper]
• Revisit PCA-based technique for Out-of-Distribution Detection (ICCV 2023) [paper] [code]
• Hierarchical Visual Categories Modeling: A Joint Representation Learning and Density Estimation Framework for Out-of-Distribution Detection (ICCV 2023) [paper]
• Out-of-Distribution Detection for Monocular Depth Estimation (ICCV 2023) [paper]
• Simple and Effective Out-of-Distribution Detection via Cosine-based Softmax Loss (ICCV 2023) [paper]
• Unsupervised Out-of-Distribution Detection with Diffusion Inpainting (ICML 2023) [paper]
• Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection (ICML 2023) [paper] [code]
• In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation (ICML 2023) [paper] [code]
• Hybrid Energy Based Model in the Feature Space for Out-of-Distribution Detection (ICML 2023) [paper] [code]
• Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability (ICML 2023) [paper] [code]
• Detecting Out-of-distribution Data through In-distribution Class Prior (ICML 2023) [paper] [code]
• Concept-based Explanations for Out-of-Distribution Detectors (ICML 2023) [paper] [code]
• Decoupling MaxLogit for Out-of-Distribution Detection (CVPR 2023) [paper] [code]
• Block Selection Method for Using Feature Norm in Out-of-Distribution Detection (CVPR 2023) [paper] [code]
• LINe: Out-of-Distribution Detection by Leveraging Important Neurons (CVPR 2023) [paper] [code]
• Uncertainty-Aware Optimal Transport for Semantically Coherent Out-of-Distribution Detection (CVPR 2023) [paper] [code]
• Rethinking Out-of-Distribution (OOD) Detection: Masked Image Modeling Is All You Need (CVPR 2023) [paper] [code]
• Balanced Energy Regularization Loss for Out-of-Distribution Detection (CVPR 2023) [paper]
• Detection of Out-of-Distribution Samples Using Binary Neuron Activation Patterns (CVPR 2023) [paper] [code]
• GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection (CVPR 2023) [paper] [code]
• OpenMix: Exploring Outlier Samples for Misclassification Detection (CVPR 2023) [paper] [code]
• Packed-Ensembles for Efficient Uncertainty Estimation (ICLR 2023) [paper] [code]
• A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet (ICLR 2023) [paper] [code]
• Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy (ICLR 2023) [paper] [code]
• Extremely Simple Activation Shaping for Out-of-Distribution Detection (ICLR 2023) [paper] [code]
• The Tilted Variational Autoencoder: Improving Out-of-Distribution Detection (ICLR 2023) [paper]
• Out-of-distribution Detection with Implicit Outlier Transformation (ICLR 2023) [paper] [code]
• Energy-based Out-of-Distribution Detection for Graph Neural Networks (ICLR 2023) [paper] [code]
• How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection? (ICLR 2023) [paper] [code]
• Harnessing Out-Of-Distribution Examples via Augmenting Content and Style (ICLR 2023) [paper]
• Fake It Until You Make It : Towards Accurate Near-Distribution Novelty Detection (ICLR 2023) [paper] [code]
• Non-parametric Outlier Synthesis (ICLR 2023) [paper] [code]
• Out-of-Distribution Detection and Selective Generation for Conditional Language Models (ICLR 2023) [paper]
• Turning the Curse of Heterogeneity in Federated Learning into a Blessing for Out-of-Distribution Detection (ICLR 2023) [paper] [code]

2022

• Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities (ICML 2022) [paper] [code]
• POEM: Out-of-Distribution Detection with Posterior Sampling (ICML 2022) [paper] [code]
• Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition (ICML 2022) [paper] [code]
• Scaling Out-of-Distribution Detection for Real-World Settings (ICML 2022) [paper] [code]
• Mitigating Neural Network Overconfidence with Logit Normalization (ICML 2022) [paper] [code]
• Out-of-Distribution Detection with Deep Nearest Neighbors (ICML 2022) [paper] [code]
• Training OOD Detectors in their Natural Habitats (ICML 2022) [paper] [code]
• Out-of-Distribution Detection via Conditional Kernel Independence Model (NeurIPS 2022) [paper] [code]
• Your Out-of-Distribution Detection Method is Not Robust! (NeurIPS 2022) [paper] [code]
• Boosting Out-of-distribution Detection with Typical Features (NeurIPS 2022) [paper] [code]
• RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection (NeurIPS 2022) [paper]
• Provably Adversarially Robust Detection of Out-of-Distribution Data (Almost) for Free (NeurIPS 2022) [paper] [code]
• Watermarking for Out-of-distribution Detection (NeurIPS 2022) [paper] [code]
• Is Out-of-Distribution Detection Learnable? (NeurIPS 2022) [paper]
• RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy & Out-of-Distribution Robustness (NeurIPS 2022) [paper] [code]
• Out-of-Distribution Detection with An Adaptive Likelihood Ratio on Informative Hierarchical VAE (NeurIPS 2022) [paper]
• GraphDE: A Generative Framework for Debiased Learning and Out-of-Distribution Detection on Graphs (NeurIPS 2022) [paper] [code]
• Density-driven Regularization for Out-of-distribution Detection (NeurIPS 2022) [paper]
• Delving into Out-of-Distribution Detection with Vision-Language Representations (NeurIPS 2022) [paper] [code]
• Beyond Mahalanobis Distance for Textual OOD Detection (NeurIPS 2022) [paper]
• SIREN: Shaping Representations for Detecting Out-of-Distribution Objects (NeurIPS 2022) [paper] [code]
• Out-of-distribution Detection with Boundary Aware Learning (ECCV 2022) [paper]
• Out-of-Distribution Detection with Semantic Mismatch under Masking (ECCV 2022) [paper] [code]
• Data Invariants to Understand Unsupervised Out-of-Distribution Detection (ECCV 2022) [paper]
• Out-of-Distribution Identification: Let Detector Tell Which I Am Not Sure (ECCV 2022) [paper]
• Deep Hybrid Models for Out-of-Distribution Detection (CVPR 2022) [paper]
• Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection (CVPR 2022) [paper]
• ViM: Out-of-Distribution With Virtual-Logit Matching (CVPR 2022) [paper] [code]
• Neural Mean Discrepancy for Efficient Out-of-Distribution Detection (CVPR 2022) [paper]
• Unknown-Aware Object Detection: Learning What You Don’t Know from Videos in the Wild (CVPR 2022) [paper] [code]
• Revisiting flow generative models for Out-of-distribution detection (ICLR 2022) [paper]
• Igeood: An Information Geometry Approach to Out-of-Distribution Detection (ICLR 2022) [paper]
• A Statistical Framework for Efficient Out of Distribution Detection in Deep Neural Networks (ICLR 2022) [paper]
• VOS: Learning What You Don't Know by Virtual Outlier Synthesis (ICLR 2022) [paper] [code]

2021

• Understanding Failures in Out-of-Distribution Detection with Deep Generative Models (ICML 2021) [paper]
• Exploring the Limits of Out-of-Distribution Detection (NeurIPS 2021) [paper]
• STEP: Out-of-Distribution Detection in the Presence of Limited In-Distribution Labeled Data (NeurIPS 2021) [paper]
• Locally Most Powerful Bayesian Test for Out-of-Distribution Detection Using Deep Generative Models (NeurIPS 2021) [paper]
• Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection (NeurIPS 2021) [paper]
• ReAct: Out-of-distribution Detection With Rectified Activations (NeurIPS 2021) [paper] [code]
• Entropy Maximization and Meta Classification for Out-of-Distribution Detection in Semantic Segmentation (ICCV 2021) [paper]
• Semantically Coherent Out-of-Distribution Detection (ICCV 2021) [paper] [code]
• Triggering Failures: Out-of-Distribution Detection by Learning From Local Adversarial Attacks in Semantic Segmentation (ICCV 2021) [paper] [code]
• MOOD: Multi-Level Out-of-Distribution Detection (CVPR 2021) [paper]
• MOS: Towards Scaling Out-of-Distribution Detection for Large Semantic Space (CVPR 2021) [paper]
• Out-of-Distribution Detection Using Union of 1-Dimensional Subspaces (CVPR 2021) [paper] [code]
• Multiscale Score Matching for Out-of-Distribution Detection (ICLR 2021) [paper] [code]
• SSD: A Unified Framework for Self-Supervised Outlier Detection (ICLR 2021) [paper] [code]

Early Works

• Energy-based Out-of-distribution Detection (NeurIPS 2020) [paper] [code]
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (NeurIPS 2020) [paper] [code]
• Why Normalizing Flows Fail to Detect Out-of-Distribution Data (NeurIPS 2020) [paper] [code]
• Likelihood Ratios for Out-of-Distribution Detection (NeurIPS 2019) [paper]
• Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem (CVPR 2019) [paper]
• Deep Anomaly Detection with Outlier Exposure (ICLR 2019) [paper] [code]
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks (NeurIPS 2018) [paper] [code]
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks (ICLR 2018) [paper]
• Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (ICLR 2018) [paper] [code]
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (NeurIPS 2018) [paper]
• Predictive Uncertainty Estimation via Prior Networks (NeurIPS 2018) [paper] [code]
• Out-of-Distribution Detection using Multiple Semantic Label Representations (NeurIPS 2018) [paper]
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (ICLR 2017) [paper]