Intra Order-preserving Functions for Calibration of Multi-Class Neural\n Networks

Predicting calibrated confidence scores for multi-class deep networks is\nimportant for avoiding rare but costly mistakes. A common approach is to learn\na post-hoc calibration function that transforms the output of the original\nnetwork into calibrated confidence scores while maintaining the network's\naccuracy. However, previous post-hoc calibration techniques work only with\nsimple calibration functions, potentially lacking sufficient representation to\ncalibrate the complex function landscape of deep networks. In this work, we aim\nto learn general post-hoc calibration functions that can preserve the top-k\npredictions of any deep network. We call this family of functions intra\norder-preserving functions. We propose a new neural network architecture that\nrepresents a class of intra order-preserving functions by combining common\nneural network components. Additionally, we introduce order-invariant and\ndiagonal sub-families, which can act as regularization for better\ngeneralization when the training data size is small. We show the effectiveness\nof the proposed method across a wide range of datasets and classifiers. Our\nmethod outperforms state-of-the-art post-hoc calibration methods, namely\ntemperature scaling and Dirichlet calibration, in several evaluation metrics\nfor the task.\n