Proper quantification of predictive uncertainty is essential for the use of machine learning in safety-critical applications. Various uncertainty measures have been proposed for this purpose, typically claiming superiority over other measures. In this paper, we argue that there is no single best measure. Instead, uncertainty quantification should be tailored to the specific application. To this end, we use a flexible family of uncertainty measures that distinguishes between total, aleatoric, and epistemic uncertainty of second-order distributions. These measures can be instantiated with specific loss functions, so-called proper scoring rules, to control their characteristics, and we show that different characteristics are useful for different tasks. In particular, we show that, for the task of selective prediction, the scoring rule should ideally match the task loss. On the other hand, for out-of-distribution detection, our results confirm that mutual information, a widely used measure of epistemic uncertainty, performs best. Furthermore, in an active learning setting, epistemic uncertainty based on zero-one loss is shown to consistently outperform other uncertainty measures.
The requirements can be installed via the following command.
pip install -r requirements.txtThe models can be trained using the training script train.py, the hyperparameters of which are set in the trainconfig.yml file.
Similarly, the pre-trained models are configured using the 'pretrain.py' file based on the settings specified in the pretrainconfig.yml file.
The experiments are performed using the main.py script for which the arguments can be passed via the command line. For example:
python3 main.py
--seed 7 \
--exp ood \
--data cifar10 \
--data_ood svhn \
--rep dropout \
--base resnet \
--measure log