Logo: to the web site of Uppsala University

Quantifying uncertainty is fundamental to reliable inference in statistical learning, especially in binary classification where models estimate probabilities for class membership. This thesis investigates the construction of Confidence Bands (CBs) for Receiver Operating Characteristic (ROC) curves, which are widely used to evaluate classifier performance. We establish a mathematical connection between ROC curves and conditional Cumulative Distribution Functions (CDFs), enabling the problem of ROC curve uncertainty quantification to be addressed through established statistical techniques for CDFs.

We develop and compare three approaches for constructing CBs for ROC curves: bootstrap CBs, Beta-Binomial CBs, and Monte Carlo CBs. The bootstrap method generates empirical CDFs through resampling, providing consistency with classical concentration inequalities but is limited by sampling bias in the calibration set. The Beta-Binomial approach adopts a Bayesian framework, constructing posterior distributions over conditional CDFs by incorporating confusion matrix elements, thus reducing sampling bias but introducing sensitivity to prior choice. Our primary contribution, the Monte Carlo CBs method, leverages the Probability Integral Transform and properties of order statistics to provide distribution-free, finite-sample guarantees for the uncertainty in ROC curves, overcoming both sampling bias and reliance on asymptotic approximations.

We formally analyze the theoretical properties of these methods, demonstrating that Monte Carlo CBs uniquely satisfy conditional coverage, finite-sample validity, distribution-free behavior, and variance adaptivity. Empirical validation on both synthetic and real-world datasets confirms that Monte Carlo CBs maintain their theoretical guarantees across varying sample sizes and significance levels. This work advances the statistical theory of uncertainty quantification for functional objects and provides practical tools for robust ROC curve analysis in binary classification.