Logo: to the web site of Uppsala University

uu.sePublications from Uppsala University
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Evaluating model calibration in classification
Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Division of Systems and Control. Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Automatic control.
Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Division of Systems and Control. Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Automatic control.ORCID iD: 0000-0001-9282-053x
Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Division of Systems and Control. Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Automatic control.ORCID iD: 0000-0002-1539-6314
Show others and affiliations
2019 (English)In: 22nd International Conference on Artificial Intelligence and Statistics, 2019, p. 3459-3467Conference paper, Published paper (Refereed)
Place, publisher, year, edition, pages
2019. p. 3459-3467
Series
Proceedings of Machine Learning Research, ISSN 2640-3498 ; 89
National Category
Probability Theory and Statistics
Identifiers
URN: urn:nbn:se:uu:diva-397519ISI: 000509687903053OAI: oai:DiVA.org:uu-397519DiVA, id: diva2:1371935
Conference
AISTATS 2019, April 16–18, Naha, Japan
Available from: 2019-04-25 Created: 2019-11-21 Last updated: 2023-04-26Bibliographically approved
In thesis
1. Deep learning applied to system identification: A probabilistic approach
Open this publication in new window or tab >>Deep learning applied to system identification: A probabilistic approach
2019 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Machine learning has been applied to sequential data for a long time in the field of system identification. As deep learning grew under the late 00's machine learning was again applied to sequential data but from a new angle, not utilizing much of the knowledge from system identification. Likewise, the field of system identification has yet to adopt many of the recent advancements in deep learning. This thesis is a response to that. It introduces the field of deep learning in a probabilistic machine learning setting for problems known from system identification.

Our goal for sequential modeling within the scope of this thesis is to obtain a model with good predictive and/or generative capabilities. The motivation behind this is that such a model can then be used in other areas, such as control or reinforcement learning. The model could also be used as a stepping stone for machine learning problems or for pure recreational purposes.

Paper I and Paper II focus on how to apply deep learning to common system identification problems. Paper I introduces a novel way of regularizing the impulse response estimator for a system. In contrast to previous methods using Gaussian processes for this regularization we propose to parameterize the regularization with a neural network and train this using a large dataset. Paper II introduces deep learning and many of its core concepts for a system identification audience. In the paper we also evaluate several contemporary deep learning models on standard system identification benchmarks. Paper III is the odd fish in the collection in that it focuses on the mathematical formulation and evaluation of calibration in classification especially for deep neural network. The paper proposes a new formalized notation for calibration and some novel ideas for evaluation of calibration. It also provides some experimental results on calibration evaluation.

Place, publisher, year, edition, pages
Uppsala University, 2019
Series
Information technology licentiate theses: Licentiate theses from the Department of Information Technology, ISSN 1404-5117 ; 2019-007
National Category
Signal Processing Probability Theory and Statistics
Research subject
Electrical Engineering with specialization in Signal Processing
Identifiers
urn:nbn:se:uu:diva-397563 (URN)
Supervisors
Available from: 2019-11-18 Created: 2019-11-21 Last updated: 2019-11-21Bibliographically approved
2. Calibration of Probabilistic Predictive Models
Open this publication in new window or tab >>Calibration of Probabilistic Predictive Models
2020 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Predicting unknown and unobserved events is a common task in many domains. Mathematically, the uncertainties arising in such prediction tasks can be described by probabilistic predictive models. Ideally, the model estimates of these uncertainties allow us to distinguish between uncertain and trustworthy predictions. This distinction is particularly important in safety-critical applications such as medical image analysis and autonomous driving.

For the probabilistic predictions to be meaningful and to allow this differentiation, they should neither be over- nor underconfident. Models that satisfy this property are called calibrated. In this thesis we study how one can measure, estimate, and statistically reason about the calibration of probabilistic predictive models.

In Paper I we discuss existing approaches for evaluating calibration in multi-class classification. We mention potential pitfalls and suggest hypothesis tests for the statistical analysis of model calibration.

In Paper II we propose a framework of calibration measures for multi-class classification. It captures common existing measures and includes a new kernel calibration error based on matrix-valued kernels. For the kernel calibration error consistent and unbiased estimators exist and asymptotic hypothesis tests for calibration can be derived. Unfortunately, by construction the framework is limited to prediction problems with finite discrete target spaces.

In Paper III we use a different approach to develop a more general framework of calibration errors that applies to any probabilistic predictive model and is not limited to classification. We show that it coincides with the framework presented in Paper II for multi-class classification. Based on scalar-valued kernels, we generalize the kernel calibration error, its estimators, and hypothesis tests to all probabilistic predictive models. For real-valued regression problems we present empirical results.

Place, publisher, year, edition, pages
Uppsala: Uppsala University, 2020. p. 266
Series
Information technology licentiate theses: Licentiate theses from the Department of Information Technology, ISSN 1404-5117 ; 2020-006
National Category
Probability Theory and Statistics
Research subject
Mathematical Statistics
Identifiers
urn:nbn:se:uu:diva-429418 (URN)
Presentation
2020-10-28, 10:00
Opponent
Supervisors
Available from: 2021-01-04 Created: 2020-12-23 Last updated: 2021-01-04Bibliographically approved
3. Deep probabilistic models for sequential and hierarchical data
Open this publication in new window or tab >>Deep probabilistic models for sequential and hierarchical data
2022 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Consider the problem where we want a computer program capable of recognizing a pedestrian on the road. This could be employed in a car to automatically apply the brakes to avoid an accident. Writing such a program is immensely difficult but what if we could instead use examples and let the program learn what characterizes a pedestrian from the examples. Machine learning can be described as the process of teaching a model (computer program) to predict something (the presence of a pedestrian) with help of data (examples) instead of through explicit programming.

This thesis focuses on a specific method in machine learning, called deep learning. This method can arguably be seen as sole responsible for the recent upswing of machine learning in academia as well as in society at large. However, deep learning requires, in human standards, a huge amount of data to perform well which can be a limiting factor.  In this thesis we describe different approaches to reduce the amount of data that is needed by encoding some of our prior knowledge about the problem into the model. To this end we focus on sequential and hierarchical data, such as speech and written language.

Representing sequential output is in general difficult due to the complexity of the output space. Here, we make use of a probabilistic approach focusing on sequential models in combination with a deep learning structure called the variational autoencoder. This is applied to a range of different problem settings, from system identification to speech modeling.

The results come in three parts. The first contribution focus on applications of deep learning to typical system identification problems, the intersection between the two areas and how they can benefit from each other. The second contribution is on hierarchical data where we promote a multiscale variational autoencoder inspired by image modeling. The final contribution is on verification of probabilistic models, in particular how to evaluate the validity of a probabilistic output, also known as calibration.

Place, publisher, year, edition, pages
Uppsala: Acta Universitatis Upsaliensis, 2022. p. 87
Series
Digital Comprehensive Summaries of Uppsala Dissertations from the Faculty of Science and Technology, ISSN 1651-6214 ; 2139
Keywords
Machine learning, Deep learning, Sequential modelling
National Category
Signal Processing
Research subject
Electrical Engineering with specialization in Signal Processing
Identifiers
urn:nbn:se:uu:diva-470433 (URN)978-91-513-1478-5 (ISBN)
Public defence
2022-05-24, Sonja Lyttkens, 101121, Ångström, Lägerhyddsvägen 1, Uppsala, 09:00 (English)
Opponent
Supervisors
Available from: 2022-05-02 Created: 2022-04-04 Last updated: 2022-06-14
4. Reliable Uncertainty Quantification in Statistical Learning
Open this publication in new window or tab >>Reliable Uncertainty Quantification in Statistical Learning
2023 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Mathematical models are powerful yet simplified abstractions used to study, explain, and predict the behavior of systems of interest. This thesis is concerned with their latter application as predictive models. Predictions of such models are often inherently uncertain, as exemplified in weather forecasting and experienced with epidemiological models during the COVID-19 pandemic. Missing information, such as incomplete atmospheric data, and the very nature of models as approximations ("all models are wrong") imply that predictions are at most approximately correct.

Probabilistic models alleviate this issue by reporting not a single point prediction ("rain"/"no rain") but a probability distribution of all possible outcomes ("80% probability of rain"), representing the uncertainty of a prediction, with the intention to be able to mark predictions as more or less trustworthy. However, simply reporting a probabilistic prediction does not guarantee that the uncertainty estimates are reliable. Calibrated models ensure that the uncertainty expressed by the predictions is consistent with the prediction task and hence the predictions are neither under- nor overconfident. Calibration is important in particular in safety-critical applications such as medical diagnostics and autonomous driving where it is crucial to be able to distinguish between uncertain and trustworthy predictions. Mathematical models do not necessarily possess this property, and in particular complex machine learning models are susceptible to reporting overconfident predictions.

The main contribution of this thesis are new statistical methods for analyzing the calibration of a model, consisting of calibration measures, their estimators, and statistical hypothesis tests based on them. These methods are presented in the five scientific papers in the second part of the thesis. In the first part the reader is introduced to probabilistic predictive models, the analysis of calibration, and positive definite kernels that form the basis of the proposed calibration measures. The contributed tools for calibration analysis cover in principle any predictive model and are applied specifically to classification models, with an arbitrary number of classes, models for regression problems, and models arising from Bayesian inference. This generality is motivated by the need for more detailed calibration analysis of increasingly complex models nowadays. To simplify the use of the statistical methods, a collection of software packages for calibration analysis written in the Julia programming language is made publicly available and supplemented with interfaces to the Python and R programming languages.

Place, publisher, year, edition, pages
Uppsala: Acta Universitatis Upsaliensis, 2023. p. 110
Series
Digital Comprehensive Summaries of Uppsala Dissertations from the Faculty of Science and Technology, ISSN 1651-6214 ; 2275
Keywords
Reliability, Calibration, Uncertainty, Probabilistic Model, Prediction, Julia
National Category
Probability Theory and Statistics
Research subject
Machine learning
Identifiers
urn:nbn:se:uu:diva-500736 (URN)978-91-513-1823-3 (ISBN)
Public defence
2023-06-14, Häggsalen (10132), Ångströmlaboratoriet, Uppsala, 09:15 (English)
Opponent
Supervisors
Available from: 2023-05-23 Created: 2023-04-26 Last updated: 2023-05-23

Open Access in DiVA

No full text in DiVA

Other links

Abstract

Authority records

Vaicenavicius, JuozasWidmann, DavidAndersson, CarlSchön, Thomas B.

Search in DiVA

By author/editor
Vaicenavicius, JuozasWidmann, DavidAndersson, CarlSchön, Thomas B.
By organisation
Division of Systems and ControlAutomatic control
Probability Theory and Statistics

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 274 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf