Logo: to the web site of Uppsala University

PURPOSE

Recent advances in machine learning have led to the development of classifiers that predict molecular subtypes of acute lymphoblastic leukemia (ALL) using RNA-sequencing (RNA-seq) data. Although these models have shown promising results, they often lack robust performance guarantees. The aim of this study was three-fold: to quantify the uncertainty of these classifiers, to provide prediction sets that control the false-negative rate (FNR), and to perform implicit error reduction by transforming incorrect predictions into uncertain predictions.

METHODS

Conformal prediction (CP) is a distribution-agnostic framework for generating statistically calibrated prediction sets whose size reflects model uncertainty. In this study, we applied an extension called conformal risk control to three RNA-seq ALL subtype classifiers. Leveraging RNA-seq data from 1,227 patient samples taken at diagnosis, we developed a multiclass conformal predictor ALLCoP, which generates statistically guaranteed FNR-controlled prediction sets.

RESULTS

ALLCoP was able to create prediction sets with specified FNR tolerances ranging from 7.5% to 30%. In a validation cohort, ALLCoP successfully reduced the FNR of the ALLIUM RNA-seq ALL subtype classifier from 8.95% to 3.5%. For patients whose subtype was not previously known, the use of ALLCoP was able to reduce the occurrence of empty predictions from 37% to 17%. Notably, up to 34% of the multiple-class prediction sets included the PAX5alt subtype, suggesting that increased prediction set size may reflect secondary aberrations and biological complexity, contributing to classifier uncertainty. Finally, ALLCoP was validated on two additional RNA-seq ALL subtype classifiers, ALLSorts and ALLCatchR.

CONCLUSION

Our results highlight the potential of CP in enhancing the use of oncologic RNA-seq subtyping classifiers and also in uncovering additional molecular aberrations of potential clinical importance.

We present an adaptive approach for robust learning from corrupted training sets. We identify corrupted and non-corrupted samples with latent Bernoulli variables and thus formulate the learning problem as maximization of the likelihood where latent variables are marginalized. The resulting problem is solved via variational inference, using an efficient Expectation-Maximization based method. The proposed approach improves over the state-of-the-art by automatically inferring the corruption level, while adding minimal computational overhead. We demonstrate our robust learning method and its parameter-free nature on a wide variety of machine learning tasks including online learning and deep learning where it adapts to different levels of noise and maintains high prediction accuracy.

Robust localization, which is the accurate measurement of the position of a user or device in the presence of obstructions or challenging environments, is a fundamental building block for numerous applications. The standard positioning methods and technologies do not provide satisfactory measurement accuracy in such challenging environments. Therefore, in this paper, a robust localization method is investigated and characterized experimentally. The robust method is capable of estimating the position of a mobile node based on distance measurements with respect to known-position anchors. The method is characterized by means of experiments using the ultra wide band ranging technology. Experimental results in a non line of sight (NLOS) scenario show that the robust localization method may reduce the error by a factor of 4 with respect to the standard method, i.e. the nonlinear least squares. In fact, the robust method results in a median error of approximately 5 cm, whereas the standard method results in a median positioning error of approximately 20 cm.

Background

In neurointensive care, increased intracranial pressure (ICP) is a feared secondary brain insult in traumatic brain injury (TBI). A system that predicts ICP insults before they emerge may facilitate early optimization of the physiology, which may in turn lead to stopping the predicted ICP insult from occurring. The aim of this study was to evaluate the performance of different artificial intelligence models in predicting the risk of ICP insults.

Methods

The models were trained to predict risk of ICP insults starting within 30 min, using the Uppsala high frequency TBI dataset. A restricted dataset consisting of only monitoring data were used, and an unrestricted dataset using monitoring data as well as clinical data, demographic data, and radiological evaluations was used. Four different model classes were compared: Gaussian process regression, logistic regression, random forest classifier, and Extreme Gradient Boosted decision trees (XGBoost).

Results

Six hundred and two patients with TBI were included (total monitoring 138,411 h). On the task of predicting upcoming ICP insults, the Gaussian process regression model performed similarly on the Uppsala high frequency TBI dataset (sensitivity 93.2%, specificity 93.9%, area under the receiver operating characteristic curve [AUROC] 98.3%), as in earlier smaller studies. Using a more flexible model (XGBoost) resulted in a comparable performance (sensitivity 93.8%, specificity 94.6%, AUROC 98.7%). Adding more clinical variables and features further improved the performance of the models slightly (XGBoost: sensitivity 94.1%, specificity of 94.6%, AUROC 98.8%).

Conclusions

Artificial intelligence models have potential to become valuable tools for predicting ICP insults in advance during neurointensive care. The fact that common off-the-shelf models, such as XGBoost, performed well in predicting ICP insults opens new possibilities that can lead to faster advances in the field and earlier clinical implementations.

In data-driven control design, an important prob-lem is to deal with uncertainty due to limited and noisydata. One way to do this is to use a min-max approach,which aims to minimize some design criteria for the worst-case scenario. However, a strategy based on this approachcan lead to overly conservative controllers. To overcome thisissue, we apply the idea of baseline regret, and it is seen thatminimizing the baseline regret under model uncertainty canguarantee safe controller improvement with less conservatismand variance in the resulting controllers. To exemplify theuse of baseline controllers, we focus on the output feedbacksetting and propose a two-step control design method; first,an uncertainty set is constructed by a data-driven systemidentification approach based on finite impulse response models;then a control design criterion based on model reference controlis used. To solve the baseline regret optimization problemefficiently, we use a convex approximation of the criterion andapply the scenario approach in optimization. The numericalexamples show that the inclusion of baseline regret indeedimproves the performance and reduces the variance of theresulting controller.

Linear regression models have a wide range of applications in statistics, signal processing, and machine learning. In this Lecture Notes column we will examine the performance of the least-squares (LS) estimator with a focus on the case when there are more parameters than training samples, which is often overlooked in textbooks on estimation.

Assessment of model fitness is a key part of machine learning. The standard paradigm of model evaluation is analysis of the average loss over future data. This is often explicit in model fitting, where we select models that minimize the average loss over training data asa surrogate, but comes with limited theoretical guarantees. In this paper, we consider the problem of characterizing a batch of out-of-sample losses of a model using a calibration dataset. We provide finite-sample limits on the out-of-sample losses that are statistically valid under quite general conditions and propose a diagonistic tool that is simple to compute andinterpret. Several numerical experiments are presented to show how the proposed  method quantifies the impact of distribution shifts, aids the analysis of regression, and enables model selection as well as hyperparameter tuning.

Measuring blade displacement using blade tip timing (BTT) enables nonintrusive monitoring of rotating blades and their vibration frequencies. The average sampling frequency of BTT is the product of the number of measurement probes and rotational frequency, which is usually far less than the blade natural frequency due to the limited number of probes. The pattern of the aliasing that arises from under-sampling is rather complex under uneven probe placement. In this article, we consider a probe placement design that is based on minimizing the maximum sidelobe level of the spectral window to suppress the aliasing frequencies in the spectrum. Based on a signal model containing both asynchronous and synchronous sinusoids, we then develop an extended version of the RELAX method (ERELAX) to estimate their parameters simultaneously. Model order selection rules are also used to determine the number of asynchronous sinusoids. The frequency ambiguity that arises from periodic nonuniform sampling (PNS) is also discussed based on the convolution in the frequency domain. Numerical simulations and results of a curved-blade experiment show that the proposed method has a mean squared estimation error less than 25% of that of two state-of-the-art methods (Block-OMP and MUSIC), requires 40% of the data length needed by the latter methods to achieve the same estimation accuracy, and has the smallest standard deviation of the reconstruction errors. Simulation codes are available at https://github.com/superjdg/RELAX_BTT.

We consider the problem of evaluating the performance of a decision policy using past observational data. The outcome of a policy is measured in terms of a loss (aka. disutility or negative reward) and the main problem is making valid inferences about its out-of-sample loss when the past data was observed under a different and possibly unknown policy. Using a sample-splitting method, we show that it is possible to draw such inferences with finite-sample coverage guarantees about the entire loss distribution, rather than just its mean. Importantly, the method takes into account model misspecifications of the past policy - including unmeasured confounding. The evaluation method can be used to certify the performance of a policy using observational data under a specified range of credible model assumptions.