Logo: to the web site of Uppsala University

uu.sePublications from Uppsala University
Change search
Link to record
Permanent link

Direct link
Alternative names
Publications (10 of 443) Show all publications
Fatima, G., Babu, P. & Stoica, P. (2024). Two New Algorithms for Maximum Likelihood Estimation of Sparse Covariance Matrices With Applications to Graphical Modeling. IEEE Transactions on Signal Processing, 72, 958-971
Open this publication in new window or tab >>Two New Algorithms for Maximum Likelihood Estimation of Sparse Covariance Matrices With Applications to Graphical Modeling
2024 (English)In: IEEE Transactions on Signal Processing, ISSN 1053-587X, E-ISSN 1941-0476, Vol. 72, p. 958-971Article in journal (Refereed) Published
Abstract [en]

In this paper, we propose two new algorithms for maximum-likelihood estimation (MLE) of high dimensional sparse covariance matrices. Unlike most of the state-of-the-art methods, which either use regularization techniques or penalize the likelihood to impose sparsity, we solve the MLE problem based on an estimated covariance graph. More specifically, we propose a two-stage procedure: in the first stage, we determine the sparsity pattern of the target covariance matrix (in other words the marginal independence in the covariance graph under a Gaussian graphical model) using the multiple hypothesis testing method of false discovery rate (FDR), and in the second stage we use either a block coordinate descent approach to estimate the non-zero values or a proximal distance approach that penalizes the distance between the estimated covariance graph and the target covariance matrix. Doing so gives rise to two different methods, each with its own advantage: the coordinate descent approach does not require tuning of any hyper-parameters, whereas the proximal distance approach is computationally fast but requires a careful tuning of the penalty parameter. Both methods are effective even in cases where the number of observed samples is less than the dimension of the data. For performance evaluation, we test the proposed methods on both simulated and real-world data and show that they provide more accurate estimates of the sparse covariance matrix than the state-of-the-art methods.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2024
Keywords
Covariance matrices, Maximum likelihood estimation, Testing, Sparse matrices, Graphical models, Tuning, Symmetric matrices, Block coordinate descent, covariance estimation, Gaussian graphical model, multiple hypothesis testing, proximal distance algorithm
National Category
Probability Theory and Statistics Signal Processing
Identifiers
urn:nbn:se:uu:diva-525046 (URN)10.1109/TSP.2024.3361082 (DOI)001167517500005 ()
Funder
Swedish Research Council
Available from: 2024-03-20 Created: 2024-03-20 Last updated: 2024-03-20Bibliographically approved
Mattsson, P., Zachariah, D. & Stoica, P. (2023). Analysis of the Minimum-Norm Least-Squares Estimator and Its Double-Descent Behavior [Lecture Notes]. IEEE signal processing magazine (Print), 40(3), 39-75
Open this publication in new window or tab >>Analysis of the Minimum-Norm Least-Squares Estimator and Its Double-Descent Behavior [Lecture Notes]
2023 (English)In: IEEE signal processing magazine (Print), ISSN 1053-5888, E-ISSN 1558-0792, Vol. 40, no 3, p. 39-75Article in journal (Refereed) Published
Abstract [en]

Linear regression models have a wide range of applications in statistics, signal processing, and machine learning. In this Lecture Notes column we will examine the performance of the least-squares (LS) estimator with a focus on the case when there are more parameters than training samples, which is often overlooked in textbooks on estimation.

Place, publisher, year, edition, pages
IEEE, 2023
Keywords
Least squares methods, Linear regression, Estimation, Machine learning, Signal processing, Behavioral sciences
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:uu:diva-504033 (URN)10.1109/MSP.2023.3242083 (DOI)000981974000005 ()
Available from: 2023-06-28 Created: 2023-06-28 Last updated: 2023-06-28Bibliographically approved
Stoica, P. & Babu, P. (2023). Low-Rank Covariance Matrix Estimation for Factor Analysis in Anisotropic Noise: Application to Array Processing and Portfolio Selection. IEEE Transactions on Signal Processing, 71, 1699-1711
Open this publication in new window or tab >>Low-Rank Covariance Matrix Estimation for Factor Analysis in Anisotropic Noise: Application to Array Processing and Portfolio Selection
2023 (English)In: IEEE Transactions on Signal Processing, ISSN 1053-587X, E-ISSN 1941-0476, Vol. 71, p. 1699-1711Article in journal (Refereed) Published
Abstract [en]

Factor analysis (FA) or principal component analysis (PCA) models the covariance matrix of the observed data as R = SST + S, where SST is the low-rank covariance matrix corresponding to the factors (aka latent variables) and S is the diagonal matrix of the noise. When the noise is anisotropic (aka nonuniform in the signal processing literature and heteroscedastic in the statistical literature), the diagonal elements of S cannot be assumed to be identical and they must be estimated jointly with the elements of SST. The problem of estimating SST and S in the above covariance model is the central theme of the present article. After stating this problem in a more formal way, we review the main existing algorithms for solving it. We then go on to show that these algorithms have reliability issues (such as lack of convergence or convergence to infeasible solutions) and therefore they may not be the best possible choice for practical applications. Next we explain how to modify one of these algorithms to improve its convergence properties and we also introduce a new method that we call FAAN (Factor Analysis for Anisotropic Noise). FAAN is a coordinate descent algorithm that iteratively maximizes the normal likelihood function, which is easy to implement in a numerically efficient manner and has excellent convergence properties as illustrated by the numerical examples presented in the article. Out of the many possible applications of FAAN we focus on the following two: direction-of-arrival (DOA) estimation using array signal processing techniques and portfolio selection for financial asset management.

Place, publisher, year, edition, pages
IEEE, 2023
Keywords
Factor model, coordinate descent, direction of arrival estimation, portfolio selection
National Category
Signal Processing Probability Theory and Statistics
Identifiers
urn:nbn:se:uu:diva-507475 (URN)10.1109/TSP.2023.3273116 (DOI)001010133500004 ()
Funder
Swedish Research Council, 2017-04610Swedish Research Council, 2016-06079Swedish Research Council, 2021-05022
Available from: 2023-07-07 Created: 2023-07-07 Last updated: 2023-07-07Bibliographically approved
Wang, Z., Stoica, P., Zachariah, D., Babu, P. & Yang, Z. (2023). Min-Max Probe Placement and Extended Relaxation Estimation Method for Processing Blade Tip Timing Signals. IEEE Transactions on Instrumentation and Measurement, 72, Article ID 3535509.
Open this publication in new window or tab >>Min-Max Probe Placement and Extended Relaxation Estimation Method for Processing Blade Tip Timing Signals
Show others...
2023 (English)In: IEEE Transactions on Instrumentation and Measurement, ISSN 0018-9456, E-ISSN 1557-9662, Vol. 72, article id 3535509Article in journal (Refereed) Published
Abstract [en]

Measuring blade displacement using blade tip timing (BTT) enables nonintrusive monitoring of rotating blades and their vibration frequencies. The average sampling frequency of BTT is the product of the number of measurement probes and rotational frequency, which is usually far less than the blade natural frequency due to the limited number of probes. The pattern of the aliasing that arises from under-sampling is rather complex under uneven probe placement. In this article, we consider a probe placement design that is based on minimizing the maximum sidelobe level of the spectral window to suppress the aliasing frequencies in the spectrum. Based on a signal model containing both asynchronous and synchronous sinusoids, we then develop an extended version of the RELAX method (ERELAX) to estimate their parameters simultaneously. Model order selection rules are also used to determine the number of asynchronous sinusoids. The frequency ambiguity that arises from periodic nonuniform sampling (PNS) is also discussed based on the convolution in the frequency domain. Numerical simulations and results of a curved-blade experiment show that the proposed method has a mean squared estimation error less than 25% of that of two state-of-the-art methods (Block-OMP and MUSIC), requires 40% of the data length needed by the latter methods to achieve the same estimation accuracy, and has the smallest standard deviation of the reconstruction errors. Simulation codes are available at https://github.com/superjdg/RELAX_BTT.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2023
Keywords
Probes, Blades, Vibrations, Vibration measurement, Frequency synchronization, Time-frequency analysis, Frequency estimation, Blade tip timing (BTT), frequency ambiguity, min-max placement, model order selection, relax
National Category
Signal Processing
Identifiers
urn:nbn:se:uu:diva-516652 (URN)10.1109/TIM.2023.3324671 (DOI)001093394400004 ()
Funder
Swedish Research Council, 2017-04610Swedish Research Council, 2016-06079Swedish Research Council, 2021-05022
Available from: 2023-11-28 Created: 2023-11-28 Last updated: 2023-11-28Bibliographically approved
Babu, P. & Stoica, P. (2023). Multiple-hypothesis testing rules for high-dimensional model selection and sparse-parameter estimation. Signal Processing, 213, Article ID 109189.
Open this publication in new window or tab >>Multiple-hypothesis testing rules for high-dimensional model selection and sparse-parameter estimation
2023 (English)In: Signal Processing, ISSN 0165-1684, E-ISSN 1872-7557, Vol. 213, article id 109189Article in journal (Refereed) Published
Abstract [en]

We consider the problem of model selection for high-dimensional sparse linear regression models. We pose the model selection problem as a multiple-hypothesis testing problem and employ the methods of false discovery rate (FDR) and familywise error rate (FER) to solve it. We also present the reformulation of the FDR/FER-based approaches as criterion-based model selection rules and establish their relation to the extended Bayesian Information Criterion (EBIC), which is a state-of-the-art high-dimensional model selection rule. We use numerical simulations to show that the proposed FDR/FER method is well suited for high-dimensional model selection and performs better than EBIC.

Place, publisher, year, edition, pages
Elsevier BV, 2023
Keywords
Model selection, Sparse parameter estimation, Mulitple hypothesis testing, FDR, FER
National Category
Signal Processing Probability Theory and Statistics Computational Mathematics
Identifiers
urn:nbn:se:uu:diva-510700 (URN)10.1016/j.sigpro.2023.109189 (DOI)001051106300001 ()
Available from: 2023-09-06 Created: 2023-09-06 Last updated: 2023-09-06Bibliographically approved
Ek, S., Zachariah, D., Johansson, F. D. & Stoica, P. (2023). Off-Policy Evaluation with Out-of-Sample Guarantees. Transactions on Machine Learning Research
Open this publication in new window or tab >>Off-Policy Evaluation with Out-of-Sample Guarantees
2023 (English)In: Transactions on Machine Learning Research, E-ISSN 2835-8856Article in journal (Refereed) Published
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:uu:diva-519244 (URN)
Available from: 2024-01-04 Created: 2024-01-04 Last updated: 2024-01-09Bibliographically approved
Osama, M., Zachariah, D., Stoica, P. & Schön, T. B. (2023). Online Learning for Prediction via Covariance Fitting: Computation, Performance and Robustness. Transactions on Machine Learning Research
Open this publication in new window or tab >>Online Learning for Prediction via Covariance Fitting: Computation, Performance and Robustness
2023 (English)In: Transactions on Machine Learning Research, E-ISSN 2835-8856Article in journal (Refereed) Published
Abstract [en]

We consider the online learning of linear smoother predictors based on a covariance model of the outcomes. To control its degrees of freedom in an appropriate manner, the covariance model parameters are often learned using cross-validation or maximum-likelihood techniques. However, neither technique is suitable when training data arrives in a streaming fashion. Here we consider a covariance-fitting method to learn the model parameters, initially used  in spectral estimation. We show that this results in a computation efficient online learning method in which the resulting predictor can be updated sequentially. We prove that, with high probability, its out-of-sample error approaches the minimum achievable level at root-$n$ rate. Moreover, we show that the resulting predictor enjoys two different robustness properties. First, it minimizes the out-of-sample error with respect to the least favourable distribution within a given Wasserstein distance from the empirical distribution. Second, it is robust against errors in the covariate training data. We illustrate the performance of the proposed method in a numerical experiment.

Place, publisher, year, edition, pages
Transactions on Machine Learning Research, 2023
National Category
Probability Theory and Statistics Engineering and Technology
Identifiers
urn:nbn:se:uu:diva-472451 (URN)
Available from: 2022-04-11 Created: 2022-04-11 Last updated: 2024-01-08Bibliographically approved
Mattsson, P., Zachariah, D. & Stoica, P. (2023). Regularized Linear Regression via Covariance Fitting. IEEE Transactions on Signal Processing, 71, 1175-1183
Open this publication in new window or tab >>Regularized Linear Regression via Covariance Fitting
2023 (English)In: IEEE Transactions on Signal Processing, ISSN 1053-587X, E-ISSN 1941-0476, Vol. 71, p. 1175-1183Article in journal (Refereed) Published
Abstract [en]

The linear minimum mean-square error estimator (LMMSE) can be viewed as a solution to a certain regularized least-squares problem formulated using model covariance matrices. However, the appropriate parameters of the model covariance matrices are unknown in many applications. This raises the question: how should we choose them using only the data? Using data-adaptive matrices obtained via the covariance fitting SPICE-methodology, we show that the empirical LMMSE is equivalent to tuned versions of various known regularized estimators - such as ridge regression, LASSO, and regularized least absolute deviation - depending on the chosen covariance structures. These theoretical results unify several important estimators under a common umbrella. Furthermore, through a number of numerical examples we show that the regularization parameters obtained via covariance fitting are close to optimal for a range of different signal conditions.

Place, publisher, year, edition, pages
IEEE, 2023
Keywords
Estimation theory, parameter estimation
National Category
Probability Theory and Statistics Signal Processing
Identifiers
urn:nbn:se:uu:diva-502506 (URN)10.1109/TSP.2023.3263363 (DOI)000976047000002 ()
Funder
Swedish Research Council, 621-2016-06079Swedish Research Council, 2018-05040Swedish Research Council, 2021-05022
Available from: 2023-05-29 Created: 2023-05-29 Last updated: 2023-05-29Bibliographically approved
Fatima, G., Babu, P. & Stoica, P. (2022). Covariance Matrix Estimation Under Positivity Constraints With Application to Portfolio Selection. IEEE Signal Processing Letters, 29, 2487-2491
Open this publication in new window or tab >>Covariance Matrix Estimation Under Positivity Constraints With Application to Portfolio Selection
2022 (English)In: IEEE Signal Processing Letters, ISSN 1070-9908, E-ISSN 1558-2361, Vol. 29, p. 2487-2491Article in journal (Refereed) Published
Abstract [en]

In this letter we propose a new method to estimate the covariance matrix under the constraint that its off-diagonal elements are non-negative, which has applications to portfolio selection in finance. We incorporate the non-negativity constraint in the maximum likelihood (ML) estimation problem and propose an algorithm based on the block coordinate descent method to solve for the ML estimate. To study the effectiveness of the proposed algorithm, we perform numerical simulations on both synthetic and real-world financial data, and show that our proposed method has better performance than that of a state-of-the-art method.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2022
Keywords
Block coordinate descent, global minimum variance portfolio, maximum-likelihood estimation, non-negative correlations, portfolio selection
National Category
Signal Processing
Identifiers
urn:nbn:se:uu:diva-492693 (URN)10.1109/LSP.2022.3226117 (DOI)000899990900001 ()
Funder
Swedish Research Council, 2017-04610Swedish Research Council, 2016-06079Swedish Research Council, 2021-05022
Available from: 2023-01-09 Created: 2023-01-09 Last updated: 2023-01-09Bibliographically approved
Stoica, P. & Babu, P. (2022). False Discovery Rate (FDR) and Familywise Error Rate (FER) Rules for Model Selection in Signal Processing Applications. IEEE Open Journal of Signal Processing, 3, 403-416
Open this publication in new window or tab >>False Discovery Rate (FDR) and Familywise Error Rate (FER) Rules for Model Selection in Signal Processing Applications
2022 (English)In: IEEE Open Journal of Signal Processing, E-ISSN 2644-1322, Vol. 3, p. 403-416Article in journal (Refereed) Published
Abstract [en]

Model selection is an omnipresent problem in signal processing applications. The Akaike information criterion (AIC) and the Bayesian information criterion (BIC) are the most commonly used solutions to this problem. These criteria have been found to have satisfactory performance in many cases and had a dominant role in the model selection literature since their introduction several decades ago, despite numerous attempts to dethrone them. Model selection can be viewed as a multiple hypothesis testing problem. This simple observation makes it possible to use for model selection a number of powerful hypothesis testing procedures that control the false discovery rate (FDR) or the familywise error rate (FER). This is precisely what we do in this paper in which we follow the lead of the proposers of the said procedures and introduce two general rules for model selection based on FDR and FER, respectively. We show in a numerical performance study that the FDR and FER rules are serious competitors of AIC and BIC with significant performance gains in more demanding cases, essentially at the same computational effort.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2022
Keywords
Signal processing, Numerical models, Predictive models, Computational modeling, Bayes methods, Sensor arrays, Maximum likelihood estimation, Model order selection, FDR, FER, AIC, BIC
National Category
Signal Processing
Identifiers
urn:nbn:se:uu:diva-489718 (URN)10.1109/OJSP.2022.3213128 (DOI)000882706000001 ()
Funder
Swedish Research Council, 2017-04610Swedish Research Council, 2016-06079Swedish Research Council, 2021-05022
Available from: 2022-12-06 Created: 2022-12-06 Last updated: 2024-01-08Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0002-7957-3711

Search in DiVA

Show all publications