What Can One do when EM Fails - Handling Missing Data with Non-parametric and Semi-parametric Models

Abstract

A common frustration in statistical estimation is when data are missing, incomplete or irregularly spaced (e.g., as with wavelets). Self-consistency is a general principle for handling such "bad" data problems under semi-parametric and/or non-parametric settings. It also provides a theoretical criterion to regulate and improve estimation procedures even when there is no missing data. Indeed, efficient estimation procedures, such as maximum likelihood estimation, are automatically self-consistent (asymptotically under square-loss). Conceptually, self-consistency is extremely appealing; it is essentially a mathematical formalization of those common-sense iterative "trial-and-error" methods. Mathematically, it is elegant, with just one fixed-point equation to solve and a general projection theorem to establish its optimality. Practically, it is straightforward to implement, as essentially it is a looped version of the regular/complete-data method. In this talk we provide a general framework for applying the self-consistency criterion to various estimation problems. We will illustrate our approach with three different missing data problems: wavelet regression, nonparametric spectral density estimation, and adatpive lasso fitting. This talk is based on joint work with Zhan Li, Xiao-Li Meng and Zhengyuan Zhu.