Farming in Data Rich Environment

Abstract

Correlated and heavy-tailed data arise frequently in a wide range of scientific and engineering problems: from genomics, medical imaging to neuroscience and finance. This talks introduce Factor-Adjusted Robust Multiple testing (FARM-test) and Factor-Adjusted Robust Model Selection (FARM-select). The former is introduced to control the false discovery proportion for large-scale simultaneous inference when variables are highly correlated and the latter deals the variable selection problems when covariates are highly correlated. We demonstrate that robust factor adjustments are extremely important in both improving the power of the tests and controlling FDP. We identify general conditions under which the proposed method produces a consistent estimate of the FDP. We also proved that factor adjustments significantly reduce the conditions needed for selection consistency. The results will be illustrated by numerical experiments. ? (based on joint work with Yuan Ke, Qiang Sun, and Wenxin Zhou and that of Yuan Ke and Kaizheng Wang) ? ?