Outliers: Identification and Treatment through use of Chebyshev's Theorem

Abstract

Outliers have traditionally presented as unwanted and troublesome elements whose influence we should protect against through robust methods of data handling and estimation. In this presentation, extreme representative outliers are seen as both inevitable and as providing necessary information about the tail ends of universe distributions, and a new method for identifying and treating outliers based on an innovative use of Chebyshev's Theorem is introduced. The method presented avoids the overly wide ranges commonly associated with the use of this theorem, and being based on Chebyshev's Theorem, does not depend on the normality or specific form of the underlying distribution. The presentation will present data and results demonstrating the importance of outliers, as well as test results that show the ability of the new method to predict corresponding universe values, using universe and drawn samples from national establishment employment data. In addition, a justification and derivation of Winsorizing that follows from considerations of the method will also be presented.