Sample Overlap in Meta-Analysis

Abstract

This paper addresses the issue of sample overlap between primary samples in meta-analysis in economics. In principle, meta-analysis should improve the estimation efficiency of a given parameter of interest by pooling estimates derived from different samples. In economics - especially in macroeconomics and related subfields - meta-analysis often combines estimates derived from overlapping samples, however. Failing to account for sampling overlap may have pernicious implications for the statistical properties of the meta-estimator. First, sample overlap introduces a pattern of correlation between primary estimates in the meta-analysis model, leading to inefficient meta-estimators. This pattern is complex and cannot, in general, be accounted for by standard techniques, such as multilevel models. Second, classical statistical inference in the meta-analysis model is invalid - standard errors are downward-biased - and cannot, in general, be corrected by standard procedures such as clustering techniques. Third, meta-estimators cease to be consistent if the fraction of overlapping primary observations does not decrease as the total number of primary observations approaches infinity. To circumvent these statistical difficulties, we propose a generalized least squares meta-estimator. We show that the elements of the variance-covariance matrix describing the structure of dependency between estimates can be computed from information typically reported in (or possible to be inferred from) the primary studies, such as primary standard errors, sample sizes, and number of overlapping observations. This variance-covariance matrix can then be used to optimally weight each estimate in the meta-analysis model (i.e., according to the amount of independent sampling information in the corresponding primary sample). We study the properties of our generalized least squares meta-estimator using? both Monte Carlo simulations and bootstrap methods on the meta-sample of output elasticities of public capital collected by Bom and Ligthart (2013). (This is a joint work with Pedro Bom)