Spurious Precision in Meta-Analysis of Observational Research

Abstract

Meta-analysis upweights studies reporting lower standard errors and hence more precision. But in observational settings common to much research in social sciences, precision is not given to the researcher. Precision must be estimated, and thus can be p-hacked to achieve statistical signi cance. Simulations and applications show that spurious precision can invalidate inverse-variance weighting and bias-correction methods based on the funnel plot. Selection models fail to solve the problem, and common cures to publication bias can become worse than the disease. We introduce a novel approach that addresses spuriousness and limits the resulting bias in meta-analysis.

Keywords: Publication bias, p-hacking, selection models, meta-regression, funnel plot, inverse-variance weighting