A Unified Hierarchical Model to Investigate the Empirical Bayes and Item Clustering Effects : A Case Study of a US National Survey

Abstract

Empirical Bayes regression procedures are commonly used in educational and psychological testing as extensions to latent variable models. The National Assessment of Educational Progress (NAEP) is an important national survey in the United States using such procedures. Student responses to questions (items) across various subject matters (e.g., reading, science, and music) are collected and analyzed by correlating with their background information such as ethnicity and parental education. NAEP applies empirical Bayes methods to models from item response theory. In the process, NAEP uses a two-stage procedure: first, item parameters are estimated, then an empirical Bayes methods is applied to estimate subgroup student proficiencies. In the second stage, item parameters are treated as known. We found that the effect of ignoring uncertainty in the item parameters on reported NAEP outcome can be substantial. Furthermore, the item response theory model NAEP uses ignores the effect of item clustering created by the design of a test form. Using Markov Chain Monte Carlo method, we simultaneously estimate all parameters of an expanded hierarchical model. The unified approach allows us to assess both the clustering and the empirical Bayes effects. (This is joint work with Steven Scott, USC)