Thurstonian Models of Paired Comparison Data

Abstract

Many paired comparison data reported in the literature are obtained in a multiple judgment setting where each judge compares all possible item pairs one at a time. Paired comparison data obtained under such a task not only allow for the identification of systematically inconsistent judges, but also provide a rich source of information about individual differences in the preference judgments. However, despite the repeated measures structure of the data, multiple judgments are analyzed frequently under the strong assumption of independence. This disregard for dependencies in the paired comparison judgments is a serious model misspecification that may lead to both incorrect statistical and substantive conclusions. Multilevel models provide a well- suited framework for the analysis of multiple paired comparison judgments because they allow for partitions of the total variation into individual differences and momentary fluctuations within each person. More specifically, the multiple judgments of a respondent represent the first level and the variability in the parameters characterizing the individual judgments constitute the second level. Particularly, Thurstonian models provide a flexible framework for the analysis of multiple paired comparison judgments because they allow testing a wide range of hypotheses about the judgments' mean and covariance structures. However, applications have been limited to a large extent by the computational intractability involved in fitting this class of models. In this presentation, Monte Carlo EM algorithm is used for the maximum likelihood estimation of the Thurstonian paired comparison models. A detailed analysis of one particular paired comparison dataset is performed to illustrate the usefulness of this approach for the interpretation of similarity and individual difference effects in preference data.