Modeling Dependency in Paired Comparison Data under a Multilevel Framework

Abstract

Paired comparison data obtained under multiple judgment setting are commonly aggregated over judges and then analyzed under the strong assumption that the paired comparison judgments are independent. Such an independence assumption seems strong given that the same items are evaluated in different pairs. As a result, two-level paired linear comparison models have been proposed to model such dependence. These models assume that the utility of an item remain invariant across trials and pair-specific variability terms are introduced to account for intransitive choice behavior. However, this invariant-utility assumption is yet another relatively strong assumption because the pairs are usually presented sequentially rather than simultaneously. In this talk, we will introduce an alternative two-level paired comparison model which considers the utilities associated with the same items across trials to be neither independent nor identical, but related. The underlying mechanism implied under such a formulation should be closer to the one undertaken in the judgmental process. The interpretation of the model parameters and its differences with the existent two-level model are discussed in detail. An extensive analysis of an experimental study illustrates the usefulness of the new approach in modeling multiple paired comparisons.