Pseudo-Gibbs Sampling for Dependence Networks

Abstract

A directed graph is either acyclic or cyclic. Dependence networks (DA) are depicted by cyclic graphs, i.e., the directed paths within the network have feed-back loops.? In statistics, a DA is called conditionally specified distribution. A DA first estimates full conditional distributions from data, then forge a joint distribution from the conditionals. The conditional approach allows a higher level of flexibility because it is computationally convenient to estimate the local distribution of one variable given the remaining variables. Similar approaches have appeared in multiple imputation since 1999. However, individually determined conditional distributions are generally not coherent with any joint distribution.? The pseudo-Gibbs sampling (PGS) method is often used to approximate joint distributions from incompatible conditional models, though its properties are mostly unknown. This talk investigates the richness and varieties of the joint distributions that PGS approximates. In short, PGS produces a large number of joint distributions with shared dependence and shared marginal distributions. As an algorithm, PGS has multiple fixed points that are as similar as possible under the constraints of incompatibility; the optimality is proved in terms of Kullback-Liebler information divergence. Our characterizations help the practitioners of multiple imputation to address some of their concerns. Moreover, properties of incompatible PGS provide a fresh perspective for understanding the original Gibbs sampler used in Bayesian computations. By the way, pseudo-Gibbs sampling is also known as chained equations imputation, stochastic relaxation, variable-by-variable imputation, regression switching, sequential regressions, partially incompatible MCMC and iterative univariate imputation.?