Gibbs Ensembles for Incompatible Dependence Networks

Abstract

Dependence networks in machine learning have been proposed for probabilistic inference, collaborative filtering, and visualization of acausal predictive relationships. Because the graph of a dependence network is cyclic, it is different from a Bayesian network. Statistically, dependence networks are equivalent to conditionally specified distributions. Because the conditional specifications are either estimated from different data bases or supplied by several experts they often are incompatible. Statisticians have long concentrated on conditional specifications that are compatible and have a unique solution, while dependence networks mostly have to deal with conditional models that are not compatible and do not have a unique solution. Finding an optimal joint distribution for an incompatible dependence network via linear inequalities is known to be very restrictive. Can some learning algorithms be helpful here? Instead of using a supervised algorithm, we propose an ensemble approach based on the Gibbs sampler. We show that the joint distributions of Gibbs samples become stationary when a deterministic pattern is used to scan the conditional distributions. However, when the scan patterns vary, the joint distributions also vary. The ensemble will consist of all the joint distributions generated from applying all possible (deterministic) scan patterns, and the final distribution is a weighted combination of all the joint distributions in the ensemble. With the option of using different weights, we show that a decent distribution can emerge from consolidating these simple-minded distributions. Several examples are used to illustrate the competitive performance of the Gibbs ensemble numerically. We observe that all of the joint distributions in the ensemble have the same marginal distributions because incompatibility does not affect the transition densities. The Gibbs ensemble is justified via the mixed parameterization: the joint distribution of the dependence network is characterized by the recurrent marginal measures and odds ratios. Gibbs ensemble provides not only the marginal measures but also a range for every odds ratio. The weighting scheme preserves the marginal distributions and finds nearly optimal values among conflicting odds ratios. In the two examples having joint distributions found by optimization, Gibbs ensemble finds the same marginal measures and a similar odds ratio for one but an improved odds ratio for the other. In both cases, solutions derived from the ensemble are optimal. The ensemble method can produce a joint distribution with errors comparable to the mathematically optimal distribution. Also, due to the incompatibility among conditional specifications, the joint distributions of certain scan patterns may attain stationarity slowly. As long as its marginal stationarity has been confirmed by a diagnostic check, the Gibbs sampler can be stopped because (1) the final solution uses the compromise of all the odds ratios of the Gibbs ensemble, (2) if a specific odds ratio becomes too large or too small---caused by immature stopping---it will automatically be under-weighted, and (3) the marginal stationarity guarantees that the final solutions will have the correct marginal measures. Some robust properties are also illustrated. In addition, we discuss a possible application to multiple imputation of multivariate data via fully conditional specifications.