Modeling a joint distribution via a collection of conditional distributions is call a conditionally specified model (CSM). Graphically, a conditional model is depicted by a cyclic directed graph with directed arrows from the conditioning variables (nodes) pointing to the conditioned variable. Computer scientists refer to CSM as dependency network (DN), and have used it for probabilistic inference and collaborative filtering. The Gibbs sampler (GS) has been the (only) computational tool for DN. In many applications, the conditional distributions are derived empirically without regard to compatibility, which makes the convergence of the GS problematic. CSM should be regarded as an unsupervised learning task and calls for an algorithmic approach; we invented the iterative conditional replacement algorithm (ICR) for this purpose. ICR works with distributions, not samples. First, closed sets of joint/marginal distributions are defined by the conditional distributions, then I-projection onto those sets are performed cyclically. Operationally, I-projection is replacing the conditional distribution and keeping the marginal distribution unchanged; accordingly, we call it ICR. Regardless of compatibility, ICR will always converge and produces one stationary distribution in each set it projects. ICR creates multiple sequences of distributions, and each sequence converges to a unique stationary distribution. We call the output of ICR mutually stationary distributions; at that point, the I-projection among them becomes ineffectual. For CSM consists of full and non-full conditionals, updating orders are critical for ICR to reach the correct distributions. For that reason, we propose two criteria to identify such updating orders, so that the dependence coded in a CSM is transferred to the stationary distributions. The collection of all mutually stationary distributions forms an ensemble, which will be combined into a joint distribution representing the CSM. We show that conditional replacement, I-projection, and conditional expectation are the same, which is also an orthogonal projection onto the Hilbert spaces of joint or marginal distributions. Therefore, ICR can be understood as a method of alternating projection.

Please click here for participating the talk online