The Chain Rule for KL Divergence

A fundamental property of the joint KL divergence is its recursive decomposition, which is analogous to the chain rule for joint entropy:

$$D_{KL}(p(x,y)\parallel q(x,y))=D_{KL}(p(x)\parallel q(x))+D_{KL}(p(y|x)\parallel q(y|x))$$

Where $D_{KL}(p(y|x)\parallel q(y|x))$ defines the conditional KL divergence, expressed as:

$$D_{KL}(p(y|x)\parallel q(y|x))=\sum _{x\in \mathcal{X}}p(x)\sum _{y\in \mathcal{Y}}p(y|x){\log }_2\frac{p(y|x)}{q(y|x)}$$

Information Theoretic Interpretation

The total divergence between two joint models is the sum of the divergence between their marginal distributions and the expected divergence between their conditional distributions.