Joint KL Divergence

Let $X$ and $Y$ be two discrete random variables defined on a joint sample space ${\mathcal{A}}_{\mathcal{X}}\times {\mathcal{A}}_{\mathcal{Y}}$. Consider two joint probability distributions $p(x,y)$ and $q(x,y)$. The joint Kullback-Leibler (KL) divergence quantifies the statistical discrepancy between these two joint probability mass functions.

Definition

The joint KL divergence is defined as the expected value of the logarithmic ratio between the joint probabilities, calculated with respect to distribution $p$:

$$D_{KL}(p_{XY}\parallel q_{XY})=\sum _{x\in {\mathcal{A}}_{\mathcal{X}}}\sum _{y\in {\mathcal{A}}_{\mathcal{Y}}}p(x,y){\log }_2\frac{p(x,y)}{q(x,y)}$$

In operator notation, this is expressed as:

$$D_{KL}(p_{XY}\parallel q_{XY})={\mathbb{E}}_{p(x,y)}\left[{\log }_2\frac{p(X,Y)}{q(X,Y)}\right]$$

The divergence is well-defined if and only if $p(x,y)\ll q(x,y)$ (joint absolute continuity), meaning that for every pair $(x,y)$, $q(x,y)=0$ implies $p(x,y)=0$.

---

Key Properties

1. Additivity under Independence

If the variables $X$ and $Y$ are independent under both distributions ($p(x,y)=p(x)p(y)$ and $q(x,y)=q(x)q(y)$), the joint divergence simplifies to the sum of the marginal divergences: $D_{KL}(p_{XY}\parallel q_{XY})=D_{KL}(p_X\parallel q_X)+D_{KL}(p_Y\parallel q_Y)$

2. Conditioning and Convexity

Conditioning does not reduce KL divergence in the same way it reduces entropy. This behavior is a direct consequence of the log-sum inequality and the convexity of the KL divergence with respect to the pair $(p,q)$.