Basics

Defintion

I (X; Y) = D_{K L} (p_{X Y} ∥ p_{X} p_{Y}) = E [lo g_{2} \frac{p _{X Y}}{p _{X} p _{Y}}] = \sum p (x, y) lo g_{2} \frac{p ( x , y )}{p ( x ) p ( y )}

The mutual information measure the degree of dependence between two r.v.s