Entropy of a Gaussian random vector

Derivation of the Differential Entropy of a Gaussian Random Vector

Assume $\mathbf{X}$ is an $n$-dimensional Gaussian random vector, denoted as $\mathbf{X}\sim \mathcal{N}(\mathbf{\mu },\Sigma )$, where $\mathbf{\mu }\in {\mathbb{R}}^n$ is the mean vector and $\Sigma \in {\mathbb{R}}^{n\times n}$ is the symmetric positive-definite covariance matrix. The multivariate probability density function (PDF) is defined as:

$$f_{\mathbf{X}}(\mathbf{x})=\frac{1}{\sqrt{(2\pi {)}^n|\Sigma |}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^T{\Sigma }^{-1}(\mathbf{x}-\mathbf{\mu })\right)$$

The differential entropy $h(\mathbf{X})$ is derived by evaluating the expected value of the negative log-likelihood:

$$\begin{array}{rlrl}h(\mathbf{X})& =\mathbb{E}\left[{\log }_2\frac{1}{f_{\mathbf{X}}(\mathbf{X})}\right]& & \text{(Definition)}\\ & =\mathbb{E}\left[{\log }_2\left(\sqrt{(2\pi {)}^n|\Sigma |}\cdot \exp \left(\frac{1}{2}(\mathbf{X}-\mathbf{\mu }{)}^T{\Sigma }^{-1}(\mathbf{X}-\mathbf{\mu })\right)\right)\right]& & \text{(PDF Substitution)}\\ & =\frac{1}{2}{\log }_2\left((2\pi {)}^n|\Sigma |\right)+\frac{1}{2}\mathbb{E}\left[(\mathbf{X}-\mathbf{\mu }{)}^T{\Sigma }^{-1}(\mathbf{X}-\mathbf{\mu })\right]{\log }_{2}e& & \text{(Linearity of }\mathbb{E})\end{array}$$

To evaluate the expectation of the quadratic form, the Trace Trick is applied. Since the quadratic form is a scalar, it is equal to its own trace ($s=\text{tr}(s)$):

$$\begin{array}{rlrl}\mathbb{E}\left[(\mathbf{X}-\mathbf{\mu }{)}^T{\Sigma }^{-1}(\mathbf{X}-\mathbf{\mu })\right]& =\mathbb{E}\left[\text{tr}\left((\mathbf{X}-\mathbf{\mu }{)}^T{\Sigma }^{-1}(\mathbf{X}-\mathbf{\mu })\right)\right]& & \text{(Trace of a scalar)}\\ & =\mathbb{E}\left[\text{tr}\left({\Sigma }^{-1}(\mathbf{X}-\mathbf{\mu })(\mathbf{X}-\mathbf{\mu }{)}^T\right)\right]& & \text{(Cyclic property)}\\ & =\text{tr}\left({\Sigma }^{-1}\mathbb{E}\left[(\mathbf{X}-\mathbf{\mu })(\mathbf{X}-\mathbf{\mu }{)}^T\right]\right)& & \text{(Linearity of tr and }\mathbb{E})\\ & =\text{tr}\left({\Sigma }^{-1}\Sigma \right)& & \text{(Definition of }\Sigma )\\ & =\text{tr}\left(I_n\right)=n& & \text{(Trace of Identity }{I}_n)\end{array}$$

Substituting this result back into the entropy expression:

$$\begin{array}{rl}h(\mathbf{X})& =\frac{1}{2}{\log }_2\left((2\pi {)}^n|\Sigma |\right)+\frac{n}{2}{\log }_{2}e\\ & =\frac{1}{2}{\log }_2\left((2\pi {)}^n|\Sigma |\right)+\frac{1}{2}{\log }_2(e^n)\\ & =\frac{1}{2}{\log }_2\left((2\pi e{)}^n|\Sigma |\right)\end{array}$$

The final expression for the differential entropy of an $n$-dimensional Gaussian vector is:

$$\boxed{h(\mathbf{X})=\frac{1}{2}{\log }_2\left((2\pi e{)}^n|\Sigma |\right)\text{[bits]}}$$

Note

This result highlights that the entropy of a multivariate Gaussian depends solely on the dimension $n$ and the determinant of the covariance matrix $|\Sigma |$, reinforcing the principle that mean shifts ($\mathbf{\mu }$) do not affect the information content.