Entropy of a Gaussian r.v.

Derivation of the Differential Entropy of a Gaussian Distribution

Assume $X$ is a Gaussian random variable, denoted as $X\sim \mathcal{N}(\mu ,{\sigma }^2)$. The probability density function (PDF) is defined as:

$$f_X(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$$

Based on the definition of differential entropy, the uncertainty associated with this distribution is derived as follows:

$$\begin{array}{rlrl}h(X)& =\mathbb{E}\left[{\log }_2\frac{1}{f(X)}\right]& & \text{(Definition)}\\ & =\mathbb{E}\left[{\log }_2\left(\frac{1}{\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(X-\mu {)}^2}{2{\sigma }^2}\right)}\right)\right]& & \text{(PDF Substitution)}\\ & =\mathbb{E}\left[{\log }_2\left(\sqrt{2\pi {\sigma }^2}\cdot \exp \left(\frac{(X-\mu {)}^2}{2{\sigma }^2}\right)\right)\right]& & \text{(Reciprocal Property)}\\ & =\mathbb{E}\left[\frac{1}{2}{\log }_2(2\pi {\sigma }^2)+\frac{(X-\mu {)}^2}{2{\sigma }^2}{\log }_{2}e\right]& & \text{(Logarithmic Properties)}\\ & =\frac{1}{2}{\log }_2(2\pi {\sigma }^2)+\frac{\mathbb{E}[(X-\mu {)}^2]}{2{\sigma }^2}{\log }_{2}e& & \text{(Linearity of }\mathbb{E})\\ & =\frac{1}{2}{\log }_2(2\pi {\sigma }^2)+\frac{{\sigma }^2}{2{\sigma }^2}{\log }_{2}e& & \text{(Definition of Variance)}\\ & =\frac{1}{2}{\log }_2(2\pi {\sigma }^2)+\frac{1}{2}{\log }_{2}e& & \text{(Simplification)}\\ & =\frac{1}{2}{\log }_2(2\pi e{\sigma }^2)& & \text{(Grouping Terms)}\end{array}$$

The final result demonstrates that the entropy is a logarithmic function of the variance:

$$\boxed{h(X)=\frac{1}{2}{\log }_2(2\pi e{\sigma }^2)\text{[bits]}}$$

Note

The result confirms that the uncertainty depends solely on the “spread” of the distribution (${\sigma }^2$) and is independent of its location ($\mu$).