Basics

Definition

While the definition of differential entropy is straightforward, several notational conventions and mathematical requirements must be addressed for a rigorous understanding.

$$h(X)=\mathbb{E}\left[{\log }_2\frac{1}{f(X)}\right]={\int }_{\mathbb{R}}f(x){\log }_2\frac{1}{f(x)}dx\text{[bits]}$$

Abuse of Notation: $h(X)$

The notation $h(X)$ is technically an abuse of notation. Unlike a standard function of a random variable, entropy does not depend on the specific values or outcomes of $X$. Instead, it is a functional of the probability density function $f_X(x)$. A more formally correct notation would be $h(f_X)$, as the entropy remains invariant regardless of whether the variable is named $X$, $Y$, or $Z$, provided they share the same distribution.

Abuse of Notation: $f(X)$

In the expression $\mathbb{E}[\log \frac{1}{f(X)}]$, the term $f(X)$ represents the density function evaluated at the random variable $X$. This makes $f(X)$ itself a random variable. The expectation is taken with respect to the density $f(x)$, effectively “averaging” the log-density over all possible realizations.

Existence of the Integral

The definition of $h(X)$ is only valid if the integral $\int f(x)\log f(x)dx$ exists and is finite. Unlike discrete entropy $H(X)$, which is always non-negative and bounded for finite alphabets, differential entropy can be positive, negative, or even $-\infty$ (e.g., for a Dirac delta distribution). For the entropy to be well-defined, the PDF must be absolutely integrable in the context of the Shannon information measure.

Support of the Integral

The integral is formally evaluated over the support of $X$, denoted as $\mathcal{S}=\{x\in \mathbb{R}:f(x)>0\}$. By convention, the term $0\log \frac{1}{0}$ is treated as $0$, which is justified by the limit ${\lim }_{p\to 0^{+}}p\log \frac{1}{p}=0$.