Basics

Definition

Let $X$ be a discrete random variable defined over the alphabet ${\mathcal{A}}_X$ and distributed according to the probability mass function $p_X(x)$.

The entropy $H(X)$ is defined as:

$$H(X)=\mathbb{E}\left[{\log }_2\frac{1}{p(X)}\right]=\sum _{x\in {\mathcal{A}}_X:p(x)\ne 0}p(x){\log }_2\frac{1}{p(x)}[\text{bits}]$$

Mathematical Nuances and Notation

To ensure rigor while maintaining readability, this definition relies on specific conventions:

1. The Support and Singularity: The condition $p(x)\ne 0$ implicitly restricts the summation to the support of the distribution, defined as ${\mathcal{S}}_X=\{x\in {\mathcal{A}}_X\mid p_X(x)>0\}$. This restriction circumvents the mathematical singularity of $\log (0)$ for impossible events. Alternatively, the summation may cover the entire alphabet ${\mathcal{A}}_X$ by adopting the standard information-theoretic convention that $0\log 0=0$ (justified by continuity, as ${\lim }_{x\to 0}x\log x=0$).

2. Dependency on the Distribution: Writing $H(X)$ is technically a convenient abuse of notation. Entropy is not determined by the specific values (labels) of $X$, but exclusively by its probability mass function (PMF). Strictly speaking, entropy is a functional of the distribution itself, making $H(p_X)$ or $H(\mathbf{p})$ the more rigorous notation.

3. Explicit PMF Notation: The term $p(x)$ is used for brevity. Formally, this should be denoted as $p_X(x)$ to explicitly attribute the probability to the random variable $X$. This distinction becomes crucial when analyzing multiple random variables simultaneously (e.g., in joint entropy).

Change of Base and Units

The choice of the logarithmic base $b$ establishes the unit of information (e.g., bits for $b=2$, nats for $b=e$, bans for $b=10$). Conversion between these distinct units is facilitated by the logarithmic change-of-base identity:

$${\log }_b(u)={\log }_b(a)\cdot {\log }_a(u)$$

By substituting this identity into the definition of entropy and invoking the linearity of the expectation operator, the conversion formula is derived as follows:

$$\begin{array}{rl}{H}_b(X)& =\mathbb{E}\left[{\log }_b\frac{1}{p(X)}\right]\\ & =\mathbb{E}\left[{\log }_b(a)\cdot {\log }_a\frac{1}{p(X)}\right]\\ & ={\log }_b(a)\cdot \mathbb{E}\left[{\log }_a\frac{1}{p(X)}\right]\\ & ={\log }_b(a)\cdot H_a(X)\end{array}$$

Thus, entropy expressed in one unit is strictly a scalar multiple of entropy expressed in another (e.g., $H_e(X)=\ln (2)\cdot H_2(X)$).

Convention: In accordance with standard information-theoretic practice, the binary logarithm ($b=2$) is adopted as the default for all subsequent analysis. Therefore, unless explicitly stated otherwise, entropy is quantified in bits.

Non-negativity of Entropy

The entropy of a discrete random variable $X$ is always non-negative: $H(X)\ge 0$This lower bound is established by analyzing the components of the definition:

1. Bounded Probabilities By definition, the probability mass function satisfies $0<p_X(x)\le 1$ for all $x$ in the support of $X$.

2. Positivity of Self-Information Consequently, the argument of the logarithm, $\frac{1}{p_X(x)}$, lies within the interval $[1,\infty )$. Since the logarithm is a monotonically increasing function with ${\log }_2(1)=0$, the variable representing self-information is non-negative:

   $${\log }_2\frac{1}{p_X(x)}\ge 0$$

3. Monotonicity of Expectation Finally, invoking the property that the expectation of a non-negative random variable is itself non-negative (i.e., if $Y\ge 0$, then $\mathbb{E}[Y]\ge 0$), the result is derived:

   $$H(X)=\mathbb{E}\left[{\log }_2\frac{1}{p(X)}\right]\ge 0$$

Note

Equality (i.e., $H(X)=0$) holds if and only if the random variable $X$ is deterministic (i.e., there exists an outcome $x$ such that $p_X(x)=1$).

---

Example: Entropy of a Bernoulli r.v

Let $X$ i.e., X is a Benoulli