Jensen's inequality

Statement

Let:

• $\mathbf{X}$ be a random vector in ${\mathbb{R}}^n$ with alphabet ${\mathcal{A}}_{\mathbf{X}}$ (i.e. the set of values that $\mathbf{X}$ can take)

• $C\subseteq {\mathbb{R}}^n$ be a convex set such that ${\mathcal{A}}_{\mathbf{X}}\subseteq C$

• $f:C\to (-\infty ,+\infty ]$ be a convex function

  Assuming that $\mathbb{E}[\mathbf{X}]$ exists and that $\mathbb{E}[f(\mathbf{X})]$ is well defined (possibly equal to $+\infty$), Jensen’s Inequality states that:

$$f(\mathbb{E}[\mathbf{X}])\le \mathbb{E}[f(\mathbf{X})]$$

If, moreover, $f$ is strictly convex, then equality holds if and only if $\mathbf{X}$ is almost surely constant, equivalently if and only if $\mathbf{X}=\mathbb{E}[\mathbf{X}]$ almost surely.

In the remainder, the scalar case $n=1$ is considered, so the random vector $\mathbf{X}$ is denoted by the random variable $X$.

Note

Although the statement holds for both discrete and continuous random variables, the proof below treats only the discrete case.

Proof (Discrete case)

Let $M=\text{card}({\mathcal{A}}_X)$ denote the number of distinct values that the random variable $X$ can assume. The proof proceeds by mathematical induction on $M$.

1. Base Case ($M=2$)

Assume that $X$ takes only two values, $x$ and $y$ in $C$, with probabilities $p$ and $1-p$ respectively.

By the definition of a convex function, the image of a convex combination is always dominated by the convex combination of the images. Therefore, it trivially follows that:

$$f(\mathbb{E}[X])=f(px+(1-p)y)\le pf(x)+(1-p)f(y)=\mathbb{E}[f(X)]$$

The base case is verified.

2. Induction Hypothesis ($M=k$)

Assume the inequality holds true for any discrete random variable taking exactly $M=k$ distinct values.

3. Inductive Step ($M=k+1$)

It must be shown that if the proposition holds for $k$, it necessarily holds for $k+1$. Consider a random variable $X$ defined by the following probability distribution:

Point in $C$	Probability
$x_1$	$p_1$
$x_2$	$p_2$
$\dots$	$\dots$
$x_k$	$p_k$
$x_{k+1}$	$p_{k+1}$

The expected value $\mathbb{E}[f(X)]$ is given by the sum over all $k+1$ states:

$$\mathbb{E}[f(X)]=\sum _{i=1}^{k+1}{p}_{i}f(x_i)$$

Isolate the $(k+1)$-th term from the rest of the summation:

$$\mathbb{E}[f(X)]=\sum _{i=1}^k{p}_{i}f(x_i)+p_{k+1}f(x_{k+1})$$

To manipulate the summation into a form where the induction hypothesis can be applied, multiply and divide the first term by $(1-p_{k+1})$, bringing the denominator inside the summation:

$$\mathbb{E}[f(X)]=(1-p_{k+1})\sum _{i=1}^k\frac{p_i}{1-p_{k+1}}f(x_i)+p_{k+1}f(x_{k+1})$$

Proof of legitimate convex combination coefficients

Before proceeding, it must be verified that the newly formed terms $\frac{p_i}{1-p_{k+1}}$ constitute a valid probability distribution. They must be strictly positive and sum to exactly $1$ to act as legitimate convex combination coefficients.

1. Positivity: Since $p_i\ge 0$ and $p_{k+1}<1$ (assuming a non-degenerate distribution), the ratio $\frac{p_i}{1-p_{k+1}}$ is intrinsically positive.
2. Summation to 1: Summing these coefficients yields:

$$\sum _{i=1}^k\frac{p_i}{1-p_{k+1}}=\frac{1}{1-p_{k+1}}\sum _{i=1}^k{p}_i$$

Knowing that the total probability must equal 1 (${\sum }_{i=1}^{k+1}{p}_i=1$), it follows that the partial sum is ${\sum }_{i=1}^k{p}_i=1-p_{k+1}$. Substituting this back provides:

$$\frac{1}{1-p_{k+1}}(1-p_{k+1})=1$$

Thus, these coefficients represent a valid probability distribution over $k$ elements.

Applying the Induction and Concluding

Since the terms inside the summation represent a valid convex combination of $k$ elements, the induction hypothesis ($M=k$) can now be applied to the sum:

$$\sum _{i=1}^k\frac{p_i}{1-p_{k+1}}f(x_i)\ge f\left(\sum _{i=1}^k\frac{p_i}{1-p_{k+1}}{x}_i\right)$$

Substituting this inequality back into the main equation yields:

$$\mathbb{E}[f(X)]\ge (1-p_{k+1})f\left(\sum _{i=1}^k\frac{p_i}{1-p_{k+1}}{x}_i\right)+p_{k+1}f(x_{k+1})$$

The resulting expression is a convex combination of exactly two images, weighted by $(1-p_{k+1})$ and $p_{k+1}$. This reduces the problem back to the base case ($M=2$). Applying the definition of convexity once more:

$$\mathbb{E}[f(X)]\ge f\left((1-p_{k+1})\left[\sum _{i=1}^k\frac{p_i}{1-p_{k+1}}{x}_i\right]+p_{k+1}{x}_{k+1}\right)$$

Simplifying the term $(1-p_{k+1})$ inside the function’s argument gives:

$$\mathbb{E}[f(X)]\ge f\left(\sum _{i=1}^k{p}_i{x}_i+p_{k+1}{x}_{k+1}\right)$$

Recombining the terms inside the argument restores the full expected value $\mathbb{E}[X]$:

$$\mathbb{E}[f(X)]\ge f\left(\sum _{i=1}^{k+1}{p}_i{x}_i\right)=f(\mathbb{E}[X])$$

This completes the proof by induction.