Softmax Properties

Consider an output layer with $K$ classes. Let

$${\mathbf{z}}^L=\left[\begin{array}{c}{z}_1^L\\ z_2^L\\ ⋮\\ z_K^L\end{array}\right]\in {\mathbb{R}}^K$$

denote the vector of logits, and let

$${\mathbf{a}}^L=\mathrm{softmax}({\mathbf{z}}^L)$$

denote the output activations, whose components are

$$a_j^L=\frac{e^{z_j^L}}{{\sum }_{m=1}^K{e}^{z_m^L}},j=1,\dots ,K.$$

As explained in Softmax Loss, the vector ${\mathbf{a}}^L$ is a probability distribution: each component is strictly positive and

$$\sum _{j=1}^K{a}_j^L=1.$$

Monotonicity and Competition Between Classes

One of the most important qualitative properties of softmax is the following:

• increasing the logit of one class increases its own output probability;
• the same perturbation decreases the probabilities assigned to all the other classes.

This statement can be made precise by studying the partial derivatives $\frac{\partial a_j^L}{\partial z_k^L}$.

Explicit Derivative Formulas

The derivative of the $j$-th softmax output with respect to the $k$-th logit is

$$\frac{\partial a_j^L}{\partial z_k^L}=a_j^L({\delta }_{jk}-a_k^L),$$

where ${\delta }_{jk}$ is the Kronecker delta. Therefore:

• for $j=k$,

$$\frac{\partial a_j^L}{\partial z_j^L}=a_j^L(1-a_j^L)>0;$$

• for $j\ne k$,

$$\frac{\partial a_j^L}{\partial z_k^L}=-a_j^L{a}_k^L<0.$$

These two formulas establish the monotonicity structure of the softmax layer:

• increasing $z_j^L$ increases $a_j^L$;
• increasing $z_j^L$ decreases every $a_k^L$ with $k\ne j$.

Intuition

The numerator $e^{z_j^L}$ tries to increase the weight assigned to class $j$, but the denominator is shared by all classes. Since the total probability mass must remain equal to $1$, any gain by one class must be compensated by losses elsewhere.

Important

The softmax layer therefore induces a genuine competition between classes. This is precisely the behavior needed in mutually exclusive multiclass classification: the classes do not make independent yes/no decisions, but compete for a single unit of probability mass.

Non-Locality of the Softmax Layer

For a sigmoid output unit, the activation is local:

$$a_j^l=\sigma (z_j^l).$$

Each output depends only on its own pre-activation.

Softmax behaves differently. In an output layer,

$$a_j^L=\frac{e^{z_j^L}}{{\sum }_{m=1}^K{e}^{z_m^L}},$$

so each output $a_j^L$ depends not only on $z_j^L$, but on all logits $z_1^L,\dots ,z_K^L$ through the common denominator.

This is why the softmax layer is non-local:

• changing $z_j^L$ affects $a_j^L$ directly;
• the same change also affects all other outputs indirectly.

The non-zero off-diagonal derivatives

$$\frac{\partial a_j^L}{\partial z_k^L}=-a_j^L{a}_k^L,j\ne k,$$

are the differential signature of this non-local coupling.

Local Versus Non-Local Output Layers

• A sigmoid output is local: each activation depends only on its own logit.
• A softmax output is non-local: every activation depends on all logits in the layer.
• This non-locality is exactly what allows the output to be interpreted as a categorical probability distribution.

Why the name “Soft-Max”

The origin of the name becomes transparent by introducing a sharpening parameter $c>0$:

$$a_j^L(c)=\frac{e^{c{z}_j^L}}{{\sum }_{m=1}^K{e}^{c{z}_m^L}}.$$

The standard softmax corresponds to $c=1$.

Why “Soft-Max”?

1. For every $c>0$, the vector ${\mathbf{a}}^L(c)$ is still a probability distribution, because $$\sum _j{a}_j^L(c)=1,a_j^L(c)>0.$$
2. Let $$m=\underset{k}{\max }{z}_k^L\text{and}M=\{k:z_k^L=m\}$$ be the set of maximizers. Then $$\underset{c\to \infty }{\lim }{a}_j^L(c)=\{\begin{array}{ll}\frac{1}{|M|},& j\in M,\\ 0,& j\notin M.\end{array}$$
3. In particular, if the maximum is unique, say at class $j^{\star }$, then $$\underset{c\to \infty }{\lim }{a}_j^L(c)=\{\begin{array}{ll}1,& j=j^{\star },\\ 0,& j\ne j^{\star }.\end{array}$$ In that case the limit is the one-hot output associated with a hardmax selector.

This explains the name: softmax is a differentiable, softened version of a hardmax or argmax-like selection mechanism. At finite $c$, probability mass is distributed across all classes; as $c$ grows, the distribution becomes increasingly concentrated on the largest logit.

Equivalent Temperature Parameter

In modern machine learning practice, especially in the context of probabilistic decoding and large language models, the same family of transformations is usually parameterized by a temperature $T>0$ rather than by a sharpening parameter $c$.

The two parameterizations are equivalent:

$$T=\frac{1}{c}⟺c=\frac{1}{T}.$$

Therefore the softmax may be written as

$$a_j^L(T)=\frac{e^{z_j^L/T}}{{\sum }_{m=1}^K{e}^{z_m^L/T}}.$$

This notation is often more intuitive in applications:

• as $T\to 0^{+}$, the distribution becomes increasingly concentrated on the largest logit and approaches a hardmax distribution when the maximizer is unique;
• as $T\to \infty$, all logits are flattened relative to one another and the distribution approaches the uniform distribution over classes.

Thus, low temperature makes the output more peaked and more deterministic, whereas high temperature makes it flatter and more random. In modern large language models, temperature is exactly the parameter used at sampling time to control how concentrated or exploratory the next-token distribution should be.

The same idea can be illustrated numerically. Consider the logits

$${\mathbf{z}}^L=\left[\begin{array}{c}2\\ 4\\ 2\\ 1\end{array}\right].$$

Then:

• a hardmax produces the one-hot vector $[0,1,0,0]$;
• a naive normalization of the positive scores would produce approximately $[0.2222,0.4444,0.2222,0.1111]$;
• the softmax output is approximately $[0.1025,0.7573,0.1025,0.0377]$.

Softmax is therefore much closer to a hard winner-take-most rule than plain normalization, while remaining fully differentiable.

Shift Invariance and Numerical Stability

Softmax is invariant under the addition of the same constant to all logits:

$$\mathrm{softmax}({\mathbf{z}}^L+\alpha 1)=\mathrm{softmax}({\mathbf{z}}^L)\text{for every }\alpha \in \mathbb{R},$$

where $1$ denotes the all-ones vector. Indeed,

$$\frac{e^{z_j^L+\alpha }}{{\sum }_m{e}^{z_m^L+\alpha }}=\frac{e^{\alpha }{e}^{z_j^L}}{e^{\alpha }{\sum }_m{e}^{z_m^L}}=\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}.$$

This property is mathematically important and computationally useful. In practice, one usually subtracts ${\max }_m{z}_m^L$ before exponentiating:

$$a_j^L=\frac{e^{z_j^L-{\max }_m{z}_m^L}}{{\sum }_k{e}^{z_k^L-{\max }_m{z}_m^L}}.$$

The result is unchanged, but numerical overflow is avoided. This is one of the reasons why modern implementations compute softmax and cross-entropy from logits in carefully stabilized form.

Derivative of Softmax and Jacobian Structure

Since softmax maps ${\mathbb{R}}^K$ to ${\mathbb{R}}^K$, its derivative is a Jacobian matrix:

$$J_{\text{softmax}}({\mathbf{z}}^L)=\frac{\partial {\mathbf{a}}^L}{\partial {\mathbf{z}}^L}=\left[\begin{array}{cccc}\frac{\partial a_1^L}{\partial z_1^L}& \frac{\partial a_1^L}{\partial z_2^L}& \cdots & \frac{\partial a_1^L}{\partial z_K^L}\\ \frac{\partial a_2^L}{\partial z_1^L}& \frac{\partial a_2^L}{\partial z_2^L}& \cdots & \frac{\partial a_2^L}{\partial z_K^L}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial a_K^L}{\partial z_1^L}& \frac{\partial a_K^L}{\partial z_2^L}& \cdots & \frac{\partial a_K^L}{\partial z_K^L}\end{array}\right].$$

To compute a generic entry, write

$$a_j^L=\frac{g_j}{h},g_j=e^{z_j^L},h=\sum _{m=1}^K{e}^{z_m^L}.$$

Then

$$\frac{\partial h}{\partial z_k^L}=e^{z_k^L},$$

while

$$\frac{\partial g_j}{\partial z_k^L}=\{\begin{array}{ll}{e}^{z_j^L},& j=k,\\ 0,& j\ne k.\end{array}$$

Applying the quotient rule gives

$$\frac{\partial a_j^L}{\partial z_k^L}=\frac{\frac{\partial g_j}{\partial z_k^L}h-g_j\frac{\partial h}{\partial z_k^L}}{h^2}.$$

Diagonal Entries

If $j=k$, then

$$\frac{\partial a_j^L}{\partial z_j^L}=\frac{e^{z_j^L}{\sum }_m{e}^{z_m^L}-e^{z_j^L}{e}^{z_j^L}}{{\left({\sum }_m{e}^{z_m^L}\right)}^2}=\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}\cdot \frac{{\sum }_m{e}^{z_m^L}-e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}=a_j^L(1-a_j^L).$$

Off-Diagonal Entries

If $j\ne k$, then

$$\frac{\partial a_j^L}{\partial z_k^L}=\frac{0\cdot {\sum }_m{e}^{z_m^L}-e^{z_j^L}{e}^{z_k^L}}{{\left({\sum }_m{e}^{z_m^L}\right)}^2}=-\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}\cdot \frac{e^{z_k^L}}{{\sum }_m{e}^{z_m^L}}=-a_j^L{a}_k^L.$$

Combining the two cases yields the compact formula

$$\frac{\partial a_j^L}{\partial z_k^L}=a_j^L({\delta }_{jk}-a_k^L).$$

Equivalently, the same derivative may be written in piecewise form as

$$\frac{\partial a_j^L}{\partial z_k^L}=\{\begin{array}{ll}{a}_j^L(1-a_j^L),& j=k,\\ -a_j^L{a}_k^L,& j\ne k.\end{array}$$

where the Kronecker delta is defined by

$${\delta }_{jk}=\{\begin{array}{ll}1,& j=k,\\ 0,& j\ne k.\end{array}$$

This is exactly the derivative used in Softmax Loss to prove that, under softmax plus negative log-likelihood,

$${\delta }_j^L=a_j^L-y_j.$$

Matrix Form of the Jacobian

If ${\mathbf{a}}^L$ is viewed as a column vector, the Jacobian can be written compactly as

$$J_{\text{softmax}}({\mathbf{z}}^L)=\mathrm{diag}({\mathbf{a}}^L)-{\mathbf{a}}^L({\mathbf{a}}^L{)}^{\top }.$$

Several important consequences follow immediately:

• the diagonal entries are positive;
• the off-diagonal entries are non-positive;
• each row sum is zero;
• each column sum is zero;
• the Jacobian is singular.

These facts encode the two defining structural constraints of softmax:

• the outputs must always sum to $1$;
• adding the same constant to all logits leaves the output unchanged.

Equivalently, softmax depends only on relative logits, not on their common offset.

Softmax in One Sentence

Softmax is a differentiable, non-local normalization map that transforms logits into a categorical probability distribution, induces competition between classes, and has Jacobian

$$J_{\text{softmax}}({\mathbf{z}}^L)=\mathrm{diag}({\mathbf{a}}^L)-{\mathbf{a}}^L({\mathbf{a}}^L{)}^{\top }.$$