Softmax Properties

Consider an output layer with $C$ classes. Let

$${\mathbf{z}}^L=\left[\begin{array}{c}{z}_1^L\\ z_2^L\\ ⋮\\ z_C^L\end{array}\right]\in {\mathbb{R}}^C$$

denote the vector of logits, and let

$${\mathbf{a}}^L=\mathrm{softmax}({\mathbf{z}}^L)$$

denote the output activations, whose components are

$$a_j^L=\frac{e^{z_j^L}}{{\sum }_{m=1}^C{e}^{z_m^L}},j=1,\dots ,C.$$

As explained in Softmax Loss, the vector ${\mathbf{a}}^L$ is a probability distribution: each component is strictly positive and

$$\sum _{j=1}^C{a}_j^L=1.$$

Monotonicity and Competition Between Classes

One of the most important qualitative properties of softmax is the following:

• increasing the logit of one class increases its own output probability;
• the same perturbation decreases the probabilities assigned to all the other classes.

This statement can be made precise by studying the partial derivatives $\frac{\partial a_j^L}{\partial z_k^L}$.

Explicit Derivative Formulas

The derivative of the $j$-th softmax output with respect to the $k$-th logit is

$$\frac{\partial a_j^L}{\partial z_k^L}=a_j^L({\delta }_{jk}-a_k^L),$$

where ${\delta }_{jk}$ is the Kronecker delta. Therefore:

• for $j=k$,

$$\frac{\partial a_j^L}{\partial z_j^L}=a_j^L(1-a_j^L)>0;$$

• for $j\ne k$,

$$\frac{\partial a_j^L}{\partial z_k^L}=-a_j^L{a}_k^L<0.$$

These two formulas establish the monotonicity structure of the softmax layer:

• increasing $z_j^L$ increases $a_j^L$;
• increasing $z_j^L$ decreases every $a_k^L$ with $k\ne j$.

Intuition

The numerator $e^{z_j^L}$ tries to increase the weight assigned to class $j$, but the denominator is shared by all classes. Since the total probability mass must remain equal to $1$, any gain by one class must be compensated by losses elsewhere.

Important

The softmax layer therefore induces a genuine competition between classes. This is precisely the behavior needed in mutually exclusive multiclass classification: the classes do not make independent yes/no decisions, but compete for a single unit of probability mass.

Non-Locality of the Softmax Layer

For a sigmoid output unit, the activation is local:

$$a_j^l=\sigma (z_j^l).$$

Each output depends only on its own pre-activation.

Softmax behaves differently. In an output layer,

$$a_j^L=\frac{e^{z_j^L}}{{\sum }_{m=1}^C{e}^{z_m^L}},$$

so each output $a_j^L$ depends not only on $z_j^L$, but on all logits $z_1^L,\dots ,z_C^L$ through the common denominator.

This is why the softmax layer is non-local:

• changing $z_j^L$ affects $a_j^L$ directly;
• the same change also affects all other outputs indirectly.

The non-zero off-diagonal derivatives

$$\frac{\partial a_j^L}{\partial z_k^L}=-a_j^L{a}_k^L,j\ne k,$$

are the differential signature of this non-local coupling.

Local Versus Non-Local Output Layers

• A sigmoid output is local: each activation depends only on its own logit.
• A softmax output is non-local: every activation depends on all logits in the layer.
• This non-locality is exactly what allows the output to be interpreted as a categorical probability distribution.

Why the name “Soft-Max”

The origin of the name becomes transparent by introducing a sharpening parameter $\beta >0$ (the inverse temperature):

$$a_j^L(\beta )=\frac{e^{\beta z_j^L}}{{\sum }_{m=1}^C{e}^{\beta z_m^L}}.$$

The standard softmax corresponds to $\beta =1$.

Why “Soft-Max”?

1. For every $\beta >0$, the vector ${\mathbf{a}}^L(\beta )$ is still a probability distribution, because $$\sum _j{a}_j^L(\beta )=1,a_j^L(\beta )>0.$$
2. Let $$m=\underset{k}{\max }{z}_k^L\text{and}M=\{k:z_k^L=m\}$$ be the set of maximizers. Then $$\underset{\beta \to \infty }{\lim }{a}_j^L(\beta )=\{\begin{array}{ll}\frac{1}{|M|},& j\in M,\\ 0,& j\notin M.\end{array}$$
3. In particular, if the maximum is unique, say at class $j^{\star }$, then $$\underset{\beta \to \infty }{\lim }{a}_j^L(\beta )=\{\begin{array}{ll}1,& j=j^{\star },\\ 0,& j\ne j^{\star }.\end{array}$$ In that case the limit is the one-hot output associated with a hardmax selector.

This explains the name: softmax is a differentiable, softened version of a hardmax or argmax-like selection mechanism. At finite $\beta$, probability mass is distributed across all classes; as $\beta$ grows, the distribution becomes increasingly concentrated on the largest logit.

Equivalent Temperature Parameter

In modern machine learning practice, especially in the context of probabilistic decoding and large language models, the same family of transformations is usually parameterized by a temperature $T>0$ rather than by the inverse temperature $\beta$.

The two parameterizations are equivalent:

$$T=\frac{1}{\beta }⟺\beta =\frac{1}{T}.$$

Therefore the softmax may be written as

$$a_j^L(T)=\frac{e^{z_j^L/T}}{{\sum }_{m=1}^C{e}^{z_m^L/T}}.$$

This notation is often more intuitive in applications:

• as $T\to 0^{+}$, the distribution becomes increasingly concentrated on the largest logit and approaches a hardmax distribution when the maximizer is unique;
• as $T\to \infty$, all logits are flattened relative to one another and the distribution approaches the uniform distribution over classes.

Thus, low temperature makes the output more peaked and more deterministic, whereas high temperature makes it flatter and more random. In modern large language models, temperature is exactly the parameter used at sampling time to control how concentrated or exploratory the next-token distribution should be.

The same idea can be illustrated numerically. Consider the logits

$${\mathbf{z}}^L=\left[\begin{array}{c}2\\ 4\\ 2\\ 1\end{array}\right].$$

Then:

• a hardmax produces the one-hot vector $[0,1,0,0]$;
• a naive normalization of the positive scores would produce approximately $[0.2222,0.4444,0.2222,0.1111]$;
• the softmax output is approximately $[0.1025,0.7573,0.1025,0.0377]$.

Softmax is therefore much closer to a hard winner-take-most rule than plain normalization, while remaining fully differentiable.

Non-Linearity

Softmax is not a linear map. In general,

$$\mathrm{softmax}(\alpha \mathbf{u}+\beta \mathbf{v})\ne \alpha \mathrm{softmax}(\mathbf{u})+\beta \mathrm{softmax}(\mathbf{v}).$$

The reason is structural: every component contains an exponential in the numerator and a shared sum of exponentials in the denominator. Both operations destroy linearity.

A simple way to see the failure of homogeneity is to scale the logits. If

$$\mathbf{z}=\left[\begin{array}{c}1\\ 2\end{array}\right],$$

then

$$\mathrm{softmax}(\mathbf{z})\approx \left[\begin{array}{c}0.269\\ 0.731\end{array}\right],\mathrm{softmax}(2\mathbf{z})\approx \left[\begin{array}{c}0.119\\ 0.881\end{array}\right].$$

Doubling the logits does not double the probabilities. It sharpens the distribution by increasing the relative separation between the logits. Probability mass moves toward the largest component.

This is the same phenomenon described in the sharpening-parameter view above: scaling all logits by a positive factor does not preserve the distribution unless all logits are equal. It changes how concentrated the output is.

Permutation Equivariance

The softmax formula treats the coordinates of the logit vector symmetrically. No class index has a privileged role in the definition: each numerator is the exponential of one logit, and the denominator is the sum of the same exponentials over all coordinates.

Let $\mathbf{P}$ be a $C\times C$ permutation matrix. Then

$$\mathrm{softmax}(\mathbf{Pz})=\mathbf{P}\mathrm{softmax}(\mathbf{z}).$$

Permuting the logits only relabels the exponentials in the numerator, while the denominator remains the same sum, merely written in a different order. Consequently, if the logits are permuted, the output probabilities are permuted in exactly the same way.

Proof

Let $\pi$ be the permutation associated with $\mathbf{P}$ under the convention

$$(\mathbf{Pz}{)}_j=z_{\pi (j)}.$$

This means that the $j$-th coordinate of the permuted vector $\mathbf{Pz}$ is obtained by taking the $\pi (j)$-th coordinate of the original vector $\mathbf{z}$. Equivalently, $\mathbf{P}$ reorders the coordinates of $\mathbf{z}$ according to $\pi$.

With this convention, the $j$-th component of $\mathrm{softmax}(\mathbf{Pz})$ is

$${\left[\mathrm{softmax}(\mathbf{Pz})\right]}_j=\frac{e^{(\mathbf{Pz}{)}_j}}{{\sum }_{m=1}^C{e}^{(\mathbf{Pz}{)}_m}}=\frac{e^{z_{\pi (j)}}}{{\sum }_{m=1}^C{e}^{z_{\pi (m)}}}.$$

Since $\pi$ is a permutation, the denominator is the same sum with reordered terms:

$$\sum _{m=1}^C{e}^{z_{\pi (m)}}=\sum _{m=1}^C{e}^{z_m}.$$

Therefore

$${\left[\mathrm{softmax}(\mathbf{Pz})\right]}_j=\frac{e^{z_{\pi (j)}}}{{\sum }_{m=1}^C{e}^{z_m}}={\left[\mathbf{P}\mathrm{softmax}(\mathbf{z})\right]}_j.$$

Since the equality holds for every coordinate $j$,

$$\mathrm{softmax}(\mathbf{Pz})=\mathbf{P}\mathrm{softmax}(\mathbf{z}).$$

For example, if

$$\mathrm{softmax}\left(\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]\right)=\left[\begin{array}{c}{p}_1\\ p_2\\ p_3\end{array}\right],$$

then

$$\mathrm{softmax}\left(\left[\begin{array}{c}3\\ 1\\ 2\end{array}\right]\right)=\left[\begin{array}{c}{p}_3\\ p_1\\ p_2\end{array}\right].$$

The distinction from invariance matters. Softmax does not collapse the vector into a scalar quantity independent of coordinate order; it returns another vector whose coordinates still correspond to classes. A permutation of the input therefore induces the same permutation of the output.

Link with entropy of a random variable

This is closely related to the symmetry of entropy, but the type of symmetry is different. Using the notation introduced in Entropy: Definition and Properties, let $X$ be a discrete random variable defined over the alphabet ${\mathcal{A}}_X$ with probability mass function $p_X(x)$. Its entropy is

$$H(X)=\sum _{x\in {\mathcal{A}}_X:p_X(x)\ne 0}{p}_X(x){\log }_2\frac{1}{p_X(x)}.$$

Entropy depends on the probability masses, not on the order in which the elements of the alphabet are listed. If the probabilities are collected into a vector

$$\mathbf{p}=\left[\begin{array}{c}{p}_X(x_1)\\ ⋮\\ p_X(x_C)\end{array}\right],$$

then reordering the alphabet corresponds to replacing $\mathbf{p}$ by $\mathbf{Pp}$ for some permutation matrix $\mathbf{P}$. The entropy is unchanged because the defining sum only reorders its terms:

$$H(\mathbf{Pp})=H(\mathbf{p}).$$

Entropy is therefore permutation invariant as a scalar functional of a distribution. Softmax is permutation equivariant because it maps one indexed vector to another indexed vector:

$$\mathrm{softmax}(\mathbf{Pz})=\mathbf{P}\mathrm{softmax}(\mathbf{z}).$$

Equivariance, not invariance

A permutation-invariant map would satisfy $f(\mathbf{Pz})=f(\mathbf{z})$. Softmax does not satisfy this in general. It satisfies the stronger coordinate-respecting statement $f(\mathbf{Pz})=\mathbf{P}f(\mathbf{z})$.

Shift Invariance and Numerical Stability

Softmax is invariant under the addition of the same constant to all logits:

$$\mathrm{softmax}({\mathbf{z}}^L+\alpha 1)=\mathrm{softmax}({\mathbf{z}}^L)\text{for every }\alpha \in \mathbb{R},$$

where $1$ denotes the all-ones vector. Indeed,

$$\frac{e^{z_j^L+\alpha }}{{\sum }_m{e}^{z_m^L+\alpha }}=\frac{e^{\alpha }{e}^{z_j^L}}{e^{\alpha }{\sum }_m{e}^{z_m^L}}=\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}.$$

This property is mathematically important and computationally useful. In practice, one usually subtracts ${\max }_m{z}_m^L$ before exponentiating:

$$a_j^L=\frac{e^{z_j^L-{\max }_m{z}_m^L}}{{\sum }_k{e}^{z_k^L-{\max }_m{z}_m^L}}.$$

The result is unchanged, but numerical overflow is avoided. This is one of the reasons why modern implementations compute softmax and cross-entropy from logits in carefully stabilized form.

Derivative of Softmax and Jacobian Structure

Since softmax maps ${\mathbb{R}}^C$ to ${\mathbb{R}}^C$, its derivative is a Jacobian matrix:

$$J_{\text{softmax}}({\mathbf{z}}^L)=\frac{\partial {\mathbf{a}}^L}{\partial {\mathbf{z}}^L}=\left[\begin{array}{cccc}\frac{\partial a_1^L}{\partial z_1^L}& \frac{\partial a_1^L}{\partial z_2^L}& \cdots & \frac{\partial a_1^L}{\partial z_C^L}\\ \frac{\partial a_2^L}{\partial z_1^L}& \frac{\partial a_2^L}{\partial z_2^L}& \cdots & \frac{\partial a_2^L}{\partial z_C^L}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial a_C^L}{\partial z_1^L}& \frac{\partial a_C^L}{\partial z_2^L}& \cdots & \frac{\partial a_C^L}{\partial z_C^L}\end{array}\right].$$

To compute a generic entry, write

$$a_j^L=\frac{g_j}{h},g_j=e^{z_j^L},h=\sum _{m=1}^C{e}^{z_m^L}.$$

Then

$$\frac{\partial h}{\partial z_k^L}=e^{z_k^L},$$

while

$$\frac{\partial g_j}{\partial z_k^L}=\{\begin{array}{ll}{e}^{z_j^L},& j=k,\\ 0,& j\ne k.\end{array}$$

Applying the quotient rule gives

$$\frac{\partial a_j^L}{\partial z_k^L}=\frac{\frac{\partial g_j}{\partial z_k^L}h-g_j\frac{\partial h}{\partial z_k^L}}{h^2}.$$

Diagonal Entries

If $j=k$, then

$$\frac{\partial a_j^L}{\partial z_j^L}=\frac{e^{z_j^L}{\sum }_m{e}^{z_m^L}-e^{z_j^L}{e}^{z_j^L}}{{\left({\sum }_m{e}^{z_m^L}\right)}^2}=\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}\cdot \frac{{\sum }_m{e}^{z_m^L}-e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}=a_j^L(1-a_j^L).$$

Off-Diagonal Entries

If $j\ne k$, then

$$\frac{\partial a_j^L}{\partial z_k^L}=\frac{0\cdot {\sum }_m{e}^{z_m^L}-e^{z_j^L}{e}^{z_k^L}}{{\left({\sum }_m{e}^{z_m^L}\right)}^2}=-\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}\cdot \frac{e^{z_k^L}}{{\sum }_m{e}^{z_m^L}}=-a_j^L{a}_k^L.$$

Combining the two cases yields the compact formula

$$\frac{\partial a_j^L}{\partial z_k^L}=a_j^L({\delta }_{jk}-a_k^L).$$

Equivalently, the same derivative may be written in piecewise form as

$$\frac{\partial a_j^L}{\partial z_k^L}=\{\begin{array}{ll}{a}_j^L(1-a_j^L),& j=k,\\ -a_j^L{a}_k^L,& j\ne k.\end{array}$$

where the Kronecker delta is defined by

$${\delta }_{jk}=\{\begin{array}{ll}1,& j=k,\\ 0,& j\ne k.\end{array}$$

This is exactly the derivative used in Softmax Loss to prove that, under softmax plus negative log-likelihood,

$${\delta }_j^L=a_j^L-y_j.$$

Matrix Form of the Jacobian

If ${\mathbf{a}}^L$ is viewed as a column vector, the Jacobian can be written compactly as

$$J_{\text{softmax}}({\mathbf{z}}^L)=\mathrm{diag}({\mathbf{a}}^L)-{\mathbf{a}}^L({\mathbf{a}}^L{)}^{\top }.$$

Several important consequences follow immediately:

• the diagonal entries are positive;
• the off-diagonal entries are non-positive;
• each row sum is zero;
• each column sum is zero;
• the Jacobian is singular.

These facts encode the two defining structural constraints of softmax:

• the outputs must always sum to $1$;
• adding the same constant to all logits leaves the output unchanged.

Equivalently, softmax depends only on relative logits, not on their common offset.

Softmax in One Sentence

Softmax is a differentiable, non-local normalization map that transforms logits into a categorical probability distribution, induces competition between classes, and has Jacobian

$$J_{\text{softmax}}({\mathbf{z}}^L)=\mathrm{diag}({\mathbf{a}}^L)-{\mathbf{a}}^L({\mathbf{a}}^L{)}^{\top }.$$