Softmax Loss

An alternative way to address the learning slowdown problem at the output layer is to replace independent sigmoid outputs with a softmax output layer.

The pre-activation of the $j$-th output neuron is still computed in the usual way:

$$z_j^L=\sum _k{w}_{jk}^L{a}_k^{L-1}+b_j^L.$$

What changes is the output activation. In a softmax layer, the activation of neuron $j$ is defined by

$$\boxed{a_j^L=\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}.}$$

The denominator contains a sum over all output neurons. Consequently, each output depends not only on its own logit $z_j^L$, but also on all the others.

Consider a network with four output neurons, with fixed values $z_1^L=-1$, $z_2^L=0$, $z_3^L=1$, and varying $z_4^L$. The following plot shows how the four softmax activations change as $z_4^L$ varies.

As $z_4^L$ increases, the corresponding activation $a_4^L$ increases, while the other activations decrease. The same phenomenon occurs for any other class: increasing one logit reallocates probability mass toward that class and away from the others.

This behavior follows directly from the defining equation. Indeed,

$$\sum _j{a}_j^L=\sum _j\frac{e^{z_j^L}}{{\sum }_m{e}^{z_m^L}}=\frac{{\sum }_j{e}^{z_j^L}}{{\sum }_m{e}^{z_m^L}}=1.$$

Moreover, every activation is strictly positive because exponentials are strictly positive. Therefore, the output of a softmax layer is a vector of positive numbers whose sum is $1$. It can therefore be interpreted as a probability distribution over the classes.

Probabilistic Interpretation

In multiclass classification with mutually exclusive classes, it is natural to interpret $a_j^L$ as the model estimate of

$$P(\text{class }j\mid x).$$

For example, in digit recognition, $a_j^L$ may be interpreted as the estimated probability that the input image belongs to class $j$.

This interpretation is specific to the softmax output layer. A layer of independent sigmoids does not generally produce outputs that sum to $1$, and therefore does not define a probability distribution over mutually exclusive classes. That setting corresponds instead to multilabel logistic loss, where each class is treated independently.

Why Softmax Produces a Probability Distribution

• The exponential in the numerator guarantees positivity.
• The shared denominator enforces normalization.
• The resulting output can therefore be interpreted as a categorical probability distribution.

---

Softmax Loss

When a softmax output layer is combined with a one-hot target vector $y$, the standard loss is the negative log-likelihood

$$\mathcal{L}=-\sum _k{y}_k\log a_k^L.$$

If the correct class is $c$, then $y_c=1$ and all other components vanish, so the loss becomes

$$\mathcal{L}=-\log a_c^L=-\log \left(\frac{e^{z_c^L}}{{\sum }_m{e}^{z_m^L}}\right).$$

This quantity is often called softmax loss. Equivalently,

$$\mathcal{L}=-z_c^L+\log \sum _m{e}^{z_m^L}.$$

This second form makes it clear that the loss compares the correct-class logit against the log-normalizer induced by all classes. In modern libraries, it is usually implemented directly from the logits for numerical stability.

Backpropagation with Softmax + Negative Log-Likelihood

Let

$$\mathcal{L}=-\sum _k{y}_k\log a_k^L,{\delta }_j^L\equiv \frac{\partial \mathcal{L}}{\partial z_j^L}.$$

Then

$$\boxed{{\delta }_j^L=a_j^L-y_j.}$$

This result follows from the chain rule together with the derivative of the softmax:

$$\frac{\partial a_k^L}{\partial z_j^L}=a_k^L({\delta }_{kj}-a_j^L).$$

Indeed,

$${\delta }_j^L=\sum _k\frac{\partial \mathcal{L}}{\partial a_k^L}\frac{\partial a_k^L}{\partial z_j^L}=-\sum _k\frac{y_k}{a_k^L}{a}_k^L({\delta }_{kj}-a_j^L),$$

and therefore

$${\delta }_j^L=-\sum _k{y}_k({\delta }_{kj}-a_j^L)=-y_j+a_j^L\sum _k{y}_k.$$

Since ${\sum }_k{y}_k=1$ for a one-hot target, it follows that

$${\delta }_j^L=a_j^L-y_j.$$

The expression ${\delta }_j^L=a_j^L-y_j$ is the key reason why softmax paired with negative log-likelihood avoids the main training pathology associated with saturated output activations. The derivative of the loss with respect to the logits collapses to a simple prediction-minus-target term, without an additional small multiplicative factor analogous to a sigmoid derivative at the output. Once this term is obtained, the rest of backpropagation proceeds exactly as usual. For a fuller discussion of the softmax Jacobian and its structural properties, see Softmax properties.

Logits Versus Probabilities at Evaluation Time

Softmax maps logits from $(-\infty ,+\infty )$ into probabilities in $(0,1)$. This is appropriate when a normalized probabilistic interpretation is required, but it also compresses score differences.

For this reason, many ranking-based evaluation metrics are often computed from the logits rather than from the post-softmax probabilities:

• logits are often preferable for ROC curves, AUC, and precision-recall analysis;
• softmax probabilities are preferable when a normalized class distribution is needed.

In practical implementations, this distinction is important:

• during training, frameworks usually accept logits and apply the softmax or log-softmax internally for numerical stability;
• during evaluation, the choice between logits and probabilities depends on the metric and on the desired interpretation.

Note

In libraries such as PyTorch, CrossEntropyLoss expects logits, not already-normalized probabilities. Mathematically, this still corresponds to softmax followed by negative log-likelihood; the softmax is simply handled inside the loss function in a numerically stable way.

Compressed Score Dynamics: Practical Consequences

Softmax converts arbitrary logits into normalized probabilities, but this normalization also compresses the score scale. As a consequence, different classes may appear more tightly clustered after softmax than they were at the logit level.

This distinction matters in practice:

• during training, softmax plus negative log-likelihood is appropriate because it defines a probabilistic loss;
• during evaluation, logits are often preferable for ranking-based analyses because they preserve a wider score dynamic range.

In ROC or precision-recall analysis, using post-softmax probabilities can sometimes produce a score distribution with less numerical spread than the corresponding logits. In such cases, the resulting curve may look less resolved than the one obtained from raw logits. This is not a universal rule, but it is a common practical reason to retain logits for evaluation while still using softmax-based losses during training.