Cross Entropy Loss

Building of a new loss

Consider again the example of the single neuron. However, the task is no longer to map its unit input to zero.

🎯 Goal

The goal is to drop the derivative terms ${\sigma }^{\prime }(z)$ from the loss function derivatives (which drive gradient descent), in order to avoid the vanishing-gradient problem in the saturation region of the sigmoid.

$$\begin{array}{rlrl}\frac{\partial {\mathcal{L}}_x}{\partial w}& =(a-y){\sigma }^{\prime }(z)x& ⟶& (a-y)x\\ & & & \\ \frac{\partial {\mathcal{L}}_x}{\partial b}& =(a-y){\sigma }^{\prime }(z)& ⟶& (a-y)\end{array}$$

${\sigma }^{\prime }(z)$ is a critical multiplicative factor: when ${\sigma }^{\prime }(z)\simeq 0$, the gradient vanishes, even if the error $(a-y)$ is large.

Problem setup

From the perspective of functional analysis, the problem can be posed as follows:
identify a loss function whose derivatives with respect to $w$ and $b$ do not depend on the derivative of the activation function.

For simplicity, let’s focus on the partial derivative $\partial {\mathcal{L}}_x/\partial b$.

By applying the chain rule

$$\frac{\partial {\mathcal{L}}_x}{\partial b}=\frac{\partial {\mathcal{L}}_x}{\partial a}\frac{\partial a}{\partial b}{|}_{a=\sigma (wx+b)}=\frac{\partial {\mathcal{L}}_x}{\partial a}\frac{\partial \sigma }{\partial z}\frac{\partial z}{\partial b}=\frac{\partial {\mathcal{L}}_x}{\partial a}{\sigma }^{\prime }(z)$$

knowing that

$${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))=a(1-a)$$

and imposing the functional analysis constraint, namely, to find a loss function ${\mathcal{L}}_x$ such that

$$\frac{\partial {\mathcal{L}}_x}{\partial b}=(a-y)$$

we get:

$$\frac{\partial {\mathcal{L}}_x}{\partial a}=\frac{\partial {\mathcal{L}}_x}{\partial b}\cdot \frac{1}{{\sigma }^{\prime }(z)}=\frac{a-y}{a(1-a)}$$

Finally, integrating w.r.t. $a$ gives the new loss function:

$$\boxed{{\mathcal{L}}_x=-[y\cdot \log (a)+(1-y)\cdot \log (1-a)]}$$

Proof of the Integration

The calculation of ${\mathcal{L}}_x$ from its partial derivative with respect to $a$ requires solving the following integral:

$${\mathcal{L}}_x=\int \frac{\partial {\mathcal{L}}_x}{\partial a}da=\int \frac{a-y}{a(1-a)}da$$

The integral is solved using the method of partial fraction decomposition. The expression is set up as follows:

$$\frac{a-y}{a(1-a)}=\frac{A}{a}+\frac{B}{1-a}$$

The coefficients $A$ and $B$ are found by multiplying both sides by the common denominator $a(1-a)$:

$$a-y=A(1-a)+B(a)$$

The values of $A$ and $B$ are then determined by substituting strategic values for $a$:

1. For $a=0$: $$0-y=A(1-0)+B(0)⟹-y=A$$
2. For $a=1$: $$1-y=A(1-1)+B(1)⟹1-y=B$$

Substituting the values of $A$ and $B$ back into the decomposed expression gives:

$$\frac{a-y}{a(1-a)}=\frac{-y}{a}+\frac{1-y}{1-a}$$

The integration can now be performed term by term:

$${\mathcal{L}}_x=\int \left(\frac{-y}{a}+\frac{1-y}{1-a}\right)da=-y\int \frac{1}{a}da+(1-y)\int \frac{1}{1-a}da$$

Solving the two simple integrals yields:

• $\int \frac{1}{a}da=\log (a)$
• $\int \frac{1}{1-a}da=-\log (1-a)$ (found via u-substitution where $u=1-a$)

Substituting these results back into the equation gives the final expression:

$${\mathcal{L}}_x=-y\log (a)+(1-y)[-\log (1-a)]=-[y\log (a)+(1-y)\log (1-a)]$$

Success

This new loss function has the advantage of eliminating the ${\sigma }^{\prime }(z)$ term from the derivatives, thus making the gradient more stable and informative, even in the presence of saturated activations.

---

MSE vs Cross Entropy

🔴 MSE, unlucky configuration $w,b$	🟢New Loss, same configuration $w,b$
Input: $1.0$ → Desired output: $0.0$	Input: $1.0$ → Desired output: $0.0$
Initial $w$: $2.0$	Initial $w$: $2.0$
Initial $b$: $2.0$	Initial $b$: $2.0$
Initial output: $0.82$	Initial output: $0.98$
Final output: $0.20$	Final output: $0.05$
Learning rate: $\eta =0.15$	Learning rate: $\eta =0.005$
Loss function: quadratic (MSE)	Loss function: Cross-Entropy
⏳ Slow learning: plateau lasting $\sim 150$ epochs	✅ Fast learning
⚠️ Near-zero initial gradient	✅ Substantial initial gradient
💤 Initially flat curve, followed by a delayed descent.	📉 The loss function decreases immediately
📌 $w$ and $b$ remain nearly constant for a considerable number of iterations.	📌 $w$ and $b$ are updated right from the start

✅ The neuron learned quickly!

By using the cross-entropy loss, the neuron was able to learn quickly, even when starting from a disadvantageous parameters configuration, exactly as hoped.
A closer look at the above plots shows that the initial slope of the loss curve is much steeper compared to the initial flat plateau observed with the MSE loss.

It is precisely this steeper slope that cross-entropy provides: it allows the neuron to avoid early stagnation, right at the stage when it should be learning the most, namely, when starting from a configuration very far from the correct target.

---

About the Learning Rate…

In the case of the MSE loss shown on the left side of the table above, the learning rate used was: $\eta =0.15$

Question

Should the same learning rate be used when switching to the new loss function (cross-entropy) ?

Strictly speaking, it is not meaningful to talk about using the same learning rate when the loss function changes: it would be like comparing apples to oranges. In both cases, the learning rate $\eta$ was empirically determined (i.e., by experimenting with different values) so as to make the learning dynamics clearly observable. For the cross-entropy loss, the chosen value was $\eta =0.005$.

The key point is not absolute speed

One might object that, since the learning rate was changed, the previous plots lose their significance.
After all, why should the speed at which the neuron learns matter if the learning rate was chosen arbitrarily?

❗But such an objection misses the central point.

The key issue is not the absolute speed of learning, but rather how the learning speed changes.

• With the quadratic loss function, learning is slower when the neuron is making large errors, and only accelerates as it approaches the desired output.
• With cross-entropy, by contrast, learning is faster precisely when the neuron is far from the correct response.

These observations hold regardless of the specific learning rate chosen.

---

Cross Entropy loss: formal derivation

Prerequisite: cross-entropy in information theory

Cross-entropy is a general quantity from information theory, not something specific to neural networks. Its definition, the decomposition $H(p,q)=H(p)+D_{KL}(p\parallel q)$, and the proof that minimising it is exactly maximum likelihood are developed in the Cross Entropy note of the information-theory section. This paragraph adapts that quantity to a neural network.

A more rigorous way to arrive at the same loss function derived above, rather than reasoning heuristically on a single neuron, is to appeal to Claude Shannon’s Information Theory, and in particular to the concept of cross-entropy $H$.

The cross-entropy $H$ between two probability distributions $p$ and $q$ over the same outcomes is

$$H(p,q)=-\sum _{x}p(x)\log q(x).$$

It measures, in bits, the cost of describing data that truly follow $p$ while assuming the distribution $q$: it is minimised when $q=p$, and grows as the two drift apart. Reading the target distribution as $p:=\mathbf{y}$ (the true labels) and the network’s output as $q:=\mathbf{a}$ (its estimate), $H(\mathbf{y},\mathbf{a})$ becomes an information distance between what the data say and what the network predicts. Adopted as a training objective, it is the cross-entropy loss:

$$\boxed{{\mathcal{L}}_{CE}=-\sum _j{y}_j\log (a_j)}$$

To intuitively motivate the superiority of the Cross-Entropy loss over Mean Squared Error (MSE), it’s useful to consider their fundamental differences. MSE is essentially a geometric ($L^2$) distance, which assumes to operate in a space where measuring an $L^2$ distance is meaningful. Its formula is:

$${\mathcal{L}}_{MSE}=\frac{1}{n}\sum _{j=1}^n(y_j-a_j{)}^2⟺\frac{1}{n}\Vert \mathbf{y}-\mathbf{a}{\Vert }_2^2$$

In contrast, ${\mathcal{L}}_{CE}$ measures the “distance” in terms of the bits of information between the desired outputs ($y_j$) and the actual outputs ($a_j$). This perspective is more consistent with the domain of neural networks, which are trained on data, that is, on information itself.

Why cross-entropy became the default loss

Three properties explain its dominance:

• it is robust to neuron saturation and the learning slowdown of MSE, because its gradient drops the ${\sigma }^{\prime }(z)$ factor (the derivation above);
• it has been the most widely used loss for training neural networks since around 2000;
• many other losses are specialisations of it: binary cross-entropy, the multilabel logistic loss and the negative log-likelihood loss are all ${\mathcal{L}}_{CE}$ instantiated to a particular label structure.

For a modern theoretical treatment, see Mao et al., Cross-entropy loss functions: theoretical analysis and applications (ICML 2023).

The Loss Function That Shaped Deep Learning

Empirically, it is observed that using ${\mathcal{L}}_{CE}$ helps neural networks converge much more effectively. This loss function was one of the key improvements that made Deep Learning practical, enabling networks to be trained faster by using a measure based on Shannon’s Information Theory in place of the $L^2$ distance.

${\mathcal{L}}_{CE}$ is a measure that approaches zero as the actual outputs become more similar to the desired outputs.

Cross-entropy in PyTorch: what actually runs

The theoretic formula ${\mathcal{L}}_{CE}=-{\sum }_j{y}_j\log a_j$ is almost never evaluated as written. PyTorch’s nn.CrossEntropyLoss fuses the softmax, the logarithm, and the negative log-likelihood into a single operation that consumes raw logits $\mathbf{z}$, never the post-softmax probabilities $\mathbf{a}$. The fusion is not a convenience: it is what keeps the computation finite.

Why the theoretic formula is never computed directly

Computing $a_j=e^{z_j}/{\sum }_k{e}^{z_k}$ and then $\log a_j$ chains two operations that each break at the extremes: $e^{z_j}$ overflows to $\infty$ for a logit as small as $z_j\approx 89$ in float32, and $\log a_j\to -\infty$ when a probability saturates to $0$. The fused loss sidesteps both by staying in logits,

$$\mathcal{L}=-z_{c^{\star }}+\log \sum _k{e}^{z_k}=-z_{c^{\star }}+z_{\max }+\log \sum _k{e}^{z_k-z_{\max }},$$

the log-sum-exp trick: subtracting the largest logit $z_{\max }$ before exponentiating forces every $e^{z_k-z_{\max }}\in (0,1]$, so the sum is exact and nothing overflows. The softmax probabilities are produced implicitly and never stored. This is the reason a classifier’s output layer must emit raw logits, with no softmax of its own.

Two silent bugs the fused design invites

Both produce wrong training with no error raised:

• Double softmax. Putting softmax or log_softmax inside the model and passing the result to CrossEntropyLoss normalises twice; the gradients flatten and learning quietly stalls. The model must output unnormalised logits.
• Probabilities instead of logits. Feeding already-normalised probabilities makes the loss take the log-softmax of a distribution, a different and incorrect objective.

The stable form keeps intact the very property that made cross-entropy worth adopting: its gradient with respect to the logits is the clean residual

$$\frac{\partial \mathcal{L}}{\partial \mathbf{z}}=\mathbf{a}-\mathbf{y},$$

so a confidently wrong logit (large $z$ on the wrong class) still produces a gradient of order one, exactly where a naive $\log (\mathrm{softmax})$ would have produced a NaN.

Three options of CrossEntropyLoss solve problems that recur constantly and that introductions rarely mention.

label_smoothing: a regulariser hidden inside the loss

Setting label_smoothing=ε replaces the hard one-hot target with a mixture: $1-\epsilon$ on the correct class and $\epsilon /C$ spread over the others. Cross-entropy against a strict one-hot target drives the correct logit toward $+\infty$, which breeds overconfident, badly calibrated predictions; the smoothed target caps that drive and leaves a finite optimal logit gap. The standard value $\epsilon =0.1$ is used in most modern vision and language models, and it routinely improves calibration, and often accuracy, for the cost of a single argument.

ignore_index and weight

• ignore_index drops chosen target positions from the loss entirely, contributing no gradient. This is what makes cross-entropy usable on padded sequences: the padding token is ignored, so short and long sequences in a batch are scored fairly. It is the standard partner of BPTT on variable-length data.
• weight rescales each class’s contribution, the simplest remedy for class imbalance: up-weighting a rare class makes its errors count more, countering a model that would otherwise collapse onto the majority class.

Finally, the target of CrossEntropyLoss may be either class indices (hard labels) or a full probability vector per row (soft labels). The same loss therefore covers ordinary classification, knowledge distillation (the targets are a teacher’s softened outputs), and mixup-style augmentation (the targets are convex combinations of one-hot vectors), with no change of interface.

When should cross-entropy be used instead of MSE?

Cross-entropy is almost always the best choice, provided that the output neurons are sigmoid neurons.

To understand why, keep in mind that during a network’s initialization phase, the weights and biases are typically assigned randomly. It’s possible for this initial choice to cause the model to be confidently wrong for some training inputs: for example, an output neuron might saturate near 1 when it should actually return $0$ (or vice-versa).

If the quadratic cost function (MSE) is being used in this scenario, learning will slow down significantly. Learning won’t stop entirely (because the network can still learn from other inputs), but the slowdown is nonetheless undesirable.