Cross Entropy Loss

Building of a new loss

Consider again the example of the single neuron. However, the task is no longer to make its unit input zeros.

🎯 Goal

The goal is to drop the derivative terms ${\sigma }^{\prime }(z)$ from the loss function derivatives (which drive gradient descent), in order to avoid the vanishing-gradient problem in the saturation region of the sigmoid.

$$\begin{array}{rlrl}\frac{\partial {\mathcal{L}}_x}{\partial w}& =(a-y){\sigma }^{\prime }(z)x& ⟶& (a-y)x\\ & & & \\ \frac{\partial {\mathcal{L}}_x}{\partial b}& =(a-y){\sigma }^{\prime }(z)& ⟶& (a-y)\end{array}$$

${\sigma }^{\prime }(z)$ is a critical multiplicative factor: when ${\sigma }^{\prime }(z)\simeq 0$, the gradient vanishes, even if the error $(a-y)$ is large.

Problem setup

From the perspective of functional analysis, the problem can be posed as follows:
identify a loss function whose derivatives with respect to $w$ and $b$ do not depend on the derivative of the activation function.

For simplicity, let’s focus on the partial derivative $\partial {\mathcal{L}}_x/\partial b$.

By applying the chain rule

$$\frac{\partial {\mathcal{L}}_x}{\partial b}=\frac{\partial {\mathcal{L}}_x}{\partial a}\frac{\partial a}{\partial b}{|}_{a=\sigma (wx+b)}=\frac{\partial {\mathcal{L}}_x}{\partial a}\frac{\partial \sigma }{\partial z}\frac{\partial z}{\partial b}=\frac{\partial {\mathcal{L}}_x}{\partial a}{\sigma }^{\prime }(z)$$

knowing that

$${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))=a(1-a)$$

and imposing the functional analysis constraint, namely, to find a loss function ${\mathcal{L}}_x$ such that

$$\frac{\partial L_x}{\partial b}=(a-y)$$

we get:

$$\frac{\partial {\mathcal{L}}_x}{\partial a}=\frac{\partial {\mathcal{L}}_x}{\partial b}\cdot \frac{1}{{\sigma }^{\prime }(z)}=\frac{a-y}{a(1-a)}$$

Finally, integrating w.r.t. $a$ gives the new loss function:

$$\boxed{{\mathcal{L}}_x=-[y\cdot \log (a)+(1-y)\cdot \log (1-a)]}$$

Proof of the Integration

The calculation of ${\mathcal{L}}_x$ from its partial derivative with respect to $a$ requires solving the following integral:

$${\mathcal{L}}_x=\int \frac{\partial {\mathcal{L}}_x}{\partial a}da=\int \frac{a-y}{a(1-a)}da$$

The integral is solved using the method of partial fraction decomposition. The expression is set up as follows:

$$\frac{a-y}{a(1-a)}=\frac{A}{a}+\frac{B}{1-a}$$

The coefficients $A$ and $B$ are found by multiplying both sides by the common denominator $a(1-a)$:

$$a-y=A(1-a)+B(a)$$

The values of $A$ and $B$ are then determined by substituting strategic values for $a$:

1. For $a=0$: $$0-y=A(1-0)+B(0)⟹-y=A$$
2. For $a=1$: $$1-y=A(1-1)+B(1)⟹1-y=B$$

Substituting the values of $A$ and $B$ back into the decomposed expression gives:

$$\frac{a-y}{a(1-a)}=\frac{-y}{a}+\frac{1-y}{1-a}$$

The integration can now be performed term by term:

$${\mathcal{L}}_x=\int \left(\frac{-y}{a}+\frac{1-y}{1-a}\right)da=-y\int \frac{1}{a}da+(1-y)\int \frac{1}{1-a}da$$

Solving the two simple integrals yields:

• $\int \frac{1}{a}da=\log (a)$
• $\int \frac{1}{1-a}da=-\log (1-a)$ (found via u-substitution where $u=1-a$)

Substituting these results back into the equation gives the final expression:

$${\mathcal{L}}_x=-y\log (a)+(1-y)[-\log (1-a)]=-[y\log (a)+(1-y)\log (1-a)]$$

This new loss function has the advantage of eliminating the ${\sigma }^{\prime }(z)$ term from the derivatives, thus making the gradient more stable and informative, even in the presence of saturated activations.

---

MSE vs Cross Entropy

🔴 MSE, unlucky configuration $w,b$	🟢New Loss, same configuration $w,b$
Input: $1.0$ → Desired output: $0.0$	Input: $1.0$ → Desired output: $0.0$
Initial $w$: $2.0$	Initial $w$: $2.0$
Initial $b$: $2.0$	Initial $b$: $2.0$
Initial output: $0.82$	Initial output: $0.98$
Final output: $0.20$	Final output: $0.05$
Learning rate: $\eta =0.15$	Learning rate: $\eta =0.005$
Loss function: quadratic (MSE)	Loss function: Cross-Entropy
⏳ Slow learning: plateau lasting $\sim 150$ epochs	✅ Fast learning
⚠️ Near-zero initial gradient	✅ Substantial initial gradient
💤 Initially flat curve, followed by a delayed descent.	📉 The loss function decreases immediately
📌 $w$ and $b$ remain nearly constant for a considerable number of iterations.	📌 $w$ and $b$ are updated right from the start

✅ The neuron learned quickly!

By using the cross-entropy loss, the neuron was able to learn quickly, even when starting from a disadvantageous parameters configuration, exactly as hoped.
A closer look at the above plots shows that the initial slope of the loss curve is much steeper compared to the initial flat plateau observed with the MSE loss.

It is precisely this steeper slope that cross-entropy provides: it allows the neuron to avoid early stagnation, right at the stage when it should be learning the most, namely, when starting from a configuration very far from the correct target.

---

About the Learning Rate…

In the case of the MSE loss shown on the left side of the table above, the learning rate used was: $\eta =0.15$

Should the same learning rate be used when switching to the new loss function (cross-entropy) ?

Strictly speaking, it is not meaningful to talk about using the same learning rate when the loss function changes: it would be like comparing apples to oranges. In both cases, the learning rate $\eta$ was empirically determined (i.e., by experimenting with different values) so as to make the learning dynamics clearly observable. For the cross-entropy loss, the chosen value was $\eta =0.005$.

🎯 The key point is not absolute speed

One might object that, since the learning rate was changed, the previous plots lose their significance.
After all, why should the speed at which the neuron learns matter if the learning rate was chosen arbitrarily?

❗But such an objection misses the central point.

The key issue is not the absolute speed of learning, but rather how the learning speed changes.

• With the quadratic loss function, learning is slower when the neuron is making large errors, and only accelerates as it approaches the desired output.
• With cross-entropy, by contrast, learning is faster precisely when the neuron is far from the correct response.

These observations hold regardless of the specific learning rate chosen.

---

Cross Entropy loss: formal derivation

A more rigorous way to arrive at the same loss function derived above, rather than reasoning heuristically on a single neuron, is to appeal to Claude Shannon’s Information Theory, and in particular to the concept of cross-entropy $H$.

By adapting the definition of cross-entropy from Information Theory to the neural network domain and interpreting it as a measure of error, the Cross-Entropy Loss is obtained:

$$\boxed{{\mathcal{L}}_{CE}=-\sum _j{y}_j\log (a_j)}$$

To intuitively motivate the superiority of the Cross-Entropy loss over Mean Squared Error (MSE), it’s useful to consider their fundamental differences. MSE is essentially a geometric ($L^2$) distance, which assumes to operate in a space where measuring an $L^2$ distance is meaningful. Its formula is:

$${\mathcal{L}}_{MSE}=\frac{1}{n}\sum _{j=1}^n(y_j-a_j{)}^2⟺\frac{1}{n}\Vert \mathbf{y}-\mathbf{a}{\Vert }_2^2$$

In contrast, ${\mathcal{L}}_{CE}$ measures the “distance” in terms of the bits of information between the desired outputs ($y_j$) and the actual outputs ($a_j$). This perspective is more consistent with the domain of neural networks, which are trained on data—that is, on information itself.

The Loss Function That Shaped Deep Learning

Empirically, it is observed that using ${\mathcal{L}}_{CE}$ helps neural networks converge much more effectively. This loss function was one of the key improvements that made Deep Learning practical, enabling networks to be trained faster by using a measure based on Shannon’s Information Theory in place of the $L^2$ distance.

${\mathcal{L}}_{CE}$ is a measure that approaches zero as the actual outputs become more similar to the desired outputs.

Cross-Entropy in Modern DL Frameworks

In Deep Learning frameworks like PyTorch, the general theoretical formula for Cross-Entropy (${\mathcal{L}}_{CE}$) is not typically implemented directly. Instead, specialized versions are provided. This is because the ${\mathcal{L}}_{CE}$ formula itself is a direct adaptation of Shannon’s cross-entropy, from Information Theory, to the context of neural networks. The implementations found within these frameworks are the practical, numerically stable specializations of that core concept.

When should cross-entropy be used instead of MSE?

Cross-entropy is almost always the best choice, provided that the output neurons are sigmoid neurons.

To understand why, keep in mind that during a network’s initialization phase, the weights and biases are typically assigned randomly. It’s possible for this initial choice to cause the model to be confidently wrong for some training inputs: for example, an output neuron might saturate near 1 when it should actually return $0$ (or vice-versa).

If the quadratic cost function (MSE) is being used in this scenario, learning will slow down significantly. Learning won’t stop entirely (because the network can still learn from other inputs), but the slowdown is nonetheless undesirable.