Cross Entropy Loss

Building of a new loss

Consider again the example of the single neuron. However, the task is no longer to make its unit input zeros.

🎯 Goal

The goal is to drop the derivative terms ${\sigma }^{\prime }(z)$ from the loss function derivatives (which drive gradient descent), in order to avoid the vanishing-gradient problem in the saturation region of the sigmoid.

$$\begin{array}{rlrl}\frac{\partial {\mathcal{L}}_x}{\partial w}& =(a-y){\sigma }^{\prime }(z)x& ⟶& (a-y)x\\ & & & \\ \frac{\partial {\mathcal{L}}_x}{\partial b}& =(a-y){\sigma }^{\prime }(z)& ⟶& (a-y)\end{array}$$

${\sigma }^{\prime }(z)$ is a critical multiplicative factor: when ${\sigma }^{\prime }(z)\simeq 0$, the gradient vanishes, even if the error $(a-y)$ is large.

Problem setup

From the perspective of functional analysis, the problem can be posed as follows:
identify a loss function whose derivatives with respect to $w$ and $b$ do not depend on the derivative of the activation function.

For simplicity, let’s focus on the partial derivative $\partial {\mathcal{L}}_x/\partial b$.

By applying the chain rule

$$\frac{\partial {\mathcal{L}}_x}{\partial b}=\frac{\partial {\mathcal{L}}_x}{\partial a}\frac{\partial a}{\partial b}{|}_{a=\sigma (wx+b)}=\frac{\partial {\mathcal{L}}_x}{\partial a}\frac{\partial \sigma }{\partial z}\frac{\partial z}{\partial b}=\frac{\partial {\mathcal{L}}_x}{\partial a}{\sigma }^{\prime }(z)$$

, knowing that

$${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))=a(1-a)$$

and imposing the functional analysis constraint, namely, to find a loss function ${\mathcal{L}}_x$ such that

$$\frac{\partial L_x}{\partial b}=(a-y)$$

we get:

$$\frac{\partial {\mathcal{L}}_x}{\partial a}=\frac{\partial {\mathcal{L}}_x}{\partial b}\cdot \frac{1}{{\sigma }^{\prime }(z)}=\frac{a-y}{a(1-a)}$$

Finally, integrating w.r.t. $a$ gives the new loss function:

$$\boxed{{\mathcal{L}}_x=-[y\cdot \log (a)+(1-y)\cdot \log (1-a)]}$$

Proof of the Integration

The calculation of ${\mathcal{L}}_x$ from its partial derivative with respect to $a$ requires solving the following integral:

$${\mathcal{L}}_x=\int \frac{\partial {\mathcal{L}}_x}{\partial a}da=\int \frac{a-y}{a(1-a)}da$$

The integral is solved using the method of partial fraction decomposition. The expression is set up as follows:

$$\frac{a-y}{a(1-a)}=\frac{A}{a}+\frac{B}{1-a}$$

The coefficients $A$ and $B$ are found by multiplying both sides by the common denominator $a(1-a)$:

$$a-y=A(1-a)+B(a)$$

The values of $A$ and $B$ are then determined by substituting strategic values for $a$:

1. For $a=0$: $$0-y=A(1-0)+B(0)⟹-y=A$$
2. For $a=1$: $$1-y=A(1-1)+B(1)⟹1-y=B$$

Substituting the values of $A$ and $B$ back into the decomposed expression gives:

$$\frac{a-y}{a(1-a)}=\frac{-y}{a}+\frac{1-y}{1-a}$$

The integration can now be performed term by term:

$${\mathcal{L}}_x=\int \left(\frac{-y}{a}+\frac{1-y}{1-a}\right)da=-y\int \frac{1}{a}da+(1-y)\int \frac{1}{1-a}da$$

Solving the two simple integrals yields:

• $\int \frac{1}{a}da=\log (a)$
• $\int \frac{1}{1-a}da=-\log (1-a)$ (found via u-substitution where $u=1-a$)

Substituting these results back into the equation gives the final expression:

$${\mathcal{L}}_x=-y\log (a)+(1-y)[-\log (1-a)]=-[y\log (a)+(1-y)\log (1-a)]$$

This new loss function has the advantage of eliminating the ${\sigma }^{\prime }(z)$ term from the derivatives, thus making the gradient more stable and informative, even in the presence of saturated activations.

---

MSE vs Cross Entropy

🔴 MSE, unlucky configuration $w,b$	🟢New Loss, same configuration $w,b$
Input: $1.0$ → Desired output: $0.0$	Input: $1.0$ → Desired output: $0.0$
Initial $w$: $2.0$	Initial $w$: $2.0$
Initial $b$: $2.0$	Initial $b$: $2.0$
Initial output: $0.82$	Initial output: $0.98$
Final output: $0.20$	Final output: $0.05$
Learning rate: $\eta =0.15$	Learning rate: $\eta =0.005$
Loss function: quadratic (MSE)	Loss function: Cross-Entropy
⏳ Slow learning: plateau lasting $\sim 150$ epochs	✅ Fast learning
⚠️ Near-zero initial gradient	✅ Substantial initial gradient
💤 Initially flat curve, followed by a delayed descent.	📉 The loss function decreases immediately
📌 $w$ and $b$ remain nearly constant for a considerable number of iterations.	📌 $w$ and $b$ are updated right from the start

✅ The neuron learned quickly!

By using the cross-entropy loss, the neuron was able to learn quickly, even when starting from a disadvantageous parameters configuration, exactly as hoped.
A closer look at the above plots shows that the initial slope of the loss curve is much steeper compared to the initial flat plateau observed with the MSE loss.

It is precisely this steeper slope that cross-entropy provides: it allows the neuron to avoid early stagnation, right at the stage when it should be learning the most, namely, when starting from a configuration very far from the correct target.

---

About the Learning Rate…

In the case of the MSE loss shown on the left side of the table above, the learning rate used was: $\eta =0.15$

Should the same learning rate be used when switching to the new loss function (cross-entropy) ?

Strictly speaking, it is not meaningful to talk about using the same learning rate when the loss function changes: it would be like comparing apples to oranges. In both cases, the learning rate $\eta$ was empirically determined (i.e., by experimenting with different values) so as to make the learning dynamics clearly observable. For the cross-entropy loss, the chosen value was $\eta =0.005$.

🎯 The key point is not absolute speed

One might object that, since the learning rate was changed, the previous plots lose their significance.
After all, why should the speed at which the neuron learns matter if the learning rate was chosen arbitrarily?

❗But such an objection misses the central point.

The key issue is not the absolute speed of learning, but rather how the learning speed changes.

• With the quadratic loss function, learning is slower when the neuron is making large errors, and only accelerates as it approaches the desired output.
• With cross-entropy, by contrast, learning is faster precisely when the neuron is far from the correct response.

These observations hold regardless of the specific learning rate chosen.