bp-mlp-2

Assumptions about the loss function

The goal of backpropagation is to compute the partial derivatives $\partial \mathcal{L}/\partial w$ and $\partial \mathcal{L}/\partial b$ of the cost function $\mathcal{L}$ with respect to any weight $w$ or bias $b$ in the network. For backpropagation to work, two main assumptions about the form of the cost function are required. Before stating those assumptions, it is useful to establish an example cost function as a reference.

Reference Model: Mean Square Error (MSE) Loss

The quadratic loss (or cost) function is defined as:

$$\mathcal{L}=\frac{1}{2n}\sum _x{\Vert y(x)-a^L(x)\Vert }^2$$

Where:

• $n$ is the total number of training examples;
• the sum is over individual training examples $x$;
• $y=y(x)$ is the corresponding desired output;
• $L$ denotes the number of layers in the network;
• $a^L=a^L(x)$ is the vector of activations output from the network when $x$ is input.

---

1st Assumption

The first assumption to make about the cost function $\mathcal{L}$, in order that backpropagation can be applied, is that the cost function can be written as an average:

$$\mathcal{L}=\frac{1}{n}\sum _x{\mathcal{L}}_x$$

over cost functions ${\mathcal{L}}_x$ for individual training examples, $x$. This is the case for the quadratic cost function, where the cost for a single training example is ${\mathcal{L}}_x=\frac{1}{2}\Vert y-a^L{\Vert }^2$. This assumption also holds true for all other cost functions encountered in this context.

Why this is needed

This assumption is necessary because backpropagation specifically enables the computation of the partial derivatives $\partial {\mathcal{L}}_x/\partial w$ and $\partial {\mathcal{L}}_x/\partial b$ for a single training example.

The global derivatives $\partial \mathcal{L}/\partial w$ and $\partial \mathcal{L}/\partial b$ are then recovered by averaging over all training examples. For notational simplicity, the training example $x$ is assumed to be fixed, and the $x$ subscript is dropped, writing the cost ${\mathcal{L}}_x$ as $\mathcal{L}$. This notation is left implicit for the duration of the derivation.

---

2nd Assumption

The second assumption made about the cost is that it can be written as a function of the outputs from the neural network:

For example, the quadratic cost function satisfies this requirement, since the quadratic cost for a single training example $x$ may be written as:

$$\mathcal{L}=\frac{1}{2}{\Vert y-a^L\Vert }^2=\frac{1}{2}\sum _j{\left(y_j-a_j^L\right)}^2$$

and thus is clearly a function of the output activations.

Why is the cost not regarded as a function of $y$?

It may be questioned why the cost is not also regarded as a function of the desired output $y$. However, since the input training example $x$ is fixed, the output $y$ is also a fixed parameter.

Specifically, $y$ cannot be modified by changing the weights and biases; it is not something which the neural network learns. Therefore, it is logical to regard $\mathcal{L}$ as a function of the output activations $a^L$ alone, with $y$ acting as a parameter that helps define that function.