BP for MLPs: notation

Scalar notations

---

Vector notations

---

The Hadamard product

The backpropagation algorithm is based on standard linear algebraic operations, such as vector addition and matrix-vector multiplication, and so on. However, one specific operation is utilized less frequently.

Definition

For two vectors $s$ and $t$ of the same dimension, the notation $s\odot t$ is used to denote the elementwise product of the two vectors. Consequently, the components of $s\odot t$ are defined as $(s\odot t{)}_j=s_j{t}_j$.

This type of elementwise multiplication is sometimes referred to as the Hadamard product or Schur product. Within this technical context, it is referred to as the Hadamard product.

Example

$$\left[\begin{array}{c}1\\ 2\end{array}\right]\odot \left[\begin{array}{c}3\\ 4\end{array}\right]=\left[\begin{array}{c}1\times 3\\ 2\times 4\end{array}\right]=\left[\begin{array}{c}3\\ 8\end{array}\right]$$

Info

Good matrix libraries typically provide optimized implementations of the Hadamard product, which is essential for the efficient implementation of the backpropagation algorithm.