MLPs do not scale well

🔴 Problem: too many parameters

Tip

A common trick in Machine Learning is to project the input feature space into a higher-dimensional feature space.
When projecting data into a higher-dimensional space is not feasible, the alternative is to project them into a lower-dimensional one (LDA, PCA, …).

Images

Consider the case where the input to an MLP is a $1000\times 1000$ pixel image.

In the context of neural networks, it is common to project into higher-dimensional spaces.
This implies that if the flattened image results in one million neurons in the input layer, then the neurons in the hidden layer are at least one million as well.
If the hidden layer has exactly one million neurons, then between the input layer and the hidden layer there are $10^6\times 10^6=10^{12}$ connections (since each neuron is fully connected to the neurons in the next layer), which is a huge number.

Note

In general, the weight matrix associated with layer $l$ of an MLP contains $n^l\times n^{l+1}$ weights.

Danger

Thus, MLPs do not scale well.

✅ Solution

: Parameters Sharing