MLPs limitations

Note

Multi-Layer Perceptrons (MLPs) are universal function approximators, yet their core architectural assumptions make them fundamentally intractable for high-dimensional, topologically structured data like images, audio, or time series.

The paradigm shift toward Convolutional Neural Networks (CNNs) is frequently misrepresented as an attempt to reverse-engineer the visual cortex. In reality, it is a strictly algorithmic evolution, driven by the imperative to resolve three precise mathematical and computational bottlenecks inherent to dense architectures.

---

🔴 The Loss of Spatial Prior

The Flattening Problem

Traditional neural networks, such as MLPs, require their inputs to be flat 1D vectors.

Whenever the input data possesses a topological structure (e.g., a 2D grid of pixels), it must be flattened before it can be multiplied by the weight matrix:

$$\text{structured input}⟶\mathbf{x}\in {\mathbb{R}}^N$$

Strictly speaking, flattening does not destroy information in an information-theoretic or set-theoretic sense, as the mapping is perfectly invertible.

The issue is purely architectural: by vectorizing the input, the network discards the spatial prior. The model no longer encodes the inductive bias necessary to exploit the inherent topological structure of the data.

Example: images and spatial priors

In a grayscale image $\mathbf{I}\in {\mathbb{R}}^{H\times W}$

• neighboring pixels are statistically highly correlated. For example, the value of a pixel at coordinates $(i,j)$ is strongly related to its immediate vertical neighbor at $(i+1,j)$.
• when the image is flattened into a $1D$ vector $\mathbf{x}$ using row-major order, the topological adjacency is destroyed. If $\mathbf{I}[i,j]$ is mapped to coordinate $x_k$, its vertical neighbor $\mathbf{I}[i+1,j]$ is mapped to $x_{k+W}$.

In the first fully connected hidden layer of an MLP,

$${\mathbf{a}}^2=\sigma (\mathbf{Wx}+\mathbf{b})=\sigma ({\mathbf{z}}^2),$$

the weight matrix $\mathbf{W}$ treats all dimensions of $\mathbf{x}$ symmetrically. Therefore there is no architectural constraint (no inductive bias) forcing the model to recognize that $x_k$ and $x_{k+W}$ represent physically adjacent locations in the original image.

Consequently, spatial locality is not encoded. The network must waste representational capacity to blindly rediscover this topological proximity by inferring it from the training data, rather than having the spatial prior built directly into its architecture.

✅ Solution

Local Receptive Field

Introducing local receptive fields restores the structural prior of the data. Instead of connecting a neuron to the entire flattened input, CNNs restrict each unit’s connections to a small, contiguous spatial neighborhood (e.g., a $3\times 3$ patch). By doing so, spatial locality is encoded directly into the architecture, eliminating the need for the network to rediscover topological proximity from scratch.

---

🔴 MLPs do not scale well

Parameter growth in dense layers

For an MLP layer with input dimension $N_{\text{in}}$ and output dimension $N_{\text{out}}$, the weight matrix contains $N_{\text{in}}\times N_{\text{out}}$ parameters, plus $N_{\text{out}}$ biases.

Therefore, the parameter footprint scales multiplicatively as $\mathcal{O}(N_{\text{in}}{N}_{\text{out}})$.

High-resolution images

Consider a modest $1000\times 1000$ image.
After flattening, the input vector size is

$$N_{\text{in}}=10^6.$$

To prevent a catastrophic information bottleneck, the first hidden layer must maintain a representational capacity comparable to the input, requiring roughly $N_{\text{out}}=10^6$ units. The corresponding dense weight matrix contains:

$$10^6\times 10^6=10^{12}$$

weights.

The VRAM Wall

In standard 32-bit floating-point precision (FP32), storing this single weight matrix requires

$$4\times 10^{12}\text{ bytes}\approx 4\text{ TB of memory}$$

Furthermore, this accounts only for the static weights. During training, tracking gradients and modern optimizer states (like Adam’s momentum and variance buffers) inflates this requirement to over 12 Terabytes for a single layer!

Important

Dense connectivity is therefore computationally and memory-wise prohibitive for high-dimensional structured inputs. The limitation is not merely inefficient; it hits a hard, insurmountable physical hardware wall.

✅ Solution

Parameter Sharing

To break the quadratic scaling bottleneck, CNNs abandon the use of independent weights for every spatial location. Instead, they employ parameter sharing.

A single, highly compact set of weights (a kernel, e.g., a $3\times 3$ matrix) is systematically swept across the entire input volume. Rather than learning $10^{12}$ independent parameters for a high-resolution image, the network learns just 9 parameters that are reused at every spatial coordinate.

This mechanism does more than just solve the VRAM crisis. It injects a profoundly powerful inductive bias: a visual feature (such as an edge or a texture) learned in one region of the image is automatically detectable anywhere else. This property, known as translation equivariance, is the theoretical cornerstone of modern computer vision.

---

🔴 The Failure of Hierarchical Composition

The reason MLPs have been proposed and studied for decades is the belief that adding more layers could lead to increasingly sophisticated levels of abstraction.

At the visual level, the human brain operates precisely in this manner: effortlessly building sophisticated abstractions from raw sensory data. In a task such as handwritten digit classification, an ideal network is expected to mirror this process by constructing abstractions progressively:

• Layer 1: Detection of local primitive strokes (e.g., vertical or horizontal segments).
• Layer 2: Composition of primitives into intermediate geometric structures (e.g., loops).
• Layer 3+: Aggregation into complete conceptual objects (e.g., the recognition of a ‘9’ as the union of a top loop and a vertical bottom stroke).

The Black Box Reality

While this human-like abstraction is the theoretical goal, traditional MLPs systematically fail to realize it. In practice, MLPs behave as unstructured black boxes. Inspection of their learned weights reveals no evidence of compositional reasoning similar to that of humans; the internal representations are neither interpretable nor easily understandable.

The Absolute Coordinate Bottleneck

This lack of interpretability is not magic, but a direct mathematical consequence of dense connectivity. Because parameters are strictly tied to absolute spatial coordinates, a generic, location-independent feature detector cannot be learned. For instance, a vertical segment located on the left side of a '$4$' is processed by one specific subset of input weights, whereas the identical segment located on the right side of a '$9$' activates an entirely different subset.

The Collapse of Abstraction

In the absence of translation equivariance (weight tying), the concept of a primitive stroke must be redundantly relearned for every possible pixel coordinate. Consequently, representational capacity is exhausted in duplicating low-level detectors, preventing the formation of a unified vocabulary of primitive shapes. Without this foundational vocabulary, intermediate structures (such as loops) are never cleanly extracted, and the intended deep cognitive hierarchy collapses into an inefficient, brute-force template matching system.

✅ Solution

Feature Maps & Downsampling

Convolutional layers resolve this by producing Feature Maps. A single shared kernel detects a “vertical stroke” universally across the entire input domain, ensuring translation equivariance. By interleaving these equivariant feature maps with Downsampling (pooling) operations, the network aggregates local features over progressively larger spatial regions. This explicit architectural mechanism forces the network to naturally build the desired compositional hierarchy: universal local edges are spatially aggregated into shapes, which are ultimately composed into global objects.