RMSProp

Hinton’s intuition: forget the distant past

Limitation of AdaGrad

The weakness of AdaGrad lies in the fact that it accumulates the entire history of squared gradients.
As training progresses, the denominator in the update rule grows unbounded, causing the effective learning rate for each parameter ${\theta }_j$ to shrink continuously.

💡 Idea (G. Hinton)

If the problem comes from summing the entire history, then stop summing it all.
Instead, assign decreasing weights to older gradients, so that recent information matters more than distant past.

Solution: use EMA

The solution builds on a widely applicable concept that goes beyond Deep Learning itself: the Exponential Moving Average (EMA).

Exponential Moving Average (EMA)

Definition

EMA definition

The Exponential Moving Average (EMA) is a type of infinite impulse response filter that applies exponentially decaying weights to past observations.
Unlike a simple moving average, which assigns equal importance to all past values within a fixed window, the EMA emphasizes recent data points while gradually “forgetting” the distant past.
This makes it particularly useful for applications such as time-series normalization and, in Deep Learning, for controlling the accumulation of gradient information.

Given a sequence of data points $x^{(t)}$ and a decay factor $\alpha \in (0,1]$, the EMA is defined recursively as:

$$\boxed{EM{A}^{(t)}=\alpha EM{A}^{(t-1)}+(1-\alpha )x^{(t)}}$$

• The most recent value $x^{(t)}$ contributes with weight $(1-\alpha )$.
• Each past contribution is attenuated by a factor of $\alpha$ per time step → the further back in time, the smaller its influence on the current EMA.

Why EMA matters for optimization

When applied to the squared gradients, the EMA provides a “limited memory” mechanism:

• It preserves per-parameter adaptivity, just as in AdaGrad.
• It avoids the unbounded accumulation that causes AdaGrad’s learning rate to shrink towards zero.
• It forms the foundation of modern adaptive optimizers such as RMSProp, Adam, and AdaDelta.

Key intuition

The EMA acts as a soft forgetting mechanism.
Instead of treating all past gradients as equally important, it ensures that recent gradients dominate the dynamics of learning, while older ones gradually fade in influence.

RMSProp in depth

Update rule

The parameter update in RMSProp follows the rule:

$${\theta }_j^{(t+1)}={\theta }_j^{(t)}-\frac{\eta }{\sqrt{v_j^{(t)}}+ϵ}{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}$$

where:

$$v_j^{(t)}=\alpha v_j^{(t-1)}+(1-\alpha )[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}{]}^2,\text{is an EMA of squared gradients with }\alpha =0.99$$

General EMA formula	Specific case: RMSProp for ${\theta }_j$	Meaning in RMSProp
$EM{A}^{(t)}$	$v_j^{(t)}$	Current value of the exponential moving average of the squared $j$-th gradient component
$EM{A}^{(t-1)}$	$v_j^{(t-1)}$	Exponential moving average, at step $t-1$, of the squared $j$-th gradient component
$x^{(t)}$	${\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}\right]}^2$	Current input: squared value of the $j$-th gradient component at step $t$
$\alpha \in (0,1)$	$\alpha \in (0,1)$	Decay rate: controls how much past information is remembered
$(1-\alpha )x^{(t)}$	$(1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}\right]}^2$	Contribution of the current $j$-th gradient component

Note

The denominator $v_j^{(t)}$ averages the squared past gradient components for parameter $j$, but does so by assigning greater weight to the more recent ones.

Why is it called RMSProp

Term	Explanation
$\text{R}$oot	The square root of the EMA of squared gradients, $\sqrt{v_j^{(t)}}$, is computed. This brings the adaptive term back to the same scale as the gradient, preventing numerical imbalance in the update. Without the square root, the denominator would grow too quickly, making the effective learning rate excessively small. The root stabilizes and normalizes the effective learning rate used in the parameter update.
$\text{M}$ean	An average (moving average) is used instead of a single gradient observation, producing a more stable estimate.
$\text{S}$quare	The gradient component is squared in the EMA formula to measure only its magnitude (the sign is irrelevant). This provides an indicator of the gradient’s energy, useful for modulating the effective learning rate applied to parameter ${\theta }_j$.
$\text{PROP}$agation	The estimate $v_j^{(t)}$ is not reset at each step but updated recursively$v_j^{(t)}=\alpha v_j^{(t-1)}+(1-\alpha )[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}{]}^2$. This way, the effect of past $j$-th gradient components propagates across iterations, making the effective learning rate for ${\theta }_j$ continuously adaptive.

Unrolling of the EMA formula

To fully understand the behavior of RMSProp, it is useful to expand recursively the definition of the exponential moving average (EMA) of the squared gradients step by step.
This makes it clear how past $j$-th gradient components gradually lose influence over time, illustrating the mechanism of exponential memory decay.

Let’s unroll the recursion for the first steps:

Step 1

$$v_j^{(1)}=\alpha v_j^{(0)}+(1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}\right]}^2$$

Step 2

$$\begin{array}{rl}{v}_j^{(2)}& =\underset{\alpha v_j^{(1)}}{\underbrace{{\alpha }^2{v}_j^{(0)}+\alpha (1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}\right]}^2}}+(1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(2)}\right]}^2\end{array}$$

Step 3

$$\begin{array}{rl}{v}_j^{(3)}& =\underset{\alpha v_j^{(2)}}{\underbrace{{\alpha }^3{v}_j^{(0)}+{\alpha }^2(1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}\right]}^2+\alpha (1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(2)}\right]}^2}}+(1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(3)}\right]}^2\end{array}$$

By induction, this leads to the explicit formula:

$$\boxed{v_j^{(t)}={\alpha }^t{v}_j^{(0)}+\sum _{k=1}^t{\alpha }^{t-k}(1-\alpha ){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(k)}\right]}^2}$$

From this expression it can be observed that:

• Each squared gradient term ${\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(k)}\right]}^2$ is multiplied by a coefficient ${\alpha }^{t-k}(1-\alpha )$,
• The further back in time a gradient component is, the smaller its impact on $v_j^{(t)}$.

Effect of exponential decay

Since $\alpha \in (0,1]$, the coefficient ${\alpha }^{t-k}(1-\alpha )$ applies an exponentially decreasing weight to past $j$-th gradient components.

Here, $t$ denotes the current iteration, while $k$ indexes a past iteration ($1\le k\le t$).
Thus, the further back in time a squared gradient term ${\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(k)}\right]}^2$ is (i.e., when $k\ll t$), the less influence it has on $v_j^{(t)}$.
Recent gradient components dominate the value of the moving average.

For example, with $\alpha =0.99$:

• ${\alpha }^1=0.99$,
• ${\alpha }^{10}\approx 0.90$,
• ${\alpha }^{100}\approx 0.366$,
• ${\alpha }^{500}\approx 0.0066$.

This shows that gradient components older than just a few hundred iterations carry virtually no weight: the optimizer effectively “forgets” the distant past, favoring an up-to-date and responsive estimate of gradient variability.

Important

This exponential decay ensures that recent gradients exert the most influence on the current value of $v_j^{(t)}$, while past information is progressively “forgotten.”
This mechanism is the very essence of RMSProp’s adaptive behavior.

RMSProp weakness

EMA is biased towards $0$

With $\alpha =0.99$ and $v_j^{(0)}=0$, the exponential moving average

$$v_j^{(t)}=\alpha v_j^{(t-1)}+(1-\alpha )[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}{]}^2$$

initially incorporates only 1% of the current squared gradient.

Iteration	$v_j^{(t)}$ computed	Weight given to current gradient
$t=1$	$0.01[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}{]}^2$	1%
$t=2$	$0.99\cdot 0.01[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}{]}^2+0.01[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(2)}{]}^2$	≈ 1%

During these very first iterations, the denominator in the RMSProp update rule

$${\theta }_j^{(t+1)}={\theta }_j^{(t)}-\frac{\eta }{\sqrt{v_j^{(t)}}+\epsilon }{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}$$

is still extremely small (close to $\epsilon$).
As a result, the effective learning rate becomes extremely large.

The risk of “wild jumps” in the loss landscape at the start of training

As a consequence, at the start of training, the RMSProp optimizer takes giant uncontrolled steps across the loss surface.
Instead of gradually descending into a promising valley, the model may:

• overshoot minima entirely,
• bounce chaotically across different regions,
• or even diverge.

This behavior can be described as the model “wandering around” the loss landscape, wasting early epochs without meaningful convergence.

Only after this short but turbulent transient phase does the EMA begin to accumulate enough recent gradient information.

Once $v_j^{(t)}$ has accumulated enough recent gradients, the denominator in the update rule grows accordingly.
This activates RMSProp’s self-normalization mechanism, which gradually reduces the effective learning rate and brings the optimizer out of its turbulent startup phase.
From that point on, RMSProp transitions into a stable regime of more controlled descent (exploitation).

At that point, if the model has not already been pushed out of promising valleys of the loss surface, the stabilized learning rate allows it to exploit one basin and converge more steadily toward a minimum.