Adam

Intro

Adam ($\text{Ada}$ptive $\text{m}$oment estimation) combines two key ideas:

• Momentum: accumulation of past gradients.
• RMSProp: accumulation of squared gradients.

In Adam, the EMA is applied to first momentum as well both to the first moment (momentum) and the second moment (squared gradients).

How Adam addresses EMA being biased toward zero

Since EMA alone is biased toward zero (i.e., it can cause extremely large effective learning rates during the very first training iterations) Adam introduces a bias-correction mechanism that stabilizes updates in the early stages of training.

Adam’s strength

Thanks to the bias correction and combined use of momentum + RMSProp, Adam achieves a more balanced and robust learning rate behavior from the very start of training: fast exploration early on without wild divergence, followed by controlled convergence.

---

Adam in depth

Adam update rule

The update rule prescribed by Adam is:

$$\boxed{{\theta }_j^{(t+1)}={\theta }_j^{(t)}-\eta \frac{{\hat{m}}_j^{(t+1)}}{\sqrt{{\hat{v}}_j^{(t+1)}}+\epsilon }}$$

where:

• ${\hat{m}}_j^{(t+1)}=\frac{m_j^{(t+1)}}{1-{\beta }_1}$: bias-corrected EMA of gradients
• ${\hat{v}}_j^{(t+1)}=\frac{v_j^{(t+1)}}{1-{\beta }_2}$: bias-corrected EMA of squared gradients
• $\eta$: base learning-rate
• $\epsilon \sim 10^{-8}$: term for numerical stability

---

EMA in Adam

Adam uses two Exponential Moving Averages (EMAs):

• one on the sequence of gradients (equivalent to Momentum),
• one on the squared gradients (same principle behind RMSProp).

The table below summarizes their respective formulas and roles.

Recursive formula
$m_j^{(t+1)}={\beta }_1{m}_j^{(t)}+(1-{\beta }_1){\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}$	Exponential Moving Average (EMA) applied to the history of gradients (borrowed from the Momentum technique). In this update formula, instead of multiplying $\eta$ by the current gradient, an EMA of past gradients is used. This reflects the same principle as Momentum: the update of the current state depends not only on the present gradient but also on gradients from previous steps, weighted by a coefficient $\mu$.
$v_j^{(t+1)}={\beta }_2{v}_j^{(t)}+(1-{\beta }_2)[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}{]}^2$	Exponential Moving Average (EMA) applied to the history of squared gradients (borrowed from RMSProp).

Default parameters (from the original paper):

$${\beta }_1=0.9,{\beta }_2=\mathrm{0.999.}$$

These two coefficients are used in the EMAs above, but more importantly, they are essential to correct the zero-bias inherent in such exponential moving averages.

---

Correcting EMA bias at initialization

EMA is biased toward zero

At the beginning of training ($t=0,1,2,\dots$), the two EMAs exhibit a bias toward zero, since they are computed over only a very small number of samples.

No bias correction scenario

If we consider the update rule without bias correction:

$${\theta }_j^{(t+1)}={\theta }_j^{(t)}-\eta \frac{m_j^{(t+1)}}{\sqrt{v_j^{(t+1)}}+\epsilon },$$

then in the very first iterations, since $m_j^{(0)}=0$ and $v_j^{(0)}=0$, the estimates $m_j^{(1)}$ and $v_j^{(1)}$ turn out to be very small, even if the gradient ${\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}$ is not.

In fact, at the first step ($t=0\to t=1$), we have:

$$\begin{array}{rl}{m}_j^{(1)}& ={\beta }_1\cdot 0+(1-{\beta }_1){\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}=(1-{\beta }_1){\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}\\ v_j^{(1)}& =(1-{\beta }_2){\left[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}\right]}^2\end{array}$$

With the values suggested in the original paper (${\beta }_1=0.9$, ${\beta }_2=0.999$), the initial coefficients are:

• $1-{\beta }_1=0.1$
• $1-{\beta }_2=0.001$

Thus, both $m_j^{(1)}$ and $v_j^{(1)}$ are heavily attenuated, making the update ${\theta }_j^{(1)}$ almost negligible:

$${\theta }_j^{(1)}={\theta }_j^{(0)}-\eta \cdot \frac{0.1{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}}{\sqrt{0.001[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}{]}^2}+\epsilon }.$$

Bias correction for EMAs at initialization

To avoid this issue, Adam introduces a bias correction through the following terms:

$$\boxed{{\hat{m}}_j^{(t+1)}=\frac{m_j^{(t+1)}}{1-{\beta }_1},{\hat{v}}_j^{(t+1)}=\frac{v_j^{(t+1)}}{1-{\beta }_2}}$$

With bias correction

Applying bias correction:

$${\hat{m}}_j^{(1)}=\frac{m_j^{(1)}}{1-{\beta }_1},{\hat{v}}_j^{(1)}=\frac{v_j^{(1)}}{1-{\beta }_2}.$$

At the first step ($m_j^{(0)}=v_j^{(0)}=0$), this becomes:

$${\hat{m}}_j^{(1)}=\frac{(1-{\beta }_1){\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}}{1-{\beta }_1}={\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)},$$ $${\hat{v}}_j^{(1)}=\frac{(1-{\beta }_2)[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}{]}^2}{1-{\beta }_2}=[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}{]}^2.$$

Thus, thanks to the correction, the update rule is already consistent with the true gradient from the very first iteration, preventing learning from stalling:

$${\theta }_j^{(1)}={\theta }_j^{(0)}-\eta \cdot \frac{{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}}{\sqrt{[{\nabla }_{{\theta }_j}{\mathcal{L}}^{(1)}{]}^2}+\epsilon }.$$

Dimensional consideration

In Deep Learning, parameters ${\theta }_j$ (and typically the loss $\mathcal{L}$) are treated as dimensionless quantities. Keeping in mind this convention, in Adam’s update rule:

• Numerator (${\hat{m}}_j$), the bias-corrected EMA of gradients, has the same units as the gradient. This ensures that the update direction is consistent with gradient descent: updates are proportional to the gradient, not to its square.
• Denominator ($\sqrt{{\hat{v}}_j}$), the bias-corrected EMA of squared gradients, also has the same units as the gradient. Dividing ${\hat{m}}_j$ by $\sqrt{{\hat{v}}_j}$ yields a dimensionless ratio.

As a result, the update step

$$\Delta {\theta }_j=-\eta \frac{{\hat{m}}_j^{(t+1)}}{\sqrt{{\hat{v}}_j^{(t+1)}}+\epsilon }$$

remains dimensionally consistent (i.e., dimensionless) and interpretable as a scaled gradient descent step.

---

Summary

Adam sets itself apart from other optimizers because it works well right from the very first training steps, thanks to the bias-correction applied to its EMAs.

• It applies bias-correction to both the first moment ($m_t$) and second moment ($v_t$) estimates, counteracting their initialization at zero.
• This prevents the denominators in the update rule from being too close to zero, ensuring stable updates even in the earliest training iterations.
• After the initial transient phase, Adam behaves similarly to other adaptive optimizers, but its key advantage lies in handling the start of training reliably, where other methods may diverge or stall.
• It also retains per-parameter adaptivity: parameters with frequent or large gradients are moderated, while rare or smaller ones are not overshadowed.

Info

Adam is a robust and self-adaptive optimizer, and has become the de facto standard for training deep networks.