Adam

1. Intro

Adam ($\text{Ada}$ptive $\text{m}$oment estimation) combines two complementary ideas:

• a first-moment EMA of the gradients, which plays the role of momentum,
• a second-moment EMA of the squared gradients, inherited from RMSProp.

Instead of updating each parameter from the raw gradient alone, Adam builds:

• a smoothed estimate of the update direction,
• a smoothed estimate of the gradient scale.

For this reason, Adam is often summarized as a combination of Momentum + RMSProp.

Core idea

Adam may be interpreted as follows:

• Momentum in the numerator: the EMA of gradients stabilizes the direction of motion.
• RMSProp in the denominator: the EMA of squared gradients rescales each parameter adaptively.
• Bias correction on top: the zero-initialization bias of both EMAs is removed during the first iterations.

Why bias correction matters

Exponential moving averages initialized at zero are systematically biased toward zero at the beginning of training. In Adam this matters twice:

• the first-moment estimate $m_t$ underestimates the true average gradient,
• the second-moment estimate $v_t$ underestimates the gradient magnitude that appears in the denominator.

Without correction, the early updates are not properly scaled.

Adam's practical strength

Strong behavior is usually observed from the very first iterations:

• rapid initial progress is often obtained,
• the step size is adapted parameter-by-parameter,
• much of the unstable startup behavior that affects adaptive methods without bias correction is avoided.

---

2. Adam in depth

Info

This note describes the core Adam algorithm itself, not the full space of implementation variants and framework-specific options.

2.1 Adam update rule

Let

$$g_j^{(t)}\triangleq {\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}$$

be the gradient of the loss with respect to parameter ${\theta }_j$ at iteration $t$.

Adam maintains two exponential moving averages:

$$\boxed{m_j^{(t)}={\beta }_1{m}_j^{(t-1)}+(1-{\beta }_1)g_j^{(t)}}$$ $$\boxed{v_j^{(t)}={\beta }_2{v}_j^{(t-1)}+(1-{\beta }_2)(g_j^{(t)}{)}^2}$$

with initialization

$$m_j^{(0)}=0,v_j^{(0)}=0.$$

Because both moving averages start at zero, Adam corrects them as follows:

$$\boxed{{\hat{m}}_j^{(t)}=\frac{m_j^{(t)}}{1-{\beta }_1^t},{\hat{v}}_j^{(t)}=\frac{v_j^{(t)}}{1-{\beta }_2^t}}$$

The parameter update is then:

$$\boxed{{\theta }_j^{(t+1)}={\theta }_j^{(t)}-\eta \frac{{\hat{m}}_j^{(t)}}{\sqrt{{\hat{v}}_j^{(t)}}+\epsilon }}$$

where:

• $\eta$ is the base learning rate,
• ${\beta }_1$ controls the memory of the first moment,
• ${\beta }_2$ controls the memory of the second moment,
• $\epsilon$ is a small numerical-stability constant.

The default values proposed in the original Adam paper are:

$${\beta }_1=0.9,{\beta }_2=0.999,\epsilon \approx 10^{-8}.$$

Note

The square root in $\sqrt{{\hat{v}}_j^{(t)}}$ is essential. Since $v_j^{(t)}$ tracks a second moment, taking the square root brings the denominator back to the same scale as the gradient.

---

2.2 EMA in Adam

Adam uses two distinct EMAs, each with a different role in the update.

Quantity	Recursive formula	Role
First moment $m_j^{(t)}$	$m_j^{(t)}={\beta }_1{m}_j^{(t-1)}+(1-{\beta }_1)g_j^{(t)}$	Tracks a smoothed version of the gradient; this stabilizes direction and plays the role of momentum
Second moment $v_j^{(t)}$	$v_j^{(t)}={\beta }_2{v}_j^{(t-1)}+(1-{\beta }_2)(g_j^{(t)}{)}^2$	Tracks a smoothed version of squared gradient magnitudes; this rescales each parameter adaptively

The qualitative effect is the following:

• if a parameter has received large recent gradients, then $v_j^{(t)}$ becomes large and the effective step on that parameter is reduced;
• if a parameter has received small or infrequent gradients, then $v_j^{(t)}$ stays smaller and that parameter is updated more aggressively.

Interpretation

In Adam, two roles that are not separated in plain SGD are handled explicitly:

• the numerator determines the update direction,
• the denominator determines the update scale for each parameter.

---

2.3 Correcting EMA bias at initialization

2.3.1 EMA is biased toward zero at the beginning

Because $m_j^{(0)}=0$ and $v_j^{(0)}=0$, the first few values of the two EMAs are pulled toward zero by construction.

If the gradient statistics are roughly stationary over the initial iterations, then:

• the average gradient does not change dramatically from one step to the next,
• the average squared gradient does not change dramatically either,
• in other words, during the very early phase of training the gradient process may be treated as having approximately stable first and second moments.

Concretely, this means that for the first few steps the following approximation is used:

$$\mathbb{E}[g_j^{(1)}]\approx \mathbb{E}[g_j^{(2)}]\approx \cdots ,\mathbb{E}[(g_j^{(1)}{)}^2]\approx \mathbb{E}[(g_j^{(2)}{)}^2]\approx \cdots$$

This assumption does not assert that the gradients are exactly constant. It is only a local approximation used to isolate the effect of zero initialization of the EMAs. Under this approximation, the bias formulas below show clearly that the factors $(1-{\beta }_1^t)$ and $(1-{\beta }_2^t)$ arise from initialization, not from some special property of the loss.

On the use of expectations such as $\mathbb{E}[g_t]$

For readers accustomed to taking expectations only of explicit random variables, the following statistical viewpoint is useful.

In full-batch gradient descent, once the dataset and the parameter vector are fixed, the gradient is deterministic. In mini-batch SGD, however, the gradient computed at step $t$ depends on the randomly selected mini-batch. Therefore, for a fixed parameter vector ${\mathbf{\theta }}^{(t)}$, the gradient may be modeled as a random variable:

$$G_j^{(t)}={\nabla }_{{\theta }_j}{\mathcal{L}}_{{\mathcal{B}}_t}({\mathbf{\theta }}^{(t)}),$$

where ${\mathcal{B}}_t$ denotes the random mini-batch sampled at iteration $t$. The actually observed gradient $g_j^{(t)}$ is then one realization of that random variable.

Under this viewpoint:

• $\mathbb{E}[G_j^{(t)}]$ is the average gradient over all possible mini-batches,
• $\mathbb{E}[(G_j^{(t)}{)}^2]$ is the average squared gradient over all possible mini-batches.

This is exactly the same kind of expectation used in probability theory for any random variable. In informal optimizer notes, the same symbol $g_j^{(t)}$ is often used both for the random quantity and for one realized value. That notation is slightly abusive, but standard.

The expectation is introduced here because the bias of an EMA is fundamentally a statistical statement about averages over repeated draws, not only a statement about one single realized training trajectory.

$$\mathbb{E}\left[m_j^{(t)}\right]\approx (1-{\beta }_1^t)\mathbb{E}\left[g_j^{(t)}\right],$$ $$\mathbb{E}\left[v_j^{(t)}\right]\approx (1-{\beta }_2^t)\mathbb{E}\left[(g_j^{(t)}{)}^2\right].$$

This is the origin of the correction factors $1-{\beta }_1^t$ and $1-{\beta }_2^t$.

Note

Bias correction does not mean that Adam becomes magically perfect in every non-stationary setting. It means that Adam removes the specific distortion induced by initializing the EMAs at zero.

2.3.2 First-step behavior

At the first iteration ($t=1$), before bias correction:

$$m_j^{(1)}=(1-{\beta }_1)g_j^{(1)},v_j^{(1)}=(1-{\beta }_2)(g_j^{(1)}{)}^2.$$

Therefore, without bias correction, the first Adam update would be:

$$\Delta {\theta }_j^{(1)}=-\eta \frac{(1-{\beta }_1)g_j^{(1)}}{\sqrt{1-{\beta }_2}|g_j^{(1)}|+\epsilon }.$$

If $\epsilon$ is negligible compared with $\sqrt{1-{\beta }_2}|g_j^{(1)}|$, this becomes approximately:

$$\Delta {\theta }_j^{(1)}\approx -\eta \frac{1-{\beta }_1}{\sqrt{1-{\beta }_2}}\mathrm{sign}\left(g_j^{(1)}\right).$$

With the default values:

$$\frac{1-{\beta }_1}{\sqrt{1-{\beta }_2}}=\frac{0.1}{\sqrt{0.001}}\approx \mathrm{3.16.}$$

The first step is therefore not “almost negligible”. If anything, it can be too large, because the denominator is underestimated even more strongly than the numerator.

Common misconception

A small $m_1$ and a small $v_1$ do not imply a tiny Adam update. Since $v_1$ appears under a square root in the denominator, the early step can actually be amplified.

2.3.3 With bias correction

Applying the bias-correction terms gives:

$${\hat{m}}_j^{(1)}=\frac{m_j^{(1)}}{1-{\beta }_1}=g_j^{(1)},{\hat{v}}_j^{(1)}=\frac{v_j^{(1)}}{1-{\beta }_2}=(g_j^{(1)}{)}^2.$$

Hence the first corrected update is:

$$\Delta {\theta }_j^{(1)}=-\eta \frac{g_j^{(1)}}{|g_j^{(1)}|+\epsilon }\approx -\eta \mathrm{sign}\left(g_j^{(1)}\right).$$

A much better-behaved update is therefore obtained:

• the update direction is aligned with the true gradient,
• the initial scale distortion caused by zero initialization is removed,
• the optimizer starts from a controlled regime rather than from a badly mis-scaled one.

Why the correction is especially important for $v_t$

Since ${\beta }_2=0.999$ is very close to $1$, the second-moment estimate is heavily biased toward zero at the start if left uncorrected. That is precisely the term sitting in the denominator. This is why bias correction is a central part of Adam rather than a minor technical addition.

2.3.4 Why the correction matters mostly at the beginning

As $t$ grows,

$${\beta }_1^t\to 0,{\beta }_2^t\to 0,$$

because both ${\beta }_1$ and ${\beta }_2$ are chosen in the interval $(0,1)$. Whenever a number satisfies $0<\beta <1$, repeated multiplication by $\beta$ makes it exponentially smaller:

$$\beta ,{\beta }^2,{\beta }^3,\dots \to 0.$$

For example:

$$0.9^{10}\approx 0.35,0.9^{100}\approx 2.7\times 10^{-5},$$

and

$$0.999^{1000}\approx 0.368,0.999^{10000}\approx 4.5\times 10^{-5}.$$

Thus, even if the decay is slow when $\beta$ is very close to $1$, the power ${\beta }^t$ still eventually vanishes.

Hence

$$1-{\beta }_1^t\to 1,1-{\beta }_2^t\to 1.$$

Therefore, the bias-correction terms matter mainly during the startup phase. After enough iterations, Adam behaves like an adaptive optimizer based on well-formed moving averages.

---

2.4 Dimensional consideration

In much of Deep Learning, parameters and losses are treated as dimensionless quantities. Even under that convention, it remains useful to check that Adam is internally consistent:

• $m_j^{(t)}$ and ${\hat{m}}_j^{(t)}$ have the same units as the gradient,
• $v_j^{(t)}$ and ${\hat{v}}_j^{(t)}$ have the units of the gradient squared,
• $\sqrt{{\hat{v}}_j^{(t)}}$ has the same units as the gradient,
• therefore ${\hat{m}}_j^{(t)}/\sqrt{{\hat{v}}_j^{(t)}}$ is dimensionless.

Thus the update

$$\Delta {\theta }_j^{(t)}=-\eta \frac{{\hat{m}}_j^{(t)}}{\sqrt{{\hat{v}}_j^{(t)}}+\epsilon }$$

is dimensionally consistent.

Info

The dimensional check also clarifies why the denominator must be $\sqrt{{\hat{v}}_t}$ rather than ${\hat{v}}_t$. Without the square root, the numerator and denominator would not live on compatible scales.

---

3. Summary

Adam is best understood as an optimizer built from three ingredients:

• momentum-like smoothing through the EMA of gradients,
• RMSProp-like adaptivity through the EMA of squared gradients,
• bias correction to neutralize the zero-initialization distortion of both EMAs.

Its main strengths are:

• stable and effective behavior from the very first iterations,
• per-parameter adaptive step sizes,
• fast practical convergence on a wide range of deep-learning problems.

Its key conceptual advantage over a plain RMSProp-style method is that the startup phase is handled much more carefully:

• the first-moment estimate is corrected,
• the second-moment estimate is corrected,
• the early effective learning rate is far better controlled.

Practical note

Adam remains one of the strongest default optimizers in Deep Learning. However, when weight decay is needed, the modern practical choice is usually AdamW, not “Adam + L2 mixed into the gradient”, because the regularization term should be decoupled from Adam’s adaptive rescaling.