Adam

1. Intro

Adam ($\text{Ada}$ptive $\text{m}$oment estimation) is the natural completion of the adaptive-optimizer line that began with AdaGrad and continued with RMSProp. It combines two complementary ideas:

• a first-moment EMA of the gradients, which plays the role of Momentum;
• a second-moment EMA of the squared gradients, inherited from RMSProp.

Instead of updating each parameter from the raw gradient alone, Adam builds:

• a smoothed estimate of the update direction,
• a smoothed estimate of the gradient scale.

For this reason, Adam is often summarized as a combination of Momentum + RMSProp, with an additional bias correction that fixes the startup transient of plain RMSProp.

Core idea

Adam may be interpreted as follows:

• Momentum in the numerator: the EMA of gradients stabilizes the direction of motion.
• RMSProp in the denominator: the EMA of squared gradients rescales each parameter adaptively.
• Bias correction on top: the zero-initialization bias of both EMAs is removed during the first iterations.

Why bias correction matters

Exponential moving averages initialized at zero are systematically biased toward zero at the beginning of training. In Adam this matters twice:

• the first-moment estimate $m_t$ underestimates the true average gradient,
• the second-moment estimate $v_t$ underestimates the gradient magnitude that appears in the denominator.

Without correction, the early updates are not properly scaled.

Adam's practical strength

Strong behavior is usually observed from the very first iterations:

• rapid initial progress is often obtained,
• the step size is adapted parameter-by-parameter,
• much of the unstable startup behavior that affects adaptive methods without bias correction is avoided.

---

2. Adam in depth

Info

This note describes the core Adam algorithm itself, not the full space of implementation variants and framework-specific options.

2.1 Adam update rule

Let

$$g_j^{(t)}\triangleq {\nabla }_{{\theta }_j}{\mathcal{L}}^{(t)}$$

be the gradient of the loss with respect to parameter ${\theta }_j$ at iteration $t$.

Adam maintains two exponential moving averages:

$$\boxed{m_j^{(t)}={\beta }_1{m}_j^{(t-1)}+(1-{\beta }_1)g_j^{(t)}}$$ $$\boxed{v_j^{(t)}={\beta }_2{v}_j^{(t-1)}+(1-{\beta }_2)(g_j^{(t)}{)}^2}$$

with initialization

$$m_j^{(0)}=0,v_j^{(0)}=0.$$

Because both moving averages start at zero, Adam corrects them as follows:

$$\boxed{{\hat{m}}_j^{(t)}=\frac{m_j^{(t)}}{1-{\beta }_1^t},{\hat{v}}_j^{(t)}=\frac{v_j^{(t)}}{1-{\beta }_2^t}}$$

The parameter update is then:

$$\boxed{{\theta }_j^{(t+1)}={\theta }_j^{(t)}-\eta \frac{{\hat{m}}_j^{(t)}}{\sqrt{{\hat{v}}_j^{(t)}}+\epsilon }}$$

where:

• $\eta$ is the base learning rate,
• ${\beta }_1$ controls the memory of the first moment,
• ${\beta }_2$ controls the memory of the second moment,
• $\epsilon$ is a small numerical-stability constant.

The default values proposed in the original Adam paper are:

$${\beta }_1=0.9,{\beta }_2=0.999,\epsilon \approx 10^{-8}.$$

Note

The square root in $\sqrt{{\hat{v}}_j^{(t)}}$ is essential. Since $v_j^{(t)}$ tracks a second moment, taking the square root brings the denominator back to the same scale as the gradient. The same dimensional argument used in RMSProp applies here unchanged.

Effective memory length of the two EMAs

Each EMA in Adam has an effective memory horizon of approximately $1/(1-\beta )$ steps (the same heuristic introduced for RMSProp’s $1/(1-\alpha )$):

• the first moment with ${\beta }_1=0.9$ remembers roughly the last $10$ steps of gradients;
• the second moment with ${\beta }_2=0.999$ remembers roughly the last $1000$ steps of squared gradients.

The asymmetry is deliberate. The direction signal is allowed to react quickly to changes in the gradient (short memory), while the per-coordinate scale signal is allowed to vary only slowly (long memory). The second-moment estimator’s much longer horizon is exactly why its bias correction is so important: with ${\beta }_2=0.999$, the uncorrected $v_j^{(t)}$ does not approach a fully-formed estimate until hundreds of iterations, and the early denominator can be systematically too small without correction.

---

2.2 EMA in Adam

Adam uses two distinct EMAs, each with a different role in the update.

Quantity	Recursive formula	Role
First moment $m_j^{(t)}$	$m_j^{(t)}={\beta }_1{m}_j^{(t-1)}+(1-{\beta }_1)g_j^{(t)}$	Tracks a smoothed version of the gradient; this stabilizes direction and plays the role of momentum
Second moment $v_j^{(t)}$	$v_j^{(t)}={\beta }_2{v}_j^{(t-1)}+(1-{\beta }_2)(g_j^{(t)}{)}^2$	Tracks a smoothed version of squared gradient magnitudes; this rescales each parameter adaptively

The qualitative effect is the following:

• if a parameter has received large recent gradients, then $v_j^{(t)}$ becomes large and the effective step on that parameter is reduced;
• if a parameter has received small or infrequent gradients, then $v_j^{(t)}$ stays smaller and that parameter is updated more aggressively.

Interpretation

In Adam, two roles that are not separated in plain SGD are handled explicitly:

• the numerator determines the update direction,
• the denominator determines the update scale for each parameter.

---

2.3 Correcting EMA bias at initialization

2.3.1 EMA is biased toward zero at the beginning

Because $m_j^{(0)}=0$ and $v_j^{(0)}=0$, the first few values of the two EMAs are pulled toward zero by construction.

If the gradient statistics are roughly stationary over the initial iterations, then:

• the average gradient does not change dramatically from one step to the next,
• the average squared gradient does not change dramatically either,
• in other words, during the very early phase of training the gradient process may be treated as having approximately stable first and second moments.

Concretely, this means that for the first few steps the following approximation is used:

$$\mathbb{E}[g_j^{(1)}]\approx \mathbb{E}[g_j^{(2)}]\approx \cdots ,\mathbb{E}[(g_j^{(1)}{)}^2]\approx \mathbb{E}[(g_j^{(2)}{)}^2]\approx \cdots$$

This assumption does not assert that the gradients are exactly constant. It is only a local approximation used to isolate the effect of zero initialization of the EMAs. Under this approximation, the bias formulas below show clearly that the factors $(1-{\beta }_1^t)$ and $(1-{\beta }_2^t)$ arise from initialization, not from some special property of the loss.

On the use of expectations such as $\mathbb{E}[g_t]$

For readers accustomed to taking expectations only of explicit random variables, the following statistical viewpoint is useful.

In full-batch gradient descent, once the dataset and the parameter vector are fixed, the gradient is deterministic. In mini-batch SGD, however, the gradient computed at step $t$ depends on the randomly selected mini-batch. Therefore, for a fixed parameter vector ${\mathbf{\theta }}^{(t)}$, the gradient may be modeled as a random variable:

$$G_j^{(t)}={\nabla }_{{\theta }_j}{\mathcal{L}}_{{\mathcal{B}}_t}({\mathbf{\theta }}^{(t)}),$$

where ${\mathcal{B}}_t$ denotes the random mini-batch sampled at iteration $t$. The actually observed gradient $g_j^{(t)}$ is then one realization of that random variable.

Under this viewpoint:

• $\mathbb{E}[G_j^{(t)}]$ is the average gradient over all possible mini-batches,
• $\mathbb{E}[(G_j^{(t)}{)}^2]$ is the average squared gradient over all possible mini-batches.

This is exactly the same kind of expectation used in probability theory for any random variable. In informal optimizer notes, the same symbol $g_j^{(t)}$ is often used both for the random quantity and for one realized value. That notation is slightly abusive, but standard.

The expectation is introduced here because the bias of an EMA is fundamentally a statistical statement about averages over repeated draws, not only a statement about one single realized training trajectory.

$$\mathbb{E}\left[m_j^{(t)}\right]\approx (1-{\beta }_1^t)\mathbb{E}\left[g_j^{(t)}\right],$$ $$\mathbb{E}\left[v_j^{(t)}\right]\approx (1-{\beta }_2^t)\mathbb{E}\left[(g_j^{(t)}{)}^2\right].$$

This is the origin of the correction factors $1-{\beta }_1^t$ and $1-{\beta }_2^t$.

Deriving the factor $1-{\beta }_1^t$ from the EMA

Unrolling the first-moment recursion from $m_j^{(0)}=0$ expresses the EMA as a weighted sum of all past gradients (the same expansion is carried out step by step for the second moment in RMSProp):

$$m_j^{(t)}=(1-{\beta }_1)\sum _{k=1}^t{\beta }_1^{t-k}{g}_j^{(k)}.$$

Taking the expectation under the local-stationarity approximation $\mathbb{E}[g_j^{(k)}]\approx \mathbb{E}[g_j^{(t)}]$ allows the constant mean to be pulled out of the sum:

$$\mathbb{E}\left[m_j^{(t)}\right]=\mathbb{E}\left[g_j^{(t)}\right](1-{\beta }_1)\sum _{k=1}^t{\beta }_1^{t-k}.$$

The remaining geometric sum is ${\sum }_{k=1}^t{\beta }_1^{t-k}=1+{\beta }_1+\cdots +{\beta }_1^{t-1}=\frac{1-{\beta }_1^t}{1-{\beta }_1}$, and the prefactor $(1-{\beta }_1)$ cancels its denominator, leaving

$$\mathbb{E}\left[m_j^{(t)}\right]=(1-{\beta }_1^t)\mathbb{E}\left[g_j^{(t)}\right].$$

Dividing $m_j^{(t)}$ by $1-{\beta }_1^t$ therefore removes the bias exactly. The identical argument applied to the squared gradients gives $\mathbb{E}[v_j^{(t)}]=(1-{\beta }_2^t)\mathbb{E}[(g_j^{(t)}{)}^2]$, which is the source of the $1-{\beta }_2^t$ correction.

Note

Bias correction addresses exactly one defect: the downward bias that zero-initialized EMAs carry during the first iterations. It removes that, and nothing beyond it. Where the gradient statistics are themselves non-stationary and drift over training, the corrected moments remain approximations of moving targets.

2.3.2 First-step behavior

At the first iteration ($t=1$), before bias correction:

$$m_j^{(1)}=(1-{\beta }_1)g_j^{(1)},v_j^{(1)}=(1-{\beta }_2)(g_j^{(1)}{)}^2.$$

Therefore, without bias correction, the first Adam update would be:

$$\Delta {\theta }_j^{(1)}=-\eta \frac{(1-{\beta }_1)g_j^{(1)}}{\sqrt{1-{\beta }_2}|g_j^{(1)}|+\epsilon }.$$

If $\epsilon$ is negligible compared with $\sqrt{1-{\beta }_2}|g_j^{(1)}|$, this becomes approximately:

$$\Delta {\theta }_j^{(1)}\approx -\eta \frac{1-{\beta }_1}{\sqrt{1-{\beta }_2}}\mathrm{sign}\left(g_j^{(1)}\right).$$

With the default values:

$$\frac{1-{\beta }_1}{\sqrt{1-{\beta }_2}}=\frac{0.1}{\sqrt{0.001}}\approx \mathrm{3.16.}$$

The first step is therefore not “almost negligible”. If anything, it can be too large, because the denominator is underestimated even more strongly than the numerator.

Common misconception

A small $m_1$ and a small $v_1$ do not imply a tiny Adam update. Since $v_1$ appears under a square root in the denominator, the early step can actually be amplified.

2.3.3 With bias correction

Applying the bias-correction terms gives:

$${\hat{m}}_j^{(1)}=\frac{m_j^{(1)}}{1-{\beta }_1}=g_j^{(1)},{\hat{v}}_j^{(1)}=\frac{v_j^{(1)}}{1-{\beta }_2}=(g_j^{(1)}{)}^2.$$

Hence the first corrected update is:

$$\Delta {\theta }_j^{(1)}=-\eta \frac{g_j^{(1)}}{|g_j^{(1)}|+\epsilon }\approx -\eta \mathrm{sign}\left(g_j^{(1)}\right).$$

A much better-behaved update is therefore obtained:

• the update direction is aligned with the true gradient,
• the initial scale distortion caused by zero initialization is removed,
• the optimizer starts from a controlled regime rather than from a badly mis-scaled one.

Why the correction is especially important for $v_t$

Since ${\beta }_2=0.999$ is very close to $1$, the second-moment estimate is heavily biased toward zero at the start if left uncorrected. That is precisely the term sitting in the denominator. This is why bias correction is a central part of Adam rather than a minor technical addition.

2.3.4 Why the correction matters mostly at the beginning

As $t$ grows,

$${\beta }_1^t\to 0,{\beta }_2^t\to 0,$$

because both ${\beta }_1$ and ${\beta }_2$ are chosen in the interval $(0,1)$. Whenever a number satisfies $0<\beta <1$, repeated multiplication by $\beta$ makes it exponentially smaller:

$$\beta ,{\beta }^2,{\beta }^3,\dots \to 0.$$

For example:

$$0.9^{10}\approx 0.35,0.9^{100}\approx 2.7\times 10^{-5},$$

and

$$0.999^{1000}\approx 0.368,0.999^{10000}\approx 4.5\times 10^{-5}.$$

Thus, even if the decay is slow when $\beta$ is very close to $1$, the power ${\beta }^t$ still eventually vanishes.

Hence

$$1-{\beta }_1^t\to 1,1-{\beta }_2^t\to 1.$$

Therefore, the bias-correction terms matter mainly during the startup phase. After enough iterations, Adam behaves like an adaptive optimizer based on well-formed moving averages.

---

2.4 Dimensional consideration

In much of Deep Learning, parameters and losses are treated as dimensionless quantities. Even under that convention, it remains useful to check that Adam is internally consistent:

• $m_j^{(t)}$ and ${\hat{m}}_j^{(t)}$ have the same units as the gradient,
• $v_j^{(t)}$ and ${\hat{v}}_j^{(t)}$ have the units of the gradient squared,
• $\sqrt{{\hat{v}}_j^{(t)}}$ has the same units as the gradient,
• therefore ${\hat{m}}_j^{(t)}/\sqrt{{\hat{v}}_j^{(t)}}$ is dimensionless.

Thus the update

$$\Delta {\theta }_j^{(t)}=-\eta \frac{{\hat{m}}_j^{(t)}}{\sqrt{{\hat{v}}_j^{(t)}}+\epsilon }$$

is dimensionally consistent.

Info

The dimensional check also clarifies why the denominator must be $\sqrt{{\hat{v}}_t}$ rather than ${\hat{v}}_t$. Without the square root, the numerator and denominator would not live on compatible scales.

---

3. Summary

Adam is built from three ingredients:

• momentum-like smoothing through the EMA of gradients in the numerator;
• RMSProp-like adaptivity through the EMA of squared gradients in the denominator;
• bias correction to neutralize the zero-initialization distortion of both EMAs during the startup phase.

The combination delivers stable, per-parameter, well-scaled updates from the very first iteration, which is the main reason Adam remains one of the strongest default optimizers in deep learning.

Memory cost: $3\times$ the parameters (one first-moment buffer and one second-moment buffer per parameter), versus $2\times$ for RMSProp and SGD with Momentum, and $1\times$ for plain SGD. This cost matters at very large scale and motivates memory-efficient variants like Adafactor and 8-bit Adam in foundation-model training; see Choosing an optimizer for the comparison table.

Adam vs AdamW when weight decay is involved

Whenever weight decay is needed, the correct choice is AdamW, not “Adam + L2 mixed into the gradient”. L2 added to Adam’s gradient is passed through Adam’s adaptive denominator, which distorts the regularization in a way that has no statistical justification. AdamW decouples the weight-decay step from the adaptive rescaling, restoring the geometric content of L2 regularization. This is treated in detail in AdamW.

Sources

Adam comes from Kingma and Ba (2014). It and the wider optimizer and regularizer literature are collected in Optimization and Regularization.