AdamW

1. Intro

AdamW is the standard modern way to combine the Adam optimizer with weight decay.

It was introduced by Ilya Loshchilov and Frank Hutter in the paper Decoupled Weight Decay Regularization (arXiv 2017, published at ICLR 2019).

The central insight of that work is extremely important:

• in SGD, L2 regularization and weight decay are equivalent,
• in Adam, they are not equivalent,
• therefore, applying L2 regularization to Adam as if it were ordinary weight decay is conceptually incorrect.

AdamW was introduced precisely to fix this problem.

What problem does AdamW solve?

AdamW solves the following issue: when the term ${\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}$ is injected directly into Adam’s gradient pipeline, the regularization term is processed by Adam’s adaptive moment machinery. As a result, the shrinkage of the weights is no longer a clean, uniform weight decay. Instead, it becomes entangled with the history of the gradients.

Notation

In this note, ${\mathbf{w}}^{(t)}$ denotes the subset of trainable parameters to which weight decay is applied. In practice, this usually means weight tensors; biases and normalization parameters are often excluded.

The coefficient used directly in the optimizer update is denoted by ${\lambda }_{\mathrm{wd}}$. This choice keeps the AdamW formulas conceptually clean, because AdamW is defined as a decoupled update rule.

To connect this note with L2 regularization, suppose the regularized objective is written using the averaged-loss convention

$$\tilde{\mathcal{L}}=\mathcal{L}+\frac{\lambda }{2n}\Vert \mathbf{w}{\Vert }_2^2.$$

Then the corresponding decay coefficient in the optimizer update is

$${\lambda }_{\mathrm{wd}}=\frac{\lambda }{n}.$$

So the present note is fully compatible with the notation used in the L2-regularization note; it simply names the direct decay coefficient explicitly.

---

2. Motivation

When L2 regularization is first learned in the context of gradient descent or SGD, the following equivalence is usually established:

• optimizing a L2-regularized loss
• applying multiplicative weight decay at each step

For SGD, this equivalence is exact. Many implementations and explanations carried this intuition over to Adam. That transfer is the mistake AdamW was designed to correct.

Historical motivation

AdamW was not introduced as a minor variant of Adam. It was introduced because a large part of the community was informally treating “Adam + L2 regularization” as if it were the same thing as “Adam + weight decay”. The paper by Loshchilov and Hutter showed that this identification is false for adaptive optimizers.

---

3. L2 regularization and weight decay equivalence in SGD

Let

$${\mathcal{L}}^{(t)}$$

denote the data loss at iteration $t$, and let

$${\mathbf{g}}^{(t)}\triangleq {\nabla }_{\mathbf{w}}{\mathcal{L}}^{(t)}$$

be its gradient evaluated at the current parameters ${\mathbf{w}}^{(t)}$.

Now consider the L2-regularized objective

$${\tilde{\mathcal{L}}}^{(t)}={\mathcal{L}}^{(t)}+\frac{{\lambda }_{\mathrm{wd}}}{2}\Vert \mathbf{w}{\Vert }_2^2.$$

Its gradient is

$${\nabla }_{\mathbf{w}}{\tilde{\mathcal{L}}}^{(t)}={\mathbf{g}}^{(t)}+{\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}.$$

Therefore, the SGD update becomes

$${\mathbf{w}}^{(t+1)}={\mathbf{w}}^{(t)}-\eta \left({\mathbf{g}}^{(t)}+{\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}\right).$$

Rearranging,

$$\boxed{{\mathbf{w}}^{(t+1)}=(1-\eta {\lambda }_{\mathrm{wd}}){\mathbf{w}}^{(t)}-\eta {\mathbf{g}}^{(t)}}$$

This is the key identity.

It shows that, for SGD:

• the term $-\eta {\mathbf{g}}^{(t)}$ is the usual optimization step on the data loss,
• the factor $(1-\eta {\lambda }_{\mathrm{wd}})$ is a pure multiplicative shrinkage of the parameters.

This multiplicative shrinkage is exactly what is meant by weight decay.

Key conclusion for SGD

In SGD, adding the L2 penalty

$$\frac{{\lambda }_{\mathrm{wd}}}{2}\Vert \mathbf{w}{\Vert }_2^2$$

to the loss is mathematically equivalent to shrinking the parameters by the factor $(1-\eta {\lambda }_{\mathrm{wd}})$ at every step.

---

4. Why this equivalence breaks in Adam

The problem appears as soon as the update is no longer a plain scalar multiple of the gradient.

Adam does not apply a single global scaling to the gradient. Instead, it builds:

• a first-moment estimate,
• a second-moment estimate,
• a coordinate-wise adaptive denominator.

If L2 regularization is naively inserted into Adam, the algorithm uses the augmented gradient

$${\tilde{\mathbf{g}}}^{(t)}={\mathbf{g}}^{(t)}+{\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}$$

and then computes

$${\mathbf{m}}^{(t)}={\beta }_1{\mathbf{m}}^{(t-1)}+(1-{\beta }_1){\tilde{\mathbf{g}}}^{(t)},$$ $${\mathbf{v}}^{(t)}={\beta }_2{\mathbf{v}}^{(t-1)}+(1-{\beta }_2)({\tilde{\mathbf{g}}}^{(t)}{)}^{\odot 2},$$ $${\hat{\mathbf{m}}}^{(t)}=\frac{{\mathbf{m}}^{(t)}}{1-{\beta }_1^t},{\hat{\mathbf{v}}}^{(t)}=\frac{{\mathbf{v}}^{(t)}}{1-{\beta }_2^t},$$

followed by

$${\mathbf{w}}^{(t+1)}={\mathbf{w}}^{(t)}-\eta {\hat{\mathbf{m}}}^{(t)}\oslash \left(\sqrt{{\hat{\mathbf{v}}}^{(t)}}+\epsilon \right),$$

where all vector operations are elementwise.

Two problems immediately appear.

1 The regularization term contaminates the moment estimates

The moments ${\mathbf{m}}^{(t)}$ and ${\mathbf{v}}^{(t)}$ are no longer statistics of the data gradient alone. They now track the mixed quantity

$${\mathbf{g}}^{(t)}+{\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}.$$

This means that the regularization term is not merely added at the end of the update. It influences:

• the EMA of gradients,
• the EMA of squared gradients,
• the bias-corrected quantities,
• the adaptive denominator itself.

In other words, the regularizer is passed through machinery that was designed to analyze the geometry of the loss gradient, not the geometry of the regularization term.

2 The decay ceases to be a clean multiplicative shrinkage

To isolate the core issue, it is useful to momentarily ignore the first-moment smoothing and view Adam as a coordinate-wise preconditioner.

Define

$${\mathbf{D}}^{(t)}\triangleq \mathrm{diag}\left(\frac{1}{\sqrt{{\hat{\mathbf{v}}}^{(t)}}+\epsilon }\right).$$

Then the adaptive step has the schematic form

$${\mathbf{w}}^{(t+1)}\approx {\mathbf{w}}^{(t)}-\eta {\mathbf{D}}^{(t)}\left({\mathbf{g}}^{(t)}+{\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}\right).$$

Rearranging,

$$\boxed{{\mathbf{w}}^{(t+1)}\approx \left(\mathbf{I}-\eta {\lambda }_{\mathrm{wd}}{\mathbf{D}}^{(t)}\right){\mathbf{w}}^{(t)}-\eta {\mathbf{D}}^{(t)}{\mathbf{g}}^{(t)}}$$

This is no longer ordinary weight decay.

Indeed:

• if ${\mathbf{D}}^{(t)}$ were a scalar multiple of the identity, the decay would be uniform;
• in Adam, ${\mathbf{D}}^{(t)}$ is coordinate-wise and time-dependent;
• therefore, each parameter is shrunk by a different effective factor. The decay has become anisotropic and history-dependent.

The conceptual failure of "Adam + L2"

The quantity ${\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}$ is supposed to express a clean preference for small weights. In coupled Adam, however, this term is rescaled according to the past gradient statistics of each coordinate. As a result, some parameters are regularized more strongly than others for reasons that come from Adam’s adaptive geometry, not from the intended meaning of weight decay.

Note

This is the precise reason why the sentence “L2 regularization is the same as weight decay” is true for SGD but false for Adam.

---

5. The fix: AdamW

AdamW solves the above problem by applying the decay outside Adam’s adaptive gradient transformation.

The moments are computed from the true data gradient only:

$${\mathbf{g}}^{(t)}={\nabla }_{\mathbf{w}}{\mathcal{L}}^{(t)},$$ $${\mathbf{m}}^{(t)}={\beta }_1{\mathbf{m}}^{(t-1)}+(1-{\beta }_1){\mathbf{g}}^{(t)},$$ $${\mathbf{v}}^{(t)}={\beta }_2{\mathbf{v}}^{(t-1)}+(1-{\beta }_2)({\mathbf{g}}^{(t)}{)}^{\odot 2},$$ $${\hat{\mathbf{m}}}^{(t)}=\frac{{\mathbf{m}}^{(t)}}{1-{\beta }_1^t},{\hat{\mathbf{v}}}^{(t)}=\frac{{\mathbf{v}}^{(t)}}{1-{\beta }_2^t}.$$

Only after the Adam step is defined is weight decay applied:

$$\boxed{{\mathbf{w}}^{(t+1)}={\mathbf{w}}^{(t)}-\eta {\hat{\mathbf{m}}}^{(t)}\oslash \left(\sqrt{{\hat{\mathbf{v}}}^{(t)}}+\epsilon \right)-\eta {\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}}$$

Equivalently,

$$\boxed{{\mathbf{w}}^{(t+1)}=(1-\eta {\lambda }_{\mathrm{wd}}){\mathbf{w}}^{(t)}-\eta {\hat{\mathbf{m}}}^{(t)}\oslash \left(\sqrt{{\hat{\mathbf{v}}}^{(t)}}+\epsilon \right)}$$

This is AdamW.

The essential distinction

In AdamW:

• the adaptive moments see only the gradient of the data loss,
• the shrinkage term $-\eta {\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}$ is applied separately,
• the meaning of weight decay is restored.

---

5.1. Why this fix makes sense mathematically

Adam and weight decay serve two different purposes:

• Adam adapts the update to the local gradient statistics of the loss,
• weight decay expresses an external preference for smaller parameter norms.

These roles should not be mixed.

If the regularization term is passed through Adam’s adaptive denominator, then the penalty on the parameters becomes distorted by quantities that were never meant to regulate the penalty itself.

AdamW enforces the correct separation:

• the loss gradient is treated adaptively,
• the weight norm is shrunk directly.

This separation is exactly what the word decoupled refers to.

Decoupling

In AdamW, the optimization step with respect to the data loss and the shrinkage step with respect to weight decay are kept as two distinct operations. That is the entire conceptual heart of AdamW.

---

6. AdamW is not “Adam with an L2 term”

This distinction is subtle but fundamental.

For SGD:

• L2 regularization
• weight decay

produce the same update rule.

For Adam:

• Adam on a L2-regularized objective
• Adam plus explicit decoupled weight decay

produce different update rules.

Therefore, AdamW should be understood as:

• Adam applied to the data loss, plus
• a separate multiplicative shrinkage of the parameters.

It is not simply a mere rewriting of the L2-regularized Adam objective.

Common terminological trap

In many codebases and tutorials, the words “L2 regularization” and “weight decay” are used interchangeably. That shortcut is acceptable for SGD, but it becomes misleading for Adam-like optimizers. AdamW exists precisely because the shortcut fails.

---

7. Coordinate-wise interpretation

For a single decayed weight parameter $w_j$, AdamW performs

$$w_j^{(t+1)}=(1-\eta {\lambda }_{\mathrm{wd}})w_j^{(t)}-\eta \frac{{\hat{m}}_j^{(t)}}{\sqrt{{\hat{v}}_j^{(t)}}+\epsilon }.$$

This formula is very revealing:

• the factor $(1-\eta {\lambda }_{\mathrm{wd}})$ applies the same decay coefficient to every parameter,
• the adaptive factor $\frac{1}{\sqrt{{\hat{v}}_j^{(t)}}+\epsilon }$ acts only on the loss-gradient step,
• the regularization term is no longer distorted by coordinate-wise gradient statistics.

This is exactly the behavior that the original notion of weight decay is supposed to represent.

---

8. Why AdamW usually works better in practice

The paper introducing AdamW emphasized two practical consequences.

Better-behaved regularization

With coupled Adam, the strength of regularization is entangled with Adam’s adaptive rescaling. With AdamW, the shrinkage mechanism is explicit and interpretable.

Cleaner hyperparameter tuning

In coupled Adam, the effect of the regularization term is distorted by the optimizer’s internal state. In AdamW, the role of ${\lambda }_{\mathrm{wd}}$ is much cleaner:

• $\eta$ controls the scale of the Adam step,
• ${\lambda }_{\mathrm{wd}}$ controls the strength of multiplicative shrinkage.

This does not mean that $\eta$ and ${\lambda }_{\mathrm{wd}}$ become mathematically unrelated. The update still contains the product $\eta {\lambda }_{\mathrm{wd}}$. However, the regularization is no longer additionally warped by the adaptive denominator and moment estimates. This makes tuning substantially more interpretable.

Note

This is one of the most important practical reasons why AdamW replaced plain Adam in many modern training pipelines.

---

9. Adam + L2 vs AdamW

Method	Moments computed from	Does adaptive scaling affect the decay term?	True decoupled weight decay?
Adam + L2	${\mathbf{g}}^{(t)}+{\lambda }_{\mathrm{wd}}{\mathbf{w}}^{(t)}$	Yes	No
AdamW	${\mathbf{g}}^{(t)}$ only	No	Yes

---

10. Implementation notes

In modern frameworks

In modern deep-learning libraries, AdamW is typically implemented directly as the decoupled rule above.

This is the optimizer that should generally be preferred whenever:

• Adam-style adaptivity is desired,
• nonzero weight decay is required,
• a modern training pipeline is being designed.

Parameters often excluded from decay

In practice, weight decay is often applied to:

• weight matrices of linear layers,
• convolution kernels,
• embedding matrices,

but often excluded from:

• biases,
• BatchNorm / LayerNorm scale and shift parameters.

The underlying practical reason is that these parameters typically do not benefit from norm shrinkage in the same way as ordinary weights.

Practical rule

If an Adam-family optimizer is used together with weight_decay, the safe default is almost always:

• use AdamW, not plain Adam,
• apply decay primarily to true weight tensors,
• exclude biases and normalization parameters unless a specific reason exists not to.

---

11. Summary

AdamW should be remembered through one central idea:

Important

Weight decay must be decoupled from Adam’s adaptive gradient transformation.

The logic of the method is:

1. Adam computes an adaptive step from the data loss gradient.
2. Weight decay shrinks the parameters separately.

This separation is necessary because:

• in SGD, L2 regularization and weight decay are equivalent,
• in Adam, they are not,
• the naive transfer of the SGD intuition to Adam is mathematically incorrect.

For this reason, AdamW is not merely a convenient variant. It is the mathematically clean way to use weight decay with Adam.

Final takeaway

Whenever Adam-style optimization and weight decay are both desired, AdamW is the principled choice.