L2 regularization in depth

The previous note introduced L2 regularization at the level of the update rule: at every gradient step, the weight $w_j$ is rescaled by a factor $(1-\eta \lambda /n)$ before the usual gradient update is applied. That view is operationally sufficient, but it leaves a deeper question unanswered:

The deeper question

Suppose training converges. What does the regularized solution $\tilde{\mathbf{w}}$ look like, as a function of the unregularized optimum ${\mathbf{w}}^{\star }$? Which components of ${\mathbf{w}}^{\star }$ does L2 shrink, and which does it leave essentially untouched?

The answer is one of the most elegant results in classical regularization theory: L2 regularization performs an anisotropic shrinkage of ${\mathbf{w}}^{\star }$ along the eigenvectors of the loss Hessian. The components of ${\mathbf{w}}^{\star }$ that lie along flat directions of the loss (small eigenvalues of the Hessian) are shrunk almost to zero, while components along steep directions (large eigenvalues) survive nearly intact. The derivation is short, the geometry is striking, and the conclusion gives the deepest intuition for why L2 regularization improves generalization.

This note follows the analysis of Goodfellow, Bengio and Courville (Deep Learning, Section 7.1.1), adapted to the notation used throughout this site.

Setup

The regularized objective, as defined in L2 regularization, is

$$\tilde{\mathcal{L}}(\mathbf{w})=\mathcal{L}(\mathbf{w})+\frac{\lambda }{2}{\mathbf{w}}^{\top }\mathbf{w}=\mathcal{L}(\mathbf{w})+\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2.$$

Notation reconciled with the previous note

The previous note writes the regularization term as $\frac{\lambda }{2n}{\sum }_w{w}^2$, with the $1/n$ factor included so that $\lambda$ stays dataset-size-independent. The analysis in this note works in scaled form: $\mathcal{L}$ here is the (already-averaged) data loss and $\lambda$ here is the rescaled coefficient that absorbs the $1/n$. Either convention gives the same algebraic result; the simpler form $\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2$ is used throughout to keep the formulas readable.

For the same reason, biases are omitted from $\mathbf{w}$ throughout this note. The previous note explains why L2 is conventionally not applied to biases; including them would not change the structure of the analysis but would clutter the notation.

Why this form is also called "weight decay" and "ridge"

The penalty $\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2$ goes by three names across communities:

• L2 regularization (machine learning, deep learning);
• weight decay (deep learning, when emphasizing the update-rule view);
• ridge regression or Tikhonov regularization (statistics, classical inverse problems).

The names refer to the same mathematical object. The latter two reflect the historical origins of the construction in linear regression and operator theory, respectively. The analysis below applies to any twice-differentiable loss; the linear-regression special case appears at the end.

The gradient and the update rule, written compactly

Differentiating the regularized loss gives the gradient

$$\nabla \tilde{\mathcal{L}}(\mathbf{w})=\nabla \mathcal{L}(\mathbf{w})+\lambda \mathbf{w}.$$

A single step of gradient descent therefore reads

$$\mathbf{w}\leftarrow \mathbf{w}-\eta (\nabla \mathcal{L}(\mathbf{w})+\lambda \mathbf{w})=(1-\eta \lambda )\mathbf{w}-\eta \nabla \mathcal{L}(\mathbf{w}).$$

This is the weight-decay form: shrink the weight by the multiplicative factor $(1-\eta \lambda )$, then apply the usual unregularized update. The rule does not push the weights inexorably to zero: the second term, the unregularized gradient, can still drive any individual weight to grow if doing so reduces $\mathcal{L}$ enough.

Per-step vs at-convergence

The update rule above describes what happens during a single step. It says nothing about where the trajectory ultimately lands. The question this note addresses is the fixed point: where does the regularized iterate $\tilde{\mathbf{w}}$ converge to, and how does it relate to the unregularized optimum ${\mathbf{w}}^{\star }$? The answer requires going past the update rule and looking at the shape of $\mathcal{L}$ around ${\mathbf{w}}^{\star }$.

Quadratic approximation around the unregularized optimum

To make the analysis tractable, the unregularized loss is approximated by a second-order Taylor expansion around the unregularized optimum ${\mathbf{w}}^{\star }=\arg {\min }_{\mathbf{w}}\mathcal{L}(\mathbf{w})$:

$$\hat{\mathcal{L}}(\mathbf{w})=\mathcal{L}({\mathbf{w}}^{\star })+\frac{1}{2}(\mathbf{w}-{\mathbf{w}}^{\star }{)}^{\top }\mathbf{H}(\mathbf{w}-{\mathbf{w}}^{\star }),$$

where $\mathbf{H}={\nabla }^2\mathcal{L}({\mathbf{w}}^{\star })$ is the Hessian of $\mathcal{L}$ evaluated at ${\mathbf{w}}^{\star }$.

Why no first-order term

A second-order Taylor expansion at a point ${\mathbf{w}}^{\star }$ is

$$\mathcal{L}(\mathbf{w})\approx \mathcal{L}({\mathbf{w}}^{\star })+\nabla \mathcal{L}({\mathbf{w}}^{\star }{)}^{\top }(\mathbf{w}-{\mathbf{w}}^{\star })+\frac{1}{2}(\mathbf{w}-{\mathbf{w}}^{\star }{)}^{\top }\mathbf{H}(\mathbf{w}-{\mathbf{w}}^{\star }).$$

At a minimum, the gradient vanishes: $\nabla \mathcal{L}({\mathbf{w}}^{\star })=0$. The first-order term therefore drops out, leaving the pure quadratic above.

A second consequence of ${\mathbf{w}}^{\star }$ being a minimum: the Hessian $\mathbf{H}$ is positive semi-definite. All its eigenvalues ${\lambda }_i$ satisfy ${\lambda }_i\ge 0$.

When is the quadratic approximation exact?

For loss functions that are already quadratic in $\mathbf{w}$, no approximation is involved: the second-order Taylor expansion equals the true loss. The most important example is linear regression with squared error, where $\mathcal{L}(\mathbf{w})=\frac{1}{2}\Vert \mathbf{Xw}-\mathbf{y}{\Vert }^2$, and the regularized version becomes exactly ridge regression. The result derived below is therefore an exact statement for ridge regression and a local approximation for any other twice-differentiable loss near its minimum.

The gradient of the quadratic approximation $\hat{\mathcal{L}}$ is

$$\nabla \hat{\mathcal{L}}(\mathbf{w})=\mathbf{H}(\mathbf{w}-{\mathbf{w}}^{\star }).$$

It vanishes precisely at $\mathbf{w}={\mathbf{w}}^{\star }$, recovering the unregularized minimum.

Solving for the regularized optimum

Adding the gradient of the regularization term $\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2$, which is $\lambda \mathbf{w}$, and setting the total gradient to zero, the regularized optimum $\tilde{\mathbf{w}}$ satisfies

$$\lambda \tilde{\mathbf{w}}+\mathbf{H}(\tilde{\mathbf{w}}-{\mathbf{w}}^{\star })=0.$$

Collecting terms,

$$(\mathbf{H}+\lambda \mathbf{I})\tilde{\mathbf{w}}=\mathbf{H}{\mathbf{w}}^{\star },$$

and assuming $\mathbf{H}+\lambda \mathbf{I}$ is invertible (it always is for $\lambda >0$, because the eigenvalues of $\mathbf{H}+\lambda \mathbf{I}$ are ${\lambda }_i+\lambda >0$),

$$\boxed{\tilde{\mathbf{w}}=(\mathbf{H}+\lambda \mathbf{I}{)}^{-1}\mathbf{H}{\mathbf{w}}^{\star }.}$$

This is the closed-form relationship between the regularized and unregularized optima for a quadratic loss. Two sanity checks:

• As $\lambda \to 0$, the regularization vanishes: $(\mathbf{H}+0\cdot \mathbf{I}{)}^{-1}\mathbf{H}={\mathbf{H}}^{-1}\mathbf{H}=\mathbf{I}$, so $\tilde{\mathbf{w}}\to {\mathbf{w}}^{\star }$.
• As $\lambda \to \infty$, the regularization dominates: $(\mathbf{H}+\lambda \mathbf{I}{)}^{-1}\mathbf{H}\approx (\lambda \mathbf{I}{)}^{-1}\mathbf{H}=\mathbf{H}/\lambda \to 0$, so $\tilde{\mathbf{w}}\to 0$.

The interesting regime is in between: what does $(\mathbf{H}+\lambda \mathbf{I}{)}^{-1}\mathbf{H}$ actually do to ${\mathbf{w}}^{\star }$? The answer becomes transparent in the basis of eigenvectors of $\mathbf{H}$.

The geometry: eigendecomposition of the Hessian

Since $\mathbf{H}$ is real and symmetric (it is a Hessian of a real-valued function), the spectral theorem guarantees an orthogonal eigendecomposition

$$\mathbf{H}=\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^{\top },$$

where $\mathbf{\Lambda }=\mathrm{diag}({\lambda }_1,\dots ,{\lambda }_d)$ collects the (non-negative) eigenvalues of $\mathbf{H}$ on its diagonal, and the columns of $\mathbf{Q}$ are the corresponding orthonormal eigenvectors ${\mathbf{q}}_i$. The matrix $\mathbf{Q}$ is therefore orthogonal: ${\mathbf{Q}}^{\top }\mathbf{Q}=\mathbf{Q}{\mathbf{Q}}^{\top }=\mathbf{I}$. The eigenvalues carry a subscript, ${\lambda }_i$, while the single unsubscripted $\lambda$ is always the regularization strength; the entire result will turn on how these two quantities compare.

A geometric reading of the eigenvalues

Each eigenvalue ${\lambda }_i$ of $\mathbf{H}$ measures the curvature of the loss along the corresponding eigenvector direction ${\mathbf{q}}_i$. Substituting $\mathbf{w}-{\mathbf{w}}^{\star }={\mathbf{q}}_i$ (a unit step along eigenvector $i$) into the quadratic approximation,

$$\hat{\mathcal{L}}({\mathbf{w}}^{\star }+{\mathbf{q}}_i)-\mathcal{L}({\mathbf{w}}^{\star })=\frac{1}{2}{\mathbf{q}}_i^{\top }\mathbf{H}{\mathbf{q}}_i=\frac{1}{2}{\lambda }_i.$$

Large ${\lambda }_i$ means a steep direction: a small step in direction ${\mathbf{q}}_i$ increases the loss a lot. Small ${\lambda }_i$ means a flat direction: a step in direction ${\mathbf{q}}_i$ barely changes the loss.

Loss landscapes of trained neural networks typically have a strongly anisotropic spectrum: a few large eigenvalues (a few directions along which the loss curves sharply) and many small eigenvalues (most directions are nearly flat). The flat directions are the ones where the data essentially does not constrain the solution: the loss has many equally good answers.

Substituting the eigendecomposition into the boxed formula,

$$\tilde{\mathbf{w}}=(\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^{\top }+\lambda \mathbf{I}{)}^{-1}\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^{\top }{\mathbf{w}}^{\star }.$$

Since $\lambda \mathbf{I}=\mathbf{Q}(\lambda \mathbf{I}){\mathbf{Q}}^{\top }$ (the identity is unchanged under any orthogonal change of basis),

$$\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^{\top }+\lambda \mathbf{I}=\mathbf{Q}(\mathbf{\Lambda }+\lambda \mathbf{I}){\mathbf{Q}}^{\top }.$$

The inverse of an orthogonal-conjugated diagonal matrix is the orthogonal-conjugated inverse:

$$(\mathbf{Q}(\mathbf{\Lambda }+\lambda \mathbf{I}){\mathbf{Q}}^{\top }{)}^{-1}=\mathbf{Q}(\mathbf{\Lambda }+\lambda \mathbf{I}{)}^{-1}{\mathbf{Q}}^{\top }.$$

Substituting back:

$$\tilde{\mathbf{w}}=\mathbf{Q}(\mathbf{\Lambda }+\lambda \mathbf{I}{)}^{-1}{\mathbf{Q}}^{\top }\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^{\top }{\mathbf{w}}^{\star }.$$

The middle ${\mathbf{Q}}^{\top }\mathbf{Q}$ collapses to $\mathbf{I}$ by orthogonality, leaving the clean expression

$$\boxed{\tilde{\mathbf{w}}=\mathbf{Q}(\mathbf{\Lambda }+\lambda \mathbf{I}{)}^{-1}\mathbf{\Lambda }{\mathbf{Q}}^{\top }{\mathbf{w}}^{\star }.}$$

This is the central result of the section. It says exactly what L2 regularization does to the optimum.

Reading the central result coordinate by coordinate

The matrix $(\mathbf{\Lambda }+\lambda \mathbf{I}{)}^{-1}\mathbf{\Lambda }$ is diagonal, since it is a product of two diagonal matrices. Its $i$-th diagonal entry is

$$\frac{{\lambda }_i}{{\lambda }_i+\lambda }.$$

The matrix ${\mathbf{Q}}^{\top }{\mathbf{w}}^{\star }$ expresses ${\mathbf{w}}^{\star }$ in the eigenbasis of $\mathbf{H}$: the $i$-th component, $({\mathbf{Q}}^{\top }{\mathbf{w}}^{\star }{)}_i$, is the projection of ${\mathbf{w}}^{\star }$ onto the $i$-th eigenvector ${\mathbf{q}}_i$.

Putting everything together, the boxed formula says:

The central insight, in one line

In the eigenbasis of the loss Hessian, L2 regularization rescales the unregularized optimum ${\mathbf{w}}^{\star }$ coordinate by coordinate, with the $i$-th coordinate multiplied by $\frac{{\lambda }_i}{{\lambda }_i+\lambda }$.

The shrinkage factor depends only on the ratio ${\lambda }_i/\lambda$ between the local curvature in that direction and the regularization strength:

• ${\lambda }_i\gg \lambda$ (steep direction): $\frac{{\lambda }_i}{{\lambda }_i+\lambda }\approx 1$. The coordinate is essentially preserved. The loss strongly opposes any shrinkage, so the regularization has little effect.
• ${\lambda }_i\ll \lambda$ (flat direction): $\frac{{\lambda }_i}{{\lambda }_i+\lambda }\approx 0$. The coordinate is shrunk almost to zero. The loss does not care about this direction, so the regularization wins.
• ${\lambda }_i\sim \lambda$: intermediate shrinkage by a factor near $1/2$.

This is anisotropic shrinkage, and it is the key to understanding why L2 regularization works.

Why this is the deep insight

A naive view of L2 regularization holds that it just “makes the weights smaller”. The boxed formula refines this view dramatically: L2 makes some weights smaller, but not uniformly, and the directions in which it shrinks aggressively are exactly the directions in which the data does not care about the answer.

A direction along which the loss is flat is a direction in which the training data leaves the model parameters underdetermined. The choice along such a direction is essentially arbitrary, and an unregularized optimizer may pick a large value for purely numerical reasons. L2 regularization breaks this indeterminacy in favour of zero, the simplest and most generalizable choice. This is the structural reason L2 improves generalization: it removes degrees of freedom the data did not constrain.

The shrinkage factors add up to something meaningful

The per-direction factors ${\lambda }_i/({\lambda }_i+\lambda )$ do more than describe the solution one coordinate at a time; their sum is itself an interpretable number,

$$\gamma =\sum _{i=1}^d\frac{{\lambda }_i}{{\lambda }_i+\lambda }=\mathrm{tr}[(\mathbf{H}+\lambda \mathbf{I}{)}^{-1}\mathbf{H}].$$

Each direction contributes a value between $0$ and $1$, near $1$ where the data fixes the coefficient and near $0$ where it does not, so $\gamma$ counts the effective number of parameters the regularized model actually uses. At $\lambda =0$ it equals $d$, the full parameter count; as $\lambda$ grows it falls steadily toward zero. In ridge regression $\gamma$ is exactly the trace of the smoother (hat) matrix, the quantity classical statistics calls the effective degrees of freedom of the fit: a single dial that reads how much model the data is really paying for.

Worked example: linear regression and ridge regression

The cleanest place to see the result in closed form is linear regression with squared error, which is exactly quadratic. The unregularized loss is

$$\mathcal{L}(\mathbf{w})=\frac{1}{2}\Vert \mathbf{Xw}-\mathbf{y}{\Vert }^2,$$

with $\mathbf{X}\in {\mathbb{R}}^{n\times d}$ the design matrix (rows are training examples) and $\mathbf{y}\in {\mathbb{R}}^n$ the targets. Direct differentiation gives the gradient $\nabla \mathcal{L}(\mathbf{w})={\mathbf{X}}^{\top }(\mathbf{Xw}-\mathbf{y})$ and the Hessian

$$\mathbf{H}={\mathbf{X}}^{\top }\mathbf{X}.$$

The unregularized optimum, when ${\mathbf{X}}^{\top }\mathbf{X}$ is invertible, is the familiar least-squares solution

$${\mathbf{w}}^{\star }=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}.$$

Adding L2 and applying the boxed formula gives ridge regression:

$$\tilde{\mathbf{w}}=({\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}.$$

The Hessian’s eigenvalues are the squared singular values of $\mathbf{X}$. The eigenvectors ${\mathbf{q}}_i$ are the right-singular vectors of $\mathbf{X}$, which are exactly the principal-component directions of the input data: ${\mathbf{q}}_1$ is the direction of largest variance in the inputs, ${\mathbf{q}}_2$ the second-largest, and so on.

L2 = shrinkage toward zero along the least informative principal components

In the linear-regression case, the central insight reads:

• The first principal components of $\mathbf{X}$ (large singular values, large ${\lambda }_i={\sigma }_i^2$) carry most of the variance in the inputs. The coefficients of ${\mathbf{w}}^{\star }$ along these directions are well-determined by the data and are barely shrunk by L2.
• The last principal components of $\mathbf{X}$ (small singular values, small ${\lambda }_i$) correspond to directions in input space along which the data has almost no spread. The coefficients of ${\mathbf{w}}^{\star }$ along these directions are poorly identified: small changes in the training data could flip them dramatically. These are exactly the coefficients that L2 shrinks aggressively toward zero.

Ridge regression is, in this sense, a soft principal-component regression: it does not hard-truncate the small-variance directions, it smoothly downweights them. The same logic governs L2 regularization in the more general nonlinear setting, where “principal components of the inputs” is replaced by “eigenvectors of the local loss Hessian”.

The feature-level reading: L2 as inflated input variance

The same result has a more elementary reading, one coordinate at a time. When the inputs are centred, $\frac{1}{n}{\mathbf{X}}^{\top }\mathbf{X}$ is the input covariance matrix, and its diagonal holds the variance of each feature, that is, of each column of $\mathbf{X}$ (recall that the rows of $\mathbf{X}$ are the $n$ training examples and its columns are the $d$ input features). Passing from $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ to $({\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I}{)}^{-1}$ adds $\lambda$ to every diagonal entry, so the estimator behaves as if each feature carried $\lambda$ more variance than it actually does. When the features are roughly uncorrelated (so ${\mathbf{X}}^{\top }\mathbf{X}$ is nearly diagonal) the effect is transparent: the weight on feature $j$ is scaled by ${\sigma }_j^2/({\sigma }_j^2+\lambda )$, with ${\sigma }_j^2$ that feature’s variance. A feature with large spread across the data keeps its weight; a feature whose values barely move has its weight pulled toward zero, because the inflated $\lambda$ now dominates its real variance. This is the per-feature shadow of the eigenvector story above: “eigenvectors of the Hessian” collapse to “individual input features” exactly when those features stop being correlated.

A visual picture

The same conclusion has a clean two-dimensional picture, shown below (the geometry behind Goodfellow, Bengio and Courville’s Figure 7.1). Take a weight space with only two coordinates, $w_1$ and $w_2$.

• The contours of the unregularized loss $\mathcal{L}$ form an ellipse centred at ${\mathbf{w}}^{\star }$, with axes aligned with the eigenvectors of $\mathbf{H}$. The eccentricity of the ellipse measures the anisotropy: a long axis (small ${\lambda }_i$) is a flat direction; a short axis (large ${\lambda }_i$) is a steep direction.
• The contours of the regularization penalty $\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2$ form circles centred at the origin.
• The regularized optimum $\tilde{\mathbf{w}}$ sits at the point of tangency of these two families, where the gradients of data loss and penalty balance.

Reading the figure

Follow $\tilde{\mathbf{w}}$ relative to ${\mathbf{w}}^{\star }$. Along the short axis of the ellipse (the steep, large-${\lambda }_i$ direction), pulling the solution toward the origin means climbing the loss quickly, so the tangency point stays much closer to ${\mathbf{w}}^{\star }$ and that coordinate is largely preserved. Along the long axis (the flat, small-${\lambda }_i$ direction), moving toward the origin costs almost no loss, so the tangency point slides far inward toward the origin and that coordinate is heavily shrunk. The regularized optimum $\tilde{\mathbf{w}}$ is dragged toward the origin, but mostly along the direction the loss does not defend.

The picture is the geometric content of the algebraic formula: L2 prefers to move along whichever direction the loss does not strongly defend.

The same shrinkage hides inside early stopping

Anisotropic shrinkage does not require an explicit penalty at all: plain gradient descent, halted early, produces almost the same effect for free. Starting from $\mathbf{w}=0$ and running $\tau$ steps on the quadratic loss, the $i$-th eigendirection of ${\mathbf{w}}^{\star }$ is reached only up to a factor

$$1-(1-\eta {\lambda }_i{)}^{\tau },$$

close to $1$ along steep directions (large ${\lambda }_i$, learned within a few steps) and close to $0$ along flat ones (small ${\lambda }_i$, learned only very slowly). That is the same profile as L2’s ${\lambda }_i/({\lambda }_i+\lambda )$: keep the steep directions, suppress the flat ones. Matching the two factors identifies $\lambda \approx 1/(\eta \tau )$, so fewer training steps act like a stronger penalty. Early stopping is, to second order, implicit weight decay, the optimization-dynamics counterpart of the static result derived here, and both improve generalization for the same reason: they decline to fit the directions the data does not pin down.

Connection to the rest of this site

The eigendecomposition view of L2 ties into several other notes:

• Random-matrix and initialization: the Xavier and He initialization note discusses the spectral radius of the recurrent / weight Jacobians under random initialization. The same eigenvalue language used there governs the analysis in this note, with $\mathbf{H}$ replacing the recurrent weight matrix.
• Gradient flow and skip connections: the Skip connections note showed that ResNets create an identity-plus-perturbation Jacobian $\mathbf{I}+\cdot$ during backpropagation. The matrix $\mathbf{H}+\lambda \mathbf{I}$ in this note has exactly the same shape: a base operator plus a scaled identity that shifts the spectrum away from zero, regularizing whatever pathology the base operator has.
• Decoupled weight decay and adaptive optimizers: the central result of this note is the geometric content of “what weight decay does to the optimum”. The AdamW note shows that this geometric content is preserved when Adam is paired with decoupled weight decay, and distorted when L2 is naively added to Adam’s gradient. The eigendecomposition view explains exactly what is being preserved or distorted: in AdamW the anisotropic shrinkage ${\lambda }_i/({\lambda }_i+\lambda )$ along each eigendirection of the loss Hessian is intact; in coupled Adam + L2, Adam’s adaptive denominator multiplies that factor by an arbitrary, history-dependent rescaling that has no statistical justification.
• The cost of inferring directions the data does not constrain: the closely related view of L2 as MAP estimation under a Gaussian prior on $\mathbf{w}$ pinpoints the same insight from a Bayesian angle. Adding $\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2$ to the negative log-likelihood is equivalent to multiplying the likelihood by a Gaussian prior $p(\mathbf{w})\propto \exp (-\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }_2^2)$; the regularized optimum is the maximum a posteriori (MAP) estimate. The prior pulls the estimate toward zero by exactly the factor ${\lambda }_i/({\lambda }_i+\lambda )$ in each eigendirection, with the same interpretation: directions the likelihood barely constrains are dominated by the prior.

Take-aways

What this analysis adds to the previous note

The previous note showed that L2 produces weight decay during training. This note shows what weight decay accomplishes when training converges:

1. L2 does not shrink all weights uniformly. It shrinks them anisotropically, with the shrinkage strength along each direction set by the ratio of the regularization coefficient $\lambda$ to the local curvature ${\lambda }_i$ of the loss.
2. Steep directions are preserved. The coefficients of ${\mathbf{w}}^{\star }$ along directions in which the data strongly constrains the model survive nearly intact under L2.
3. Flat directions are zeroed. The coefficients of ${\mathbf{w}}^{\star }$ along directions in which the data is essentially silent are shrunk toward zero, removing degrees of freedom the data did not pin down.
4. In the linear-regression special case, this is ridge regression, and the eigenvectors of $\mathbf{H}$ are the principal components of the inputs: L2 is a soft principal-component shrinkage.
5. The same intuition transfers to deep nonlinear networks locally around any minimum: L2 removes the indeterminacy in the parameter directions the data does not constrain, producing solutions that depend less on the idiosyncrasies of the training set.

The complementary mechanism of Dropout achieves a similar generalization benefit by a completely different route (injecting noise into activations rather than penalizing weights). The two are usually combined in practice when both are useful, exactly because they regularize different aspects of the model.