Step and Exponential decay

1. Intro

Step Decay and Exponential Decay are the two foundational learning-rate schedules in Deep Learning. Both implement the same broad idea introduced in Learning rate scheduling:

• begin with a learning rate large enough to support meaningful exploration;
• reduce it over time;
• so that late training proceeds with smaller, more controlled updates.

The essential difference is how the reduction is distributed over time:

• Step Decay changes the learning rate at discrete milestones, producing a piecewise-constant staircase profile;
• Exponential Decay reduces it at every step, producing a smooth monotone curve.

Same idea, different cadence

These are not competing mathematical curiosities. They are two answers to a single practical question: should the learning rate decrease abruptly at specific times, or continuously at every step? Both are multiplicative schedules; they differ only in the frequency at which the multiplicative reduction is applied. The smoother alternative that interpolates between the two regimes is Cosine Annealing, treated in its own note.

Scheduler steps are not intrinsically epochs

Throughout this note, all hyperparameters (step_size, $\gamma$, scheduler indices) are expressed in scheduler steps, i.e.\ in units of how often scheduler.step() is called. If step() runs once per epoch, the hyperparameters count epochs; if once per iteration, they count iterations. The general discussion of this convention is in Learning rate scheduling §3; the practical consequence for these two schedules is that stepping per batch while choosing step_size as if it counted epochs is the most common implementation bug.

2. Step Decay

2.1 Definition

Step Decay multiplies the learning rate by a fixed factor $\gamma \in (0,1)$ every $r$ scheduler steps, where $r$ is the interval between two consecutive drops. The result is a staircase profile: the learning rate stays constant on each plateau, then drops abruptly to the next plateau.

2.2 Closed-form schedule

Let ${\eta }_0$ be the initial learning rate, $\gamma \in (0,1)$ the multiplicative decay factor, $r$ the step interval, and $n$ the scheduler step index. The closed form is

$$\boxed{{\eta }_n={\eta }_0{\gamma }^{\lfloor n/r\rfloor }.}$$

The floor expression $\lfloor n/r\rfloor$ counts how many drops have already occurred by step $n$:

• for $n<r$ no drop has happened, the exponent is $0$, and ${\eta }_n={\eta }_0$;
• after the first crossing of a threshold $n=r$, the exponent becomes $1$ and ${\eta }_n=\gamma {\eta }_0$;
• after $k$ thresholds, the exponent is $k$ and ${\eta }_n={\gamma }^k{\eta }_0$.

Equivalent step-index reformulation

Writing $i=\lfloor n/r\rfloor$ for the number of drops so far, the schedule simplifies to the pure geometric sequence

$${\eta }_i={\eta }_0{\gamma }^i,$$

which is immediate from ${\eta }_1=\gamma {\eta }_0$, ${\eta }_2=\gamma {\eta }_1={\gamma }^2{\eta }_0$, …, ${\eta }_i={\gamma }^i{\eta }_0$. The two formulas describe the same schedule at two different levels: the time-indexed form ${\eta }_n={\eta }_0{\gamma }^{\lfloor n/r\rfloor }$ tells the learning rate at every scheduler step $n$, while the drop-indexed form ${\eta }_i={\eta }_0{\gamma }^i$ tells the learning rate after $i$ drops have happened. The floor expression in the first form simply determines when the index $i$ should advance.

2.3 What Step Decay is and is not

Step Decay strengths

Step Decay is often the right choice when:

• a small number of explicit, controlled regime changes is desired;
• prior experiments already suggest good milestone locations (the epoch numbers where the loss curve flattens);
• manual interpretability of the schedule matters;
• a simple baseline is needed before moving to smoother families.

Main weakness: abruptness and milestone sensitivity

At each milestone the learning rate changes discontinuously, and the optimization dynamics can change suddenly. Training therefore becomes sensitive to the exact placement of the milestones: if step_size is too small or $\gamma$ is too low, the learning rate collapses too early and learning slows down prematurely; if step_size is too large, the late-stage exploitation never begins. The right milestones are essentially a hyperparameter that has to be tuned per task.

Step Decay vs ReduceLROnPlateau

Step Decay is time-driven: drops happen at predetermined scheduler-step counts. PyTorch’s ReduceLROnPlateau is metric-driven: drops happen when a monitored quantity (typically validation loss) stops improving for a configured number of epochs. The two are different mechanisms and address different needs; this note covers only the time-driven family.

An indexing-convention subtlety

Different texts and implementations write the Step Decay schedule with slightly different floor expressions, for example

$${\eta }_n={\eta }_0{\gamma }^{\lfloor n/r\rfloor }\text{or}{\eta }_n={\eta }_0{\gamma }^{\lfloor (1+n)/r\rfloor }.$$

The difference is not conceptual: it concerns when the first drop is considered to happen. The left form keeps ${\eta }_0$ for $n\in \{0,1,\dots ,r-1\}$ and drops at $n=r$; the right form keeps ${\eta }_0$ only at $n=0$ and drops at $n=r-1$. Neither is “more correct”; they encode different choices of when the schedule’s first reduction lands.

The practical point: the convention used in the mathematics must match the convention used in the implementation. Two implementations using the two formulas above with the same $r$ produce schedules that are off by exactly one drop event, and the resulting bug looks like a near-miss on the training curve rather than an obvious crash. PyTorch’s StepLR uses the convention ${\eta }_n={\eta }_0{\gamma }^{\lfloor n/r\rfloor }$, the first form above.

3. Exponential Decay

3.1 Definition

Exponential Decay multiplies the learning rate by the same factor $\gamma \in (0,1)$ at every scheduler step. Unlike Step Decay, the reduction happens continuously rather than in discrete plateaus.

3.2 Closed-form schedule

The per-step recursion ${\eta }_n=\gamma {\eta }_{n-1}$ has the immediate closed form

$$\boxed{{\eta }_n={\eta }_0{\gamma }^n.}$$

The same schedule can equivalently be written as a continuous exponential

$${\eta }_n={\eta }_0{e}^{-kn},k=-\ln (\gamma ),\gamma =e^{-k}.$$

The two forms are mathematically identical; the exponential parameterization is occasionally convenient because it makes the time constant $1/k$ explicit (the number of scheduler steps over which the learning rate decays by a factor of $e\approx 2.718$).

3.3 What Exponential Decay is and is not

Exponential Decay strengths

Exponential Decay is often the right choice when:

• a smooth schedule is preferred over discontinuous drops;
• the learning rate should decrease steadily throughout training without explicit milestones;
• a single-parameter decay mechanism (only $\gamma$ to tune) is desired.

Main weakness: less explicit control over regime changes

The reduction is always active, so the schedule may become too conservative if $\gamma$ is chosen too small: the optimizer enters the low-learning-rate exploitation regime gradually but starts giving up step magnitude immediately, with no extended high-LR exploration plateau. Step Decay can express the rule “stay at the high LR for $r$ steps, then drop”; Exponential Decay cannot.

4. The two schedules in one picture

Step Decay and Exponential Decay are closely related. Both are multiplicative; they differ in frequency:

• Step Decay applies the factor $\gamma$ every $r$ steps: the same geometric decay is concentrated into explicit jumps;
• Exponential Decay applies the factor $\gamma$ every step: the same geometric decay is spread continuously over time.

Step Decay as a coarsened Exponential Decay

A useful way to think about the difference: a Step Decay with interval $r$ and factor $\gamma$ has the same per-drop multiplier as an Exponential Decay with factor ${\gamma }^{1/r}$, but the contraction is buffered at the drop boundary instead of being smeared across $r$ steps. Step Decay changes the learning rate through regime shifts; Exponential Decay changes it through small continuous contractions.

5. Tuning $\gamma$ from a target final learning rate

In practice, the most natural way to choose $\gamma$ is not to pick it directly but to specify the desired endpoint of training and solve for $\gamma$.

If the final learning rate ${\eta }_N$ is known in advance, what value of $\gamma$ achieves it?

5.1 Exponential Decay

Training runs for $N$ scheduler steps and the goal is to go from ${\eta }_0$ to ${\eta }_N$. Inverting ${\eta }_N={\eta }_0{\gamma }^N$:

$$\boxed{\gamma ={\left(\frac{{\eta }_N}{{\eta }_0}\right)}^{1/N}.}$$

5.2 Step Decay

If $K$ decay events occur during training, ${\eta }_K={\eta }_0{\gamma }^K$, so

$$\boxed{\gamma ={\left(\frac{{\eta }_K}{{\eta }_0}\right)}^{1/K}.}$$

Here $K$ counts drops, not scheduler steps. If training lasts $T$ scheduler steps with drops every $r$ steps, $K\approx \lfloor T/r\rfloor$.

Target-final-LR thinking

This viewpoint is often more interpretable than tuning $\gamma$ directly by trial and error. It converts scheduler design into four explicit choices:

1. the initial learning rate ${\eta }_0$;
2. the desired final learning rate ${\eta }_N$ (or ${\eta }_K$);
3. the training horizon $N$ (or the number of drops $K$);
4. then solve for $\gamma$.

6. PyTorch

Both schedules live in torch.optim.lr_scheduler and follow the same two PyTorch conventions that apply to every scheduler:

• scheduler.step() must be called after optimizer.step();
• all scheduler hyperparameters are in scheduler-step units (see §1).

6.1 StepLR

StepLR multiplies the learning rate of each parameter group by gamma every step_size scheduler steps.

import torch
import torch.optim as optim
from torch.optim.lr_scheduler import StepLR
 
model = torch.nn.Linear(10, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
 
# Halve the learning rate every 10 scheduler steps
scheduler = StepLR(optimizer, step_size=10, gamma=0.5)
 
for epoch in range(1, 31):
    # ... training loop over mini-batches ...
    optimizer.step()
    scheduler.step()

The resulting schedule:

epoch range	learning rate
1 to 9	$0.1000$
10 to 19	$0.0500$
20 to 29	$0.0250$
$\ge 30$	$0.0125$

6.2 ExponentialLR

ExponentialLR multiplies the learning rate by gamma at every scheduler step.

import torch
import torch.optim as optim
from torch.optim.lr_scheduler import ExponentialLR
 
model = torch.nn.Linear(10, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
 
# Reduce LR by 10% each scheduler step
scheduler = ExponentialLR(optimizer, gamma=0.9)
 
for epoch in range(1, 6):
    # ... training loop over mini-batches ...
    optimizer.step()
    scheduler.step()

The resulting schedule:

epoch	learning rate
1	$0.0900$
2	$0.0810$
3	$0.0729$
4	$0.0656$
5	$0.0590$

Both code blocks are epoch-based only because step() is called once per epoch. Calling it once per batch turns the same schedulers into iteration-based variants, with the hyperparameter values then needing to be expressed in iterations.

7. Comparison and decision

Aspect	Step Decay	Exponential Decay
Decay profile	piecewise-constant staircase	smooth monotone curve
Update frequency	every step_size steps	every step
Optimization dynamics	stable plateaus, abrupt regime changes	gradual continuous contraction
Main hyperparameters	step_size, $\gamma$	$\gamma$
Main strength	explicit milestone-based exploitation control	smooth reduction without discontinuities
Main weakness	sensitivity to milestone placement	no extended high-LR plateau; can be too aggressive if $\gamma$ small

When neither is enough

If both schedules feel too rigid or too sensitive to their hyperparameters, the natural next step is Cosine Annealing, which provides a smooth nonlinear decay with a long high-LR plateau at the start and a slow tail near ${\eta }_{\min }$. For the full decision framework (when to add warm-up, when to consider restarts, when to keep things simple), see Choosing an LR scheduler.

8. Summary

Final takeaway

Step Decay and Exponential Decay embody the same fundamental idea of multiplicative learning-rate reduction. The real choice between them is between discrete milestone-based exploitation (Step Decay) and continuous smooth decay (Exponential Decay). The better schedule is the one whose transition pattern best matches the optimization dynamics of the training problem at hand; both are simple baselines against which any more elaborate scheduler should be compared.