Step and Exponential decay

Intro

Step Decay and Exponential Decay form two of the foundational paradigms upon which many modern learning rate scheduling policies are built.

Both rely on the principle of a systematic, multiplicative reduction of the learning rate. Their essential difference, however, lies in how this reduction unfolds over time.

• Step Decay enforces the decrease at fixed, discrete intervals, producing a piecewise-constant, “staircase-like” profile.
• Exponential Decay, by contrast, applies the reduction at every epoch, yielding a smooth, monotonically decreasing curve.

These contrasting dynamics embody alternative pathways to the same overarching objective:

Goal

Begin with a high learning rate to foster broad exploration of the parameter space, then gradually temper it to allow fine-grained exploitation and convergence without overshooting, following the principle of the exploration–exploitation trade-off.

---

Step Decay

It is a learning rate scheduling policy based on the training epochs.

Underlying principle

Specifically, it is a strategy that reduces the learning rate $\eta$ by a multiplicative factor $\gamma <1$ every step_size epochs.

From the graph above, it can be observed that:

• The learning rate starts with a base value, for example ${\eta }_0=0.001$.
• After each fixed interval (step_size = 4 epochs, in the figure), the learning rate is multiplied by $\gamma$ (e.g., $\gamma =0.5$).
• This produces a staircase-like profile, where each sharp drop is followed by a plateau of stability.

Formulation	Equation	Variables & Notes
General (compact form)	${\eta }_n={\eta }_0\cdot {\gamma }^{\lfloor \frac{1+n}{r}\rfloor }$	$n\to$ epoch (or iteration) index ${\eta }_0\to$ initial learning rate $\gamma \to$ decay factor ($0<\gamma <1$) (e.g. $\gamma =0.5$ halves the LR) $r\to$ step size (number of epochs between updates) $\lfloor \cdot \rfloor \to$ floor function (applies decay only at multiples of $r$) 👉 Directly links LR to epoch index $n$, capturing both plateaus and drops. Example: if $r=4$, $\gamma =0.5$, and $n=12$, then ${\eta }_{12}={\eta }_0\cdot 0.5^{\lfloor (1+12)/4\rfloor }={\eta }_0\cdot 0.5^3$.
Equivalent step-wise	${\eta }_i={\eta }_0\cdot {\gamma }^i,i=0,1,2,\dots$	$i$ = index of the performed decay step (not epoch number) $\gamma \to$ multiplicative decay factor 👉 More intuitive: tracks how many decay steps have occurred, not every epoch. Example: if $r=4$ and epoch $n=12$, then $i=\lfloor \frac{1+12}{4}\rfloor =3$ → ${\eta }_3={\eta }_0\cdot {\gamma }^3$.

Step-by-step derivation

The step-wise formulation can be derived by unfolding the recursive update:

$$\begin{array}{rlrl}{\eta }_1& =\gamma \cdot {\eta }_0& & \\ {\eta }_2& =\gamma \cdot {\eta }_1& & ={\gamma }^2\cdot {\eta }_0\\ ⋮& ⋮& & ⋮\\ {\eta }_i& =\gamma \cdot {\eta }_{i-1}& & ={\gamma }^i\cdot {\eta }_0\end{array}$$

This shows how the recursive multiplication by $\gamma$ at each step naturally leads to the closed-form expression

$${\eta }_i={\eta }_0\cdot {\gamma }^i.$$

Info

This strategy is often combined with monitoring validation metrics, as in PyTorch’s ReduceLROnPlateau function, which reduces the learning rate only if performance does not improve.

A Note on Formulation Subtlety

It is worth noting that while the formula ${\eta }_n={\eta }_0\cdot {\gamma }^{\lfloor (1+n)/r\rfloor }$ is valid, some popular Deep Learning libraries (such as PyTorch’s StepLR) use a slightly different convention, often equivalent to ${\eta }_n={\eta }_0\cdot {\gamma }^{\lfloor n/r\rfloor }$. The primary difference is the timing of the first decay: the formula presented here applies the first drop at epoch $n=r-1$, whereas the floor(n/r) convention applies it at epoch $n=r$. This distinction is important to keep in mind when translating theoretical models into practical code.

Hyperparameter Tuning Challenge

The choice of step_size ($r$) and gamma ($\gamma$) is a critical hyperparameter tuning task. A step_size that is too small or a gamma value that is too low can cause the learning rate to decay prematurely, potentially halting the learning process before the model converges to an optimal solution.

---

Exponential Decay

It is a learning rate scheduling policy that decreases the learning rate $\eta$ at every epoch, yielding a smoother and more gradual decay compared to the staircase profile of Step Decay.

Underlying principle

The learning rate is multiplied by a factor $\gamma$ ($0<\gamma <1$) at each epoch.

From the graph above, it can be observed that:

• The learning rate starts from a base value, e.g., ${\eta }_0=0.001$.
• At each epoch, the learning rate is multiplied by $\gamma$ (i.e., $L{R}_t=L{R}_{t-1}\cdot \gamma$).
• This produces a smooth, monotonically decreasing curve, unlike the staircase-like profile of Step Decay.

Formulation	Equation	Variables & Notes
General (per-epoch form)	${\eta }_n={\eta }_0\cdot {\gamma }^n$	$n\to$ epoch index $(0,1,2,\cdots )$ ${\eta }_0\to$ initial learning rate $\gamma \to$ decay factor per epoch ($0<\gamma <1$) (controls the speed of the decay) Most common in ML libraries. 👉 At each epoch, the LR is multiplied by $\gamma$. Example: If ${\eta }_0=0.1$, $\gamma =0.95$, then at $n=10$: ${\eta }_{10}=0.1\cdot 0.95^{10}\approx 0.0599$.
Equivalent continuous form	${\eta }_n={\eta }_0\cdot e^{-kn}$	$k\to$ decay rate (a positive constant) $e\to$ Euler’s number ($\approx 2.718$) Equivalent to the discrete one if $\gamma =e^{-k}$ (or $k=-\ln (\gamma )$). 👉 Highlights the connection to natural decay processes. Example: If $\gamma =0.95$, then $k=-\ln (0.95)\approx 0.0513$, and at $n=10$: ${\eta }_{10}=0.1\cdot e^{-0.0513\cdot 10}\approx 0.0599$.

Note

Exponential Decay is particularly useful when a progressive and gradual reduction is preferred, avoiding the abrupt drops typical of Step Decay.

Hyperparameter Tuning Challenge

Tuning the decay factor gamma ($\gamma$) is crucial. An overly aggressive decay (a low gamma value) can diminish the learning rate too quickly, preventing the model from exploring the parameter space effectively in the later stages of training.

---

Implementation with PyTorch

Here is how to implement these schedulers using PyTorch’s torch.optim.lr_scheduler.

Step Decay (StepLR)

This scheduler decays the learning rate of each parameter group by gamma every step_size epochs.

import torch 
import torch.optim as optim 
from torch.optim.lr_scheduler import StepLR 
 
# Dummy model and optimizer 
model = torch.nn.Linear(10, 2) 
optimizer = optim.SGD(model.parameters(), lr=0.1) # Initial LR = 0.1 
 
# Scheduler: Halve the learning rate every 10 epochs 
# gamma = 0.5, step_size = 10 
scheduler = StepLR(optimizer, step_size=10, gamma=0.5) 
 
print(f"Initial LR: {optimizer.param_groups[0]['lr']:.4f}") 
 
# Simulate training loop for 30 epochs 
for epoch in range(1, 31): 
    # Training steps would go here 
    # optimizer.step() 
    
    # Update the learning rate 
    scheduler.step() 
    
    if epoch % 5 == 0: 
        print(f"Epoch {epoch}: Current LR = {optimizer.param_groups[0]['lr']:.4f}") 
        
 
# Expected output: 
# Initial LR: 0.1000 
# Epoch 5: Current LR = 0.1000 
# Epoch 10: Current LR = 0.0500 <- First drop 
# Epoch 15: Current LR = 0.0500 
# Epoch 20: Current LR = 0.0250 <- Second drop 
# Epoch 25: Current LR = 0.0250 
# Epoch 30: Current LR = 0.0125 <- Third drop

Exponential Decay (ExponentialLR)

This scheduler decays the learning rate of each parameter group by gamma at every single epoch.

import torch
import torch.optim as optim
from torch.optim.lr_scheduler import ExponentialLR
 
# Dummy model and optimizer
model = torch.nn.Linear(10, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1) # Initial LR = 0.1
 
# Scheduler: Reduce LR by 10% each epoch (gamma = 0.9)
scheduler = ExponentialLR(optimizer, gamma=0.9)
 
print(f"Initial LR: {optimizer.param_groups[0]['lr']:.4f}")
 
# Simulate training loop for 5 epochs
for epoch in range(1, 6):
    # Training steps...
    # optimizer.step()
 
    # Update the learning rate
    scheduler.step()
    
    print(f"Epoch {epoch}: Current LR = {optimizer.param_groups[0]['lr']:.4f}")
 
# Expected output:
# Initial LR: 0.1000
# Epoch 1: Current LR = 0.0900  (0.1 * 0.9)
# Epoch 2: Current LR = 0.0810  (0.09 * 0.9)
# Epoch 3: Current LR = 0.0729  (0.081 * 0.9)
# Epoch 4: Current LR = 0.0656  (0.0729 * 0.9)
# Epoch 5: Current LR = 0.0590  (0.0656 * 0.9)

---

Step vs Exponential Decay

Aspect	Step Decay	Exponential Decay
Decay Profile	Piecewise-constant, staircase-like decay	Smooth, monotonically decreasing curve
Update Frequency	Applied every step_size epochs	Applied at every epoch
Decay Dynamics	More aggressive at the drop, yet stable between steps	More gentle, ideal for a progressive reduction without abrupt jumps
Key Parameters	step_size, $\gamma$	$\gamma$