BPTT Problems

The derivation of Backpropagation Through Time shows that the gradients of an unfolded RNN contain products of recurrent Jacobians. Training a vanilla RNN runs into three distinct problems, and it helps to separate them at the outset:

1. The cost of unrolling the full sequence. A full forward and backward pass over $T$ time steps costs $\mathcal{O}(T)$ in both time and memory. This problem is computational, and it is addressed by the chunking strategy of BPTT Variants (truncated BPTT, in its stateful and stateless modes).
2. Exploding gradients. Repeated recurrent multiplication can amplify the backward signal exponentially with the temporal distance.
3. Vanishing gradients. The same repeated multiplication can instead suppress the backward signal exponentially.

This note studies the second and third problems: the stability of the gradient as it flows backward through time. The first problem, scaling, is treated separately in BPTT Variants.

Historical perspective

The vanishing gradient problem was a major obstacle for recurrent models throughout the 1990s. At the time, the architectural mechanisms that eventually mitigated it (gated cells, and later residual or skip connections) had not yet been developed. This is part of why training deep recurrent models was long considered so difficult.

The instability becomes especially visible when the unrolled computation spans many time steps: even $T=50$ or $T=100$ can already make the vanishing gradient problem severe.

---

The Recurrent Jacobian Product

Two factors determine how the magnitude of a backpropagated gradient changes as it moves backward:

• the weight matrices it is multiplied by;
• the activation functions, or more precisely their derivatives.

In a vanilla RNN both factors are applied repeatedly, because the same recurrent transition is traversed once per time step. The critical term, already isolated in the BPTT derivation, is the derivative that connects a later hidden state to an earlier one,

$$\frac{\partial {\mathbf{h}}_{t+1}}{\partial {\mathbf{h}}_k}=\prod _{j=k}^t\frac{\partial {\mathbf{h}}_{j+1}}{\partial {\mathbf{h}}_j}=\frac{\partial {\mathbf{h}}_{t+1}}{\partial {\mathbf{h}}_t}\frac{\partial {\mathbf{h}}_t}{\partial {\mathbf{h}}_{t-1}}\cdots \frac{\partial {\mathbf{h}}_{k+1}}{\partial {\mathbf{h}}_k}.$$

The product from $j=k$ to $j=t$ contains $t-k+1$ factors: the longer the temporal gap, the more Jacobians the gradient must pass through.

The recurrent term, and its MLP analogue

The repeated multiplication by ${\mathbf{W}}_{hh}$ that appears in this product is also what drives the error-signal recursion derived in BPTT. Combining its two equations

$${\mathbf{\delta }}_t^h={\mathbf{W}}_{yh}^{\top }{\mathbf{\delta }}_t^o+{\mathbf{W}}_{hh}^{\top }{\mathbf{\delta }}_{t+1}^a,{\mathbf{\delta }}_t^a={\varphi }_h^{\prime }({\mathbf{a}}_t)\odot {\mathbf{\delta }}_t^h,$$

gives the compact recursion

$$\boxed{{\mathbf{\delta }}_t^a={\varphi }_h^{\prime }({\mathbf{a}}_t)\odot \left({\mathbf{W}}_{yh}^{\top }{\mathbf{\delta }}_t^o+{\mathbf{W}}_{hh}^{\top }{\mathbf{\delta }}_{t+1}^a\right)}.$$

The red term is the recurrent contribution: the same matrix ${\mathbf{W}}_{hh}^{\top }$ is encountered again and again as the error signal moves backward through time. Isolating this temporal path, and dropping the local output contribution for intuition,

$${\mathbf{\delta }}_t^a\approx {\varphi }_h^{\prime }({\mathbf{a}}_t)\odot \left({\mathbf{W}}_{hh}^{\top }{\mathbf{\delta }}_{t+1}^a\right),$$

which is exactly the MLP error-backpropagation equation ${\mathbf{\delta }}^l=(({\mathbf{W}}^{l+1}{)}^{\top }{\mathbf{\delta }}^{l+1})\odot {\sigma }^{\prime }({\mathbf{z}}^l)$, with one crucial difference. In an MLP each layer has its own weight matrix; here the same recurrent matrix is multiplied at every step. Repeated multiplication by one fixed matrix is precisely what makes RNN gradients prone to explode or vanish.

---

The Local Recurrent Jacobian

For a vanilla RNN, the pre-activation and hidden state at time $j+1$ are

$${\mathbf{a}}_{j+1}={\mathbf{W}}_{xh}{\mathbf{x}}_{j+1}+{\mathbf{W}}_{hh}{\mathbf{h}}_j+{\mathbf{b}}_h,{\mathbf{h}}_{j+1}={\varphi }_h({\mathbf{a}}_{j+1}).$$

Since ${\varphi }_h$ is applied element-wise, the Jacobian $\partial {\mathbf{h}}_{j+1}/\partial {\mathbf{a}}_{j+1}$ is diagonal (derived component by component in BPTT). Composing it with the linear step $\partial {\mathbf{a}}_{j+1}/\partial {\mathbf{h}}_j={\mathbf{W}}_{hh}$ via the chain rule gives the local recurrent Jacobian

$${\mathbf{J}}_j:=\frac{\partial {\mathbf{h}}_{j+1}}{\partial {\mathbf{h}}_j}=\mathrm{diag}\left({\varphi }_h^{\prime }({\mathbf{a}}_{j+1})\right){\mathbf{W}}_{hh}.$$

Every step therefore contributes the same two factors: a diagonal matrix carrying the activation derivatives, and the recurrent weight matrix ${\mathbf{W}}_{hh}$. The full product becomes

$$\frac{\partial {\mathbf{h}}_{t+1}}{\partial {\mathbf{h}}_k}=\prod _{j=k}^t{\mathbf{J}}_j=\prod _{j=k}^t\mathrm{diag}\left({\varphi }_h^{\prime }({\mathbf{a}}_{j+1})\right){\mathbf{W}}_{hh}.$$

The magnitude of this product is what decides between exploding and vanishing gradients. Two complementary lenses describe it: an eigenvalue intuition and a rigorous norm bound.

---

How the Product Grows or Shrinks

The single quantity that measures how much ${\mathbf{W}}_{hh}$ can stretch a vector is its spectral norm, the largest singular value

$${\sigma }_{\max }({\mathbf{W}}_{hh})=\underset{\Vert \mathbf{v}{\Vert }_2=1}{\max }\Vert {\mathbf{W}}_{hh}\mathbf{v}{\Vert }_2=\sqrt{{\lambda }_{\max }\left({\mathbf{W}}_{hh}^{\top }{\mathbf{W}}_{hh}\right)}.$$

The first expression reads directly: among all unit-length vectors $\mathbf{v}$, ${\sigma }_{\max }$ is the largest length that ${\mathbf{W}}_{hh}\mathbf{v}$ can reach, i.e., the largest factor by which ${\mathbf{W}}_{hh}$ can stretch any vector. The second expression is the standard identity relating the largest singular value to the eigenvalues of ${\mathbf{W}}_{hh}^{\top }{\mathbf{W}}_{hh}$: it is the square root of the largest such eigenvalue. This is the natural quantity to track, because during BPTT the error signal is multiplied by ${\mathbf{W}}_{hh}$, inside the Jacobian, once per step.

Eigenvalue intuition

Suppose, for intuition, that a local Jacobian ${\mathbf{J}}_j$ is diagonalizable with eigenvalues ${\lambda }_1,\dots ,{\lambda }_n$ and eigenvectors ${\mathbf{v}}_1,\dots ,{\mathbf{v}}_n$. A perturbation along ${\mathbf{v}}_i$ is scaled by ${\lambda }_i$ at each step, so across $r$ recurrent steps it is scaled by ${\lambda }_i^r$. Consequently:

• if the dominant eigenvalue magnitude is below $1$, repeated multiplication suppresses the component and the gradient tends to vanish;
• if it is above $1$, repeated multiplication amplifies the component and the gradient can explode.

This is only intuition: in a real RNN the diagonal factor $\mathrm{diag}({\varphi }_h^{\prime }({\mathbf{a}}_{j+1}))$ depends on the activations, so the product is one of time-varying matrices, not a power of a single fixed matrix.

Norm-based bound

A rigorous bound, valid without diagonalizability or a fixed matrix, comes from matrix norms. The key property is submultiplicativity: for any two matrices, the norm of a product is at most the product of the norms, $\Vert \mathbf{AB}\Vert \le \Vert \mathbf{A}\Vert \Vert \mathbf{B}\Vert$. Applied to the local Jacobian ${\mathbf{J}}_j=\mathrm{diag}({\varphi }_h^{\prime }({\mathbf{a}}_{j+1})){\mathbf{W}}_{hh}$,

$$\Vert {\mathbf{J}}_j\Vert \le \Vert \mathrm{diag}\left({\varphi }_h^{\prime }({\mathbf{a}}_{j+1})\right)\Vert \Vert {\mathbf{W}}_{hh}\Vert ={\gamma }_h{\sigma }_{\max }({\mathbf{W}}_{hh}).$$

The factor $\Vert {\mathbf{W}}_{hh}\Vert ={\sigma }_{\max }({\mathbf{W}}_{hh})$ bounds the recurrent contribution. The other factor is the norm of a diagonal matrix, which equals the largest magnitude among its diagonal entries, because a diagonal matrix simply scales each coordinate independently and the largest stretch is the largest scaling factor. Here those entries are the activation derivatives ${\varphi }_h^{\prime }({\mathbf{a}}_{j+1})$, so their largest possible value is ${\gamma }_h$: namely ${\gamma }_h\le 1$ for $\tanh$, and ${\gamma }_h\le 0.25$ for the logistic sigmoid.

The product over $t-k+1$ steps is bounded by applying submultiplicativity once per factor: the norm of the whole product is at most the product of the per-step norms,

$$\Vert \frac{\partial {\mathbf{h}}_{t+1}}{\partial {\mathbf{h}}_k}\Vert =\Vert \prod _{j=k}^t{\mathbf{J}}_j\Vert \le \prod _{j=k}^t\Vert {\mathbf{J}}_j\Vert \le ({\gamma }_h{\sigma }_{\max }({\mathbf{W}}_{hh}){)}^{t-k+1}.$$

Define the effective per-step scale

$$\gamma ={\gamma }_h{\sigma }_{\max }({\mathbf{W}}_{hh}).$$

If $\gamma <1$, the bound decays exponentially with the temporal gap and the gradient vanishes. If $\gamma >1$, the bound grows exponentially and the gradient can explode. The exponent $t-k+1$ is what turns a small per-step deviation from $1$ into an exponential effect across time.

---

The Two Failure Modes

The same recurrent structure produces opposite pathologies depending on the value of the effective scale $\gamma ={\gamma }_h{\sigma }_{\max }({\mathbf{W}}_{hh})$.

Exploding Gradients

When the recurrent matrix has a large spectral norm, so that the effective scale exceeds $1$,

$$\gamma ={\gamma }_h{\sigma }_{\max }({\mathbf{W}}_{hh})>1⟹\text{exploding gradients},$$

the repeated recurrent multiplication amplifies the backward signal exponentially. The gradient can become extremely large, making the first-order approximation used by gradient descent unreliable and the parameter updates numerically unstable.

Solution: Gradient Clipping

The standard optimizer-level fix is gradient clipping: after backpropagation but before the update, the gradient vector $\mathbf{g}$ is rescaled if its norm exceeds a threshold $M$,

$$\mathbf{g}\leftarrow \frac{M}{\Vert \mathbf{g}{\Vert }_2}\mathbf{g}\text{if }\Vert \mathbf{g}{\Vert }_2>M.$$

Clipping does not remove the recurrent cause of the explosion; it caps the resulting step so it cannot become numerically destructive. The geometric justification (cliffs in the loss surface) and the choice of the threshold $M$ are developed in Gradient Clipping.

Vanishing Gradients

When the effective scale falls below $1$,

$$\gamma ={\gamma }_h{\sigma }_{\max }({\mathbf{W}}_{hh})<1⟹\text{vanishing gradients},$$

the repeated multiplication suppresses the backward signal. Because ${\gamma }_h\le 1$ for $\tanh$, a spectral norm ${\sigma }_{\max }({\mathbf{W}}_{hh})<1$ already guarantees vanishing, and the effect is even stronger when $\tanh$ saturates and ${\varphi }_h^{\prime }({\mathbf{a}}_t)$ drops toward $0$. Early hidden states then receive almost no learning signal from later losses.

Solution: Gated Memory and Attention Mechanism

Vanishing gradients are harder to fix with a post-processing trick, because the signal has already been attenuated before the optimizer sees it. The effective solution is architectural.

LSTMs and GRUs modify the recurrent cell with gates and a memory path that let information and gradients persist across long temporal distances; the structural motivation is developed in the limitations of vanilla RNNs. Later, attention mechanisms provide an even shorter route: instead of forcing all long-range information through every recurrent step, the model combines distant hidden states directly when needed.

Two further remedies are complementary to the two solutions above:

• Careful initialization of ${\mathbf{W}}_{hh}$ keeps ${\sigma }_{\max }({\mathbf{W}}_{hh})$ near $1$ from the start. The variance-preserving schemes of Xavier and He initialization are designed precisely to place the network at the edge between the vanishing and exploding regimes; regularization plays a similar role by indirectly bounding the spectral norm during training.
• Non-saturating activations such as ReLU and its variants (surveyed here) reduce the activation-derivative attenuation. They do not by themselves solve the problem, however, since the recurrent matrix ${\mathbf{W}}_{hh}$ can still amplify or suppress the gradient regardless of the activation.

---

Consequences for Long-Term Dependencies

Vanishing gradients are not exclusive to RNNs; they also occur in deep feedforward networks. RNNs are especially vulnerable because unrolling them through time creates a very deep computational graph with repeated use of the same recurrent transformation, so a per-step deviation of $\gamma$ from $1$ compounds over the whole sequence.

The practical consequence is a difficulty in learning long-range dependencies. Information from the distant past may be present in the hidden state in principle, but the learning signal needed to adjust the parameters that would exploit it becomes too weak to be useful. This single limitation is the central reason vanilla RNNs were eventually displaced by gated cells and, later, by attention.