Stacked RNNs

What this note builds

A single recurrent layer applies one nonlinear transformation per step, so its expressivity per step is capped by that one cell. Stacking lifts the cap by composing $L$ layers vertically: the per-step computation becomes a depth-$L$ feedforward network with stateful inputs. This note develops the construction as a 2-D grid over time and depth, shows why the gradient must now survive both axes (and why plain vanilla cells fail at it), and distils the practical recipe that made stacked LSTMs the workhorse of pre-Transformer speech and NLP: gated cells, two to four layers, between-layer dropout, and residual connections on depth.

Reading the figure

The figure unrolls the stack over time into the $(t,\ell )$ grid: each row is a layer, each column a time step. Two kinds of arrow move information through it: solid horizontal arrows are the recurrence within a layer (the time axis), and dashed vertical arrows lift a layer’s output up to the next layer (the depth axis). The bottom row reads the raw input ${\mathbf{x}}_t$ at each step; every row above reads features of features from the row below, and the top row emits the sequence ${\mathbf{h}}_t^L$ that downstream modules consume. This stacked abstraction is what the slide calls “hierarchical sequential features”, the recurrent analogue of adding hidden layers to an MLP, and the reason stacks are standard in speech and NLP. (Colours: states at the current step $t$ are $\text{green}$, those carried from earlier steps $\text{blue}$.)

A single recurrent layer, whether vanilla RNN, LSTM or GRU, applies the same nonlinear transformation to its input at every time step. The expressive power available per step is fixed by the width and shape of that one cell. Stacking is the natural answer to a simple question: what if more expressivity per step is needed than one cell can deliver?

A stacked RNN answers by composing $L$ recurrent layers vertically. At every time step, the output of layer $\ell$ becomes the input of layer $\ell +1$, while each layer continues to carry its own hidden state forward in time. The result is a 2-D computation grid with two independent axes: time (horizontal) and depth (vertical).

Definition

Let ${\mathbf{h}}_t^{(\ell )}\in {\mathbb{R}}^{n_{\ell }}$ denote the hidden state of layer $\ell$ at time $t$, with $n_{\ell }$ the width of layer $\ell$. The input to layer $\ell$ at time $t$ is the hidden state of the previous layer at the same time:

$${\mathbf{h}}_t^{(0)}={\mathbf{x}}_t,{\mathbf{h}}_t^{(\ell )}=f^{(\ell )}({\mathbf{h}}_t^{(\ell -1)},{\mathbf{h}}_{t-1}^{(\ell )})\text{for }\ell =1,\dots ,L.$$

Here $f^{(\ell )}$ is the recurrent cell of layer $\ell$, with its own parameters ${\mathbf{\theta }}^{(\ell )}$. The cell can be any recurrent unit: vanilla, LSTM, GRU, or a mix. The final output sequence used by any downstream module (loss, attention, prediction head) is conventionally ${\mathbf{h}}_t^{(L)}$, the top-layer hidden state.

Dimensions used in this note

Colours follow the figure: states at the current step $t$ in $\text{green}$, states carried from the previous step $t-1$ in $\text{blue}$.

symbol	role	shape
${\mathbf{x}}_t$	input	$n_{\text{inputs}}$
${\mathbf{h}}_t^{(\ell )}$	hidden state of layer $\ell$ at time $t$	$n_{\ell }$
${\mathbf{c}}_t^{(\ell )}$	cell state of layer $\ell$ at time $t$ (LSTM only)	$n_{\ell }$
${\mathbf{\theta }}^{(\ell )}$	parameters of layer $\ell$‘s cell	depends on cell type

Layer widths $n_{\ell }$ may differ across layers, though in practice they are usually kept equal except possibly for the first layer (which adapts to $n_{\text{inputs}}$).

For an LSTM stack, each layer carries both a hidden state ${\mathbf{h}}_t^{(\ell )}$ and a cell state ${\mathbf{c}}_t^{(\ell )}$. Only the hidden state of layer $\ell$ feeds layer $\ell +1$; the cell state stays inside the layer, exposed only via the read-out ${\mathbf{h}}_t^{(\ell )}={\mathbf{o}}_t^{(\ell )}\odot \tanh ({\mathbf{c}}_t^{(\ell )})$.

Why stack at all

A single recurrent layer with $n_{\text{neurons}}$ units has the expressive power of one nonlinear function of $({\mathbf{x}}_t,{\mathbf{h}}_{t-1})$. No matter how wide $n_{\text{neurons}}$ is made, there is exactly one $\tanh$ (or one sigmoid-tanh-gated combination) between input and output at each step.

Stacking $L$ layers turns the per-step computation into a depth-$L$ feedforward network whose inputs are stateful. The same intuition that motivates deep MLPs over wide single-layer networks transfers directly: features at higher layers can be features of features, and the hierarchy is built once per time step rather than rebuilt at every step. Empirically, this is what makes stacked LSTMs and GRUs work well on tasks like acoustic modelling and machine translation, where the relevant structure at each time step is itself hierarchical (phonemes → words → phrases).

Higher layers are deeper, not slower

A natural misreading of “hierarchy” is that the upper layers run on a coarser timescale, updating less often and capturing longer patterns by construction. They do not. Every layer updates at every time step, consuming the layer below’s output each step, so stacking adds feature abstraction (depth), not temporal resolution. Any tendency of higher layers to track longer-range structure is emergent, not built in. Genuinely multi-timescale recurrence needs explicit machinery (clockwork RNNs, hierarchical multiscale RNNs, or strided/dilated stacking) that updates different layers at different rates on purpose.

Depth and time are orthogonal axes of expressivity

A useful mental model: a stacked RNN’s computation graph is a rectangular grid of cells indexed by $(t,\ell )$. Each cell takes input from its left neighbour (the layer’s own past, ${\mathbf{h}}_{t-1}^{(\ell )}$) and from its bottom neighbour (the previous layer’s current output, ${\mathbf{h}}_t^{(\ell -1)}$). Width adds parallel features at a given $(t,\ell )$; depth adds compositional features across $\ell$; time adds context across $t$.

The three axes do different work, and trading them against each other has different consequences. Adding width helps with representational capacity at a fixed depth; adding depth helps with feature compositionality at a fixed width; adding context window (longer sequences) helps with long-range dependency, but only if the gradient survives the BPTT product.

Gradient flow along two axes

In a single-layer RNN the gradient must survive time (the $t$-direction). In a stacked RNN it must additionally survive depth (the $\ell$-direction). At every position $(t,\ell )$ in the grid the local Jacobian is roughly

$$\frac{\partial {\mathbf{h}}_t^{(\ell )}}{\partial {\mathbf{h}}_{t-1}^{(\ell )}}\text{(temporal)},\frac{\partial {\mathbf{h}}_t^{(\ell )}}{\partial {\mathbf{h}}_t^{(\ell -1)}}\text{(vertical)}.$$

A gradient signal from the loss at time $T$ in the top layer to the input at time $1$ in the bottom layer must traverse a path of length $T+L-2$ in this grid, and each step of the path is a multiplication by one of the two Jacobians. Both axes are subject to the vanishing/exploding pathology of BPTT.

Stacking with vanilla RNNs amplifies the problem

A stack of vanilla RNNs has gradients that vanish or explode along two axes at once. If ${\gamma }_{\text{time}}<1$ is the temporal contraction factor of one layer and ${\gamma }_{\text{depth}}<1$ the vertical contraction across one layer, the gradient magnitude across a path of length $T$ steps and $L$ layers is bounded above by ${\gamma }_{\text{time}}^T\cdot {\gamma }_{\text{depth}}^L$. Doubling the depth has the same effect on the gradient as doubling the time horizon.

This is why stacked vanilla RNNs are essentially never used in practice. They train poorly even at modest depths. The standard recipe is to use LSTM or GRU cells at every layer, which fixes the temporal axis with the gated additive update, and (optionally) to add residual or skip connections between layers, which fixes the vertical axis with the same trick applied to depth.

Practical guidance

A few empirical regularities that have held up across many sequence-modelling benchmarks:

• Use gated cells (LSTM or GRU) at every layer. Vanilla cells stack badly; the same vanishing-gradient logic that rules them out at long horizons rules them out at depth.
• Two to four layers is the sweet spot for most tasks. A single layer is often a strong baseline; three or four layers reliably improves on it; beyond four, returns diminish quickly and training becomes brittle. The deepest stacked LSTMs in production reached eight layers, in speech-recognition systems and in the encoder of Google’s neural translation system, an encoder-decoder, but only with careful initialization and residual connections.
• Layer widths usually match. Choosing $n_{\ell }=n_{\text{neurons}}$ for all $\ell$ simplifies the architecture without measurable cost in most settings.
• Dropout goes between layers, not inside the recurrence. Dropping units across the time axis breaks the very dependencies the recurrence is meant to learn; dropping units between layer $\ell$ and layer $\ell +1$ regularizes without disturbing temporal information flow. This is the recipe of Zaremba, Sutskever and Vinyals (2014).
• Residual or skip connections help past three layers. Adding ${\mathbf{h}}_t^{(\ell )}\leftarrow {\mathbf{h}}_t^{(\ell )}+{\mathbf{h}}_t^{(\ell -1)}$ (assuming matching widths) turns the depth axis into a residual stream, with the same benefits the cell state line provides on the time axis.

Parameter count

Each layer is an independent recurrent cell with its own parameters. For a stack of $L$ identical LSTM layers of width $n_{\text{neurons}}$, with input width $n_{\text{inputs}}$ at the bottom and $n_{\text{neurons}}$ everywhere above,

$$P_{\text{stacked LSTM}}=4(n_{\text{neurons}}{n}_{\text{inputs}}+n_{\text{neurons}}^2+n_{\text{neurons}})+(L-1)\cdot 4(n_{\text{neurons}}^2+n_{\text{neurons}}^2+n_{\text{neurons}}).$$

where:

• the first term is the bottom layer, which sees $n_{\text{inputs}}$-dimensional input;
• the second is the sum across the remaining $L-1$ layers, each of which sees an $n_{\text{neurons}}$-dimensional input from the layer below.

For a stack of GRU layers, multiply the same expression by $3/4$ (see GRU for the parameter accounting per layer).

A note on terminology

The literature uses a few interchangeable terms for the same idea: stacked RNN, multilayer RNN, deep RNN. They all mean the construction described above, with one set of independent parameters per layer. The term “deep RNN” is sometimes also used in a narrower sense for architectures that add depth inside the per-step transition function (e.g. Pascanu, Gulcehre, Cho and Bengio, 2014), but this is uncommon today. In modern usage, “depth” in a recurrent model means vertical stacking.

The trick of stacking layers is orthogonal to the choice of recurrent cell: vanilla, LSTM, GRU and any other recurrent unit can be stacked in exactly the same way, with the same equations and the same caveats about gradient flow along depth. The same is true of bidirectionality, the topic of the next note: any cell, any stack, can be made bidirectional.