Forget Gate

The forget gate is the first of the three gates that act on the cell state. It produces a vector ${\mathbf{f}}_t\in (0,1{)}^{n_{\text{neurons}}}$ that scales the previous cell state ${\mathbf{c}}_{t-1}$ coordinate by coordinate before the new content is added:

$${\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}.$$

This is the only mechanism by which the LSTM can erase information from its memory. Without it, the cell state would be a strictly accumulating buffer; with it, the cell state becomes a finite, refreshable register whose contents the network learns to manage.

Definition

The forget gate is a single fully connected layer of width $n_{\text{neurons}}$ with sigmoid activation. It consumes the current input ${\mathbf{x}}_t\in {\mathbb{R}}^{n_{\text{inputs}}}$ and the previous hidden state ${\mathbf{h}}_{t-1}\in {\mathbb{R}}^{n_{\text{neurons}}}$, and returns

$${\mathbf{f}}_t=\sigma ({\mathbf{W}}_{xf}{\mathbf{x}}_t+{\mathbf{W}}_{hf}{\mathbf{h}}_{t-1}+{\mathbf{b}}_f)\in (0,1{)}^{n_{\text{neurons}}}.$$

The sigmoid is applied element-wise, so each coordinate of ${\mathbf{f}}_t$ is an independent value in $(0,1)$, and the product ${\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}$ rescales each coordinate of ${\mathbf{c}}_{t-1}$ by its own learned factor.

"Element-wise sigmoid" written out

Let ${\mathbf{z}}_t={\mathbf{W}}_{xf}{\mathbf{x}}_t+{\mathbf{W}}_{hf}{\mathbf{h}}_{t-1}+{\mathbf{b}}_f\in {\mathbb{R}}^{n_{\text{neurons}}}$ denote the pre-activation. The notation $\sigma ({\mathbf{z}}_t)$ means: apply the scalar sigmoid $\sigma (z)=1/(1+e^{-z})$ to each coordinate independently,

$$\sigma ({\mathbf{z}}_t)=(\sigma (z_t^{(1)}),\sigma (z_t^{(2)}),\dots ,\sigma (z_t^{(n_{\text{neurons}})})).$$

This is a vector-valued function of a vector input, $\sigma :{\mathbb{R}}^{n_{\text{neurons}}}\to (0,1{)}^{n_{\text{neurons}}}$, that does not mix coordinates: the output at slot $k$ depends only on the input at slot $k$. The coordinates of ${\mathbf{f}}_t$ are therefore independent of each other given the pre-activation, even though the pre-activation itself is computed from a dense affine map. The same applies to $\tanh (\cdot )$ throughout the LSTM.

Dimensions used in this note

symbol	role	single example	mini-batch ($B$)
${\mathbf{x}}_t,{\mathbf{X}}_t$	current input	$n_{\text{inputs}}$	$n_{\text{inputs}}\times B$
${\mathbf{h}}_{t-1},{\mathbf{H}}_{t-1}$	previous hidden state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{c}}_{t-1},{\mathbf{C}}_{t-1}$	previous cell state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{f}}_t,{\mathbf{F}}_t$	forget gate	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{W}}_{xf}$	input-to-hidden weights	$n_{\text{neurons}}\times n_{\text{inputs}}$	idem
${\mathbf{W}}_{hf}$	hidden-to-hidden weights	$n_{\text{neurons}}\times n_{\text{neurons}}$	idem
${\mathbf{b}}_f$	bias	$n_{\text{neurons}}$	$n_{\text{neurons}}$ (broadcast across columns)

The product ${\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}$ (and ${\mathbf{F}}_t\odot {\mathbf{C}}_{t-1}$ in mini-batch form) acts on operands of identical shape: no broadcasting is involved in the $\odot$.

Pointwise reading of the gate

If the pre-activation vector evaluates to $\mathbf{z}=[-2,0,2]$, the gate fires as

$${\mathbf{f}}_t=\sigma (\mathbf{z})=[0.119,0.500,0.881].$$

Applied component-wise to ${\mathbf{c}}_{t-1}$, this keeps roughly 88% of the third coordinate, halves the second, and erases nearly 90% of the first. The forget gate is not a binary mask; it is a continuous, per-coordinate volume knob.

Why sigmoid, and not anything else

The choice of $\sigma$ is dictated by the role of ${\mathbf{f}}_t$ in the cell state update.

• The output must lie in $[0,1]$, because ${\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}$ is meant to rescale, not to flip sign or amplify. A value of $1$ keeps the corresponding coordinate intact; a value of $0$ erases it; intermediate values smoothly interpolate between the two.
• The function must be smooth and differentiable, so that the gating decision is learnable end-to-end. A hard $\{0,1\}$ mask would block gradient flow.
• The function must be monotone, so that “raising the pre-activation” unambiguously means “keep more”. This makes the optimization landscape interpretable: each component of ${\mathbf{W}}_{xf}$ has a clear sign of effect.

The sigmoid is essentially the unique smooth choice that satisfies all three at once. It also gives the gate a clear interpretation as the Jacobian of the cell-state recurrence: differentiating the cell-state update with respect to ${\mathbf{c}}_{t-1}$ gives exactly $\mathrm{diag}({\mathbf{f}}_t)$, so the forget gate is the recurrent Jacobian (see Cell state and Gradient in LSTM).

The forget gate decides how long memories live

Setting $f_t^{(k)}\approx 1$ at every step keeps coordinate $k$ of the cell state alive indefinitely; the contribution of $c_0^{(k)}$ to $c_T^{(k)}$ persists almost undiminished. Setting $f_t^{(k)}\approx 0$ at a particular step wipes that coordinate in a single step.

Because ${\mathbf{f}}_t$ depends on $({\mathbf{x}}_t,{\mathbf{h}}_{t-1})$, the network can condition the preservation of each memory slot on the current context. A language model can keep “the subject is plural” while reading the verb, then forget it at the end of the clause; a control system can keep “the previous action” until the consequence is observed, then clear it. This is what the vanilla RNN could not do at all: there, preservation was a knife-edge of the spectrum; here, it is a decision the model takes at every step.

A worked example: gender of the subject

Consider a character-level language model reading the sentence “Alice was walking and she was happy.”

While processing the word Alice, the input gate writes into one coordinate of ${\mathbf{c}}_t$ a representation of the grammatical gender of the subject (this is the job of the input gate). Several time steps later, when the model encounters she, that same coordinate is still needed to disambiguate the pronoun. The forget gate is what keeps it alive in the meantime: at every intermediate step, the corresponding component of ${\mathbf{f}}_t$ stays close to $1$, so the gender information passes through the cell-state line essentially unchanged.

Now suppose the sentence ends and a new, unrelated one begins. At the period (or whatever boundary signal is encoded in ${\mathbf{x}}_t$ and ${\mathbf{h}}_{t-1}$), the network can drive the same component of ${\mathbf{f}}_t$ to a value near $0$: the previous subject’s gender is deleted in a single step, freeing the slot for the next sentence’s subject. The cell state has a finite capacity, on the order of $n_{\text{neurons}}$ slots, and the forget gate is what makes that capacity reusable.

Mini-batch form

In practice the cell processes a mini-batch of $B$ sequences in parallel. Input, hidden state, and gate become matrices whose columns are the per-example vectors:

$${\mathbf{X}}_t\in {\mathbb{R}}^{n_{\text{inputs}}\times B},{\mathbf{H}}_{t-1}\in {\mathbb{R}}^{n_{\text{neurons}}\times B},{\mathbf{F}}_t\in {\mathbb{R}}^{n_{\text{neurons}}\times B}.$$

The weights ${\mathbf{W}}_{xf},{\mathbf{W}}_{hf}$ and the bias ${\mathbf{b}}_f$ are shared across all examples in the batch (and, in the unrolled cell, across all time steps), so their shapes do not change. The vectorized forget gate reads

$${\mathbf{F}}_t=\sigma ({\mathbf{W}}_{xf}{\mathbf{X}}_t+{\mathbf{W}}_{hf}{\mathbf{H}}_{t-1}+{\mathbf{b}}_f),$$

with ${\mathbf{b}}_f$ added to every column via broadcasting and $\sigma$ applied element-wise. This is the form actually executed on GPU and the one assumed in all subsequent equations.

On the "concatenated weight matrix" notation

Many references write the gate as ${\mathbf{f}}_t=\sigma ({\mathbf{W}}_f[{\mathbf{x}}_t;{\mathbf{h}}_{t-1}]+{\mathbf{b}}_f)$ with a single matrix ${\mathbf{W}}_f=[{\mathbf{W}}_{xf}{\mathbf{W}}_{hf}]$ acting on the vertical concatenation of ${\mathbf{x}}_t$ and ${\mathbf{h}}_{t-1}$. Block-matrix multiplication makes the two forms identical:

$$\left[\begin{array}{cc}{\mathbf{W}}_{xf}& {\mathbf{W}}_{hf}\end{array}\right]\left[\begin{array}{c}{\mathbf{x}}_t\\ {\mathbf{h}}_{t-1}\end{array}\right]={\mathbf{W}}_{xf}{\mathbf{x}}_t+{\mathbf{W}}_{hf}{\mathbf{h}}_{t-1}.$$

The compact form is convenient on paper and matches the way modern frameworks store the parameters (a single fused weight tensor for all four gates), but it hides the fact that the input and the previous hidden state are conceptually distinct sources of context. The explicit form is used throughout these notes.

A practical detail: initializing the forget bias

A common and consequential trick: initialize ${\mathbf{b}}_f\approx 1$ rather than $0$ (Jozefowicz, Zaremba and Sutskever, 2015). With zero bias, the sigmoid starts at $0.5$, so a freshly-initialized LSTM forgets half of its memory at every step; across, say, twenty steps the residual is $0.5^{20}\approx 10^{-6}$, and the cell state path is effectively dead at initialization. Setting ${\mathbf{b}}_f\approx 1$ shifts the sigmoid to roughly $\sigma (1)\approx 0.73$, so the cell state survives initialization long enough for the gradient signal to teach the gate when it actually needs to forget. This is the kind of detail that decides whether an LSTM trains at all on long sequences.

Saturated gates stop learning

A sigmoid gate has derivative ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$, which collapses toward zero as the pre-activation $z$ grows large in magnitude. A gate pushed deep into saturation, $f_t^{(k)}$ pinned at almost exactly $0$ or $1$, therefore receives almost no gradient and can become stuck: it can no longer learn to change its decision. This is the reason the forget-bias trick targets ${\mathbf{b}}_f\approx 1$, where $\sigma (1)\approx 0.73$ sits on the responsive part of the curve, rather than a large value such as $5$, where $\sigma (5)\approx 0.993$ would begin in saturation and resist ever learning to forget. The same caution applies to all four gates: a healthy LSTM keeps its gates off the flat extremes for most of training, and a coordinate whose gate saturates early effectively freezes that coordinate’s behaviour for the rest of the run.

The next gate, the input gate, decides what is added to the cell state once the forget step has rescaled it.