Input Gate

The forget gate decides what to erase from the previous cell state. The input gate decides what to write into the new one. Writing is split into two questions that the LSTM keeps strictly separate:

1. What could be written? A candidate update ${\tilde{\mathbf{c}}}_t\in (-1,1{)}^{n_{\text{neurons}}}$, proposed by a tanh layer.
2. How much of it should actually be written, coordinate by coordinate? A gate ${\mathbf{i}}_t\in (0,1{)}^{n_{\text{neurons}}}$, produced by a sigmoid layer.

The two are combined element-wise into the additive contribution ${\mathbf{i}}_t\odot {\tilde{\mathbf{c}}}_t$, and the cell state is updated as

$${\mathbf{c}}_t=\underset{\text{kept from the past}}{\underbrace{{\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}}}+\underset{\text{added now}}{\underbrace{{\mathbf{i}}_t\odot {\tilde{\mathbf{c}}}_t}}.$$

This separation of what and how much is the architectural device that makes the cell’s memory both bidirectional in sign and selectively addressable. Both properties are unusual, and both are essential.

The candidate update ${\tilde{\mathbf{c}}}_t$

The candidate is computed by a single fully connected layer of width $n_{\text{neurons}}$ with tanh activation:

$${\tilde{\mathbf{c}}}_t=\tanh ({\mathbf{W}}_{xc}{\mathbf{x}}_t+{\mathbf{W}}_{hc}{\mathbf{h}}_{t-1}+{\mathbf{b}}_c)\in (-1,1{)}^{n_{\text{neurons}}}.$$

It depends on the same input pair $({\mathbf{x}}_t,{\mathbf{h}}_{t-1})$ as every other gate in the cell. Its parameters ${\mathbf{W}}_{xc},{\mathbf{W}}_{hc},{\mathbf{b}}_c$ are independent of those of the forget and input gates.

Dimensions used in this note

symbol	role	single example	mini-batch ($B$)
${\mathbf{x}}_t,{\mathbf{X}}_t$	current input	$n_{\text{inputs}}$	$n_{\text{inputs}}\times B$
${\mathbf{h}}_{t-1},{\mathbf{H}}_{t-1}$	previous hidden state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{c}}_{t-1},{\mathbf{C}}_{t-1}$	previous cell state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{c}}_t,{\mathbf{C}}_t$	new cell state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{f}}_t,{\mathbf{F}}_t$	forget gate (from Forget gate)	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{i}}_t,{\mathbf{I}}_t$	input gate	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\tilde{\mathbf{c}}}_t,{\tilde{\mathbf{C}}}_t$	candidate update	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{W}}_{xi},{\mathbf{W}}_{xc}$	input-to-hidden weights	$n_{\text{neurons}}\times n_{\text{inputs}}$	idem
${\mathbf{W}}_{hi},{\mathbf{W}}_{hc}$	hidden-to-hidden weights	$n_{\text{neurons}}\times n_{\text{neurons}}$	idem
${\mathbf{b}}_i,{\mathbf{b}}_c$	biases	$n_{\text{neurons}}$	$n_{\text{neurons}}$ (broadcast across columns)

The element-wise products ${\mathbf{i}}_t\odot {\tilde{\mathbf{c}}}_t$ and ${\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}$ act on operands of identical shape: no broadcasting is involved in the $\odot$.

Why tanh and not sigmoid for the candidate

The forget and input gates use sigmoid because they play the role of multiplicative masks: values in $[0,1]$ act as continuous on/off switches, and there is no situation in which “negative keeping” or “negative writing” would make sense.

The candidate plays a different role. It is added to the cell state, so its sign matters: each coordinate of ${\tilde{\mathbf{c}}}_t$ can either push up or push down the corresponding coordinate of ${\mathbf{c}}_t$. A bounded, zero-centred range $(-1,1)$ is the natural choice. Sigmoid would constrain the candidate to non-negative values, breaking the symmetry and forcing the cell state to drift in one direction; tanh keeps the additive updates centred around zero, which keeps the cell state itself bounded in expectation.

So the role determines the activation: masks use sigmoid, contents use tanh. The same logic governs the choice of tanh in the vanilla RNN recurrence (see the “Why tanh?” callout there).

The input gate ${\mathbf{i}}_t$

The input gate is again a sigmoid layer with the standard structure:

$${\mathbf{i}}_t=\sigma ({\mathbf{W}}_{xi}{\mathbf{x}}_t+{\mathbf{W}}_{hi}{\mathbf{h}}_{t-1}+{\mathbf{b}}_i)\in (0,1{)}^{n_{\text{neurons}}}.$$

Coordinate by coordinate, $i_t^{(k)}$ controls how much of the candidate ${\tilde{c}}_t^{(k)}$ is admitted into the corresponding slot of the cell state. A value near $1$ accepts the candidate in full; a value near $0$ rejects it; intermediate values blend it in partially.

Two questions, two networks, one input

A single network producing values in $(-1,1)$ could in principle express both “what to write” and “how much” in one shot: a small magnitude would mean “write little”, a large one would mean “write a lot”. The LSTM deliberately refuses this conflation.

The reason is that gradient flow distinguishes the two. The candidate ${\tilde{\mathbf{c}}}_t$ is added to ${\mathbf{c}}_t$, so its gradient receives the full upstream signal multiplied by ${\mathbf{i}}_t$; the gate ${\mathbf{i}}_t$ enters multiplicatively, so its gradient is multiplied by ${\tilde{\mathbf{c}}}_t$ instead. Splitting the two networks gives the optimizer two independent levers for the same write operation, and it is consistently easier to train than a single fused layer. Modern attention-based architectures revisit exactly this trick, splitting content from routing through separate projections.

How the two combine

The new contribution to the cell state is the element-wise product

$${\mathbf{i}}_t\odot {\tilde{\mathbf{c}}}_t\in (-1,1{)}^{n_{\text{neurons}}}.$$

It is bounded coordinate-wise: even if every gate fires fully and every candidate saturates, no single update can add more than $1$ in magnitude to any slot of the cell state. The full cell state update,

$${\mathbf{c}}_t={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot {\tilde{\mathbf{c}}}_t,$$

then has a clean interpretation: each slot of memory is independently decayed by the forget gate and incremented by the gated candidate. The two operations commute with the coordinate structure, so the LSTM’s memory is, internally, $n_{\text{neurons}}$ scalar registers operating in parallel, each one with its own learned decay and its own learned write enable.

Each coordinate is an independent learned register

The cell state is not a single distributed representation in the way a vanilla hidden state is. Because the recurrence factorizes across coordinates,

$$c_t^{(k)}=f_t^{(k)}{c}_{t-1}^{(k)}+i_t^{(k)}{\tilde{c}}_t^{(k)},$$

the network can in principle dedicate different slots to different memories with different lifetimes: a slot with $f_t^{(k)}\approx 1$ everywhere becomes a near-permanent register; a slot with $f_t^{(k)}\approx 0$ at every boundary becomes a working-memory scratchpad. The gates that affect slot $k$ at time $t$ still depend on all coordinates of ${\mathbf{x}}_t$ and ${\mathbf{h}}_{t-1}$ (the $k$-th row of the weight matrices is dense), but the update rule applied to each slot is independent of the others.

This is why probing studies on trained LSTMs occasionally find single coordinates of ${\mathbf{c}}_t$ that track interpretable variables: line-position counters, quote-depth, sentence sentiment. The architecture explicitly affords such coordinate-level specialization.

Why the recurrence factorizes coordinate-wise

By the coordinate independence of $\odot$, $(\mathbf{a}\odot \mathbf{b}{)}^{(k)}=a^{(k)}{b}^{(k)}$, so the $k$-th coordinate of $\mathbf{a}\odot \mathbf{b}$ depends only on the $k$-th coordinates of $\mathbf{a}$ and $\mathbf{b}$. Applying this to both terms of the cell state update,

$${\mathbf{c}}_t={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot {\tilde{\mathbf{c}}}_t,$$

gives, component by component,

$$c_t^{(k)}=f_t^{(k)}{c}_{t-1}^{(k)}+i_t^{(k)}{\tilde{c}}_t^{(k)}.$$

Slot $k$ of the new cell state depends on slot $k$ of the previous cell state and on slot $k$ of the candidate; coordinates do not mix. The coupling across coordinates lives entirely upstream, inside the four affine maps that produce ${\mathbf{f}}_t,{\mathbf{i}}_t,{\tilde{\mathbf{c}}}_t,{\mathbf{o}}_t$ from $({\mathbf{x}}_t,{\mathbf{h}}_{t-1})$. Replace the recurrence’s $\odot$ with a general matrix multiplication and the property is lost: this is exactly what a vanilla RNN does, and it is why the vanilla RNN cannot dedicate isolated slots to isolated memories.

A language-model example, continued

Returning to the example from the forget gate note: while reading “Alice was walking”, the input gate writes a new value into the “subject gender” slot of the cell state. Concretely, the candidate coordinate ${\tilde{c}}_t^{(k)}$ might take a value near $+1$ to encode “female subject”, and the input gate coordinate $i_t^{(k)}$ fires near $1$ to admit it. The forget gate has, at the same step, already cleared whatever was previously stored in slot $k$, so the new gender simply replaces the old one. From that point until the gate decides to clear or overwrite slot $k$ again, the value rides the cell-state highway untouched.

The crucial property here is selectivity. At the same time step, the input gate may write into slot $k$ while leaving slot $k^{\prime }$ (say, “current verb tense”) completely alone, because $i_t^{(k^{\prime })}\approx 0$. A vanilla RNN cannot do this: every step overwrites every coordinate of its hidden state in lockstep.

Mini-batch form

For a mini-batch of $B$ examples, vectors become matrices with one column per example. The shared weights are unchanged; the gate equations read

$$\begin{array}{rl}{\tilde{\mathbf{C}}}_t& =\tanh ({\mathbf{W}}_{xc}{\mathbf{X}}_t+{\mathbf{W}}_{hc}{\mathbf{H}}_{t-1}+{\mathbf{b}}_c),\\ {\mathbf{I}}_t& =\sigma ({\mathbf{W}}_{xi}{\mathbf{X}}_t+{\mathbf{W}}_{hi}{\mathbf{H}}_{t-1}+{\mathbf{b}}_i),\end{array}$$

with ${\tilde{\mathbf{C}}}_t,{\mathbf{I}}_t\in {\mathbb{R}}^{n_{\text{neurons}}\times B}$, and ${\mathbf{b}}_c,{\mathbf{b}}_i\in {\mathbb{R}}^{n_{\text{neurons}}}$ broadcasting across columns. As always in the LSTM, the parameters are shared across both the batch dimension and the unrolled time dimension.

The last gate, the output gate, decides what fraction of the newly updated cell state ${\mathbf{c}}_t$ becomes the externally visible hidden state ${\mathbf{h}}_t$.