Output Gate

After the forget gate has scaled the previous memory and the input gate has added new content, the cell state ${\mathbf{c}}_t$ is complete: it holds everything the network has chosen to remember at time $t$. The output gate answers the last remaining question: which parts of that memory should be exposed as the hidden state ${\mathbf{h}}_t$?

This is the only gate whose target is not the cell state itself. Its job is to decide what the cell says to the rest of the world, while the cell state stays free to hold information that may not be needed right now but will be later.

Definition

The output gate is again a sigmoid layer of width $n_{\text{neurons}}$:

$${\mathbf{o}}_t=\sigma ({\mathbf{W}}_{xo}{\mathbf{x}}_t+{\mathbf{W}}_{ho}{\mathbf{h}}_{t-1}+{\mathbf{b}}_o)\in (0,1{)}^{n_{\text{neurons}}}.$$

The hidden state is then read out from the cell state through a tanh and a pointwise product:

$${\mathbf{h}}_t={\mathbf{o}}_t\odot \tanh ({\mathbf{c}}_t)\in (-1,1{)}^{n_{\text{neurons}}}.$$

Dimensions used in this note

symbol	role	single example	mini-batch ($B$)
${\mathbf{x}}_t,{\mathbf{X}}_t$	current input	$n_{\text{inputs}}$	$n_{\text{inputs}}\times B$
${\mathbf{h}}_{t-1},{\mathbf{H}}_{t-1}$	previous hidden state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{c}}_t,{\mathbf{C}}_t$	current cell state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{h}}_t,{\mathbf{H}}_t$	new hidden state	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{o}}_t,{\mathbf{O}}_t$	output gate	$n_{\text{neurons}}$	$n_{\text{neurons}}\times B$
${\mathbf{W}}_{xo}$	input-to-hidden weights	$n_{\text{neurons}}\times n_{\text{inputs}}$	idem
${\mathbf{W}}_{ho}$	hidden-to-hidden weights	$n_{\text{neurons}}\times n_{\text{neurons}}$	idem
${\mathbf{b}}_o$	bias	$n_{\text{neurons}}$	$n_{\text{neurons}}$ (broadcast across columns)

The product ${\mathbf{o}}_t\odot \tanh ({\mathbf{c}}_t)$ acts on operands of identical shape: no broadcasting is involved in the $\odot$.

Two things deserve attention.

• A tanh is applied to ${\mathbf{c}}_t$ before gating. The cell state lives in an unbounded space (its components are sums of bounded increments, but they can accumulate over many steps); the hidden state must be bounded, because it is fed back into the gates at the next step and into any downstream prediction head. The tanh is the bounding step. Importantly, it acts only on the read-out, not on the cell state itself, so the long-term memory remains in its native unbounded range.
• The output gate ${\mathbf{o}}_t$ then selects, coordinate by coordinate, how much of each bounded slot to expose. A value of $1$ exposes the slot in full; a value of $0$ hides it from view.

The Jacobian of the read-out

Differentiating ${\mathbf{h}}_t={\mathbf{o}}_t\odot \tanh ({\mathbf{c}}_t)$ with respect to ${\mathbf{c}}_t$ (with ${\mathbf{o}}_t$ held fixed, since it depends on ${\mathbf{x}}_t$ and ${\mathbf{h}}_{t-1}$ only) gives, component by component,

$$\frac{\partial h_t^{(k)}}{\partial c_t^{(j)}}=o_t^{(k)}\cdot (1-{\tanh }^2{c}_t^{(k)}){\delta }_{jk},$$

using $\frac{d}{dz}\tanh (z)=1-{\tanh }^2(z)$ and the coordinate independence of $\odot$. Stacked into a matrix,

$$\frac{\partial {\mathbf{h}}_t}{\partial {\mathbf{c}}_t}=\mathrm{diag}({\mathbf{o}}_t\odot (1-{\tanh }^2{\mathbf{c}}_t)).$$

This Jacobian is again diagonal: the read-out couples each slot of ${\mathbf{h}}_t$ to the same slot of ${\mathbf{c}}_t$, never to a different one. The coordinate independence that the cell state enjoys (see Cell state and Input gate) is therefore preserved by the read-out as well. Cross-coordinate mixing reappears only in the next step, when ${\mathbf{h}}_t$ is consumed by the four affine maps to produce the next round of gates.

Cell state stores, hidden state speaks

The pair $({\mathbf{c}}_t,{\mathbf{h}}_t)$ implements a clean separation of concerns that no single-state recurrence can express:

• ${\mathbf{c}}_t$ is the archive: large, unbounded, persistent, and not directly observed by downstream layers. The network can use it to hold information that will only become relevant many steps later.
• ${\mathbf{h}}_t$ is the dispatch: a bounded, selected projection of the archive, customized at every step to the current task. It is what the next-step gates see and what any prediction head consumes.

The output gate is the interface between the two. It is what lets the LSTM remember privately and report selectively.

Why this separation matters

A vanilla RNN, and any single-state recurrence, must pack everything into one vector: features it currently needs to output, features it will need later but not now, and features it must condition future gating decisions on. These three roles place incompatible demands on the same representation:

• Outputs benefit from being bounded (to keep prediction heads well-behaved).
• Long-term memory benefits from being unbounded (to let gradient flow without saturation, as derived in Cell state).
• Future gating decisions benefit from being selectively visible, not from seeing everything at once.

The output gate resolves all three. The cell state stays unbounded, so its gradient flows freely; the hidden state is bounded by tanh and selected by ${\mathbf{o}}_t$, so prediction heads and downstream gates see a clean, focused signal; and the network can choose, at every step, to keep some information dark by setting the corresponding component of ${\mathbf{o}}_t$ near zero.

${\mathbf{h}}_t$ is a compressed view of ${\mathbf{c}}_t$

In information-theoretic terms, ${\mathbf{h}}_t={\mathbf{o}}_t\odot \tanh ({\mathbf{c}}_t)$ is a lossy projection of the cell state. Setting some components of ${\mathbf{o}}_t$ to zero discards that information from the externally visible signal, even though it remains intact inside ${\mathbf{c}}_t$. The next step still has access to it through the cell-state line, but no other component of the network does.

This anticipates the design philosophy of modern attention-based architectures: maintain a rich internal state, and emit at each step only the selected summary that downstream computation needs.

A language-model example, continued

Continuing the example from the input gate note: by the time the model has finished reading “Alice was walking and”, the cell state contains, among other things, the gender of the subject in slot $k$. While processing the next, uninformative tokens (“walking”, “and”), that gender information is not yet needed for any prediction; it must be kept alive in ${\mathbf{c}}_t$ but does not need to be exposed in ${\mathbf{h}}_t$.

The output gate accomplishes this by setting $o_t^{(k)}\approx 0$ at those steps: the gender slot stays in the archive, invisible to the next-step gates and to the prediction head. When the model reaches the position immediately before “she”, where the pronoun must be predicted, the output gate fires $o_t^{(k)}\approx 1$ on the same slot: the gender becomes visible to the head, and the correct pronoun is generated. Throughout, the value in slot $k$ of ${\mathbf{c}}_t$ has not changed; only its visibility has.

This is the kind of behaviour that no single-state recurrence can implement: the value and the visibility are entirely separate quantities, controlled by different parameters.

Why a sigmoid, again

The output gate uses the same sigmoid as the forget and input gates, for the same three reasons given in Forget gate: bounded range $[0,1]$, smooth differentiability, and monotonicity. The output gate is a mask, not a content, so a $(0,1)$ output is the right range. Critically, ${\mathbf{o}}_t$ multiplies $\tanh ({\mathbf{c}}_t)$ pointwise, exactly as ${\mathbf{f}}_t$ multiplied ${\mathbf{c}}_{t-1}$: same mathematical structure, different operand, different role.

Mini-batch form

The vectorized form for a mini-batch of $B$ examples is, as always, obtained by replacing the per-example vectors by matrices whose columns are the per-example values:

$${\mathbf{O}}_t=\sigma ({\mathbf{W}}_{xo}{\mathbf{X}}_t+{\mathbf{W}}_{ho}{\mathbf{H}}_{t-1}+{\mathbf{b}}_o),{\mathbf{H}}_t={\mathbf{O}}_t\odot \tanh ({\mathbf{C}}_t),$$

with ${\mathbf{O}}_t,{\mathbf{H}}_t\in {\mathbb{R}}^{n_{\text{neurons}}\times B}$. The bias ${\mathbf{b}}_o$ broadcasts across the $B$ columns of the affine result; the element-wise product ${\mathbf{O}}_t\odot \tanh ({\mathbf{C}}_t)$ acts on matrices of identical shape, with no broadcasting.

All four gate operations are now defined. Putting all together assembles them into the complete forward pass of one LSTM cell.