Attention as a General Aggregation Pattern

The soft-attention pipeline was introduced as a fix for a specific problem in recurrent sequence modelling: how to summarize a variable-length sequence of hidden states into a fixed-shape vector with content-dependent weights. The mechanism itself, a learned weighted combination of a set of feature vectors, is structurally far more general than the use case that motivated it.

This note surveys four representative applications of the same three-step pipeline (score, normalize, combine) outside of recurrent sequence summarization, then states the unifying abstraction.

The pattern

Whenever a model has to combine a set or sequence of feature vectors into a fixed-shape representation, with mixing weights that should depend on the actual content of those vectors and on the downstream task, soft attention is essentially the only option that is differentiable, parameter-efficient, and architecturally agnostic to the size of the set. The applications below differ in what the set is (channels, spatial locations, graph nodes, image regions) and how the score is computed, but the three-step pipeline is unchanged.

Channel attention in CNNs: Squeeze-and-Excitation

The general channel-attention recipe

Every channel-attention module has the same two-step shape: map each channel to a scalar weight, then recalibrate (rescale) the channels by those weights. The published variants differ in just two independent choices.

• How to pool each channel into a scalar: global average pooling alone (the original SENet), or average and max pooling fed through a shared MLP (CBAM’s channel module, detailed later in this note).
• How to normalize the resulting scores: a sigmoid, giving independent gates that need not sum to one, or a softmax, making the channels compete for a fixed budget (the distinction is unpacked in the next callout).

Optionally, once recalibrated the channels can be collapsed by a weighted sum into a single aggregated map. The walkthrough below uses the canonical SENet recipe (average pooling, MLP, sigmoid); the diagram above shows the average-plus-max variant.

A convolutional feature map $\mathbf{F}\in {\mathbb{R}}^{C\times H\times W}$ has $C$ channels, each one a 2-D map of spatial activations. The standard implicit assumption of a convolutional block is that all channels are equally important: the next layer sees the full $\mathbf{F}$ unchanged. Empirically this is a poor assumption. For any given input, some channels carry information highly relevant to the task while others are nearly silent or actively distracting.

Squeeze-and-Excitation (Hu, Shen and Sun, 2018) is the soft-attention pipeline applied to channels rather than to sequence positions. The “set” being attended over is the set of $C$ channels; each channel is summarized into a single scalar by global average pooling, the scalars are passed through a small bottleneck MLP to produce a per-channel weight, and the weights rescale the channels.

The three steps map exactly onto the soft-attention pipeline:

1. Score. Each channel is reduced to a scalar by global average pooling over $H\times W$: $$z_c=\frac{1}{HW}\sum _{h=1}^H\sum _{w=1}^W{F}_{c,h,w},c=1,\dots ,C.$$ The resulting vector $\mathbf{z}\in {\mathbb{R}}^C$ passes through a small 2-layer MLP with bottleneck width $C/r$ (typical reduction ratio $r\in \{8,16\}$), producing per-channel scores.
2. Normalize. The MLP’s output is squashed by a sigmoid, not softmax, giving independent per-channel weights $\mathbf{\alpha }\in (0,1{)}^C$ that do not sum to 1. This is a deliberate departure from the soft-attention default: the channels are not competing for a fixed attention budget, they are independently gated.
3. Combine. Each channel of $\mathbf{F}$ is rescaled by its weight, giving the recalibrated feature map ${\tilde{F}}_{c,h,w}={\alpha }_c\cdot F_{c,h,w}$.

Soft attention with sigmoid instead of softmax

The softmax in the canonical soft-attention pipeline enforces ${\sum }_t{\alpha }_t=1$, which is appropriate when the model has a fixed budget of attention to spend across positions. Channel attention drops this constraint by replacing softmax with sigmoid: each channel decides independently whether to amplify or attenuate, and the network can in principle enhance many channels at once or suppress them all. The resulting operation is best thought of as a learned per-channel gate, structurally analogous to the gates of an LSTM.

Squeeze-and-Excitation became a standard block in image classification CNNs and was a component of the winning entry in the ILSVRC 2017 competition. The cost is small (an additional MLP per block, with parameter count $\mathcal{O}(C^2/r)$); the accuracy gain is consistent across architectures.

---

Spatial attention and CBAM

Squeeze-and-Excitation answered which feature maps matter, collapsing space to score channels. The complementary question is where: within the retained channels, which spatial locations carry the signal? Spatial attention attends over the $H\times W$ grid of positions, producing a 2-D map that rescales the activations location by location.

The Convolutional Block Attention Module (CBAM, Woo et al., 2018) combines the two into one lightweight, architecture-agnostic block, applying channel attention first, then spatial attention.

Reading the overview figure

CBAM is two sequential sub-modules, channel then spatial. The intermediate feature map is multiplied first by the channel map and then by the spatial map, emerging refined, and the block is dropped in at every convolutional block of a deep network. The two stages are complementary by design: channel attention asks what is meaningful (which feature detectors to trust), spatial attention asks where the meaningful content sits.

Formally, for an input feature map $\mathbf{F}\in {\mathbb{R}}^{C\times H\times W}$,

$${\mathbf{F}}^{\prime }={\mathbf{M}}_c(\mathbf{F})\odot \mathbf{F},{\mathbf{F}}^{\prime \prime }={\mathbf{M}}_s({\mathbf{F}}^{\prime })\odot {\mathbf{F}}^{\prime },$$

where ${\mathbf{M}}_c\in {\mathbb{R}}^{C\times 1\times 1}$ is the channel map and ${\mathbf{M}}_s\in {\mathbb{R}}^{1\times H\times W}$ the spatial map. The element-wise product $\odot$ broadcasts the missing dimension: the channel map is copied across every spatial location, the spatial map across every channel. ${\mathbf{F}}^{\prime \prime }$ is the refined output.

The channel sub-module: Squeeze-and-Excitation, refined

CBAM’s channel map is the SE construction with one upgrade. SE squeezed each channel by average pooling alone; CBAM squeezes with both average and max pooling, passes each descriptor through the same shared bottleneck MLP, sums the two, and applies a sigmoid:

$${\mathbf{M}}_c(\mathbf{F})=\sigma (\text{MLP}(\text{AvgPool}(\mathbf{F}))+\text{MLP}(\text{MaxPool}(\mathbf{F})))=\sigma ({\mathbf{W}}_1{\mathbf{W}}_0{\mathbf{F}}_{\text{avg}}^c+{\mathbf{W}}_1{\mathbf{W}}_0{\mathbf{F}}_{\text{max}}^c),$$

with the bottleneck ${\mathbf{W}}_0\in {\mathbb{R}}^{C/r\times C}$ (reduction ratio $r$, ReLU applied after it) and ${\mathbf{W}}_1\in {\mathbb{R}}^{C\times C/r}$, both shared across the two pooled descriptors ${\mathbf{F}}_{\text{avg}}^c,{\mathbf{F}}_{\text{max}}^c\in {\mathbb{R}}^{C\times 1\times 1}$, and $\sigma$ the sigmoid.

Why max-pooling, on top of SE's average

Average pooling reports how broadly a channel fires across the image; max pooling reports its single strongest response. These are different clues: a faint feature spread everywhere and a sharp feature firing in one spot can share an average yet differ wildly in their max. Routing both through the shared MLP lets the channel score use the distinctive-object evidence that pure averaging washes out, and CBAM’s ablations confirm that both together beat either alone. The channel sub-module is, in one line, SE with a parallel max-pool branch.

The spatial sub-module: pool along channels, then convolve

The spatial map is the mirror image of the channel map. Where the channel module pooled over space to score each channel, the spatial module pools over channels to score each location. Average- and max-pooling along the channel axis give two $1\times H\times W$ maps; these are concatenated and passed through a single $7\times 7$ convolution, then a sigmoid:

$${\mathbf{M}}_s(\mathbf{F})=\sigma (f^{7\times 7}([\text{AvgPool}(\mathbf{F});\text{MaxPool}(\mathbf{F})]))=\sigma (f^{7\times 7}([{\mathbf{F}}_{\text{avg}}^s;{\mathbf{F}}_{\text{max}}^s])),$$

where ${\mathbf{F}}_{\text{avg}}^s,{\mathbf{F}}_{\text{max}}^s\in {\mathbb{R}}^{1\times H\times W}$ are the channel-pooled maps and $f^{7\times 7}$ a convolution with a $7\times 7$ kernel. The deliberately large kernel gives each location a wide receptive field, so the map can flag salient regions rather than isolated pixels.

Reading the sub-module figure: a symmetric squeeze

The figure shows both halves. The channel sub-module (top) takes the max-pooled and average-pooled descriptors through a shared network; the spatial sub-module (bottom) takes the analogous two maps, pooled along the channel axis, and forwards them to a convolution. The symmetry is exact: the channel module squeezes space (global pooling over $H\times W$, then an MLP over channels), the spatial module squeezes channels (pooling over $C$, then a convolution over space). Each discards the axis it is not scoring.

Both sub-modules are the soft-attention pipeline with sigmoid gating

Each map is built by the three steps seen throughout this note: pool to a compact descriptor, score it with a small learned network (an MLP for channels, a convolution for space), squash with a sigmoid. As with SE, the sigmoid rather than a softmax makes the weights independent gates in $(0,1)$ rather than a competitive budget summing to one: any channel or location can be amplified or suppressed on its own. CBAM is two such gates stacked, one over the channel set and one over the spatial set, which is why it slots into the same aggregation story as every other example here.

CBAM is cheap and backbone-agnostic, so it drops into existing architectures with no structural surgery. In a residual network it sits on the convolution output inside each block, before the skip connection rejoins:

Placement and ordering inside a network

The figure shows the exact position: CBAM refines the convolution output, $\mathbf{F}\to {\mathbf{F}}^{\prime }\to {\mathbf{F}}^{\prime \prime }$, before the residual addition $\oplus$, so the identity skip path is left untouched. Two further choices were settled empirically: the sub-modules help more in sequence than in parallel, and channel-first (channel then spatial, as written above) slightly beats spatial-first. The gains are consistent across architectures and across both classification and detection, at negligible parameter cost.

---

Image-captioning attention: “Show, Attend and Tell”

The first major application of soft attention outside of recurrent sequence summarization, and arguably the work that introduced the mechanism to a wide audience, is Show, Attend and Tell (Xu et al., 2015). The task is image captioning: given an image, generate a natural-language description, one word at a time.

The pipeline has four stages:

• 1. An input image.
• 2. A CNN backbone acting as the encoder extracts a convolutional feature map; Xu et al. used VGGNet, but any backbone works, since the attention mechanism is indifferent to how the grid was produced.
• 3. An LSTM decoder generates the caption word by word, attending to that grid at every step.
• 4. The caption is emitted token by token.

The link between stage 2 and the mechanism is the whole point. At its last spatial layer the backbone produces a $14\times 14$ grid of $512$-dimensional vectors. Flattening that grid gives $L=196$ location vectors $\{{\mathbf{a}}_1,\dots ,{\mathbf{a}}_L\}$, each ${\mathbf{a}}_i\in {\mathbb{R}}^D$ with $D=512$ and $L=H\times W$. This is precisely the set the attention aggregates over: not a sequence of words, but a grid of image regions.

At each decoding step $t$, the decoder produces a query (its current hidden state ${\mathbf{h}}_t^{\text{dec}}$) and uses soft attention to select which spatial regions to look at when generating the next word:

$$e_{t,i}=f_{\text{att}}({\mathbf{h}}_t^{\text{dec}},{\mathbf{a}}_i),{\alpha }_{t,i}=\frac{\exp (e_{t,i})}{{\sum }_{j=1}^L\exp (e_{t,j})},{\mathbf{c}}_t=\sum _{i=1}^L{\alpha }_{t,i}{\mathbf{a}}_i.$$

Here $f_{\text{att}}$ is the attention scoring function (the alignment model): a small learned network that takes the decoder’s query ${\mathbf{h}}_t^{\text{dec}}$ together with one location vector ${\mathbf{a}}_i$ and returns a scalar compatibility score $e_{t,i}$, large when region $i$ is relevant to the word about to be generated. It is the query-conditional generalization of the query-free scorer $s(\mathbf{h})$ from the soft-attention note: the score now depends on both the query and the candidate, not on the candidate alone. In Show, Attend and Tell $f_{\text{att}}$ is a one-hidden-layer MLP, the additive (Bahdanau-style) form of the score, developed in the additive-attention note, where the query / key / value vocabulary is also pinned down.

The context vector ${\mathbf{c}}_t$ is concatenated with the previous word’s embedding and fed into the LSTM decoder for step $t$, and the next word is produced. Because the attention distribution ${\mathbf{\alpha }}_t$ lives over the $L$ image regions rather than over abstract positions, it can be painted back onto the image as a heatmap: the bright regions show where the model is looking while it generates that particular word.

Reading the figure: attention tracks the word

Each panel overlays the attention map ${\mathbf{\alpha }}_t$ for one generated word of “A woman is throwing a frisbee in a park”. The mass moves with the content being described: it lands on the disk when the model emits “frisbee”, on the person for “woman”, and spreads across the background for “park”. This is the spatial counterpart of the token-level salience heatmap from the sentiment example, with image regions in place of sequence positions.

The pitfall: these smooth blobs are an upsampling artefact

The organic, almost pixel-precise look of the highlights is misleading. The attention distribution has only $L=196$ entries, one per cell of the coarse $14\times 14$ grid. The smooth, finely-curved heatmaps are produced by interpolating that tiny $14\times 14$ map up to full image resolution purely for display.

The model does not localise at pixel granularity: it distinguishes $196$ regions, and the apparent crispness is a rendering choice, not a property of the attention. Reading more spatial precision into the picture than $196$ cells can carry is a common misinterpretation.

The behaviour is consistent across images: the highlighted region tracks the underlined content word, whether it is a frisbee, a dog, or a stop sign.

Compelling heatmaps are not certified explanations

These visualizations are the single most cited piece of evidence for the slogan “attention is interpretable”, and they are genuinely informative: they show where the context vector ${\mathbf{c}}_t$ draws its mass. But to read the map as the model’s reason for a word is to treat it as a post hoc explanation (Latin, “after this”): a story told about the output after it has been produced, by inspecting the model from the outside, rather than a mechanism guaranteed faithful by the way the model actually computes. Like a saliency map, it reveals what the model attended to, not why it emitted a given word, and a map that looks convincing can still be unfaithful to the real decision. The same caution raised for token-level attention (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019) applies here; the interpretability of attention and its limits are taken up in the explainability note.

The architectural shape that became Transformers

The construction above is query-conditional soft attention: the score $e_{t,i}$ depends on both the query ${\mathbf{h}}_t^{\text{dec}}$ and the key ${\mathbf{a}}_i$. This is the architectural shape that was eventually generalized into the modern Transformer attention block, with three crucial additions:

• the query, key, and value are produced by separate learned projections of the input;
• the score is computed by scaled dot product rather than by a learned MLP;
• the mechanism is self-attention (the queries come from the same sequence as the keys and values) rather than encoder-decoder cross-attention.

The “Show, Attend and Tell” model is, in retrospect, the immediate ancestor of the cross-attention block in encoder-decoder Transformers, three years before “Attention is all you need”. The soft-attention pipeline introduced in the previous note is the same idea stripped of the query.

---

Graph attention networks (GAT)

In a graph neural network, each node $v$ aggregates information from its neighbors $\mathcal{N}(v)$ to compute its updated representation. The original Graph Convolutional Network (Kipf and Welling, 2017) uses a uniform average over the neighbors, weighted by the inverse square root of the node degrees. This is the graph analogue of the rejected averaging fix in sentiment analysis: every neighbor contributes equally, regardless of how informative it is for the task.

The Graph Attention Network (Veličković et al., 2018) replaces the uniform average with soft attention over neighbors:

$$e_{vu}=\text{LeakyReLU}({\mathbf{w}}^{\top }[\mathbf{W}{\mathbf{h}}_v;\mathbf{W}{\mathbf{h}}_u]),{\alpha }_{vu}=\frac{\exp (e_{vu})}{{\sum }_{u^{\prime }\in \mathcal{N}(v)}\exp (e_{v{u}^{\prime }})},{\mathbf{h}}_v^{\prime }=\sigma \left(\sum _{u\in \mathcal{N}(v)}{\alpha }_{vu}\mathbf{W}{\mathbf{h}}_u\right).$$

The set being attended over is $\mathcal{N}(v)$, the neighborhood of node $v$.

• the score $e_{vu}$ depends on both the query node $v$ and the neighbor $u$;
• the softmax normalizes over neighbors;
• the context ${\mathbf{h}}_v^{\prime }$ is a learned weighted aggregation that downweights uninformative neighbors.

GAT consistently outperforms its uniform-averaging counterpart on node classification benchmarks, particularly on graphs where neighborhoods are large and heterogeneous.

The same pattern reappears in many subsequent graph models. As in the image case, the attention map over neighbors is directly interpretable as a “where is this node looking” diagnostic.

---

A handful of other settings

A few additional applications, mentioned briefly to indicate the breadth of the pattern.

• Pointer Networks (Vinyals, Fortunato and Jaitly, 2015) use attention scores as outputs: the model “points” to a position in the input by emitting its attention distribution, rather than emitting a token from a fixed vocabulary. This is the natural construction for tasks where the output is a permutation or selection over the input (e.g., the travelling-salesman problem, convex-hull computation, extractive summarization).
• Memory Networks (Sukhbaatar, Szlam, Weston and Fergus, 2015) maintain an external memory of fixed slots and use soft attention to read from it: the query is the current state of the controller, the keys are the memory slot identifiers, the values are the contents. This is essentially the construction the modern Transformer’s key-value attention generalizes.
• Neural Turing Machines (Graves, Wayne and Danihelka, 2014) preceded memory networks and used soft attention to both read from and write to a differentiable external memory. The construction is mathematically elaborate but conceptually the same three-step pipeline.
• Multi-instance learning uses attention to aggregate predictions across the instances of a bag in weakly-supervised settings, with the attention weights identifying which instances drove the bag-level prediction (Ilse, Tomczak and Welling, 2018).

---

The unifying abstraction

The soft-attention building block

Soft attention is the differentiable, learned, content-dependent generalization of “aggregate a set of feature vectors into a single one”. Any time a model faces such an aggregation problem, soft attention is the architectural default, parameterized by:

• the set being aggregated (sequence positions, channels, spatial locations, neighbours, memory slots);
• the score function (linear projection, MLP, dot product, learned similarity, possibly query-conditional);
• the normalization (softmax for competitive allocation, sigmoid for independent gating);
• the combination (weighted sum, weighted concatenation, gated multiplication).

Modern attention-based architectures are, in essence, soft attention applied recursively and at scale: many heads, many layers, query-conditional scoring, learned projections for keys and values. None of these refinements changes the underlying logic introduced in the previous note; they sharpen and parallelize it.

Attention is a learned kernel smoother

Stripped to its arithmetic, the pipeline is a classical object. With scores $s_t$ and weights ${\alpha }_t\propto \exp (s_t)$, the context vector $\mathbf{c}={\sum }_t{\alpha }_t{\mathbf{h}}_t$ is a kernel-weighted average of the ${\mathbf{h}}_t$, the softmax acting as a similarity kernel. This is exactly Nadaraya-Watson kernel regression (1964), the textbook non-parametric estimator that predicts a target as a similarity-weighted average of stored outputs.

Query-conditional attention, ${\alpha }_t\propto \exp (\mathrm{sim}(\mathbf{q},{\mathbf{k}}_t))$, is Nadaraya-Watson with a learned, query-dependent kernel and learned values.

This reading explains why attention generalises so freely across the settings above: kernel smoothing is defined for any set of points carrying a notion of similarity, which is precisely why the same three steps work over tokens, channels, pixels, and graph nodes. The modern contribution is not the averaging, which is sixty years old, but making the kernel and the values learned and differentiable.

What comes next

The soft-attention pipeline has now been seen as both a fix for a specific failure mode of recurrent sequence summarization and as a generic aggregation primitive that recurs across vision, graphs, and external memory. The natural next question is what happens when the architectural reliance on a recurrent encoder is dropped entirely: when the only mechanism the model uses to mix information across positions is attention. The answer is the Transformer, and it is the subject of the next module.