DenseNet

Abstract

DenseNet (Huang, Liu, van der Maaten, and Weinberger, 2017) takes the residual idea to its limit: instead of one skip connection per block, every layer is connected to every layer after it inside a dense block. The twist that makes this work, and makes it cheap, is that the connections concatenate features rather than add them, so nothing is overwritten and every feature is reused. The result matches ResNet’s accuracy with a fraction of the parameters, and won the CVPR 2017 best-paper award. This note builds the idea from its one real difference with ResNet, explains the counterintuitive parameter saving, and reads off the architecture.

The idea: connect every layer to every later layer

ResNet connects a block to the one just before it through a single shortcut. DenseNet generalizes the move: inside a dense block, layer $\ell$ takes as input the feature maps of all preceding layers,

$${\mathbf{x}}_{\ell }=H_{\ell }([{\mathbf{x}}_0,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{\ell -1}]),$$

where $[\cdot ]$ is concatenation along the channel axis and $H_{\ell }$ is a composite of batch normalization, a ReLU, and a $3\times 3$ convolution. A dense block of $L$ layers therefore has $\frac{L(L+1)}{2}$ direct connections rather than $L$: every layer can see, unaltered, everything computed before it, and every layer’s output is available, unaltered, to everything after it. This is the slide’s “extremization of ResNet”: ResNet’s skip reaches back one block, DenseNet’s reach back to every previous layer.

The one real difference from ResNet: concatenate, not add

The dense wiring looks like more of the same, but the operation that combines the connections is different, and that difference is the whole story.

• ResNet adds. A block computes $F(\mathbf{x})+\mathbf{x}$. The shortcut and the new features land in the same tensor, summed. There is a single running state that each block reads and overwrites a little.
• DenseNet concatenates. A layer reads $[{\mathbf{x}}_0,\dots ,{\mathbf{x}}_{\ell -1}]$ and appends its own new maps. Nothing is summed and nothing is overwritten; the tensor grows, and every feature ever produced stays addressable.

A running state versus a feature bank

Add-versus-concatenate is a choice between two ways to carry information through depth. ResNet keeps a single running state, the residual stream, that each block nudges: a feature can be refined, but it can also be written over. DenseNet keeps an accumulating feature bank: each layer reads the whole bank, contributes a few new entries, and erases nothing, so the final classifier sees features from every depth and abstraction level at once. The Transformer inherited ResNet’s running stream; DenseNet’s feature bank is the alternative most later networks set aside, used where parameter efficiency matters most.

Why dense connectivity uses fewer parameters

Dense connectivity has more connections than ResNet but fewer parameters. The reason is feature reuse.

Because every layer already has direct access to all earlier feature maps, it never has to reproduce or carry them forward. It only has to compute a small number of new feature maps. That number is the growth rate $k$, deliberately tiny (typically $k=12$ to $32$), so the layers are very narrow. After $\ell$ layers a dense block holds

$$k_0+\ell k$$

channels: the width grows linearly in depth, $k$ per layer, with none of the wide jumps of VGG or ResNet.

The saving comes from reuse, not from fewer wires

The connections themselves are free, concatenation has no weights; the parameters live in the convolutions, and dense connectivity lets those convolutions be tiny. In ResNet, addition can overwrite, so each layer must be wide enough to both preserve and transform the representation; in DenseNet, concatenation preserves for free, so each layer can be a thin specialist that adds only $k$ new maps on top of everything already available. A DenseNet-121 reaches the accuracy of a ResNet-50 with roughly a third of its parameters, for exactly this reason.

Keeping the growing input in check

Concatenation makes each layer’s input grow, so two refinements keep the cost flat. A bottleneck (a $1\times 1$ convolution before the $3\times 3$, the Inception trick again) caps the channels the $3\times 3$ has to process; and compression at the transition layers halves the channel count between dense blocks. The variant with both is named DenseNet-BC.

Dense blocks and transition layers

A DenseNet alternates two kinds of stage, visible in the diagram as the boxed dense blocks separated by plain convolution-and-pooling steps:

• a dense block, where the densely connected layers live and the spatial resolution stays fixed;
• a transition layer between dense blocks: a $1\times 1$ convolution that halves the number of feature maps (compression), followed by a $2\times 2$ average pooling that halves the resolution.

The full network is a stem convolution, then dense blocks separated by transition layers, ending, like GoogLeNet and ResNet, in global average pooling and a single linear classifier rather than a heavy dense head. The ImageNet models use four dense blocks; the diagram illustrates the pattern with three. The thin curved arrows inside each block are the dense connections, every node feeding every later node.

Why downsampling is exiled to the transitions

Concatenation only works if all the maps being stacked share the same height and width. A dense block therefore cannot contain a downsampling step, so every change of resolution is pushed out into the transition layers between blocks. This is why the architecture is a strict alternation, a dense block at one resolution then a transition to the next, and why a DenseNet has only as many resolution levels as it has dense blocks.

The variants

Variant	Layers	Parameters
DenseNet-121	$121$	$\approx 8$M
DenseNet-169	$169$	$\approx 14$M
DenseNet-201	$201$	$\approx 20$M
DenseNet-264	$264$	$\approx 33$M

Set these against ResNet (ResNet-50 at $\approx 24$M, ResNet-152 at $\approx 59$M) and the point of the design is plain: comparable accuracy, far fewer parameters, and the gap widens with depth, which is what the slide means by “dramatic reduction of parameters with deep networks”.

Two deeper consequences

Every layer is close to both the input and the loss

Dense connectivity gives each layer a short path to the input (through the concatenations) and to the loss (the classifier reads its features directly). The gradient therefore reaches even the earliest layers with little attenuation, a built-in form of deep supervision, the benefit GoogLeNet’s auxiliary classifiers chased explicitly and got here for free. DenseNet fights the vanishing gradient even harder than ResNet, by keeping every layer’s path to the loss short.

Few parameters, much memory

DenseNet shows clearly that parameter count, compute, and memory are three different costs. It is extraordinarily parameter-efficient, and yet training it is memory-hungry: every layer’s output must be kept alive for the concatenations and for backpropagation, so the activation memory grows roughly with the square of the depth, and a naive implementation that re-materializes each concatenation makes it worse. This, more than accuracy, is why ResNet stayed the default backbone in practice despite DenseNet’s smaller models. A network can be small to store and still expensive to train, which is why parameter count alone is a poor guide to a model’s real cost.

Two answers to one question

DenseNet and ResNet are the two answers to one question: how should information flow through a deep network. ResNet’s additive residual stream became the universal default, the spine of the Transformer and of nearly every large model since. DenseNet’s concatenative feature bank is the alternative most later networks set aside, used where its parameter efficiency pays. Knowing both, and that they differ in exactly one operation, add versus concatenate, is the clearest way to see what a skip connection is for.