Negative Log-Likelihood Loss

Multiclass exclusive classification is the most common classification setting in practice: each example belongs to exactly one of $C$ categories. Digit recognition (one digit per image), object classification (one label per crop), and language modelling (one next token) are all of this form.

Binary problems are often an idealisation

Strictly, many “binary” problems are a coarse approximation of a graded reality. In medical imaging, for instance, there is a whole spectrum of nuance between the extremes benign and malignant. Multiclass exclusive classification is the natural generalisation when the categories are several but still mutually exclusive.

The network has one output neuron per category, and the desired output for a given input is a one-hot vector: a $1$ in the position of the correct class, $0$ everywhere else.

$$\mathbf{y}=(0,0,\underset{\text{correct class}}{1},0,0).$$

The output must be a distribution: softmax

Because the classes are mutually exclusive, the $C$ outputs should form a genuine probability distribution: non-negative and summing to one. Independent sigmoids cannot do this (their outputs need not sum to one, as shown for the multilabel case). The construction that does is the softmax, which couples the outputs,

$$a_j=\mathrm{softmax}(\mathbf{z}{)}_j=\frac{e^{z_j}}{{\sum }_{k=1}^C{e}^{z_k}},\sum _{j=1}^C{a}_j=1.$$

The softmax makes the classes compete for a single unit of probability: raising one output lowers the others. Its properties are developed in Softmax and Cross-Entropy and Softmax properties.

From cross-entropy to negative log-likelihood

The loss for this setting is the cross-entropy between the one-hot target and the softmax output,

$${\mathcal{L}}_{CE}=-\sum _{j=1}^C{y}_j\log a_j.$$

Here the one-hot structure of $\mathbf{y}$ does something clean: every $y_j$ is zero except for the correct class $c^{\star }$, where $y_{c^{\star }}=1$. The whole sum therefore collapses to a single term,

$$\boxed{{\mathcal{L}}_{NLL}=-\log a_{c^{\star }},}$$

the negative log-likelihood of the correct class. The loss depends only on the probability the model assigned to the right answer: it ignores how the remaining mass is distributed among the wrong classes, and cares only that $a_{c^{\star }}$ be pushed toward $1$.

Why this is "negative log-likelihood"

The softmax output $a_{c^{\star }}$ is the probability the model assigns to the observed label. Its likelihood over a dataset is ${\prod }_i{a}_{c^{\star }}^{(i)}$; taking the negative logarithm turns the product into the sum $-{\sum }_i\log a_{c^{\star }}^{(i)}$. Minimising the NLL loss is therefore exactly maximising the likelihood of the correct labels under the model’s categorical distribution. This is the multiclass counterpart of the Bernoulli likelihood behind BCE and the Gaussian likelihood behind MSE: all three losses are negative log-likelihoods, differing only in the assumed output distribution.

Reading the single term confirms the behaviour wanted of a classification loss: $-\log a_{c^{\star }}$ is $0$ when the model is certain and correct ($a_{c^{\star }}\to 1$) and grows without bound as it becomes confidently wrong ($a_{c^{\star }}\to 0$).

Over a dataset the objective is the average,

$${\mathcal{L}}_{NLL}(\mathbf{\theta })=-\frac{1}{N}\sum _{i=1}^N\log a_{c_i^{\star }}^{(i)}.$$

The gradient: the clean residual again

Pairing softmax with this loss reproduces the cancellation that made BCE work. The gradient of the loss with respect to the logits is the signed residual between the predicted distribution and the one-hot target,

$$\frac{\partial {\mathcal{L}}_{NLL}}{\partial \mathbf{z}}=\mathbf{a}-\mathbf{y},$$

with no softmax Jacobian left to attenuate it: confidently wrong predictions receive the largest gradient. The derivation, in which the softmax derivative and the log derivative cancel, is carried out in Softmax and Cross-Entropy and reused in Backpropagation Through Time.

In PyTorch: a naming subtlety worth knowing

PyTorch splits this loss across two objects, and confusing them is a common source of silent bugs.

import torch.nn as nn
 
# Option A: log-softmax in the model, NLLLoss as the criterion
model_tail = nn.LogSoftmax(dim=1)        # outputs log-probabilities
criterion  = nn.NLLLoss()                # expects log-probabilities + integer class index
loss = criterion(model_tail(logits), target)   # target: class indices, not one-hot
 
# Option B (preferred): raw logits straight into CrossEntropyLoss
criterion = nn.CrossEntropyLoss()        # = LogSoftmax + NLLLoss, fused and stable
loss = criterion(logits, target)         # target: class indices

NLLLoss expects log-probabilities, not probabilities

Despite its name, nn.NLLLoss does not apply a logarithm. It expects inputs that are already log-probabilities (the output of LogSoftmax) and simply selects and negates the entry of the true class. Feeding it raw probabilities, or raw logits, gives a wrong loss with no error raised. The robust pattern is nn.CrossEntropyLoss fed raw logits: it fuses LogSoftmax and NLLLoss into one numerically stable operation (via the log-sum-exp trick), so the softmax is never materialised and log(0) cannot occur. The target in both cases is a tensor of class indices, not a one-hot matrix.

The family at a glance

The four classification-and-regression losses of this section are one idea, the negative log-likelihood of a chosen output distribution, specialised to four answer shapes.

Loss	Output activation	Target	Output distribution
MSE	none (real value)	real number	Gaussian
BCE	one sigmoid	$\{0,1\}$	Bernoulli
Multilabel logistic	sigmoid per class	multi-hot	product of Bernoullis
NLL	softmax	one-hot	categorical

The same principle, minimise the negative log-likelihood of the data, generates all four; the choice among them is a statement about what kind of quantity the network is predicting.