Multilabel Logistic Loss

This note specialises the cross-entropy loss to multilabel classification: a multi-class problem in which the classes are not mutually exclusive, so a single example can carry several labels at once.

One item, several labels

On an e-commerce platform a single product can be tagged simultaneously as electronics, laptop, and gaming. The labels do not compete: assigning one does not forbid the others.

The model produces one output neuron per class, and each output $a_j$ estimates, independently, the probability that the example belongs to class $j$:

• each $a_j$ is an independent probability for class $j$;
• there is no normalisation constraint, so ${\sum }_j{a}_j\ne 1$ in general;
• an example may belong to zero, one, or several classes at the same time.

Why a sigmoid on each output neuron

The sigmoid maps any real pre-activation $z_j\in (-\infty ,+\infty )$ into $(0,1)$, so each output can be read as a probability. While the sigmoid is rarely used in hidden layers today (replaced by ReLU and its family), it remains the natural choice for an output neuron whose value must be a standalone probability. Multilabel classification is the textbook case: each sigmoid output $a_j$ independently estimates whether class $j$ is present.

Independent sigmoids do not form a distribution

A layer of independent sigmoid outputs does not produce a probability distribution, because the activations need not sum to one. A concrete check with two output neurons:

$$z_1=2\Rightarrow a_1=\sigma (2)\approx 0.88,z_2=3\Rightarrow a_2=\sigma (3)\approx 0.95,a_1+a_2\approx 1.83>1.$$

Sigmoid outputs are independent gates, not a distribution

Each sigmoid acts on its own neuron, with no coupling between outputs, so the vector ${\mathbf{a}}^L$ is a set of independent per-class probabilities rather than a single distribution over classes. A genuine distribution over mutually exclusive classes requires a softmax layer, which couples the outputs so they sum to one. The choice between the two is the choice between multilabel (this note) and multiclass-exclusive (NLL) classification.

Cross-entropy for the multilabel case

Multilabel classification = many independent binary problems

A multilabel problem is a collection of independent binary classification problems, one per class, solved in parallel by the same network.

For each class $j$ the output neuron estimates $a_j$, the probability that the example belongs to class $j$; the complement $1-a_j$ is the probability that it does not (a one-versus-all split, with all other classes lumped into “not $j$”). Each neuron therefore carries its own binary cross-entropy, and the total loss is their sum over the $C$ output neurons:

$$\boxed{{\mathcal{L}}_{ML}=\sum _{j=1}^C{\mathcal{L}}_{BCE,j}=-\sum _{j=1}^C[y_j\log a_j+(1-y_j)\log (1-a_j)]}$$

This is the multilabel logistic loss, ${\mathcal{L}}_{ML}$. Each term is a complete BCE on one class: it rewards the model for raising $a_j$ toward $1$ on present labels and lowering it toward $0$ on absent ones, independently for every class.

Symbol	Meaning
$a_j$	probability that the example belongs to class $j$
$1-a_j$	probability that it does not belong to class $j$
${\mathcal{L}}_{ML}$	sum of the per-class binary cross-entropies

Why "logistic"

The name refers to the activation: each $a_j=\sigma (z_j)=1/(1+e^{-z_j})$ is produced by a sigmoid, also called the logistic function in statistics. For this reason the loss is exposed in frameworks such as PyTorch as BCEWithLogitsLoss: the optimiser works directly on the raw logits $z_j$, and the loss applies the sigmoid internally, which is both more numerically stable and more efficient than applying the sigmoid as a separate step.

Numerical instability of log-based losses

Losses built on logarithms, binary and multilabel cross-entropy alike, can become unstable at the extremes of their inputs. When $a_j$ reaches exactly $0$ or $1$, a term $\log (0)=-\infty$ appears and, propagated through backpropagation, produces NaN gradients that derail training. Two standard remedies:

• clamp with an $\epsilon$: replace $\log (a_j)$ by $\log (a_j+\epsilon )$ and $\log (1-a_j)$ by $\log (1-a_j+\epsilon )$, with a small $\epsilon$ such as $10^{-8}$;
• better, use the logits form (BCEWithLogitsLoss), which fuses the sigmoid and the logarithm into a single stable expression and never materialises the dangerous intermediate $a_j$.

Multilabel versus multiclass-exclusive

It is worth setting the two sister losses side by side, because the choice of loss encodes a modelling assumption about the labels.

Property	Multilabel (this note)	Multiclass-exclusive (NLL)
Output activation	sigmoid per neuron	single softmax
Outputs sum to one?	no, independent	yes, coupled
Labels per example	zero, one, or many	exactly one
Loss	${\sum }_j{\text{BCE}}_j$	$-\log a_{c^{\star }}$
Target	binary vector (any number of $1$s)	one-hot vector

Choosing the multilabel loss when the classes are genuinely exclusive wastes the strong prior that exactly one class is correct; choosing softmax+NLL when several labels can co-occur forces a single winner where there should be many. The loss is part of the model.

In PyTorch

import torch.nn as nn
 
# logits: raw network outputs, shape (batch, n_classes)
# target: multi-hot float tensor, same shape, with a 1 for every present label
criterion = nn.BCEWithLogitsLoss()       # sigmoid + BCE, summed over classes, fused
loss = criterion(logits, target)

BCEWithLogitsLoss applied to a vector of logits is exactly ${\mathcal{L}}_{ML}$: an independent binary cross-entropy per class, computed in a numerically stable way. The corresponding loss for the mutually-exclusive case is the negative log-likelihood loss, paired with a softmax output.