Guiding Diffusion Models: Classifier and Classifier-Free Guidance

In the previous sections, we explored how diffusion models can generate diverse samples by learning to reverse a noising process. However, often we want to control what the model generates, for example, generating an image of a specific object or style. This is known as conditional generation. Guidance techniques are methods to steer this generation process towards desired attributes.

The core idea behind many guidance techniques is to modify the sampling process, specifically how the model predicts the less noisy sample $x_{t-1}$ from $x_t$. Recall from DDPMs (Section 1.6) that the model ${ϵ}_{\theta }(x_t,t)$ predicts the noise added to $x_0$ to get $x_t$. The reverse process then uses this predicted noise to estimate the mean of $p_{\theta }(x_{t-1}|x_t)$.

$${\mu }_{\theta }(x_t,t)=\frac{1}{\sqrt{{\alpha }_t}}\left(x_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{\theta }(x_t,t)\right)$$

Guidance methods typically alter the effective noise prediction ${ϵ}_{\theta }(x_t,t)$ or, equivalently, the score ${\nabla }_{x_t}\log p(x_t)$, to incorporate the desired condition $y$.

3.1. The Role of Score Matching and Noise Prediction

To fully appreciate why predicting the noise $ϵ$ is so effective and how it relates to guiding the generation process, it’s helpful to briefly touch upon the concept of score-based generative models (also known as score matching).

The “score” of a data distribution $p(x)$ at a point $x$ is defined as the gradient of the log-probability with respect to the data: ${\nabla }_x\log p(x)$. Score-based models aim to learn this score function for the data distribution at different noise levels.

The key intuition connecting this to noise prediction in DDPMs is that the score of the noised data distribution $q(x_t|x_0)$ can be shown to be proportional to the negative of the noise $ϵ$ that was added to $x_0$ to obtain $x_t$ (when conditioned on $x_0$). Specifically, for $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$, we have ${\nabla }_{x_t}\log q(x_t|x_0)=-\frac{ϵ}{\sqrt{1-{\bar{\alpha }}_t}}$.

So, a model ${ϵ}_{\theta }(x_t,t)$ that is trained to predict the noise $ϵ$ is implicitly learning a scaled version of the score ${\nabla }_{x_t}\log q(x_t|x_0)$. This is why the terms “noise prediction model” and “score-based model” are often used interchangeably in the context of diffusion models, as they are learning essentially the same underlying quantity. The equations for classifier guidance, which explicitly use the score ${\nabla }_{x_t}\log p(x_t|y)$, highlight this connection.

While a deep dive into score-based models is beyond the scope of this note, understanding this connection provides a richer perspective on why diffusion models work and how guidance mechanisms are formulated. For those interested in a comprehensive exploration that covers VAEs, DDPMs, DDIMs, and Score-Based Models in detail, the following guide is an excellent resource:

• Tutorial on Diffusion Models for Imaging and Vision (Chan, 2025)

3.2. Classifier Guidance

Classifier guidance, introduced by Dhariwal and Nichol (2021), uses a separate, pre-trained classifier $p_{\varphi }(y|x_t)$ to guide the diffusion sampling process. The classifier is trained to predict the class $y$ of a noisy image $x_t$.

The Core Idea: Modifying the Score

The goal is to sample from the conditional distribution $p(x_t|y)$. Using Bayes’ theorem:

$$p(x_t|y)=\frac{p(y|x_t)p(x_t)}{p(y)}$$

Taking the logarithm and then the gradient with respect to $x_t$:

$${\nabla }_{x_t}\log p(x_t|y)={\nabla }_{x_t}\log p(x_t)+{\nabla }_{x_t}\log p(y|x_t)$$

Let’s break this down:

• ${\nabla }_{x_t}\log p(x_t)$: This is the score of the unconditional distribution that our diffusion model has learned to approximate. If our model ${ϵ}_{\theta }(x_t,t)$ predicts the noise, this score is related by $s_{\theta }(x_t,t)\approx -{ϵ}_{\theta }(x_t,t)/\sqrt{1-{\bar{\alpha }}_t}$.
• ${\nabla }_{x_t}\log p(y|x_t)$: This is the gradient of the log-likelihood of the condition $y$ given $x_t$. This term is provided by the external classifier $p_{\varphi }(y|x_t)$. It “points” $x_t$ in a direction that makes it more recognizable as class $y$ by the classifier.

The guided score $s_{guided}(x_t,t,y)$ is then a combination, often with a guidance scale $\lambda$ (also denoted $s$ or guidance_scale):

$$s_{guided}(x_t,t,y)=s_{\theta }(x_t,t)+\lambda {\nabla }_{x_t}\log p_{\varphi }(y|x_t)$$

This modified score then leads to an adjusted noise prediction ${ϵ}_{\theta }^{\prime }(x_t,t,y)$ that is used in the DDPM sampling step:

$${ϵ}_{\theta }^{\prime }(x_t,t,y)={ϵ}_{\theta }(x_t,t)-\lambda \sqrt{1-{\bar{\alpha }}_t}{\nabla }_{x_t}\log p_{\varphi }(y|x_t)$$

The term $\lambda \sqrt{1-{\bar{\alpha }}_t}$ is often absorbed into a single guidance scale hyperparameter. The sign depends on whether the gradient is added to the score or subtracted from the noise.

How it Works in Practice

During each step $t$ of the reverse diffusion process:

1. The diffusion model predicts the unconditional noise ${ϵ}_{\theta }(x_t,t)$.
2. The current noisy image $x_t$ and the target class $y$ are fed to the classifier $p_{\varphi }(y|x_t)$.
3. The gradient of the log probability of class $y$ with respect to the input $x_t$, i.e., ${\nabla }_{x_t}\log p_{\varphi }(y|x_t)$, is computed. This requires $x_t$ to have gradients enabled.
4. The unconditional noise prediction is adjusted using this gradient and the guidance scale.
5. The sampling step proceeds using this adjusted noise.

Pseudo-code for Classifier Guidance

# classifier_model: Pre-trained image classification model p_phi(y|x_t)
# model: Unconditional diffusion model epsilon_theta(x_t, t)
# scheduler: DDPM scheduler for timesteps and updates
# y: Target class label
# guidance_scale: Strength of the classifier guidance
 
input = get_noise_from_standard_normal_distribution(...)  # Initial noisy image x_T
 
for t in tqdm(scheduler.timesteps):  # Iterate from T down to 0
    # Ensure input requires gradients for the classifier
    input_with_grad = input.detach().requires_grad_(True)
 
    # 1. Predict unconditional noise
    with torch.no_grad():
        noise_pred_uncond = model(input, t).sample  # epsilon_theta(x_t, t)
 
    # 2. Get classifier gradient
    log_probs_y = classifier_model(input_with_grad, t).log_softmax(dim=-1)[:, y]  
    # log p_phi(y|x_t)
    
    # 3. Compute gradient for guidance
    class_gradient = classifier_model.get_class_guidance(input_with_grad, y)  
    # nabla_{x_t} log p_phi(y|x_t)
 
    # 4. Adjust noise prediction
    noise_pred_guided = noise_pred_uncond + class_gradient * guidance_scale
 
    # 5. Perform sampling step
    input = scheduler.step(noise_pred_guided, t, input).prev_sample

Note on Pseudo-code:

The exact implementation details, especially the sign and scaling of class_gradient, can vary. The key is that the classifier’s gradient steers the sampling. The classifier_model.get_class_guidance encapsulates the gradient computation ${\nabla }_{x_t}\log p_{\varphi }(y|x_t)$.

Pros and Cons of Classifier Guidance

Pros:

• Can guide any pre-trained unconditional diffusion model without retraining it.
• Allows leveraging powerful, off-the-shelf classifiers.

Cons:

• Requires a separate classifier model, which must be robust to noisy inputs $x_t$ (often requires training the classifier on noisy data).

• The guidance is limited to the classes the classifier was trained on.
• Can be computationally more expensive due to classifier forward/backward passes at each step.
• Guidance can sometimes lead to adversarial examples for the classifier, resulting in artifacts if the guidance scale is too high.

3.3. Classifier-Free Guidance (CFG)

Classifier-Free Guidance (CFG), proposed by Ho and Salimans (2022), offers a way to guide diffusion models without needing an external classifier. It has become a very popular and effective technique.

The Core Idea: Jointly Trained Conditional Model

The key idea is to train a single diffusion model ${ϵ}_{\theta }(x_t,t,y)$ that is conditioned on $y$ (e.g., class label, text embedding). During training, this model is occasionally fed a null condition $\varnothing$ (e.g., a zero vector for class embeddings, or an empty string embedding for text). This means the model learns both conditional generation $p(x_t|y)$ and unconditional generation $p(x_t)$ (when $y=\varnothing$).

At sampling time, the model makes two predictions:

1. ${ϵ}_{\theta }(x_t,t,y)$: The noise prediction conditioned on the desired $y$.
2. ${ϵ}_{\theta }(x_t,t,\varnothing )$: The noise prediction for unconditional generation.

The final noise prediction ${ϵ}_{\theta }^{\prime }$ used for sampling is an extrapolation from the unconditional prediction in the direction of the conditional one:

$${ϵ}_{\theta }^{\prime }(x_t,t,y,\lambda )={ϵ}_{\theta }(x_t,t,\varnothing )+\lambda ({ϵ}_{\theta }(x_t,t,y)-{ϵ}_{\theta }(x_t,t,\varnothing ))$$

Here, $\lambda$ is the guidance scale (often denoted $w$ or guidance_scale).

• If $\lambda =0$, we get unconditional generation: ${ϵ}_{\theta }^{\prime }={ϵ}_{\theta }(x_t,t,\varnothing )$.
• If $\lambda =1$, we get standard conditional generation: ${ϵ}_{\theta }^{\prime }={ϵ}_{\theta }(x_t,t,y)$.
• If $\lambda >1$, the generation is pushed further in the direction of $y$, often improving sample quality and adherence to the condition, at the cost of diversity.

Equivalently,

$${ϵ}_{\theta }^{\prime }(x_t,t,y,\lambda )=(1-\lambda ){ϵ}_{\theta }(x_t,t,\varnothing )+\lambda {ϵ}_{\theta }(x_t,t,y).$$

How it Works in Practice

1. Training – Train a conditional diffusion model ${ϵ}_{\theta }(x_t,t,y)$. With some probability (e.g., 10-20 %), replace the true condition $y$ with a null/empty condition $\varnothing$.
2. Sampling – At each step $t$:
   • Compute ${ϵ}_{cond}={ϵ}_{\theta }(x_t,t,y)$ and ${ϵ}_{uncond}={ϵ}_{\theta }(x_t,t,\varnothing )$.
   • Combine them: ${ϵ}_{guided}={ϵ}_{uncond}+\lambda ({ϵ}_{cond}-{ϵ}_{uncond})$.
   • Use ${ϵ}_{guided}$ in the DDPM sampling step.

Pseudo-code for Classifier-Free Guidance (Text-to-Image Example)

# model: conditional diffusion model epsilon_theta(x_t, t, y_embedding)
# scheduler: DDPM scheduler
# text_condition: e.g. "a photo of a cat"
# guidance_scale: CFG strength
# text_encoder: encodes text to embeddings
 
cond_emb = text_encoder.encode(text_condition)
uncond_emb = text_encoder.encode("")  # empty string
 
x = torch.randn_like(image_shape)  # x_T ~ N(0,I)
 
for t in scheduler.timesteps:
    with torch.no_grad():
        eps_uncond = model(x, t, encoder_hidden_states=uncond_emb).sample
        eps_cond   = model(x, t, encoder_hidden_states=cond_emb).sample
    eps_guided = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
    x = scheduler.step(eps_guided, t, x).prev_sample