Computer Vision MT25, Generative models

We have some dataset $\mathcal D = \{ x _ i \mid 1 \le i \le N \}$ which is assumed to come from some underlying distribution $p _ \text{data}(x)$. We aim to learn a distribution $p _ \text{model}(x)$ we can sample from so that $p _ \text{data}$ and $p _ \text{model}$ are similar.

• Discriminative: $p(y \mid x)$
• Generative: $p(x)$
• Conditional generative: $p(x \mid y)$.

Consider the following form of Bayes rule:

Here:

• $p(x \mid y)$ is a conditional generative model
• $p(y \mid x)$ is a discriminative model
• $p(y)$ is a prior over labels
• $p(x)$ is a generative model

• Explicit: We obtain some representation of $p _ \text{model}(x)$ and can determine the density of samples
• Implicit: We may sample from $p _ \text{model}(x)$ but cannot determine the density of samples exactly

• Explicit models :
  • Tractable density: Autoregressive models, factorise $p(\pmb x) = \prod _ i p(x _ i \mid x _ {< i})$ and evaluate exactly.
  • Approximate density: VAEs, bound $p(\pmb x)$ via an evidence lower bound, optimised through an encoder-decoder pair.
• Implicit models:
  • Direct sampling:
    • GANs: a generator network and a discriminator network, trained adversarially
    • Diffusion models: produce samples via iterative denoising from Gaussian noise
  • Markov chain:
    • GSNs (not covered explicitly in the course)

• Convert the image into a sequence of pixels
• Predict the sequence of pixels by factorising the joint distribution into a series of one-dimensional distributions that only depends on the previous pixels

In order to preserve the autoregressive property, predictions are not allowed to depend on the future pixels, otherwise a model could cheat.

You need to preserve the autoregressive property that the prediction for a pixel is not allowed to depend on the values of the next pixels. Mask A is used for the first layer, and Mask B is used for all subsequent layers.

The difference between having a $0$ or $1$ in the middle of the mask is due to the centre of the computed features after the first layer now longer not conveying information about the next pixels.

It instead generated the images auto-regressively in “token-space”, using tokens learned by a VQ-VAE.

• Train a CNN encoder to generate continuous representations of images
• At training time, replace activations with the closest vector from a learned codebook
• Backpropagate as if nothing happened

Convert samples from a simple distribution (e.g. Gaussian noise) into samples from the data distribution by integrating along a learned velocity field.

Concretely: a neural field $f _ \theta : \mathbb R^d \to \mathbb R^d$ outputs, for every point in space, the velocity at which a sample at that location should move. Starting from $x _ 0 \sim p _ \text{init}$, take discrete Euler steps

until $x _ 0 \sim p _ \text{data}$

Generate an image in small stops from some noise $\epsilon$ by learning a denoising process.

Learn how to generate $p _ \theta (x _ {t-1} \mid x _ t)$.

where $\beta _ t$ is a “variance schedule”, which is often fixed.

We rewrite to give the update

where $\alpha _ t = 1 - \beta _ t$ and $\bar \alpha _ t = \prod^t _ {i = 1} \alpha _ i$ (note: the lecture slide has $\sum$, but I think this is a typo). Applying this recursively yields

Two closure rules:

• Independent sum: $X + Y \sim \mathcal N(\mu _ x + \mu _ y,\ \sigma _ x^2 + \sigma _ y^2)$ — means and variances add (variances, not standard deviations; the variance rule needs independence).
• Affine transform: $aX + b \sim \mathcal N(a\mu _ x + b,\ a^2\sigma _ x^2)$ — scaling by $a$ multiplies the mean by $a$ and the variance by $a^2$.

Apply both to $\sqrt\lambda\, X + \sqrt\omega\, Y$: each term is an affine scaling ($a = \sqrt\lambda$ or $\sqrt\omega$, $b = 0$), giving $\sqrt\lambda\, X \sim \mathcal N(\sqrt\lambda\,\mu _ x,\ \lambda\sigma _ x^2)$ and $\sqrt\omega\, Y \sim \mathcal N(\sqrt\omega\,\mu _ y,\ \omega\sigma _ y^2)$; then add means and variances:

Key point: a $\sqrt\lambda$ in amplitude contributes $\lambda$ (not $\sqrt\lambda$) to the variance. For standard Gaussians this is exactly $\sqrt a\,\epsilon + \sqrt b\,\epsilon' = \sqrt{a+b}\,\bar\epsilon$ with $\bar\epsilon \sim \mathcal N(\pmb 0, I)$ — the merge step that makes the diffusion noise terms telescope (∆diffusion-closed-form-derivation).

By induction on $t$.

Base case ($t = 1$): $\bar\alpha _ 1 = \alpha _ 1$, so the recurrence $x _ 1 = \sqrt{\alpha _ 1}\,x _ 0 + \sqrt{1-\alpha _ 1}\,\epsilon _ 0$ is already the claim.

Inductive step: assume $x _ {t-1} = \sqrt{\bar\alpha _ {t-1}}\,x _ 0 + \sqrt{1-\bar\alpha _ {t-1}}\,\tilde\epsilon$ with $\tilde\epsilon \sim \mathcal N(\pmb 0, I)$ independent of $\epsilon _ {t-1}$. Then

• $(\star 1)$: the forward recurrence.
• $(\star 2)$: substitute the inductive hypothesis for $x _ {t-1}$ and distribute $\sqrt{\alpha _ t}$.
• $(\star 3)$: on the signal term $\alpha _ t \bar\alpha _ {t-1} = \bar\alpha _ t$; the two independent noise terms merge into a single $\epsilon \sim \mathcal N(\pmb 0, I)$ scaled by the root of their summed variances (the Q3.1 lemma).
• $(\star 4)$: simplify, $\alpha _ t(1-\bar\alpha _ {t-1}) + (1-\alpha _ t) = 1 - \alpha _ t\bar\alpha _ {t-1} = 1 - \bar\alpha _ t$.

This closes the induction.

Why it works: the coefficients satisfy $(\sqrt{\bar\alpha _ t})^2 + (\sqrt{1-\bar\alpha _ t})^2 = 1$ at every $t$, so each merge preserves unit total variance — the noise scale is exactly what is left once the signal is attenuated by $\sqrt{\bar\alpha _ t}$.

Do diffusion in latent space rather than pixel space.

Algorithm:

1. Sample $x _ T \sim \mathcal N(0, I)$, starting from pure Gaussian noise.
2. For $t = T, T-1, \ldots, 1$:
   1. Sample $z \sim \mathcal N(0, I)$ if $t > 1$, else $z = 0$ (so there is no noise on the last step).
   2. Update:

	- where $\beta _ t$ is a noise schedule, $\alpha _ t = 1 - \beta _ t$ is a signal retention factor, and $\overline \alpha _ t = \prod^t _ {i=1} \alpha _ i$ is the cumulative signal
3. Return $x _ 0$.

Explanation: Each step subtracts a scaled version of the predicted noise to denoise $x _ t$ a bit, then re-ads a smaller amount of fresh Gaussian noise $\sqrt{\beta _ t} z$. The fresh noise at each intermediate step keeps the trajectory stochastic.

• Predict noise: minimise $\|f(x _ t, t) - \epsilon\| _ 2^2$.
• Predict clean image: minimise $\|f(x _ t, t) - x _ 0\|^2 _ 2$.

These two forms are equivalent. In practice, predicting noise tends to work slightly better, since:

• Target diversity: $\epsilon$ targets are different at every step, giving the model varied supervision throughout training.
• Memorisation risk: an $x _ 0$-prediction model has the same target $x _ 0$ for every $t$, which makes overfitting / memorisation of the specific training images more of an issue.

Setup:

• Feature extractor $f : \text{Image} \to \mathbb R^d$, typically Inception v3.
• Compute features $\{f(I) \mid I \in \mathcal I _ r\}$ and $\{f(I) : I \in \mathcal I _ g\}$.
• Fit a Gaussian to each set: $\mathcal N(\mu _ r, \Sigma _ r)$ to real features, $\mathcal N(\mu _ g, \Sigma _ g)$ to generated features.

The FID is then

The first term is the cost of moving between the mean of the real distribution and of the generated distribution. The second term is the cost of updating the covariance $\Sigma _ r$ to $\Sigma _ g$.

Setup: train a VQ-VAE encoder-decoder pair $(\mathcal E, \mathcal D)$ on the image dataset.

• $\mathcal E : \text{image} \to \text{latent}$, mapping $3 \times H \times W$ to $D \times H' \times W'$.
• $\mathcal D : \text{latent} \to \text{image}$, mapping $D \times H' \times W'$ back to $3 \times H \times W$.
• $D \times H' \times W'$ are chosen so that the latent representation is smaller.

Diffusion: Diffusion runs on $z = \mathcal E(x)$, not on $x$:

• Forward process: $q(z _ t \mid z _ {t-1}) = \mathcal N(z _ t \mid \sqrt{1-\beta _ t} z _ {t-1}, \beta _ t I)$
• Reverse process: a denoising U-Net $\epsilon _ \theta (z _ t, t, c)$ predicts the noise added after a step $t$.
• After sampling completes, $z _ 0$ is decoded back to a pixel-space image via $\tilde x = \mathcal D(z _ 0)$.

1. Conditional diffusion model: Every denoising step takes a conditioning input $c$, so that the model can generate images guided by different prompts.
2. Latent diffusion: The diffusion process runs on a low-dimensional latent $z = \mathcal E(x)$ from a VQ-VAE, not on raw pixels (∆stable-diffusion-latent-diffusion).
3. U-Net architecture: The denoising network $\epsilon _ \theta (z _ t, t, c)$ is a CNN encoder-decoder with skip connections, taking noisy latent + timestep + conditioning as inputs.
4. Cross-attention layers: inserted between U-Net conv blocks, attending to the encoded conditioning $\tau _ \theta(c)$ (∆stable-diffusion-cross-attention). The U-Net also receives the timestep via a Fourier-basis embedding (∆stable-diffusion-time-dependency).

Conditioning encoder $\tau _ \theta$: a separate network that maps the raw prompt to a sequence of tokens:

• Text prompt: a text encoder
• Semantic map / layout: the corresponding encoder
• Image: an image encoder

The output is $\tau _ \theta(c) \in \mathbb R^{M \times d}$, an $M$-token representation in the U-Net’s hidden dimension.

Cross-attention layers: inserted between the U-Net’s conv blocks at each spatial resolution. As in any attention layer, $Q$, $K$, $V$ are learned linear projections of their sources (per head, matrices $W _ Q, W _ K, W _ V$):

• Queries $Q$: projected from the U-Net’s spatial features at that resolution, one query per spatial location.
• Keys and values $K$, $V$: projected from $\tau _ \theta(c)$, and only from $\tau _ \theta(c)$. The U-Net features go into $Q$ alone, never into $K$ or $V$.
• Output: $\text{softmax}(QK^\top / \sqrt d) V$, mixed back into the U-Net features via the residual stream.

Where does conditioning enter the U-Net? The prompt $c$ enters only via these cross-attention layers. The timestep $t$ is a separate conditioning signal injected differently: a Fourier embedding passed through an MLP and added per-channel into each ResBlock (∆stable-diffusion-time-dependency). Spatially-aligned conditioning (e.g. an inpainting mask) can additionally be concatenated channel-wise with the noisy latent at the U-Net input.

Fourier-basis time embedding: Convert the scalar $t$ into a vector

where $\omega _ 1, \ldots, \omega _ K$ are a fixed set of frequencies.

MLP: Pass the Fourier embedding through an MLP to produce $f(t) \in \mathbb R^C$, one value per channel.

Add to U-Net features: at each layer of the U-Net, broadcast $f(t)$ across the spatial dimensions and add per-channel:

• VAE (Kingma et al., ICLR 2014): the encoder $E$ outputs $(\mu, \sigma)$ for a continuous Gaussian posterior $q(z \mid x) = \mathcal N(\mu, \sigma^2)$. A sample $z \sim q(z \mid x)$ is decoded by $D$. Trained on the ELBO: reconstruction loss minus KL divergence to a prior.
• VQ-VAE (van den Oord et al., 2017): the encoder produces a continuous feature map $z _ e(x)$, but the bottleneck quantises each activation vector to its nearest entry in a learned codebook $\{e _ 1, \ldots, e _ K\}$. The decoder operates on the quantised $z _ q(x)$. Gradients flow through the quantiser via the straight-through estimator.

VQ-VAE’s discrete latent is what makes it pair naturally with autoregressive models on tokens (DALL-E) and with diffusion in latent space (Stable Diffusion’s VAE-like compressor).

Consider two paths in space — one walked by a dog, one walked by its owner, each parameterised by time $t \in [0, 1]$ and constrained to move only forward. The Frechet distance is the shortest possible leash that allows both to traverse their respective paths, where the leash length at time $t$ is the distance between the two positions.

Equivalently in measure-theoretic terms (Wasserstein-2 / earth-mover’s): treat each distribution as a pile of dirt and consider the minimum total transport cost to reshape one into the other under squared distance.

FID applies this idea in Inception-v3 feature space: it fits Gaussians $\mathcal N(\mu _ r, \Sigma _ r)$ and $\mathcal N(\mu _ g, \Sigma _ g)$ to real and generated features, and computes the closed-form Wasserstein-2 distance between those Gaussians.

• Explicit models (autoregressive, VAEs) directly compute $\log p _ \text{model}(\pmb x)$ for any input $\pmb x$. You can take a held-out test set and average $\log p$ across it: the model with higher average held-out log-likelihood wins. Single number, comparable across models.
• Implicit models (GANs, diffusion, GSNs) only let you sample from $p _ \text{model}$. We cannot evaluate the density at a particular point, so there is no test-set log-likelihood. Instead we resort to indirect proxies:
  • Human evaluation: ask people which images they prefer. Subjective, vague (realism? prompt fidelity? aesthetics?), expensive, hard to scale.
  • FID: compute a Wasserstein-2 distance between Gaussian fits to Inception features of real vs generated images. Aligns somewhat with human judgement but is gameable and not a true likelihood.
  • Inception Score, Precision/Recall for generative models, CMMD, etc.: each has known failure modes.

So while explicit models have a single principled comparison metric (held-out NLL), the field has no clean winner for evaluating implicit models — FID is the de-facto standard despite its weaknesses.