Computer Vision MT25, Neural networks
Flashcards
Residual connections
One part of a neural network could be interpreted as calculating
\[y = f(x)\]
With residual connections, you instead calculate
\[y = f(x) + x\]
Can you give some intuition for why this might be easier to train?
- In the first option, the network needs to learn to pass on all the information in $x$ to the following layer.
- In the second option, the network only needs to learn the changes in the representation $f(x) = y - x$.
- This means gradients flow much easier through the network.
Batch normalisation
Consider a single layer with ReLU
\[y = \max(0, Wx + b)\]
After initialisation, $W$ and $b$ can be hard to learn (e.g. if $x$ is small, then $W$ needs to be large, and if $x$ is negative, $b$ needs to large to avoid $0$ gradient from ReLU).
Given $x \in \mathbb R^{B \times d}$, how does batch normalisation solve some of these problems, and what is $y \in \mathbb R^{B \times d}$?
Batch normalisation rescales the pre-activations using batch statistics, so they are “well-behaved” for optimisation. For each feature $j$, $\text{BN}(z)$ calculates
- $\mu _ j = \frac 1 B \sum^B _ {i=1} z _ {i,j}$
- $\sigma^2 _ j = \frac 1 B \sum^B _ {i=1} (z _ {i,j} - \mu _ j)^2$
- $\tilde z _ {i,j} = \frac{z _ {i, j} - \mu _ j}{\sqrt{\sigma^2 _ j + \epsilon}}$
- $\text{BN}(z) _ {i, j} = \gamma _ j \tilde z _ {i,j} + \beta _ j$, where $\gamma, \beta \in \mathbb R^d$ are learned (note here that $\gamma _ j$ is a scalar, so this isn’t quite another linear layer inside BN).
In matrix form, with broadcasting across the batch dimension:
\[\text{BN}(z) = \gamma \cdot \frac{z-\mu}{\sqrt{\sigma^2 + \epsilon}} + \beta\]The full layer applied BN between the linear map and the ReLU:
\[y = \max(0, \text{BN}(Wx + b)).\]After batch normalisation, what do you expect to be true about 50% of activations in a ReLU network?
They are $0$.
Where does batch normalisation typically go in a neural network?
After fully connected or convolution layers, and before nonlinearities.
Why can batch normalisation be a large source of bugs?
It behaves differently during training and testing.
Batch normalisation requires statistics to be computed from batches of the input data. How are these statistics found at test time?
They are fixed.
@Visualise the difference between:
- Batch norm
- Layer norm
- Instance norm
- Group norm
Transfer learning
What is transfer learning?
You train a some model on a large dataset for some related task, and then fine-tune on your task that has less data.
Softmax and temperature
@Define the $\text{softmax}(Y, \tau)$ with temperature function, @state three results that intuitively relate it to just taking the argmax of some set of predictions, and @visualise how varying $\tau$ affects the distribution derived from the following input data:
.
where $\hat Y$ is the vector of predictions for each class. We have the results that:
- It maintains the relative ordering: $\text{softmax} _ i(\hat Y, \tau _ 1) < \text{softmax} _ j(\hat Y, \tau _ 1) \implies \text{softmax} _ i(\hat Y, \tau _ 2) < \text{softmax} _ j(\hat Y, \tau _ 2)$
- As $\tau \to \infty$, softmax becomes a uniform distribution.
- As $\tau \to 0$, softmax becomes argmax (as one-hot).
@State the SGD with momentum update rule, and explain what momentum buys you over plain SGD.
where $\rho$ is the momentum coefficient (typically $\sim 0.9$), $\lambda$ is the learning rate, and $g _ t$ is the gradient at step $t$. The new weight is
\[w _ {t+1} = w _ t + \Delta _ t\]Useful because:
- Accelerates progress when consecutive gradients point in the same direction, they reinforce in $\Delta w _ {t-1}$.
- Dampens oscillations across narrow ravines.
SGD with momentum gives the weight update
\[\Delta w _ t = \rho \Delta w _ {t-1} - \lambda g _ t.\]
What does Adam add on top of this, and what does the full update rule look like?
Adam keeps two exponentially decaying averages of past gradients:
- $m _ t = \beta _ 1 m _ {t-1} + (1 - \beta _ 1) g _ t$, the first moment (a running mean, similar to momentum).
- $v _ t = \beta _ 2 v _ {t-1} + (1 - \beta _ 2) g _ t^2$, the second moment (per-coordinate scale of the gradient)
Both are initialised to $0$ and so are biased towards zero early in training. We apply a bias correction to obtain:
- $\hat m _ t = \frac{m _ t}{1 - \beta _ 1^t}$
- $\hat v _ t = \frac{v _ t}{1-\beta _ 2^t}$
This gives the update
\[\Delta w _ t = -\frac{\lambda}{\sqrt{\hat v _ t} + \varepsilon} \hat m _ t.\]The innovation of Adam is a per-coordinate scaling by $1 / \sqrt{\hat v _ t}$, which gives each parameter its own effective learning rate based on the magnitude of its recent gradients.
Bite-sized
At test time, BatchNorm uses fixed statistics — typically an exponential moving average of $(\mu, \sigma^2)$ accumulated over training mini-batches. This makes test-time output deterministic and independent of any other inputs in the test batch.
A key practical advantage of BatchNorm at test time: since $\mu, \sigma^2$ are fixed constants, the BN operation $(x - \mu)/\sqrt{\sigma^2 + \varepsilon}$ followed by the learnable affine rescale $\gamma x' + \beta$ can be fused with the preceding linear / convolution layer, giving BN effectively zero compute or memory overhead at inference.
@Justify why BatchNorm requires reasonably large batches at training time.
BatchNorm computes per-feature mean and variance from the current mini-batch: $\mu _ j = \frac{1}{B} \sum _ {i=1}^B x _ {i,j}$ and similarly for $\sigma^2 _ j$. These are noisy estimates of the true population statistics.
If $B$ is too small (e.g. $B = 1$ or $B = 2$):
- $\sigma^2$ has very high variance and can even be zero, making $1/\sqrt{\sigma^2 + \varepsilon}$ huge and unstable.
- The forward and backward passes depend strongly on the random other samples in the batch, hurting training.
- The accumulated EMA used at test time becomes a bad estimate of the population statistics, so train/test behaviour diverges.
This is why BN is problematic for small-batch settings (e.g. high-resolution image segmentation), and why LayerNorm/GroupNorm are preferred in transformer-scale training where per-GPU batch sizes are tiny.
If the learnable BatchNorm parameters are set to $\gamma _ j = \sqrt{\sigma _ j^2 + \varepsilon}$ and $\beta _ j = \mu _ j$ (i.e. the training-set statistics), then BN reduces to the identity function. So BN can in principle learn to do nothing — it’s a strict generalisation of “no normalisation”.
@Describe why fine-tuning typically uses a lower learning rate than training-from-scratch, and what’s usually frozen.
The pre-trained weights already encode useful features (e.g. edge/blob/colour detectors in early CNN layers, semantic concepts in deeper layers). A high learning rate would quickly destroy this hard-won initialisation by making large gradient updates.
A lower learning rate (often $10\times$ or $100\times$ smaller than training-from-scratch) preserves the broad structure of the pre-trained features while adapting them to the new task.
What’s frozen:
- Feature extractor backbone (the body of the network) is often fully frozen for small target datasets. Only a fresh task-specific head is trained.
- Early conv layers are frozen even when later layers are fine-tuned, since first-layer filters are nearly task-independent (edges, colours) and don’t benefit from adaptation.
- For larger target datasets, you can unfreeze more or all layers, but still use a smaller learning rate than from-scratch training.
Strong evidence that CNN/transformer first-layer filters are task-independent: they almost always look the same (Gabor-like oriented edges, blobs, colour-opponent patches) regardless of what task or dataset the model was trained on. This justifies always pre-training on a large image dataset (e.g. ImageNet, JFT) and reusing the early layers.
ReLU sets roughly 50% of activations to zero (assuming pre-activations are zero-mean, which BatchNorm helps ensure). This sparsity is one motivation for placing BN immediately before the nonlinearity: zero-mean pre-activations let ReLU zero out roughly half its inputs, producing the desired sparse activation pattern.
@Describe one mini-recipe for training a deep model on a small dataset by using pre-training and transfer learning.
- Pick a backbone pre-trained on a large dataset relevant to your modality (ImageNet for natural images, CLIP for vision-language, etc.).
- Replace the head with a new task-specific head (e.g. a fresh linear layer with the right number of output classes), randomly initialised.
- Freeze the backbone and train only the head for a few epochs at a moderate learning rate — this fits the head to the pre-trained features without disturbing them.
- Unfreeze the backbone and continue training at a much lower learning rate (e.g. $10^{-5}$ vs $10^{-3}$ for the head). Optionally use discriminative learning rates: deeper layers get smaller LRs than the head.
- Use heavy augmentations and small batches with care, monitoring validation loss for overfitting.
This is the standard recipe behind nearly every applied CV system on data-limited problems.