Computer Vision MT25, Unsupervised computer vision

• Dataset $\mathcal D = \{x _ i \mid 1 \le i \le N\}$, inputs only
• Dataset is split into training / validation $\mathcal D = \mathcal D _ T \sqcup \mathcal D _ V$
• We aim to learn some $f$ which will be useful for a downstream task (e.g. assigning clusters)
• We have some downstream task $T = \{(\chi _ i, \nu _ i) \mid 1 \le i \le M \}$ (e.g. classification)

1. Recovery: Let $M$ be some transformation. Train $f$ to undo $M$, i.e. so that $f(M(x)) \approx x$.
2. Bottleneck: Let $g$ be a function from some restricted class (e.g. a low-dimensional encoder). Train $f$ to recover $x$ from $g(x)$, i.e. so that $f(g(x)) \approx x$.
3. Dataset: E.g. using patches of an image and unrelated patches from the dataset to learn unsupervised image representations.
4. Invariance: Make sure $f$ is (approximately) invariant under some class of transformations to the input, so that $f(\pi(x)) \approx f(x)$.
5. Equivariance: Make sure $f$ is (approximately) equivariant under some class of transformations to the input, so that $f(\pi(x)) \approx \pi'(f(x))$.
6. Transformation estimation: Given a transformed input $x$, estimate the transformation: $f(\pi(x, \theta)) \approx \theta$
7. Generative: Learn to generate samples from the dataset.

• Noise (i.e. $f$ learns denoising)
• Occlusion (i.e. $f$ learns inpainting)
• Grayscale

• Low dimensional
• Sparse
• Activations from a dictionary

Given two patches from an image, determine the spatial relationship between the two patches.

• Student network and teacher network, both with the same architecture
• These networks are randomly initialised
• Each input $x$ is split into two components
  • $x _ 1$ for the student network, a random crop or transformation of the full input
  • $x _ 2$, the full input
• The output probability distributions are compared via cross-entropy loss and gradient descent optimises the parameters of the student
• The teacher’s parameters are an exponentially weighted moving average of the student

This encourages local-to-global features to be learned.

Centring: We centre the teacher’s pre-softmax logits:

where the average is over a running batch of teacher outputs.

This subtracts off the running mean of the teacher outputs. Without it, the EMA dynamics push the teacher towards a constant function, which is one mode of collapse.

Sharpening: We sharpen the teacher’s output distribution by decreasing the softmax temperature:

with $\tau _ T$ small. This makes the teacher’s distribution more peaked. Without it, centring alone tends to push the teacher towards the uniform distribution, which is another mode of collapse.

It provides a rich set of features for images, which can be used for downstream tasks.

Supervision but where the outputs come from a different task.

Setup: Take an unlabelled image $x$, apply a random rotation $r \in \{0^\circ, 90^\circ, 180^\circ, 270^\circ\}$ to produce $\tilde x = r(x)$, and train a network $f$ to predict which rotation was applied via cross-entropy over $4$ classes.

Why it works: To classify the rotation correct, $f$ must learn features that depend on global spatial layout

Pipeline (per training image $x$):

1. Apply two random augmentations $t, t' \sim \mathcal T$ (a fixed policy) to produce two views $\tilde x _ i = t(x), \tilde x _ j = t'(x)$.
2. Pass each through a shared encoder $f$ to get representations $h _ i = f(\tilde x _ i), h _ j = f(\tilde x _ j)$.
3. Pass each representation through a non-linear projection head $g$ (a small MLP) to get $z _ i = g(h _ i)$, $z _ j = g(h _ j)$.
4. Train to maximise the agreement between $z _ i$ and $z _ j$ via a contrastive loss (∆contrastive-loss): the positives are $(z _ i, z _ j)$, the negatives are all other batch elements.

Non-linear projection head: The contrastive loss forces $z$ to be invariant to whatever augmentations are used (e.g. colour jitter would make $z$ colour-invariant). This invariance is good for the loss, but discards information downstream tasks may need (e.g. classifying a red vs green apple). Inserting a non-linear $g$ between $h$ and $z$ lets $g$ absorb the “invariance pressure” while $h$ retains the richer features. Downstream users can take $h$ and discard $g$.

Example augmentations:

In unsupervised classification, a clustering algorithm produces $N$ cluster IDs with no semantic meaning. To evaluate against ground-truth labels, we need an optimal 1-to-1 assignment between clusters and labels.

Hungarian matching: given an $N \times N$ cost matrix $C$ where $C _ {ij}$ is the error of matching cluster $i$ to label $j$, find a permutation matrix $P$ minimising

in $O(N^3)$ time (i.e. the sum of the diagonal).

This separates the grouping (the model’s job) with the semantics (Hungarian’s job, naming the clusters). It finds the best possible assignment of clusters to true labels to minimise the error.

An alternating optimisation between learning a model and assigning pseudo-labels, both minimising the cross-entropy

• Learning step: fix labels $q$ and update CNN weights via $\min _ p H(q, p)$, which is equivalent to standard supervised training with the current pseudo-labels.
• Labelling step: fix the model $p$ and update label assignments via $\min _ q H(q, p)$, subject to $\sum _ i q(y \mid x _ i) = N/K$ for every class $y$ (each cluster receives an equal share of samples). Without this constraint, $\min _ q H$ collapses to the trivial solution of putting every sample in the single most-likely class.

Why the labelling step is optimal transport: with the equal-size constraint, the labelling problem has the structure of a discrete OT problem. Treat the $N$ samples as sources (mass $1$ each) and the $K$ classes as targets (each demanding mass $N/K$). The cost of assigning sample $i$ to class $y$ is the entry

so $H(q, p) = \tfrac{1}{N}\sum _ {i,y} q(y \mid x _ i)\, C _ {iy}$ is exactly the total transport cost. Minimising it over $q$ subject to the two marginals $\sum _ y q(y \mid x _ i) = 1$ (each sample fully assigned) and $\sum _ i q(y \mid x _ i) = N/K$ (equal class sizes) is the Kantorovich OT problem. Solved efficiently with the Sinkhorn-Knopp algorithm (entropy-regularised OT, Cuturi 2013): scale rows and columns of $\exp(-C/\varepsilon)$ iteratively to match the marginals, $O(NK)$ per iteration.

Iterate the two steps to convergence to obtain an unsupervised classifier. After training, ∆hungarian-matching maps cluster IDs to ground-truth labels for evaluation.

If your CNN architecture ends with a global average pooling layer before the rotation classifier, the rotation-prediction task becomes trivial in an unhelpful way:

GAP is invariant to spatial permutation — averaging across all spatial positions discards the spatial layout of the feature map. But the rotation classifier needs spatial layout to figure out which way is “up”. So the classifier ends up keying on spatial-invariant cues like aspect ratio of objects or colour gradients (e.g. sky tends to be lighter at the top), rather than learning genuine visual features.

The fix: don’t use global pooling before the rotation head. Use a head that preserves spatial information (flatten + FC, or attention over spatial positions). Then the network is forced to learn features that respect spatial layout, which generalise better to downstream tasks like classification.

This is the standard worked example of how pretext task design choices interact with architecture to determine whether self-supervision actually transfers.

• Jigsaw puzzle solving (Noroozi & Favaro, ECCV 2016): split an image into a 3×3 grid, shuffle the patches, train the network to predict the permutation (chosen from 1000 fixed permutations, treated as 1000-way classification).
• Context prediction (Doersch, Gupta, Efros, ICCV 2015): given a centre patch and one of its 8 neighbours, predict the relative position. 8-way classification.
• Inpainting (Pathak et al., CVPR 2016): mask out a random rectangular region and train the network to fill it in. Forces semantic understanding.
• Colourisation (Zhang, Isola, Efros, ECCV 2016): convert image to grayscale and train the network to predict the colour (or chromatic) channels. Forces understanding of object-level colour priors.

All four follow the recovery template: apply a known transformation, train the network to undo it. The learned features (encoder activations) transfer to downstream tasks like classification.

For a function $f$ and a transformation $\pi$:

• Invariance: $f(\pi(x)) = f(x)$ — applying the transformation to the input doesn’t change the output.
  • Example: an image classifier should be invariant to small translations of the image — a cat shifted by 5 pixels is still a cat.
• Equivariance: $f(\pi(x)) = \pi'(f(x))$ for some “matching” transformation $\pi'$ on the output space. Applying $\pi$ to the input changes the output in a predictable way.
  • Example: a semantic segmentation network should be equivariant to translation — translating the image translates the output mask by the same amount.

Convolutional layers are translation-equivariant by construction. Pooling adds partial translation invariance. Self-attention is permutation-equivariant unless positional encodings break the symmetry.

In unsupervised learning, both are used as objectives: SimCLR enforces invariance under data augmentations; rotation-prediction enforces equivariance with $\pi' = $ the rotation label.

The contrastive loss $\mathcal L(z _ i, z _ j) = -\log \frac{\exp \mathcal S(z _ i, z _ j)}{\sum _ k \exp \mathcal S(z _ i, z _ k)}$ forces $z$ to be invariant to whatever transformations are used during augmentation. If we augment with colour distortion, then $z$ must produce the same output for “red car” and “blue car” — i.e. colour information is destroyed.

But downstream classification needs colour information (distinguishing “red apple” from “green apple”, for example). If we used $h = z$ (no projection head), the encoder would have to destroy this information.

The fix is to insert a non-linear MLP $g$ between $h$ and $z$. The contrastive loss invariance pressure operates only on $z$; $h$ can retain richer features that are useful downstream. After training, the projection head $g$ is discarded and $h$ is used as the representation.

This was one of SimCLR’s biggest empirical findings: using $g$ rather than $h$ for the contrastive loss substantially improves downstream transfer.

The lecture’s own Twitter-poll-style discussion shows there’s no universally agreed definition. Common positions:

• “They are the same”: both produce a useful representation without human-annotated labels for the target task.
• “Self-supervised uses problem-specific principles to derive supervisory signal from the data itself” (e.g. predict-the-rotation, predict-the-relative-patch): so the code looks like supervised learning but with synthetic labels.
• “Unsupervised is older techniques that don’t work; self-supervised is newer methods that do”: half-joking but contains a kernel of truth — the modern self-supervised paradigm (SimCLR, MoCo, DINO) genuinely outperforms classical unsupervised methods (k-means, PCA, autoencoders) on representation learning.

A reasonable resolution: self-supervised is a technique (deriving labels from the data), and unsupervised is a goal (no human labels for the downstream task). Self-supervised methods are unsupervised methods that achieve this via the labels-from-data trick.

But: don’t lose sleep over the terminology distinction. Most papers use them interchangeably.