Computer Vision MT25, Representation learning

Generally you learn a function from the input to task output, in representational learning you learn a general-purpose representation which can then be used for downstream tasks.

• Supervised: Given a specific task, learn a domain-specific representation which is often constrained to this particular task.
• Unsupervised: Given only the data, find a representation for it which often does not align exactly with the task.

• It’s hard to find the “discriminative signature” for a problem
• Even if you find the discriminative signature, it can be hard to implement programmatically
• Many different signals need to be combined, which itself is a difficult problem

The SIFT descriptor.

• Use a dataset with known point correspondences
• Extract patches around keypoints
• Use positive examples of matching keypoints and negative examples using random keypoints
• Train a model to predict the similarity between two patches

All three map a pair of patches to a similarity score. They differ in how early the two patches interact, and whether a reusable per-patch descriptor falls out.

2-channel (left): stack the two patches into a single 2-channel image and feed it through one network topped by a decision layer. The patches interact from the first layer, making it the most expressive. The drawback is that there is no separable descriptor, so the whole network must be re-run for every pair.

Siamese / pseudo-siamese (middle): one branch network per patch, whose outputs are concatenated and passed to a decision network. Siamese ties the two branches’ weights (shared), pseudo-siamese leaves them unshared. Each branch output is a reusable descriptor, so one patch can be compared against many cheaply.

Central-surround two-stream (right): a multi-resolution design. Each patch is split into a downsampled surround crop and a high-resolution central crop. A surround stream and a central stream, each a siamese pair of branches, process these separately, and both streams feed a shared decision network.

• LIFT: Learned Invariant Feature Transform
• In SIFT, the keypoints to choose and their corresponding descriptors are determined by a hand-crafted algorithm.
• In LIFT, the keypoints to choose and their corresponding descriptors are determined using a trained model.

• Might not generalise to unseen domains
• Typically slower than hand-crafted
• Less interpretable

A set of samples $\{ x _ i \}$, subdivided into positive $\{ x _ {i, j}^+ \}$ and negative $\{ x _ {i, k}^- \}$ examples.

Let $J _ i$ be the number of positive examples and $K _ i$ be the number of negative examples. Then:

where $\mathcal S _ {\cos}$ denotes cosine similarity. Since these sums might be large, it’s typically approximated by choosing randomly one positive and negative example:

It forces positive pairs to be almost identical and negative examples to be fully dissimilar.

where $\epsilon$ is some constant.

Intuitively, this is enforcing a relative order on the similarities of positive and negative examples.

This requires three evaluations of $f$ per loss calculation, which might be slow if $f$ is a complicated neural network.

If training over a batch, then you can save calculating the embeddings for lots of negative samples by using the embeddings of the other batch elements as negative examples.

where:

• $\phi _ i = f(x _ i)$ and $\phi _ i^+ = f(x _ i^+)$ are the embeddings of the anchor and its positive
• $\mathcal S$ is a similarity function, typically cosine similarity $\mathcal S(u, v) = u^\top v / (\|u\| \, \|v\|)$, or just a dot product
• the sum in denominator runs over all $B$ batch elements (the other $B-1$ elements act as negatives)

Imagine you had a classifier that computed the embeddings $\phi _ i$ of the input, and then at the final layer computed the similarities across all the classes. Assuming the similarity function $\mathcal S$ were fixed, contrastive loss is the cross-entropy loss you would use to tune the embedding model.

Then we use cross-entropy loss to make the diagonal entries close to $1$ and the off-diagonal entries $0$.

:

# img_emb : image model embedding [n, dim]
# txt_emb : text model embedding [n, dim]
# t_prime, b : learnable temperature and bias
# n : mini-batch size

t = exp(t_prime)
zimg = l2_normalize(img_emb)
ztxt = l2_normalize(txt_emb)
logits = dot(zimg, ztxt.T) * t + b
labels = 2 * eye(n) - ones(n)   # -1 with diagonal 1
l = -sum(log_sigmoid(labels * logits)) / n

It is focussed on efficiency and multi-GPU training.

Ranking loss generalises the triplet loss (∆triplet-loss-definition) from one positive and one negative per anchor to many positives and many negatives.

• Goal: every positive should be more similar to the anchor than every negative, so all positives outrank all negatives in similarity.
• Ranked positives: sometimes the positives themselves carry an ordering (some are more relevant than others), and the loss should reproduce that ranking, not merely push every positive above every negative.
• Construction: a ranking loss is typically assembled from pairwise terms (e.g. a triplet or hinge term for each positive-negative pair) summed over all pairs.

Why it matters: this is the natural loss for retrieval, where the output is an ordered list and getting the order right, not just which items are returned, is what counts.

In retrieval problems, there is either not a predefined set of classes or the set of classes is too large to train a classifier. For representation learning we don’t require a list of classes, and can just instead return similar examples.

Without negatives, a representation-learning objective that only enforces “matching pairs should be close” can be minimised trivially by collapsing the embedding to a constant: $f(\pmb x) = \pmb c$ for all $\pmb x$. Then any pair has cosine similarity 1, the positive-pair loss is zero, and the model has learned nothing useful.

Negatives prevent collapse by adding the opposing pressure that non-matching pairs should be dissimilar. This forces the model to use the embedding space’s full geometry — and explains why contrastive losses (InfoNCE, NT-Xent, triplet) all involve a negative term.

DINO avoids explicit negatives but only by using centring + sharpening of the teacher distribution — a different anti-collapse trick. The principle is the same: something has to push apart non-matching content.

A photo of an apple with a sticky note saying “iPod” stuck to it. CLIP’s classification:

• Prompt “an apple”: 42% confidence on iPod image, 78% on real apple. Without the text being relevant, CLIP still leans correctly toward “apple” — but only weakly.
• Prompt “a picture of an apple”: 56% / 91%.
• Prompt “an ipod”: 60% confidence on iPod-labelled apple, 0% on plain apple.
• Prompt “an apple with a note saying ‘ipod’”: 99% / 0% — i.e. CLIP can correctly identify the whole adversarial configuration if you describe it explicitly.

The key takeaway: CLIP’s training data has many images that contain text, so CLIP has implicitly learned that the text in an image is a strong cue for the image’s class. A sticky note saying “iPod” overrides the visual content. This is an adversarial attack by text and works against many production VLMs.

Goal: learn a common embedding space for two modalities (e.g. image + text, image + audio, vision + 3D), where matching cross-modal pairs land close together.

Recipe:

1. One encoder per modality: $f _ A : \mathcal D _ A \to \mathbb R^d$, $f _ B : \mathcal D _ B \to \mathbb R^d$. Architectures differ (CNN for images, transformer for text, etc.) but the output dimension $d$ is shared.
2. Training pairs: data is naturally paired across modalities (image with caption, video with soundtrack, photo with text description).
3. Contrastive loss across modalities: pull matching cross-modal pairs together (high similarity), push non-matching pairs apart. CLIP-style symmetric InfoNCE is the standard form.
4. Optional projection heads: small MLPs after each encoder to project into a shared “contrastive space” without disturbing the deeper features (SimCLR-style).

Resulting embedding space lets you do zero-shot classification (compare image embedding to text embeddings of class names) and cross-modal retrieval (find images matching a text query, or vice versa).

• Cosine similarity loss enforces an absolute target: positive pairs should approach similarity 1, negative pairs should approach similarity −1 (or 0, depending on normalisation). This is a strong constraint that may not be satisfiable — e.g. two views of the same object will never have identical feature vectors after data augmentation, so the loss can never reach 0.
• Triplet loss $\max(0, \mathcal S(\phi, \phi^-) - \mathcal S(\phi, \phi^+) + \varepsilon)$ enforces a relative target: positive should just be more similar than negative by some margin $\varepsilon$. This is a much weaker constraint that always has feasible solutions.
• The triplet loss is also zero once the margin is satisfied, so easy-correctly-classified triplets do not pull on training, and the loss focuses on hard examples (those still violating the margin).

This makes triplet loss numerically more stable and converges better in practice, especially when paired with hard-negative mining to find difficult-but-informative triplets.