Computer Vision MT25, Interpreting vision models

• Recipient: Explanations need to adapt to the recipient of the information.
• Content: Explanations provide different types of information.
• Purpose: Explanations differ based on use-cases.

• Post-hoc analysis: Explanations are derived from a fixed, pre-trained model via analysis.
• Transparent models: The model is specifically constructed such that some mechanism has semantic meaning.
• Learned explanations: The model is trained to deliver explanations together with predictions.

• Pro: There is no impact on performance.
• Con: Typically very difficult.
• Con: Explanations are often local around predictions.

• Pro: Does not require difficult post-hoc analysis.
• Con: Requires a task-specific architecture.
• Con: Can affect performance, there is a trade-off between explainability vs capabilities.

• Pro: Explanations can be very semantic.
• Con: Might need meta-explanations; you are always unsure if the explanation is a true explanation.
• Con: Can affect performance.

Use PCA to reduce to $2 \times 1000$ and plot.

Compute the activations, use PCA, then visualise the embedding with class labels.

A non-linear (although less interpretable) embedding, sort of like PCA.

1. See if it makes a randomly initialised network appear interpretable.
2. Train another model on the same data but random labels, and see if this model also seems interpretable.

Search for inputs that maximise class probabilities.

A black-box interpretability method where you occlude a part of the image and measure the change in response; the bigger the change, the more important the region.

If the image is already black there, it will make no difference.

Plot the gradient magnitude $ \vert \nabla _ x f(x) \vert _ 1$, i.e. see which direction does the input need to change in order to affect the output the most.

• RemOve And Retrain
• Run your attribution method, delete X% of the most important pixels
• Retrain your network on this data, see how much performance changes

Clever Hans was a German horse (1895-1916) that appeared to do arithmetic by tapping its hoof. A formal investigation revealed the horse was actually reading subtle cues from its trainer’s body language. The trainer was unaware of providing such cues.

The CV analogue: a classifier appears to perform a task well, but is actually exploiting a spurious correlation in the training data rather than the intended visual content.

Lecture 9’s worked example: a horse classifier on PASCAL VOC achieves ~90% accuracy. Visualising the heatmap reveals it’s not looking at the horse — it’s keying on the copyright notice that appears in the lower part of many horse photographs. Across the dataset average, the most “activated” region of horse images is exactly where copyright notices sit.

So the model has “learned” not the visual concept of horse, but the visual concept of copyright-notice-likely-on-horse-photo. This is why test-set accuracy alone is insufficient to validate a model.

• Black-box: only the inputs and outputs of the model are observed. The attribution must be inferred from how the output changes when the input is perturbed. Example: occlusion, where a sliding patch of the image is blocked and the change in target-class probability is recorded. Slow (many forward passes) and depends on the form of perturbation.
• White-box: the internal weights and computations of the model are accessible. Attribution leverages this access directly. Example: gradient methods, which take $ \vert \nabla _ {\pmb x} f(\pmb x) \vert _ 1$ to find the input directions that change the output most. Fast (one back-pass) and continuous, but sensitive to ReLU/pooling choices.

The lecture argues white-box methods are typically harder to make robust (they can be largely independent of the network weights — failing sanity checks), whereas black-box methods are easier to interpret but slower.

• Stochasticity: t-SNE is initialised randomly and optimised via gradient descent; each run produces a different embedding for the same data. Different seeds can produce visually quite different cluster shapes, so any particular t-SNE figure must not be over-interpreted as canonical.
• Distances are not preserved: t-SNE optimises for local neighbourhood preservation only. Inter-cluster distances and absolute scale are essentially meaningless — two clusters that look far apart in t-SNE may actually be close in the original space, and vice versa.
• Cluster sizes are not informative: t-SNE tends to equalise cluster sizes, so a small visual cluster does not necessarily reflect a small number of points.
• Non-linear: unlike PCA, there is no analytic interpretation of the t-SNE coordinates.

So t-SNE is good for exploratory hints about clustering structure, but bad for quantitative claims about distances or proportions.

The gradient-ascent procedure $\pmb x \leftarrow \pmb x + \eta \nabla _ {\pmb x} p(\text{class} \mid \pmb x)$ has no constraint that $\pmb x$ should look like a natural image. So the optimisation finds an adversarial example: a high-frequency, low-amplitude perturbation pattern that happens to maximally activate the target class. These look like random-coloured noise to a human.

To get class-canonical visualisations, one must add regularisers that push $\pmb x$ toward the natural-image manifold:

• Smoothness regulariser (total variation $L _ 1$ on image gradients) reduces high-frequency noise.
• Image jittering: randomly shift $\pmb x$ by a few pixels each iteration, forcing the optimum to be robust to small spatial perturbations.
• More sophisticated regularisers (Olah et al., Distill.pub) give significantly better visualisations.

This same observation underlies adversarial examples generally: neural network output landscapes are extremely non-smooth in input space.

ViT-style attention maps offer many choices for what to visualise:

• which layer (out of $\sim 12$),
• which token’s attention pattern,
• which attention head (out of $\sim 12$),
• whether to look at a single head or aggregate.

That’s many candidate visualisations per input. With this much freedom, it is almost always possible to find some configuration where the attention map appears to “highlight the right thing” — irrespective of whether the model is genuinely attending to that region for its decision.

So a single pretty attention-map figure can be cherry-picked, and on its own provides weak evidence about what the model is doing. Robust interpretability claims need cross-layer/head aggregation, sanity checks, and quantitative metrics like ROAR rather than single-image visual storytelling.