Computer Vision MT25, Precision and recall
Flashcards
Suppose a binary classifier produces predictions, and we partition the predictions into:
- $\text{TP}$ (true positive): predicted positive and actually positive.
- $\text{FP}$ (false positive): predicted positive but actually negative.
- $\text{FN}$ (false negative): predicted negative but actually positive.
- $\text{TN}$ (true negative): predicted negative and actually negative.
@Define precision and recall.
In words:
- Precision answers “of all the items I claimed were positive, what fraction actually were?”
- Recall answers “of all the items that actually were positive, what fraction did I find?”
There is a natural trade-off. Raising the decision threshold makes fewer positive predictions, so precision tends to go up (fewer FPs) and recall tends to go down (more FNs). Lowering the threshold does the opposite.
Explain what is being plotted on a graph like this.
The precision versus recall for many different thresholding values.
What is the AP metric?
The area under the precision-recall curve for a given classifier.
One way to evaluate an object detection model is to compute the average precision by calculating the area under the precision-recall curve. Typically, this will depend on some choice made about what to count as a false positive in terms of the threshold used by the IoU metric.
Given this, can you define the $\text{AP}$ metric in full generality and explain how it is an average, average, average precision?
where
\[\text{AP}(\text{class}) = \frac{1}{\#\text{IoU thresholds}} \sum _ {\text{iou} \in \text{thresholds}} \text{AP}(\text{class}, \text{iou})\]and $\text{AP}(\text{class}, \text{iou})$ is the area under the corresponding precision-recall curve.
It is an:
- average: average over the classes
- average: average over the IoU thresholds
- average: average over the precision at different recall levels
precision.
@State the four cells of the binary-classification confusion matrix and define error rate, false positive rate, and false negative rate from them.
- TP (true positive): predicted positive, outcome was negative.
- FP (false positive): predicted positive, outcome was negative.
- FN (false negative): predicted negative, outcome was positive.
- TN (true negative): predicted negative, outcome was negative.
Error rate, how often the prediction is wrong:
\[\text{error rate} = \frac{\text{FP} + \text{FN}}{\text{TN} + \text{FP} + \text{FN} + \text{TP}}\]False positive rate, among actual negatives, how often they were predicted positive:
\[\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}\]False negative rate, among actual positives, how often they were predicted negative:
\[\text{FNR} = \frac{\text{FN}}{\text{FN} + \text{TP}}\]Bite-sized
@Describe the precision-recall trade-off under varying confidence threshold and what extreme thresholds correspond to.
Sweeping the confidence threshold of a classifier from very high to very low:
- Very high threshold: only the most confident predictions count as positive. Almost all of these are true positives, so precision is high but recall is low (many genuine positives below the threshold are missed).
- Very low threshold: almost everything counts as positive. Recall is near 1 (you catch all true positives), but precision is low (most of your predictions are false alarms).
In between, you get the precision-recall curve — a one-parameter family of (precision, recall) operating points, one per threshold. The curve starts in the upper-left (high precision, low recall) and decays toward the lower-right (low precision, high recall).
This trade-off is why a single threshold-dependent metric (precision or recall or accuracy at one threshold) is misleading. The AP metric averages performance over all thresholds.
A precision-recall curve sweeps the confidence threshold of the classifier. Each threshold gives one (precision, recall) point: lower thresholds give more predictions (higher recall, lower precision); higher thresholds give fewer predictions (higher precision, lower recall).
@Justify why AP (area under the PR curve) is preferred over simple accuracy as an object-detection metric.
Object detection has heavy class imbalance: most candidate regions are background, and most pixels are not part of any object. So accuracy is dominated by the trivially-easy “predict no object everywhere” baseline, and provides no useful signal on actual detection quality.
AP fixes this:
- Ignores true negatives: the formula $\mathrm{precision} = \mathrm{TP}/(\mathrm{TP}+\mathrm{FP})$ and $\mathrm{recall} = \mathrm{TP}/(\mathrm{TP}+\mathrm{FN})$ both omit TN. So the trivial all-negative baseline scores zero AP rather than near-1 accuracy.
- Threshold-independent: AP integrates over confidence thresholds, so a single number captures behaviour across the precision-recall trade-off.
- Combines with per-class and per-IoU averaging to give mAP, the standard object-detection summary metric.
So AP/mAP is robust to class imbalance and gives a more interpretable summary of detector quality than any threshold-dependent metric.
In the AP triple-average, the innermost level (the area under the precision-recall curve for a single class at a single IoU threshold) averages precision over different recall levels, with one (precision, recall) point per confidence threshold. The two outer levels then average over IoU thresholds and over classes.
@Define each of TP, FP, FN, TN in plain English using the example of an email-spam classifier.
For a spam classifier predicting “spam” or “not spam”:
- TP (true positive): the email is spam and the classifier predicted “spam”. Correct catch.
- FP (false positive): the email is not spam but the classifier predicted “spam”. Wrongly flagging a legitimate email as spam — the more harmful error from the user’s perspective.
- FN (false negative): the email is spam but the classifier predicted “not spam”. Wrongly letting spam through.
- TN (true negative): the email is not spam and the classifier predicted “not spam”. Correctly leaving a legitimate email alone.
For spam, users care strongly about minimising FP (don’t lose my legitimate emails) and care less about FN (a bit of spam in the inbox is annoying but not destructive). For cancer screening, the priorities flip: FN is catastrophic, FP is acceptable. This stakeholder framing is the same idea behind Lecture 20’s COMPAS analysis.