# Notes - Computer Vision MT25, Optical flow

> Source: https://ollybritton.com/notes/uni/part-c/mt25/computer-vision/notes/optical-flow/ · Updated: 2026-05-20 · Tags: uni, notes

- [Course - Computer Vision MT25](https://ollybritton.com/notes/uni/part-c/mt25/computer-vision/)
	- [Notes - Computer Vision MT25, Video](https://ollybritton.com/notes/uni/part-c/mt25/computer-vision/notes/video/)

### Flashcards
Suppose we have an image at frame $t$ of a video given by $I(x, y, t)$, and we wish to calculate the optical flow $F(x, y, t)$ for this frame. What do we hope is satisfied mathematically by the computed $F(x, y, t)$?::

Writing $F(x, y, t) = (\Delta x, \Delta y)$, then $I(x + \Delta x, y + \Delta y, t + \Delta t) = I(x, y, t)$, i.e. the optical flow tells us where the pixel will move in the next frame.

Suppose:

- We have a pixel at $(x, y, t)$ with intensity $I(x, y, t)$
- This pixel has been moved by $\Delta x$, $\Delta y$ in space during a timestep $\Delta t$
- We assume the pixel does not change intensity
- The time and movement is small

Under these assumptions, @state the motion constraint equation.::

$$
\nabla \pmb I^\top \pmb \mu = -\frac{\partial I}{\partial t}
$$
where

- $\nabla \pmb I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right]^\top$
- $\pmb \mu = \left[ \frac{\Delta x}{\Delta t}, \frac{\Delta y}{\Delta t} \right]^\top$

Suppose:

- We have a pixel at $(x, y, t)$ with intensity $I(x, y, t)$
- This pixel has been moved by $\Delta x$, $\Delta y$ in space during a timestep $\Delta t$
- We assume the pixel does not change intensity
- The time and movement is small

Under these assumptions, derive the motion constraint equation
$$
\nabla \pmb I^\top \pmb \mu = -\frac{\partial I}{\partial t}
$$
where:

- $\nabla \pmb I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right]^\top$
- $\pmb \mu = \left[ \frac{\Delta x}{\Delta t}, \frac{\Delta y}{\Delta t} \right]^\top$

::

Under the assumption the time and movement is small, we have the Taylor approximation
$$
I(x + \Delta x, y + \Delta y, t + \Delta t) \approx  I(x, y, t) + \frac{\partial I}{\partial x} \Delta x + \frac{\partial I}{\partial y} \Delta y + \frac{\partial I}{\partial t} \Delta t
$$
Since we want $I(x + \Delta x, y + \Delta y, t + \Delta t) =  I(x, y, t)$, we have
$$
\frac{\partial I}{\partial x} \Delta x + \frac{\partial I}{\partial y} \Delta y + \frac{\partial I}{\partial t} \Delta t = 0
$$
Dividing through by $\Delta t$, we obtain
$$
\frac{\partial I}{\partial x} \frac{\Delta x}{\Delta t} + \frac{\partial I}{\partial y} \frac{\Delta y}{\Delta t} + \frac{\partial I}{\partial t} = 0
$$
which after substitutions is equivalent to the required form.

@prove~

The ∆motion-constraint-equation
$$
\nabla I^\top \pmb \mu = -\frac{\partial I}{\partial t}
$$
with per-pixel flow vector $\pmb \mu = (\mu_x, \mu_y)^\top$ is one equation in two unknowns and so cannot be solved pointwise (the ∆bite-aperture-problem).

Rather than solving it directly, @state the *Horn-Schunck objective* (Horn & Schunck, 1981), which turns flow estimation into a well-posed variational problem by adding a smoothness penalty on the flow field.

::

For a flow field $\pmb \mu : R \to \mathbb R^2$ on the image domain $R \subset \mathbb R^2$, with component scalar fields $\mu_x, \mu_y : R \to \mathbb R$ (these are *components* of $\pmb \mu$, *not* partial derivatives), the Horn-Schunck objective is
$$
\min_{\pmb \mu} \iint_R \underbrace{\left( \nabla I^\top \pmb \mu + \frac{\partial I}{\partial t} \right)^2}_{\text{brightness-constancy residual}} + \alpha \underbrace{\left( \|\nabla \mu_x\|^2 + \|\nabla \mu_y\|^2 \right)}_{\text{smoothness penalty}} \, \text dx \, \text dy
$$
where:

- $\nabla I = (\partial_x I, \partial_y I)^\top$ is the *spatial* image gradient.
- $\nabla \mu_x = (\partial_x \mu_x, \partial_y \mu_x)^\top$ is the *spatial* gradient of the scalar field $\mu_x$ (similarly $\mu_y$); $\|\cdot\|$ is the Euclidean norm.
- $\alpha > 0$ trades off data fidelity against smoothness. Pixels where $\nabla I \approx 0$ (textureless regions) get their flow from the smoothness term coupling them to neighbours, resolving the aperture problem.

**Source**: Lecture 13, **Regularisation** slide; Horn & Schunck, *Determining Optical Flow*, Artificial Intelligence 17 (1981).

Suppose:

- We have a pixel at $(x, y, t)$ with intensity $I(x, y, t)$
- This pixel has been moved by $\Delta x$, $\Delta y$ in space during a timestep $\Delta t$
- We assume the pixel does not change intensity
- The time and movement is small

Under these assumptions, derive the motion constraint equation
$$
\nabla \pmb I^\top \pmb \mu = -\frac{\partial I}{\partial t}
$$
where:

- $\nabla \pmb I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right]^\top$
- $\pmb \mu = \left[ \frac{\Delta x}{\Delta t}, \frac{\Delta y}{\Delta t} \right]^\top$

Rather than solving this directly, we instead use an objective which adds a regularisation parameter to make close pixels have similar flow:
$$
\min_{\pmb \mu} \iint_R \left[ \left( \nabla I^\top \pmb \mu + \frac{\partial I}{\partial t} \right)^2 + \alpha\left( |\nabla \mu_x|^2 + |\nabla \mu_y|^2 \right) \right] \text dx \text dy
$$
@Visualise how this affects the computed optical flow.::

![Screenshot 2025-11-13 at 16.03.39.png](https://ollybritton.com/assets/attachments/img/Screenshot 2025-11-13 at 16.03.39.png)

What are the main problems with intensity-based optical flow?::

- It struggles in uniform-coloured regions
- It is difficult to regularise
- It is difficult to evaluate

@Define the aperture problem, explain its geometric origin, and state the standard ways of resolving it.::

**Definition**: when estimating the 2D motion of a pixel from intensity information alone, the *aperture problem* is the fact that a single pixel observation can only determine the motion component *normal* to the local image gradient. The component *along* the gradient (i.e. along the edge direction) is undetermined.

**Geometric origin**: the brightness-constancy motion-constraint equation
$$\nabla I^\top \pmb \mu = -\frac{\partial I}{\partial t}$$
is *one equation in two unknowns* ($\mu_x, \mu_y$). It constrains $\pmb \mu$ to lie on a 1D affine subspace of $\mathbb R^2$ (a line normal to $\nabla I$ in velocity space). Equivalently, moving a long straight edge parallel to itself produces no temporal intensity change at any pixel along it, so motion along an edge is *invisible* to local intensity-difference measurements.

**Where the name comes from**: peering at a moving object through a small aperture, you only see a tiny patch, and you can only detect motion normal to whatever edge happens to pass through that patch. Motion sliding along the edge is hidden by the aperture's blind spots.

**Standard resolutions**:

- *Smoothness prior* (Horn-Schunck, ∆horn-schunck-objective): assume the flow field is locally smooth, coupling many under-determined per-pixel equations into a globally well-posed problem.
- *Local patch with varied gradient orientations* (Lucas-Kanade, ∆lk-update-rule-statement): collect the constraint equations from many pixels in a patch. If the local gradients $\nabla I$ point in different directions, the structure tensor $\pmb M = \sum \nabla I \nabla I^\top$ is invertible and the $2 \times 2$ system has a unique solution.
- *Feature tracking on corner-like keypoints*: corners and blobs have rich 2D gradient content by construction, sidestepping the aperture problem entirely (∆bite-good-keypoint-properties).

**Connection to LK invertibility**: in pure-translation LK, the structure-tensor matrix $\pmb M$ is invertible *iff* the template has gradient content in two non-parallel directions, which is exactly the condition for the aperture problem to be solvable locally (∆bite-lk-translation-2x2-system).

**Source**: Lec 13, **Regularisation** and **Recap: Template Tracking** slides; Marr & Ullman 1981.

@exam~

### Bite-sized

The Horn-Schunck algorithm for optical flow was introduced in Horn & Schunck, "Determining Optical Flow", *Artificial Intelligence* 17 (1981). It combines the brightness-constancy motion constraint with a smoothness regulariser to produce dense optical flow.

**Source**: Lecture 13, **Optical Flow -- The Beginnings** slide.

@bite~

@Justify the *aperture problem*: why the brightness-constancy equation alone cannot determine the full 2D flow vector.

::

The motion constraint $\nabla I^\top \pmb\mu = -\partial I / \partial t$ is *one equation in two unknowns* ($\mu_x, \mu_y$). It only constrains the component of $\pmb\mu$ in the direction of the image gradient $\nabla I$.

Geometrically: a single pixel observation only tells you motion *normal* to the edge passing through it. The motion *along* the edge is invisible — moving a long edge parallel to itself produces no temporal intensity change at any pixel along it.

This is the classic *aperture problem* (Marr & Ullman 1981): peering at a moving object through a small aperture, you can only detect motion normal to local edges.

Resolutions:

- Add a smoothness prior (Horn-Schunck), so neighbouring pixels share their motion.
- Use a local patch of multiple pixels with different gradient orientations (Lucas-Kanade), so you have multiple equations in the two unknowns.
- Use feature-tracking on corner-like keypoints with rich 2D gradient content.

**Source**: Lecture 13, **Regularisation** slide; implicit in **Limitations of Classical Optical Flow**.

@bite~

FlowNet (Dosovitskiy et al., ICCV 2015) was the first deep-learning approach to optical flow: a Siamese encoder with a *correlation layer* that computes $c(\pmb x_1, \pmb x_2) = \sum_{\pmb o} \langle \pmb f_1(\pmb x_1 + \pmb o), \pmb f_2(\pmb x_2 + \pmb o) \rangle$, followed by a decoder producing the flow. Trained on FlyingThings3D synthetic data.

**Source**: Lecture 1, **Optical Flow -- Learned: FlowNet** slide.

@bite~

RAFT (Teed & Deng, ECCV 2020) is the modern state-of-the-art classical optical flow architecture. Its key innovations are: (i) multi-scale 4D correlation volumes precomputed from feature maps, (ii) iterative GRU-based refinement (10+ iterations) of the flow estimate, (iii) a context encoder providing static features from frame 1.

**Source**: Lecture 1, **Optical Flow -- Now(ish): RAFT** slide.

@bite~

@Justify why optical flow is hard to evaluate quantitatively, and how synthetic datasets address this.

::

*Why hard*: ground-truth optical flow is essentially impossible to obtain for real scenes. You'd need pixel-accurate correspondences between consecutive frames — and that requires knowing the full 3D motion of every scene element, which is the problem you're trying to solve.

*Synthetic-dataset workaround*:

- **FlyingThings3D** (Mayer et al., CVPR 2016): randomly textured 3D objects flying past a camera in a synthetic scene. Since the renderer controls all 3D motion, exact per-pixel flow can be exported. Used to train FlowNet and RAFT.
- **MPI Sintel** (Butler et al., ECCV 2012): uses scenes from the open-source animated movie *Sintel* with photorealistic rendering. Provides ground-truth flow including motion blur, atmospheric effects, and large displacements.

These synthetic datasets enable supervised training of dense optical-flow networks despite the impossibility of collecting real-world ground truth. Sim-to-real transfer happens to work well for optical flow because it's a *low-level* vision task — local intensity gradients look similar across synthetic and real domains.

**Source**: Lecture 1, **Optical Flow -- Flying Things** and **Optical Flow -- Sintel** slides.

@bite~

@Describe what optical flow predicts conceptually, and how this differs from *point tracking* and from *stereo disparity*.

::

- **Optical flow**: a *dense* 2D displacement field $F(x, y, t) = (\Delta x, \Delta y)$ predicting where every pixel will be in the next frame. Output has the same resolution as the input. Tracks for one timestep only.
- **Point tracking** (e.g. PIPs, TAP-Net, CoTracker): tracks a *sparse* set of pre-specified query points across many frames. Each query point gets a long trajectory; other pixels are not tracked. Long-term but sparse.
- **Stereo disparity**: dense 1D displacement field along the epipolar line for a pair of *simultaneous* images from two cameras. The constraint is purely spatial (epipolar geometry) rather than temporal.

All three are *correspondence problems* (see [Notes - Computer Vision MT25, Correspondences](https://ollybritton.com/notes/uni/part-c/mt25/computer-vision/notes/correspondences/)), and modern architectures (RAFT for OF, CoTracker for PT, RAFT-stereo for disparity) share a common design language: correlation volumes + iterative refinement.

**Source**: Lecture 1, **Motion Estimation Summary** slide; Lecture 13, **A Note on Correspondences** slide.

@bite~

The *smoothness* (regularisation) term in the Horn-Schunck objective $\alpha (|\nabla \mu_x|^2 + |\nabla \mu_y|^2)$ uses the spatial gradient $\nabla$ (i.e. $\frac{\partial}{\partial x}, \frac{\partial}{\partial y}$ of the flow itself), not the Laplacian $\Delta$ . The regulariser penalises the flow field for changing rapidly between neighbouring pixels, encouraging smooth flow even where the intensity equation is underdetermined.

**Source**: Lecture 13, **Regularisation** slide.

@bite~

@State the strategy for deriving the optical-flow motion constraint equation (∆motion-constraint-derivation), in 3 steps without algebra.

::

- *Taylor-expand* $I(x + \Delta x, y + \Delta y, t + \Delta t)$ around $(x, y, t)$ to first order using the small-motion-and-small-time assumption, giving $I + \partial_x I \cdot \Delta x + \partial_y I \cdot \Delta y + \partial_t I \cdot \Delta t$.
- *Apply brightness constancy* $I(x + \Delta x, y + \Delta y, t + \Delta t) = I(x, y, t)$: the constant terms cancel, leaving the linear equation $\partial_x I \cdot \Delta x + \partial_y I \cdot \Delta y + \partial_t I \cdot \Delta t = 0$.
- *Divide by $\Delta t$* and identify $(\Delta x / \Delta t, \Delta y / \Delta t)$ as the per-second flow $\pmb \mu$, giving $\nabla I^\top \pmb \mu = -\partial_t I$.

The whole derivation is "first-order Taylor + brightness constancy + rescale time". Both load-bearing assumptions (linearisation valid, intensity preserved) are flagged on ∆bite-brightness-constancy-assumption; the aperture problem (one equation, two unknowns) follows directly from the resulting form.

**Source**: Lecture 13, **Optical Flow -- The Beginnings** slide.

@bite~ @proofsupport~

### Visualising the aperture problem

The ∆motion-constraint-equation $I_x u + I_y v + I_t = 0$ is a single line in velocity space, which is exactly the ∆aperture-problem: through one edge only the flow normal to it is recoverable, and the true velocity can slide anywhere along that line. Drag the true motion and the grating seen through the aperture looks unchanged. Switch to a corner and the second constraint line pins the velocity to the intersection — why Lucas-Kanade needs a corner or textured patch.

{% include optical_flow.liquid %}

---
Olly Britton — https://ollybritton.com. Machine-readable index: https://ollybritton.com/llms.txt