Computer Vision MT25, Optical flow

Created: November 11, 2025 | Updated: November 13, 2025 | About these notes | View in context | Study these flashcards

Flashcards

Suppose we have an image at frame $t$ of a video given by $I(x, y, t)$, and we wish to calculate the optical flow $F(x, y, t)$ for this frame. What do we hope is satisfied mathematically by the computed $F(x, y, t)$?
  • $F(x, y, t) = (\Delta x, \Delta y)$
  • $I(x + \Delta x, y + \Delta y, t + \Delta t) = I(x, y, t)$

i.e. the optical flow tells us where the pixel will move in the next frame.

Suppose:

  • We have a pixel at $(x, y, t)$ with intensity $I(x, y, t)$
  • This pixel has been moved by $\Delta x$, $\Delta y$ in space during a timestep $\Delta t$
  • We assume the pixel does not change intensity
  • The time and movement is small

Under these assumptions, @state the motion constraint equation.
\[\nabla \pmb I^\top \pmb \mu = -\frac{\partial I}{\partial t}\]

where

  • $\nabla \pmb I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right]^\top$
  • $\pmb \mu = \left[ \frac{\Delta x}{\Delta t}, \frac{\Delta y}{\Delta t} \right]^\top$

Suppose:

  • We have a pixel at $(x, y, t)$ with intensity $I(x, y, t)$
  • This pixel has been moved by $\Delta x$, $\Delta y$ in space during a timestep $\Delta t$
  • We assume the pixel does not change intensity
  • The time and movement is small

Under these assumptions, derive the motion constraint equation

\[\nabla \pmb I^\top \pmb \mu = -\frac{\partial I}{\partial t}\]

where:

  • $\nabla \pmb I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right]^\top$
  • $\pmb \mu = \left[ \frac{\Delta x}{\Delta t}, \frac{\Delta y}{\Delta t} \right]^\top$

Under the assumption the time and movement is small, we have the Taylor approximation

\[I(x + \Delta x, y + \Delta y, t + \Delta t) \approx I(x, y, t) + \frac{\partial I}{\partial x} \Delta x + \frac{\partial I}{\partial y} \Delta y + \frac{\partial I}{\partial t} \Delta t\]

Since we want $I(x + \Delta x, y + \Delta y, t + \Delta t) = I(x, y, t)$, we have

\[\frac{\partial I}{\partial x} \Delta x + \frac{\partial I}{\partial y} \Delta y + \frac{\partial I}{\partial t} \Delta t = 0\]

Dividing through by $\Delta t$, we obtain

\[\frac{\partial I}{\partial x} \frac{\Delta x}{\Delta t} + \frac{\partial I}{\partial y} \frac{\Delta y}{\Delta t} + \frac{\partial I}{\partial t} = 0\]

which after substitutions is equivalent to the required form.

@prove~

Suppose:

  • We have a pixel at $(x, y, t)$ with intensity $I(x, y, t)$
  • This pixel has been moved by $\Delta x$, $\Delta y$ in space during a timestep $\Delta t$
  • We assume the pixel does not change intensity
  • The time and movement is small

Under these assumptions, derive the motion constraint equation

\[\nabla \pmb I^\top \pmb \mu = -\frac{\partial I}{\partial t}\]

where:

  • $\nabla \pmb I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right]^\top$
  • $\pmb \mu = \left[ \frac{\Delta x}{\Delta t}, \frac{\Delta y}{\Delta t} \right]^\top$

Rather than solving this directly, @state the objective which adds a regularisation parameter to make close pixels have similar flow.
\[\min _ {\pmb \mu} \iint _ R \left[ \left( \Delta I^\top \pmb \mu + \frac{\partial I}{\partial t} \right)^2 + \alpha\left( \vert \nabla \mu _ x \vert ^2 + \vert \nabla \mu _ y \vert ^2 \right) \right] \text dx \text dy\]

Suppose:

  • We have a pixel at $(x, y, t)$ with intensity $I(x, y, t)$
  • This pixel has been moved by $\Delta x$, $\Delta y$ in space during a timestep $\Delta t$
  • We assume the pixel does not change intensity
  • The time and movement is small

Under these assumptions, derive the motion constraint equation

\[\nabla \pmb I^\top \pmb \mu = -\frac{\partial I}{\partial t}\]

where:

  • $\nabla \pmb I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right]^\top$
  • $\pmb \mu = \left[ \frac{\Delta x}{\Delta t}, \frac{\Delta y}{\Delta t} \right]^\top$

Rather than solving this directly, we instead use an objective which adds a regularisation parameter to make close pixels have similar flow:

\[\min _ {\pmb \mu} \iint _ R \left[ \left( \Delta I^\top \pmb \mu + \frac{\partial I}{\partial t} \right)^2 + \alpha\left( \vert \nabla \mu _ x \vert ^2 + \vert \nabla \mu _ y \vert ^2 \right) \right] \text dx \text dy\]

@Visualise how this affects the computed optical flow.

What are the main problems with intensity-based optical flow?
  • It struggles in uniform-coloured regions
  • It is difficult to regularise
  • It is difficult to evaluate


Related posts