Computer Vision MT25, Camera models

The observation that one image is not enough to determine the true location of objects in the image (even given perfect knowledge about the camera itself)).

The coordinate system where:

• The optical centre is at the origin (i.e. the location of the pinhole in the pinhole camera model, or the effective centre of projection more generally)
• The $z$ axis is the optical axis, perpendicular to the image plane
• The $xy$ plane is parallel to the image plane, $x$ is horizontal and $y$ is vertical

Setup (the ∆camera-coordinate-system): the camera is the optical centre $O$ (the pinhole), placed at the origin. The optical axis is the $z$-axis, and the image plane is perpendicular to it at distance $f$ (the focal length). A scene point $P = (x, y, z)$ sits at depth $z$; its image $P'$ is where the ray $P \to O$ meets the image plane.

Projection: the diagram is the side ($z$-$y$) view. Triangle $OP$ (legs $z$ and $y$) is similar to triangle $OP'$ (legs $f$ and $y'$), so $y' / f = y / z$, and likewise for $x$. Hence

The depth $z$ in the denominator is what makes distant objects appear smaller.

Matrix form: in homogeneous coordinates the whole projection is one matrix multiply, with the perspective division by $z$ applied last:

It is where the distance from the centre of the projection to the image plane is infinite.

• The camera centre is at the origin
• The principal axis is the $z$-axis
• $x$ and $y$ axes of the image plane are parallel to axes of the world

Camera calibration is the problem of determining the transformation from the world coordinate system to the image coordinate system.

3pt] 0^{\mathsf T} & 1 \end{pmatrix}\ (4\times 4) \end{pmatrix} \begin{pmatrix} \text{3D}\ \text{point}\ \boldsymbol{X}\ (4\times 1) \end{pmatrix}

\pmb x \quad\cong\quad \mathbf K[\mathbf R \mid \pmb t] \pmb X \quad\cong\quad \mathbf P \mathbf X

$$

• $\pmb p$: The point where the principal axis (i.e. the one perpendicular to the image plane coming from the world) intersects the image plane
• In the normalised coordinate system: the centre of the image
• In the image coordinate system: the corner of the image

3pt] \pmb 0^{\top} & 1 \end{bmatrix} \end{aligned}

$$

where:

• $\mathbf R$ and $\mathbf t$ is a rotation that maps from the world coordinates to the normalised camera coordinates
• $f$ is the focal length
• $p _ x, p _ y$ is the location of the principal point in image coordinates
• $m _ x, m _ y$ are the number of pixels per $m$ in the horizontal and vertical directions

Forget all the parameters in $\mathbf P$ and instead consider it as

Given $n$ points with known 3D coordinates $\mathbf X _ i$ and image projections $\pmb x _ i$, we have

Writing

therefore

or equivalently

Collecting this into a matrix equation, we obtain

Repeating this for the $n$ points, we obtain

where

which can be solved via least squares.

Linear calibration recovers $\mathbf P$ has 11 free parameters, but not in the factored form $\mathbf P = \mathbf K[\mathbf R \mid t]$, so you can’t directly obtain the explicit intrinsic and extrinsic parameters.

You instead formulate the problem of determining the parameters via a loss function and solve via nonlinear optimisation methods.

A 3D rotation is any matrix $R \in \mathbb R^{3 \times 3}$ satisfying $R^\top R = I _ 3$ (orthogonal) and $\det R = +1$ (orientation-preserving). These are the defining properties of $\mathrm{SO}(3)$.

Rotations about the coordinate axes (right-hand rule, counter-clockwise viewed from the positive end of the axis):

Mnemonic: the $1$ sits in the row and column of the rotation axis, and the other $2 \times 2$ block is the standard 2D rotation matrix (∆bite-2d-rotation-matrix), with the sign convention flipped for $R _ y$ to keep the right-hand rule.

Degrees of freedom: 3. The upper bound comes from counting: 9 entries minus 6 orthonormality constraints (three for unit-length columns, three for pairwise orthogonality). The matching lower bound comes from the Euler-angle parametrisation: any rotation factors as $R = R _ z(\alpha) R _ y(\beta) R _ x(\gamma)$, a smooth surjection $\mathbb R^3 \to \mathrm{SO}(3)$, so $\dim \ge 3$.

Useful corollary: $R^{-1} = R^\top$, so inverting a rotation is free.

The extrinsic matrix $[R \mid \pmb t]$ is the rigid transformation that converts world coordinates into camera coordinates:

• $R \in \mathrm{SO}(3)$ is the rotation that aligns the world axes with the camera axes. Equivalently, its rows are the world-frame coordinates of the camera’s $x, y, z$ basis vectors.
• $\pmb t \in \mathbb R^3$ is the position of the world origin expressed in camera coordinates. It is where the world origin appears from the camera’s point of view.

Common confusion: $\pmb t$ is not the camera position in the world. The camera position in world coordinates, $\pmb C$, is the world point that gets mapped to the camera’s own origin: $\pmb 0 = R\pmb C + \pmb t$, giving

So the camera-in-world is $\pmb C$, but the world-in-camera is $\pmb t$, and the two are related by the rotation $R^\top$ and a sign flip. Treating $\pmb t$ as the camera position will reliably break reconstructions.

Why $[R \mid \pmb t]$ is the natural form for the extrinsics: the camera projection $\pmb x \cong K[R \mid \pmb t]\pmb X$ first sends a world point through the rigid transformation $\pmb X \mapsto R\pmb X + \pmb t$ to put it in camera coordinates, then applies $K$ to project it into pixels. The decomposition is “where the point is from my perspective, then how my sensor digitises it”, which is the natural pipeline of the camera.

The camera centre $\tilde{\pmb C} \in \mathbb R^4$ in homogeneous world coordinates is the right null vector of $P$:

Geometric reason: a generic 3D point $\pmb X$ projects to a 2D image point via the ray from $\pmb X$ through the camera centre $\pmb C$, and the projection picks out where this ray meets the image plane. For $\pmb X = \pmb C$ the ray collapses to zero length, so the projection is undefined. Algebraically, “undefined projection” corresponds to $P\tilde{\pmb C} = \pmb 0$, the zero vector in homogeneous coordinates, which represents no valid projective point. So the camera centre is the unique point in the right null space of $P$.

Algebraic verification from the factored form: with $P = K[R \mid \pmb t]$, set $\tilde{\pmb C} = \binom{\pmb C}{1}$ where $\pmb C = -R^\top \pmb t$. Then

so $\pmb C = -R^\top \pmb t$ in inhomogeneous world coordinates.

Computational note: in the linear-calibration setting where $P$ is recovered as a flat $3 \times 4$ matrix without explicit $K, R, \pmb t$, you get $\tilde{\pmb C}$ from the SVD of $P$ as the right singular vector corresponding to its zero (smallest) singular value, then dehomogenise.

• Radial distortion: the non-linear loss can incorporate lens-distortion parameters so the model fits real (non-pinhole) lenses.
• Constraints: known focal length, orthogonality of $\mathbf R$, visibility of points, etc., can be enforced directly in the objective.
• Initialisation: run linear calibration first, then perform QR-decomposition of $\mathbf P = [\mathbf P _ {1{:}3} \mid \pmb p _ 4]$ to extract initial estimates of $\mathbf K$ and $\mathbf R$, then refine non-linearly.

The weak-perspective camera is the limit of perspective projection when the distance from the camera centre to the image plane (focal length / depth ratio) becomes very large — equivalently, the limit as $f \to \infty$ while subject distance also grows so depth variation $\Delta z / z \to 0$.

In this limit the projection becomes essentially $(x, y, z) \mapsto (f \, x / \bar z, \, f \, y / \bar z)$ for a single representative depth $\bar z$, so it acts like a scaled orthographic projection: no perspective foreshortening within the object.

Good approximation when: long focal lengths, faraway subjects, or shallow scenes (object depth small compared to overall distance). The lecture demonstrates this with a sequence of photos of the same building taken at increasing focal lengths — the perspective convergence steadily disappears.

• Parameterise freely: forget the factored form $\mathbf P = \mathbf K[\mathbf R \mid \pmb t]$ and treat the $3 \times 4$ projection matrix as 12 unknown entries $p _ {ij}$ (11 DoF after the up-to-scale equivalence).
• Translate each correspondence into linear equations: the projective equality $\pmb x _ i \cong \mathbf P \mathbf X _ i$ kills the unknown scale by cross-multiplying — each 2D-3D pair gives 2 linearly independent rows in the entries of $\mathbf P$.
• Stack into a homogeneous system: collecting all $n$ correspondences gives $\mathbf A \pmb p = \pmb 0$ with $\mathbf A \in \mathbb R^{2n \times 12}$; need $n \ge 6$ to over-determine the 11 DoF.
• Solve as homogeneous LS: take $\pmb p$ to be the right singular vector of $\mathbf A$ associated with the smallest singular value (equivalently, the smallest-eigenvalue eigenvector of $\mathbf A^\top \mathbf A$), then reshape back into a $3 \times 4$ matrix.

This is the same “stack-and-SVD” recipe as the homography solve (∆homography-linear-system-solution) and the eight-point algorithm (∆eight-point-algorithm-derivation) — the only differences are the dimensions and the post-solve rank-enforcement step.