Computer Vision MT25, Video

You can train on short clips with low resolution and low FPS, and then at test time run the model on overlapping clips and average the predictions.

• Predict for each frame independently with an image model
• Average the probabilities for each frame together

The observation that for many actions, only one frame of a video of that activity is needed to classify that activity.

• Pooling
• Averaging

Low-level motion is often lost after compressing a frame into a feature vector.

Use 3D convolutions to slowly compress the temporal information.

• Single input image in the first stream: $3 \times H \times W$
• Stack of optical flow in the $x$ and $y$ directions across all the whole $T$ frames in the video: $[2 \times (T - 1)] \times H \times W$.

The tokens are spatio-temporal tokens: each token comes from a patch in space and time.

Tubelets.

We use two new primitives:

• ∆spatial-attention: attention performed spatially across the image, independently for each time step
• ∆temporal-attention: attention performed temporally across the video, independently for each spatial location

Then we have:

1. Factorised encoder:
   • Two separate transformer encoders: one spatial (per frame), one temporal (across frames).
   • Apply the spatial encoder to each frame independently to produce a per-frame summary, then apply the temporal encoder to the sequence of summaries.
2. Factorised self-attention:
   • Within each transformer block, alternate a spatial-attention layer and a temporal attention layer.
   • Each block does both, but at different sub-layers.
3. Factorised dot-product:
   • Spatial and temporal attention computed in parallel (not alternating), then fused.
   • Each attention head is dedicated to either spatial or temporal attention, and the results are concatenated.

• Permute: $B \times H \times W \times T \times D$
• Flatten: $BHW \times T \times D$
• Attention: $BHW \times T \times D \to BHW \times T \times D$ (so here the height and width basically become part of the batch, and are processed independently for each spatial location)
• Unflatten: $B \times H \times W \times T \times D$
• Permute: $B \times T \times D \times H \times W$

• Permute: $B \times T \times H \times W \times D$
• Flatten: $BT \times HW \times D$
• Attention: $BT \times HW \times D \to BT \times HW \times D$ (so here the time effectively becomes part of the batch, and each time step gets processed independently)
• Unflatten: $B \times T \times H \times W \times D$
• Permute: $B \times T \times D \times H \times W$

Full attention (every token attends to every other token across space and time):

• Total tokens: $THW$
• Complexity: $O((THW)^2)$

Factorised attention (alternating temporal and spatial tokens):

• Temporal attention: There is one $T \times T$ attention per $HW$ locations. Complexity $O(T^2 \times HW)$
• Spatial attention: Process each frame separately, attending across $HW$ spatial tokens. One $HW \times HW$ attention for each $T$ frames. Complexity $O(T \cdot (HW)^2)$.
• Total: $O(T^2 HW + T(HW)^2)$

The factorisation replaces a single $(THW)^2$ term with the sum of two smaller terms.

• Training: extract many short clips from each video (e.g. 16 frames at low FPS and low spatial resolution — e.g. $112 \times 112$). Each clip is one training sample. This makes individual forward/backward passes affordable and lets the GPU process whole batches.
• Testing: process overlapping clips across the test video and average the per-clip predictions. The overlap ensures no portion of the video is missed; the averaging smooths over per-clip noise.

This mirrors the patch-then-aggregate pattern used elsewhere in CV (e.g. sliding-window classification, dense prediction). The principle: don’t try to process the whole signal at once — break it into manageable pieces and aggregate.

A 2D convolution takes an input of shape $(C, H, W)$, convolves with $N$ filters of shape $(C, h, w)$, and produces an output of shape $(N, H, W)$ (with appropriate padding). The convolution slides spatially in 2D but operates over all channels.

A 3D convolution generalises this to a third dimension (depth or time):

• Input: $(C, D, H, W)$ where $D$ is the depth/time dimension.
• Filters: $N$ filters of shape $(C, d, h, w)$.
• Output: $(N, D, H, W)$ (with appropriate padding).

The convolution now slides in three directions: depth, height, and width. The channel dimension is still consumed by the dot product, just as in 2D.

Naming convention: “3D conv” refers to spatial-and-time, not 4D. The channel dimension is “ignored” in the naming. So a 3D conv has 5D weight tensors and 4D feature tensors.

3D conv pooling, fully connected layers, and activations all work analogously.

A full attention layer over a $T \times H \times W$ spatial-temporal token grid has $THW$ tokens, so naive self-attention is $O((THW)^2)$ in operations. For a modest $T = 16, H = W = 14$ this is $(16 \cdot 196)^2 \approx 10^7$ operations per layer, which is barely affordable; doubling any dimension makes it infeasible.

Factorised attention separates the costs:

• Spatial attention: at each timestep, attend across $HW$ spatial tokens. Cost per timestep is $(HW)^2$; total $T \cdot (HW)^2$.
• Temporal attention: at each spatial location, attend across $T$ temporal tokens. Cost per location is $T^2$; total $T^2 \cdot HW$.

Sum: $O(T^2 HW + T (HW)^2)$. Compare with $(THW)^2 = T^2 H^2 W^2$:

• The full cost is proportional to $T^2 H^2 W^2$.
• The factorised cost is proportional to $T(HW)^2 + T^2(HW) = (T + HW) \cdot THW$ (ish).

So we go from quadratic in $THW$ to roughly linear in $THW$ multiplied by the smaller factor. For typical video sizes where $HW \gg T$, factorised attention is ~$\frac{1}{T}$ as expensive.

Trade-off: each block can no longer model arbitrary spatio-temporal interactions in a single attention pass. But by stacking spatial + temporal attentions, the model can still represent all such interactions across depth — just at much lower cost per block.