Optical flow

The concept of optic flow has roots as far back as Euclid's Optics, but its modern formulation arose from Second World War research into pilot vision during landing. Several researchers arrived at the idea independently; James J. Gibson gave it its most influential treatment, publishing his theory in 1947 and created the term "optic flow" in 1950.

The term optical flow is also used by roboticists, encompassing related techniques from image processing and control of navigation including motion detection, object segmentation, time-to-contact information, focus of expansion calculations, luminance, motion compensated encoding, and stereo disparity measurement.

Estimation

Optical flow can be estimated in a number of ways. Broadly, optical flow estimation approaches can be divided into machine learning based models (sometimes called data-driven models), classical models (sometimes called knowledge-driven models) which do not use machine learning and hybrid models which use aspects of both learning based models and classical models.

Classical models

Many classical models use the intuitive assumption of brightness constancy; that even if a point moves between frames, its brightness stays constant. To formalise this intuitive assumption, consider two consecutive frames from a video sequence, with intensity ${\displaystyle I(x,y,t)}$, where ${\displaystyle (x,y)}$ refer to pixel coordinates and ${\displaystyle t}$ refers to time. In this case, the brightness constancy constraint is

${\displaystyle I(x,y,t)-I(x+u,y+v,t+1)=0,}$

where ${\displaystyle \mathbf{w}:=(u,v)}$ is the displacement vector between a point in the first frame and the corresponding point in the second frame. By itself, the brightness constancy constraint cannot be solved for ${\displaystyle u}$ and ${\displaystyle v}$ at each pixel, since there is only one equation and two unknowns. This is known as the aperture problem. Therefore, additional constraints must be imposed to estimate the flow field.

Regularized models

Perhaps the most natural approach to addressing the aperture problem is to apply a smoothness constraint or a regularization constraint to the flow field. One can combine both of these constraints to formulate estimating optical flow as an optimization problem, where the goal is to minimize the cost function of the form,

${\displaystyle E={\iint }_{\mathrm{\Omega }}\mathrm{\Psi }(I(x+u,y+v,t+1)-I(x,y,t))+\alpha \mathrm{\Psi }(|\mathrm{\nabla }u|)+\alpha \mathrm{\Psi }(|\mathrm{\nabla }v|)dxdy,}$

where ${\displaystyle \mathrm{\Omega }}$ is the extent of the images ${\displaystyle I(x,y)}$, ${\displaystyle \mathrm{\nabla }}$ is the gradient operator, ${\displaystyle \alpha }$ is a constant, and ${\displaystyle \mathrm{\Psi }()}$ is a loss function.

This optimisation problem is difficult to solve owing to its non-linearity. To address this issue, one can use a variational approach and linearise the brightness constancy constraint using a first order Taylor series approximation. Specifically, the brightness constancy constraint is approximated as,

${\displaystyle \frac{\mathrm{\partial }I}{\mathrm{\partial }x}u+\frac{\mathrm{\partial }I}{\mathrm{\partial }y}v+\frac{\mathrm{\partial }I}{\mathrm{\partial }t}=0.}$

For convenience, the derivatives of the image, ${\displaystyle \frac{\mathrm{\partial }I}{\mathrm{\partial }x}}$, ${\displaystyle \frac{\mathrm{\partial }I}{\mathrm{\partial }y}}$ and ${\displaystyle \frac{\mathrm{\partial }I}{\mathrm{\partial }t}}$ are often condensed to become ${\displaystyle I_x}$, ${\displaystyle I_y}$ and ${\displaystyle I_t}$. Doing so, allows one to rewrite the linearised brightness constancy constraint as,

${\displaystyle I_{x}u+I_{y}v+I_t=0.}$

The optimization problem can now be rewritten as

${\displaystyle E={\iint }_{\mathrm{\Omega }}\mathrm{\Psi }(I_{x}u+I_{y}v+I_t)+\alpha \mathrm{\Psi }(|\mathrm{\nabla }u|)+\alpha \mathrm{\Psi }(|\mathrm{\nabla }v|)dxdy.}$

For the choice of ${\displaystyle \mathrm{\Psi }(x)=x^2}$, this method is the same as the Horn-Schunck method. Of course, other choices of cost function have been used such as ${\displaystyle \mathrm{\Psi }(x)=\sqrt{x^2+{ϵ}^2}}$, which is a differentiable variant of the ${\displaystyle L^1}$ norm.

To solve the aforementioned optimization problem, one can use the Euler-Lagrange equations to provide a system of partial differential equations for each point in ${\displaystyle I(x,y,t)}$. In the simplest case of using ${\displaystyle \mathrm{\Psi }(x)=x^2}$, these equations are,

${\displaystyle I_x(I_{x}u+I_{y}v+I_t)-\alpha \mathrm{\Delta }u=0,}$

${\displaystyle I_y(I_{x}u+I_{y}v+I_t)-\alpha \mathrm{\Delta }v=0,}$

where ${\displaystyle \mathrm{\Delta }=\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{x}^2}+\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{y}^2}}$ denotes the Laplace operator. Since the image data is made up of discrete pixels, these equations are discretised. Doing so yields a system of linear equations which can be solved for ${\displaystyle (u,v)}$ at each pixel, using an iterative scheme such as Gauss-Seidel.

Although, linearising the brightness constancy constraint simplifies the optimisation problem significantly, the linearisation is only valid for small displacements and/or smooth images. To avoid this problem, a multi-scale or coarse-to-fine approach is often used. In such a scheme, the images are initially downsampled and the linearised Euler-Lagrange equations are solved at the reduced resolution. The estimated flow field at this scale is then used to initialise the process at next scale. This initialisation process is often performed by warping one frame using the current estimate of flow field so that it is as similar to other as possible.

An alternate approach is to discretize the optimisation problem and then perform a search of the possible ${\displaystyle (u,v)}$ values without linearising it. This search is often performed using Max-flow min-cut theorem algorithms, linear programming or belief propagation methods.

These regularized methods typically require manual tuning of the Lagrange multiplier, the so-called regularization parameters. There has been some progress in the automatic determination of these parameters in the context of optical flow applied to particle image velocimetry (PIV) data.

Parametric models

Instead of applying the regularization constraint on a point by point basis as per a regularized model, one can group pixels into regions and estimate the motion of these regions. This is known as a parametric model, since the motion of these regions is parameterized. In formulating optical flow estimation in this way, one makes the assumption that the motion field in each region be fully characterised by a set of parameters. Therefore, the goal of a parametric model is to estimate the motion parameters that minimise a loss function which can be written as,

${\displaystyle \hat{\mathit{\alpha }}=\arg \underset{\mathit{\alpha }}{min}\sum _{(x,y)\in \mathcal{R}}g(x,y)\rho (x,y,I_1,I_2,u_{\mathit{\alpha }},v_{\mathit{\alpha }}),}$

where ${\displaystyle \mathit{\alpha }}$ is the set of parameters determining the motion in the region ${\displaystyle \mathcal{R}}$, ${\displaystyle \rho ()}$ is data cost term, ${\displaystyle g()}$ is a weighting function that determines the influence of pixel ${\displaystyle (x,y)}$ on the total cost, and ${\displaystyle I_1}$ and ${\displaystyle I_2}$ are frames 1 and 2 from a pair of consecutive frames.

The simplest parametric model is the Lucas-Kanade method. This uses rectangular regions and parameterises the motion as purely translational. The Lucas-Kanade method uses the original brightness constancy constraint as the data cost term and selects ${\displaystyle g(x,y)=1}$. This yields the local loss function,

${\displaystyle \hat{\mathit{\alpha }}=\arg \underset{\mathit{\alpha }}{min}\sum _{(x,y)\in \mathcal{R}}|I(x+u_{\mathit{\alpha }},y+v_{\mathit{\alpha }},t+1)-I(x,y,t)|.}$

Other possible local loss functions include the negative normalized cross-correlation between the two frames.

Learning-based models

Instead of seeking to model optical flow directly, one can train a machine learning system to estimate optical flow. Since 2015, when FlowNet was proposed, learning based models have been applied to optical flow and have gained prominence. Initially, these approaches were based on Convolutional Neural Networks arranged in a U-Net architecture, often utilizing encoder-decoder or feature pyramid structures, such as PWC-Net, which integrated cost volumes (a 4D tensor representing the matching costs between all pairs of pixels in two feature maps) and warping (the process of spatially transforming one image based on a predicted flow field) to refine flow estimates across multiple scales. However, with the advent of transformer architecture in 2017, transformer based models have gained prominence. A significant shift occurred with the introduction of RAFT (Recurrent All-Pairs Field Transforms), which replaced coarse-to-fine pyramids with a single GRU-based state that iteratively updates the flow field. By maintaining a constant feature resolution at 1/8 of the input, RAFT significantly improved the preservation of fine details and robustness to fast motion compared to previous bottleneck-heavy designs, influencing a wide range of subsequent models that adopt similar iterative update mechanisms.

However, the all-pairs correlation used in such models is computationally expensive; for high-resolution content like FullHD or 4K, global matching can require more than 32 GB of VRAM, making it impractical for consumer-grade GPUs. To address this, efficiency-focused methods such as Flow1D, MeFlow, and Memfof have been developed. While these approaches generally optimize memory usage by decomposing the 2D search space, the latter optimizes correlation volumes for high-resolution multi-frame sequences, providing a practical implementation for standard GPUs.

Most learning-based approaches to optical flow use supervised learning. In this case, many frame pairs of video data and their corresponding ground-truth flow fields are used to optimise the parameters of the learning-based model to accurately estimate optical flow. This process often relies on vast synthetic training datasets, such as FlyingChairs and FlyingThings3D, due to the number of parameters involved. The models are then evaluated on benchmarks like MPI Sintel, KITTI, and the high-resolution Spring dataset. However, models trained exclusively on synthetic data often struggle with the domain gap when applied to real-world footage.

To address this, some learning-based optical flow approaches use self-supervised learning (sometimes called unsupervised learning) to reduce the need for large datasets with ground-truth data and leverage real-world footage without labels during training. Instead of training models to minimise the differences between estimated and ground-truth flow fields, they are trained to achieve learning objectives such as brightness constancy and smoothness of the flow field. More recently, methods like CroCo have introduced cross-view completion pre-training. Forcing the network to predict the masked regions of one image using the full second image teaches the model a strong geometric understanding and better generalization than models trained solely on task-specific labels.