Blob detection

The difference of Gaussians approach

From the fact that the scale space representation ${\displaystyle L(x,y,t)}$ satisfies the diffusion equation

${\displaystyle {\mathrm{\partial }}_{t}L=\frac{1}{2}{\mathrm{\nabla }}^{2}L}$

it follows that the Laplacian of the Gaussian operator ${\displaystyle {\mathrm{\nabla }}^{2}L(x,y,t)}$ can also be computed as the limit case of the difference between two Gaussian smoothed images (scale space representations)

${\displaystyle {\mathrm{\nabla }}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}^{2}L(x,y;t)\approx \frac{t}{\mathrm{\Delta }t}\left(L(x,y;t+\mathrm{\Delta }t)-L(x,y;t)\right)}$.

In the computer vision literature, this approach is referred to as the difference of Gaussians (DoG) approach. Besides minor technicalities, however, this operator is in essence similar to the Laplacian and can be seen as an approximation of the Laplacian operator. In a similar fashion as for the Laplacian blob detector, blobs can be detected from scale-space extrema of differences of Gaussians—(Lindeberg 2012, 2015) for the explicit relation between the difference-of-Gaussian operator and the scale-normalized Laplacian operator. This approach is for instance used in the scale-invariant feature transform (SIFT) algorithm—(Lowe 2004).

The determinant of the Hessian

By considering the scale-normalized determinant of the Hessian, also referred to as the Monge–Ampère operator,

${\displaystyle det{H}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}L=t^2\left(L_{xx}{L}_{yy}-L_{xy}^2\right)}$

where ${\displaystyle HL}$ denotes the Hessian matrix of the scale-space representation ${\displaystyle L}$ and then detecting scale-space maxima of this operator one obtains another straightforward differential blob detector with automatic scale selection which also responds to saddles (Lindeberg 1994, 1998)

${\displaystyle (\hat{x},\hat{y};\hat{t})={\mathrm{argmaxlocal}}_{(x,y;t)}((det{H}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}L)(x,y;t))}$.

The blob points ${\displaystyle (\hat{x},\hat{y})}$ and scales ${\displaystyle \hat{t}}$ are also defined from an operational differential geometric definitions that leads to blob descriptors that are covariant with translations, rotations and rescalings in the image domain. In terms of scale selection, blobs defined from scale-space extrema of the determinant of the Hessian (DoH) also have slightly better scale selection properties under non-Euclidean affine transformations than the more commonly used Laplacian operator (Lindeberg 1994, 1998, 2015). In simplified form, the scale-normalized determinant of the Hessian computed from Haar wavelets is used as the basic interest point operator in the SURF descriptor (Bay et al. 2006) for image matching and object recognition.

A detailed analysis of the selection properties of the determinant of the Hessian operator and other closely scale-space interest point detectors is given in (Lindeberg 2013a) showing that the determinant of the Hessian operator has better scale selection properties under affine image transformations than the Laplacian operator. In (Lindeberg 2013b, 2015) it is shown that the determinant of the Hessian operator performs significantly better than the Laplacian operator or its difference-of-Gaussians approximation, as well as better than the Harris or Harris-Laplace operators, for image-based matching using local SIFT-like or SURF-like image descriptors, leading to higher efficiency values and lower 1-precision scores.

The hybrid Laplacian and determinant of the Hessian operator (Hessian-Laplace)

A hybrid operator between the Laplacian and the determinant of the Hessian blob detectors has also been proposed, where spatial selection is done by the determinant of the Hessian and scale selection is performed with the scale-normalized Laplacian (Mikolajczyk and Schmid 2004):

${\displaystyle (\hat{x},\hat{y})={\mathrm{argmaxlocal}}_{(x,y)}((detHL)(x,y;t))}$

${\displaystyle \hat{t}={\mathrm{argmaxminlocal}}_t(({\mathrm{\nabla }}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}^{2}L)(\hat{x},\hat{y};t))}$

This operator has been used for image matching, object recognition as well as texture analysis.

Affine-adapted differential blob detectors

The blob descriptors obtained from these blob detectors with automatic scale selection are invariant to translations, rotations and uniform rescalings in the spatial domain. The images that constitute the input to a computer vision system are, however, also subject to perspective distortions. To obtain blob descriptors that are more robust to perspective transformations, a natural approach is to devise a blob detector that is invariant to affine transformations. In practice, affine invariant interest points can be obtained by applying affine shape adaptation to a blob descriptor, where the shape of the smoothing kernel is iteratively warped to match the local image structure around the blob, or equivalently a local image patch is iteratively warped while the shape of the smoothing kernel remains rotationally symmetric (Lindeberg and Garding 1997; Baumberg 2000; Mikolajczyk and Schmid 2004, Lindeberg 2008). In this way, we can define affine-adapted versions of the Laplacian/Difference of Gaussian operator, the determinant of the Hessian and the Hessian-Laplace operator (see also Harris-Affine and Hessian-Affine).

Spatio-temporal blob detectors

The determinant of the Hessian operator has been extended to joint space-time by Willems et al. and Lindeberg, leading to the following scale-normalized differential expression:

${\displaystyle det(H_{(x,y,t),\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}L)=s^{2{\gamma }_s}{\tau }^{{\gamma }_{\tau }}\left(L_{xx}{L}_{yy}{L}_{tt}+2L_{xy}{L}_{xt}{L}_{yt}-L_{xx}{L}_{yt}^2-L_{yy}{L}_{xt}^2-L_{tt}{L}_{xy}^2\right).}$

In the work by Willems et al., a simpler expression corresponding to ${\displaystyle {\gamma }_s=1}$ and ${\displaystyle {\gamma }_{\tau }=1}$ was used. In Lindeberg, it was shown that ${\displaystyle {\gamma }_s=5/4}$ and ${\displaystyle {\gamma }_{\tau }=5/4}$ implies better scale selection properties in the sense that the selected scale levels obtained from a spatio-temporal Gaussian blob with spatial extent ${\displaystyle s=s_0}$ and temporal extent ${\displaystyle \tau ={\tau }_0}$ will perfectly match the spatial extent and the temporal duration of the blob, with scale selection performed by detecting spatio-temporal scale-space extrema of the differential expression.

The Laplacian operator has been extended to spatio-temporal video data by Lindeberg, leading to the following two spatio-temporal operators, which also constitute models of receptive fields of non-lagged vs. lagged neurons in the LGN:

${\displaystyle {\mathrm{\partial }}_{t,\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}({\mathrm{\nabla }}_{(x,y),\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}^{2}L)=s^{{\gamma }_s}{\tau }^{{\gamma }_{\tau }/2}(L_{xxt}+L_{yyt}),}$

${\displaystyle {\mathrm{\partial }}_{tt,\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}({\mathrm{\nabla }}_{(x,y),\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}^{2}L)=s^{{\gamma }_s}{\tau }^{{\gamma }_{\tau }}(L_{xxtt}+L_{yytt}).}$

For the first operator, scale selection properties call for using ${\displaystyle {\gamma }_s=1}$ and ${\displaystyle {\gamma }_{\tau }=1/2}$, if we want this operator to assume its maximum value over spatio-temporal scales at a spatio-temporal scale level reflecting the spatial extent and the temporal duration of an onset Gaussian blob. For the second operator, scale selection properties call for using ${\displaystyle {\gamma }_s=1}$ and ${\displaystyle {\gamma }_{\tau }=3/4}$, if we want this operator to assume its maximum value over spatio-temporal scales at a spatio-temporal scale level reflecting the spatial extent and the temporal duration of a blinking Gaussian blob.

Grey-level blobs, grey-level blob trees and scale-space blobs

A natural approach to detect blobs is to associate a bright (dark) blob with each local maximum (minimum) in the intensity landscape. A main problem with such an approach, however, is that local extrema are very sensitive to noise. To address this problem, Lindeberg (1993, 1994) studied the problem of detecting local maxima with extent at multiple scales in scale space. A region with spatial extent defined from a watershed analogy was associated with each local maximum, as well a local contrast defined from a so-called delimiting saddle point. A local extremum with extent defined in this way was referred to as a grey-level blob. Moreover, by proceeding with the watershed analogy beyond the delimiting saddle point, a grey-level blob tree was defined to capture the nested topological structure of level sets in the intensity landscape, in a way that is invariant to affine deformations in the image domain and monotone intensity transformations. By studying how these structures evolve with increasing scales, the notion of scale-space blobs was introduced. Beyond local contrast and extent, these scale-space blobs also measured how stable image structures are in scale-space, by measuring their scale-space lifetime.

It was proposed that regions of interest and scale descriptors obtained in this way, with associated scale levels defined from the scales at which normalized measures of blob strength assumed their maxima over scales could be used for guiding other early visual processing. An early prototype of simplified vision systems was developed where such regions of interest and scale descriptors were used for directing the focus-of-attention of an active vision system. While the specific technique that was used in these prototypes can be substantially improved with the current knowledge in computer vision, the overall general approach is still valid, for example in the way that local extrema over scales of the scale-normalized Laplacian operator are nowadays used for providing scale information to other visual processes.

Lindeberg's watershed-based grey-level blob detection algorithm

For the purpose of detecting grey-level blobs (local extrema with extent) from a watershed analogy, Lindeberg developed an algorithm based on pre-sorting the pixels, alternatively connected regions having the same intensity, in decreasing order of the intensity values. Then, comparisons were made between nearest neighbours of either pixels or connected regions.

For simplicity, consider the case of detecting bright grey-level blobs and let the notation "higher neighbour" stand for "neighbour pixel having a higher grey-level value". Then, at any stage in the algorithm (carried out in decreasing order of intensity values) is based on the following classification rules:

1. If a region has no higher neighbour, then it is a local maximum and will be the seed of a blob. Set a flag which allows the blob to grow.
2. Else, if it has at least one higher neighbour, which is background, then it cannot be part of any blob and must be background.
3. Else, if it has more than one higher neighbour and if those higher neighbours are parts of different blobs, then it cannot be a part of any blob, and must be background. If any of the higher neighbors are still allowed to grow, clear their flag which allows them to grow.
4. Else, it has one or more higher neighbours, which are all parts of the same blob. If that blob is still allowed to grow then the current region should be included as a part of that blob. Otherwise the region should be set to background.

Compared to other watershed methods, the flooding in this algorithm stops once the intensity level falls below the intensity value of the so-called delimiting saddle point associated with the local maximum. However, it is rather straightforward to extend this approach to other types of watershed constructions. For example, by proceeding beyond the first delimiting saddle point a "grey-level blob tree" can be constructed. Moreover, the grey-level blob detection method was embedded in a scale space representation and performed at all levels of scale, resulting in a representation called the scale-space primal sketch.

This algorithm with its applications in computer vision is described in more detail in Lindeberg's thesis as well as the monograph on scale-space theory partially based on that work. Earlier presentations of this algorithm can also be found in . More detailed treatments of applications of grey-level blob detection and the scale-space primal sketch to computer vision and medical image analysis are given in .

Maximally stable extremal regions (MSER)

Matas et al. (2002) were interested in defining image descriptors that are robust under perspective transformations. They studied level sets in the intensity landscape and measured how stable these were along the intensity dimension. Based on this idea, they defined a notion of maximally stable extremal regions and showed how these image descriptors can be used as image features for stereo matching.

There are close relations between this notion and the above-mentioned notion of grey-level blob tree. The maximally stable extremal regions can be seen as making a specific subset of the grey-level blob tree explicit for further processing.