U-Net · Lablypedia

The U-Net architecture has also been employed in diffusion models for iterative image denoising. This technology underlies many modern image generation models, such as DALL-E, Midjourney, and Stable Diffusion.

U-Net is also being explored for language models. Tokenization is not a separate step, allowing the model to more easily understand spelling and concurrently vectorizing / tokenizing higher level concepts.

Description

The U-Net architecture stems from the so-called "fully convolutional network".

The main idea is to supplement a usual contracting network by successive layers, where pooling operations are replaced by upsampling operators. Hence these layers increase the resolution of the output. A successive convolutional layer can then learn to assemble a precise output based on this information.

One important modification in U-Net is that there are a large number of feature channels in the upsampling part, which allow the network to propagate context information to higher resolution layers. As a consequence, the expansive path is more or less symmetric to the contracting part, and yields a u-shaped architecture. The network only uses the valid part of each convolution without any fully connected layers. Therefore, the network can be trained and be run with a mix of input resolutions. To predict the pixels in the border region of the image, the missing context beyond the image boundary is extrapolated by mirroring the input image at the image boundaries. The original U-Net paper further suggested a tiling strategy, where large images are cut into overlapping tiles that are processed independently. This enables processing high-resolution images that would otherwise exceed the available GPU memory. Recently, there had also been an interest in receptive field based U-Net models for medical image segmentation.

4 more chapters are free with a Lably account.

Network architecture

The network consists of a contracting path and an expansive path, which gives it the u-shaped architecture. The contracting path is a typical convolutional network that consists of repeated application of convolutions, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased. The expansive pathway combines the feature and spatial information through a sequence of up-convolutions and concatenations with high-resolution features from the contracting path.

Applications

There are many applications of U-Net in biomedical image segmentation, such as the segmentation of different organs and bodily systems within computed tomography (CT) and magnetic resonance imaging (MRI) scans. Specific cases include brain image segmentation (''BRATS'') and liver image segmentation ("siliver07") as well as protein binding site prediction. U-Net implementations have also found use in the physical sciences, for example in the analysis of micrographs of materials. Variations of the U-Net have also been applied for medical image reconstruction. Here are some variants and applications of U-Net as follows:

Pixel-wise regression using U-Net and its application on pansharpening;
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation;
TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation.
Image-to-image translation to estimate fluorescent stains
In binding site prediction of protein structure.

History

U-Net was created by Olaf Ronneberger, Philipp Fischer, Thomas Brox in 2015 and reported in the paper "U-Net: Convolutional Networks for Biomedical Image Segmentation". It is an improvement and development of FCN: Evan Shelhamer, Jonathan Long, Trevor Darrell (2014). "Fully convolutional networks for semantic segmentation".

Implementations

Tensorflow Unet by J Akeret (2017)
U-Net source code from Pattern Recognition and Image Processing at Computer Science Department of the University of Freiburg, Germany.