LabelMe

Annotation Tool

The LabelMe annotation tool provides a means for users to contribute to the project. The tool can be accessed anonymously or by logging into a free account. To access the tool, users must have a compatible web browser with JavaScript support. When the tool is loaded, it chooses a random image from the LabelMe dataset and displays it on the screen. If the image already has object labels associated with it, they will be overlaid on top of the image in polygon format. Each distinct object label is displayed in a different color.

If the image is not completely labeled, the user can use the mouse to draw a polygon containing an object in the image. For example, in the adjacent image, if a person was standing in front of the building, the user could click on a point on the border of the person, and continue clicking along the outside edge until returning to the starting point. Once the polygon is closed, a bubble pops up on the screen which allows the user to enter a label for the object. The user can choose whatever label the user thinks best describes the object. If the user disagrees with the previous labeling of the image, the user can click on the outline polygon of an object and either delete the polygon completely or edit the text label to give it a new name.

As soon as changes are made to the image by the user, they are saved and openly available for anyone to download from the LabelMe dataset. In this way, the data is always changing due to contributions by the community of users who use the tool. Once the user is finished with an image, the Show me another image link can be clicked and another random image will be selected to display to the user.

Problems with the data

The LabelMe dataset has some problems. Some are inherent in the data, such as the objects in the images not being uniformly distributed with respect to size and image location. This is due to the images being primarily taken by humans who tend to focus the camera on interesting objects in a scene. However, cropping and rescaling the images randomly can simulate a uniform distribution. Other problems are caused by the amount of freedom given to the users of the annotation tool. Some problems that arise are:

• The user can choose which objects in the scene to outline. Should an occluded person be labeled? Should an occluded part of an object be included when outlining the object? Should the sky be labeled?
• The user has to describe the shape of the object themselves by outlining a polygon. Should the fingers of a hand on a person be outlined with detail? How much precision must be used when outlining objects?
• The user chooses what text to enter as the label for the object. Should the label be person, man, or pedestrian?

The creators of LabelMe decided to leave these decisions up to the annotator. The reason for this is that they believe people will tend to annotate the images according to what they think is the natural labeling of the images. This also provides some variability in the data, which can help researchers tune their algorithms to account for this variability.

Extending the data

Using WordNet

Since the text labels for objects provided in LabelMe come from user input, there is a lot of variation in the labels used (as described above). Because of this, analysis of objects can be difficult. For example, a picture of a dog might be labeled as dog, canine, hound, pooch, or animal. Ideally, when using the data, the object class dog at the abstract level should incorporate all of these text labels.

WordNet is a database of words organized into a structural way. It allows assigning a word to a category, or in WordNet language: a sense. Sense assignment is not easy to do automatically. When the authors of LabelMe tried automatic sense assignment, they found that it was prone to a high rate of error, so instead they assigned words to senses manually. At first, this may seem like a daunting task since new labels are added to the LabelMe project continuously. To the right is a graph comparing the growth of polygons to the growth of words (descriptions). As you can see, the growth of words is small compared with the continuous growth of polygons, and therefore is easy enough to keep up to date manually by the LabelMe team.

Once WordNet assignment is done, searches in the LabelMe database are much more effective. For example, a search for animal might bring up pictures of dogs, cats and snakes. However, since the assignment was done manually, a picture of a computer mouse labeled as mouse would not show up in a search for animals. Also, if objects are labeled with more complex terms like dog walking, WordNet still allows the search of dog to return these objects as results. WordNet makes the LabelMe database much more useful.

Object-part hierarchy

Having a large dataset of objects where overlap is allowed provides enough data to try and categorize objects as being a part of another object. For example, most of the labels assigned wheel are probably part of objects assigned to other labels like car or bicycle. These are called part labels. To determine if label P is a part label for label O:

• Let ${\displaystyle {\mathrm{I}}_{\mathrm{O}}}$ denote the set of images containing an object (e.g. car)
• Let ${\displaystyle {\mathrm{I}}_{\mathrm{P}}}$ denote the set of images containing a part (e.g. wheel)
• Let the overlap score between object O and part P, ${\displaystyle {\mathrm{S}}_{\mathrm{O},\mathrm{P}}}$, be defined as the ratio of the intersection area to the area of the part polygon. (e.g. ${\displaystyle \frac{\mathrm{A}(\mathrm{O}\cap \mathrm{P})}{\mathrm{A}(\mathrm{P})}}$)
• Let ${\displaystyle {\mathrm{I}}_{\mathrm{O},\mathrm{P}}\subseteq {\mathrm{I}}_{\mathrm{P}}}$ denote the images where object and part polygons have ${\displaystyle {\mathrm{S}}_{\mathrm{O},\mathrm{P}}>\beta }$ where ${\displaystyle \beta }$ is some threshold value. The authors of LabelMe use ${\displaystyle \beta =0.5}$
• The object-part score for a candidate label is ${\displaystyle \frac{{\mathrm{N}}_{\mathrm{O},\mathrm{P}}}{{\mathrm{N}}_{\mathrm{P}}+\alpha }}$ where ${\displaystyle {\mathrm{N}}_{\mathrm{O},\mathrm{P}}}$ and ${\displaystyle {\mathrm{N}}_{\mathrm{P}}}$ are the number of images in ${\displaystyle {\mathrm{I}}_{\mathrm{O},\mathrm{P}}}$ and ${\displaystyle {\mathrm{I}}_{\mathrm{P}}}$, respectively, and ${\displaystyle \alpha }$ is a concentration parameter. The authors of LabelMe use ${\displaystyle \alpha =5}$.

This algorithm allows the automatic classification of parts of an object when the part objects are frequently contained within the outer object.

Object depth ordering

Another instance of object overlap is when one object is actually on top of the other. For example, an image might contain a person standing in front of a building. The person is not a part label as above since the person is not part of the building. Instead, they are two separate objects that happen to overlap. To automatically determine which object is the foreground and which is the background, the authors of LabelMe propose several options:

• If an object is completely contained within another object, then the inner object must be in the foreground. Otherwise, it would not be visible in the image. The only exception is with transparent or translucent objects, but these occur rarely.
• One of the objects could be labeled as something that cannot be in the foreground. Examples are sky, ground, or road.
• The object with more polygon points inside the intersecting area is most likely the foreground. The authors tested this hypothesis and found it to be highly accurate.
• Histogram intersection can be used. To do this, a color histogram in the intersecting areas is compared to the color histogram of the two objects. The object with the closer color histogram is assigned as the foreground. This method is less accurate than counting the polygon points.

Matlab Toolbox

The LabelMe project provides a set of tools for using the LabelMe dataset from Matlab. Since research is often done in Matlab, this allows the integration of the dataset with existing tools in computer vision. The entire dataset can be downloaded and used offline, or the toolbox allows dynamic downloading of content on demand.