Lably
Skip to content

Jaccard index

Measure of similarity and diversity between sets

  • 17 min read
  • 7 chapters

Related on Lably

Through Statistic, Sampling (statistics) and Measure (mathematics)

The concept was first introduced by Grove Karl Gilbert in 1884 as the “ratio of verification” in contexts of geological prediction evaluation. and now is often called the critical success index in meteorology. It was later developed independently by Paul Jaccard, originally giving the French name coefficient de communauté (coefficient of community), and independently formulated again by Taffee Tadashi Tanimoto. Thus, it is also called Tanimoto index or Tanimoto coefficient in some fields.

Overview

The Jaccard index measures similarity between finite non-empty sample sets and is defined as the size of the intersection divided by the size of the union of the sample sets:

J(A,B)=|A∩B||A∪B|=|A∩B||A|+|B|−|A∩B|.

The Jaccard index can be interpreted as a normalized measure of overlap between two sets, where the intersection represents shared elements and the union represents the total set of distinct elements. By definition, 0≤J(A,B)≤1. If the sets A and B have no elements in common, their intersection is empty, so |A∩B|=0 and therefore J(A,B)=0. The other extreme is that the two sets are equal. In that case A∩B=A∪B=A=B, so then J(A,B)=1. The Jaccard index is widely used in computer science, ecology, genomics and other sciences where binary or binarized data are used.


Both the exact solution and approximation methods are available for hypothesis testing with the Jaccard index. Jaccard similarity also applies to bags, i.e., multisets. This has a similar formula, but the symbols used represent bag intersection and bag sum (not union). The maximum value is 1/2.

J(A,B)=|A∩B||A⊎B|=|A∩B||A|+|B|.

The Jaccard distance, which measures dissimilarity between sample sets, is complementary to the Jaccard index and is obtained by subtracting the Jaccard index from 1 or, equivalently, by dividing the difference of the sizes of the union and the intersection of two sets by the size of the union:

dJ(A,B)=1−J(A,B)=|A∪B|−|A∩B||A∪B|.

An alternative interpretation of the Jaccard distance is as the ratio of the size of the symmetric difference A△B=(A∪B)−(A∩B) to the union. Jaccard distance is commonly used to calculate an n × n matrix for clustering and multidimensional scaling of n sample sets. These distance measures are commonly used in cluster analysis for grouping similar observations.

This distance is a metric on the collection of all finite sets.

There is also a version of the Jaccard distance for measures, including probability measures. If μ is a measure on a measurable space X, then we define the Jaccard index by

Jμ(A,B)=μ(A∩B)μ(A∪B),

and the Jaccard distance by

dμ(A,B)=1−Jμ(A,B)=μ(A△B)μ(A∪B).

The definition is not well-defined when μ(A∪B)=0 or μ(A∪B)=∞.

The MinHash min-wise independent permutations locality sensitive hashing scheme may be used to efficiently compute an accurate estimate of the Jaccard similarity index of pairs of sets, where each set is represented by a constant-sized signature derived from the minimum values of a hash function.

The Jaccard index is particularly useful for analyzing large-scale and sparse datasets in modern data mining applications.

Sign in to keep reading

6 more chapters are free with a Lably account.

More in Computing and communication