Tf–idf

Definition

1. The tf–idf is the product of two statistics, term frequency and inverse document frequency. There are various ways for determining the exact values of both statistics.
2. A formula that aims to define the importance of a keyword or phrase within a document or a web page.

Term frequency

Term frequency, tf(t,d), is the relative frequency of term t within document d,

${\displaystyle \mathrm{t}\mathrm{f}(t,d)=\frac{f_{t,d}}{\sum _{t^{\prime }\in d}{f}_{t^{\prime },d}}}$,

where f_t,d is the raw count of a term in a document, i.e., the number of times that term t occurs in document d. Note the denominator is simply the total number of terms in document d (counting each occurrence of the same term separately). There are various other ways to define term frequency:

• the raw count itself: tf(t,d) = f_t,d
• Boolean "frequencies": tf(t,d) = 1 if t occurs in d and 0 otherwise;
• logarithmically scaled frequency: tf(t,d) = log (1 + f_t,d);
• augmented frequency, to prevent a bias towards longer documents, e.g. raw frequency divided by the raw frequency of the most frequently occurring term in the document:

${\displaystyle \mathrm{t}\mathrm{f}(t,d)=0.5+0.5\cdot \frac{f_{t,d}}{max\{f_{t^{\prime },d}:t^{\prime }\in d\}}}$

Inverse document frequency

The inverse document frequency is a measure of how much information the word provides, i.e., how common or rare it is across all documents. It is the logarithmically scaled inverse fraction of the documents that contain the word (obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient):

${\displaystyle \mathrm{i}\mathrm{d}\mathrm{f}(t,D)=\log \frac{N}{n_t}}$

with

• ${\displaystyle D}$: is the set of all documents in the corpus
• ${\displaystyle N=|D|}$: total number of documents in the corpus
• ${\displaystyle n_t=|\{d\in D:t\in d\}|}$ : number of documents where the term ${\displaystyle t}$ appears (i.e., ${\displaystyle \mathrm{t}\mathrm{f}(t,d)\ne 0}$). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the numerator to ${\displaystyle 1+N}$ and the denominator to ${\displaystyle 1+n_t}$.

Term frequency–inverse document frequency

Then tf–idf is calculated as

${\displaystyle \mathrm{t}\mathrm{f}\mathrm{i}\mathrm{d}\mathrm{f}(t,d,D)=\mathrm{t}\mathrm{f}(t,d)\cdot \mathrm{i}\mathrm{d}\mathrm{f}(t,D)}$

A high weight in tf–idf is reached by a high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents; the weights hence tend to filter out common terms. Since the ratio inside the idf's log function is always greater than or equal to 1, the value of idf (and tf–idf) is greater than or equal to 0. As a term appears in more documents, the ratio inside the logarithm approaches 1, bringing the idf and tf–idf closer to 0.

Justification of idf

Idf was introduced as "term specificity" by Karen Spärck Jones in a 1972 paper. Although it has worked well as a heuristic, its theoretical foundations have been troublesome for at least three decades afterward, with many researchers trying to find information theoretic justifications for it.

Spärck Jones's own explanation did not propose much theory, aside from a connection to Zipf's law. Attempts have been made to put idf on a probabilistic footing, by estimating the probability that a given document d contains a term t as the relative document frequency,

${\displaystyle P(t|D)=\frac{|\{d\in D:t\in d\}|}{N},}$

so that we can define idf as

${\displaystyle \begin{array}{cc}\mathrm{i}\mathrm{d}\mathrm{f}& =-\log P(t|D)\\ & =\log \frac{1}{P(t|D)}\\ & =\log \frac{N}{|\{d\in D:t\in d\}|}\end{array}}$

Namely, the inverse document frequency is the logarithm of "inverse" relative document frequency.

This probabilistic interpretation in turn takes the same form as that of self-information. However, applying such information-theoretic notions to problems in information retrieval leads to problems when trying to define the appropriate event spaces for the required probability distributions: not only documents need to be taken into account, but also queries and terms.

Link with information theory

Both term frequency and inverse document frequency can be formulated in terms of information theory; it helps to understand why their product has a meaning in terms of joint informational content of a document. A characteristic assumption about the distribution ${\displaystyle p(d,t)}$ is that:

${\displaystyle p(d|t)=\frac{1}{|\{d\in D:t\in d\}|}}$

This assumption and its implications, according to Aizawa: "represent the heuristic that tf–idf employs."

The conditional entropy of a "randomly chosen" document in the corpus ${\displaystyle D}$, conditional to the fact it contains a specific term ${\displaystyle t}$ (and assuming that all documents have equal probability to be chosen) is:

${\displaystyle H(\mathcal{D}|\mathcal{T}=t)=-\sum _d{p}_{d|t}\log p_{d|t}=-\log \frac{1}{|\{d\in D:t\in d\}|}=\log \frac{|\{d\in D:t\in d\}|}{|D|}+\log |D|=-\mathrm{i}\mathrm{d}\mathrm{f}(t)+\log |D|}$

In terms of notation, ${\displaystyle \mathcal{D}}$ and ${\displaystyle \mathcal{T}}$ are "random variables" corresponding to respectively draw a document or a term. The mutual information can be expressed as

${\displaystyle M(\mathcal{T};\mathcal{D})=H(\mathcal{D})-H(\mathcal{D}|\mathcal{T})=\sum _t{p}_t\cdot (H(\mathcal{D})-H(\mathcal{D}|W=t))=\sum _t{p}_t\cdot \mathrm{i}\mathrm{d}\mathrm{f}(t)}$

The last step is to expand ${\displaystyle p_t}$, the unconditional probability to draw a term, with respect to the (random) choice of a document, to obtain:

${\displaystyle M(\mathcal{T};\mathcal{D})=\sum _{t,d}{p}_{t|d}\cdot p_d\cdot \mathrm{i}\mathrm{d}\mathrm{f}(t)=\sum _{t,d}\mathrm{t}\mathrm{f}(t,d)\cdot \frac{1}{|D|}\cdot \mathrm{i}\mathrm{d}\mathrm{f}(t)=\frac{1}{|D|}\sum _{t,d}\mathrm{t}\mathrm{f}(t,d)\cdot \mathrm{i}\mathrm{d}\mathrm{f}(t).}$

This expression shows that summing the Tf–idf of all possible terms and documents recovers the mutual information between documents and term taking into account all the specificities of their joint distribution. Each Tf–idf hence carries the "bit of information" attached to a term x document pair.

Link with statistical theory

Tf–idf is closely related to the negative logarithmically transformed p-value from a one-tailed formulation of Fisher's exact test when the underlying corpus documents satisfy certain idealized assumptions. More recently, tf–idf variants were shown to arise as components in the test statistic of a penalized likelihood-ratio test for word burstiness based on a beta-binomial statistical language model. In this framework, the null hypothesis models term occurrences using a binomial distribution, while the alternative hypothesis models word burstiness using a beta-binomial distribution with a gamma-distributed penalty term placed on the beta-binomial precision parameter. The resulting test statistic contains the tf–idf variants binary term frequency–inverse document frequency (btf–idf) and term frequency–inverse collection frequency (tf–icf), thereby establishing a direct connection between tf–idf family term-weighting schemes and statistical hypothesis testing.

Example of tf–idf

Suppose that we have term count tables of a corpus consisting of only two documents:

"This is a sample A."

"This is another example, another example, example."

The calculation of tf–idf for the term "this" is performed as follows:

In its raw frequency form, tf is just the frequency of the "this" for each document. In each document, the word "this" appears once; but as the document 2 has more words, its relative frequency is smaller.

${\displaystyle \mathrm{t}\mathrm{f}({}^{″}\mathsf{t}\mathsf{h}\mathsf{i}{\mathsf{s}}^{″},d_1)=\frac{1}{5}=0.2}$

${\displaystyle \mathrm{t}\mathrm{f}({}^{″}\mathsf{t}\mathsf{h}\mathsf{i}{\mathsf{s}}^{″},d_2)=\frac{1}{7}\approx 0.14}$

An idf is constant per corpus, and accounts for the ratio of documents that include the word "this". In this case, we have a corpus of two documents and all of them include the word "this".

${\displaystyle \mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{t}\mathsf{h}\mathsf{i}{\mathsf{s}}^{″},D)=\log \left(\frac{2}{2}\right)=0}$

So tf–idf is zero for the word "this", which implies that the word is not very informative as it appears in all documents.

${\displaystyle \mathrm{t}\mathrm{f}\mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{t}\mathsf{h}\mathsf{i}{\mathsf{s}}^{″},d_1,D)=0.2\times 0=0}$

${\displaystyle \mathrm{t}\mathrm{f}\mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{t}\mathsf{h}\mathsf{i}{\mathsf{s}}^{″},d_2,D)=0.14\times 0=0}$

The word "example" is more interesting - it occurs three times, but only in the second document:

${\displaystyle \mathrm{t}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},d_1)=\frac{0}{5}=0}$

${\displaystyle \mathrm{t}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},d_2)=\frac{3}{7}\approx 0.429}$

${\displaystyle \mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},D)=\log \left(\frac{2}{1}\right)=0.301}$

Finally,

${\displaystyle \mathrm{t}\mathrm{f}\mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},d_1,D)=\mathrm{t}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},d_1)\times \mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},D)=0\times 0.301=0}$

${\displaystyle \mathrm{t}\mathrm{f}\mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},d_2,D)=\mathrm{t}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},d_2)\times \mathrm{i}\mathrm{d}\mathrm{f}({}^{″}\mathsf{e}\mathsf{x}\mathsf{a}\mathsf{m}\mathsf{p}\mathsf{l}{\mathsf{e}}^{″},D)=0.429\times 0.301\approx 0.129}$

(using the base 10 logarithm).

Beyond terms

The idea behind tf–idf also applies to entities other than terms. In 1998, the concept of idf was applied to citations. The authors argued that "if a very uncommon citation is shared by two documents, this should be weighted more highly than a citation made by a large number of documents". In addition, tf–idf was applied to "visual words" with the purpose of conducting object matching in videos, and entire sentences. However, the concept of tf–idf did not prove to be more effective in all cases than a plain tf scheme (without idf). When tf–idf was applied to citations, researchers could find no improvement over a simple citation-count weight that had no idf component.

Derivatives

A number of term-weighting schemes have derived from tf–idf. One of them is TF–PDF (term frequency * proportional document frequency). TF–PDF was introduced in 2001 in the context of identifying emerging topics in the media. The PDF component measures the difference of how often a term occurs in different domains. Another derivate is TF–IDuF. In TF–IDuF, idf is not calculated based on the document corpus that is to be searched or recommended. Instead, idf is calculated on users' personal document collections. The authors report that TF–IDuF was equally effective as tf–idf but could also be applied in situations when, e.g., a user modeling system has no access to a global document corpus. The DELTA TF-IDF derivative uses the difference in importance of a term across two specific classes, like positive and negative sentiment. For example, it can assign a high score to a word like "excellent" in positive reviews and a low score to the same word in negative reviews. This helps identify words that strongly indicate the sentiment of a document, potentially leading to improved accuracy in text classification tasks.

External links and suggested reading

• Gensim is a Python library for vector space modeling and includes tf–idf weighting.
• Anatomy of a search engine Archived 2009-03-10 at the Wayback Machine
• tf–idf and related definitions as used in Lucene
• TfidfTransformer in scikit-learn
• Text to Matrix Generator (TMG) MATLAB toolbox that can be used for various tasks in text mining (TM) specifically i) indexing, ii) retrieval, iii) dimensionality reduction, iv) clustering, v) classification. The indexing step offers the user the ability to apply local and global weighting methods, including tf–idf.
• Term-frequency explained Explanation of term-frequency