Bag-of-words model

The bag-of-words model is commonly used in methods of document classification where, for example, the (frequency of) occurrence of each word is used as a feature for training a classifier. It has also been used for computer vision.

An early reference to "bag of words" in a linguistic context can be found in Zellig Harris's 1954 article on Distributional Structure.

Definition

The following models a text document using bag-of-words. Here are two simple text documents:

<span id="mwJw"></span>(1) John likes to watch movies. Mary likes movies too.

<span id="mwKg"></span>(2) Mary also likes to watch football games.

Based on these two text documents, a list is constructed as follows for each document:

<span id="mwLg"></span>"John","likes","to","watch","movies","Mary","likes","movies","too" "Mary","also","likes","to","watch","football","games"

Representing each bag-of-words as a JSON object, and attributing to the respective JavaScript variable:

<span id="mwNA"></span><span class="nx" id="mwNQ">BoW1</span><span class="w" id="mwNg"> </span><span class="o" id="mwNw">=</span><span class="w" id="mwOA"> </span><span class="p" id="mwOQ">{</span><span class="s2" id="mwOg">"John"</span><span class="o" id="mwOw">:</span><span class="mf" id="mwPA">1</span><span class="p" id="mwPQ">,</span><span class="s2" id="mwPg">"likes"</span><span class="o" id="mwPw">:</span><span class="mf" id="mwQA">2</span><span class="p" id="mwQQ">,</span><span class="s2" id="mwQg">"to"</span><span class="o" id="mwQw">:</span><span class="mf" id="mwRA">1</span><span class="p" id="mwRQ">,</span><span class="s2" id="mwRg">"watch"</span><span class="o" id="mwRw">:</span><span class="mf" id="mwSA">1</span><span class="p" id="mwSQ">,</span><span class="s2" id="mwSg">"movies"</span><span class="o" id="mwSw">:</span><span class="mf" id="mwTA">2</span><span class="p" id="mwTQ">,</span><span class="s2" id="mwTg">"Mary"</span><span class="o" id="mwTw">:</span><span class="mf" id="mwUA">1</span><span class="p" id="mwUQ">,</span><span class="s2" id="mwUg">"too"</span><span class="o" id="mwUw">:</span><span class="mf" id="mwVA">1</span><span class="p" id="mwVQ">};</span> <span class="nx" id="mwVg">BoW2</span><span class="w" id="mwVw"> </span><span class="o" id="mwWA">=</span><span class="w" id="mwWQ"> </span><span class="p" id="mwWg">{</span><span class="s2" id="mwWw">"Mary"</span><span class="o" id="mwXA">:</span><span class="mf" id="mwXQ">1</span><span class="p" id="mwXg">,</span><span class="s2" id="mwXw">"also"</span><span class="o" id="mwYA">:</span><span class="mf" id="mwYQ">1</span><span class="p" id="mwYg">,</span><span class="s2" id="mwYw">"likes"</span><span class="o" id="mwZA">:</span><span class="mf" id="mwZQ">1</span><span class="p" id="mwZg">,</span><span class="s2" id="mwZw">"to"</span><span class="o" id="mwaA">:</span><span class="mf" id="mwaQ">1</span><span class="p" id="mwag">,</span><span class="s2" id="mwaw">"watch"</span><span class="o" id="mwbA">:</span><span class="mf" id="mwbQ">1</span><span class="p" id="mwbg">,</span><span class="s2" id="mwbw">"football"</span><span class="o" id="mwcA">:</span><span class="mf" id="mwcQ">1</span><span class="p" id="mwcg">,</span><span class="s2" id="mwcw">"games"</span><span class="o" id="mwdA">:</span><span class="mf" id="mwdQ">1</span><span class="p" id="mwdg">};</span>

Each key is the word, and each value is the number of occurrences of that word in the given text document.

The order of elements is free, so, for example {"too":1,"Mary":1,"movies":2,"John":1,"watch":1,"likes":2,"to":1} is also equivalent to BoW1. It is also what we expect from a strict JSON object representation.

Note: if another document is like a union of these two,

<span id="mwfw"></span>(3) John likes to watch movies. Mary likes movies too. Mary also likes to watch football games.

its JavaScript representation will be:

<span id="mwgw"></span><span class="nx" id="mwhA">BoW3</span><span class="w" id="mwhQ"> </span><span class="o" id="mwhg">=</span><span class="w" id="mwhw"> </span><span class="p" id="mwiA">{</span><span class="s2" id="mwiQ">"John"</span><span class="o" id="mwig">:</span><span class="mf" id="mwiw">1</span><span class="p" id="mwjA">,</span><span class="s2" id="mwjQ">"likes"</span><span class="o" id="mwjg">:</span><span class="mf" id="mwjw">3</span><span class="p" id="mwkA">,</span><span class="s2" id="mwkQ">"to"</span><span class="o" id="mwkg">:</span><span class="mf" id="mwkw">2</span><span class="p" id="mwlA">,</span><span class="s2" id="mwlQ">"watch"</span><span class="o" id="mwlg">:</span><span class="mf" id="mwlw">2</span><span class="p" id="mwmA">,</span><span class="s2" id="mwmQ">"movies"</span><span class="o" id="mwmg">:</span><span class="mf" id="mwmw">2</span><span class="p" id="mwnA">,</span><span class="s2" id="mwnQ">"Mary"</span><span class="o" id="mwng">:</span><span class="mf" id="mwnw">2</span><span class="p" id="mwoA">,</span><span class="s2" id="mwoQ">"too"</span><span class="o" id="mwog">:</span><span class="mf" id="mwow">1</span><span class="p" id="mwpA">,</span><span class="s2" id="mwpQ">"also"</span><span class="o" id="mwpg">:</span><span class="mf" id="mwpw">1</span><span class="p" id="mwqA">,</span><span class="s2" id="mwqQ">"football"</span><span class="o" id="mwqg">:</span><span class="mf" id="mwqw">1</span><span class="p" id="mwrA">,</span><span class="s2" id="mwrQ">"games"</span><span class="o" id="mwrg">:</span><span class="mf" id="mwrw">1</span><span class="p" id="mwsA">};</span>

So, as we see in the bag algebra, the "union" of two documents in the bags-of-words representation is, formally, the disjoint union, summing the multiplicities of each element.

Word order

The BoW representation of a text removes all word ordering. For example, the BoW representation of "man bites dog" and "dog bites man" are the same, so any algorithm that operates with a BoW representation of text must treat them in the same way. Despite this lack of syntax or grammar, BoW representation is fast and may be sufficient for simple tasks that do not require word order. For instance, for document classification, if the words "stocks" "trade" "investors" appears multiple times, then the text is likely a financial report, even though it would be insufficient to distinguish between

Yesterday, investors were rallying, but today, they are retreating.

and

Yesterday, investors were retreating, but today, they are rallying.

and so the BoW representation would be insufficient to determine the detailed meaning of the document.