Information content

Properties

Monotonically decreasing function of probability

For a given probability space, the measurement of rarer events are intuitively more "surprising", and yield more information content than more "common" events. Thus, self-information is a strictly decreasing monotonic function of the probability, or sometimes called an "antitonic" function.

While standard probabilities are represented by real numbers in the interval ${\displaystyle [0,1]}$, self-information values are non-negative extended real numbers in the interval ${\displaystyle [0,\mathrm{\infty }]}$. Specifically:

• An event with probability ${\displaystyle Pr(x)=1}$ (a certain event) has an information content of ${\displaystyle \mathrm{I}(x)=-{\log }_b(1)=0}$. Its occurrence is perfectly unsurprising and reveals no new information.
• An event with probability ${\displaystyle Pr(x)=0}$ (an impossible event) has an information content of ${\displaystyle \mathrm{I}(x)=-{\log }_b(0)}$, which is undefined but is taken to be ${\displaystyle \mathrm{\infty }}$ by convention. This reflects that observing an event believed to be impossible would be infinitely surprising.

This monotonic relationship is fundamental to the use of information content as a measure of uncertainty. For example, learning that a one-in-a-million lottery ticket won provides far more information than learning it lost (See also Lottery mathematics.) This also establishes an intuitive connection to concepts like statistical dispersion; events that are far from the mean or typical outcome (and thus have low probability in many common distributions) have high self-information.

Relationship to log-odds

The Shannon information is closely related to the log-odds. The log-odds of an event ${\displaystyle x}$, with probability ${\displaystyle p(x)}$, is defined as the logarithm of the odds, ${\displaystyle \frac{p(x)}{1-p(x)}}$. This can be expressed as a difference of two information content values:$${\displaystyle {\displaystyle \begin{array}{cc}\text{log-odds}(x)& ={\log }_b\left(\frac{p(x)}{1-p(x)}\right)\\ & ={\log }_b(p(x))-{\log }_b(1-p(x))\\ & =\mathrm{I}(\mathrm{\neg }x)-\mathrm{I}(x),\end{array}}}$$where ${\displaystyle \mathrm{\neg }x}$ denotes the event not ${\displaystyle x}$.

This expression can be interpreted as the amount of information gained (or surprise) from learning the event did not occur, minus the information gained from learning it did occur. This connection is particularly relevant in statistical modeling where log-odds are the core of the logit function and logistic regression.

Additivity of independent events

The information content of two independent events is the sum of each event's information content. This property is known as additivity in mathematics. Consider two independent random variables ${\displaystyle X}$ and ${\displaystyle Y}$ with probability mass functions ${\displaystyle p_X(x)}$ and ${\displaystyle p_Y(y)}$. The joint probability of observing the outcome ${\displaystyle (x,y)}$ is given by the product of the individual probabilities due to independence:$${\displaystyle p_{X,Y}(x,y)=Pr(X=x,Y=y)=p_X(x)p_Y(y)}$$The information content of this joint event is:$${\displaystyle {\displaystyle \begin{array}{cc}{\mathrm{I}}_{X,Y}(x,y)& =-{\log }_b\left[p_{X,Y}(x,y)\right]\\ & =-{\log }_b\left[p_X(x)p_Y(y)\right]\\ & =-{\log }_b\left[p_X(x)\right]-{\log }_b\left[p_Y(y)\right]\\ & ={\mathrm{I}}_X(x)+{\mathrm{I}}_Y(y),\end{array}}}$$This additivity makes information content a more mathematically convenient measure than probability in many applications, such as in coding theory where the amount of information needed to describe a sequence of independent symbols is the sum of the information needed for each symbol.

The corresponding property for likelihoods is that the log-likelihood of independent events is the sum of the log-likelihoods of each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information.

Relationship to entropy

The Shannon entropy of the random variable ${\displaystyle X}$ is defined as:$${\displaystyle {\displaystyle \begin{array}{cc}\mathrm{H}(X)& =\sum _x-p_X\left(x\right)\log p_X\left(x\right)\\ & =\sum _x{p}_X\left(x\right){\mathrm{I}}_X(x)\stackrel{\underset{\mathrm{d}\mathrm{e}\mathrm{f}}{}}{=}\mathrm{E}\left[{\mathrm{I}}_X(X)\right],\end{array}}}$$by definition equal to the expected information content of measurement of ${\displaystyle X}$.

The expectation is taken over the discrete values over its support.

Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies ${\displaystyle \mathrm{H}(X)=\mathrm{I}(X;X)}$, where ${\displaystyle \mathrm{I}(X;X)}$ is the mutual information of ${\displaystyle X}$ with itself.

For continuous random variables the corresponding concept is differential entropy.

Examples

Fair coin toss

Consider the Bernoulli trial of tossing a fair coin ${\displaystyle X}$. The probabilities of the events of the coin landing as heads ${\displaystyle \text{H}}$ and tails ${\displaystyle \text{T}}$ (see fair coin and obverse and reverse) are one half each, $p_X(\text{H})=p_X(\text{T})=\frac{1}{2}=0.5$. Upon measuring the variable as heads, the associated information gain is $${\displaystyle {\mathrm{I}}_X(\text{H})=-{\log }_2{p}_X(\text{H})=-{\log }_2\frac{1}{2}=1,}$$so the information gain of a fair coin landing as heads is 1 shannon. Likewise, the information gain of measuring tails ${\displaystyle T}$ is$${\displaystyle {\mathrm{I}}_X(T)=-{\log }_2{p}_X(\text{T})=-{\log }_2\frac{1}{2}=1\text{ Sh}.}$$

Fair die roll

Suppose we have a fair six-sided die. The value of a die roll is a discrete uniform random variable ${\displaystyle X\sim \mathrm{D}\mathrm{U}[1,6]}$ with probability mass function $${\displaystyle p_X(k)=\{\begin{array}{ll}\frac{1}{6},& k\in \{1,2,3,4,5,6\}\\ 0,& \text{otherwise}\end{array}}$$The probability of rolling a 4 is $p_X(4)=\frac{1}{6}$, as for any other valid roll. The information content of rolling a 4 is thus$${\displaystyle {\mathrm{I}}_X(4)=-{\log }_2{p}_X(4)=-{\log }_2\frac{1}{6}\approx 2.585\text{Sh}}$$of information.

Two independent, identically distributed dice

Suppose we have two independent, identically distributed random variables $X,Y\sim \mathrm{D}\mathrm{U}[1,6]$ each corresponding to an independent fair 6-sided dice roll. The joint distribution of ${\displaystyle X}$ and ${\displaystyle Y}$ is$${\displaystyle \begin{array}{cc}{p}_{X,Y}\left(x,y\right)& =Pr(X=x,Y=y)=p_X(x)p_Y(y)\\ & =\{\begin{array}{ll}{\displaystyle \frac{1}{36},}& x,y\in [1,6]\cap \mathbb{N}\\ 0& \text{otherwise.}\end{array}\end{array}}$$

The information content of the random variate ${\displaystyle (X,Y)=(2,4)}$ is $${\displaystyle \begin{array}{cc}{\mathrm{I}}_{X,Y}(2,4)& =-{\log }_2\left[p_{X,Y}(2,4)\right]={\log }_{2}36=2{\log }_{2}6\\ & \approx 5.169925\text{ Sh},\end{array}}$$ and can also be calculated by additivity of events $${\displaystyle \begin{array}{cc}{\mathrm{I}}_{X,Y}(2,4)& =-{\log }_2\left[p_{X,Y}(2,4)\right]=-{\log }_2\left[p_X(2)\right]-{\log }_2\left[p_Y(4)\right]\\ & =2{\log }_{2}6\\ & \approx 5.169925\text{ Sh}.\end{array}}$$

Information from frequency of rolls

If we receive information about the value of the dice without knowledge of which die had which value, we can formalize the approach with so-called counting variables $${\displaystyle C_k:={\delta }_k(X)+{\delta }_k(Y)=\{\begin{array}{ll}0,& \mathrm{\neg }(X=k\vee Y=k)\\ 1,& X=k⊻Y=k\\ 2,& X=k\wedge Y=k\end{array}}$$ for ${\displaystyle k\in \{1,2,3,4,5,6\}}$, then $\sum _{k=1}^6{C}_k=2$ and the counts have the multinomial distribution $${\displaystyle \begin{array}{cc}f(c_1,\dots ,c_6)& =Pr(C_1=c_1\text{ and }\dots \text{ and }{C}_6=c_6)\\ & =\{\begin{array}{ll}{\displaystyle \frac{1}{18}\frac{1}{c_1!\cdots c_k!}},& \text{when }\sum _{i=1}^6{c}_i=2\\ 0& \text{otherwise,}\end{array}\\ & =\{\begin{array}{ll}\frac{1}{18},& \text{when 2 }{c}_k\text{ are }1\\ \frac{1}{36},& \text{when exactly one }{c}_k=2\\ 0,& \text{otherwise.}\end{array}\end{array}}$$

To verify this, the 6 outcomes $(X,Y)\in {\left\{(k,k)\right\}}_{k=1}^6=\left\{(1,1),(2,2),(3,3),(4,4),(5,5),(6,6)\right\}$ correspond to the event ${\displaystyle C_k=2}$ and a total probability of ⁠1/6⁠. These are the only events that are faithfully preserved with identity of which dice rolled which outcome because the outcomes are the same. Without knowledge to distinguish the dice rolling the other numbers, the other $(\genfrac{}{}{0ex}{}{6}{2})=15$ combinations correspond to one die rolling one number and the other die rolling a different number, each having probability ⁠1/18⁠. Indeed, $6\cdot \frac{1}{36}+15\cdot \frac{1}{18}=1$, as required.

Unsurprisingly, the information content of learning that both dice were rolled as the same particular number is more than the information content of learning that one die was one number and the other was a different number. Take for examples the events ${\displaystyle A_k=\{(X,Y)=(k,k)\}}$ and ${\displaystyle B_{j,k}=\{c_j=1\}\cap \{c_k=1\}}$ for ${\displaystyle j\ne k,1\le j,k\le 6}$. For example, ${\displaystyle A_2=\{X=2\text{ and }Y=2\}}$ and ${\displaystyle B_{3,4}=\{(3,4),(4,3)\}}$.

The information contents are $${\displaystyle \mathrm{I}(A_2)=-{\log }_2\frac{1}{36}=5.169925\text{ Sh}}$$ $${\displaystyle \mathrm{I}\left(B_{3,4}\right)=-{\log }_2\frac{1}{18}=4.169925\text{ Sh}}$$

Let $\text{Same}=\bigcup _{i=1}^6{A}_i$ be the event that both dice rolled the same value and ${\displaystyle \text{Diff}=\overline{\text{Same}}}$ be the event that the dice differed. Then $Pr(\text{Same})=\frac{1}{6}$ and $Pr(\text{Diff})=\frac{5}{6}$. The information contents of the events are $${\displaystyle \mathrm{I}(\text{Same})=-{\log }_2\frac{1}{6}=2.5849625\text{ Sh}}$$ $${\displaystyle \mathrm{I}(\text{Diff})=-{\log }_2\frac{5}{6}=0.2630344\text{ Sh}.}$$

Information from sum of dice

The probability mass or density function (collectively probability measure) of the sum of two independent random variables is the convolution of each probability measure. In the case of independent fair 6-sided dice rolls, the random variable ${\displaystyle Z=X+Y}$ has probability mass function $p_Z(z)=p_X(x)\ast p_Y(y)=\frac{6-|z-7|}{36}$, where ${\displaystyle \ast }$ represents the discrete convolution. The outcome ${\displaystyle Z=5}$ has probability $p_Z(5)=\frac{4}{36}=\frac{1}{9}$. Therefore, the information asserted is$${\displaystyle {\mathrm{I}}_Z(5)=-{\log }_2\frac{1}{9}={\log }_{2}9\approx 3.169925\text{ Sh}.}$$

General discrete uniform distribution

Generalizing the § Fair dice roll example above, consider a general discrete uniform random variable (DURV) ${\displaystyle X\sim \mathrm{D}\mathrm{U}[a,b];a,b\in \mathbb{Z},b\ge a.}$ For convenience, define $N:=b-a+1$. The probability mass function is $${\displaystyle p_X(k)=\{\begin{array}{ll}\frac{1}{N},& k\in [a,b]\cap \mathbb{Z}\\ 0,& \text{otherwise}.\end{array}}$$In general, the values of the DURV need not be integers, or for the purposes of information theory even uniformly spaced; they need only be equiprobable. The information gain of any observation ${\displaystyle X=k}$ is$${\displaystyle {\mathrm{I}}_X(k)=-{\log }_2\frac{1}{N}={\log }_{2}N\text{ Sh}.}$$

Special case: constant random variable

If ${\displaystyle b=a}$ above, ${\displaystyle X}$ degenerates to a constant random variable with probability distribution deterministically given by ${\displaystyle X=b}$ and probability measure the Dirac measure $p_X(k)={\delta }_b(k)$. The only value ${\displaystyle X}$ can take is deterministically ${\displaystyle b}$, so the information content of any measurement of ${\displaystyle X}$ is$${\displaystyle {\mathrm{I}}_X(b)=-{\log }_{2}1=0.}$$In general, there is no information gained from measuring a known value.

Categorical distribution

Generalizing all of the above cases, consider a categorical discrete random variable with support $\mathcal{S}=\{s_i{\}}_{i=1}^N$ and probability mass function given by

$${\displaystyle p_X(k)=\{\begin{array}{ll}{p}_i,& k=s_i\in \mathcal{S}\\ 0,& \text{otherwise}.\end{array}}$$

For the purposes of information theory, the values ${\displaystyle s\in \mathcal{S}}$ do not have to be numbers; they can be any mutually exclusive events on a measure space of finite measure that has been normalized to a probability measure ${\displaystyle p}$. Without loss of generality, we can assume the categorical distribution is supported on the set $[N]=\left\{1,2,\dots ,N\right\}$; the mathematical structure is isomorphic in terms of probability theory and therefore information theory as well.

The information of the outcome ${\displaystyle X=x}$ is given

$${\displaystyle {\mathrm{I}}_X(x)=-{\log }_2{p}_X(x).}$$

From these examples, it is possible to calculate the information of any set of independent DRVs with known distributions by additivity.

Derivation

By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information.

For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin:

Weather forecast for tonight: dark. ] Continued dark overnight, with widely scattered light by morning.

Assuming that one does not reside near the polar regions, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.

Accordingly, the amount of self-information ${\displaystyle \mathrm{I}}$ contained in a message conveying an occurrence of event, ${\displaystyle {\omega }_n}$, depends only on the probability ${\displaystyle Pr({\omega }_n)}$ of that event.$${\displaystyle \mathrm{I}({\omega }_n)=f(Pr({\omega }_n)),}$$for some function ${\displaystyle f}$ to be determined. If ${\displaystyle Pr({\omega }_n)=1}$, then ${\displaystyle \mathrm{I}({\omega }_n)=0}$. If ${\displaystyle Pr({\omega }_n)<1}$, then ${\displaystyle \mathrm{I}({\omega }_n)>0}$.

Further, by definition, the measure of self-information is nonnegative and additive. If an event ${\displaystyle C}$ is the intersection of two independent events ${\displaystyle A}$ and ${\displaystyle B}$, then the information of event ${\displaystyle C}$ occurring is the sum of the amounts of information of the individual events ${\displaystyle A}$ and ${\displaystyle B}$:$${\displaystyle \mathrm{I}(C)=\mathrm{I}(A\cap B)=\mathrm{I}(A)+\mathrm{I}(B).}$$Because of the independence of events ${\displaystyle A}$ and ${\displaystyle B}$, the probability of event ${\displaystyle C}$ is:$${\displaystyle Pr(C)=Pr(A\cap B)=Pr(A)\cdot Pr(B).}$$Relating the probabilities to the function ${\displaystyle f}$:$${\displaystyle f(Pr(A)\cdot Pr(B))=f(Pr(A))+f(Pr(B)).}$$This is a functional equation. The only continuous functions ${\displaystyle f}$ with this property are the logarithm functions. Therefore, ${\displaystyle f(p)}$ must be of the form:$${\displaystyle f(p)=K{\log }_b(p),}$$for some base ${\displaystyle b}$ and constant ${\displaystyle K}$. Since a low-probability event must correspond to high information content, the constant ${\displaystyle K}$ must be negative. We can write ${\displaystyle K=-1}$ and absorb any scaling into the base ${\displaystyle b}$ of the logarithm. This gives the final form:$${\displaystyle \mathrm{I}({\omega }_n)=-{\log }_b(Pr({\omega }_n))={\log }_b\left(\frac{1}{Pr({\omega }_n)}\right).}$$The smaller the probability of event ${\displaystyle {\omega }_n}$, the larger the quantity of self-information associated with the message that the event indeed occurred. If the above logarithm is base 2, the unit of ${\displaystyle I({\omega }_n)}$ is shannon. This is the most common practice. When using the natural logarithm of base ${\displaystyle e}$, the unit will be the nat. For the base 10 logarithm, the unit of information is the hartley.

As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 shannons (probability 1/16), and the information content associated with getting a result other than the one specified would be ${\displaystyle -{\log }_2(15/16)\approx 0.09}$ shannons. See above for detailed examples.