Independence (probability theory)

When dealing with collections of more than two events, two notions of independence need to be distinguished. The events are called pairwise independent if any two events in the collection are independent of each other, while mutual independence (or collective independence) of events means, informally speaking, that each event is independent of any combination of other events in the collection. A similar notion exists for collections of random variables. Mutual independence implies pairwise independence, but not the other way around. In the standard literature of probability theory, statistics, and stochastic processes, independence without further qualification usually refers to mutual independence.

Definition

For events

Two events

Two events ${\displaystyle A}$ and ${\displaystyle B}$ are independent (often written as ${\displaystyle A\perp B}$ or ${\displaystyle A\perp \perp B}$, where the latter symbol often is also used for conditional independence) if and only if their joint probability equals the product of their probabilities:

${\displaystyle A\cap B\ne \mathrm{\varnothing }}$ indicates that two independent events ${\displaystyle A}$ and ${\displaystyle B}$ have common elements in their sample space so that they are not mutually exclusive (mutually exclusive if and only if (iff) ${\displaystyle A\cap B=\mathrm{\varnothing }}$). Why this defines independence is made clear by rewriting with conditional probabilities ${\displaystyle P(A\mid B)=\frac{P(A\cap B)}{P(B)}}$ as the probability at which the event ${\displaystyle A}$ occurs provided that the event ${\displaystyle B}$ has or is assumed to have occurred:

${\displaystyle \mathrm{P}(A\cap B)=\mathrm{P}(A)\mathrm{P}(B)⟺\mathrm{P}(A\mid B)=\frac{\mathrm{P}(A\cap B)}{\mathrm{P}(B)}=\mathrm{P}(A).}$

and similarly

${\displaystyle \mathrm{P}(A\cap B)=\mathrm{P}(A)\mathrm{P}(B)⟺\mathrm{P}(B\mid A)=\frac{\mathrm{P}(A\cap B)}{\mathrm{P}(A)}=\mathrm{P}(B).}$

Thus, the occurrence of ${\displaystyle B}$ does not affect the probability of ${\displaystyle A}$, and vice versa. In other words, ${\displaystyle A}$ and ${\displaystyle B}$ are independent of each other. Although the derived expressions may seem more intuitive, they are not the preferred definition, as the conditional probabilities may be undefined if ${\displaystyle \mathrm{P}(A)}$ or ${\displaystyle \mathrm{P}(B)}$ are 0. Furthermore, the preferred definition makes clear by symmetry that when ${\displaystyle A}$ is independent of ${\displaystyle B}$, ${\displaystyle B}$ is also independent of ${\displaystyle A}$.

Odds

Stated in terms of odds, two events are independent if and only if the odds ratio of ⁠${\displaystyle A}$⁠ and ⁠${\displaystyle B}$⁠ is unity (1). Analogously with probability, this is equivalent to the conditional odds being equal to the unconditional odds:

${\displaystyle O(A\mid B)=O(A)\text{ and }O(B\mid A)=O(B),}$

or to the odds of one event, given the other event, being the same as the odds of the event, given the other event not occurring:

${\displaystyle O(A\mid B)=O(A\mid \mathrm{\neg }B)\text{ and }O(B\mid A)=O(B\mid \mathrm{\neg }A).}$

The odds ratio can be defined as

${\displaystyle O(A\mid B):O(A\mid \mathrm{\neg }B),}$

or symmetrically for odds of ⁠${\displaystyle B}$⁠ given ⁠${\displaystyle A}$⁠, and thus is 1 if and only if the events are independent.

More than two events

A finite set of events ${\displaystyle \{A_i{\}}_{i=1}^n}$ is pairwise independent if every pair of events is independent—that is, if and only if for all distinct pairs of indices ${\displaystyle m,k}$,

A finite set of events is mutually independent if every event is independent of any intersection of the other events—that is, if and only if for every ${\displaystyle k\le n}$ and for every k indices ${\displaystyle 1\le i_1<\cdots <i_k\le n}$,

This is called the multiplication rule for independent events. It is not a single condition involving only the product of all the probabilities of all single events; it must hold true for all subsets of events.

For more than two events, a mutually independent set of events is (by definition) pairwise independent; but the converse is not necessarily true.

Log probability and information content

Stated in terms of log probability, two events are independent if and only if the log probability of the joint event is the sum of the log probability of the individual events:

${\displaystyle \log \mathrm{P}(A\cap B)=\log \mathrm{P}(A)+\log \mathrm{P}(B)}$

In information theory, negative log probability is interpreted as information content, and thus two events are independent if and only if the information content of the combined event equals the sum of information content of the individual events:

${\displaystyle \mathrm{I}(A\cap B)=\mathrm{I}(A)+\mathrm{I}(B)}$

See Information content § Additivity of independent events for details.

For real valued random variables

Two random variables

Two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are independent if and only if (iff) the elements of the π-system generated by them are independent; that is to say, for every ${\displaystyle x}$ and ${\displaystyle y}$, the events ${\displaystyle \{X\le x\}}$ and ${\displaystyle \{Y\le y\}}$ are independent events (as defined above in Eq.1). That is, ${\displaystyle X}$ and ${\displaystyle Y}$ with cumulative distribution functions ${\displaystyle F_X(x)}$ and ${\displaystyle F_Y(y)}$, are independent iff the combined random variable ${\displaystyle (X,Y)}$ has a joint cumulative distribution function

More generally and equivalently if ${\displaystyle X}$ and ${\displaystyle Y}$ are real valued, if the pair of random variables ${\displaystyle (X,Y)}$ has values in ${\displaystyle \mathcal{X}\times \mathcal{Y}}$ with joint probability distribution ${\displaystyle P_{X,Y}}$ and marginals ${\displaystyle P_X}$ and ${\displaystyle P_Y}$ we have the equality of measures

${\displaystyle P_{X,Y}(d(x,y))=P_X(dx)P_Y(dy),}$

i.e. for every Borel set ${\displaystyle A\subseteq \mathcal{X}\times \mathcal{Y}}$ we have

${\displaystyle P_{X,Y}(A)={\int }_A{P}_{X,Y}(d(x,y))={\int }_A{P}_X(dx)P_Y(dy)}$

where ${\displaystyle P_{X,Y}(A)=P\left((X,Y)\in A\right)}$. If ${\displaystyle X}$ and ${\displaystyle Y}$ are discrete valued this simplifies to

${\displaystyle P_{X,Y}(x_i,y_j)=P_X(x_i)P_Y(y_j)\text{ for all }i=1,..,|\mathcal{X}|,j=1,..,|\mathcal{Y}|,}$

while if ${\displaystyle X}$ and ${\displaystyle Y}$ are real valued and have probability densities ${\displaystyle p_X(x)}$ and ${\displaystyle p_Y(y)}$ and joint probability density ${\displaystyle p_{X,Y}(x,y)}$ it becomes

${\displaystyle p_{X,Y}(x,y)=p_X(x)p_Y(y)\text{for almost all }(x,y)\in {\mathbb{R}}^2.}$

where "almost all" means all except for a set of measure zero.

More than two random variables

A finite set of ${\displaystyle n}$ random variables ${\displaystyle \{X_1,\dots ,X_n\}}$ is pairwise independent if and only if every pair of random variables is independent. Even if the set of random variables is pairwise independent, it is not necessarily mutually independent as defined next.

A finite set of ${\displaystyle n}$ random variables ${\displaystyle \{X_1,\dots ,X_n\}}$ is mutually independent if and only if for any sequence of numbers ${\displaystyle \{x_1,\dots ,x_n\}}$, the events ${\displaystyle \{X_1\le x_1\},\dots ,\{X_n\le x_n\}}$ are mutually independent events (as defined above in Eq.3). This is equivalent to the following condition on the joint cumulative distribution function ${\displaystyle F_{X_1,\dots ,X_n}(x_1,\dots ,x_n)}$. A finite set of ${\displaystyle n}$ random variables ${\displaystyle \{X_1,\dots ,X_n\}}$ is mutually independent if and only if

It is not necessary here to require that the probability distribution factorizes for all possible ${\displaystyle k}$-element subsets as in the case for ${\displaystyle n}$ events. This is not required because e.g. ${\displaystyle F_{X_1,X_2,X_3}(x_1,x_2,x_3)=F_{X_1}(x_1)\cdot F_{X_2}(x_2)\cdot F_{X_3}(x_3)}$ implies ${\displaystyle F_{X_1,X_3}(x_1,x_3)=F_{X_1}(x_1)\cdot F_{X_3}(x_3)}$.

Similarly to the case of two random variables, the general formulation of independence can be done measure-theoretically. A set of random variables ${\displaystyle (X_1,\dots ,X_n)}$ with values in measurable spaces ${\displaystyle {\mathcal{X}}_1,\times \dots \times {\mathcal{X}}_n}$ with joint distribution ${\displaystyle P_{X_1,\dots X_n}}$, and marginals ${\displaystyle P_{X_1}\dots P_{X_n}}$ are independent if and only if

${\displaystyle P_{X_1,\dots X_n}(d(x_1,\dots x_n))=P_{X_1}(d{x}_1)\cdots P_{X_n}(d{x}_n)}$

which again means that for every Borel set ${\displaystyle A\subseteq {\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_n}$, we have

${\displaystyle P_{X_1,\dots ,X_n}(A)={\int }_A{P}_{X_1,\dots X_n}(d(x_1,\dots x_n))={\int }_A{P}_{X_1}(d{x}_1)\cdots P_{X_n}(d{x}_n),}$

The definition is exactly equivalent to the one above when the values of the random variables are real numbers. It has the advantage of working also for complex-valued random variables or for random variables taking values in other measurable spaces (which includes topological spaces endowed by appropriate σ-algebras).

For real valued random vectors

Two random vectors ${\displaystyle \mathbf{X}=(X_1,\dots ,X_m{)}^{\mathrm{T}}}$ and ${\displaystyle \mathbf{Y}=(Y_1,\dots ,Y_n{)}^{\mathrm{T}}}$ are called independent if

where ${\displaystyle F_{\mathbf{X}}(\mathbf{x})}$ and ${\displaystyle F_{\mathbf{Y}}(\mathbf{y})}$ denote the cumulative distribution functions of ${\displaystyle \mathbf{X}}$ and ${\displaystyle \mathbf{Y}}$ and ${\displaystyle F_{\mathbf{X}\mathbf{,}\mathbf{Y}}(\mathbf{x}\mathbf{,}\mathbf{y})}$ denotes their joint cumulative distribution function. Independence of ${\displaystyle \mathbf{X}}$ and ${\displaystyle \mathbf{Y}}$ is often denoted by ${\displaystyle \mathbf{X}\perp \perp \mathbf{Y}}$. Written component-wise, ${\displaystyle \mathbf{X}}$ and ${\displaystyle \mathbf{Y}}$ are called independent if

${\displaystyle F_{X_1,\dots ,X_m,Y_1,\dots ,Y_n}(x_1,\dots ,x_m,y_1,\dots ,y_n)=F_{X_1,\dots ,X_m}(x_1,\dots ,x_m)\cdot F_{Y_1,\dots ,Y_n}(y_1,\dots ,y_n)\text{for all }{x}_1,\dots ,x_m,y_1,\dots ,y_n.}$

For stochastic processes

For one stochastic process

The definition of independence may be extended from random vectors to a stochastic process. Therefore, it is required for an independent stochastic process that the random variables obtained by sampling the process at any ${\displaystyle n}$ times ${\displaystyle t_1,\dots ,t_n}$ are independent random variables for any ${\displaystyle n}$.

Formally, a stochastic process ${\displaystyle {\left\{X_t\right\}}_{t\in \mathcal{T}}}$ is called independent, if and only if for all ${\displaystyle n\in \mathbb{N}}$ and for all ${\displaystyle t_1,\dots ,t_n\in \mathcal{T}}$

where ${\displaystyle F_{X_{t_1},\dots ,X_{t_n}}(x_1,\dots ,x_n)=\mathrm{P}(X(t_1)\le x_1,\dots ,X(t_n)\le x_n)}$. Independence of a stochastic process is a property within a stochastic process, not between two stochastic processes.

For two stochastic processes

Independence of two stochastic processes is a property between two stochastic processes ${\displaystyle {\left\{X_t\right\}}_{t\in \mathcal{T}}}$ and ${\displaystyle {\left\{Y_t\right\}}_{t\in \mathcal{T}}}$ that are defined on the same probability space ${\displaystyle (\mathrm{\Omega },\mathcal{F},P)}$. Formally, two stochastic processes ${\displaystyle {\left\{X_t\right\}}_{t\in \mathcal{T}}}$ and ${\displaystyle {\left\{Y_t\right\}}_{t\in \mathcal{T}}}$ are said to be independent if for all ${\displaystyle n\in \mathbb{N}}$ and for all ${\displaystyle t_1,\dots ,t_n\in \mathcal{T}}$, the random vectors ${\displaystyle (X(t_1),\dots ,X(t_n))}$ and ${\displaystyle (Y(t_1),\dots ,Y(t_n))}$ are independent, i.e. if

Independent σ-algebras

The definitions above (Eq.1 and Eq.2) are both generalized by the following definition of independence for σ-algebras. Let ${\displaystyle (\mathrm{\Omega },\mathrm{\Sigma },\mathrm{P})}$ be a probability space and let ${\displaystyle \mathcal{A}}$ and ${\displaystyle \mathcal{B}}$ be two sub-σ-algebras of ${\displaystyle \mathrm{\Sigma }}$. ${\displaystyle \mathcal{A}}$ and ${\displaystyle \mathcal{B}}$ are said to be independent if, whenever ${\displaystyle A\in \mathcal{A}}$ and ${\displaystyle B\in \mathcal{B}}$,

${\displaystyle \mathrm{P}(A\cap B)=\mathrm{P}(A)\mathrm{P}(B).}$

Likewise, a finite family of σ-algebras ${\displaystyle ({\tau }_i{)}_{i\in I}}$, where ${\displaystyle I}$ is an index set, is said to be independent if and only if

${\displaystyle \mathrm{\forall }{\left(A_i\right)}_{i\in I}\in {\prod }_{i\in I}{\tau }_i:\mathrm{P}\left({\bigcap }_{i\in I}{A}_i\right)={\prod }_{i\in I}\mathrm{P}\left(A_i\right)}$

and an infinite family of σ-algebras is said to be independent if all its finite subfamilies are independent.

The new definition relates to the previous ones very directly:

• Two events are independent (in the old sense) if and only if the σ-algebras that they generate are independent (in the new sense). The σ-algebra generated by an event ${\displaystyle E\in \mathrm{\Sigma }}$ is, by definition,

${\displaystyle \sigma (\{E\})=\{\mathrm{\varnothing },E,\mathrm{\Omega }\setminus E,\mathrm{\Omega }\}.}$

• Two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ defined over ${\displaystyle \mathrm{\Omega }}$ are independent (in the old sense) if and only if the σ-algebras that they generate are independent (in the new sense). The σ-algebra generated by a random variable ${\displaystyle X}$ taking values in some measurable space ${\displaystyle S}$ consists, by definition, of all subsets of ${\displaystyle \mathrm{\Omega }}$ of the form ${\displaystyle X^{-1}(U)}$, where ${\displaystyle U}$ is any measurable subset of ${\displaystyle S}$.

Using this definition, it is easy to show that if ${\displaystyle X}$ and ${\displaystyle Y}$ are random variables and ${\displaystyle Y}$ is constant, then ${\displaystyle X}$ and ${\displaystyle Y}$ are independent, since the σ-algebra generated by a constant random variable is the trivial σ-algebra ${\displaystyle \{\varnothing ,\mathrm{\Omega }\}}$. Probability zero events cannot affect independence so independence also holds if ${\displaystyle Y}$ is only Pr-almost surely constant.