Law of total variance

Examples

Example 1 (Exam scores)

Suppose five students take an exam scored 0–100. Let Y = student’s score and X indicate whether the student is *international* or *domestic*:

• Mean and variance for international: ${\displaystyle \mathrm{E}[Y\mid X=\text{Intl}]=50,\mathrm{Var}(Y\mid X=\text{Intl})\approx \mathrm{1266.7.}}$
• Mean and variance for domestic: ${\displaystyle \mathrm{E}[Y\mid X=\text{Dom}]=50,\mathrm{Var}(Y\mid X=\text{Dom})=100.}$

Both groups share the same mean (50), so the explained variance ${\displaystyle \mathrm{Var}(\mathrm{E}[Y\mid X])}$ is 0, and the total variance equals the average of the within-group variances (weighted by group size), i.e. 800.

Example 2 (Mixture of two Gaussians)

Let X be a coin flip taking values Heads with probability h and Tails with probability 1−h. Given Heads, ${\displaystyle Y\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}({\mu }_h,{\sigma }_h^2)}$; given Tails, ${\displaystyle Y\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}({\mu }_t,{\sigma }_t^2)}$. Then ${\displaystyle \mathrm{E}[\mathrm{Var}(Y\mid X)]=h{\sigma }_h^2+(1-h){\sigma }_t^2,}$ ${\displaystyle \mathrm{Var}(\mathrm{E}[Y\mid X])=h(1-h)({\mu }_h-{\mu }_t{)}^2,}$ so ${\displaystyle \mathrm{Var}(Y)=h{\sigma }_h^2+(1-h){\sigma }_t^2+h(1-h)({\mu }_h-{\mu }_t{)}^2.}$

Example 3 (Dice and coins)

Consider a two-stage experiment:

1. Roll a fair die (values 1–6) to choose one of six biased coins.
2. Flip that chosen coin; let Y=1 if Heads, 0 if Tails.

Then ${\displaystyle \mathrm{E}[Y\mid X=i]=p_i,\mathrm{Var}(Y\mid X=i)=p_i(1-p_i).}$ The overall variance of Y becomes $${\displaystyle \mathrm{Var}(Y)=\mathrm{E}[p_X(1-p_X)]+\mathrm{Var}(p_X),}$$ with ${\displaystyle p_X}$ uniform on ${\displaystyle \{p_1,\dots ,p_6\}.}$

Proof

Discrete/finite proof

Let ${\displaystyle (X_i,Y_i)}$, ${\displaystyle i=1,\dots ,n}$, be observed pairs. Define ${\displaystyle \overline{Y}=\mathrm{E}[Y].}$ Then $${\displaystyle \begin{array}{cc}\mathrm{Var}(Y)& =\frac{1}{n}\sum _{i=1}^n{\left(Y_i-\overline{Y}\right)}^2\\ & =\frac{1}{n}\sum _{i=1}^n{\left[\left(Y_i-{\overline{Y}}_{X_i}\right)+\left({\overline{Y}}_{X_i}-\overline{Y}\right)\right]}^2,\end{array}}$$ where ${\displaystyle {\overline{Y}}_{X_i}=\mathrm{E}[Y\mid X=X_i].}$ Expanding the square and noting the cross term cancels in summation yields: ${\displaystyle \mathrm{Var}(Y)=\mathrm{E}\left[\mathrm{Var}(Y\mid X)\right]+\mathrm{Var}\left(\mathrm{E}[Y\mid X]\right).}$

General case

Using ${\displaystyle \mathrm{Var}(Y)=\mathrm{E}[Y^2]-\mathrm{E}[Y{]}^2}$ and the law of total expectation: $${\displaystyle \begin{array}{cc}\mathrm{E}[Y^2]& =\mathrm{E}\left[\mathrm{E}(Y^2\mid X)\right]\\ & =\mathrm{E}\left[\mathrm{Var}(Y\mid X)+(\mathrm{E}[Y\mid X]{)}^2\right].\end{array}}$$ Subtract ${\displaystyle \mathrm{E}[Y{]}^2={\left(\mathrm{E}[\mathrm{E}(Y\mid X)]\right)}^2}$ and regroup to arrive at ${\displaystyle \mathrm{Var}(Y)=\mathrm{E}\left[\mathrm{Var}(Y\mid X)\right]+\mathrm{Var}\left(\mathrm{E}[Y\mid X]\right).}$

Applications

Analysis of Variance (ANOVA)

In a one-way analysis of variance, the total sum of squares (proportional to ${\displaystyle \mathrm{Var}(Y)}$) is split into a “between-group” sum of squares (${\displaystyle \mathrm{Var}(\mathrm{E}[Y\mid X])}$) plus a “within-group” sum of squares (${\displaystyle \mathrm{E}[\mathrm{Var}(Y\mid X)]}$). The F-test examines whether the explained component is sufficiently large to indicate X has a significant effect on Y.

Regression and R²

In linear regression and related models, if ${\displaystyle \hat{Y}=\mathrm{E}[Y\mid X],}$ the fraction of variance explained is $${\displaystyle \begin{array}{cc}{R}^2=\frac{\mathrm{Var}(\hat{Y})}{\mathrm{Var}(Y)}& =\frac{\mathrm{Var}(\mathrm{E}[Y\mid X])}{\mathrm{Var}(Y)}\\ & =1-\frac{\mathrm{E}[\mathrm{Var}(Y\mid X)]}{\mathrm{Var}(Y)}.\end{array}}$$ In the simple linear case (one predictor), ${\displaystyle R^2}$ also equals the square of the Pearson correlation coefficient between X and Y.

Machine learning and Bayesian inference

In many Bayesian and ensemble methods, one decomposes prediction uncertainty via the law of total variance. For a Bayesian neural network with random parameters ${\displaystyle \theta }$: ${\displaystyle \mathrm{Var}(Y)=\mathrm{E}\left[\mathrm{Var}(Y\mid \theta )\right]+\mathrm{Var}\left(\mathrm{E}[Y\mid \theta ]\right),}$ often referred to as “aleatoric” (within-model) vs. “epistemic” (between-model) uncertainty.

Actuarial science

Credibility theory uses the same partitioning: the expected value of process variance (EVPV), ${\displaystyle \mathrm{E}[\mathrm{Var}(Y\mid X)],}$ and the variance of hypothetical means (VHM), ${\displaystyle \mathrm{Var}(\mathrm{E}[Y\mid X]).}$ The ratio of explained to total variance determines how much “credibility” to give to individual risk classifications.

Information theory

For jointly Gaussian ${\displaystyle (X,Y)}$, the fraction ${\displaystyle \mathrm{Var}(\mathrm{E}[Y\mid X])/\mathrm{Var}(Y)}$ relates directly to the mutual information ${\displaystyle I(Y;X).}$ In non-Gaussian settings, a high explained-variance ratio still indicates significant information about Y contained in X.

Generalizations

The law of total variance generalizes to multiple or nested conditionings. For example, with two conditioning variables ${\displaystyle X_1}$ and ${\displaystyle X_2}$: $${\displaystyle \mathrm{Var}(Y)=\mathrm{E}\left[\mathrm{Var}(Y\mid X_1,X_2)\right]+\mathrm{E}\left[\mathrm{Var}(\mathrm{E}[Y\mid X_1,X_2]\mid X_1)\right]+\mathrm{Var}(\mathrm{E}[Y\mid X_1]).}$$ More generally, the law of total cumulance extends this approach to higher moments.