Pooled variance

Definition and computation

The pooled variance is an estimate of the fixed common variance ${\displaystyle {\sigma }^2}$ underlying various populations that have different means.

We are given a set of sample variances ${\displaystyle s_i^2}$, where the populations are indexed ${\displaystyle i=1,\dots ,m}$,

${\displaystyle s_i^2}$ = ${\displaystyle \frac{1}{n_i-1}\sum _{j=1}^{n_i}{\left(y_{i,j}-{\overline{y}}_i\right)}^2.}$

Assuming uniform sample sizes, ${\displaystyle n_i=n}$, then the pooled variance ${\displaystyle s_p^2}$ can be computed by the arithmetic mean:

${\displaystyle s_p^2=\frac{\sum _{i=1}^m{s}_i^2}{m}=\frac{s_1^2+s_2^2+\cdots +s_m^2}{m}.}$

If the sample sizes are non-uniform, then the pooled variance ${\displaystyle s_p^2}$ can be computed by the weighted average, using as weights ${\displaystyle w_i=n_i-1}$ the respective degrees of freedom (see also: Bessel's correction):

${\displaystyle s_p^2=\frac{\sum _{i=1}^m(n_i-1)s_i^2}{\sum _{i=1}^m(n_i-1)}=\frac{(n_1-1)s_1^2+(n_2-1)s_2^2+\cdots +(n_m-1)s_m^2}{n_1+n_2+\cdots +n_m-m}.}$

The distribution of ${\displaystyle \frac{s_p^2(\sum _i{n}_i-m)}{{\sigma }^2}}$ is ${\displaystyle {\chi }^2(\sum _i{n}_i-m)}$.

Proof. When there is a single mean, the distribution of ${\displaystyle (y_1-\overline{y},\dots ,y_n-\overline{y})}$ is a gaussian in ${\displaystyle {\mathrm{\Delta }}_{n-1}}$, the ${\displaystyle (n-1)}$-dimensional simplex, with standard deviation ${\displaystyle \sigma }$. Where there are multiple means, the distribution of ${\displaystyle (y_{1,1}-{\overline{y}}_1,\dots ,y_{1,n_1}-{\overline{y}}_1,\dots ,y_{m,1}-{\overline{y}}_m,\dots ,y_{m,n_m}-{\overline{y}}_m)}$ is a gaussian in ${\displaystyle {\mathrm{\Delta }}_{n_1-1}\times \cdots \times {\mathrm{\Delta }}_{n_m-1}}$.

Variants

The unbiased least squares estimate of ${\displaystyle {\sigma }^2}$ (as presented above), and the biased maximum likelihood estimate below:

${\displaystyle s_p^2=\frac{\sum _{i=1}^N(n_i-1)s_i^2}{\sum _{i=1}^N{n}_i},}$

are used in different contexts. The former can give an unbiased ${\displaystyle s_p^2}$ to estimate ${\displaystyle {\sigma }^2}$ when the two groups share an equal population variance. The latter one can give a more efficient ${\displaystyle s_p^2}$ to estimate ${\displaystyle {\sigma }^2}$, although subject to bias. Note that the quantities ${\displaystyle s_i^2}$ in the right hand sides of both equations are the unbiased estimates.

Example

Consider the following set of data for y obtained at various levels of the independent variable x.

The number of trials, mean, variance and standard deviation are presented in the next table.

These statistics represent the variance and standard deviation for each subset of data at the various levels of x. If we can assume that the same phenomena are generating random error at every level of x, the above data can be “pooled” to express a single estimate of variance and standard deviation. In a sense, this suggests finding a mean variance or standard deviation among the five results above. This mean variance is calculated by weighting the individual values with the size of the subset for each level of x. Thus, the pooled variance is defined by

${\displaystyle s_p^2=\frac{(n_1-1)s_1^2+(n_2-1)s_2^2+\cdots +(n_k-1)s_k^2}{(n_1-1)+(n_2-1)+\cdots +(n_k-1)}}$

where n₁, n₂, . . ., n_k are the sizes of the data subsets at each level of the variable x, and s₁², s₂², . . ., s_k² are their respective variances.

The pooled variance of the data shown above is therefore:

${\displaystyle s_p^2=2.764}$

Effect on precision

Pooled variance is an estimate when there is a correlation between pooled data sets or the average of the data sets is not identical. Pooled variation is less precise the more non-zero the correlation or distant the averages between data sets.

The variation of data for non-overlapping data sets is:

${\displaystyle {\sigma }_X^2=\frac{\sum _i\left[(N_{X_i}-1){\sigma }_{X_i}^2+N_{X_i}{\mu }_{X_i}^2\right]-\left[\sum _i{N}_{X_i}\right]{\mu }_X^2}{\sum _i{N}_{X_i}-1}}$

where the mean is defined as:

${\displaystyle {\mu }_X=\frac{\sum _i{N}_{X_i}{\mu }_{X_i}}{\sum _i{N}_{X_i}}}$

Given a biased maximum likelihood defined as:

${\displaystyle s_p^2=\frac{\sum _{i=1}^k(n_i-1)s_i^2}{\sum _{i=1}^k{n}_i},}$

Then the error in the biased maximum likelihood estimate is:

${\displaystyle \begin{array}{cc}\text{Error}& =s_p^2-{\sigma }_X^2\\ & =\frac{\sum _i(N_{X_i}-1)s_i^2}{\sum _i{N}_{X_i}}-\frac{1}{\sum _i{N}_{X_i}-1}\left(\sum _i\left[(N_{X_i}-1){\sigma }_{X_i}^2+N_{X_i}{\mu }_{X_i}^2\right]-\left[\sum _i{N}_{X_i}\right]{\mu }_X^2\right)\end{array}}$

Assuming N is large such that:

${\displaystyle \sum _i{N}_{X_i}\approx \sum _i{N}_{X_i}-1}$

Then the error in the estimate reduces to:

${\displaystyle \begin{array}{cc}E& =-\frac{\left(\sum _i\left[N_{X_i}{\mu }_{X_i}^2\right]-\left[\sum _i{N}_{X_i}\right]{\mu }_X^2\right)}{\sum _i{N}_{X_i}}\\ & ={\mu }_X^2-\frac{\sum _i\left[N_{X_i}{\mu }_{X_i}^2\right]}{\sum _i{N}_{X_i}}\end{array}}$

Or alternatively:

${\displaystyle \begin{array}{cc}E& ={\left[\frac{\sum _i{N}_{X_i}{\mu }_{X_i}}{\sum _i{N}_{X_i}}\right]}^2-\frac{\sum _i\left[N_{X_i}{\mu }_{X_i}^2\right]}{\sum _i{N}_{X_i}}\\ & =\frac{{\left[\sum _i{N}_{X_i}{\mu }_{X_i}\right]}^2-\sum _i{N}_{X_i}\sum _i\left[N_{X_i}{\mu }_{X_i}^2\right]}{{\left[\sum _i{N}_{X_i}\right]}^2}\end{array}}$

Aggregation of standard deviation data

Rather than estimating pooled standard deviation, the following is the way to exactly aggregate standard deviation when more statistical information is available.

Population-based statistics

The populations of sets, which may overlap, can be calculated simply as follows:

${\displaystyle \begin{array}{cccc}& & N_{X\cup Y}& =N_X+N_Y-N_{X\cap Y}\end{array}}$

The populations of sets, which do not overlap, can be calculated simply as follows:

${\displaystyle \begin{array}{cccc}X\cap Y=\varnothing & \Rightarrow & N_{X\cap Y}& =0\\ & \Rightarrow & N_{X\cup Y}& =N_X+N_Y\end{array}}$

Standard deviations of non-overlapping (X ∩ Y = ∅) sub-populations can be aggregated as follows if the size (actual or relative to one another) and means of each are known:

${\displaystyle \begin{array}{cc}{\mu }_{X\cup Y}& =\frac{N_X{\mu }_X+N_Y{\mu }_Y}{N_X+N_Y}\\ {\sigma }_{X\cup Y}& =\sqrt{\frac{N_X{\sigma }_X^2+N_Y{\sigma }_Y^2}{N_X+N_Y}+\frac{N_X{N}_Y}{(N_X+N_Y{)}^2}({\mu }_X-{\mu }_Y{)}^2}\end{array}}$

For example, suppose it is known that the average American man has a mean height of 70 inches with a standard deviation of three inches and that the average American woman has a mean height of 65 inches with a standard deviation of two inches. Also assume that the number of men, N, is equal to the number of women. Then the mean and standard deviation of heights of American adults could be calculated as

${\displaystyle \begin{array}{cc}\mu & =\frac{N\cdot 70+N\cdot 65}{N+N}=\frac{70+65}{2}=67.5\\ \sigma & =\sqrt{\frac{3^2+2^2}{2}+\frac{(70-65{)}^2}{2^2}}=\sqrt{12.75}\approx 3.57\end{array}}$

For the more general case of M non-overlapping populations, X₁ through X_M, and the aggregate population $X=\bigcup _i{X}_i$,

${\displaystyle \begin{array}{cc}{\mu }_X& =\frac{\sum _i{N}_{X_i}{\mu }_{X_i}}{\sum _i{N}_{X_i}}\\ {\sigma }_X& =\sqrt{\frac{\sum _i{N}_{X_i}{\sigma }_{X_i}^2}{\sum _i{N}_{X_i}}+\frac{\sum _{i<j}{N}_{X_i}{N}_{X_j}({\mu }_{X_i}-{\mu }_{X_j}{)}^2}{(\sum _i{N}_{X_i}{)}^2}}\end{array}}$,

where

${\displaystyle X_i\cap X_j=\varnothing ,\mathrm{\forall }i<j.}$

If the size (actual or relative to one another), mean, and standard deviation of two overlapping populations are known for the populations as well as their intersection, then the standard deviation of the overall population can still be calculated as follows:

${\displaystyle \begin{array}{cc}{\mu }_{X\cup Y}& =\frac{1}{N_{X\cup Y}}\left(N_X{\mu }_X+N_Y{\mu }_Y-N_{X\cap Y}{\mu }_{X\cap Y}\right)\\ {\sigma }_{X\cup Y}& =\sqrt{\frac{1}{N_{X\cup Y}}\left(N_X[{\sigma }_X^2+{\mu }_X^2]+N_Y[{\sigma }_Y^2+{\mu }_Y^2]-N_{X\cap Y}[{\sigma }_{X\cap Y}^2+{\mu }_{X\cap Y}^2]\right)-{\mu }_{X\cup Y}^2}\end{array}}$

If two or more sets of data are being added together datapoint by datapoint, the standard deviation of the result can be calculated if the standard deviation of each data set and the covariance between each pair of data sets is known:

${\displaystyle {\sigma }_X=\sqrt{\sum _i{\sigma }_{X_i}^2+2\sum _{i,j}\mathrm{cov}(X_i,X_j)}}$

For the special case where no correlation exists between any pair of data sets, then the relation reduces to the root sum of squares:

${\displaystyle \begin{array}{cc}& \mathrm{cov}(X_i,X_j)=0,\mathrm{\forall }i<j\\ \Rightarrow & {\sigma }_X=\sqrt{\sum _i{\sigma }_{X_i}^2}.\end{array}}$

Sample-based statistics

Standard deviations of non-overlapping (X ∩ Y = ∅) sub-samples can be aggregated as follows if the actual size and means of each are known:

${\displaystyle \begin{array}{cc}{\mu }_{X\cup Y}& =\frac{1}{N_{X\cup Y}}\left(N_X{\mu }_X+N_Y{\mu }_Y\right)\\ {\sigma }_{X\cup Y}& =\sqrt{\frac{1}{N_{X\cup Y}-1}\left([N_X-1]{\sigma }_X^2+N_X{\mu }_X^2+[N_Y-1]{\sigma }_Y^2+N_Y{\mu }_Y^2-[N_X+N_Y]{\mu }_{X\cup Y}^2\right)}\end{array}}$

For the more general case of M non-overlapping data sets, X₁ through X_M, and the aggregate data set $X=\bigcup _i{X}_i$,

${\displaystyle \begin{array}{cc}{\mu }_X& =\frac{1}{\sum _i{N}_{X_i}}\left(\sum _i{N}_{X_i}{\mu }_{X_i}\right)\\ {\sigma }_X& =\sqrt{\frac{1}{\sum _i{N}_{X_i}-1}\left(\sum _i\left[(N_{X_i}-1){\sigma }_{X_i}^2+N_{X_i}{\mu }_{X_i}^2\right]-\left[\sum _i{N}_{X_i}\right]{\mu }_X^2\right)}\end{array}}$

where

${\displaystyle X_i\cap X_j=\varnothing ,\mathrm{\forall }i<j.}$

If the size, mean, and standard deviation of two overlapping samples are known for the samples as well as their intersection, then the standard deviation of the aggregated sample can still be calculated. In general,

${\displaystyle \begin{array}{cc}{\mu }_{X\cup Y}& =\frac{1}{N_{X\cup Y}}\left(N_X{\mu }_X+N_Y{\mu }_Y-N_{X\cap Y}{\mu }_{X\cap Y}\right)\\ {\sigma }_{X\cup Y}& =\sqrt{\frac{[N_X-1]{\sigma }_X^2+N_X{\mu }_X^2+[N_Y-1]{\sigma }_Y^2+N_Y{\mu }_Y^2-[N_{X\cap Y}-1]{\sigma }_{X\cap Y}^2-N_{X\cap Y}{\mu }_{X\cap Y}^2-[N_X+N_Y-N_{X\cap Y}]{\mu }_{X\cup Y}^2}{N_{X\cup Y}-1}}\end{array}}$