Cochran's theorem

Alternative formulation

The following version is often seen when considering linear regression. Suppose that ${\displaystyle Y\sim N_n(0,{\sigma }^2{I}_n)}$ is a standard multivariate normal random vector (here ${\displaystyle I_n}$ denotes the n-by-n identity matrix), and if ${\displaystyle A_1,\dots ,A_k}$ are all n-by-n symmetric matrices with ${\displaystyle \sum _{i=1}^k{A}_i=I_n}$. Then, on defining ${\displaystyle r_i=\mathrm{rank}(A_i)}$, any one of the following conditions implies the other two:

• ${\displaystyle \sum _{i=1}^k{r}_i=n,}$
• ${\displaystyle Y^T{A}_{i}Y\sim {\sigma }^2{\chi }_{r_i}^2}$ (thus the ${\displaystyle A_i}$ are positive semidefinite)
• ${\displaystyle Y^T{A}_{i}Y}$ is independent of ${\displaystyle Y^T{A}_{j}Y}$ for ${\displaystyle i\ne j.}$

Examples

Sample mean and sample variance

If X₁, ..., X_n are independent normally distributed random variables with mean μ and standard deviation σ then

${\displaystyle U_i=\frac{X_i-\mu }{\sigma }}$

is standard normal for each i. Note that the total Q is equal to sum of squared Us as shown here:

${\displaystyle \sum _i{Q}_i=\sum _{jik}{U}_j{B}_{jk}^{(i)}{U}_k=\sum _{jk}{U}_j{U}_k\sum _i{B}_{jk}^{(i)}=\sum _{jk}{U}_j{U}_k{\delta }_{jk}=\sum _j{U}_j^2}$

which stems from the original assumption that ${\displaystyle B_1+B_2\dots =I}$. So instead we will calculate this quantity and later separate it into Q_i's. It is possible to write

${\displaystyle \sum _{i=1}^n{U}_i^2=\sum _{i=1}^n{\left(\frac{X_i-\overline{X}}{\sigma }\right)}^2+n{\left(\frac{\overline{X}-\mu }{\sigma }\right)}^2}$

(here ${\displaystyle \overline{X}}$ is the sample mean). To see this identity, multiply throughout by ${\displaystyle {\sigma }^2}$ and note that

${\displaystyle \sum (X_i-\mu {)}^2=\sum (X_i-\overline{X}+\overline{X}-\mu {)}^2}$

and expand to give

${\displaystyle \sum (X_i-\mu {)}^2=\sum (X_i-\overline{X}{)}^2+\sum (\overline{X}-\mu {)}^2+2\sum (X_i-\overline{X})(\overline{X}-\mu ).}$

The third term is zero because it is equal to a constant times

${\displaystyle \sum (\overline{X}-X_i)=0,}$

and the second term has just n identical terms added together. Thus

${\displaystyle \sum (X_i-\mu {)}^2=\sum (X_i-\overline{X}{)}^2+n(\overline{X}-\mu {)}^2,}$

and hence

${\displaystyle \sum {\left(\frac{X_i-\mu }{\sigma }\right)}^2=\sum {\left(\frac{X_i-\overline{X}}{\sigma }\right)}^2+n{\left(\frac{\overline{X}-\mu }{\sigma }\right)}^2=\stackrel{Q_1}{\overbrace{\sum _i{\left(U_i-\frac{1}{n}\sum _j{U}_j\right)}^2}}+\stackrel{Q_2}{\overbrace{\frac{1}{n}{\left(\sum _j{U}_j\right)}^2}}=Q_1+Q_2.}$

Now ${\displaystyle B^{(2)}=\frac{J_n}{n}}$ with ${\displaystyle J_n}$ the matrix of ones which has rank 1. In turn ${\displaystyle B^{(1)}=I_n-\frac{J_n}{n}}$ given that ${\displaystyle I_n=B^{(1)}+B^{(2)}}$. This expression can be also obtained by expanding ${\displaystyle Q_1}$ in matrix notation. It can be shown that the rank of ${\displaystyle B^{(1)}}$ is ${\displaystyle n-1}$ as the addition of all its rows is equal to zero. Thus the conditions for Cochran's theorem are met.

Cochran's theorem then states that Q₁ and Q₂ are independent, with chi-squared distributions with n − 1 and 1 degree of freedom respectively. This shows that the sample mean and sample variance are independent. This can also be shown by Basu's theorem, and in fact this property characterizes the normal distribution – for no other distribution are the sample mean and sample variance independent.

Distributions

The result for the distributions is written symbolically as

${\displaystyle \sum {\left(X_i-\overline{X}\right)}^2\sim {\sigma }^2{\chi }_{n-1}^2.}$

${\displaystyle n(\overline{X}-\mu {)}^2\sim {\sigma }^2{\chi }_1^2,}$

Both these random variables are proportional to the true but unknown variance σ². Thus their ratio does not depend on σ² and, because they are statistically independent. The distribution of their ratio is given by

${\displaystyle \frac{n{\left(\overline{X}-\mu \right)}^2}{\frac{1}{n-1}\sum {\left(X_i-\overline{X}\right)}^2}\sim \frac{{\chi }_1^2}{\frac{1}{n-1}{\chi }_{n-1}^2}\sim F_{1,n-1}}$

where F_{1,n − 1} is the F-distribution with 1 and n − 1 degrees of freedom (see also Student's t-distribution). The final step here is effectively the definition of a random variable having the F-distribution.

Estimation of variance

To estimate the variance σ², one estimator that is sometimes used is the maximum likelihood estimator of the variance of a normal distribution

${\displaystyle {\hat{\sigma }}^2=\frac{1}{n}\sum {\left(X_i-\overline{X}\right)}^2.}$

Cochran's theorem shows that

${\displaystyle \frac{n{\hat{\sigma }}^2}{{\sigma }^2}\sim {\chi }_{n-1}^2}$

and the properties of the chi-squared distribution show that

${\displaystyle \begin{array}{cc}E\left(\frac{n{\hat{\sigma }}^2}{{\sigma }^2}\right)& =E\left({\chi }_{n-1}^2\right)\\ \frac{n}{{\sigma }^2}E\left({\hat{\sigma }}^2\right)& =(n-1)\\ E\left({\hat{\sigma }}^2\right)& =\frac{{\sigma }^2(n-1)}{n}\end{array}}$

Proof

Claim: Let ${\displaystyle X}$ be a standard Gaussian in ${\displaystyle {\mathbb{R}}^n}$, then for any symmetric matrices ${\displaystyle Q,Q^{\prime }}$, if ${\displaystyle X^{T}QX}$ and ${\displaystyle X^T{Q}^{\prime }X}$ have the same distribution, then ${\displaystyle Q,Q^{\prime }}$ have the same eigenvalues (up to multiplicity).

Proof

Let the eigenvalues of ${\displaystyle Q}$ be ${\displaystyle {\lambda }_1,...,{\lambda }_n}$, then calculate the characteristic function of ${\displaystyle X^{T}QX}$. It comes out to be

${\displaystyle \varphi (t)={\left(\prod _j(1-2i{\lambda }_{j}t)\right)}^{-1/2}}$

(To calculate it, first diagonalize ${\displaystyle Q}$, change into that frame, then use the fact that the characteristic function of the sum of independent variables is the product of their characteristic functions.)

For ${\displaystyle X^{T}QX}$ and ${\displaystyle X^T{Q}^{\prime }X}$ to be equal, their characteristic functions must be equal, so ${\displaystyle Q,Q^{\prime }}$ have the same eigenvalues (up to multiplicity).

Claim: ${\displaystyle I=\sum _i{B}_i}$.

Proof

${\displaystyle U^T(I-\sum _i{B}_i)U=0}$. Since ${\displaystyle (I-\sum _i{B}_i)}$ is symmetric, and ${\displaystyle U^T(I-\sum _i{B}_i)U{=}^d{U}^{T}0U}$, by the previous claim, ${\displaystyle (I-\sum _i{B}_i)}$ has the same eigenvalues as 0.

Lemma: If ${\displaystyle \sum _i{M}_i=I}$, all ${\displaystyle M_i}$ symmetric, and have eigenvalues 0, 1, then they are simultaneously diagonalizable.

Proof

Fix i, and consider the eigenvectors v of ${\displaystyle M_i}$ such that ${\displaystyle M_{i}v=v}$. Then we have ${\displaystyle v^{T}v=v^{T}Iv=v^{T}v+\sum _{j\ne i}{v}^T{M}_{j}v}$. We note that the ${\displaystyle M_j}$s are positive semi-definite, since their eigenvectors are all non-negative; so, for all ${\displaystyle j}$s we must have ${\displaystyle v^T{M}_{j}v=0}$. Thus we obtain a split of ${\displaystyle {\mathbb{R}}^N}$ into ${\displaystyle V\oplus V^{\perp }}$, such that the 1-eigenspace ${\displaystyle V}$ of ${\displaystyle M_i}$ is contained in the 0-eigenspaces of ${\displaystyle M_j}$ for ${\displaystyle j\ne i}$. Now induct by moving into ${\displaystyle V^{\perp }}$. If ${\displaystyle M_i\ne 0}$ for all ${\displaystyle i}$s, then a basis diagonalizing simultaneously each ${\displaystyle M_i}$ is given by ${\displaystyle (v_{1,1},\dots ,v_{r_1,1},\dots ,v_{1,k},\dots ,v_{r_k,k})}$, where ${\displaystyle (v_{1,i},\dots ,v_{r_i,i})}$ is a basis of the 1-eigenspace of ${\displaystyle M_i}$.

Now we prove the original theorem. We prove that the three cases are equivalent by proving that each case implies the next one in a cycle (${\displaystyle 1\to 2\to 3\to 1}$).

Proof

Case: All ${\displaystyle Q_i}$ are independent

Fix some ${\displaystyle i}$, define ${\displaystyle C_i=I-B_i=\sum _{j\ne i}{B}_j}$, and diagonalize ${\displaystyle B_i}$ by an orthogonal transform ${\displaystyle O}$. Then consider ${\displaystyle O{C}_i{O}^T=I-O{B}_i{O}^T}$. It is diagonalized as well.

Let ${\displaystyle W=OU}$, then it is also standard Gaussian. Then we have

${\displaystyle Q_i=W^T(O{B}_i{O}^T)W;\sum _{j\ne i}{Q}_j=W^T(I-O{B}_i{O}^T)W}$

Inspect their diagonal entries, to see that ${\displaystyle Q_i\perp \sum _{j\ne i}{Q}_j}$ implies that their nonzero diagonal entries are disjoint.

Thus all eigenvalues of ${\displaystyle B_i}$ are 0, 1, so ${\displaystyle Q_i}$ is a ${\displaystyle {\chi }^2}$ dist with ${\displaystyle r_i}$ degrees of freedom.

Case: Each ${\displaystyle Q_i}$ is a ${\displaystyle {\chi }^2(r_i)}$ distribution.

Fix any ${\displaystyle i}$, diagonalize it by orthogonal transform ${\displaystyle O}$, and reindex, so that ${\displaystyle O{B}_i{O}^T=diag({\lambda }_1,...,{\lambda }_{r_i},0,...,0)}$. Then ${\displaystyle Q_i=\sum _j{\lambda }_j{U^{\prime }}_j^2}$ for some ${\displaystyle U_j^{\prime }}$, a spherical rotation of ${\displaystyle U_i}$.

Since ${\displaystyle Q_i\sim {\chi }^2(r_i)}$, we get all ${\displaystyle {\lambda }_j=1}$. So all ${\displaystyle B_i⪰0}$, and have eigenvalues ${\displaystyle 0,1}$.

So diagonalize them simultaneously, add them up, to find ${\displaystyle \sum _i{r}_i=N}$.

Case: ${\displaystyle r_1+\cdots +r_k=N}$.

We first show that the matrices B⁽ⁱ⁾ can be simultaneously diagonalized by an orthogonal matrix and that their non-zero eigenvalues are all equal to +1. Once that's shown, take this orthogonal transform to this simultaneous eigenbasis, in which the random vector ${\displaystyle [U_1,...,U_N{]}^T}$ becomes ${\displaystyle [U_1^{\prime },...,U_N^{\prime }{]}^T}$, but all ${\displaystyle U_i^{\prime }}$ are still independent and standard Gaussian. Then the result follows.

Each of the matrices B⁽ⁱ⁾ has rank r_i and thus r_i non-zero eigenvalues. For each i, the sum ${\displaystyle C^{(i)}\equiv \sum _{j\ne i}{B}^{(j)}}$ has at most rank ${\displaystyle \sum _{j\ne i}{r}_j=N-r_i}$. Since ${\displaystyle B^{(i)}+C^{(i)}=I_{N\times N}}$, it follows that C⁽ⁱ⁾ has exactly rank N − r_i.

Therefore B⁽ⁱ⁾ and C⁽ⁱ⁾ can be simultaneously diagonalized. This can be shown by first diagonalizing B⁽ⁱ⁾, by the spectral theorem. In this basis, it is of the form:

${\displaystyle \left[\begin{array}{ccccccc}{\lambda }_1& 0& 0& \cdots & \cdots & & 0\\ 0& {\lambda }_2& 0& \cdots & \cdots & & 0\\ 0& 0& \ddots & & & & ⋮\\ ⋮& ⋮& & {\lambda }_{r_i}& & \\ ⋮& ⋮& & & 0& \\ 0& ⋮& & & & \ddots \\ 0& 0& \dots & & & & 0\end{array}\right].}$

Thus the lower ${\displaystyle (N-r_i)}$ rows are zero. Since ${\displaystyle C^{(i)}=I-B^{(i)}}$, it follows that these rows in C⁽ⁱ⁾ in this basis contain a right block which is a ${\displaystyle (N-r_i)\times (N-r_i)}$ unit matrix, with zeros in the rest of these rows. But since C⁽ⁱ⁾ has rank N − r_i, it must be zero elsewhere. Thus it is diagonal in this basis as well. It follows that all the non-zero eigenvalues of both B⁽ⁱ⁾ and C⁽ⁱ⁾ are +1. This argument applies for all i, thus all B⁽ⁱ⁾ are positive semidefinite.

Moreover, the above analysis can be repeated in the diagonal basis for ${\displaystyle C^{(1)}=B^{(2)}+\sum _{j>2}{B}^{(j)}}$. In this basis ${\displaystyle C^{(1)}}$ is the identity of an ${\displaystyle (N-r_1)\times (N-r_1)}$ vector space, so it follows that both B⁽²⁾ and ${\displaystyle \sum _{j>2}{B}^{(j)}}$ are simultaneously diagonalizable in this vector space (and hence also together with B⁽¹⁾). By iteration it follows that all B-s are simultaneously diagonalizable.

Thus there exists an orthogonal matrix ${\displaystyle S}$ such that for all ${\displaystyle i}$, ${\displaystyle S^{\mathrm{T}}{B}^{(i)}S\equiv B^{(i)\mathrm{\prime }}}$ is diagonal, where any entry ${\displaystyle B_{x,y}^{(i)\mathrm{\prime }}}$ with indices ${\displaystyle x=y}$, ${\displaystyle \sum _{j=1}^{i-1}{r}_j<x=y\le \sum _{j=1}^i{r}_j}$, is equal to 1, while any entry with other indices is equal to 0.