Quadratic form (statistics)

Variance in the Gaussian case

In general, the variance of a quadratic form depends greatly on the distribution of ${\displaystyle \epsilon }$. However, if ${\displaystyle \epsilon }$ does follow a multivariate normal distribution, the variance of the quadratic form becomes particularly tractable. Assume for the moment that ${\displaystyle \mathrm{\Lambda }}$ is a symmetric matrix. Then,

${\displaystyle \mathrm{var}\left[{\epsilon }^T\mathrm{\Lambda }\epsilon \right]=2\mathrm{tr}\left[\mathrm{\Lambda }\mathrm{\Sigma }\mathrm{\Lambda }\mathrm{\Sigma }\right]+4{\mu }^T\mathrm{\Lambda }\mathrm{\Sigma }\mathrm{\Lambda }\mu }$.

In fact, this can be generalized to find the covariance between two quadratic forms on the same ${\displaystyle \epsilon }$ (once again, ${\displaystyle {\mathrm{\Lambda }}_1}$ and ${\displaystyle {\mathrm{\Lambda }}_2}$ must both be symmetric):

${\displaystyle \mathrm{cov}\left[{\epsilon }^T{\mathrm{\Lambda }}_1\epsilon ,{\epsilon }^T{\mathrm{\Lambda }}_2\epsilon \right]=2\mathrm{tr}\left[{\mathrm{\Lambda }}_1\mathrm{\Sigma }{\mathrm{\Lambda }}_2\mathrm{\Sigma }\right]+4{\mu }^T{\mathrm{\Lambda }}_1\mathrm{\Sigma }{\mathrm{\Lambda }}_2\mu }$.

In addition, a quadratic form such as this follows a generalized chi-squared distribution.

Computing the variance in the non-symmetric case

The case for general ${\displaystyle \mathrm{\Lambda }}$ can be derived by noting that

${\displaystyle {\epsilon }^T{\mathrm{\Lambda }}^T\epsilon ={\epsilon }^T\mathrm{\Lambda }\epsilon }$

so

${\displaystyle {\epsilon }^T\tilde{\mathrm{\Lambda }}\epsilon ={\epsilon }^T\left(\mathrm{\Lambda }+{\mathrm{\Lambda }}^T\right)\epsilon /2}$

is a quadratic form in the symmetric matrix ${\displaystyle \tilde{\mathrm{\Lambda }}=\left(\mathrm{\Lambda }+{\mathrm{\Lambda }}^T\right)/2}$, so the mean and variance expressions are the same, provided ${\displaystyle \mathrm{\Lambda }}$ is replaced by ${\displaystyle \tilde{\mathrm{\Lambda }}}$ therein.

Examples of quadratic forms

In the setting where one has a set of observations ${\displaystyle y}$ and an operator matrix ${\displaystyle H}$, then the residual sum of squares can be written as a quadratic form in ${\displaystyle y}$:

${\displaystyle \text{RSS}=y^T(I-H{)}^T(I-H)y.}$

For procedures where the matrix ${\displaystyle H}$ is symmetric and idempotent, and the errors are Gaussian with covariance matrix ${\displaystyle {\sigma }^{2}I}$, ${\displaystyle \text{RSS}/{\sigma }^2}$ has a chi-squared distribution with ${\displaystyle k}$ degrees of freedom and noncentrality parameter ${\displaystyle \lambda }$, where

${\displaystyle k=\mathrm{tr}\left[(I-H{)}^T(I-H)\right]}$

${\displaystyle \lambda ={\mu }^T(I-H{)}^T(I-H)\mu /2}$

may be found by matching the first two central moments of a noncentral chi-squared random variable to the expressions given in the first two sections. If ${\displaystyle Hy}$ estimates ${\displaystyle \mu }$ with no bias, then the noncentrality ${\displaystyle \lambda }$ is zero and ${\displaystyle \text{RSS}/{\sigma }^2}$ follows a central chi-squared distribution.