Residual sum of squares

Matrix expression for the OLS residual sum of squares

The general regression model with n observations and k explanators, the first of which is a constant unit vector whose coefficient is the regression intercept, is

${\displaystyle y=X\beta +e}$

where y is an n × 1 vector of dependent variable observations, each column of the n × k matrix X is a vector of observations on one of the k explanators, ${\displaystyle \beta }$ is a k × 1 vector of true coefficients, and e is an n× 1 vector of the true underlying errors. The ordinary least squares estimator for ${\displaystyle \beta }$ is

${\displaystyle X\hat{\beta }=y⟺}$

${\displaystyle X^{\mathrm{T}}X\hat{\beta }=X^{\mathrm{T}}y⟺}$

${\displaystyle \hat{\beta }=(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}y.}$

The residual vector ${\displaystyle \hat{e}=y-X\hat{\beta }=y-X(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}y}$; so the residual sum of squares is:

${\displaystyle \mathrm{RSS}={\hat{e}}^{\mathrm{T}}\hat{e}=\Vert \hat{e}{\Vert }^2}$,

(equivalent to the square of the norm of residuals). In full:

${\displaystyle \mathrm{RSS}=y^{\mathrm{T}}y-y^{\mathrm{T}}X(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}y=y^{\mathrm{T}}[I-X(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}]y=y^{\mathrm{T}}[I-H]y}$,

where H is the hat matrix, or the projection matrix in linear regression.

Relation with Pearson's product-moment correlation

The least-squares regression line is given by

${\displaystyle y=ax+b}$,

where ${\displaystyle b=\overline{y}-a\overline{x}}$ and ${\displaystyle a=\frac{S_{xy}}{S_{xx}}}$, where ${\displaystyle S_{xy}=\sum _{i=1}^n(\overline{x}-x_i)(\overline{y}-y_i)}$ and ${\displaystyle S_{xx}=\sum _{i=1}^n(\overline{x}-x_i{)}^2.}$

Therefore,

${\displaystyle \begin{array}{cc}\mathrm{RSS}& =\sum _{i=1}^n(y_i-f(x_i){)}^2=\sum _{i=1}^n(y_i-(a{x}_i+b){)}^2=\sum _{i=1}^n(y_i-a{x}_i-\overline{y}+a\overline{x}{)}^2\\ & =\sum _{i=1}^n(a(\overline{x}-x_i)-(\overline{y}-y_i){)}^2=a^2{S}_{xx}-2a{S}_{xy}+S_{yy}=S_{yy}-a{S}_{xy}=S_{yy}\left(1-\frac{S_{xy}^2}{S_{xx}{S}_{yy}}\right)\end{array}}$

where ${\displaystyle S_{yy}=\sum _{i=1}^n(\overline{y}-y_i{)}^2.}$

The Pearson product-moment correlation is given by ${\displaystyle r=\frac{S_{xy}}{\sqrt{S_{xx}{S}_{yy}}};}$ therefore, ${\displaystyle \mathrm{RSS}=S_{yy}(1-r^2).}$