Explained sum of squares

Partitioning in simple linear regression

The following equality, stating that the total sum of squares (TSS) equals the residual sum of squares (SSE, the sum of squared errors of prediction) plus the explained sum of squares (SSR, the sum of squares due to regression or explained sum of squares), is generally true in simple linear regression:

${\displaystyle \sum _{i=1}^n{\left(y_i-\overline{y}\right)}^2=\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2+\sum _{i=1}^n{\left({\hat{y}}_i-\overline{y}\right)}^2.}$

Simple derivation

${\displaystyle \begin{array}{c}(y_i-\overline{y})=(y_i-{\hat{y}}_i)+({\hat{y}}_i-\overline{y}).\end{array}}$

Square both sides and sum over all i:

${\displaystyle \sum _{i=1}^n(y_i-\overline{y}{)}^2=\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2+\sum _{i=1}^n({\hat{y}}_i-\overline{y}{)}^2+\sum _{i=1}^{n}2({\hat{y}}_i-\overline{y})(y_i-{\hat{y}}_i).}$

Here is how the last term above is zero from simple linear regression

${\displaystyle \widehat{y_i}=\hat{a}+\hat{b}{x}_i}$

${\displaystyle \overline{y}=\hat{a}+\hat{b}\overline{x}}$

${\displaystyle \hat{b}=\frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}}$

So,

${\displaystyle \widehat{y_i}-\overline{y}=\hat{b}(x_i-\overline{x})}$

${\displaystyle y_i-{\hat{y}}_i=(y_i-\overline{y})-({\hat{y}}_i-\overline{y})=(y_i-\overline{y})-\hat{b}(x_i-\overline{x})}$

Therefore,

${\displaystyle \begin{array}{cc}& \sum _{i=1}^{n}2({\hat{y}}_i-\overline{y})(y_i-{\hat{y}}_i)=2\hat{b}\sum _{i=1}^n(x_i-\overline{x})(y_i-{\hat{y}}_i)\\ =& 2\hat{b}\sum _{i=1}^n(x_i-\overline{x})((y_i-\overline{y})-\hat{b}(x_i-\overline{x}))\\ =& 2\hat{b}\left(\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})-\sum _{i=1}^n(x_i-\overline{x}{)}^2\frac{\sum _{j=1}^n(x_j-\overline{x})(y_j-\overline{y})}{\sum _{j=1}^n(x_j-\overline{x}{)}^2}\right)\\ =& 2\hat{b}(0)=0\end{array}}$

Partitioning in the general ordinary least squares model

The general regression model with n observations and k explanators, the first of which is a constant unit vector whose coefficient is the regression intercept, is

${\displaystyle y=X\beta +e}$

where y is an n × 1 vector of dependent variable observations, each column of the n × k matrix X is a vector of observations on one of the k explanators, ${\displaystyle \beta }$ is a k × 1 vector of true coefficients, and e is an n × 1 vector of the true underlying errors. The ordinary least squares estimator for ${\displaystyle \beta }$ is

${\displaystyle \hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}y.}$

The residual vector ${\displaystyle \hat{e}}$ is ${\displaystyle y-X\hat{\beta }=y-X(X^{T}X{)}^{-1}{X}^{T}y}$, so the residual sum of squares ${\displaystyle {\hat{e}}^T\hat{e}}$ is, after simplification,

${\displaystyle RSS=y^{T}y-y^{T}X(X^{T}X{)}^{-1}{X}^{T}y.}$

Denote as ${\displaystyle \overline{y}}$ the constant vector all of whose elements are the sample mean ${\displaystyle y_m}$ of the dependent variable values in the vector y. Then the total sum of squares is

${\displaystyle TSS=(y-\overline{y}{)}^T(y-\overline{y})=y^{T}y-2y^T\overline{y}+{\overline{y}}^T\overline{y}.}$

The explained sum of squares, defined as the sum of squared deviations of the predicted values from the observed mean of y, is

${\displaystyle ESS=(\hat{y}-\overline{y}{)}^T(\hat{y}-\overline{y})={\hat{y}}^T\hat{y}-2{\hat{y}}^T\overline{y}+{\overline{y}}^T\overline{y}.}$

Using ${\displaystyle \hat{y}=X\hat{\beta }}$ in this, and simplifying to obtain ${\displaystyle {\hat{y}}^T\hat{y}=y^{T}X(X^{T}X{)}^{-1}{X}^{T}y}$, gives the result that TSS = ESS + RSS if and only if ${\displaystyle y^T\overline{y}={\hat{y}}^T\overline{y}}$. The left side of this is ${\displaystyle y_m}$ times the sum of the elements of y, and the right side is ${\displaystyle y_m}$ times the sum of the elements of ${\displaystyle \hat{y}}$, so the condition is that the sum of the elements of y equals the sum of the elements of ${\displaystyle \hat{y}}$, or equivalently that the sum of the prediction errors (residuals) ${\displaystyle y_i-{\hat{y}}_i}$ is zero. This can be seen to be true by noting the well-known OLS property that the k × 1 vector ${\displaystyle X^T\hat{e}=X^T[I-X(X^{T}X{)}^{-1}{X}^T]y=0}$: since the first column of X is a vector of ones, the first element of this vector ${\displaystyle X^T\hat{e}}$ is the sum of the residuals and is equal to zero. This proves that the condition holds for the result that TSS = ESS + RSS.

In linear algebra terms, we have ${\displaystyle RSS=\Vert y-\hat{y}{\Vert }^2}$, ${\displaystyle TSS=\Vert y-\overline{y}{\Vert }^2}$, ${\displaystyle ESS=\Vert \hat{y}-\overline{y}{\Vert }^2}$. The proof can be simplified by noting that ${\displaystyle {\hat{y}}^T\hat{y}={\hat{y}}^{T}y}$. The proof is as follows:

${\displaystyle {\hat{y}}^T\hat{y}=y^{T}X(X^{T}X{)}^{-1}{X}^{T}X(X^{T}X{)}^{-1}{X}^{T}y=y^{T}X(X^{T}X{)}^{-1}{X}^{T}y={\hat{y}}^{T}y,}$

Thus,

${\displaystyle \begin{array}{cc}TSS& =\Vert y-\overline{y}{\Vert }^2=\Vert y-\hat{y}+\hat{y}-\overline{y}{\Vert }^2\\ & =\Vert y-\hat{y}{\Vert }^2+\Vert \hat{y}-\overline{y}{\Vert }^2+2⟨y-\hat{y},\hat{y}-\overline{y}⟩\\ & =RSS+ESS+2y^T\hat{y}-2{\hat{y}}^T\hat{y}-2y^T\overline{y}+2{\hat{y}}^T\overline{y}\\ & =RSS+ESS-2y^T\overline{y}+2{\hat{y}}^T\overline{y}\end{array}}$

which again gives the result that TSS = ESS + RSS, since ${\displaystyle (y-\hat{y}{)}^T\overline{y}=0}$.