Polynomial regression

Definition and example

The goal of regression analysis is to model the expected value of a dependent variable y in terms of the value of an independent variable (or vector of independent variables) x. In simple linear regression, the model

${\displaystyle y={\beta }_0+{\beta }_{1}x+\epsilon ,}$

is used, where ε is an unobserved random error with mean zero conditioned on a scalar variable x. In this model, for each unit increase in the value of x, the conditional expectation of y increases by β₁ units.

In many settings, such a linear relationship may not hold. For example, if we are modeling the yield of a chemical synthesis in terms of the temperature at which the synthesis takes place, we may find that the yield improves by a different amount for each unit increase in temperature. Or we may find that the yield decreases with increasing temperature (but only for a certain range of temperatures) and increases with increasing temperature in a different range of temperatures. In this case, we might propose a quadratic model of the form

${\displaystyle y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+\epsilon .}$

In this model, when the temperature is increased from x to x + 1 units, the expected yield changes by ${\displaystyle {\beta }_1+{\beta }_2(2x+1).}$ (This can be seen by obtaining the derivative with respect to x of the regression formula.) For infinitesimal changes in x, the effect on y is given by the total derivative with respect to x: ${\displaystyle {\beta }_1+2{\beta }_{2}x.}$ The fact that the change in yield depends on x is what makes the relationship between x and y nonlinear even though the model is linear in the parameters to be estimated.

In general, we can model the expected value of y as an n^th degree polynomial, yielding the general polynomial regression model

${\displaystyle y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+{\beta }_3{x}^3+\cdots +{\beta }_n{x}^n+\epsilon .}$

Conveniently, these models are all linear from the point of view of estimation, since the regression function is linear in terms of the unknown parameters β₀, β₁, .... Therefore, for least squares analysis, the computational and inferential problems of polynomial regression can be completely addressed using the techniques of multiple regression. This is done by treating x, x², ... as being distinct independent variables in a multiple regression model.

Matrix form and calculation of estimates

The polynomial regression model

${\displaystyle y_i={\beta }_0+{\beta }_1{x}_i+{\beta }_2{x}_i^2+\cdots +{\beta }_m{x}_i^m+{\epsilon }_i(i=1,2,\dots ,n)}$

can be expressed in matrix form in terms of a design matrix ${\displaystyle \mathbf{X}}$, a response vector ${\displaystyle \overrightarrow{y}}$, a parameter vector ${\displaystyle \overrightarrow{\beta }}$, and a vector ${\displaystyle \overrightarrow{\epsilon }}$ of random errors. The i-th row of ${\displaystyle \mathbf{X}}$ and ${\displaystyle \overrightarrow{y}}$ will contain the x and y value for the i-th data sample. Then the model can be written as a system of linear equations:

${\displaystyle \left[\begin{array}{c}{y}_1\\ y_2\\ y_3\\ ⋮\\ y_n\end{array}\right]=\left[\begin{array}{ccccc}1& x_1& x_1^2& \dots & x_1^m\\ 1& x_2& x_2^2& \dots & x_2^m\\ 1& x_3& x_3^2& \dots & x_3^m\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& x_n& x_n^2& \dots & x_n^m\end{array}\right]\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_m\end{array}\right]+\left[\begin{array}{c}{\epsilon }_1\\ {\epsilon }_2\\ {\epsilon }_3\\ ⋮\\ {\epsilon }_n\end{array}\right],}$

which when using pure matrix notation is written as

${\displaystyle \overrightarrow{y}=\mathbf{X}\overrightarrow{\beta }+\overrightarrow{\epsilon }.}$

The vector of estimated polynomial regression coefficients (using ordinary least squares estimation) is

${\displaystyle \hat{\overrightarrow{\beta }}=({\mathbf{X}}^{\mathsf{T}}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathsf{T}}\overrightarrow{y},}$

assuming m < n which is required for the matrix to be invertible; then since ${\displaystyle \mathbf{X}}$ is a Vandermonde matrix, the invertibility condition is guaranteed to hold if all the ${\displaystyle x_i}$ values are distinct. This is the unique least-squares solution. In other words, it realizes the minimum of the distance ${\displaystyle \Vert \overrightarrow{\epsilon }\Vert }$ between the sample ${\displaystyle y_i}$ and the corresponding value of the polynomial, ${\displaystyle \sum _{k=1}^m{\beta }_k{x}_i^k}$, that is, it realizes the minimum

${\displaystyle \underset{\overrightarrow{\beta }}{min}\sum _{i=1}^n{\left(y_i-\sum _{k=1}^m{\beta }_k{x}_i^k\right)}^2.}$

Expanded formulas

The above matrix equations explain the behavior of polynomial regression well. However, to physically implement polynomial regression for a set of xy point pairs, more detail is useful. The below matrix equations for polynomial coefficients are expanded from regression theory without derivation and easily implemented.

${\displaystyle \left[\begin{array}{ccccc}\sum _{i=1}^n{x}_i^0& \sum _{i=1}^n{x}_i^1& \sum _{i=1}^n{x}_i^2& \cdots & \sum _{i=1}^n{x}_i^m\\ \sum _{i=1}^n{x}_i^1& \sum _{i=1}^n{x}_i^2& \sum _{i=1}^n{x}_i^3& \cdots & \sum _{i=1}^n{x}_i^{m+1}\\ \sum _{i=1}^n{x}_i^2& \sum _{i=1}^n{x}_i^3& \sum _{i=1}^n{x}_i^4& \cdots & \sum _{i=1}^n{x}_i^{m+2}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \sum _{i=1}^n{x}_i^m& \sum _{i=1}^n{x}_i^{m+1}& \sum _{i=1}^n{x}_i^{m+2}& \dots & \sum _{i=1}^n{x}_i^{2m}\end{array}\right]\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ {\beta }_2\\ \cdots \\ {\beta }_m\end{array}\right]=\left[\begin{array}{c}\sum _{i=1}^n{y}_i{x}_i^0\\ \sum _{i=1}^n{y}_i{x}_i^1\\ \sum _{i=1}^n{y}_i{x}_i^2\\ \cdots \\ \sum _{i=1}^n{y}_i{x}_i^m\end{array}\right]}$

After solving the above system of linear equations for ${\displaystyle {\beta }_0\text{ through }{\beta }_m}$, the regression polynomial may be constructed as follows:

${\displaystyle \begin{array}{cc}& \hat{y}={\beta }_0{x}^0+{\beta }_1{x}^1+{\beta }_2{x}^2+\cdots +{\beta }_m{x}^m\\ & \\ & \text{Where:}\\ & n=\text{number of }{x}_i{y}_i\text{ variable pairs in the data}\\ & m=\text{order of the polynomial to be used for regression}\\ & {\beta }_{(0-m)}=\text{polynomial coefficient for each corresponding }{x}^{(0-m)}\\ & \hat{y}=\text{estimated y variable based on the polynomial regression calculations.}\end{array}}$

Interpretation

Although polynomial regression is technically a special case of multiple linear regression, the interpretation of a fitted polynomial regression model requires a somewhat different perspective. It is often difficult to interpret the individual coefficients in a polynomial regression fit, since the underlying monomials can be highly correlated. For example, x and x² have correlation around 0.97 when x is uniformly distributed on the interval (0, 1). Although the correlation can be reduced by using orthogonal polynomials, it is generally more informative to consider the fitted regression function as a whole. Point-wise or simultaneous confidence bands can then be used to provide a sense of the uncertainty in the estimate of the regression function.

Alternative approaches

Polynomial regression is one example of regression analysis using basis functions to model a functional relationship between two quantities. More specifically, it replaces ${\displaystyle x\in {\mathbb{R}}^{d_x}}$ in linear regression with polynomial basis ${\displaystyle \phi (x)\in {\mathbb{R}}^{d_{\phi }}}$, e.g. ${\displaystyle [1,x]\stackrel{\phi }{\to }[1,x,x^2,\dots ,x^d]}$. A drawback of polynomial bases is that the basis functions are "non-local", meaning that the fitted value of y at a given value x = x₀ depends strongly on data values with x far from x₀. In modern statistics, polynomial basis-functions are used along with new basis functions, such as splines, radial basis functions, and wavelets. These families of basis functions offer a more parsimonious fit for many types of data.

The goal of polynomial regression is to model a non-linear relationship between the independent and dependent variables (technically, between the independent variable and the conditional mean of the dependent variable). This is similar to the goal of nonparametric regression, which aims to capture non-linear regression relationships. Therefore, non-parametric regression approaches such as smoothing can be useful alternatives to polynomial regression. Some of these methods make use of a localized form of classical polynomial regression. An advantage of traditional polynomial regression is that the inferential framework of multiple regression can be used (this also holds when using other families of basis functions such as splines).

A final alternative is to use kernelized models such as support vector regression with a polynomial kernel.

If residuals have unequal variance, a weighted least squares estimator may be used to account for that.