Goodness of fit

Regression analysis

In regression analysis, more specifically regression validation, the following topics relate to goodness of fit:

• Coefficient of determination (the R-squared measure of goodness of fit);
• Lack-of-fit sum of squares;
• Mallows's Cp criterion
• Prediction error
• Reduced chi-square

Categorical data

The following are examples that arise in the context of categorical data.

Pearson's chi-square test

Pearson's chi-square test uses a measure of goodness of fit which is the sum of differences between observed and expected outcome frequencies (that is, counts of observations), each squared and divided by the expectation:

$${\displaystyle {\chi }^2=\sum _{i=1}^n{\frac{(O_i-E_i)}{E_i}}^2}$$ where:

• O_i = an observed count for bin i
• E_i = an expected count for bin i, asserted by the null hypothesis.

The expected frequency is calculated by: $${\displaystyle E_i=(F(Y_u)-F(Y_l))N}$$ where:

• F = the cumulative distribution function for the probability distribution being tested.
• Y_u = the upper limit for bin i,
• Y_l = the lower limit for bin i, and
• N = the sample size

The resulting value can be compared with a chi-square distribution to determine the goodness of fit. The chi-square distribution has (k − c) degrees of freedom, where k is the number of non-empty bins and c is the number of estimated parameters (including location and scale parameters and shape parameters) for the distribution plus one. For example, for a 3-parameter Weibull distribution, c = 4.

Binomial case

A binomial experiment is a sequence of independent trials in which the trials can result in one of two outcomes, success or failure. There are n trials each with probability of success, denoted by p. Provided that np_i ≫ 1 for every i (where i = 1, 2, ..., k), then

$${\displaystyle {\chi }^2=\sum _{i=1}^k\frac{(N_i-n{p}_i{)}^2}{n{p}_i}=\sum _{\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s}}^{}\frac{(\mathrm{O}-\mathrm{E}{)}^2}{\mathrm{E}}.}$$

This has approximately a chi-square distribution with k − 1 degrees of freedom. The fact that there are k − 1 degrees of freedom is a consequence of the restriction $\sum N_i=n$. We know there are k observed bin counts, however, once any k − 1 are known, the remaining one is uniquely determined. Basically, one can say, there are only k − 1 freely determined bin counts, thus k − 1 degrees of freedom.

G-test

G-tests are likelihood-ratio tests of statistical significance that are increasingly being used in situations where Pearson's chi-square tests were previously recommended.

The general formula for G is

${\displaystyle G=2\sum _i{O}_i\cdot \ln \left(\frac{O_i}{E_i}\right),}$

where $O_i$ and $E_i$ are the same as for the chi-square test, $\ln$ denotes the natural logarithm, and the sum is taken over all non-empty bins. Furthermore, the total observed count should be equal to the total expected count:$${\displaystyle \sum _i{O}_i=\sum _i{E}_i=N}$$where $N$ is the total number of observations.

G-tests have been recommended at least since the 1981 edition of the popular statistics textbook by Robert R. Sokal and F. James Rohlf.