Parametric statistics

Parameter estimation

In the typical setting of parameter estimation, the chosen parametric family of distributions has either a density function or a density mass function ${\displaystyle L_{\theta }}$, called the likelihood depending on the parameters ${\displaystyle \theta =({\theta }_1,\dots ,{\theta }_p)}$, and a finite sample of independent and identically distributed data ${\displaystyle X_1,\dots ,X_n}$ is given. The goal is to estimate ${\displaystyle \theta }$ (or a function thereof) based on the observed data.

Frequentist approaches

In a frequentist approach, the data is assumed to be distributed according to ${\displaystyle L_{{\theta }^{\ast }}}$ for some "true" parameter ${\displaystyle {\theta }^{\ast }}$. The goal is to recover ${\displaystyle {\theta }^{\ast }}$ from the observed data. The most common methods for parameter estimation are the following.

• Maximum Likelihood estimation (MLE): The model parameters are chosen such that the probability (or probability density) of the given observation is maximal. Under certain assumptions on the likelihood, the MLE is (strongly) consistent, asymptotically efficient and asymptotically normal.
• Method of moments (MoM): If the model parameters can be expressed as functions ${\displaystyle g_1,\dots ,g_p}$ of the moments of the distribution, then the moment estimates of the parameters are ${\displaystyle g_1\left(\sum _{i=1}^n{X}_i,\dots ,\sum _{i=1}^n{X}_i^p\right),\dots ,g_p\left(\sum _{i=1}^n{X}_i,\dots ,\sum _{i=1}^n{X}_i^p\right)}$. If the functions ${\displaystyle g_1,\dots ,g_p}$ are continuous, then the moment estimator is strongly consistent (due to the strong law of large numbers). If the functions ${\displaystyle g_1,\dots ,g_p}$ are even differentiable, then the moment estimator is also asymptotically normal (due to the central limit theorem and the delta method).
• Least square estimation (LSE): This method applies to a regression setting, where the data arises in pairs ${\displaystyle (X_1,Y_1),\dots ,(X_n,Y_n)}$ and a regression function ${\displaystyle f}$ is to be determined. The model parameters are chosen such that the sum of squared differences ${\displaystyle \sum _{i=1}^n(Y_i-f_{\theta }(X_i){)}^2}$ between the observed data and the model prediction is minimal. In fact, LSE is a special case of MLE, where the conditional distribution of ${\displaystyle Y}$ given ${\displaystyle X}$ is normally distributed.

Bayesian approaches

In a Bayesian approach, the data is not assumed to be generated by a distribution ${\displaystyle L_{{\theta }^{\ast }}}$ for some true ${\displaystyle {\theta }^{\ast }}$. Instead, the set of all possible (or plausible) model parameters is initially weighted with an à-priori distribution ${\displaystyle \pi }$ reflecting the statistician's prior belief. Given the observed data, the parameter distribution is updated via Bayes' rule, yielding the à-posteriori distribution ${\displaystyle p_{\theta }}$ that is proportional to the likelihood ${\displaystyle L_{\theta }}$ times the prior ${\displaystyle \pi }$. Bayesian estimators therefore rather give a best estimate given the statisticians beliefs.

• Posterior mean estimation: The estimator takes the average of the posterior distribution.
• Posterior median estimation: The estimator takes the median of the posterior distribution.
• Maximum à-posteriori estimation (MAP): The estimator takes the parameter that maximizes the posterior distribution. If the prior is the uniform distribution, then MAP estimation is equivalent to maximum likelihood estimation.

Optimality of estimators

Uniformly minimum-variance unbiased estimators

Uniformly minimum-variance unbiased estimators (UMVUE), sometimes called best unbiased estimators as well, are estimators that have minimum variance among all unbiased estimators. Due to the bias-variance decomposition, they are optimal in the sense that they minimise the mean squared error among all unbiased estimators.

Let's assume a function of the model parameters ${\displaystyle q(\theta )}$ needs to be estimated (e.g., mean, variance, median, or ${\displaystyle \theta }$ itself). Most generally, if an UMVU estimator of ${\displaystyle q(\theta )}$ with finite variance exists, then it must be unique due to the Rao–Blackwell theorem. If there exists a complete sufficient statistic ${\displaystyle T}$ in the chosen statistical model, which is often easy to determine using the Neyman–Fisher theorem, then it holds the Lehmann–Scheffé theorem: An estimators of the form ${\displaystyle f(T)}$ that is unbiased, is automatically UMVU. If such an estimator has finite variance for all ${\displaystyle \theta }$, then it is also the unique UMVUE.

Best linear unbiased estimator

A best linear unbiased estimator (BLUE) has minimum variance among all unbiased estimators ${\displaystyle \hat{\theta }}$ that are linear in the data, that is,$${\displaystyle \hat{\theta }=\sum _{i=1}^n{\alpha }_i{X}_i,}$$for some coefficients ${\displaystyle {\alpha }_1,\dots ,{\alpha }_n\in \mathbb{R}}$. In a sense, BLUEs are UMVU estimators with the additional restriction of being linear.

In the regression setting, the Gauss–Markov theorem states that the least squares estimator is the unique BLUE. This theorem also shows (considering the special case of constant regression) that the empirical mean is a BLUE for estimating the mean.

Information inequality and exponential families

In regular statistical models, it can be shown that the variance of unbiased estimators cannot be arbitrarily small: Any unbiased estimator ${\displaystyle T}$ of the quantity ${\displaystyle q(\theta )}$ is bounded from below by the universal bound$${\displaystyle {\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }[T(X)]\ge \mathrm{\nabla }q(\theta )I(\theta {)}^{-1}\mathrm{\nabla }q(\theta {)}^T,}$$where ${\displaystyle I(\theta )}$ is the Fisher-information matrix of the statistical model. The right hand-side is the so-called Cramér–Rao bound.

Estimators for which the variance equals the Cramér–Rao bound are called efficient and are almost UMVU, since they minimize the variance over all regular unbiased estimators (there might be non-regular estimators with smaller variance). On the contrary, not every UMVUE is efficient. In fact, an estimator ${\displaystyle T}$ is efficient if and only if (i) the statistical model is an exponential family and (ii) ${\displaystyle T}$ is the natural sufficient statistic.

Parametric models

The choice of the model, that is, the probability distribution from which the data is assumed to be drawn in density estimation problems or an assumed functional depence between pairs of data ${\displaystyle X}$ and ${\displaystyle Y}$ in regression/classification problems, lies at the core of parametric procedures. Here is a list of common models used in practice.

Density estimation

• Exponential families (e.g. normal distribution, exponential distribution, log-normal distribution, Gamma distribution, Chi-squared distribution, Erlang distribution, Beta distribution, Gumbel distribution, Pareto distribution, (Negative-)Binomial distribution, Poisson distribution, geometric distribution)
• Laplace distribution
• Uniform distribution
• Weibull distribution

Regression

• linear model (special cases thereof are ANOVA and ANCOVA)
• generalized linear model (GLM)
• neural networks

Classification

• logistic regression
• linear discriminant analysis (LDA)
• quadratic discriminant analysis (QDA)
• neural networks

Example

The normal family of distributions all have the same general shape and are parameterized by mean and standard deviation. That means that if the mean and standard deviation are known and if the distribution is normal, the probability of any future observation lying in a given range is known.

Suppose that we have a sample of 99 test scores with a mean of 100 and a standard deviation of 1. If we assume all 99 test scores are random observations from a normal distribution, then we predict there is a 1% chance that the 100th test score will be higher than 102.33 (that is, the mean plus 2.33 standard deviations), assuming that the 100th test score comes from the same distribution as the others. Parametric statistical methods are used to compute the 2.33 value above, given 99 independent observations from the same normal distribution.

A non-parametric estimate of the same thing is the maximum of the first 99 scores. We don't need to assume anything about the distribution of test scores to reason that before we gave the test it was equally likely that the highest score would be any of the first 100. Thus there is a 1% chance that the 100th score is higher than any of the 99 that preceded it.

History

Parametric statistics was mentioned by R. A. Fisher in his work Statistical Methods for Research Workers in 1925, which created the foundation for modern statistics.