Method of moments (statistics)

It starts by expressing the population moments (i.e., the expected values of powers of the random variable under consideration) as functions of the parameters of interest. Those expressions are then set equal to the sample moments. The number of such equations is the same as the number of parameters to be estimated. Those equations are then solved for the parameters of interest. The solutions are estimates of those parameters. The method of moments was first introduced by Karl Pearson in 1895.

Method

Suppose that the parameter $θ$ = ( $θ_{1}, θ_{2}, \dots, θ_{k}$ ) characterizes the distribution $f_{W} (w; θ)$ of the random variable $W$ . Suppose the first $k$ moments of the true distribution (the "population moments") can be expressed as functions of the $θ$ s:

$\begin{matrix} m_{1} & \equiv E [W] = g_{1} (θ_{1}, θ_{2}, \dots, θ_{k}), \\ m_{2} & \equiv E [W^{2}] = g_{2} (θ_{1}, θ_{2}, \dots, θ_{k}), \\ ⋮ \\ m_{k} & \equiv E [W^{k}] = g_{k} (θ_{1}, θ_{2}, \dots, θ_{k}) . \end{matrix}$

Suppose a sample of size $n$ is drawn, resulting in the values $w_{1}, \dots, w_{n}$ . For $j = 1, \dots, k$ , let ${\hat{m}}_{j} = \frac{1}{n} \sum_{i = 1}^{n} w_{i}^{j}$ be the j-th sample moment, an estimate of $m_{j}$ . The method of moments estimator for $θ_{1}, θ_{2}, \dots, θ_{k}$ denoted by ${\hat{θ}}_{1}, {\hat{θ}}_{2}, \dots, {\hat{θ}}_{k}$ is defined to be a solution (if one exists) to the equations:

$\begin{matrix} {\hat{m}}_{1} & = g_{1} ({\hat{θ}}_{1}, {\hat{θ}}_{2}, \dots, {\hat{θ}}_{k}), \\ {\hat{m}}_{2} & = g_{2} ({\hat{θ}}_{1}, {\hat{θ}}_{2}, \dots, {\hat{θ}}_{k}), \\ ⋮ \\ {\hat{m}}_{k} & = g_{k} ({\hat{θ}}_{1}, {\hat{θ}}_{2}, \dots, {\hat{θ}}_{k}) . \end{matrix}$

The method described here for single random variables generalizes in an obvious manner to multiple random variables leading to multiple choices for moments to be used. Different choices generally lead to different solutions.

4 more chapters are free with a Lably account.

Properties

If it exists, a moment estimator ${\hat{θ}}^{(n)} = ({\hat{θ}}_{1}, \dots, {\hat{θ}}_{k})$ based on an independent sample of $n$ observations drawn from the distribution $f_{W} (w; θ)$ has the following statistical properties.

Consistency

Provided that the function $g = (g_{1}, \dots, g_{k})$ is invertible and continuous, and the first $k$ moments exist ( $E [| W |^{k}] < \infty$ ), the moment estimator is strongly consistent. This is a direct consequence of the strong law of large numbers.

In the typical case, where the second moment $E [({\hat{θ}}^{(n)})^{2}] = V a r ({\hat{θ}}^{(n)}) + E [{\hat{θ}}^{(n)}]^{2}$ does not explode as $n$ goes to infinity, the above implies (by the Vitali convergence theorem) that the moment estimator is also unbiased in the limit, that is,

$lim_{n \to \infty} E [{\hat{θ}}^{(n)}] = θ .$

Asymptotic normality

If the function $g = (g_{1}, \dots, g_{k})$ is even differentiable and $E [| W^{2} |^{k}] < \infty$ , then the moment estimator is also asymptotically normal in the sense $\sqrt{n} ({\hat{θ}}^{(n)} - E_{θ} [{\hat{θ}}^{(n)}]) \overset{d}{⟶} N (0, (\nabla g^{- 1} (θ))^{T} {C o v}_{θ} (W) \nabla g^{- 1} (θ)),$ where ${C o v}_{θ} (W)$ is the covariance matrix of $W$ . This is a consequence of the central limit theorem and the delta method.

Advantages and disadvantages

On the one hand, the method of moments is fairly simple and broadly applicable, since it does not require any assumptions on the data distribution besides existence of the first few moments. This is particularly useful in settings, where the general shape of the distribution may not be known (e.g. utility functions). Moreover, the resulting estimators are almost always consistent and asymptotically unbiased.

On the other hand, the moment equations do not always have a solution, in which case the method does not provide any estimator. In other cases, there might exist multiple solutions. In view of parameter estimation in a parametric model, moment estimators are, in general, not asymptotically efficient, in contrast to maximum likelihood estimators. Moreover, the method does not necessarily lead to sufficient estimators, i.e., they sometimes fail to take into account all relevant information in the sample.

Alternative method of moments

The equations to be solved in the method of moments (MoM) are in general nonlinear and there are no generally applicable guarantees that tractable solutions exist. But there is an alternative approach to using sample moments to estimate data model parameters in terms of known dependence of model moments on these parameters, and this alternative requires the solution of only linear equations or, more generally, tensor equations. This alternative is referred to as the Bayesian-Like MoM (BL-MoM), and it differs from the classical MoM in that it uses optimally weighted sample moments. Considering that the MoM is typically motivated by a lack of sufficient knowledge about the data model to determine likelihood functions and associated a posteriori probabilities of unknown or random parameters, it is odd that there exists a type of MoM that is Bayesian-Like. But the particular meaning of Bayesian-Like leads to a problem formulation in which required knowledge of a posteriori probabilities is replaced with required knowledge of only the dependence of model moments on unknown model parameters, which is exactly the knowledge required by the traditional MoM. The BL-MoM also uses knowledge of a priori probabilities of the parameters to be estimated, when available, but otherwise uses uniform priors.

The BL-MoM has been reported on in only the applied statistics literature in connection with parameter estimation and hypothesis testing using observations of stochastic processes for problems in Information and Communications Theory and, in particular, communications receiver design in the absence of knowledge of likelihood functions or associated a posteriori probabilities and references therein. In addition, the restatement of this receiver design approach for stochastic process models as an alternative to the classical MoM for any type of multivariate data is available in tutorial form at the university website. The applications in and references demonstrate some important characteristics of this alternative to the classical MoM, and a detailed list of relative advantages and disadvantages is given in, but the literature is missing direct comparisons in specific applications of the classical MoM and the BL-MoM.

Examples

Poisson distribution

Let $W \sim P o i s s (λ)$ be Poisson distributed with parameter $λ > 0$ . The first moment (that is, the mean) is

$m_{1} = E [W] = λ .$

Since this is already an explicit equation for $λ$ , the moment estimator is simply the sample mean.

Normal distribution

Let $W \sim N (μ, σ^{2})$ be normally distributed with unknown parameters $μ \in R$ and $σ^{2} > 0$ . Since two parameters must be determined, two equations are needed. The first two moments are given by

$\begin{matrix} m_{1} = E [W] = μ, \\ m_{2} = E [W^{2}] = V a r (W) + E [W]^{2} = σ^{2} + μ^{2} . \end{matrix}$

Solving the equations for the parameters yields

$\begin{matrix} μ & = m_{1}, \\ σ^{2} & = m_{2} - m_{1}^{2} . \end{matrix}$

Given a sample $X_{1}, \dots, X_{n} \sim N (μ, σ^{2})$ , the corresponding estimators are

$\begin{matrix} \hat{μ} & = \frac{1}{n} \sum_{i = 1}^{n} X_{i} = \bar{X}, \\ {\hat{σ}}^{2} & = \frac{1}{n} \sum_{i = 1}^{n} X_{i}^{2} - \frac{1}{n^{2}} {(\sum_{i = 1}^{n} X_{i})}^{2} = \frac{1}{n} \sum_{i = 1}^{n} (X_{i} - \bar{X})^{2}, \end{matrix}$

which are the sample mean and the (biased) sample variance. In this case, they coincide with the maximum likelihood estimators.

Uniform distribution

Consider the uniform distribution on the interval $[a, b]$ , $U (a, b)$ , with unknown parameters $- \infty < a < b < \infty$ . If $W \sim U (a, b)$ then we have

$\begin{matrix} m_{1} & = E [W] & = & \frac{1}{2} (a + b) \\ m_{2} & = E [W^{2}] & = & \frac{1}{3} (a^{2} + a b + b^{2}) \end{matrix}$

Solving these equations gives

$\begin{matrix} \hat{a} & = m_{1} - \sqrt{3 (m_{2} - m_{1}^{2})} \\ \hat{b} & = m_{1} + \sqrt{3 (m_{2} - m_{1}^{2})} \end{matrix}$

Given a set of samples ${w_{i}}$ we can use the sample moments ${\hat{m}}_{1}$ and ${\hat{m}}_{2}$ in these formulae in order to estimate $a$ and $b$ .

Note, however, that this method can produce inconsistent results in some cases. For example, the set of samples ${0, 0, 0, 0, 1}$ results in the estimate $\hat{a} = \frac{1}{5} (1 - 2 \sqrt{3}) = - 0.4928$ , $\hat{b} = \frac{1}{5} (1 + 2 \sqrt{3}) = 0.8928$ . Since $\hat{b} < 1$ it is impossible for the set ${0, 0, 0, 0, 1}$ to have been drawn from $U (\hat{a}, \hat{b})$ in this case.

Binomial distribution

Let $W \sim B i n (n, p)$ be binomially distributed with unknown parameters $n \in N$ and $p \in (0, 1)$ . The first two moments read as

$\begin{matrix} m_{1} = E [W] = n p, \\ m_{2} = E [W^{2}] = n p (1 - p) + n^{2} p^{2} . \end{matrix}$

Solving these equations for the parameters yields $\begin{matrix} p = \frac{m_{1} + m_{1}^{2} - m_{2}}{m_{1}} = 1 + m_{1} - \frac{m_{2}}{m_{1}}, \\ n = \frac{m_{1}}{p} = \frac{m_{1}^{2}}{m_{1} + m_{1}^{2} - m_{2}} . \end{matrix}$

Inserting the estimators ${\hat{m}}_{1}$ and ${\hat{m}}_{2}$ for $m_{1}$ and $m_{2}$ gives the corresponding moment estimators $\hat{p}$ and $\hat{n}$ .

This is a negative example, where the moment estimators have multiple issues: First, the estimator $\hat{n}$ does not necessarily provide integers, although this can be easily resolved by rounding to the nearest integer. A more serious issue is that both estimators can lead to negative estimates, namely when ${\hat{m}}_{1} + {\hat{m}}_{1}^{2} - {\hat{m}}_{2} < 0$ , which is equivalent to the sample variance $S_{n}^{2} = {\hat{m}}_{2} - {\hat{m}}_{1}^{2}$ being larger than the sample mean ${\hat{m}}_{1}$ . This can happen, for example, in small data sets with large variation. In such cases, the moment estimators are useless.

Nonparametric estimation of mean and variance

The method of moment does not actually need any assumptions on the data distribution besides the existence of a certain number of moments. For the estimation of the mean $μ$ and the variance $σ^{2}$ using observations $X_{1}, \dots, X_{n}$ , assuming only $E [X_{i}^{2}] < \infty$ , the very same derivation as for the normal distribution (see above) applies, yielding the estimators

$\begin{matrix} \hat{μ} & = \frac{1}{n} \sum_{i = 1}^{n} X_{i} = \bar{X}, \\ {\hat{σ}}^{2} & = \frac{1}{n} \sum_{i = 1}^{n} (X_{i} - \bar{X})^{2} . \end{matrix}$

Estimation of polynomial probability densities

Another example application of the method of moments is to estimate polynomial probability density distributions. In this case, an approximating polynomial of order $N$ is defined on an interval $[a, b]$ . The method of moments then yields a system of equations, whose solution involves the inversion of a Hankel matrix.