Maximum a posteriori estimation

Computation

MAP estimates can be computed in several ways:

1. Analytically, when the mode(s) of the posterior density can be given in closed form. This is the case when conjugate priors are used.
2. Via numerical optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
3. Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.
4. Via a Monte Carlo method using simulated annealing

Limitations

While only mild conditions are required for MAP estimation to be a limiting case of Bayes estimation (under the 0–1 loss function), it is not representative of Bayesian methods in general. This is because MAP estimates are point estimates, and depend on the arbitrary choice of reference measure, whereas Bayesian methods are characterized by the use of distributions to summarize data and draw inferences: thus, Bayesian methods tend to report the posterior mean or median instead, together with credible intervals. This is both because these estimators are optimal under squared-error and linear-error loss respectively—which are more representative of typical loss functions—and for a continuous posterior distribution there is no loss function which suggests the MAP is the optimal point estimator.

In many types of models, such as mixture models, the posterior may be multi-modal. In such a case, the usual recommendation is that one should choose the highest mode: this is not always feasible (global optimization is a difficult problem), nor in some cases even possible (such as when identifiability issues arise). Furthermore, the highest mode may be uncharacteristic of the majority of the posterior, especially in many dimensions.

Finally, unlike ML estimators, the MAP estimate is not invariant under reparameterization. Switching from one parameterization to another involves introducing a Jacobian that impacts on the location of the maximum. In contrast, Bayesian posterior expectations are invariant under reparameterization.

As an example of the difference between Bayes estimators mentioned above (mean and median estimators) and using a MAP estimate, consider the case where there is a need to classify inputs ${\displaystyle x}$ as either positive or negative (for example, loans as risky or safe). Suppose there are just three possible hypotheses about the correct method of classification ${\displaystyle h_1}$, ${\displaystyle h_2}$ and ${\displaystyle h_3}$ with posteriors 0.4, 0.3 and 0.3 respectively. Suppose given a new instance, ${\displaystyle x}$, ${\displaystyle h_1}$ classifies it as positive, whereas the other two classify it as negative. Using the MAP estimate for the correct classifier ${\displaystyle h_1}$, ${\displaystyle x}$ is classified as positive, whereas the Bayes estimators would average over all hypotheses and classify ${\displaystyle x}$ as negative.

Example

Suppose that we are given a sequence ${\displaystyle (x_1,\dots ,x_n)}$ of IID ${\displaystyle N(\mu ,{\sigma }_v^2)}$ random variables and a prior distribution of ${\displaystyle \mu }$ is given by ${\displaystyle N({\mu }_0,{\sigma }_m^2)}$. We wish to find the MAP estimate of ${\displaystyle \mu }$. Note that the normal distribution is its own conjugate prior, so we will be able to find a closed-form solution analytically.

The function to be maximized is then given by

${\displaystyle g(\mu )f(x\mid \mu )=\pi (\mu )L(\mu )=\frac{1}{\sqrt{2\pi }{\sigma }_m}\exp \left(-\frac{1}{2}{\left(\frac{\mu -{\mu }_0}{{\sigma }_m}\right)}^2\right)\prod _{j=1}^n\frac{1}{\sqrt{2\pi }{\sigma }_v}\exp \left(-\frac{1}{2}{\left(\frac{x_j-\mu }{{\sigma }_v}\right)}^2\right),}$

which is equivalent to minimizing the following function of ${\displaystyle \mu }$:

${\displaystyle \sum _{j=1}^n{\left(\frac{x_j-\mu }{{\sigma }_v}\right)}^2+{\left(\frac{\mu -{\mu }_0}{{\sigma }_m}\right)}^2.}$

Thus, we see that the MAP estimator for μ is given by

${\displaystyle {\hat{\mu }}_{\mathrm{M}\mathrm{A}\mathrm{P}}=\frac{{\sigma }_m^{2}n}{{\sigma }_m^{2}n+{\sigma }_v^2}\left(\frac{1}{n}\sum _{j=1}^n{x}_j\right)+\frac{{\sigma }_v^2}{{\sigma }_m^{2}n+{\sigma }_v^2}{\mu }_0=\frac{{\sigma }_m^2\left(\sum _{j=1}^n{x}_j\right)+{\sigma }_v^2{\mu }_0}{{\sigma }_m^{2}n+{\sigma }_v^2}.}$

which turns out to be a linear interpolation between the prior mean and the sample mean weighted by their respective covariances.

The case of ${\displaystyle {\sigma }_m\to \mathrm{\infty }}$ is called a non-informative prior and leads to an improper probability distribution; in this case ${\displaystyle {\hat{\mu }}_{\mathrm{M}\mathrm{A}\mathrm{P}}\to {\hat{\mu }}_{\mathrm{M}\mathrm{L}\mathrm{E}}.}$