Bayesian statistics

Bayesian statistical methods use Bayes' theorem to compute and update probabilities after obtaining new data. Bayes' theorem describes the conditional probability of an event based on data as well as prior information or beliefs about the event or conditions related to the event. For example, in Bayesian inference, Bayes' theorem can be used to estimate the parameters of a probability distribution or statistical model. Since Bayesian statistics treats probability as a degree of belief, Bayes' theorem can directly assign a probability distribution that quantifies the belief to the parameter or set of parameters.

Bayesian statistics is named after Thomas Bayes, who formulated a specific case of Bayes' theorem in a paper published in 1763. In several papers spanning from the late 18th to the early 19th centuries, Pierre-Simon Laplace developed the Bayesian interpretation of probability. Laplace used methods now considered Bayesian to solve a number of statistical problems. While many Bayesian methods were developed by later authors, the term "Bayesian" was not commonly used to describe these methods until the 1950s. Throughout much of the 20th century, Bayesian methods were viewed unfavorably by many statisticians due to philosophical and practical considerations. Many of these methods required much computation, and most widely used approaches during that time were based on the frequentist interpretation. However, with the advent of powerful computers and new algorithms like Markov chain Monte Carlo, Bayesian methods have gained increasing prominence in statistics in the 21st century.

Bayes' theorem

Bayes' theorem is used in Bayesian methods to update probabilities, which are degrees of belief, after obtaining new data. Given two events ${\displaystyle A}$ and ${\displaystyle B}$, the conditional probability of ${\displaystyle A}$ given that ${\displaystyle B}$ is true is expressed as follows:

$${\displaystyle P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}}$$

where ${\displaystyle P(B)\ne 0}$. Although Bayes' theorem is a fundamental result of probability theory, it has a specific interpretation in Bayesian statistics. In the above equation, ${\displaystyle A}$ usually represents a proposition (such as the statement that a coin lands on heads fifty percent of the time) and ${\displaystyle B}$ represents the evidence, or new data that is to be taken into account (such as the result of a series of coin flips). ${\displaystyle P(A)}$ is the prior probability of ${\displaystyle A}$ which expresses one's beliefs about ${\displaystyle A}$ before evidence is taken into account. The prior probability may also quantify prior knowledge or information about ${\displaystyle A}$. ${\displaystyle P(B\mid A)}$ is the likelihood function, which can be interpreted as the probability of the evidence ${\displaystyle B}$ given that ${\displaystyle A}$ is true. The likelihood quantifies the extent to which the evidence ${\displaystyle B}$ supports the proposition ${\displaystyle A}$. ${\displaystyle P(A\mid B)}$ is the posterior probability, the probability of the proposition ${\displaystyle A}$ after taking the evidence ${\displaystyle B}$ into account. Essentially, Bayes' theorem updates one's prior beliefs ${\displaystyle P(A)}$ after considering the new evidence ${\displaystyle B}$.

The probability of the evidence ${\displaystyle P(B)}$ can be calculated using the law of total probability. If ${\displaystyle \{A_1,A_2,\dots ,A_n\}}$ is a partition of the sample space, which is the set of all outcomes of an experiment, then,

$${\displaystyle P(B)=P(B\mid A_1)P(A_1)+P(B\mid A_2)P(A_2)+\cdots +P(B\mid A_n)P(A_n)=\sum _{i}P(B\mid A_i)P(A_i)}$$

When there are an infinite number of outcomes, it is necessary to integrate over all outcomes to calculate ${\displaystyle P(B)}$ using the law of total probability. Often, ${\displaystyle P(B)}$ is difficult to calculate as the calculation would involve sums or integrals that would be time-consuming to evaluate, so often only the product of the prior and likelihood is considered, since the evidence does not change in the same analysis. The posterior is proportional to this product:

$${\displaystyle P(A\mid B)\propto P(B\mid A)P(A)}$$

The maximum a posteriori, which is the mode of the posterior and is often computed in Bayesian statistics using mathematical optimization methods, remains the same. The posterior can be approximated even without computing the exact value of ${\displaystyle P(B)}$ with methods such as Markov chain Monte Carlo or variational Bayesian methods.

Construction

The classical textbook equation for the posterior in Bayesian statistics is usually stated as $${\displaystyle \pi (\theta \mid x)=\mathcal{L}(x\mid \theta )\cdot \frac{\pi (\theta )}{{\int }_{\mathrm{\Theta }}\mathcal{L}(x\mid {\theta }^{\prime })\cdot \pi ({\theta }^{\prime })d{\theta }^{\prime }}}$$ where ${\displaystyle \pi (\theta \mid x)}$ is the updated probability of ${\displaystyle \theta }$ being the true parameter after collecting the data ${\displaystyle x}$, ${\displaystyle \mathcal{L}(x\mid \theta )}$ is the likelihood of collecting the data ${\displaystyle x}$ given the parameter ${\displaystyle \theta }$, ${\displaystyle \pi (\theta )}$ is the prior belief of ${\displaystyle \theta }$'s likelihood and the integral in the denominator gives the probability of collecting the data ${\displaystyle x}$.

Mathematically, this version of Bayes' theorem can be constructed in the following way: Suppose ${\displaystyle (\mathrm{\Omega },{\mathrm{\Sigma }}_{\mathrm{\Omega }},\{P_{\theta }\mid \theta \in \mathrm{\Theta }\})}$ to be some parametric statistical model and ${\displaystyle (\mathrm{\Theta },{\mathrm{\Sigma }}_{\mathrm{\Theta }},\pi )}$ to be a probability space over the parameter space. We can construct a new probability space ${\displaystyle (\mathrm{\Theta }\times \mathrm{\Omega },{\mathrm{\Sigma }}_{\mathrm{\Theta }}\otimes {\mathrm{\Sigma }}_{\mathrm{\Omega }},Q)}$ where ${\displaystyle Q}$ is a sort of product measure defined as: ${\displaystyle Q(M):=(\pi \otimes P_{\cdot })(M)={\int }_{\mathrm{\Theta }}{P}_{{\theta }^{\prime }}(M_{{\theta }^{\prime }})d\pi ({\theta }^{\prime }),}$ where ${\displaystyle M_{{\theta }^{\prime }}:=\{\omega \in \mathrm{\Omega }\mid ({\theta }^{\prime },\omega )\in M\})}$.

Now, let ${\displaystyle A_{\theta }:=\{\theta \}\times \mathrm{\Omega }}$ and ${\displaystyle B_x:=\mathrm{\Theta }\times \{x\}}$, then we get: ${\displaystyle Q(\theta )=Q(A_{\theta })={\int }_{\{\theta \}}{P}_{{\theta }^{\prime }}(\mathrm{\Omega })d\pi ({\theta }^{\prime })=\pi (\{\theta \})\cdot P_{\theta }(\mathrm{\Omega })=\pi (\theta )}$

and hence

${\displaystyle Q(x\mid \theta )=\frac{Q(B_x\cap A_{\theta })}{Q(A_{\theta })}=\frac{\pi (\theta )\cdot P_{\theta }(\{x\})}{\pi (\theta )}=P_{\theta }(x)}$

both as empirically might be expected. Thus, Bayes' theorem states:

${\displaystyle Q(\theta \mid x)=P_{\theta }(x)\cdot \frac{\pi (\theta )}{Q(x)}}$

If ${\displaystyle \pi \ll \lambda }$ (i.e. absolutely continuous w.r.t. the Lebesgue measure ), then there exists a density such that ${\displaystyle \pi (\theta )=\frac{d\pi }{d\lambda }(\theta )}$ and we can write:

${\displaystyle Q(x)={\int }_{\mathrm{\Theta }}{P}_{{\theta }^{\prime }}(x)d\pi ({\theta }^{\prime })={\int }_{\mathrm{\Theta }}{P}_{{\theta }^{\prime }}(x)\cdot \pi ({\theta }^{\prime })d{\theta }^{\prime }}$

Else, if ${\displaystyle \pi \ll \nu }$ (absolutely continuous w.r.t. counting measure), analogous we can write:

${\displaystyle Q(x)={\int }_{\mathrm{\Theta }}{P}_{{\theta }^{\prime }}(x)\cdot \pi ({\theta }^{\prime })d\nu ({\theta }^{\prime })=\sum _i{P}_{{\theta }_i}(x)\cdot \pi ({\theta }_i)}$

Thus, by identifying ${\displaystyle Q(\theta \mid x)}$ with ${\displaystyle \pi (\theta \mid x)}$ and ${\displaystyle \mathcal{L}(x\mid \theta )}$ with ${\displaystyle P_{\theta }(x)}$ we arrive at the classical equation stated above.