Frequentist probability

Scope

The frequentist interpretation is a philosophical approach to the definition and use of probabilities; it is one of several such approaches. It does not claim to capture all connotations of the concept 'probable' in colloquial speech of natural languages.

As an interpretation, it is not in conflict with the mathematical axiomatization of probability theory; rather, it provides guidance for how to apply mathematical probability theory to real-world situations. It offers distinct guidance in the construction and design of practical experiments, especially when contrasted with the Bayesian interpretation. As to whether this guidance is useful, or is apt to mis-interpretation, has been a source of controversy. This is particularly true when the frequency interpretation of probability is mistakenly assumed to be the only possible basis for frequentist inference. So, for example, a list of mis-interpretations of the meaning of p-values accompanies the article on p-values; controversies are detailed in the article on statistical hypothesis testing. The Jeffreys–Lindley paradox shows how different interpretations, applied to the same data set, can lead to different conclusions about the 'statistical significance' of a result.

As Feller notes:

There is no place in our system for speculations concerning the probability that the sun will rise tomorrow. Before speaking of it we should have to agree on an (idealized) model which would presumably run along the lines "out of infinitely many worlds one is selected at random ..." Little imagination is required to construct such a model, but it appears both uninteresting and meaningless.

History

The frequentist view may have been foreshadowed by Aristotle, in Rhetoric, when he wrote:

the probable is that which for the most part happens — Aristotle Rhetoric

Poisson (1837) clearly distinguished between objective and subjective probabilities. Soon thereafter a flurry of nearly simultaneous publications by Mill, Ellis (1843) and Ellis (1854), Cournot (1843), and Fries introduced the frequentist view. Venn (1866, 1876, 1888) provided a thorough exposition two decades later. These were further supported by the publications of Boole and Bertrand. By the end of the 19th century the frequentist interpretation was well established and perhaps dominant in the sciences. The following generation established the tools of classical inferential statistics (significance testing, hypothesis testing and confidence intervals) all based on frequentist probability.

Alternatively, Bernoulli understood the concept of frequentist probability and published a critical proof (the weak law of large numbers) posthumously (Bernoulli, 1713). He is also credited with some appreciation for subjective probability (prior to and without Bayes' theorem). Gauss and Laplace used frequentist (and other) probability in derivations of the least squares method a century later, a generation before Poisson. Laplace considered the probabilities of testimonies, tables of mortality, judgments of tribunals, etc. which are unlikely candidates for classical probability. In this view, Poisson's contribution was his sharp criticism of the alternative "inverse" (subjective, Bayesian) probability interpretation. Any criticism by Gauss or Laplace was muted and implicit. (However, note that their later derivations of least squares did not use inverse probability.)

Major contributors to "classical" statistics in the early 20th century included Fisher, Neyman, and Pearson. Fisher contributed to most of statistics and made significance testing the core of experimental science, although he was critical of the frequentist concept of "repeated sampling from the same population"; Neyman formulated confidence intervals and contributed heavily to sampling theory; Neyman and Pearson paired in the creation of hypothesis testing. All valued objectivity, so the best interpretation of probability available to them was frequentist.

All were suspicious of "inverse probability" (the available alternative) with prior probabilities chosen by using the principle of indifference. Fisher said, "... the theory of inverse probability is founded upon an error, [referring to Bayes' theorem] and must be wholly rejected." While Neyman was a pure frequentist, Fisher's views of probability were unique: Both Fisher and Neyman had nuanced view of probability. von Mises offered a combination of mathematical and philosophical support for frequentism in the era.

Etymology

According to the Oxford English Dictionary, the term frequentist was first used by M.G. Kendall in 1949, to contrast with Bayesians, whom he called non-frequentists. Kendall observed

3. ... we may broadly distinguish two main attitudes. One takes probability as 'a degree of rational belief', or some similar idea...the second defines probability in terms of frequencies of occurrence of events, or by relative proportions in 'populations' or 'collectives';

...

12. It might be thought that the differences between the frequentists and the non-frequentists (if I may call them such) are largely due to the differences of the domains which they purport to cover.

...

I assert that this is not so ... The essential distinction between the frequentists and the non-frequentists is, I think, that the former, in an effort to avoid anything savouring of matters of opinion, seek to define probability in terms of the objective properties of a population, real or hypothetical, whereas the latter do not. [emphasis in original]

"The Frequency Theory of Probability" was used a generation earlier as a chapter title in Keynes (1921).

The historical sequence:

1. Probability concepts were introduced and much of the mathematics of probability derived (prior to the 20th century)
2. classical statistical inference methods were developed
3. the mathematical foundations of probability were solidified and current terminology was introduced (all in the 20th century).

The primary historical sources in probability and statistics did not use the current terminology of classical, subjective (Bayesian), and frequentist probability.

Alternative views

Probability theory is a branch of mathematics. While its roots reach centuries into the past, it reached maturity with the axioms of Andrey Kolmogorov in 1933. The theory focuses on the valid operations on probability values rather than on the initial assignment of values; the mathematics is largely independent of any interpretation of probability.

Applications and interpretations of probability are considered by philosophy, the sciences and statistics. All are interested in the extraction of knowledge from observations—inductive reasoning. There are a variety of competing interpretations; All have problems. The frequentist interpretation does resolve difficulties with the classical interpretation, such as any problem where the natural symmetry of outcomes is not known. It does not address other issues, such as the dutch book.

• Classical probability assigns probabilities based on physical idealized symmetry (dice, coins, cards). The classical definition is at risk of circularity: Probabilities are defined by assuming equality of probabilities. In the absence of symmetry the utility of the definition is limited.

• Subjective (Bayesian) probability (a family of competing interpretations) considers degrees of belief: All practical "subjective" probability interpretations are so constrained to rationality as to avoid most subjectivity. Real subjectivity is repellent to some definitions of science which strive for results independent of the observer and analyst. Other applications of Bayesianism in science (e.g. logical Bayesianism) embrace the inherent subjectivity of many scientific studies and objects and use Bayesian reasoning to place boundaries and context on the influence of subjectivities on all analysis. The historical roots of this concept extended to such non-numeric applications as legal evidence.

• Propensity probability views probability as a causative phenomenon rather than a purely descriptive or subjective one.