Probability space

Definition

In short, a probability space is a measure space such that the measure of the whole space is equal to one.

The expanded definition is the following: a probability space is a triple ${\displaystyle (\mathrm{\Omega },\mathcal{F},P)}$ consisting of:

• the sample space ${\displaystyle \mathrm{\Omega }}$ – an arbitrary non-empty set,
• the σ-algebra ${\displaystyle \mathcal{F}\subseteq 2^{\mathrm{\Omega }}}$ (also called σ-field) – a set of subsets of ${\displaystyle \mathrm{\Omega }}$, called events, such that:
  • ${\displaystyle \mathcal{F}}$ contains the sample space: ${\displaystyle \mathrm{\Omega }\in \mathcal{F}}$,
  • ${\displaystyle \mathcal{F}}$ is closed under complements: if ${\displaystyle A\in \mathcal{F}}$, then also ${\displaystyle (\mathrm{\Omega }\setminus A)\in \mathcal{F}}$,
  • ${\displaystyle \mathcal{F}}$ is closed under countable unions: if ${\displaystyle A_i\in \mathcal{F}}$ for ${\displaystyle i=1,2,\dots }$, then also $(\bigcup _{i=1}^{\mathrm{\infty }}{A}_i)\in \mathcal{F}$
    • The corollary from the previous two properties and De Morgan's law is that ${\displaystyle \mathcal{F}}$ is also closed under countable intersections: if ${\displaystyle A_i\in \mathcal{F}}$ for ${\displaystyle i=1,2,\dots }$, then also $(\bigcap _{i=1}^{\mathrm{\infty }}{A}_i)\in \mathcal{F}$
• the probability measure ${\displaystyle P:\mathcal{F}\to [0,1]}$ – a function on ${\displaystyle \mathcal{F}}$ such that:
  • P is countably additive (also called σ-additive): if ${\displaystyle \{A_i{\}}_{i=1}^{\mathrm{\infty }}\subseteq \mathcal{F}}$ is a countable collection of pairwise disjoint sets, then $P(\bigcup _{i=1}^{\mathrm{\infty }}{A}_i)=\sum _{i=1}^{\mathrm{\infty }}P(A_i),$
  • the measure of the entire sample space is equal to one: ${\displaystyle P(\mathrm{\Omega })=1}$.

Discrete case

For a countable sample space ${\displaystyle \mathrm{\Omega }}$, probabilities can be ascribed to elements of ${\displaystyle \mathrm{\Omega }}$ by a discrete probability distribution ${\displaystyle p:\mathrm{\Omega }\to [0,1]}$ such that $\sum _{\omega \in \mathrm{\Omega }}p(\omega )=1,$ whereby all subsets of ${\displaystyle \mathrm{\Omega }}$ can be treated as events (thus, ${\displaystyle \mathcal{F}=2^{\mathrm{\Omega }}}$ is the power set) and the probability measure has the form

The greatest σ-algebra ${\displaystyle \mathcal{F}=2^{\mathrm{\Omega }}}$ describes the complete information. In general, a σ-algebra ${\displaystyle \mathcal{F}\subseteq 2^{\mathrm{\Omega }}}$ corresponds to a finite or countable partition ${\displaystyle \mathrm{\Omega }=B_1\cup B_2\cup \dots }$, the general form of an event ${\displaystyle A\in \mathcal{F}}$ being ${\displaystyle A=B_{k_1}\cup B_{k_2}\cup \dots }$.

The case ${\displaystyle p(\omega )=0}$ is permitted by the definition, but rarely used, since such ${\displaystyle \omega }$ can safely be excluded from the sample space.

General case

If Ω is uncountable, still, it may happen that P(ω) ≠ 0 for some ω; such ω are called atoms. They are an at most countable (maybe empty) set, whose probability is the sum of probabilities of all atoms. If this sum is equal to 1 then all other points can safely be excluded from the sample space, returning us to the discrete case. Otherwise, if the sum of probabilities of all atoms is between 0 and 1, then the probability space decomposes into a discrete (atomic) part (maybe empty) and a non-atomic part.

Non-atomic case

If P(ω) = 0 for all ω ∈ Ω, then Ω must be uncountable (otherwise the axiom P(Ω) = 1 would be rendered impossible) and equation (1) does not apply: the probability of a set is not necessarily the sum over the probabilities of its elements, as summation is only defined for countable numbers of elements. This makes the probability space theory much more technical. A formulation stronger than summation, measure theory is applicable. Initially the probabilities are ascribed to some "generator" sets (see the examples). Then a limiting procedure allows assigning probabilities to sets that are limits of sequences of generator sets, or limits of limits, and so on. All these sets are the σ-algebra ${\displaystyle \mathcal{F}}$. For technical details see Carathéodory's extension theorem. Sets belonging to ${\displaystyle \mathcal{F}}$ are called measurable. In general they are much more complicated than generator sets, but much better than non-measurable sets.

Complete probability space

A probability space ${\displaystyle (\mathrm{\Omega },\mathcal{F},P)}$ is said to be a complete probability space if for all ${\displaystyle B\in \mathcal{F}}$ with ${\displaystyle P(B)=0}$ and all ${\displaystyle A\subset B}$ one has ${\displaystyle A\in \mathcal{F}}$. Often, the study of probability spaces is restricted to complete probability spaces.

Examples

Discrete examples

Example 1

If the experiment consists of just one flip of a fair coin, then the outcome is either heads or tails: ${\displaystyle \mathrm{\Omega }=\{\text{H},\text{T}\}}$. The σ-algebra ${\displaystyle \mathcal{F}=2^{\mathrm{\Omega }}}$ contains ${\displaystyle 2^2=4}$ events, namely: ${\displaystyle \{\text{H}\}}$ ("heads"), ${\displaystyle \{\text{T}\}}$ ("tails"), ${\displaystyle \{\}}$ ("neither heads nor tails"), and ${\displaystyle \{\text{H},\text{T}\}}$ ("either heads or tails"); in other words, ${\displaystyle \mathcal{F}=\{\{\},\{\text{H}\},\{\text{T}\},\{\text{H},\text{T}\}\}}$. There is a fifty percent chance of tossing heads and fifty percent for tails, so the probability measure in this example is ${\displaystyle P(\{\})=0}$, ${\displaystyle P(\{\text{H}\})=0.5}$, ${\displaystyle P(\{\text{T}\})=0.5}$, ${\displaystyle P(\{\text{H},\text{T}\})=1}$.

Example 2

The fair coin is tossed three times. There are 8 possible outcomes: Ω = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT} (here "HTH" for example means that first time the coin landed heads, the second time tails, and the last time heads again). The complete information is described by the σ-algebra ${\displaystyle \mathcal{F}=2^{\mathrm{\Omega }}}$ of 2⁸ = 256 events, where each of the events is a subset of Ω.

Alice knows the outcome of the second toss only. Thus her incomplete information is described by the partition Ω = A₁ ⊔ A₂ = {HHH, HHT, THH, THT} ⊔ {HTH, HTT, TTH, TTT}, where ⊔ is the disjoint union, and the corresponding σ-algebra ${\displaystyle {\mathcal{F}}_{\text{Alice}}=\{\{\},A_1,A_2,\mathrm{\Omega }\}}$. Bryan knows only the total number of tails. His partition contains four parts: Ω = B₀ ⊔ B₁ ⊔ B₂ ⊔ B₃ = {HHH} ⊔ {HHT, HTH, THH} ⊔ {TTH, THT, HTT} ⊔ {TTT}; accordingly, his σ-algebra ${\displaystyle {\mathcal{F}}_{\text{Bryan}}}$ contains 2⁴ = 16 events.

The two σ-algebras are incomparable: neither ${\displaystyle {\mathcal{F}}_{\text{Alice}}\subseteq {\mathcal{F}}_{\text{Bryan}}}$ nor ${\displaystyle {\mathcal{F}}_{\text{Bryan}}\subseteq {\mathcal{F}}_{\text{Alice}}}$; both are sub-σ-algebras of 2^Ω.

Example 3

If 100 voters are to be drawn randomly from among all voters in California and asked whom they will vote for governor, then the set of all sequences of 100 Californian voters would be the sample space Ω. We assume that sampling without replacement is used: only sequences of 100 different voters are allowed. For simplicity an ordered sample is considered, that is a sequence (Alice, Bryan) is different from (Bryan, Alice). We also take for granted that each potential voter knows exactly his/her future choice, that is he/she does not choose randomly.

Alice knows only whether or not Arnold Schwarzenegger has received at least 60 votes. Her incomplete information is described by the σ-algebra ${\displaystyle {\mathcal{F}}_{\text{Alice}}}$ that contains: (1) the set of all sequences in Ω where at least 60 people vote for Schwarzenegger; (2) the set of all sequences where fewer than 60 vote for Schwarzenegger; (3) the whole sample space Ω; and (4) the empty set ∅.

Bryan knows the exact number of voters who are going to vote for Schwarzenegger. His incomplete information is described by the corresponding partition Ω = B₀ ⊔ B₁ ⊔ ⋯ ⊔ B₁₀₀ and the σ-algebra ${\displaystyle {\mathcal{F}}_{\text{Bryan}}}$ consists of 2¹⁰¹ events.

In this case, Alice's σ-algebra is a subset of Bryan's: ${\displaystyle {\mathcal{F}}_{\text{Alice}}\subset {\mathcal{F}}_{\text{Bryan}}}$. Bryan's σ-algebra is in turn a subset of the much larger "complete information" σ-algebra 2^Ω consisting of 2^{n(n−1)⋯(n−99)} events, where n is the number of all potential voters in California.

Non-atomic examples

Example 4

A number between 0 and 1 is chosen at random, uniformly. Here Ω = [0,1], ${\displaystyle \mathcal{F}}$ is the σ-algebra of Borel sets on Ω, and P is the Lebesgue measure on [0,1].

In this case, the open intervals of the form (a,b), where 0 < a < b < 1, could be taken as the generator sets. Each such set can be ascribed the probability of P((a,b)) = (b − a), which generates the Lebesgue measure on [0,1], and the Borel σ-algebra on Ω.

Example 5

A fair coin is tossed endlessly. Here one can take Ω = {0,1}^∞, the set of all infinite sequences of numbers 0 and 1. Cylinder sets {(x₁, x₂, ...) ∈ Ω : x₁ = a₁, ..., x_n = a_n} may be used as the generator sets. Each such set describes an event in which the first n tosses have resulted in a fixed sequence (a₁, ..., a_n), and the rest of the sequence may be arbitrary. Each such event can be naturally given the probability of 2⁻ⁿ.

These two non-atomic examples are closely related: a sequence (x₁, x₂, ...) ∈ {0,1}^∞ leads to the number 2⁻¹x₁ + 2⁻²x₂ + ⋯ ∈ [0,1]. This is not a one-to-one correspondence between {0,1}^∞ and [0,1] however: it is an isomorphism modulo zero, which allows for treating the two probability spaces as two forms of the same probability space. In fact, all non-pathological non-atomic probability spaces are the same in this sense. They are so-called standard probability spaces. Basic applications of probability spaces are insensitive to standardness. However, non-discrete conditioning is easy and natural on standard probability spaces, otherwise it becomes obscure.

Related concepts

Probability distribution

Random variables

A random variable X is a measurable function X: Ω → S from the sample space Ω to another measurable space S called the state space.

If A ⊂ S, the notation Pr(X ∈ A) is a commonly used shorthand for ${\displaystyle P(\{\omega \in \mathrm{\Omega }:X(\omega )\in A\})}$.

Defining the events in terms of the sample space

If Ω is countable, we almost always define ${\displaystyle \mathcal{F}}$ as the power set of Ω, i.e. ${\displaystyle \mathcal{F}=2^{\mathrm{\Omega }}}$ which is trivially a σ-algebra and the biggest one we can create using Ω. We can therefore omit ${\displaystyle \mathcal{F}}$ and just write (Ω,P) to define the probability space.

On the other hand, if Ω is uncountable and we use ${\displaystyle \mathcal{F}=2^{\mathrm{\Omega }}}$ we get into trouble defining our probability measure P because ${\displaystyle \mathcal{F}}$ is too "large", i.e. there will often be sets to which it will be impossible to assign a unique measure. In this case, we have to use a smaller σ-algebra ${\displaystyle \mathcal{F}}$, for example the Borel algebra of Ω, which is the smallest σ-algebra that makes all open sets measurable.

Conditional probability

Kolmogorov's definition of probability spaces gives rise to the natural concept of conditional probability. Every set A with non-zero probability (that is, P(A) > 0) defines another probability measure $${\displaystyle P(B\mid A)=\frac{P(B\cap A)}{P(A)}}$$ on the space. This is usually pronounced as the "probability of B given A".

For any event A such that P(A) > 0, the function Q defined by Q(B) = P(B | A) for all events B is itself a probability measure.

Independence

Two events, A and B are said to be independent if P(A ∩ B) = P(A) P(B).

Two random variables, X and Y, are said to be independent if any event defined in terms of X is independent of any event defined in terms of Y. Formally, they generate independent σ-algebras, where two σ-algebras G and H, which are subsets of F are said to be independent if any element of G is independent of any element of H.

Mutual exclusivity

Two events, A and B are said to be mutually exclusive or disjoint if the occurrence of one implies the non-occurrence of the other, i.e., their intersection is empty. This is a stronger condition than the probability of their intersection being zero.

If A and B are disjoint events, then P(A ∪ B) = P(A) + P(B). This extends to a (finite or countably infinite) sequence of events. However, the probability of the union of an uncountable set of events is not the sum of their probabilities. For example, if Z is a normally distributed random variable, then P(Z = x) is 0 for any x, but P(Z ∈ R) = 1.

The event A ∩ B is referred to as "A and B", and the event A ∪ B as "A or B".