Conditional probability

Example

Suppose that somebody secretly rolls two fair six-sided dice, and we wish to compute the probability that the face-up value of the first one is 2, given the information that their sum is no greater than 5.

• Let D₁ be the value rolled on dice 1.
• Let D₂ be the value rolled on dice 2.

Probability that D₁ = 2

Table 1 shows the sample space of 36 combinations of rolled values of the two dice, each of which occurs with probability 1/36, with the numbers displayed in the red and dark gray cells being D₁ + D₂.

D₁ = 2 in exactly 6 of the 36 outcomes; thus P(D₁ = 2) = 6⁄36 = 1⁄6:

Probability that D₁ + D₂ ≤ 5

Table 2 shows that D₁ + D₂ ≤ 5 for exactly 10 of the 36 outcomes, thus P(D₁ + D₂ ≤ 5) = 10⁄36:

Probability that D₁ = 2 given that D₁ + D₂ ≤ 5

Table 3 shows that for 3 of these 10 outcomes, D₁ = 2.

Thus, the conditional probability P(D₁ = 2 | D₁+D₂ ≤ 5) = 3⁄10 = 0.3:

Here, in the earlier notation for the definition of conditional probability, the conditioning event B is that D₁ + D₂ ≤ 5, and the event A is D₁ = 2. We have ${\displaystyle P(A\mid B)=\frac{P(A\cap B)}{P(B)}=\frac{3/36}{10/36}=\frac{3}{10},}$ as seen in the table.

Use in inference

In statistical inference, the conditional probability is an update of the probability of an event based on new information. The new information can be incorporated as follows:

• Let A, the event of interest, be in the sample space, say (X,P).
• The occurrence of the event A knowing that event B has or will have occurred, means the occurrence of A as it is restricted to B, i.e. ${\displaystyle A\cap B}$.
• Without the knowledge of the occurrence of B, the information about the occurrence of A would simply be P(A)
• The probability of A knowing that event B has or will have occurred, will be the probability of ${\displaystyle A\cap B}$ relative to P(B), the probability that B has occurred.
• This results in $P(A\mid B)=P(A\cap B)/P(B)$ whenever P(B) > 0 and 0 otherwise.

This approach results in a probability measure that is consistent with the original probability measure and satisfies all the Kolmogorov axioms. This conditional probability measure also could have resulted by assuming that the relative magnitude of the probability of A with respect to X will be preserved with respect to B (cf. a Formal Derivation below).

The wording "evidence" or "information" is generally used in the Bayesian interpretation of probability. The conditioning event is interpreted as evidence for the conditioned event. That is, P(A) is the probability of A before accounting for evidence E, and P(A|E) is the probability of A after having accounted for evidence E or after having updated P(A). This is consistent with the frequentist interpretation, which is the first definition given above.

Example

When Morse code is transmitted, there is a certain probability that the "dot" or "dash" that was received is erroneous. This is often taken as interference in the transmission of a message. Therefore, it is important to consider when sending a "dot", for example, the probability that a "dot" was received. This is represented by: ${\displaystyle P(\text{dot sent }|\text{ dot received})=P(\text{dot received }|\text{ dot sent})\frac{P(\text{dot sent})}{P(\text{dot received})}.}$ In Morse code, the ratio of dots to dashes is 3:4 at the point of sending, so the probabilities of a "dot" and "dash" are ${\displaystyle P(\text{dot sent})=\frac{3}{7}\text{ and }P(\text{dash sent})=\frac{4}{7}}$. If it is assumed that the probability that a dot is transmitted as a dash is 1/10, and that the probability that a dash is transmitted as a dot is likewise 1/10, then Bayes's rule can be used to calculate ${\displaystyle P(\text{dot received})}$.

${\displaystyle P(\text{dot received})=P(\text{dot received}\cap \text{dot sent})+P(\text{dot received}\cap \text{dash sent})}$

${\displaystyle P(\text{dot received})=P(\text{dot received}\mid \text{dot sent})P(\text{dot sent})+P(\text{dot received}\mid \text{dash sent})P(\text{dash sent})}$

${\displaystyle P(\text{dot received})=\frac{9}{10}\times \frac{3}{7}+\frac{1}{10}\times \frac{4}{7}=\frac{31}{70}}$

Now, ${\displaystyle P(\text{dot sent}\mid \text{dot received})}$ can be calculated:

${\displaystyle P(\text{dot sent}\mid \text{dot received})=P(\text{dot received}\mid \text{dot sent})\frac{P(\text{dot sent})}{P(\text{dot received})}=\frac{9}{10}\times \frac{\frac{3}{7}}{\frac{31}{70}}=\frac{27}{31}}$

Information, conditional probability, and statistical independence

The concepts of conditional probability and statistical independence can be understood through the idea of how new information changes an individual's uncertainty about an event.

Example 1

Dependent events (six-sided die)

Consider the experiment of rolling a fair six-sided die, where the sample space is ${\displaystyle S=\{1,2,3,4,5,6\}}$. Let event A denote the occurrence of an even number, so ${\displaystyle A=\{2,4,6\}}$, and let event B denote the occurrence of a number less than or equal to 3, so ${\displaystyle B=\{1,2,3\}}$. Before any additional information is available, the probabilities of these events are ${\displaystyle P(A)=\frac{3}{6}=\frac{1}{2}}$ and ${\displaystyle P(B)=\frac{3}{6}=\frac{1}{2}}$. The intersection of the two events is ${\displaystyle A\cap B=\{2\}}$, and therefore ${\displaystyle P(A\cap B)=\frac{1}{6}}$. If the events were statistically independent, we would expect ${\displaystyle P(A\cap B)=P(A)P(B)}$. However, ${\displaystyle \frac{1}{6}\ne \frac{1}{2}\times \frac{1}{2}=\frac{1}{4}}$, indicating that events A and B are not independent but are instead statistically dependent.

Now suppose that an observer reveals that event A has occurred; that is, the outcome is known to be an even number. This new information reduces the effective sample space from ${\displaystyle S=\{1,2,3,4,5,6\}}$ to ${\displaystyle A=\{2,4,6\}}$. Within this reduced sample space, event B can occur only when the outcome is 2, so the relevant event becomes ${\displaystyle A\cap B=\{2\}}$. Consequently, the conditional probability of event B given that event A has occurred is ${\displaystyle P(B\mid A)=\frac{P(A\cap B)}{P(A)}=\frac{1/6}{1/2}=\frac{1}{3}}$. Thus, the probability of event B changes from ${\displaystyle \frac{1}{2}}$ to ${\displaystyle \frac{1}{3}}$ after the additional information is provided. This change in probability demonstrates that knowledge of event A affects the likelihood of event B, confirming that the two events are statistically dependent. In such situations, additional information reduces uncertainty and can significantly influence rational decision-making, prediction, and betting behaviour.

Example 2

Independent events (eight-sided die)

Consider the experiment of rolling a fair eight-sided die with sample space ${\displaystyle S=\{1,2,3,4,5,6,7,8\}}$. Let event A represent the outcome being less than 5, so that ${\displaystyle A=\{1,2,3,4\}}$, and let event B represent the outcome being either 3, 4, 5, or 6, so that ${\displaystyle B=\{3,4,5,6\}}$. Initially, the probabilities of these events are ${\displaystyle P(A)=\frac{4}{8}=\frac{1}{2}}$ and ${\displaystyle P(B)=\frac{4}{8}=\frac{1}{2}}$. The intersection of the two events is ${\displaystyle A\cap B=\{3,4\}}$, and therefore ${\displaystyle P(A\cap B)=\frac{2}{8}=\frac{1}{4}}$. Since ${\displaystyle P(A)P(B=\frac{1}{2}\times \frac{1}{2}=\frac{1}{4}}$, we obtain ${\displaystyle P(A\cap B)=P(A)P(B)}$, which confirms that events A and B are statistically independent.

Now suppose that an observer reveals that event (A) has occurred. This information reduces the effective sample space from ${\displaystyle S=\{1,2,3,4,5,6,7,8\}}$ to ${\displaystyle A=\{1,2,3,4\}}$. Within this reduced sample space, the outcomes that also satisfy event B are ${\displaystyle A\cap B=\{3,4\}}$. Consequently, the conditional probability of event B given that event A has occurred is ${\displaystyle P(B\mid A)=\frac{P(A\cap B)}{P(A)}=\frac{1/4}{1/2}=\frac{1}{2}}$. Notice that this conditional probability is equal to the original probability of event B, that is, ${\displaystyle P(B\mid A)=P(B)}$. Therefore, learning that event A has occurred does not change the likelihood of event B. The information provided by event A has no effect on the uncertainty associated with event B, demonstrating the fundamental property of statistical independence: knowledge of one event does not alter the probability of the other event.

Interpretation

Statistical independence can therefore be interpreted as the absence of informational value between events. If knowledge of one event changes the probability of another event, the events are dependent. Conversely, if the probability remains unchanged after obtaining new information, the events are independent.

Formally, two events A and B are independent if and only if:

${\displaystyle P(B\mid A)=P(B),}$ which is equivalent to: ${\displaystyle P(A\cap B)=P(A)P(B).}$

In this sense, independence implies that observing one event provides no additional information about the occurrence of the other event.

Statistical independence

Events A and B are defined to be statistically independent if the probability of the intersection of A and B is equal to the product of the probabilities of A and B:

${\displaystyle P(A\cap B)=P(A)P(B).}$

If P(B) is not zero, then this is equivalent to the statement that

${\displaystyle P(A\mid B)=P(A).}$

Similarly, if P(A) is not zero, then

${\displaystyle P(B\mid A)=P(B)}$

is also equivalent. Although the derived forms may seem more intuitive, they are not the preferred definition as the conditional probabilities may be undefined, and the preferred definition is symmetrical in A and B. Independence does not refer to a disjoint event.

It should also be noted that given the independent event pair [A,B] and an event C, the pair is defined to be conditionally independent if

${\displaystyle P(AB\mid C)=P(A\mid C)P(B\mid C).}$

This theorem is useful in applications where multiple independent events are being observed.

Independent events vs. mutually exclusive events

The concepts of mutually independent events and mutually exclusive events are separate and distinct. The following table contrasts results for the two cases (provided that the probability of the conditioning event is not zero).

In fact, mutually exclusive events cannot be statistically independent (unless both of them are impossible), since knowing that one occurs gives information about the other (in particular, that the latter will certainly not occur).

Independent vs. Positive association vs. Negative association of events

Common fallacies

These fallacies should not be confused with Robert K. Shope's 1978 "conditional fallacy", which deals with counterfactual examples that beg the question.

Assuming conditional probability is of similar size to its inverse

In general, it cannot be assumed that P(A|B) ≈ P(B|A). This can be an insidious error, even for those who are highly conversant with statistics. The relationship between P(A|B) and P(B|A) is given by Bayes' theorem:

${\displaystyle \begin{array}{cc}P(B\mid A)& =\frac{P(A\mid B)P(B)}{P(A)}\\ \iff \frac{P(B\mid A)}{P(A\mid B)}& =\frac{P(B)}{P(A)}\end{array}}$

That is, P(A|B) ≈ P(B|A) only if P(B)/P(A) ≈ 1, or equivalently, P(A) ≈ P(B).

Assuming marginal and conditional probabilities are of similar size

In general, it cannot be assumed that P(A) ≈ P(A|B). These probabilities are linked through the law of total probability:

${\displaystyle P(A)=\sum _{n}P(A\cap B_n)=\sum _{n}P(A\mid B_n)P(B_n).}$

where the events ${\displaystyle (B_n)}$ form a countable partition of ${\displaystyle \mathrm{\Omega }}$.

This fallacy may arise through selection bias. For example, in the context of a medical claim, let S_C be the event that a sequela (chronic disease) S occurs as a consequence of circumstance (acute condition) C. Let H be the event that an individual seeks medical help. Suppose that in most cases, C does not cause S (so that P(S_C) is low). Suppose also that medical attention is only sought if S has occurred due to C. From experience of patients, a doctor may therefore erroneously conclude that P(S_C) is high. The actual probability observed by the doctor is P(S_C|H).

Over- or under-weighting priors

Not taking prior probability into account partially or completely is called base rate neglect. The reverse, insufficient adjustment from the prior probability is conservatism.

Formal derivation

Formally, P(A | B) is defined as the probability of A according to a new probability function on the sample space, such that outcomes not in B have probability 0 and that it is consistent with all original probability measures.

Let Ω be a discrete sample space with elementary events {ω}, and let P be the probability measure with respect to the σ-algebra of Ω. Suppose we are told that the event B ⊆ Ω has occurred. A new probability distribution (denoted by the conditional notation) is to be assigned on {ω} to reflect this. All events that are not in B will have null probability in the new distribution. For events in B, two conditions must be met: the probability of B is one and the relative magnitudes of the probabilities must be preserved. The former is required by the axioms of probability, and the latter stems from the fact that the new probability measure has to be the analog of P in which the probability of B is one—and every event that is not in B, therefore, has a null probability. Hence, for some scale factor α, the new distribution must satisfy:

1. ${\displaystyle \omega \in B:P(\omega \mid B)=\alpha P(\omega )}$
2. ${\displaystyle \omega \notin B:P(\omega \mid B)=0}$
3. ${\displaystyle \sum _{\omega \in \mathrm{\Omega }}P(\omega \mid B)=1.}$

Substituting 1 and 2 into 3 to select α:

${\displaystyle \begin{array}{cc}1& =\sum _{\omega \in \mathrm{\Omega }}P(\omega \mid B)\\ & =\sum _{\omega \in B}P(\omega \mid B)+{\cancel{\sum _{\omega \notin B}P(\omega \mid B)}}^0\\ & =\alpha \sum _{\omega \in B}P(\omega )\\ & =\alpha \cdot P(B)\\ \Rightarrow \alpha & =\frac{1}{P(B)}\end{array}}$

So the new probability distribution is

1. ${\displaystyle \omega \in B:P(\omega \mid B)=\frac{P(\omega )}{P(B)}}$
2. ${\displaystyle \omega \notin B:P(\omega \mid B)=0}$

Now for a general event A,

${\displaystyle \begin{array}{cc}P(A\mid B)& =\sum _{\omega \in A\cap B}P(\omega \mid B)+{\cancel{\sum _{\omega \in A\cap B^c}P(\omega \mid B)}}^0\\ & =\sum _{\omega \in A\cap B}\frac{P(\omega )}{P(B)}\\ & =\frac{P(A\cap B)}{P(B)}\end{array}}$