Conditional Probability

Overview

If $A$ and $B$ are events with $P (B) > 0$ , then the conditional probability of $A$ given $B$ , denoted by $P (A | B)$ , is defined as

P (A | B) = \frac{P (A \cap B)}{P (B)} .

Let $B_{1}, B_{2}, \dots, B_{n}$ be a set of observed events. Generalizing, we say

P (A | B_{1}, B_{2}, \dots, B_{n}) = \frac{A \cap B_{1} \cap B_{2} \cap \dots \cap B_{n}}{B_{1} \cap B_{2} \cap \dots \cap B_{n}} .

$P (A)$ is the prior probability of $A$ whereas $P (A | B)$ is the posterior probability of $A$ . The symbol bar used to separate the event whose probability is being updated is called the conditioning bar.

Bayes' Theorem

Let $⟨ S, P ⟩$ be a probability space and $A, B \subseteq S$ be events with positive probabilities. Then Bayes' theorem states that

P (A | B) = \frac{P (B | A) P (A)}{P (B)} .

Odds Form

For any events $A$ and $B$ with positive probabilities, the odds of $A$ after conditioning on $B$ are

\frac{P (A | B)}{P (A^{C} | B)} = \frac{P (B | A)}{P (B | A^{C})} \frac{P (A)}{P (A^{C})} .

In other words, the posterior odds $P (A | B) / P (A^{C} | B)$ are equal to the prior odds $P (A) / P (A^{C})$ times the likelihood ratio $P (B | A) / P (B | A^{C})$ .

Law of Total Probability

Let $A_{1}, \dots, A_{n}$ be a partition of sample space $S$ , with $P (A_{i}) > 0$ for all $1 \leq i \leq n$ . Then

P (B) = \sum_{i = 1}^{n} P (B | A_{i}) P (A_{i}) .

Coherency

Bayesian reasoning is coherent meaning new information, i.e. evidence, can be incorporated sequentially or simultaneously without affecting the result.

Reasoning

Reasoning refers to the decision-making process made in the wake of probabilistic events. A number of biases exist innately that yield "paradoxes" hard to reason about, especially in light of conditional data.

Consider $P (A | B)$ . If $B$ is perceived as a cause of the occurrence of $A$ , we refer to $B$ as a causal datum. If $A$ is instead treated as a possible cause of $B$ , we refer to $B$ as a diagnostic datum. Otherwise we refer to $B$ as an incidental datum.

Notice in normative analysis, the distinction between causal, diagnostic, and incidental is irrelevant. This is purely a psychological perspective that explains biases humans tend to have around conditional reasoning.

Cromwell's Rule

Cromwell's rule is the principle stating one should avoid assigning probabilities of $0$ or $1$ to any event (except mathematical certainties).

Let $B$ with $P (B) > 0$ . If $P (A) = 1$ , then $P (A | B) = 1$ . Likewise, if $P (A) = 0$ , then $P (A | B) = 0$ . In other words, if someone were to believe an event is to occur with absolute certainty, no amount of evidence can be used to change their mind.