The things we can assign probabilities to are called random experiments. The (finite) set of possible outcomes to a random experiment is called the sample space (we will usually denote this set as \(S\)). Examples:
An event is any subset of the sample space. For example, when rolling a fair six sided dice, we can define event \(A\) as the event when getting an even number, and event \(B\) as the event when getting a number less than or equal to \(3\):
We can portray this in a Venn diagram:
The intersection, \(A \cap B\), is \(\{2\}\), and the union, \(A \cup B\), is \(\{1,2,3,4,6\}\).
The cardinality of some set \(E\), denoted as \(|E|\), defines the number of elements in a set. For example, \(|B| = 3\) and \(|A \cup B| = 5\).
The probability of an event is the number of outcomes in that event divided by the total number of outcomes. Examples:
If I roll a dice and I told you that event \(B\) definitely happened (i.e. the outcome is less than or equal to 3), but I don't tell you what the outcome actually is. What is the likelihood that \(A\) actually happens?
If we know event \(B\) definitely happened, then for event \(A\) to happen, the outcome has to be 2. Conditional probability is the probability of a event given another event already took place. The conditional probability of \(A\) given \(B\) is denoted as \(P(A|B)\), and since \(B\) has three elements and since only one of them belongs in \(A\), then:
The definition of a conditional probability: The probability of \(X\) given \(Y\) is:
This can be written another way:
Consider another scenario:
Here, \(P(E) = 5/10 = 1/2 = 0.5\), and \(P(E|F) = \frac{3}{6} = 0.5\). This means \(P(E|F)=P(E)\). Similarly, since \(P(F) = 6/10 = 0.6\) and \(P(F|E) = 3/5 = 0.6 \), then \(P(F)=P(F|E)\). If the probability of an events, like \(E\), doesn't change when the another event, like \(F\), happens, then the events are independent.
Claim: If \(P(X|Y)=P(X)\), then \(P(Y|X)=P(Y)\). Based on our definition of conditional probability, we know:
If \(P(X|Y)=P(X)\), then \(P(X|Y)P(Y) = P(Y|X)P(Y)\) impplies \(P(X)P(Y) = P(Y|X)P(X)\). For the LHS and RHS to match, \(P(Y|X)\) would have to be \(P(Y)\). This also means that if \(X\) and \(Y\) are independent, then \(P(X)P(Y) = P(X \cap Y)\).
Now let's focus on union. The probability of \(P(E \cup F)\) is \(\frac{|E \cup F|}{|S|} = \frac{8}{10} = 0.8\). Let's try to get the value of the formula for the union \((|E \cup F|)\). If we individually add the number of elements in each set, we get \(|E| + |F| = 5 + 6 = 11\), but this is not the number of elements in \(E \cup F\) (which is 8). This is because the intersection we counted twice, so we should subtract it once:
This means:
If \(|X \cap Y|=0\), then the events \(X\) and \(Y\) are mutually exclusive. Consider the example of rolling the dice we had above, with \(A=\{2,4,6\}\), and let \(C = \{5\}\). This means events \(A\) and \(C\) are mutually exclusive because \(|A \cap C| = 0\). This also means \(P(A \cap C) = P(A) + P(C)\).
In summary, to find the probability of \(X\) and \(Y\):
To find the probability of \(X\) or \(Y\):