4  Independence

Reference: Chapter 4. Resnick (2005).

Definition 4.1 (\(\color{magenta}{\textbf{Independent events}}\))
Let’s consider a probability space \((\Omega, \mathcal{B}, \mathbb{P})\) and two events \(A, B \in \mathcal{B}\). The events \(A\) and \(B\) are said to be independent if \[ A \perp B \iff \mathbb{P}(A {\color{blue}{\cap}} B) = \mathbb{P}(A) \mathbb{P}(B) \text{.} \tag{4.1}\] Similarly, a finite sequence of events \(A_1, A_2, \dots , A_n \in \mathcal{B}\), is said to be independent, if for all \(2 \le j \le n\) and \(1 \le k_1 \le k_2 \le \dots \le k_j \le n\) the following factorization holds \[ A_{k_1} \perp A_{k_2} \perp \dots \perp A_{k_n} \iff \mathbb{P}(A_{k_1} {\color{blue}{\cap}} A_{k_2} {\color{blue}{\cap}} \dots {\color{blue}{\cap}} A_{k_n}) = \prod_{j = 1}^{n} \mathbb{P}(A_{k_j}) \text{.} \]

Example 4.1 Consider the probability space \((\Omega, \mathcal{P}(\Omega), \mathbb{P})\), where \(\Omega = \{1,2,3,4,5,6\}\) and for every \(\omega_i \in \Omega\) we have a constant probability \(\mathbb{P}(\omega_i) = \frac{1}{6} \; \; \forall i\). Consider the events \(A_1 = \{1,2,3,4\}\) and \(A_2 = A_3 = \{4,5,6\}\), are these events independent? Note that the events have probabilities \(\mathbb{P}(A_1) = \frac{2}{3}\), \(\mathbb{P}(A_2) = \mathbb{P}(A_3) =\frac{1}{2}\). Consider all the events \(A_1, A_2, A_3\), then the intersection of those sets gives \([A_1 {\color{blue}{\cap}} A_2 {\color{blue}{\cap}} A_3] = \{4\}\) that has probability \(\mathbb{P}(\{4\}) = \frac{1}{6}\). Then we can compute the product of the probabilities of the single events: \[ \frac{1}{6}= \mathbb{P}([A_1 {\color{blue}{\cap}} A_2 {\color{blue}{\cap}} A_3]) = \mathbb{P}(A_1) \mathbb{P}(A_2) \mathbb{P}(A_3) = \frac{2}{3} \frac{1}{2} \frac{1}{2} = \frac{1}{6} \text{.} \] Hence the events \(A_1 {\color{blue}{\cap}} A_2 {\color{blue}{\cap}} A_3\) are independent. Consider now only the events \(A_2 {\color{blue}{\cap}} A_3\), the probability of the joint set, namely \([ A_2 {\color{blue}{\cap}} A_3] = \{4, 5, 6\}\), is \(\mathbb{P}(\{4, 5, 6\}) = \frac{1}{2}\). However the product of the probabilities of the single events gives a different result: \[ \frac{1}{2}= \mathbb{P}([A_2 {\color{blue}{\cap}} A_3]) \neq \mathbb{P}(A_2) \mathbb{P}(A_3) = \frac{1}{2} \frac{1}{2} = \frac{1}{4} \text{.} \] Thus, the events \(A_2\) and \(A_3\) are NOT independent.

Proposition 4.1 (\(\color{magenta}{\textbf{Independence and complementation}}\))
If two events \(A\) and \(B\) are independent, then also \(A\) and \(B^{\mathsf{c}}\), \(B\) and \(A^{\mathsf{c}}\), \(A^{\mathsf{c}}\) and \(B^{\mathsf{c}}\) are independent.

Proof. In order to prove that \(A\) and \(B^{\mathsf{c}}\) are independent let’s write the event \(A\) as union of disjoint events (Equation 1.9). Then since \(A\) and \(B\) are assumed to be independent: \[ \begin{aligned} \mathbb{P}(A) & {} = \mathbb{P}(A {\color{blue}{\cap}} B) + \mathbb{P}(A{\color{blue}{\cap}} B^{\mathsf{c}}) = \\ & = \mathbb{P}(A)\mathbb{P}(B) + \mathbb{P}(A{\color{blue}{\cap}} B^{\mathsf{c}}) \end{aligned} \] Recovering \(\mathbb{P}(A{\color{blue}{\cap}} B^{\mathsf{c}})\) one obtain: \[ \begin{aligned} \mathbb{P}(A{\color{blue}{\cap}} B^{\mathsf{c}}) & {} = \mathbb{P}(A) - \mathbb{P}(A)\mathbb{P}(B) = \\ & = \mathbb{P}(A) (1-\mathbb{P}(B)) = \\ & = \mathbb{P}(A) \mathbb{P}(B^{\mathsf{c}}) \end{aligned} \] The same follows for \(A^{\mathsf{c}}\) and \(B\). Now let’s consider the case of \(A^{\mathsf{c}}\) and \(B^{\mathsf{c}}\). Using the same trick done previously \(A^{\mathsf{c}} = [A^{\mathsf{c}} {\color{blue}{\cap}} B]{\color{red}{\sqcup}} [A^{\mathsf{c}}{\color{blue}{\cap}} B^{\mathsf{c}}]\). Since we have already proven that \(B^{\mathsf{c}}\) and \(A\) are independent, we can write: \[ \begin{aligned} \mathbb{P}(A^{\mathsf{c}}) & {} = \mathbb{P}([A^{\mathsf{c}} {\color{blue}{\cap}} B]{\color{red}{\sqcup}} [A^{\mathsf{c}}{\color{blue}{\cap}} B^{\mathsf{c}}]) = \\ & = \mathbb{P}(A^{\mathsf{c}} {\color{blue}{\cap}} B) + \mathbb{P}(A^{\mathsf{c}} {\color{blue}{\cap}} B^{\mathsf{c}}) = \\ & = \mathbb{P}(A^{\mathsf{c}})\mathbb{P}(B) + \mathbb{P}(A^{\mathsf{c}} {\color{blue}{\cap}} B^{\mathsf{c}}) \end{aligned} \] Recovering \(\mathbb{P}(A^{\mathsf{c}} {\color{blue}{\cap}} B^{\mathsf{c}})\) gives the result, i.e. \[ \begin{aligned} \mathbb{P}(A^{\mathsf{c}}{\color{blue}{\cap}} B^{\mathsf{c}}) & {} = \mathbb{P}(A^{\mathsf{c}}) - \mathbb{P}(A^{\mathsf{c}})\mathbb{P}(B) = \\ & = \mathbb{P}(A^{\mathsf{c}}) (1-\mathbb{P}(B)) = \\ & = \mathbb{P}(A^{\mathsf{c}}) \mathbb{P}(B^{\mathsf{c}}) \end{aligned} \]

Definition 4.2 (\(\color{magenta}{\textbf{Independent classes}}\))
Let’s consider a probability space \((\Omega, \mathcal{B}, \mathbb{P})\) and let \(\mathcal{C}_i \in \mathcal{B}\) with \(i = 1, \dots, n\) be a classes of independent events. Then, for any choice of \(A_1, \dots, A_n\) with \(A_i \in \mathcal{C}_i\) and \(i = 1, \dots, n\), we have that the events \(A_1, \dots, A_n\) are independent according to the Definition 4.1.

4.1 Independent random variables

Theorem 4.1 (\(\color{magenta}{\textbf{Basic Criterion}}\))
Let’s consider a probability space \((\Omega, \mathcal{B}, \mathbb{P})\) and let \(\mathcal{C}_i\) with \(i = 1, \dots, n\) be a non-empty class of events such that

  1. \(\mathcal{C}_i\) is a \(\pi\)-system (see here);
  2. \(\mathcal{C}_i\) with \(i = 1, \dots, n\) are independent according to Definition 4.2.

Then, the \(\sigma\)-fields \(\sigma(\mathcal{C}_1), \dots \sigma(\mathcal{C}_n)\) are independent.

Definition 4.3 (\(\color{magenta}{\textbf{Independent random variables}}\))
Let’s consider a probability space \((\Omega, \mathcal{B}, \mathbb{P})\) and let \(\{X_t\}_{t \in \mathcal{T}}\) be a family of independent random variables. Then, the family of sigma fields \[ \{\sigma(X_t)\}_{t \in \mathcal{T}} \text{,} \] are independent.

The result in Theorem 4.1 is very important to prove the independence between two or more random variables. In fact, according to Definition 4.3, to prove independence we should verify that all the \(\sigma\)-fields generated by \(X_t\) are independent. If one is able to prove that there exists a class of independent events \(\mathcal{C}_t\) that generate \(\sigma(X_t)\) and that is a \(\pi\)-system, then one can immediately conclude that the random variables are independent.

4.2 Independence theorems

Theorem 4.2 (\(\color{magenta}{\textbf{Borel-Cantelli Lemma}}\))
Let \(\{A_n\}\) be a sequence of any events. If \[ \sum_{n} \mathbb{P}(A_n) < \infty \text{,} \] then \[ \mathbb{P}(A_n, \text{ i.o.}) = \mathbb{P}(\underset{n\to\infty}{\text{limsup}} A_n) = 0 \text{.} \]

Proof. To prove the Borel-Cantelli lemma, let’s start by the definition, i.e. \[ [A_n, \text{ i.o.}] = \underset{n\to\infty}{\text{limsup}} A_n = \bigcap_{n=1}^{\infty} \bigcup_{k=n}^{\infty} A_k \text{.} \] Then, let’s define \(B_n = \bigcup_{k=n}^{\infty} A_k\), such that \(B_1 \subset B_2 \subset \dots\) is an increasing sequence of events. Therefore, applying the probability on both sides \[ \begin{aligned} {} & \underset{n\to\infty}{\text{limsup}} A_n = \lim_{n\to \infty} \uparrow B_n \\ & \implies \mathbb{P}(\underset{n\to\infty}{\text{limsup}} A_n) = \mathbb{P}(\lim_{n\to \infty} \uparrow B_n) = \lim_{n\to \infty} \uparrow \mathbb{P}(B_n) \end{aligned} \] Then, assuming \(\sum_{n} \mathbb{P}(A_n) < \infty\) and recalling the definition of \(B_n\) and of sub-additivity (Equation 2.1) one obtain the following inequality: \[ \lim_{n\to \infty} \uparrow \mathbb{P}(B_n) \le \lim_{n\to \infty} \uparrow \sum_{k = n}^{\infty} \mathbb{P}(B_k) = 0 \text{.} \] Note that the result follows from the assumption of convergence of the series \(\sum_{n = 1}^{\infty} \mathbb{P}(A_n) < \infty\). In fact if a series converges, this implies that from a certain \(n\) on wards \(\mathbb{P}(A_n) = 0\), that is equivalent to say \[ \sum_{n = 1}^{\infty} \mathbb{P}(A_n) < \infty \iff \lim_{N \to \infty} \sum_{k = n}^{N} \mathbb{P}(A_k) = 0 \text{.} \]

Proposition 4.2 (\(\color{magenta}{\textbf{Borel-Zero-One Law}}\))
Let \(\{A_n\}\) be a sequence of independent events, then \[ \mathbb{P}(A_n, \text{ i.o.}) = \begin{cases} 0 & \text{iff} \; \sum_{n} \mathbb{P}(A_n) < \infty \\ 1 & \text{iff} \; \sum_{n} \mathbb{P}(A_n) = \infty \end{cases} \]