1  Basic set theory

Reference: Chapter 1. Resnick (2005).

In probability, an event is interpreted as a collection of possible outcomes of a random experiment.

Definition 1.1 ($\mathbf{\text{Random experiment}}$)
A random experiment is any repeatable procedure that results in one out of a well-defined set of possible outcomes.

• The set of possible outcomes is called sample space and denoted with $\mathrm{\Omega }$. An element of $\mathrm{\Omega }$ is denoted as $\omega$ and is an outcome
• A set of zero or more outcomes $\omega$ is an event $A$, where we denote with $\mathcal{F}$ a class of subsets $A\subset \mathrm{\Omega }$.
• A map (function) that goes from the space of the events $\mathcal{F}$ to the space of probabilities (real numbers in $[0,1]$) is the probability law and is denoted with $\mathbb{P}$.

Together, sample space, event space and probability law characterize a random experiment.

There are several definitions related to sets and their operation.

Definition 1.2 ($\mathbf{\text{Complementation}}$)
The complement of a set $A$ is denoted by $A^{\mathsf{c}}$ and represents the set of elements that do not belong to $A$, i.e. $$\begin{array}{}\text{(1.1)}& A^{\mathsf{c}}=\{\omega \in \mathrm{\Omega }:\omega \notin A\}\text{.}\end{array}$$

Definition 1.3 ($\mathbf{\text{Containment}}$)
A set $A$ is said to be contained in a set $B$ if every element of $A$ is also an element of $B$. Formally, $$\begin{array}{}\text{(1.2)}& A\subset B⟺\omega \in A⟹\omega \in B\mathrm{\forall }\omega \in \mathrm{\Omega }\text{.}\end{array}$$

Definition 1.4 ($\mathbf{\text{Equality}}$)
Given two sets, $A$ is equal to $B$, written $A=B$, if and only if every element of $A$ is an element of $B$ and every element of $B$ is an element of $A$. Formally, $$\begin{array}{}\text{(1.3)}& A\subset B\text{and}B\subset A\text{.}\end{array}$$

1.1 Set operations

Let’s now state some elementary operations between sets.

Figure 1.1: Elementary set operations.

Definition 1.5 ($\mathbf{\text{Union}}$)
The union of two sets, written $A\cup B$, is the set of $\omega$ that belongs either to $A$ or $B$, i.e. $$\begin{array}{}\text{(1.4)}& A\cup B=\{\omega \in \mathrm{\Omega }:\omega \in A\text{ or }\omega \in B\}\text{.}\end{array}$$

As a consequence of the definition of the union the following relations holds true, i.e.  $$\begin{array}{rlrl}& A\cup A=A& & A\cup \mathrm{\Omega }=\mathrm{\Omega }\\ & A\cup \mathrm{\varnothing }=A& & A\bigsqcup A^{\mathsf{c}}=\mathrm{\Omega }\end{array}$$

Definition 1.6 ($\mathbf{\text{Intersection}}$)
The intersection of $A$ and $B$ is written $A\cap B$ and is the set of elements that belongs at the same time to $A$ and $B$. $$A\cap B=\{\omega \in \mathrm{\Omega }:\omega \in A\text{ and }\omega \in B\}\text{.}$$

As a consequence of the definition of the intersection the following relations holds, i.e.  $$\begin{array}{rlrl}& A\cap A=A& & A\cap \mathrm{\Omega }=A\\ & A\cap \mathrm{\varnothing }=\mathrm{\varnothing }& & A\sqcap A^{\mathsf{c}}=\mathrm{\varnothing }\end{array}$$ Moreover, let’s state the distributive laws of the union and the intersection, i.e.  $$\begin{array}{}\text{(1.5)}& \begin{array}{rlrl}& \mathbf{Intersection}.& & (A\cup B)\cap C=(A\cap C)\cup (B\cap C)\\ & \mathbf{Union}.& & (A\cap B)\cup C=(A\cup C)\cap (B\cup C)\end{array}\end{array}$$ And the De Morgan’s laws: $$\begin{array}{}\text{(1.6)}& \begin{array}{rlrl}& \mathbf{Intersection}.& & (A\cap B{)}^{\mathsf{c}}=(A^{\mathsf{c}}\cup B^{\mathsf{c}})\\ & \mathbf{Union}.& & (A\cup B{)}^{\mathsf{c}}=(A^{\mathsf{c}}\cap B^{\mathsf{c}})\end{array}\end{array}$$

Definition 1.7 ($\mathbf{\text{Difference}}$)
The difference between two sets $A$ and $B$, written $A-B$ (or also $A/B$), is the set of elements of $A$ that do not belong to $B$. Formally $$\begin{array}{}\text{(1.7)}& A-B=A\cap B^{\mathsf{c}}=\{\omega \in \mathrm{\Omega }:\omega \in A\text{ and }\omega \notin B\}\text{.}\end{array}$$

Disjoint reppresentation of a set

Given two set $A$ and $B$, then each one can be written as the union of disjoint sets. In fact, their union can be decomposed into the union of three disjoint sets, i.e.  $$\begin{array}{}\text{(1.8)}& A\bigsqcup B=(A\cap B)\bigsqcup (A\cap B^{\mathsf{c}})\bigsqcup (A^{\mathsf{c}}\cap B)\text{,}\end{array}$$ and therefore for example the set $A$ can be written as $$\begin{array}{}\text{(1.9)}& A=(A\cap B)\bigsqcup (A-B)=(A\cap B)\bigsqcup (A\cap B^{\mathsf{c}})\text{.}\end{array}$$

Definition 1.8 ($\mathbf{\text{Symmetric difference}}$)
The symmetric difference between two sets $A$ and $B$ is written $A\mathrm{\Delta }B$ and is the union of elements of $A$ that do not belong to $B$ and of elements of $B$ that do not belong to $A$, i.e. $$\begin{array}{rl}A\mathrm{\Delta }B& =(A-B)\bigsqcup (B-A)=\\ & =(A\cap B^{\mathsf{c}})\bigsqcup (A^{\mathsf{c}}\cap B)=\\ & =\{\omega :\omega \in A,\omega \notin B\}\bigsqcup \{\omega :\omega \in B,\omega \notin A\}\end{array}$$

Proposition 1.1 Given two set $A,B$, the symmetric difference can be written as $$A\mathrm{\Delta }B=(A\bigsqcup B)\cap (A^{\mathsf{c}}\bigsqcup B^{\mathsf{c}})\text{.}$$

Proof: Proposition 1.1

Proof. Let’s denote with $C=A^{\mathsf{c}}\cap B$, then apply the distributive law of the union twice (Equation 1.5) and develop the computations, i.e. $$\begin{array}{rl}A\mathrm{\Delta }B& =(A\cap B^{\mathsf{c}})\bigsqcup (A^{\mathsf{c}}\cap B)=\\ & =(A\cap B^{\mathsf{c}})\bigsqcup C=\\ & =(A\cup C)\cap (B^{\mathsf{c}}\cup C)=\\ & =[A\cup (A^{\mathsf{c}}\cap B)]\cap [B^{\mathsf{c}}\cup (A^{\mathsf{c}}\cap B)]=\\ & =[((A\bigsqcup A^{\mathsf{c}})\cap (A\cup B)]\cap (B^{\mathsf{c}}\cup A^{\mathsf{c}})\cap (B^{\mathsf{c}}\bigsqcup B)=\\ & =(A\cup B)\cap (A^{\mathsf{c}}\cup B^{\mathsf{c}})\end{array}$$

1.2 Indicator function

Definition 1.9 ($\mathbf{\text{Indicator function}}$)
An indicator function is a function that associate an $\omega \in A\subset \mathrm{\Omega }$ to a real number, i.e. either 0 or 1. It is a tool that allows to transfer a computation from the set domain into the real numbers domain, i.e. $\{0,1\}$. Formally, ${\mathbb{1}}_A(\omega ):\mathrm{\Omega }\to \{0,1\}$, i.e.  $${\mathbb{1}}_A(\omega )=\{\begin{array}{l}1\omega \in A\\ 0\omega \in A^{\mathsf{c}}\end{array}\text{.}$$

Indicator function in $\mathbb{R}$

Remark 1.1. If $x\in \mathbb{R}$, then the equivalent operator an indicator function is the heavyside function in a point $a\in \mathbb{R}$ (see here), i.e. $$\begin{array}{}\text{(1.10)}& H_a(x)=H(x-a)=\{\begin{array}{ll}0& x<a\\ 1& x\ge a\end{array}\text{.}\end{array}$$ The first derivative of the heavyside with respect to $x$ is the dirac delta function (see here), i.e. $$\begin{array}{}\text{(1.11)}& \frac{d}{dx}{H}_a(x)={\delta }_a(x)=\delta (x-a)=\{\begin{array}{ll}1& x=a\\ 0& \text{othewise}\end{array}\text{.}\end{array}$$ A fundamental property of the dirac delta is that for a general function $f$ Thanks to a property of the diract function, i.e.  $$\begin{array}{}\text{(1.12)}& {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(y)\delta (y-a)dy=f(a)\text{.}\end{array}$$

Proposition 1.2 The containment between two sets can be equivalently written in terms of indicator functions: $$A\subset B⟺{\mathbb{1}}_A(\omega )\le {\mathbb{1}}_B(\omega ),\mathrm{\forall }\omega \in \mathrm{\Omega }\text{.}$$

Proof: Proposition 1.2

Proof. Let’s start by assuming $A\subset B$ and let’s distinguish two main cases.

1. Assuming $\omega \in A$ implies that $\omega \in B$, and therefore one have an equality $1={\mathbb{1}}_A\le {\mathbb{1}}_B=1$.
2. Assuming $\omega \in A^{\mathsf{c}}$ implies $[\omega \in B]\cup [\omega \in B^{\mathsf{c}}]$. In this situation for both cases one will have that ${\mathbb{1}}_A\le {\mathbb{1}}_B$, in fact:
   • Considering $\omega \in B$ implies that $0={\mathbb{1}}_A<{\mathbb{1}}_B=1$.
   • Considering $\omega \in B^{\mathsf{c}}$ implies that $0={\mathbb{1}}_A\le {\mathbb{1}}_B=0$.

Hence, assuming $A\subset B$ implies that ${\mathbb{1}}_A(\omega )\le {\mathbb{1}}_B(\omega )$ for all $\omega \in \mathrm{\Omega }$. Now let’s assume the contrary: ${\mathbb{1}}_A\le {\mathbb{1}}_B$ and let’s again distinguish in two main cases:

1. Assuming $\omega \in A$, i.e. ${\mathbb{1}}_A=1$, the inequality ${\mathbb{1}}_A\le {\mathbb{1}}_B$ holds and since the indicator function is bounded by 1 by definition it is possible to write $1={\mathbb{1}}_A\le {\mathbb{1}}_B\le 1$. Therefore, one obtain ${\mathbb{1}}_B=1$ and so $\omega \in B$.
2. Assuming $\omega \in A^{\mathsf{c}}$, i.e. ${\mathbb{1}}_A=0$, the inequality ${\mathbb{1}}_A\le {\mathbb{1}}_B$ holds and it is possible to write $0={\mathbb{1}}_A\le {\mathbb{1}}_B\le 1$. Hence, when $\omega \in A^{\mathsf{c}}$, there are two possible cases, i.e. 
   • ${\mathbb{1}}_B=1$, but this implies that $\omega \in B$.
   • ${\mathbb{1}}_B=0$, but this implies that $\omega \in B^{\mathsf{c}}$.

When an $\omega \in A$ implies that $\omega \in B$, but the contrary do not holds true. Hence, it is possible to conclude that $A\subset B$.

1.3 Limits of sets

Let’s define the infimum ($inf$) and the supremum ($sup$) of a sequence of sets $\{A_n{\}}_{n\ge 1}$ as $$\underset{k\ge n}{inf}=\bigcap _{k=n}^{\mathrm{\infty }}{A}_k\underset{k\ge n}{sup}=\bigcup _{k=n}^{\mathrm{\infty }}{A}_k\text{,}$$ Informally, the infimum of a sequence of sets is the smallest set in $k=n,\dots ,\mathrm{\infty }$, on the other hand, the supremum of a sequence of sets is the biggest set in $k=n,\dots ,\mathrm{\infty }$.

Then, the $liminf$ is defined as $$\underset{n\to \mathrm{\infty }}{lim}inf{A}_n=\underset{n\ge 1}{sup}\underset{k\ge n}{inf}{A}_k=\bigcup _{n=1}^{\mathrm{\infty }}\bigcap _{k=n}^{\mathrm{\infty }}{A}_k\text{,}$$ The liminf (limit inferior of sets) is the set of all elements that eventually always belong to the sequence $\{A_n{\}}_{n\ge 1}$. By definition, the limit of the infimum ($liminf$) is the biggest (union) among all the smallest (intersection) sets. In other words, $x\in lim inf{A}_n$ if there exists some index $N$ such that for all $n\ge N$, we have $x\in A_n$.

Instead, the $limsup$ is defined as $$\underset{n\to \mathrm{\infty }}{lim}sup{A}_n=\underset{n\ge 1}{inf}\underset{k\ge n}{sup}{A}_k=\bigcap _{n=1}^{\mathrm{\infty }}\bigcup _{k=n}^{\mathrm{\infty }}{A}_k\text{.}$$ The limsup (limit superior of sets) is the set of all elements that belong infinitely often to the sequence $\{A_n{\}}_{n\ge 1}$. By definition, the limit of the supremum ($limsup$) is the smallest (intersection) among all the biggest (union) sets. In other words, $x\in lim sup{A}_n$ if for infinitely many $n$, $x\in A_n$.

Remark 1.2. By De Morgan’s laws (Equation 1.6): $$(\underset{n\to \mathrm{\infty }}{lim}sup{A}_n{)}^{\mathsf{c}}={\left(\bigcap _{n=1}^{\mathrm{\infty }}\bigcup _{k=n}^{\mathrm{\infty }}{A}_k\right)}^{\mathsf{c}}=\bigcup _{n=1}^{\mathrm{\infty }}\bigcap _{k=n}^{\mathrm{\infty }}{A}_k^{\mathsf{c}}=\underset{n\to \mathrm{\infty }}{lim}inf{A}_n^{\mathsf{c}}\text{,}$$ and similarly $$(\underset{n\to \mathrm{\infty }}{lim}inf{A}_n{)}^{\mathsf{c}}={\left(\bigcup _{n=1}^{\mathrm{\infty }}\bigcap _{k=n}^{\mathrm{\infty }}{A}_k\right)}^{\mathsf{c}}=\bigcap _{n=1}^{\mathrm{\infty }}\bigcup _{k=n}^{\mathrm{\infty }}{A}_k^c=\underset{n\to \mathrm{\infty }}{lim}sup{A}_n^c\text{.}$$

1.4 Monotone Sequences

Let’s define a sequence of sets $\{A_n{\}}_{n\ge 1}$ as monotone non-decreasing if $A_1\subset A_2\subset \dots$, while we define it monotone non-increasing if $A_1\supset A_2\supset \dots$. In general, a monotone non-decreasing sequence is denoted with $A_n\uparrow$, while a non-increasing one with $A_n\downarrow$. In general, the limit of a monotone sequence always exists.

Definition 1.10 ($\mathbf{\text{Limit of Monotone Sequences}}$)
Let’s consider a monotone sequence of sets $\{A_n\}$. Then, if

1. $A_n\uparrow$ is a sequence of non-decreasing sets, i.e. $A_1\subset A_2\subset \dots$, then the limit exists, i.e.  $$A_n\uparrow ⟹\underset{n\to \mathrm{\infty }}{lim}{A}_n=\bigcup _{n=1}^{\mathrm{\infty }}{A}_n\text{.}$$

2. $A_n\downarrow$ is a sequence of non-increasing sets, i.e. $A_1\supset A_2\supset \dots$, then the limit exists, i.e.  $$A_n\downarrow ⟹\underset{n\to \mathrm{\infty }}{lim}{A}_n=\bigcap _{n=1}^{\mathrm{\infty }}{A}_n\text{.}$$

1.5 Fields and $\sigma$-fields

Definition 1.11 ($\mathbf{\text{Field}}$)
Let’s define a field $\mathcal{A}$ as a non-empty class of subsets of $\mathrm{\Omega }$ closed under complementation, finite union and intersection. The minimal requirements for a class of subsets to be a field are:

1. $\mathrm{\Omega }\in \mathcal{A}$, i.e. the sample space if in $\mathcal{A}$.
2. $A\in \mathcal{A}⟹A^{\mathsf{c}}\in \mathcal{A}$, i.e. if a set $A$ is in $\mathcal{A}$, then also its complement is in $\mathcal{A}$.
3. $A,B\in \mathcal{A}⟹A\cup B\in \mathcal{A}$, i.e. if two sets $A$ and $B$ are in $\mathcal{A}$, then also their union (or intersection) is in $\mathcal{A}$.

Definition 1.12 ($\sigma \mathbf{\text{-field}}$)
Let’s define a $\sigma$-field $\mathcal{B}$ as a non-empty class of subsets of $\mathrm{\Omega }$ closed under complementation, countable union and intersection. The minimal requirements for a class of subsets to be a $\sigma$-field are:

1. $\mathrm{\Omega }\in \mathcal{B}$, i.e. the sample space if in $\mathcal{B}$.
2. $B\in \mathcal{B}⟹B^{\mathsf{c}}\in \mathcal{B}$, i.e. if a set $B$ is in $\mathcal{B}$, then also its complement is in $\mathcal{B}$.
3. $B_i\in \mathcal{B},i\ge 1⟹\bigcup _{n\ge 1}{B}_n\in \mathcal{B}$, i.e. if a countable sequence of sets $B_i$ is in $\mathcal{B}$, then also its countable union (or intersection) is in $\mathcal{B}$.

Field vs $\sigma$-Field

The main difference between a field and a $\sigma$-field is in the third property of the definitions. A field is closed under finite union, namely the union of a finite sequence of events $A_n$ indexed by $n\in \{0,1,2,\dots ,n\}$ (property 3 of Definition 1.11). On the other hand, a $\sigma$-field is closed under countable union, namely the union of an infinite sequence of events $A_n$ indexed by $n\in \{0,1,2,\dots ,n,n+1,\dots \}$ (property 3. of the Definition 1.12).

Exercise 1.1 Let the sample space be ${\mathrm{\Omega }}^{\prime }=\{-1,0,1\}$. Generate a $\sigma$-algebra according to Definition 1.12

Solution: Exercise 1.1

Solution 1.1. Let the sample space be ${\mathrm{\Omega }}^{\prime }=\{-1,0,1\}$. A natural choice of $\sigma$–algebra on a finite set is the power set: $$\mathcal{P}({\mathrm{\Omega }}^{\prime })=\{\varnothing ,\{-1\},\{0\},\{1\},\{-1,0\},\{-1,1\},\{0,1\},{\mathrm{\Omega }}^{\prime }\}.$$

Resnick, Sidney I. 2005. A Probability Path. Birkhauser. https://link.springer.com/book/10.1007/978-0-8176-8409-9.