8  Convergence concepts

Let’s consider a sequence of real number, say $a_n$, then stating that the associated series converges, formally $\sum _{k=1}^{\mathrm{\infty }}{\alpha }_k<\mathrm{\infty }$, implies that from a certain $k$ awards $a_k=0$, i.e.  $$\sum _{k=1}^{\mathrm{\infty }}{\alpha }_k<\mathrm{\infty }⟺\underset{N\to \mathrm{\infty }}{lim}\sum _{k=n}^N{a}_k=0$$

8.1 Types of convergence

Definition 8.1 ($\mathbf{\text{Pointwise}}$)
A sequence of random variables $\{X_n{\}}_{n\ge 1}$ is said to be convergent point wise to a limit $X$ iff for all $\omega \in \mathrm{\Omega }$: $$X_n(\omega )\underset{n\to \mathrm{\infty }}{⟶}X(\omega )⟺\underset{n\to \mathrm{\infty }}{lim}{X}_n(\omega )=X(\omega )$$ This kind of definition requires that convergence happen for every $\omega \in \mathrm{\Omega }$.

Definition 8.2 ($\mathbf{\text{Almost Surely}}$)
A sequence of random variables $\{X_n{\}}_{n\ge 1}$ is said to be convergent almost surely to a limit $X$ iff: $$\mathbb{P}\{\omega \in \mathrm{\Omega }:\underset{n\to \mathrm{\infty }}{lim}{X}_n(\omega )=X(\omega )\}=1$$ Usually, such kind of convergence is denoted as: $$X_n(\omega )\stackrel{\text{a.s.}}{\underset{n\to \mathrm{\infty }}{⟶}}X(\omega )$$

In other terms, an almost surely convergence implies the relation must holds for all $\omega \in \mathrm{\Omega }$ with the exception of some $\omega$’s, that are in $\mathrm{\Omega }$, but whose probability of occurrence is zero.

Definition 8.3 ($\mathbf{\text{In Probability}}$)
A sequence of random variables $\{X_n{\}}_{n\ge 1}$ is said to be convergent in probability to a limit $X$ if, for a fixed $ϵ>0$: $$\underset{n\to \mathrm{\infty }}{lim}\mathbb{P}\{\omega \in \mathrm{\Omega }:|X_n(\omega )-X(\omega )|>ϵ\}=0$$ Usually, such kind of convergence is denoted as: $$X_n(\omega )\stackrel{\text{p}}{\underset{n\to \mathrm{\infty }}{⟶}}X(\omega )$$

Definition 8.4 ($L_p$)
A sequence of events $X_n$ such that: $$\mathbb{E}\{|X_n{|}^p\}<\mathrm{\infty },\mathbb{E}\{|X{|}^p\}<\mathrm{\infty }\text{,}$$ is said to be convergent in $L_p$, with $p>0$, to a random variable $X$ iff $$X_n(\omega )\stackrel{L_p}{\underset{n\to \mathrm{\infty }}{⟶}}X(\omega )⟺\underset{n\to \mathrm{\infty }}{lim}\mathbb{E}\{|X_n-X{|}^p\}=0$$ Usually, such kind of convergence is denoted as: $$X_n\underset{n\to \mathrm{\infty }}{\stackrel{L_p}{⟶}}X$$

Note that, it can be proved that there is no relation between almost sure convergence and $L_p$ convergence, i.e. one do not imply the other and viceversa. However, a convergence in a bigger space, say $q>s$ implies the convergence in the smaller space, i.e.  $$X_n\underset{n\to \mathrm{\infty }}{\stackrel{L_q}{⟶}}X⟹X_n\underset{n\to \mathrm{\infty }}{\stackrel{L_p}{⟶}}X,0<p<q$$

Definition 8.5 ($\mathbf{\text{In Distribution}}$)

A sequence of random variables $X_n$ is said to be convergent in distribution to a random variable $X$ if the distribution of ${\mathbb{F}}_{X_n}$ to ${\mathbb{F}}_X$ , i.e.  $$\underset{n\to \mathrm{\infty }}{lim}{\mathbb{F}}_{X_n}(x)={\mathbb{F}}_X(x)\mathrm{\forall }x$$ where $x$ is a continuity point of $\mathbb{F}$. Usually, such kind of convergence is denoted as: $$X_n(\omega )\stackrel{\text{d}}{\underset{n\to \mathrm{\infty }}{⟶}}X(\omega )$$

In other terms, we have convergence $\mathbf{\text{in distribution}}$ if the distribution of $X_n$, namely ${\mathbb{F}}_{X_n}$, converges as $n\to \mathrm{\infty }$ to the distribution of $X$, namely ${\mathbb{F}}_X$. Note that the convergence in distribution is not related with probability space but involves only the distribution functions.

8.2 Laws of Large Numbers

There are many versions of laws of large numbers (LLN). In general, a sequence $\{X_n{\}}_{n\ge 1}$ is said to satisfy a LLN iff: $${\overline{X}}_n=\frac{S_n}{n}=\frac{1}{n}\sum _{i=1}^n{X}_i⟶X$$

Strong vs weak laws of large numbers

In general, if convergence happens almost surely (Definition 8.2) we speak about strong laws of large numbers (SLLN). Otherwise, if convergence happens in probability we speak about weak laws of large numbers (WLLN). A crucial difference to be noted is that when convergence happens almost surely we are dealing with a limit of a sequence of sets (limit is inside $\mathbb{P}$), instead if convergence happens in probability we are dealing with a limit of a sequence of real numbers in $[0,1]$ (limit is outside $\mathbb{P}$).

8.2.1 Strong Laws of Large Numbers

Definition 8.6 ($\mathbf{\text{Kolmogorov SLLN}}$)
Let’s consider a sequence of IID random variables $\{X_n{\}}_{n\ge 1}$. Then, there exist a constant $c\in \mathbb{R}$ such that: $${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{S_n}{n}\stackrel{\text{a.s.}}{\underset{n\to \mathrm{\infty }}{⟶}}c$$ Then, if $\mathbb{E}\{|X_1|\}<\mathrm{\infty }$ in which case $c=\mathbb{E}\{|X_1|\}$.

Definition 8.7 ($\mathbf{\text{SLLN without independence}}$)
Let’s consider a sequence of identically distributed random variables $\{X_n{\}}_{n\ge 1}$, i.e. $\mathbb{E}\{X_n\}=\mathbb{E}\{X_1\}$ for all $n$, such that:

1. $\mathbb{E}\{X^2\}<c$ where $c>0$ is a constant independent from $n$.
2. $\mathbb{C}v\{X_i,X_j\}=0\mathrm{\forall }i\ne j$.

$${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{S_n}{n}\stackrel{\text{a.s.}}{\underset{n\to \mathrm{\infty }}{⟶}}\mathbb{E}\{X_1\}$$

Note that the existence of the first moment and the fact that it is finite, i.e. $\mathbb{E}\{X_1\}<\mathrm{\infty }$, implies that there exists the characteristic function of the random variable in zero, i.e. $\mathrm{\exists }{\varphi }_{X_1}^{\prime }(0)$. On the other hand, the existence of the characteristic function in zero do not ensure that the first moment is finite.

8.2.2 Weak Laws of Large Numbers

Let’s repeat a random experiment many times, every time ensuring the same conditions in such a way that the sequence of the experiment are IID. Then, each random variable $X_i$ comes from the same population with a unknown mean $\mathbb{E}\{X\}$ and variance $\mathbb{V}\{X\}$. Thanks to the WLLN and repeating the experiment many times we have that the sample mean of the experiment converges in probability to the true mean in population. Convergence in probability means that: $$\underset{n\to \mathrm{\infty }}{lim}\mathbb{P}\{\omega \in \mathrm{\Omega }:|\frac{1}{n}\sum _{i=1}^n{X}_i(\omega )-\mathbb{E}\{X(\omega )\}|>ϵ\}=0$$

Definition 8.8 ($\mathbf{\text{WLLN with variances}}$)
Given a sequence of independent and identically distributed random variables $\{X_n{\}}_{n\ge 1}$ such that:

1. $\mathbb{E}\{X_1\}=\mu$.
2. $\mathbb{E}\{X_1^2\}<\mathrm{\infty }$.

$${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{S_n}{n}\stackrel{\text{p}}{\underset{n\to \mathrm{\infty }}{⟶}}\mathbb{E}\{X_1\}=\mu$$

Proof WLLN with variances

Proof. Let’s consider the random variable ${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i$, then since by assumption the mean and variance are finite, let’s apply the Chebychev inequality (Equation 5.5), i.e.  $$\mathbb{P}(|{\overline{X}}_n-\mu |\ge \lambda )\le \frac{1}{{\lambda }^2}\mathbb{V}\{{\overline{X}}_n-\mu \}$$ Using a well known scaling property of variance let’s simplify it as: $$\begin{array}{rlrl}\mathbb{V}\{{\overline{X}}_n-\mu \}& =\mathbb{V}\{\frac{1}{n}\sum _{i=1}^n{X}_i-\mu \}=& & (\text{Constant})\\ & =\mathbb{V}\left\{\frac{1}{n}\sum _{i=1}^n{X}_i\right\}=& & (\text{Scaling})\\ & =\frac{1}{n^2}\mathbb{V}\left\{\sum _{i=1}^n{X}_i\right\}=& & (\text{Independence})\\ & =\frac{1}{n^2}\sum _{i=1}^n\mathbb{V}\{X_i\}=& & (\text{Identically distribution})\\ & =\frac{n{\sigma }^2}{n^2}=\frac{{\sigma }^2}{n}\end{array}$$ Therefore the Chebychev inequality became $$\mathbb{P}(|{\overline{X}}_n-\mu |\ge \lambda )\le \frac{{\sigma }^2}{n{\lambda }^2}$$ Taking the limit as $n\to \mathrm{\infty }$ proves the convergence in probability, i.e.  $$\underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(|{\overline{X}}_n-\mu |\ge \lambda )\le \underset{n\to \mathrm{\infty }}{lim}\frac{{\sigma }^2}{n{\lambda }^2}=0$$

Definition 8.9 ($\mathbf{\text{Khintchin’s WLLN under first moment hypothesis}}$)
Given a sequence of independent and identically distributed random variables $\{X_n{\}}_{n\ge 1}$ such that:

1. $\mathbb{E}\{X_1\}<\mathrm{\infty }$.
2. $\mathbb{E}\{X_n\}=\mu$.

$${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{S_n}{n}\stackrel{\text{p}}{\underset{n\to \mathrm{\infty }}{⟶}}\mathbb{E}\{X_1\}=\mu$$

Definition 8.10 ($\mathbf{\text{Feller’s WLLN without first moment}}$)
Given a sequence of independent and identically distributed random variables $\{X_n{\}}_{n\ge 1}$ such that: $$\underset{x\to \mathrm{\infty }}{lim}x\mathbb{P}\{|X_1|>x\}=0$$ then $${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{S_n}{n}\stackrel{\text{p}}{\underset{n\to \mathrm{\infty }}{⟶}}\mathbb{E}\{X_1{\mathbb{1}}_{[|X_1|\le n]}\}$$ Note that this result makes not assumptions about a finite first moment.

SLLN (without independence) implies WLLN

Let’s verity that under the assumptions of the SLLN without independence (Definition 8.7) we will always have convergence in probability, i.e. $${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i\stackrel{\text{p}}{\underset{n\to \mathrm{\infty }}{⟶}}\mathbb{E}\{X_1\}$$

Proof. Using Chebychev inequality (Equation 5.5), fix an $\epsilon >0$ such that: $$\mathbb{P}(|{\overline{X}}_n-\mathbb{E}\{X_1\}|>\epsilon )\le \frac{\mathbb{V}\{{\overline{X}}_n\}}{{\epsilon }^2}$$ Let’s explicit the computations, i.e.  $$\begin{array}{rl}\frac{\mathbb{V}\{{\overline{X}}_n\}}{{\epsilon }^2}& =\frac{1}{n^2{\epsilon }^2}\mathbb{V}\left\{\sum _{i=1}^n{X}_i\right\}=\\ & =\frac{1}{n^2{\epsilon }^2}[\sum _{i=1}^n\mathbb{V}\{X_i\}+\sum _{i=1}^n\sum _{j\ne i}^n\mathbb{C}v\{X_i,X_j\}]\end{array}$$ By assumption the covariances are zero $\mathbb{C}v\{X_i,X_j\}=0\mathrm{\forall }i\ne j$. Moreover, since $\mathbb{V}\{X_i\}=\mathbb{E}\{X_i^2\}-\mathbb{E}\{X_i{\}}^2$ it is possible to upper bound the variance with the second moment, namely $\mathbb{V}\{X_i\}\le \mathbb{E}\{X_i^2\}$, i.e.  $$\frac{1}{n^2{\epsilon }^2}\sum _{i=1}^n\mathbb{V}\{X_i\}\le \frac{1}{n^2{\epsilon }^2}\sum _{i=1}^n\mathbb{E}\{X_i^2\}$$ Since by the assumption of the SLLN we have that $\mathbb{E}\{X^2\}<c$ where $c>0$ is a constant independent from $n$ we can further upper bound the probability by: $$\frac{1}{n^2{\epsilon }^2}\sum _{i=1}^n\mathbb{E}\{X_i^2\}\le \frac{1}{n^2{\epsilon }^2}\sum _{i=1}^{n}c=\frac{nc}{n^2{\epsilon }^2}=\frac{c}{n{\epsilon }^2}$$ Finally if we take the limit for $n\to \mathrm{\infty }$ it is equal to zero implying convergence in probability: $$0\le \underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(|{\overline{X}}_n-\mathbb{E}\{X_1\}|>\epsilon )\le \underset{n\to \mathrm{\infty }}{lim}\frac{c}{n{\epsilon }^2}=0$$

8.3 Central Limit Theorem

Theorem 8.1 ($\mathbf{\text{Central Limit Theorem (CLT) - IID case}}$)
Let’s consider a sequence of $n$ random variables, $X_n=(X_1,\dots ,X_n)$, where each $X_i$ is independent and identically distributed (IID), i.e. $$\begin{array}{rl}{X}_i\sim \text{IID}(\mu ,{\sigma }^2)& ⟹\mathbb{E}\{X_i\}=\mathbb{E}\{X_1\}=\mu \\ & ⟹\mathbb{V}\{X_i\}=\mathbb{V}\{X_1\}={\sigma }^2\end{array}$$ Then, let’s define a random variable, namely $S_n$, given by the sum of all the $X_i$, i.e.
$$S_n=\sum _{i=1}^n{X}_i$$ It is easy to see that due to the fact that the random variables are IID the moments of $S_n$ are: $$\mathbb{E}\{S_n\}=n\mathbb{E}\{X_1\}=n\mu ,\mathbb{V}\{S_n\}=n\mathbb{V}\{X_1\}=n{\sigma }^2$$ Hence, the standardized variable $Z_n$ on large samples is normally distributed, i.e.
$$Z_n=\frac{S_n-\mathbb{E}\{S_n\}}{\sqrt{\mathbb{V}\{S_n\}}}=\frac{\sum _{i=1}^n{X}_i-n\mu }{\sqrt{n}\sigma }\stackrel{d}{\underset{n\to \mathrm{\infty }}{\sim }}\mathcal{N}(0,1)$$