5  Expectation

The expectation represents a central value of a random variable and has a measure theory counterpart as a Lebesgue-Stieltjes integral of $X$ with respect to a (probability) measure $\mathbb{P}$. This kind of integration is defined in steps. First it is shown the integration of simple functions and then extended to more general random variables. In general, let’s consider a probability space $(\mathrm{\Omega },\mathcal{B},\mathbb{P})$ and a random variable $X$ such that $$X:(\mathrm{\Omega },\mathcal{B})⟶(\overline{\mathbb{R}},\mathcal{B}(\overline{\mathbb{R}}))$$ where $\overline{\mathbb{R}}=[-\mathrm{\infty },\mathrm{\infty }]$. Then, the expectation of $X$ is denoted as: $$\mathbb{E}\{X\}={\int }_{\mathrm{\Omega }}Xd\mathbb{P}={\int }_{\mathrm{\Omega }}X(\omega )\mathbb{P}(d\omega )$$ as the Lebesgue-Stieltjes integral of $X$ with respect to the (probability) measure $\mathbb{P}$.

5.1 Simple functions

In general a random variable $X(\omega )$ is simple if it has a finite range. Let’s consider a probability space $(\mathrm{\Omega },\mathcal{B},\mathbb{P})$ and consider a $\mathcal{B}/\mathcal{B}(\mathbb{R})$-measurable simple function $X:\mathrm{\Omega }\to \mathbb{R}$, i.e.  $$\begin{array}{}\text{(5.1)}& X(\omega )=\sum _{i=1}^n{a}_i{\mathbb{1}}_{A_i}(\omega )\text{,}\end{array}$$ where $a_i\in \mathbb{R}$ and $A_i\in \mathcal{B}$ are a disjoint partition of the sample space, i.e. $\underset{i=1}{\overset{n}{⨆}}{A}_i=\mathrm{\Omega }$. Let’s denote the set of all simple functions on $\mathrm{\Omega }$ as $\mathcal{E}$. In this settings, $\mathcal{E}$ is a vector space. This implies that he following two properties holds.

1. Constant: given a simple function $X\in \mathcal{E}$, then $\alpha X\in \mathcal{E}$. In fact: $$\begin{array}{}\text{(5.2)}& \begin{array}{rl}\alpha X& =\sum _{i=1}^n\alpha a_i{\mathbb{1}}_{A_i}=\sum _{i=1}^n{a}_i^{\ast }{\mathbb{1}}_{A_i}\in \mathcal{E}\end{array}\end{array}$$ where $a_i^{\ast }=\alpha a_i$.

2. Linearity: given two simple function $X,Y\in \mathcal{E}$, then $X+Y\in \mathcal{E}$. In fact: $$\begin{array}{}\text{(5.3)}& \begin{array}{rl}X+Y& =\sum _{i=1}^n{a}_i{\mathbb{1}}_{A_i}+\sum _{j=1}^m{b}_j{\mathbb{1}}_{B_j}=\\ & =\sum _{i=1}^n\sum _{j=1}^m(a_i+b_j){\mathbb{1}}_{A_i}{\mathbb{1}}_{B_j}=\\ & =\sum _{i=1}^n\sum _{j=1}^m(a_i+b_j){\mathbb{1}}_{A_i\cap B_j}\end{array}\end{array}$$ where the sequence of sets $\{A_i{B}_{j}1\le i\le n\text{and}1\le j\le m\}$ form a disjoint partition of $\mathrm{\Omega }$.

3. Product: given two simple function $X,Y\in \mathcal{E}$, then $XY\in \mathcal{E}$. In fact: $$\begin{array}{}\text{(5.4)}& \begin{array}{rl}XY& =\sum _{i=1}^n{a}_i{\mathbb{1}}_{A_i}\sum _{j=1}^m{b}_j{\mathbb{1}}_{B_j}=\\ & =\sum _{i=1}^n\sum _{j=1}^m(a_i{b}_j){\mathbb{1}}_{A_i}{\mathbb{1}}_{B_j}=\\ & =\sum _{i=1}^n\sum _{j=1}^m(a_i{b}_j){\mathbb{1}}_{A_i\cap B_j}\end{array}\end{array}$$

5.1.1 Measurability

Simple functions are the building blocks in the definition of the expectation in terms of Lebesgue-Stieltjes integral. In fact a known theorem called Measurability theorem shows that any measurable function can be approximated by a sequence of simple functions.

Theorem 5.1 Suppose that $X(\omega )\ge 0$ for all $\omega \in \mathrm{\Omega }$. Then, $X$ is $\mathcal{B}/\mathcal{B}(\mathbb{R})$ measurable if and only if there exists simple functions $X_n\in \mathcal{E}$ and $$0\le X_n\uparrow X⟺X=\underset{n\to \mathrm{\infty }}{lim}\uparrow X_n$$

5.2 Expectation of Simple Functions

The expectation of a simple function $X$ is defined as: $$\mathbb{E}\{X\}=\sum _{i=1}^n{a}_i\mathbb{P}(A_i)$$ where $|a_i|<\mathrm{\infty }$.

5.2.1 Properties

1. Non-negativity: If $X\ge 0$ and $X\in \mathcal{E}$ then $\mathbb{E}\{X\}\ge 0$

Expectation of a simple function is non-negative

Proof. By definition of simple functions

2. Linearity: the expectation of simple function is linear, i.e.  $$\mathbb{E}\{\alpha X+\beta Y\}=\alpha \mathbb{E}\{X\}+\beta \mathbb{E}\{Y\}$$

Expectation of a simple function is linear

Proof. Let’s consider two simple functions, i.e.  $$X(\omega )=\sum _{i=1}^n{a}_i{\mathbb{1}}_{A_i}(\omega )\text{and}Y(\omega )=\sum _{j=1}^m{b}_j{\mathbb{1}}_{B_j}(\omega )\text{,}$$ and let’s fix $\alpha ,\beta \in \mathbb{R}$. Then, by the second property of the vector space $\mathcal{E}$ (Equation 5.3) it is possible to write: $$\alpha X+\beta Y=\sum _{i=1}^n\sum _{j=1}^m(\alpha a_i+\beta b_j){\mathbb{1}}_{A_i\cap B_j}$$ Then, taking the expectation on both sides:
$$\begin{array}{rl}\mathbb{E}\{\alpha X+\beta Y\}& =\sum _{i=1}^n\sum _{j=1}^m(\alpha a_i+\beta b_j)\mathbb{P}(A_i\cap B_j)=\\ & =\sum _{i=1}^n\alpha a_i\sum _{j=1}^m\mathbb{P}(A_i\cap B_j)+\sum _{j=1}^m\beta b_j\sum _{i=1}^n\mathbb{P}(A_i\cap B_j)\end{array}$$ Fixing $i$, the sequence $A_i\cap B_j$ for $j=1,\dots ,n$ is composed by disjoint events since by definition $B_j$ are disjoint. Hence, applying $\sigma$-additivity it is possible to write: $$\begin{array}{rl}\sum _{j=1}^m\mathbb{P}(A_i\cap B_j)& =\mathbb{P}\left(\underset{j=1}{\overset{m}{⨆}}{A}_i\cap B_j\right)=\\ & =\mathbb{P}\left(A_i\cap \left(\underset{j=1}{\overset{m}{⨆}}{B}_j\right)\right)=\\ & =\mathbb{P}\left(A_i\cap \mathrm{\Omega }\right)=\mathbb{P}\left(A_i\right)\end{array}$$ Therefore, the expectation simplifies in: $$\begin{array}{rl}\mathbb{E}\{\alpha X+\beta Y\}& =\sum _{i=1}^n\alpha a_i\mathbb{P}(A_i)+\sum _{j=1}^m\beta b_j\mathbb{P}(B_j)=\\ & =\alpha \mathbb{E}\{X\}+\beta \mathbb{E}\{Y\}\end{array}$$

5.3 Review of inequalities

5.3.1 Modulus inequality

Definition 5.1 ($\mathbf{\text{Modulus Inequality}}$)
Let’s consider a random variable $X\in {\mathcal{L}}_1$, where ${\mathcal{L}}_1$ stands for the set of integrable random variables, i.e.  $${\mathcal{L}}_1=\{X:\mathrm{\Omega }\to \mathbb{R}:X\text{ is a r.v. },\mathbb{E}\{|X|\}<\mathrm{\infty }\}$$ Then, the modulus inequality states that: $$|\mathbb{E}\{X\}|\le \mathbb{E}\{|X|\}$$

5.3.2 Markov inequality

Definition 5.2 ($\mathbf{\text{Markov Inequality}}$)
Let’s consider a random variable $X\in {\mathcal{L}}_1$ and fix a $\lambda >0$, then by the Markov inequality: $$\mathbb{P}(|X|\ge \lambda )\le \frac{1}{\lambda }\mathbb{E}\{|X|\}$$

5.3.3 Chebychev inequality

Definition 5.3 ($\mathbf{\text{Chebychev Inequality}}$)
Consider a random variable $X$ with first and second moment finite, i.e.  $$\mathbb{E}\{|X|\}<\mathrm{\infty },\mathbb{V}\{|X|\}<\mathrm{\infty }$$ then by the Chebychev inequality: $$\begin{array}{}\text{(5.5)}& \mathbb{P}(X\ge \lambda )\le \frac{1}{{\lambda }^2}\mathbb{E}\{|X{|}^2\}\end{array}$$

5.3.4 Holder inequality

Definition 5.4 ($\mathbf{\text{Holder Inequality}}$)
Let’s consider two numbers $p$ and $q$ such that $$p>1,q>1,\frac{1}{p}+\frac{1}{q}=1$$ and let’s consider two random variables $X$ and $Y$ such that: $$\mathbb{E}\{|X{|}^p\}<\mathrm{\infty },\mathbb{E}\{|Y{|}^q\}<\mathrm{\infty }$$ Then, $$\begin{array}{}\text{(5.6)}& |\mathbb{E}\{XY\}|\le \mathbb{E}\{|XY|\}\le (\mathbb{E}\{|X{|}^p\}{)}^{\frac{1}{p}}(\mathbb{E}\{|Y{|}^q\}{)}^{\frac{1}{q}}\end{array}$$ In terms of norms: $$||XY|{|}_1\le ||X|{|}_p||Y|{|}_q$$

5.3.5 Schwartz inequality

Definition 5.5 ($\mathbf{\text{Schwartz Inequality}}$)
Consider two random variables $X,Y\in {\mathcal{L}}_2$, i.e. with first and second moment finite, i.e.  $$\mathbb{E}\{|X|\}<\mathrm{\infty },\mathbb{E}\{X^2\}<\mathrm{\infty }$$ Then $$\begin{array}{}\text{(5.7)}& |\mathbb{E}\{XY\}|\le \mathbb{E}\{|XY|\}\le \sqrt{\mathbb{E}\{X^2\}\mathbb{E}\{Y^2\}}\end{array}$$ In terms of norms: $$||XY|{|}_1\le ||X|{|}_2||Y|{|}_2$$ Note that this is a special case of Holder inequality (Equation 5.6) with $p=q=2$.

5.3.6 Minkowski inequality

Definition 5.6 ($\mathbf{\text{Minkowski Inequality}}$)
For $1\le p<\mathrm{\infty }$ let’s consider two random variables $X,Y\in {\mathcal{L}}_p$, then $X+Y\in {\mathcal{L}}_p$ and $$\begin{array}{}\text{(5.8)}& ||X+Y|{|}_p\le ||X|{|}_p+||Y|{|}_p\end{array}$$

Note that the triangular inequality is a special case of Minkowski inequality with $p=1$, i.e.  $$\begin{array}{}\text{(5.9)}& |X+Y|\le |X|+|Y|\end{array}$$

5.3.7 Jensen inequality

Definition 5.7 ($\mathbf{\text{Jensen Inequality}}$)
Let’s consider a convex function $u:\mathbb{R}\to \mathbb{R}$. Suppose that $\mathbb{E}\{X\}<\mathrm{\infty }$ and $\mathbb{E}\{|u(X)|\}<\mathrm{\infty }$, then $$\begin{array}{}\text{(5.10)}& \mathbb{E}\{u(X)\}\ge u(\mathbb{E}\{X\})\end{array}$$ if $u$ is concave the results revert, i.e.  $$\begin{array}{}\text{(5.11)}& \mathbb{E}\{u(X)\}\le u(\mathbb{E}\{X\})\end{array}$$