6  Conditional expectation

Theorem 6.1 ($\mathbf{\text{Radom Nikodym}}$)
Consider a measure space $(\mathrm{\Omega },\mathcal{B})$ and two measures $\mu$, $\nu$ such that $\mu$ is $\sigma$-finite (Definition 31.4) and $\mu <<\nu$ (Definition 31.1). Then there exists a measurable function $X:\mathrm{\Omega }\to \mathbb{R}$ such that: $$\mu (B)={\int }_{A}Xd\nu \mathrm{\forall }B\in \mathcal{B}$$

Definition 6.1 ($\mathbf{\text{Conditional expectation}}$)
Given a probability space $(\mathrm{\Omega },\mathcal{B},\mathbb{P})$, consider $\mathcal{G}$ as a sub $\sigma$-field of $\mathcal{B}$, i.e. $\mathcal{G}\subset \mathcal{B}$. Let’s consider a random variable $X:\mathrm{\Omega }\to \mathbb{R}$ with finite expectation $\mathbb{E}\{|X|\}<+\mathrm{\infty }$. We define a conditional expectation for $X$ given $\mathcal{G}$, any random variable $Y=\mathbb{E}\{X|\mathcal{G}\}$ such that:

1. $Y$ has finite expectation, i.e. $\mathbb{E}\{|Y|\}<+\mathrm{\infty }$.
2. $Y$ is $\mathcal{G}$-measurable.
3. $\mathbb{E}\{{\mathbb{1}}_{A}Y\}=\mathbb{E}\{{\mathbb{1}}_{A}X\}$, $\mathrm{\forall }A\in \mathcal{G}$, namely if $X$ and $Y$ are restricted to $A\in \mathcal{G}$, then their expectation coincides.

A $\sigma$-field can be used to describe our state of information. It means that, $\mathrm{\forall }A\in \mathcal{G}$ we already know if the event A has occurred or not. Therefore, when we insert in $\mathcal{G}$ the events that we know were already occurred, we are saying that the random variable $Y$ is $\mathcal{G}$-measurable, i.e. the value of $Y$ is not stochastic once we know the information contained in $\mathcal{G}$. Moreover, the random variable $Y=\mathbb{E}\{X|\mathcal{G}\}$ represent a prediction of the random variable $X$, given the information contained in the sub $\sigma$-field $\mathcal{G}$.

Definition 6.2 ($\mathbf{\text{Predictor}}$)
Consider $Z$ any $\mathcal{G}$-measurable random variable. Then $Z$ can be interpreted as a predictor of another random variable $X$ under the information contained in the $\sigma$-field $\mathcal{G}$. However, when we substitute $X$ with its prediction, namely $Z$, we make an error given by the difference $X-Z$. In the special case in which $\mathbb{E}\{|Z{|}^2\}<\mathrm{\infty }$, we can take as error function the mean squared error, i.e. $$\mathbb{E}\{{\text{error}}^2\}=\mathbb{E}\{(X-Z{)}^2\}$$ We say that the conditional expectation $\mathbb{E}\{X|\mathcal{G}\}$ is the best predictor in the sense that: $$\mathbb{E}\{(X-\mathbb{E}\{X|\mathcal{G}\}{)}^2\}=\underset{Z\in \mathcal{Z}}{min}\mathbb{E}\{(X-Z{)}^2\}$$ Hence, $\mathbb{E}\{X|\mathcal{G}\}$ is the best predictor that minimize the mean squared error over the class $\mathcal{Z}$ composed by $\mathcal{G}$-measurable functions with finite second moment, formally
$$\mathcal{Z}=\{Z\mathcal{G}\text{-measurable}\text{and}\mathbb{E}\{|Z{|}^2\}<\mathrm{\infty }\}$$

6.1 Properties of conditional expectation

Here we state some useful properties of conditional expectation:

1. Linearity: $\mathbb{E}\{aX+bY|\mathcal{G}\}=a\mathbb{E}\{X|\mathcal{G}\}+b\mathbb{E}\{Y|\mathcal{G}\}$, for all constants $a,b\in \mathbb{R}$.
2. Positive: $X\ge 0⟹\mathbb{E}\{X|\mathcal{G}\}\ge 0$.
3. Measurability: If $Y$ is $\mathcal{G}$-measurable, then $\mathbb{E}\{XY|\mathcal{G}\}=Y\mathbb{E}\{X|\mathcal{G}\}$.
4. Constant: $\mathbb{E}\{a|\mathcal{G}\}=a\mathrm{\forall }a\in \mathbb{R}$. In general, if $X$ is $\mathcal{G}$-measurable then $\mathbb{E}\{X|\mathcal{G}\}=X$, i.e. is not stochastic.
5. Independence: If $X$ is independent from the $\sigma$-field $\mathcal{G}$, then $\mathbb{E}\{X|\mathcal{G}\}=\mathbb{E}\{X\}$.
6. Chain rule: consider two two sub $\sigma$-fields of $\mathcal{B}$ such that ${\mathcal{G}}_{\mathcal{1}}\subset {\mathcal{G}}_{\mathcal{2}}$, then we can write: $$\mathbb{E}\{X|{\mathcal{G}}_{\mathcal{1}}\}=\mathbb{E}\{\mathbb{E}\{X|{\mathcal{G}}_{\mathcal{2}}\}|{\mathcal{G}}_{\mathcal{1}}\}$$ Remember that, when using this property it is mandatory to take the conditional expectation before with respect to the greatest $\sigma$-field, i.e. the one that contains more information (in this case ${\mathcal{G}}_{\mathcal{2}}$), and then with respect to the smallest one (in this case ${\mathcal{G}}_{\mathcal{1}}$).

6.2 Conditional probability

Definition 6.3 ($\mathbf{\text{Conditional probability}}$)
Given a probability space $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$, consider $\mathcal{G}$ as a sub $\sigma$-field of $\mathcal{F}$, i.e. $\mathcal{G}\subset \mathcal{F}$. Then the general definition of the conditional probability of an event $A$ given $\mathcal{G}$ is: $$\begin{array}{}\text{(6.1)}& \mathbb{P}(A|\mathcal{G})=\mathbb{E}({\mathbb{1}}_A|\mathcal{G})\end{array}$$ Instead, the elementary definition do not consider the conditioning with respect to a $\sigma$-field, but instead with respect to a single event $B$. In practice, take an event $B\in \mathcal{F}$ such that $0<\mathbb{P}(B)<1$, then $\mathrm{\forall }A\in \mathcal{F}$ the conditional probability of $A$ given $B$ is defined as: $$\begin{array}{}\text{(6.2)}& \mathbb{P}(A|B)=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)},\mathbb{P}(A|B^c)=\frac{\mathbb{P}(A\cap B^c)}{\mathbb{P}(B^c)}\end{array}$$

Elementary and the general definition are equivalent

The elementary (Equation 6.2) and the general (Equation 6.1) definitions are equivalent, in fact consider a sub $\sigma$-field $\mathcal{G}$ which provides only the information concerning whenever $\omega$ is in $B$ or not. A $\sigma$-field of this kind will have the form ${\mathcal{G}}_B=\{\mathrm{\Omega },\mathrm{\varnothing },B,B^c\}$. Then, consider a ${\mathcal{G}}_B$-measurable function, $f:\mathrm{\Omega }\to \mathbb{R}$, such that: $$f(\omega )=\{\begin{array}{l}\alpha \omega \in B\\ \beta \omega \in B^c\end{array}$$ It remains to find $\alpha$ and $\beta$ in the following expression: $$\mathbb{P}(A|{\mathcal{G}}_B)=\mathbb{E}\{{\mathbb{1}}_A|{\mathcal{G}}_B\}=\alpha {\mathbb{1}}_B+\beta {\mathbb{1}}_{B^c}$$ Note that, the joint probability of $A$ and $B$ can be obtained as: $$\begin{array}{rl}\mathbb{P}(A\cap B)& =\mathbb{E}\{{\mathbb{1}}_A{\mathbb{1}}_B\}=\mathbb{E}\{\mathbb{E}\{{\mathbb{1}}_A{\mathbb{1}}_B|{\mathcal{G}}_B\}\}=\mathbb{E}\{\mathbb{E}\{{\mathbb{1}}_A|{\mathcal{G}}_B\}{\mathbb{1}}_B\}=\\ & =\mathbb{E}\{\mathbb{P}(A|{\mathcal{G}}_B){\mathbb{1}}_B\}=\\ & =\mathbb{E}\{(\alpha {\mathbb{1}}_B+\beta {\mathbb{1}}_{B^c}){\mathbb{1}}_B\}=\\ & =\alpha \mathbb{E}\{{\mathbb{1}}_B\}+\beta \mathbb{E}\{{\mathbb{1}}_{B^c}{\mathbb{1}}_B\}=\\ & =\alpha \mathbb{P}(B)\end{array}$$ Hence, we obtain: $$\mathbb{P}(A\cap B)=\alpha \mathbb{P}(B)⟹\alpha =\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)}$$ Equivalently for $\mathbb{P}(A\cap B^c)$ it is possible to prove that: $$\mathbb{P}(A\cap B^c)=\beta \mathbb{P}(B^c)⟹\beta =\frac{\mathbb{P}(A\cap B^c)}{\mathbb{P}(B^c)}$$

Finally it is possible to write the conditional probability in the general definition as a linear combination of conditional probabilities defined accordingly to the elementary one, i.e.  $$\mathbb{P}(A|{\mathcal{G}}_B)=\mathbb{P}(A|B){\mathbb{1}}_B+\mathbb{P}(A|B^c){\mathbb{1}}_{B^c}$$

Conditional probability

Example 6.1 Let’s continue from the example Example 2.1, let’s say that we observe $X(\omega )=\{+1\}$, then we ask ourselves, what is the probability that in the next extraction $X(\omega )=\{0\}$? The chances that with 52 cards we obtain $X(\omega )=\{0\}$ is approximately $\frac{3}{13}\approx 23.08\mathrm{\%}$ (see Example 3.1). Then, given the fact that the extracted card originates $X(\omega )=\{+1\}$ we have that the probability, conditional to the fact that in the first extraction we had a card $\{+1\}$, that in the next extraction we have $\{0\}$ is $\frac{12}{51}=23.52\mathrm{\%}$. Let’s now investigate the chances that in the next extraction $X(\omega )=\{+1\}$ given that in the previous was $\{+1\}$. The unconditional probability is $\frac{20}{52}\approx 38.46\mathrm{\%}$, the conditional probability will be $\frac{19}{51}\approx 37.25\mathrm{\%}$.

Conditional probability: numerical example

Example 6.2 Let’s consider two random variables $X(\omega )$ and $Y(\omega )$ taking values in $\{0,1\}$. The marginal probabilities $\mathbb{P}(X=0)=0.6$ and $\mathbb{P}(Y=0)=0.29$. Let’s consider the matrix of joint events and probabilities, i.e.  $$\left(\begin{array}{cc}[X=0]\cap [Y=0]& [X=0]\cap [Y=1]\\ [X=1]\cap [Y=0]& [X=1]\cap [Y=1]\end{array}\right)\stackrel{\mathbb{P}}{⟶}\left(\begin{array}{cc}0.17& 0.43\\ 0.12& 0.28\end{array}\right)$$ Then, by definition the conditional probabilities are defined as: $$\mathbb{P}(X=0|Y=0)=\frac{\mathbb{P}(X=0\cap Y=0)}{\mathbb{P}(Y=0)}=\frac{0.17}{0.29}\approx 58.63\mathrm{\%}$$ and $$\mathbb{P}(X=0|Y=1)=\frac{\mathbb{P}(X=0\cap Y=1)}{\mathbb{P}(Y=1)}=\frac{0.43}{1-0.29}\approx 60.56\mathrm{\%}$$ Considering $Y$ instead: $$\mathbb{P}(Y=0|X=0)=\frac{\mathbb{P}(Y=0\cap X=0)}{\mathbb{P}(X=0)}=\frac{0.17}{0.6}\approx 28.33\mathrm{\%}$$ and $$\mathbb{P}(Y=0|X=1)=\frac{\mathbb{P}(Y=0\cap X=1)}{\mathbb{P}(X=1)}=\frac{0.12}{1-0.6}\approx 30\mathrm{\%}$$ Then, it is possible to express the marginal probability of $X$ as: $$\begin{array}{rl}\mathbb{P}(X=0)& =\mathbb{E}\{\mathbb{P}(X=0|Y)\}=\\ & =\mathbb{P}(X=0|Y=0)\mathbb{P}(Y=0)+\mathbb{P}(X=0|Y=1)\mathbb{P}(Y=1)=\\ & =0.5863\cdot 0.29+0.6056\cdot (1-0.29)\approx 60\mathrm{\%}\end{array}$$ And similarly for $Y$ $$\begin{array}{rl}\mathbb{P}(Y=0)& =\mathbb{E}\{\mathbb{P}(Y=0|X)\}=\\ & =\mathbb{P}(Y=0|X=0)\mathbb{P}(X=0)+\mathbb{P}(Y=0|X=1)\mathbb{P}(X=1)=\\ & =0.2833\cdot 0.6+0.30\cdot (1-0.6)\approx 29\mathrm{\%}\end{array}$$