6  Conditional expectation

Reference: Pietro Rigo (2023)

Theorem 6.1 ($\mathbf{\text{Radom Nikodym}}$)
Consider a measure space $(\mathrm{\Omega },\mathcal{B})$ and two measures $\mu$, $\nu$ such that $\mu$ is $\sigma$-finite (Definition 30.4) and $\mu <<\nu$ (Definition 30.1). Then there exists a measurable function $X:\mathrm{\Omega }\to \mathbb{R}$ such that: $$\mu (B)={\int }_{A}Xd\nu \mathrm{\forall }B\in \mathcal{B}$$

Definition 6.1 ($\mathbf{\text{Conditional expectation}}$)
Given a probability space $(\mathrm{\Omega },\mathcal{B},\mathbb{P})$. Let’s consider a sub $\sigma$-field of $\mathcal{B}$, i.e. $\mathcal{G}\subset \mathcal{B}$ and a random variable $X:\mathrm{\Omega }\to \mathbb{R}$ with finite expectation $\mathbb{E}\{|X|\}<+\mathrm{\infty }$. Then, the conditional expectation of $X$ given $\mathcal{G}$ is any random variable $$Z=\mathbb{E}\{X\mid \mathcal{G}\}\text{,}$$ such that:

1. $Z$ has finite expectation, i.e. $\mathbb{E}\{|Y|\}<+\mathrm{\infty }$.
2. $Z$ is $\mathcal{G}$-measurable.
3. $\mathbb{E}\{{\mathbb{1}}_{A}Z\}=\mathbb{E}\{{\mathbb{1}}_{A}X\}$, $\mathrm{\forall }A\in \mathcal{G}$, namely if $X$ and $Z$ are restricted to $A\in \mathcal{G}$, then their expectation coincides.

A $\sigma$-field can be used to describe our state of information. It means that, $\mathrm{\forall }A\in \mathcal{G}$ we already know if the event $A$ has occurred or not. Therefore, when we insert in $\mathcal{G}$ the events that we know were already occurred, we are saying that the random variable $Z$ is $\mathcal{G}$-measurable, i.e. the value of $Z$ is not stochastic once we know the information contained in $\mathcal{G}$. In this context, one can see the random variable $Y=\mathbb{E}\{X\mid \mathcal{G}\}$ as the prediction of $X$, given the information contained in the sub $\sigma$-field $\mathcal{G}$.

Definition 6.2 ($\mathbf{\text{Predictor}}$)
Consider $Z$ any $\mathcal{G}$-measurable random variable. Then $Z$ can be interpreted as a predictor of another random variable $X$ under the information contained in the $\sigma$-field $\mathcal{G}$. When we substitute $X$ with its prediction $Z$, we make an error given by the difference $X-Z$. In the special case in which $\mathbb{E}\{|Z{|}^2\}<\mathrm{\infty }$ and using as error function the mean squared error, i.e.  $$\mathbb{E}\{{\text{error}}^2\}=\mathbb{E}\{(X-Z{)}^2\}\text{,}$$ then it is possible to prove that the conditional expectation $\mathbb{E}\{X\mid \mathcal{G}\}$ represent the best predictor of $X$ in the sense that it minimized the mean squared error, i.e.  $$\mathbb{E}\{(X-\mathbb{E}\{X\mid \mathcal{G}\}{)}^2\}=\underset{Z\in \mathcal{Z}}{min}[\mathbb{E}\{(X-Z{)}^2\}]\text{.}$$ Hence, $\mathbb{E}\{X|\mathcal{G}\}$ is the best predictor that minimize the mean squared error over the class $\mathcal{Z}$ composed by $\mathcal{G}$-measurable functions with finite second moment, i.e. $$Z=\mathbb{E}\{X\mid \mathcal{G}\}=\underset{Z\in \mathcal{Z}}{\text{argmin}}[\mathbb{E}\{(X-Z{)}^2\}]\text{,}$$ where $\mathcal{Z}=\{Z:Z\text{ is }\mathcal{G}\text{-measurable}\text{and}\mathbb{E}\{|Z{|}^2\}<\mathrm{\infty }\}$.

6.1 Properties of conditional expectation

Here we state some useful properties of conditional expectation:

1. Linearity: The conditional expectation is linear for all all constants $a,b\in \mathbb{R}$, i.e.  $$\mathbb{E}\{aX+bY\mid \mathcal{G}\}=a\mathbb{E}\{X\mid \mathcal{G}\}+b\mathbb{E}\{Y\mid \mathcal{G}\}\text{.}$$
2. Positive: $X\ge 0$ implies that $\mathbb{E}\{X\mid \mathcal{G}\}\ge 0$.
3. Measurability: If $Y$ is $\mathcal{G}$-measurable, then $\mathbb{E}\{XY\mid \mathcal{G}\}=Y\mathbb{E}\{X\mid \mathcal{G}\}$. In general, if $X$ is $\mathcal{G}$-measurable then $\mathbb{E}\{X\mid \mathcal{G}\}=X$, i.e. $X$ is not stochastic.
4. Constant: The conditional expectation of a constant is a constant, i.e.  $$\mathbb{E}\{a\mid \mathcal{G}\}=a\mathrm{\forall }a\in \mathbb{R}\text{.}$$
5. Independence: If $X$ is independent from the $\sigma$-field $\mathcal{G}$, then $\mathbb{E}\{X\mid \mathcal{G}\}=\mathbb{E}\{X\}$.
6. Chain rule: If one consider two sub $\sigma$-fields of $\mathcal{B}$ such that ${\mathcal{G}}_{\mathcal{1}}\subset {\mathcal{G}}_{\mathcal{2}}$, then we can write: $$\begin{array}{}\text{(6.1)}& \mathbb{E}\{X\mid {\mathcal{G}}_{\mathcal{1}}\}=\mathbb{E}\{\mathbb{E}\{X\mid {\mathcal{G}}_{\mathcal{2}}\}\mid {\mathcal{G}}_{\mathcal{1}}\}⟺{\mathcal{G}}_{\mathcal{1}}\subset {\mathcal{G}}_{\mathcal{2}}\text{.}\end{array}$$ Remember that, when using the chain rule it is mandatory to take the conditional expectation before with respect to the greatest $\sigma$-field, i.e. the one that contains more information (in this case ${\mathcal{G}}_{\mathcal{2}}$), and then with respect to the smallest one (in this case ${\mathcal{G}}_{\mathcal{1}}$).

6.2 Conditional probability

Proposition 6.1 ($\mathbf{\text{Conditional probability}}$)
Given a probability space $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$, consider $\mathcal{G}$ as a sub $\sigma$-field of $\mathcal{F}$, i.e. $\mathcal{G}\subset \mathcal{F}$. Then the general definition of the conditional probability of an event $A$, given $\mathcal{G}$, is $$\begin{array}{}\text{(6.2)}& \mathbb{P}(A\mid \mathcal{G})=\mathbb{E}({\mathbb{1}}_A\mid \mathcal{G})\text{.}\end{array}$$ Instead, the elementary definition do not consider the conditioning with respect to a $\sigma$-field, but instead with respect to a single event $B$. In practice, take an event $B\in \mathcal{F}$ such that $0<\mathbb{P}(B)<1$, then $\mathrm{\forall }A\in \mathcal{F}$ the conditional probability of $A$ given $B$ is defined as: $$\begin{array}{}\text{(6.3)}& \mathbb{P}(A\mid B)=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)}\text{,}\mathbb{P}(A\mid B^c)=\frac{\mathbb{P}(A\cap B^c)}{\mathbb{P}(B^c)}\text{.}\end{array}$$

Proof: Proposition 6.1

Proof. The elementary (Equation 6.3) and the general (Equation 6.2) definitions are equivalent, in fact consider a sub $\sigma$-field $\mathcal{G}$ which provides only the information concerning whenever $\omega$ is in $B$ or not. A $\sigma$-field of this kind will have the form ${\mathcal{G}}_B=\{\mathrm{\Omega },\mathrm{\varnothing },B,B^c\}$. Then, consider a ${\mathcal{G}}_B$-measurable function, $f:\mathrm{\Omega }\to \mathbb{R}$, such that: $$f(\omega )=\{\begin{array}{l}\alpha \omega \in B\\ \beta \omega \in B^c\end{array}$$ It remains to find $\alpha$ and $\beta$ in the following expression: $$\mathbb{P}(A\mid {\mathcal{G}}_B)=\mathbb{E}\{{\mathbb{1}}_A\mid {\mathcal{G}}_B\}=\alpha {\mathbb{1}}_B+\beta {\mathbb{1}}_{B^c}\text{.}$$ Note that, the joint probability of $A$ and $B$ can be obtained as: $$\begin{array}{rl}\mathbb{P}(A\cap B)& =\mathbb{E}\{{\mathbb{1}}_A{\mathbb{1}}_B\}=\mathbb{E}\{\mathbb{E}\{{\mathbb{1}}_A{\mathbb{1}}_B\mid {\mathcal{G}}_B\}\}=\\ & =\mathbb{E}\{\mathbb{E}\{{\mathbb{1}}_A\mid {\mathcal{G}}_B\}{\mathbb{1}}_B\}=\\ & =\mathbb{E}\{\mathbb{P}(A\mid {\mathcal{G}}_B){\mathbb{1}}_B\}=\\ & =\mathbb{E}\{(\alpha {\mathbb{1}}_B+\beta {\mathbb{1}}_{B^c}){\mathbb{1}}_B\}=\\ & =\alpha \mathbb{E}\{{\mathbb{1}}_B\}+\beta \mathbb{E}\{{\mathbb{1}}_{B^c}{\mathbb{1}}_B\}=\\ & =\alpha \mathbb{P}(B)\end{array}$$ Hence, we obtain: $$\mathbb{P}(A\cap B)=\alpha \mathbb{P}(B)⟹\alpha =\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)}\text{.}$$ Equivalently for $\mathbb{P}(A\cap B^c)$ it is possible to prove that: $$\mathbb{P}(A\cap B^c)=\beta \mathbb{P}(B^c)⟹\beta =\frac{\mathbb{P}(A\cap B^c)}{\mathbb{P}(B^c)}\text{.}$$ Finally, thanks to this result it is possible to express the conditional probability in the general definition (Equation 6.2) as a linear combination of conditional probabilities defined accordingly to the elementary definition (Equation 6.3), i.e. $$\mathbb{P}(A\mid {\mathcal{G}}_B)=\mathbb{P}(A\mid B){\mathbb{1}}_B+\mathbb{P}(A\mid B^c){\mathbb{1}}_{B^c}\text{.}$$

Exercise 6.1 Let’s continue from the Exercise 3.1 and let’s say that we observe $X(\omega )=\{+1\}$, then we ask ourselves, what is the probability that in the next extraction $X(\omega )=\{0\}$?

Solution: Exercise 6.1

Solution 6.1. The chances that with 52 cards we obtain $X(\omega )=\{0\}$ is approximately $\frac{12}{52}\approx 23.08\mathrm{\%}$ (see Solution 3.2), while $X(\omega )=\{1\}$ is approximately $\frac{20}{52}\approx 38.46\mathrm{\%}$.

Hence, if we have extracted a card that originates an $X(\omega )=\{+1\}$, then in the deck remain only 19 possible cards that gives $\{+1\}$ while the number of total cards reduces to 51. Thus, the conditional probability on another 1 decreases $$\mathbb{P}(X({\omega }_2)=\{+1\}\mid X({\omega }_1)=\{+1\})=\frac{19}{51}=37.25\mathrm{\%}\text{.}$$ On the other hand, the conditional probability of a 0 increases $$\mathbb{P}(X({\omega }_2)=\{0\}\mid X({\omega }_1)=\{+1\})=\frac{12}{51}=23.52\mathrm{\%}\text{.}$$

Proposition 6.2 ($\mathbf{\text{Conditional probability and independent events}}$)
If two events $A$ and $B$ are independent and $\mathbb{P}(B)>0$, then $$A\perp B⟺\mathbb{P}(A\mid B)=\mathbb{P}(A)\text{.}$$

Proof: Proposition 6.2

Proof. To prove the result in Proposition 6.2 we consider both sides of the expression.

1. (LHS $⟹$ RHS) Let’s assume that $A$ and $B$ are independent, i.e. $A\perp B$. Then, by definition (Definition 4.1) under independence holds the decomposition in Equation 4.1, i.e. $$\mathbb{P}(A\cap B)=\mathbb{P}(A)\mathbb{P}(B)⟹\mathbb{P}(A\mid B)=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)}=\mathbb{P}(A)\text{.}$$

2. (RHS $⟹$ LHS) Let’s assume that the following holds, i.e. $\mathbb{P}(A\mid B)=\mathbb{P}(A)$ and $\mathbb{P}(B)>0$. Then, by definition of conditional probability (Equation 6.3) the joint probability is defined as: $$\mathbb{P}(A\cap B)=\mathbb{P}(A\mid B)\mathbb{P}(B)=\mathbb{P}(A)\mathbb{P}(B)\text{,}$$ that is exactly the definition of independence (Definition 4.1).

Theorem 6.2 ($\mathbf{\text{Bayes’ Theorem}}$)
Let’s consider a partition of disjoint events $\{A_1,\dots ,A_n\}$ with each $A_n\subset \mathrm{\Omega }$ and such that ${\bigsqcup }_{i=1}^n{A}_i=\mathrm{\Omega }$. Then, given any event $B\subset \mathrm{\Omega }$ with probability greater than zero, $\mathbb{P}(B)>0$. Then, for any $j\in \{1,2,\dots ,n\}$ the conditional probability of the event $A_j$ given $B$ is defined as: $$\mathbb{P}(A_j\mid B)=\frac{\mathbb{P}(B\mid A_j)\mathbb{P}(A_j)}{\sum _{i=1}^n\mathbb{P}(B\mid A_i)\mathbb{P}(A_i)}\text{.}$$

Conditional probability: numerical example

Example 6.1 Let’s consider two random variables $X(\omega )$ and $Y(\omega )$ taking values in $\{0,1\}$. The marginal probabilities $\mathbb{P}(X=0)=0.6$ and $\mathbb{P}(Y=0)=0.29$. Let’s consider the matrix of joint events and probabilities, i.e.  $$\left(\begin{array}{cc}[X=0]\cap [Y=0]& [X=0]\cap [Y=1]\\ [X=1]\cap [Y=0]& [X=1]\cap [Y=1]\end{array}\right)\stackrel{\mathbb{P}}{⟶}\left(\begin{array}{cc}0.17& 0.43\\ 0.12& 0.28\end{array}\right)\text{.}$$ Then, by definition the conditional probabilities are defined as: $$\mathbb{P}(X=0\mid Y=0)=\frac{\mathbb{P}(X=0\cap Y=0)}{\mathbb{P}(Y=0)}=\frac{0.17}{0.29}\approx 58.63\mathrm{\%}\text{,}$$ and $$\mathbb{P}(X=0\mid Y=1)=\frac{\mathbb{P}(X=0\cap Y=1)}{\mathbb{P}(Y=1)}=\frac{0.43}{1-0.29}\approx 60.56\mathrm{\%}\text{.}$$ Considering $Y$ instead: $$\mathbb{P}(Y=0\mid X=0)=\frac{\mathbb{P}(Y=0\cap X=0)}{\mathbb{P}(X=0)}=\frac{0.17}{0.6}\approx 28.33\mathrm{\%}\text{,}$$ and $$\mathbb{P}(Y=0\mid X=1)=\frac{\mathbb{P}(Y=0\cap X=1)}{\mathbb{P}(X=1)}=\frac{0.12}{1-0.6}\approx 30\mathrm{\%}\text{,}$$ Then, it is possible to express the marginal probability of $X$ as: $$\begin{array}{rl}\mathbb{P}(X=0)& =\mathbb{E}\{\mathbb{P}(X=0\mid Y)\}=\\ & =\mathbb{P}(X=0\mid Y=0)\mathbb{P}(Y=0)+\mathbb{P}(X=0\mid Y=1)\mathbb{P}(Y=1)=\\ & =0.5863\cdot 0.29+0.6056\cdot (1-0.29)\approx 60\mathrm{\%}\end{array}$$ And similarly for $Y$ $$\begin{array}{rl}\mathbb{P}(Y=0)& =\mathbb{E}\{\mathbb{P}(Y=0\mid X)\}=\\ & =\mathbb{P}(Y=0\mid X=0)\mathbb{P}(X=0)+\mathbb{P}(Y=0\mid X=1)\mathbb{P}(X=1)=\\ & =0.2833\cdot 0.6+0.30\cdot (1-0.6)\approx 29\mathrm{\%}\end{array}$$

Exercise 6.2 ($\mathbf{\text{Monty Hall Problem}}$)
You are on a game show where there are three closed doors: behind one is a car (the prize) and behind the other two are goats.

The rules are simple:

1. You choose one door (say Door 1).
2. The host, who knows where the car is, opens one of the other doors, always revealing a goat.
3. You are then offered the chance to stay with your original choice or switch to the other unopened door.

Question: Is in your interests to switch door? (See 21 Blackjack)

Solution: Exercise 6.2

Solution 6.2. Before any door is opened, the probability that the car is behind each door is $$\mathbb{P}(\text{car behind door 1})=\mathbb{P}(\text{car behind door 2})=\mathbb{P}(\text{car behind door 3})=\frac{1}{3}=33.\overline{3}\mathrm{\%}\text{.}$$ Suppose you picked Door 1. The conductor opens (say) Door 3, revealing a goat. Now the conditional probabilities are:

• If the car is behind Door 1: Monty could open either Door 2 or Door 3 with equal probability.

• If the car is behind Door 2: Monty is forced to open Door 3.

• If the car is behind Door 3: Monty is forced to open Door 2 (so this case is impossible if Monty opens Door 3).

Apply Bayes’ Rule: $$\mathbb{P}(\text{car behind door 1}\mid \text{Monty opens door 3})=\frac{\frac{1}{3}\cdot \frac{1}{2}}{\frac{1}{3}\cdot \frac{1}{2}+\frac{1}{3}\cdot 1}=\frac{1/6}{1/6+1/3}=\frac{1}{3}=33.\overline{3}\mathrm{\%}\text{.}$$ On the other hand, the other door has probabvility of winning of $$\mathbb{P}(\text{car behind door 2}\mid \text{Monty opens door 3})=\frac{\frac{1}{3}\cdot 1}{\frac{1}{3}\cdot \frac{1}{2}+\frac{1}{3}\cdot 1}=\frac{1/3}{1/6+1/3}=\frac{2}{3}=66.\overline{6}\mathrm{\%}\text{.}$$ After Monty opens a goat door, the probability the car is behind your original choice is still 33$\mathrm{\%}$, while the probability it is behind the other unopened door is 66$\mathrm{\%}$, almost double . Therefore, switching doubles the chances of winning.

6.2.1 Conditional variance

Proposition 6.3 ($\mathbf{\text{Conditional variance}}$)
Let’s consider two random variable $X$ and $Y$ with finite second moment. Then, the total variance can be expressed as: $$\mathbb{V}\{X\}=\mathbb{E}\{\mathbb{V}\{X\mid Y\}\}+\mathbb{V}\{\mathbb{E}\{X\mid Y\}\}⟺\mathbb{V}\{Y\}=\mathbb{E}\{\mathbb{V}\{Y\mid X\}\}+\mathbb{V}\{\mathbb{E}\{Y\mid X\}\}$$

Proof: Proposition 6.3

Proof. By definition, the variance of a random variable reads: $$\mathbb{V}\{X\}=\mathbb{E}\{X^2\}-\mathbb{E}\{X{\}}^2\text{.}$$ Applying the chain rule (Equation 6.1) one can write $$\mathbb{V}\{X\}=\mathbb{E}\{\mathbb{E}\{X^2\mid Y\}\}-\mathbb{E}\{\mathbb{E}\{X\mid Y\}{\}}^2\text{.}$$ Then, add and subtract $\mathbb{E}\{\mathbb{E}\{X\mid Y{\}}^2\}$ gives $$\mathbb{V}\{X\}=\mathbb{E}\{\mathbb{E}\{X^2\mid Y\}\}-\mathbb{E}\{\mathbb{E}\{X\mid Y\}{\}}^2+\mathbb{E}\{\mathbb{E}\{X\mid Y{\}}^2\}-\mathbb{E}\{\mathbb{E}\{X\mid Y{\}}^2\}\text{.}$$ Group the first and fourth terms and the second and third to obtain $$\mathbb{V}\{X\}=\mathbb{E}\{\mathbb{E}\{X^2\mid Y\}-\mathbb{E}\{X\mid Y{\}}^2\}+\mathbb{E}\{\mathbb{E}\{X\mid Y{\}}^2\}-\mathbb{E}\{\mathbb{E}\{X\mid Y\}{\}}^2\text{,}$$ that simplifies as $$\mathbb{V}\{X\}=\mathbb{E}\{\mathbb{V}\{X\mid Y\}\}+\mathbb{V}\{\mathbb{E}\{X\mid Y\}\}\text{.}$$

Pietro Rigo. 2023. “Appunti Di Probabilità (PhD Courses).”