What is the Law of Iterated Expectation?
To find the expected value of a random variable $Y$, it is often best to first think about the value of another random variable $X$, and form a prediction about $Y$ in light of this information.
The law of iterated expectation says that to find the overall expected value of $Y$, we can first form a prediction based on $X$, and then take the expected value of this prediction. That is, we make a “prediction about our prediction”.
The law of iterated expectation is a topic students often find confusing, but it is very useful to master. We present the key intuition with our earlier example.
Suppose again we have a bag with three balls, labelled $1,2,3$. We pick a ball at random from the bag, and let the number shown be $X$. Then, we flip a fair coin $X$ times. Let $Y$ be the number of heads obtained in total.
Then we have:
$$\mathbb{E}(Y \mid X=1) = 0.5 $$
$$\mathbb{E}(Y \mid X=2) = 1 \ \ \ $$
$$\mathbb{E}(Y \mid X=3) = 1.5 $$
Which we can summarise as:
$$\mathbb{E}(Y \mid X=x) = 0.5x, \quad x = 1,2,3 $$
Replacing $x$ with its random counterpart $X$, we get the “random predictor” $\mathbb{E}(Y \mid X) = 0.5X$. This is itself a random variable, and a function of $X$. The law of iterated expectation says that the expected value of this random variable is the same as that of $Y$.
That is; we have
$$\mathbb{E}(Y) = \mathbb{E}( 0.5 X) $$
This should make sense: we’re getting on average $0.5$ Heads for each time we will have to flip the coin, and there will be $X$ coin flips.
Next, we calculate:
$$ \mathbb{E}(X)= 1 \times \mathbb{P}(X=1) + 2 \times \mathbb{P}(X=2) + 3 \times \mathbb{P}(X=3 ) = 1 \times \frac{1}{3} + 2 \times \frac{1}{3} + 3 \times \frac{1}{3} = 2 $$
So, we conclude:
$$\mathbb{E}(Y) = \mathbb{E}( 0.5 X) = 0.5 \mathbb{E}(X) = 0.5 \times 2 = 1 $$
And in general, we have:
$$\mathbb{E}(Y) = \mathbb{E}( \mathbb{E}(Y \mid X) ) $$
Where the “inner” expectation $\mathbb{E}(Y \mid X)$ is a function of $X$, and the “outer” expectation over $X$ finds the expected value of this function.