What is a Moment Generating Function?
The moment generating function of a random variable $X$ is a power series whose coefficients are related to the moments of $X$. Equivalently, it is defined as:
$$M_X(t) = \mathbb{E}(e^{tX})$$
Given a sequence $(a_0, a_1, a_2, \dots ) $, its “exponential generating function” is defined as a power series where the coefficient of $t^n$ is precisely $\frac{a_n}{n!}$. That is:
$$f(t)=a_0 + \frac{a_1}{1!}t + \frac{a_2}{2!}t^2 + \frac{a_3}{3!}t^3 + \dots $$
Now, let $X$ be a random variable. We have the sequence of its “moments”:
$$\mathbb{E}(X^0)=1, \ \ \mathbb{E}(X^1)=\mu_X, \ \ \mathbb{E}(X^2), \ \ \mathbb{E}(X^3), \ \ \dots $$
The moment generating function of $X$ is precisely the exponential generating function of this sequence. That is:
$$ M_X(t) = 1+ \frac{\mathbb{E}(X)}{1!}t + \frac{\mathbb{E}(X^2)}{2!}t^2 + \frac{\mathbb{E}(X^3)}{3!}t^3 + \dots $$
Knowing the MGF of a distribution uniquely pins down which probability distribution we have. Often we therefore try to recognise the distribution of a complex random variable by looking at its moment generating function.
What are Moment Generating Functions For?
If we know the MGF of a random variable, we can use this to recover the moments. The method is similar to that for PGFs.
We take the above formula, differentiate several times with respect to $t$, thus moving the moment we want into the “constant” position, and finally set $t=0$ to clear away the other terms.
Suppose, for instance, we want the second moment. We differentiate once:
$$ M’_X(t) = \mathbb{E}(X)+ \frac{\mathbb{E}(X^2)}{1!}t + \frac{\mathbb{E}(X^3)}{2!}t^2 + \frac{\mathbb{E}(X^4)}{3!}t^3 + \dots $$
Note that the factorial terms are partly cancelled by the powers coming down.
Then, we differentiate again:
$$ M’’_X(t) = \mathbb{E}(X^2)+ \frac{\mathbb{E}(X^3)}{1!}t + \frac{\mathbb{E}(X^4)}{2!}t^2 + \frac{\mathbb{E}(X^5)}{3!}t^3 + \dots $$
Finally, we set $t=0$, leaving just the first term remaining:
$$M’’_X(0) = \mathbb{E}(X^2)$$
For instance, if $X$ is binomially distributed, $X \sim \text{Bin}(n,p)$, it is possible to show that
$$M_X(t) = (1-p + pe^t)^n$$
Differentiating using the chain rule, we have:
$$M_X’(t) = n(1-p + pe^t)^{(n-1)} (pe^t)$$
Then setting $t=0$, we obtain:
$$ \mathbb{E}(X) = M’_X(0) = n(1-p+p)^{n-1}p = np $$
The second moment and variance can be obtained by differentiating again.
How Do we Find Moment Generating Functions?
Recall the exponential series:
$$e^y = 1 + y + \frac{y^2}{2!} + \frac{y^3}{3!} + \dots $$
This looks quite like our MGF formula. Indeed, letting $y=tX$ and taking an expectation, we have
$$\mathbb{E}(e^{tX}) = \mathbb{E} \left( 1+tX+\frac{(tX)^2}{2!}+\frac{(tX)^3}{3!}+\dots \right) $$
$$ = 1+ \frac{\mathbb{E}(X)}{1!}t + \frac{\mathbb{E}(X^2)}{2!}t^2 + \frac{\mathbb{E}(X^3)}{3!}t^3 + \dots = M_X(t) $$
So, we can find MGFs by looking at $\mathbb{E}(e^{tX})$ and doing a sum or integral.
As with probability generating functions, once we have worked out the MGFs of some random variables, we can extend this to sums and linear transformations, using this expected value formula.
If $X$ and $Y$ are independent, we have:
$$ M_{X+Y}(t) = \mathbb{E}(e^{(X+Y)t}) = \mathbb{E}(e^{Xt}e^{Yt}) = \mathbb{E}(e^{Xt})\mathbb{E}(e^{Yt}) = M_X(t) \ M_Y(t)$$
Transforming $X$ to $aX+b$, we have:
$$ M_{aX+b}(t) = \mathbb{E}(e^{(aX+b)t}) = e^{bt} \ \mathbb{E}(e^{atX})= e^{bt} M_X(at) $$
Again, this is very useful when working out the MGF of the average of a series of independent variables $X_1,X_2, \dots X_n$, each with the same distribution. In statistics, this is called a “random sample”.