What is Statistics?

Statistics is about collecting and analysing data to draw conclusions, often in the real world.

In probability theory, we do theoretical calculations to find things like the expected value of a fair die roll.

For instance, letting $X$ represent the outcome of a fair die roll, we can calculate:

$$\mathbb{E}(X) = 1\times \frac{1}{6}+2\times \frac{1}{6}+3\times \frac{1}{6}+4\times \frac{1}{6}+5\times \frac{1}{6}+6\times \frac{1}{6}=3.5$$

However, suppose instead we were not sure whether or not our die was fair, perhaps wanting to test this. Now we are in a position where we do not know the probability distribution, so cannot do the above calculations.

We may nevertheless still want to know about the expected value of this unknown distribution.

A reasonable plan would be to start by rolling the die a number of times – let’s say $100$.

We obtain the following data, where $x_i$ is the $i^{th}$ roll:

$$x_1=2,x_2=4,x_3=2,x_4=2, \dots , x_{98}=4, x_{99}=4, x_{100}=6$$

Naturally, we could then take the average of these rolls:

$$\frac{x_1+x_2+\dots+x_{100}}{100} = \frac{2+4+\dots +6}{100}=3.42$$

This is different from the expected value of $3.5$ from a fair die roll. However, this doesn’t prove the die is not fair, since we shouldn’t expect exactly $3.5$, even from a fair die. In the real world, numbers are fickle like that!

We can use this approach any time we want to try to figure out an unknown like $\mathbb{E}(X)$.

In general, we call the process “estimation” and the number $3.42$ our “estimate”.

This example illustrates the difference between probability and statistics: in probability we work with known distributions, whereas in statistics we generally work with data to try to infer things about unknown distributions.

In statistics, we also use probability theory to check up on whether estimation methods like this are likely to work well or not.