Sample variance

The sample variance of a random variable demonstrates two aspects of estimator bias: firstly, the naive estimator is biased, which can be corrected by a scale factor; second, the unbiased estimator is not optimal in terms of mean squared error (MSE), which can be minimized by using a different scale factor, resulting in a biased estimator with lower MSE than the unbiased estimator. Concretely, the naive estimator sums the squared deviations and divides by n, which is biased. Dividing instead by n − 1 yields an unbiased estimator. Conversely, MSE can be minimized by dividing by a different number (depending on distribution), but this results in a biased estimator. This number is always larger than n − 1, so this is known as a shrinkage estimator, as it "shrinks" the unbiased estimator towards zero; for the normal distribution the optimal value is n + 1.

Suppose X₁, ..., X_n are independent and identically distributed (i.i.d.) random variables with expectation μ and variance σ². If the sample mean and uncorrected sample variance are defined as

${\displaystyle {\overline {X}}\,={\frac {1}{n}}\sum _{i=1}^{n}X_{i}\qquad S^{2}={\frac {1}{n}}\sum _{i=1}^{n}{\big (}X_{i}-{\overline {X}}\,{\big )}^{2}\qquad }$

then S² is a biased estimator of σ², because

${\displaystyle {\begin{aligned}\operatorname {E} [S^{2}]&=\operatorname {E} \left[{\frac {1}{n}}\sum _{i=1}^{n}{\big (}X_{i}-{\overline {X}}{\big )}^{2}\right]=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}{\bigg (}(X_{i}-\mu )-({\overline {X}}-\mu ){\bigg )}^{2}{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}{\bigg (}(X_{i}-\mu )^{2}-2({\overline {X}}-\mu )(X_{i}-\mu )+({\overline {X}}-\mu )^{2}{\bigg )}{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}-{\frac {2}{n}}({\overline {X}}-\mu )\sum _{i=1}^{n}(X_{i}-\mu )+{\frac {1}{n}}({\overline {X}}-\mu )^{2}\sum _{i=1}^{n}1{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}-{\frac {2}{n}}({\overline {X}}-\mu )\sum _{i=1}^{n}(X_{i}-\mu )+{\frac {1}{n}}({\overline {X}}-\mu )^{2}\cdot n{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}-{\frac {2}{n}}({\overline {X}}-\mu )\sum _{i=1}^{n}(X_{i}-\mu )+({\overline {X}}-\mu )^{2}{\bigg ]}\\[8pt]\end{aligned}}}$

To continue, we note that by subtracting ${\displaystyle \mu }$ from both sides of ${\displaystyle {\overline {X}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}}$, we get

${\displaystyle {\begin{aligned}{\overline {X}}-\mu ={\frac {1}{n}}\sum _{i=1}^{n}X_{i}-\mu ={\frac {1}{n}}\sum _{i=1}^{n}X_{i}-{\frac {1}{n}}\sum _{i=1}^{n}\mu \ ={\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu ).\\[8pt]\end{aligned}}}$

Meaning, (by cross-multiplication) ${\displaystyle n\cdot ({\overline {X}}-\mu )=\sum _{i=1}^{n}(X_{i}-\mu )}$. Then, the previous becomes:

${\displaystyle {\begin{aligned}\operatorname {E} [S^{2}]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}-{\frac {2}{n}}({\overline {X}}-\mu )\sum _{i=1}^{n}(X_{i}-\mu )+({\overline {X}}-\mu )^{2}{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}-{\frac {2}{n}}({\overline {X}}-\mu )\cdot n\cdot ({\overline {X}}-\mu )+({\overline {X}}-\mu )^{2}{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}-2({\overline {X}}-\mu )^{2}+({\overline {X}}-\mu )^{2}{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}-({\overline {X}}-\mu )^{2}{\bigg ]}\\[8pt]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}{\bigg ]}-\operatorname {E} {\bigg [}({\overline {X}}-\mu )^{2}{\bigg ]}\\[8pt]&=\sigma ^{2}-\operatorname {E} {\bigg [}({\overline {X}}-\mu )^{2}{\bigg ]}=\left(1-{\frac {1}{n}}\right)\sigma ^{2}<\sigma ^{2}.\end{aligned}}}$

This can be seen by noting the following formula, which follows from the Bienaymé formula, for the term in the inequality for the expectation of the uncorrected sample variance above: ${\displaystyle \operatorname {E} {\big [}({\overline {X}}-\mu )^{2}{\big ]}={\frac {1}{n}}\sigma ^{2}}$.

In other words, the expected value of the uncorrected sample variance does not equal the population variance σ², unless multiplied by a normalization factor. The sample mean, on the other hand, is an unbiased^[5] estimator of the population mean μ.^[3]

Note that the usual definition of sample variance is ${\displaystyle S^{2}={\frac {1}{n-1}}\sum _{i=1}^{n}(X_{i}-{\overline {X}}\,)^{2}}$, and this is an unbiased estimator of the population variance.

Algebraically speaking, ${\displaystyle \operatorname {E} [S^{2}]}$ is unbiased because:

${\displaystyle {\begin{aligned}\operatorname {E} [S^{2}]&=\operatorname {E} \left[{\frac {1}{n-1}}\sum _{i=1}^{n}{\big (}X_{i}-{\overline {X}}{\big )}^{2}\right]={\frac {n}{n-1}}\operatorname {E} \left[{\frac {1}{n}}\sum _{i=1}^{n}{\big (}X_{i}-{\overline {X}}{\big )}^{2}\right]\\[8pt]&={\frac {n}{n-1}}\left(1-{\frac {1}{n}}\right)\sigma ^{2}=\sigma ^{2},\\[8pt]\end{aligned}}}$

where the transition to the second line uses the result derived above for the biased estimator. Thus ${\displaystyle \operatorname {E} [S^{2}]=\sigma ^{2}}$, and therefore ${\displaystyle S^{2}={\frac {1}{n-1}}\sum _{i=1}^{n}(X_{i}-{\overline {X}}\,)^{2}}$ is an unbiased estimator of the population variance, σ². The ratio between the biased (uncorrected) and unbiased estimates of the variance is known as Bessel's correction.

The reason that an uncorrected sample variance, S², is biased stems from the fact that the sample mean is an ordinary least squares (OLS) estimator for μ: ${\displaystyle {\overline {X}}}$ is the number that makes the sum ${\displaystyle \sum _{i=1}^{n}(X_{i}-{\overline {X}})^{2}}$ as small as possible. That is, when any other number is plugged into this sum, the sum can only increase. In particular, the choice ${\displaystyle \mu \neq {\overline {X}}}$ gives,

${\displaystyle {\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-{\overline {X}})^{2}<{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2},}$

and then

${\displaystyle {\begin{aligned}\operatorname {E} [S^{2}]&=\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-{\overline {X}})^{2}{\bigg ]}<\operatorname {E} {\bigg [}{\frac {1}{n}}\sum _{i=1}^{n}(X_{i}-\mu )^{2}{\bigg ]}=\sigma ^{2}.\end{aligned}}}$

The above discussion can be understood in geometric terms: the vector ${\displaystyle {\vec {C}}=(X_{1}-\mu ,\ldots ,X_{n}-\mu )}$ can be decomposed into the "mean part" and "variance part" by projecting to the direction of ${\displaystyle {\vec {u}}=(1,\ldots ,1)}$ and to that direction's orthogonal complement hyperplane. One gets ${\displaystyle {\vec {A}}=({\overline {X}}-\mu ,\ldots ,{\overline {X}}-\mu )}$ for the part along ${\displaystyle {\vec {u}}}$ and ${\displaystyle {\vec {B}}=(X_{1}-{\overline {X}},\ldots ,X_{n}-{\overline {X}})}$ for the complementary part. Since this is an orthogonal decomposition, Pythagorean theorem says ${\displaystyle |{\vec {C}}|^{2}=|{\vec {A}}|^{2}+|{\vec {B}}|^{2}}$, and taking expectations we get ${\displaystyle n\sigma ^{2}=n\operatorname {E} \left[({\overline {X}}-\mu )^{2}\right]+n\operatorname {E} [S^{2}]}$, as above (but times ${\displaystyle n}$). If the distribution of ${\displaystyle {\vec {C}}}$ is rotationally symmetric, as in the case when ${\displaystyle X_{i}}$ are sampled from a Gaussian, then on average, the dimension along ${\displaystyle {\vec {u}}}$ contributes to ${\displaystyle |{\vec {C}}|^{2}}$ equally as the ${\displaystyle n-1}$ directions perpendicular to ${\displaystyle {\vec {u}}}$, so that ${\displaystyle \operatorname {E} \left[({\overline {X}}-\mu )^{2}\right]={\frac {\sigma ^{2}}{n}}}$ and ${\displaystyle \operatorname {E} [S^{2}]={\frac {(n-1)\sigma ^{2}}{n}}}$. This is in fact true in general, as explained above.

Estimating a Poisson probability

A far more extreme case of a biased estimator being better than any unbiased estimator arises from the Poisson distribution.^[6][7] Suppose that X has a Poisson distribution with expectation λ. Suppose it is desired to estimate

${\displaystyle \operatorname {P} (X=0)^{2}=e^{-2\lambda }\quad }$

with a sample of size 1. (For example, when incoming calls at a telephone switchboard are modeled as a Poisson process, and λ is the average number of calls per minute, then e^−2λ is the probability that no calls arrive in the next two minutes.)

Since the expectation of an unbiased estimator δ(X) is equal to the estimand, i.e.

${\displaystyle \operatorname {E} (\delta (X))=\sum _{x=0}^{\infty }\delta (x){\frac {\lambda ^{x}e^{-\lambda }}{x!}}=e^{-2\lambda },}$

the only function of the data constituting an unbiased estimator is

${\displaystyle \delta (x)=(-1)^{x}.\,}$

To see this, note that when decomposing e^−λ from the above expression for expectation, the sum that is left is a Taylor series expansion of e^−λ as well, yielding e^−λe^−λ = e^−2λ (see Characterizations of the exponential function).

If the observed value of X is 100, then the estimate is 1, although the true value of the quantity being estimated is very likely to be near 0, which is the opposite extreme. And, if X is observed to be 101, then the estimate is even more absurd: It is −1, although the quantity being estimated must be positive.

The (biased) maximum likelihood estimator

${\displaystyle e^{-2{X}}\quad }$

is far better than this unbiased estimator. Not only is its value always positive but it is also more accurate in the sense that its mean squared error

${\displaystyle e^{-4\lambda }-2e^{\lambda (1/e^{2}-3)}+e^{\lambda (1/e^{4}-1)}\,}$

is smaller; compare the unbiased estimator's MSE of

${\displaystyle 1-e^{-4\lambda }.\,}$

The MSEs are functions of the true value λ. The bias of the maximum-likelihood estimator is:

${\displaystyle e^{-2\lambda }-e^{\lambda (1/e^{2}-1)}.\,}$

Maximum of a discrete uniform distribution

The bias of maximum-likelihood estimators can be substantial. Consider a case where n tickets numbered from 1 through to n are placed in a box and one is selected at random, giving a value X. If n is unknown, then the maximum-likelihood estimator of n is X, even though the expectation of X given n is only (n + 1)/2; we can be certain only that n is at least X and is probably more. In this case, the natural unbiased estimator is 2X − 1.

Example: Estimation of population variance

For example,^[15] suppose an estimator of the form

${\displaystyle T^{2}=c\sum _{i=1}^{n}\left(X_{i}-{\overline {X}}\,\right)^{2}=cnS^{2}}$

is sought for the population variance as above, but this time to minimise the MSE:

${\displaystyle {\begin{aligned}\operatorname {MSE} =&\operatorname {E} \left[(T^{2}-\sigma ^{2})^{2}\right]\\=&\left(\operatorname {E} \left[T^{2}-\sigma ^{2}\right]\right)^{2}+\operatorname {Var} (T^{2})\end{aligned}}}$

If the variables X₁ ... X_n follow a normal distribution, then nS²/σ² has a chi-squared distribution with n − 1 degrees of freedom, giving:

${\displaystyle \operatorname {E} [nS^{2}]=(n-1)\sigma ^{2}{\text{ and }}\operatorname {Var} (nS^{2})=2(n-1)\sigma ^{4}.}$

and so

${\displaystyle \operatorname {MSE} =(c(n-1)-1)^{2}\sigma ^{4}+2c^{2}(n-1)\sigma ^{4}}$

With a little algebra it can be confirmed that it is c = 1/(n + 1) which minimises this combined loss function, rather than c = 1/(n − 1) which minimises just the square of the bias.

More generally it is only in restricted classes of problems that there will be an estimator that minimises the MSE independently of the parameter values.

However it is very common that there may be perceived to be a bias–variance tradeoff, such that a small increase in bias can be traded for a larger decrease in variance, resulting in a more desirable estimator overall.