Derivation

The derivation of the bias–variance decomposition for squared error proceeds as follows.^[9][10] For notational convenience, we abbreviate ${\displaystyle f=f(x)}$, ${\displaystyle {\hat {f}}={\hat {f}}(x;D)}$ and we drop the ${\displaystyle D}$ subscript on our expectation operators.

Let us write the mean-squared error of our model:

${\displaystyle {\text{MSE}}\triangleq \operatorname {E} {\big [}(y-{\hat {f}})^{2}{\big ]}=\operatorname {E} {\big [}y^{2}-2y{\hat {f}}+{\hat {f}}^{2}{\big ]}=\operatorname {E} {\big [}y^{2}{\big ]}-2\operatorname {E} {\big [}y{\hat {f}}{\big ]}+\operatorname {E} {\big [}{\hat {f}}^{2}{\big ]}}$

First,

${\displaystyle {\begin{aligned}\operatorname {E} {\big [}{\hat {f}}^{2}{\big ]}&=\operatorname {Var} ({\hat {f}})+\operatorname {E} [{\hat {f}}]^{2}&&{\text{since }}\operatorname {Var} [X]\triangleq \operatorname {E} {\Big [}(X-\operatorname {E} [X])^{2}{\Big ]}=\operatorname {E} [X^{2}]-\operatorname {E} [X]^{2}{\text{ for any random variable }}X\end{aligned}}}$

Secondly, since we model ${\displaystyle y=f+\varepsilon }$, we show that

${\displaystyle {\begin{aligned}\operatorname {E} {\big [}y^{2}{\big ]}&=\operatorname {E} {\big [}(f+\varepsilon )^{2}{\big ]}\\&=\operatorname {E} [f^{2}]+2\operatorname {E} [f\varepsilon ]+\operatorname {E} [\varepsilon ^{2}]&&{\text{by linearity of }}\operatorname {E} \\&=f^{2}+2f\operatorname {E} [\varepsilon ]+\operatorname {E} [\varepsilon ^{2}]&&{\text{since }}f{\text{ does not depend on the data}}\\&=f^{2}+2f\cdot 0+\sigma ^{2}&&{\text{since }}\varepsilon {\text{ has zero mean and variance }}\sigma ^{2}\end{aligned}}}$

Lastly,

${\displaystyle {\begin{aligned}\operatorname {E} {\big [}y{\hat {f}}{\big ]}&=\operatorname {E} {\big [}(f+\varepsilon ){\hat {f}}{\big ]}\\&=\operatorname {E} [f{\hat {f}}]+\operatorname {E} [\varepsilon {\hat {f}}]&&{\text{by linearity of }}\operatorname {E} \\&=\operatorname {E} [f{\hat {f}}]+\operatorname {E} [\varepsilon ]\operatorname {E} [{\hat {f}}]&&{\text{since }}{\hat {f}}{\text{ and }}\varepsilon {\text{ are independent}}\\&=f\operatorname {E} [{\hat {f}}]&&{\text{since }}\operatorname {E} [\varepsilon ]=0\end{aligned}}}$

Eventually, we plug these 3 formulas in our previous derivation of ${\displaystyle {\text{MSE}}}$ and thus show that:

${\displaystyle {\begin{aligned}{\text{MSE}}&=f^{2}+\sigma ^{2}-2f\operatorname {E} [{\hat {f}}]+\operatorname {Var} [{\hat {f}}]+\operatorname {E} [{\hat {f}}]^{2}\\&=(f-\operatorname {E} [{\hat {f}}])^{2}+\sigma ^{2}+\operatorname {Var} {\big [}{\hat {f}}{\big ]}\\[5pt]&=\operatorname {Bias} [{\hat {f}}]^{2}+\sigma ^{2}+\operatorname {Var} {\big [}{\hat {f}}{\big ]}\end{aligned}}}$

Finally, MSE loss function (or negative log-likelihood) is obtained by taking the expectation value over ${\displaystyle x\sim P}$:

${\displaystyle {\text{MSE}}=\operatorname {E} _{x}{\bigg \{}\operatorname {Bias} _{D}[{\hat {f}}(x;D)]^{2}+\operatorname {Var} _{D}{\big [}{\hat {f}}(x;D){\big ]}{\bigg \}}+\sigma ^{2}.}$

k-nearest neighbors

In the case of k-nearest neighbors regression, when the expectation is taken over the possible labeling of a fixed training set, a closed-form expression exists that relates the bias–variance decomposition to the parameter k:^{[8]: 37, 223}

${\displaystyle \operatorname {E} [(y-{\hat {f}}(x))^{2}\mid X=x]=\left(f(x)-{\frac {1}{k}}\sum _{i=1}^{k}f(N_{i}(x))\right)^{2}+{\frac {\sigma ^{2}}{k}}+\sigma ^{2}}$

where ${\displaystyle N_{1}(x),\dots ,N_{k}(x)}$ are the k nearest neighbors of x in the training set. The bias (first term) is a monotone rising function of k, while the variance (second term) drops off as k is increased. In fact, under "reasonable assumptions" the bias of the first-nearest neighbor (1-NN) estimator vanishes entirely as the size of the training set approaches infinity.^[12]

In regression

The bias–variance decomposition forms the conceptual basis for regression regularization methods such as Lasso and ridge regression. Regularization methods introduce bias into the regression solution that can reduce variance considerably relative to the ordinary least squares (OLS) solution. Although the OLS solution provides non-biased regression estimates, the lower variance solutions produced by regularization techniques provide superior MSE performance.

In classification

The bias–variance decomposition was originally formulated for least-squares regression. For the case of classification under the 0-1 loss (misclassification rate), it is possible to find a similar decomposition.^[16][17] Alternatively, if the classification problem can be phrased as probabilistic classification, then the expected squared error of the predicted probabilities with respect to the true probabilities can be decomposed as before.^[18]

It has been argued that as training data increases, the variance of learned models will tend to decrease, and hence that as training data quantity increases, error is minimized by methods that learn models with lesser bias, and that conversely, for smaller training data quantities it is ever more important to minimize variance.^[19]

In reinforcement learning

Even though the bias–variance decomposition does not directly apply in reinforcement learning, a similar tradeoff can also characterize generalization. When an agent has limited information on its environment, the suboptimality of an RL algorithm can be decomposed into the sum of two terms: a term related to an asymptotic bias and a term due to overfitting. The asymptotic bias is directly related to the learning algorithm (independently of the quantity of data) while the overfitting term comes from the fact that the amount of data is limited.^[20]

In human learning

While widely discussed in the context of machine learning, the bias–variance dilemma has been examined in the context of human cognition, most notably by Gerd Gigerenzer and co-workers in the context of learned heuristics. They have argued (see references below) that the human brain resolves the dilemma in the case of the typically sparse, poorly-characterised training-sets provided by experience by adopting high-bias/low variance heuristics. This reflects the fact that a zero-bias approach has poor generalisability to new situations, and also unreasonably presumes precise knowledge of the true state of the world. The resulting heuristics are relatively simple, but produce better inferences in a wider variety of situations.^[21]

Geman et al.^[12] argue that the bias–variance dilemma implies that abilities such as generic object recognition cannot be learned from scratch, but require a certain degree of "hard wiring" that is later tuned by experience. This is because model-free approaches to inference require impractically large training sets if they are to avoid high variance.