Minimum mean square error estimation

The most common risk function used for Bayesian estimation is the mean square error (MSE), also called squared error risk. The MSE is defined by

${\displaystyle \mathrm {MSE} =E\left[({\widehat {\theta }}(x)-\theta )^{2}\right],}$

where the expectation is taken over the joint distribution of ${\displaystyle \theta }$ and ${\displaystyle x}$.

Bayes estimators for conjugate priors

If there is no inherent reason to prefer one prior probability distribution over another, a conjugate prior is sometimes chosen for simplicity. A conjugate prior is defined as a prior distribution belonging to some parametric family, for which the resulting posterior distribution also belongs to the same family. This is an important property, since the Bayes estimator, as well as its statistical properties (variance, confidence interval, etc.), can all be derived from the posterior distribution.

Conjugate priors are especially useful for sequential estimation, where the posterior of the current measurement is used as the prior in the next measurement. In sequential estimation, unless a conjugate prior is used, the posterior distribution typically becomes more complex with each added measurement, and the Bayes estimator cannot usually be calculated without resorting to numerical methods.

Following are some examples of conjugate priors.

• If ${\displaystyle x|\theta }$ is Normal, ${\displaystyle x|\theta \sim N(\theta ,\sigma ^{2})}$, and the prior is normal, ${\displaystyle \theta \sim N(\mu ,\tau ^{2})}$, then the posterior is also Normal and the Bayes estimator under MSE is given by

${\displaystyle {\widehat {\theta }}(x)={\frac {\sigma ^{2}}{\sigma ^{2}+\tau ^{2}}}\mu +{\frac {\tau ^{2}}{\sigma ^{2}+\tau ^{2}}}x.}$

• If ${\displaystyle x_{1},...,x_{n}}$ are iid Poisson random variables ${\displaystyle x_{i}|\theta \sim P(\theta )}$, and if the prior is Gamma distributed ${\displaystyle \theta \sim G(a,b)}$, then the posterior is also Gamma distributed, and the Bayes estimator under MSE is given by

${\displaystyle {\widehat {\theta }}(X)={\frac {n{\overline {X}}+a}{n+b}}.}$

• If ${\displaystyle x_{1},...,x_{n}}$ are iid uniformly distributed ${\displaystyle x_{i}|\theta \sim U(0,\theta )}$, and if the prior is Pareto distributed ${\displaystyle \theta \sim Pa(\theta _{0},a)}$, then the posterior is also Pareto distributed, and the Bayes estimator under MSE is given by

${\displaystyle {\widehat {\theta }}(X)={\frac {(a+n)\max {(\theta _{0},x_{1},...,x_{n})}}{a+n-1}}.}$

Alternative risk functions

Risk functions are chosen depending on how one measures the distance between the estimate and the unknown parameter. The MSE is the most common risk function in use, primarily due to its simplicity. However, alternative risk functions are also occasionally used. The following are several examples of such alternatives. We denote the posterior generalized distribution function by ${\displaystyle F}$.

Example

A typical example is estimation of a location parameter with a loss function of the type ${\displaystyle L(a-\theta )}$. Here ${\displaystyle \theta }$ is a location parameter, i.e., ${\displaystyle p(x|\theta )=f(x-\theta )}$.

It is common to use the improper prior ${\displaystyle p(\theta )=1}$ in this case, especially when no other more subjective information is available. This yields

${\displaystyle p(\theta |x)={\frac {p(x|\theta )p(\theta )}{p(x)}}={\frac {f(x-\theta )}{p(x)}}}$

so the posterior expected loss

${\displaystyle E[L(a-\theta )|x]=\int {L(a-\theta )p(\theta |x)d\theta }={\frac {1}{p(x)}}\int L(a-\theta )f(x-\theta )d\theta .}$

The generalized Bayes estimator is the value ${\displaystyle a(x)}$ that minimizes this expression for a given ${\displaystyle x}$. This is equivalent to minimizing

${\displaystyle \int L(a-\theta )f(x-\theta )d\theta }$ for a given ${\displaystyle x.}$        (1)

In this case it can be shown that the generalized Bayes estimator has the form ${\displaystyle x+a_{0}}$, for some constant ${\displaystyle a_{0}}$. To see this, let ${\displaystyle a_{0}}$ be the value minimizing (1) when ${\displaystyle x=0}$. Then, given a different value ${\displaystyle x_{1}}$, we must minimize

${\displaystyle \int L(a-\theta )f(x_{1}-\theta )d\theta =\int L(a-x_{1}-\theta ')f(-\theta ')d\theta '.}$        (2)

This is identical to (1), except that ${\displaystyle a}$ has been replaced by ${\displaystyle a-x_{1}}$. Thus, the expression minimizing is given by ${\displaystyle a-x_{1}=a_{0}}$, so that the optimal estimator has the form

${\displaystyle a(x)=a_{0}+x.\,\!}$

Example

The following is a simple example of parametric empirical Bayes estimation. Given past observations ${\displaystyle x_{1},\ldots ,x_{n}}$ having conditional distribution ${\displaystyle f(x_{i}|\theta _{i})}$, one is interested in estimating ${\displaystyle \theta _{n+1}}$ based on ${\displaystyle x_{n+1}}$. Assume that the ${\displaystyle \theta _{i}}$'s have a common prior ${\displaystyle \pi }$ which depends on unknown parameters. For example, suppose that ${\displaystyle \pi }$ is normal with unknown mean ${\displaystyle \mu _{\pi }\,\!}$ and variance ${\displaystyle \sigma _{\pi }\,\!.}$ We can then use the past observations to determine the mean and variance of ${\displaystyle \pi }$ in the following way.

First, we estimate the mean ${\displaystyle \mu _{m}\,\!}$ and variance ${\displaystyle \sigma _{m}\,\!}$ of the marginal distribution of ${\displaystyle x_{1},\ldots ,x_{n}}$ using the maximum likelihood approach:

${\displaystyle {\widehat {\mu }}_{m}={\frac {1}{n}}\sum {x_{i}},}$

${\displaystyle {\widehat {\sigma }}_{m}^{2}={\frac {1}{n}}\sum {(x_{i}-{\widehat {\mu }}_{m})^{2}}.}$

Next, we use the law of total expectation to compute ${\displaystyle \mu _{m}}$ and the law of total variance to compute ${\displaystyle \sigma _{m}^{2}}$ such that

${\displaystyle \mu _{m}=E_{\pi }[\mu _{f}(\theta )]\,\!,}$

${\displaystyle \sigma _{m}^{2}=E_{\pi }[\sigma _{f}^{2}(\theta )]+E_{\pi }[(\mu _{f}(\theta )-\mu _{m})^{2}],}$

where ${\displaystyle \mu _{f}(\theta )}$ and ${\displaystyle \sigma _{f}(\theta )}$ are the moments of the conditional distribution ${\displaystyle f(x_{i}|\theta _{i})}$, which are assumed to be known. In particular, suppose that ${\displaystyle \mu _{f}(\theta )=\theta }$ and that ${\displaystyle \sigma _{f}^{2}(\theta )=K}$; we then have

${\displaystyle \mu _{\pi }=\mu _{m}\,\!,}$

${\displaystyle \sigma _{\pi }^{2}=\sigma _{m}^{2}-\sigma _{f}^{2}=\sigma _{m}^{2}-K.}$

Finally, we obtain the estimated moments of the prior,

${\displaystyle {\widehat {\mu }}_{\pi }={\widehat {\mu }}_{m},}$

${\displaystyle {\widehat {\sigma }}_{\pi }^{2}={\widehat {\sigma }}_{m}^{2}-K.}$

For example, if ${\displaystyle x_{i}|\theta _{i}\sim N(\theta _{i},1)}$, and if we assume a normal prior (which is a conjugate prior in this case), we conclude that ${\displaystyle \theta _{n+1}\sim N({\widehat {\mu }}_{\pi },{\widehat {\sigma }}_{\pi }^{2})}$, from which the Bayes estimator of ${\displaystyle \theta _{n+1}}$ based on ${\displaystyle x_{n+1}}$ can be calculated.

Admissibility

Bayes rules having finite Bayes risk are typically admissible. The following are some specific examples of admissibility theorems.

• If a Bayes rule is unique then it is admissible.^[5] For example, as stated above, under mean squared error (MSE) the Bayes rule is unique and therefore admissible.
• If θ belongs to a discrete set, then all Bayes rules are admissible.
• If θ belongs to a continuous (non-discrete) set, and if the risk function R(θ,δ) is continuous in θ for every δ, then all Bayes rules are admissible.

By contrast, generalized Bayes rules often have undefined Bayes risk in the case of improper priors. These rules are often inadmissible and the verification of their admissibility can be difficult. For example, the generalized Bayes estimator of a location parameter θ based on Gaussian samples (described in the "Generalized Bayes estimator" section above) is inadmissible for ${\displaystyle p>2}$; this is known as Stein's phenomenon.

Asymptotic efficiency

Let θ be an unknown random variable, and suppose that ${\displaystyle x_{1},x_{2},\ldots }$ are iid samples with density ${\displaystyle f(x_{i}|\theta )}$. Let ${\displaystyle \delta _{n}=\delta _{n}(x_{1},\ldots ,x_{n})}$ be a sequence of Bayes estimators of θ based on an increasing number of measurements. We are interested in analyzing the asymptotic performance of this sequence of estimators, i.e., the performance of ${\displaystyle \delta _{n}}$ for large n.

To this end, it is customary to regard θ as a deterministic parameter whose true value is ${\displaystyle \theta _{0}}$. Under specific conditions,^[6] for large samples (large values of n), the posterior density of θ is approximately normal. In other words, for large n, the effect of the prior probability on the posterior is negligible. Moreover, if δ is the Bayes estimator under MSE risk, then it is asymptotically unbiased and it converges in distribution to the normal distribution:

${\displaystyle {\sqrt {n}}(\delta _{n}-\theta _{0})\to N\left(0,{\frac {1}{I(\theta _{0})}}\right),}$

where I(θ₀) is the Fisher information of θ₀. It follows that the Bayes estimator δ_n under MSE is asymptotically efficient.

Another estimator which is asymptotically normal and efficient is the maximum likelihood estimator (MLE). The relations between the maximum likelihood and Bayes estimators can be shown in the following simple example.