Rao–Blackwell theorem

The theorem

Mean-squared-error version

One case of Rao–Blackwell theorem states:

The mean squared error of the Rao–Blackwell estimator does not exceed that of the original estimator.

In other words,

${\displaystyle \mathrm{E}\left[({\delta }_1(X)-\theta {)}^2\right]\le \mathrm{E}\left[(\delta (X)-\theta {)}^2\right].}$

The essential tools of the proof besides the definition above are the law of total expectation and the fact that for any random variable ${\displaystyle Y}$, ${\displaystyle \mathrm{E}\left[Y^2\right]}$ cannot be less than ${\displaystyle {\left(\mathrm{E}[Y]\right)}^2}$. That inequality is a case of Jensen's inequality, although it may also be shown to follow instantly from the frequently mentioned fact that

${\displaystyle 0\le \mathrm{Var}[Y]=\mathrm{E}\left[(Y-\mathrm{E}[Y]{)}^2\right]=\mathrm{E}\left[Y^2\right]-{\left(\mathrm{E}[Y]\right)}^2.}$

More precisely, the mean square error of the Rao-Blackwell estimator has the following decomposition

${\displaystyle \mathrm{E}[({\delta }_1(X)-\theta {)}^2]=\mathrm{E}[(\delta (X)-\theta {)}^2]-\mathrm{E}\left[\mathrm{Var}[\delta (X)\mid T(X)]\right]}$

Since ${\displaystyle \mathrm{E}\left[\mathrm{Var}[\delta (X)\mid T(X)]\right]\ge 0}$, the Rao-Blackwell theorem immediately follows.

Convex loss generalization

The more general version of the Rao–Blackwell theorem speaks of the "expected loss" or risk function:

${\displaystyle \mathrm{E}[L({\delta }_1(X))]\le \mathrm{E}[L(\delta (X))]}$

where the "loss function" ${\displaystyle L}$ may be any convex function. If the loss function is twice-differentiable, as in the case for mean-squared-error, then we have the sharper inequality

${\displaystyle \mathrm{E}[L(\delta (X))]-\mathrm{E}[L({\delta }_1(X))]\ge \frac{1}{2}{\mathrm{E}}_T\left[\underset{x}{inf}{L}^{″}(x)\mathrm{Var}[\delta (X)\mid T]\right].}$

Properties

The improved estimator is unbiased if and only if the original estimator is unbiased, as may be seen at once by using the law of total expectation. The theorem holds regardless of whether biased or unbiased estimators are used.

The theorem seems very weak: it says only that the Rao–Blackwell estimator is no worse than the original estimator. In practice, however, the improvement is often enormous.

Examples

A Poisson-distribution example

Phone calls arrive at a switchboard according to a Poisson process at an average rate of λ per minute. This rate is not observable, but the numbers ${\displaystyle X_1,\dots ,X_n}$ of phone calls that arrived during n successive one-minute periods are observed. It is desired to estimate the probability e^−λ that the next one-minute period passes with no phone calls.

An extremely crude estimator of the desired probability is

${\displaystyle {\delta }_0=\{\begin{array}{cc}1& \text{if}{X}_1=0,\\ 0& \text{otherwise,}\end{array}}$

i.e., it estimates this probability to be 1 if no phone calls arrived in the first minute and zero otherwise. Despite the apparent limitations of this estimator, the result given by its Rao–Blackwellization is a very good estimator.

The sum

${\displaystyle S_n=\sum _{i=1}^n{X}_i=X_1+\cdots +X_n}$

can be readily shown to be a sufficient statistic for λ, i.e., the conditional distribution of the data ${\displaystyle X_1,\dots ,X_n}$, depends on λ only through this sum. Therefore, we find the Rao–Blackwell estimator

${\displaystyle {\delta }_1=\mathrm{E}\left[{\delta }_0\mid S_n=s_n\right].}$

After doing some algebra we have

${\displaystyle \begin{array}{cc}{\delta }_1& =\mathrm{E}\left[{\mathbf{1}}_{\{X_1=0\}}|\sum _{i=1}^n{X}_i=s_n\right]\\ & =P\left[X_1=0|\sum _{i=1}^n{X}_i=s_n\right]\\ & =P\left[X_1=0,\sum _{i=2}^n{X}_i=s_n\right]\times P{\left[\sum _{i=1}^n{X}_i=s_n\right]}^{-1}\\ & =e^{-\lambda }\frac{{\left((n-1)\lambda \right)}^{s_n}{e}^{-(n-1)\lambda }}{s_n!}\times {\left(\frac{(n\lambda {)}^{s_n}{e}^{-n\lambda }}{s_n!}\right)}^{-1}\\ & =\frac{{\left((n-1)\lambda \right)}^{s_n}{e}^{-n\lambda }}{s_n!}\times \frac{s_n!}{(n\lambda {)}^{s_n}{e}^{-n\lambda }}\\ & ={\left(1-\frac{1}{n}\right)}^{s_n}\end{array}}$

Since the expected total number of calls arriving during the first n minutes is nλ, one might not be surprised if this estimator has a fairly high probability (if n is big, by WLLN, the sample average converges in probability to the parameter λ) of being close to

${\displaystyle {\left(1-\frac{1}{n}\right)}^{n\lambda }\approx e^{-\lambda }.}$

So ${\displaystyle {\delta }_1}$ is clearly a very much improved estimator of that last quantity. In fact, since ${\displaystyle S_n}$ is complete and ${\displaystyle {\delta }_0}$ is unbiased, ${\displaystyle {\delta }_1}$ is the unique minimum variance unbiased estimator by the Lehmann–Scheffé theorem.

A uniform-distribution example

Suppose n independent positive samples are taken from a uniform distribution with an unknown upper bound ${\displaystyle \theta }$, that is, ${\displaystyle X_1,\dots ,X_n\sim U(0,\theta )\text{ i.i.d.},}$ and the aim is to estimate this parameter ${\displaystyle \theta }$. This is a continuous (and thus simpler) version of the German tank problem.

In this setting, the maximal value statistic ${\displaystyle M\triangleq T(X)\triangleq {max}_i{X}_i}$ is sufficient, as given ${\displaystyle M}$, one (random) sample would equal it, and the others would be uniformly and independently distributed between 0 and ${\displaystyle M}$; this statistic is also complete.

There are two natural estimators for the upper bound ${\displaystyle \theta }$, which improve as n grows. The first is twice the sample mean. This is unbiased (as ${\displaystyle \mathrm{E}[X_i]=\theta /2}$ for each ${\displaystyle i}$), but is not necessarily consistent - it may even be lower than the observed maximum ${\displaystyle M}$, and then clearly also lower than ${\displaystyle \theta }$. Thus it cannot be optimal (i.e., of lowest variance).

The second natural estimator is ${\displaystyle M}$ itself, but it is obviously too low and thus biased; intuitively, it should be "inflated". The factor by which it may be inflated in order to become unbiased can be found using the beta distribution of the uniform order statistics, but just doing that is unmotivated and has no further guarantees.

We can then apply the Rao-Blackwell procedure using the first of the suggestions above as the crude estimator. Actually, as the procedure is linear, we can even just use ${\displaystyle \delta (X)\triangleq 2X_1}$ - it would be a bad estimator by itself (not using the whole information, its error does not improve with n), but that is perfectly fine for Rao-Blackwellization. Moving from the mean of the samples to just picking one of them keeps the logic the same, only casting it in the language of probabilities rather than expectations.

So, what is the conditional expectation of ${\displaystyle X_1}$ given ${\displaystyle M}$? It has a probability of 1/n to be the maximal observed value ${\displaystyle M}$ itself, otherwise it is uniformly distributed in ${\displaystyle [0,M]}$ and thus has an expectation of ${\displaystyle M/2}$. Hence, finally, the Rao-Blackwell estimator is

${\displaystyle {\delta }_1(X)=\mathrm{E}[\delta (X)\mid T(X)]=\mathrm{E}[2X_1\mid M]=2\left(\frac{1}{n}\cdot M+\frac{n-1}{n}\cdot \frac{M}{2}\right)=\frac{n+1}{n}M}$.

This is indeed the correctly-inflated version of ${\displaystyle M}$ we were hoping for (inflating less and less as the number of samples n grows). It is unbiased, and again, by the Lehmann–Scheffé theorem, it is the (unique) minimum-variance such estimator.

Idempotence

Rao–Blackwellization is an idempotent operation. Using it to improve the already improved estimator does not obtain a further improvement, but merely returns as its output the same improved estimator.

Completeness and Lehmann–Scheffé minimum variance

If the conditioning statistic is both complete and sufficient, and the starting estimator is unbiased, then the Rao–Blackwell estimator is the unique "best unbiased estimator": see Lehmann–Scheffé theorem.

An example of an improvable Rao–Blackwell improvement, when using a minimal sufficient statistic that is not complete, was provided by Galili and Meilijson in 2016. Let ${\displaystyle X_1,\dots ,X_n}$ be a random sample from a scale-uniform distribution ${\displaystyle X\sim U\left((1-k)\theta ,(1+k)\theta \right),}$ with unknown mean ${\displaystyle E[X]=\theta }$ and known design parameter ${\displaystyle k\in (0,1)}$. In the search for "best" possible unbiased estimators for ${\displaystyle \theta ,}$ it is natural to consider ${\displaystyle X_1}$ as an initial (crude) unbiased estimator for ${\displaystyle \theta }$ and then try to improve it. Since ${\displaystyle X_1}$ is not a function of ${\displaystyle T=\left(X_{(1)},X_{(n)}\right)}$, the minimal sufficient statistic for ${\displaystyle \theta }$ (where ${\displaystyle X_{(1)}=min(X_i)}$ and ${\displaystyle X_{(n)}=max(X_i)}$), it may be improved using the Rao–Blackwell theorem as follows:

${\displaystyle {\hat{\theta }}_{RB}=E_{\theta }\left[X_1|X_{(1)},X_{(n)}\right]=\frac{X_{(1)}+X_{(n)}}{2}.}$

However, the following unbiased estimator can be shown to have lower variance:

${\displaystyle {\hat{\theta }}_{LV}=\frac{1}{2\left(k^2\frac{n-1}{n+1}+1\right)}\left[(1-k)X_{(1)}+(1+k)X_{(n)}\right].}$

And in fact, it could be even further improved when using the following estimator:

${\displaystyle {\hat{\theta }}_{BAYES}=\frac{n+1}{n}\left[1-\frac{\frac{\left(\frac{X_{(1)}}{1-k}\right)}{\left(\frac{X_{(n)}}{1+k}\right)}-1}{{\left[\frac{\left(\frac{X_{(1)}}{1-k}\right)}{\left(\frac{X_{(n)}}{1+k}\right)}\right]}^{n+1}-1}\right]\frac{X_{(n)}}{1+k}}$

The model is a scale model. Optimal equivariant estimators can then be derived for loss functions that are invariant.