Lehmann–Scheffé theorem

Statement

As above, let ${\displaystyle X=(X_1,X_2,\dots ,X_n)}$ be a vector of random samples from a distribution ${\displaystyle {\mathbb{P}}_{\theta }}$ for some parameter ${\displaystyle \theta \in \mathrm{\Theta }}$ and ${\displaystyle \mathrm{\Theta }}$ an arbitrary set.

Assume that there exists a complete, sufficient statistic ${\displaystyle T}$ for the family of distributions ${\displaystyle ({\mathbb{P}}_{\theta }{)}_{\theta \in \mathrm{\Theta }}}$. Then, the following two equivalent statements hold:

• There exists at most one measurable function ${\displaystyle h}$ such that ${\displaystyle T^{\ast }=h(T(X))}$ is unbiased for ${\displaystyle g(\theta )}$ and ${\displaystyle {\mathbb{E}}_{\theta }[T^{\ast }(X{)}^2]<\mathrm{\infty }}$ for all ${\displaystyle \theta \in \mathrm{\Theta }}$, in which case ${\displaystyle T^{\ast }(X)}$ is the unique UMVUE for ${\displaystyle g(\theta )}$.
• For any unbiased estimator ${\displaystyle S(X)}$, if it exists, with ${\displaystyle {\mathbb{E}}_{\theta }[S(X{)}^2]<\mathrm{\infty }}$ for all ${\displaystyle \theta \in \mathrm{\Theta }}$, the estimator ${\displaystyle T^{\ast }(X)={\mathbb{E}}_{\theta }[S(X)\mid T(X)]}$ is the unique UMVUE for ${\displaystyle g(\theta )}$.

In fact, the theorem does not state that unbiased estimators exist in the first place. However, if they do, then there exists a unique square-integrable UMVUE. Moreover, the estimator ${\displaystyle T^{\ast }(X)={\mathbb{E}}_{\theta }[S(X)\mid T(X)]}$ does neither depend on ${\displaystyle \theta }$, since ${\displaystyle T}$ is sufficient, nor on ${\displaystyle S}$, since ${\displaystyle T}$ is also complete.

Proof

In the following, the dependence of an estimator on the data ${\displaystyle X}$ will not be written out explicitly, i.e., we write ${\displaystyle S}$ instead of ${\displaystyle S(X)}$.

First of all, if there is no unbiased estimator for ${\displaystyle g(\theta )}$, then there is obviously no UMVUE, and if all unbiased estimator are not square-integrable, then their variances are infinity and the statement is trivial. Thus, we focus on the case where a square-integrable unbiased estimator exists.

Uniqueness of ${\displaystyle h}$: Let ${\displaystyle S=h_1(T)}$ and ${\displaystyle R=h_2(T)}$ be unbiased estimators of ${\displaystyle g(\theta )}$ for some measurable functions ${\displaystyle h_1}$ and ${\displaystyle h_2}$. For the expectation of the difference it holds$${\displaystyle {\mathbb{E}}_{\theta }[h_1(T)-h_2(T)]=g(\theta )-g(\theta )=0.}$$Since ${\displaystyle T}$ is complete, this implies ${\displaystyle h_1-h_2=0}$. Thus, ${\displaystyle T^{\ast }=h_1(T)=:h(T)}$ is the unique unbiased estimator that is a function of ${\displaystyle T}$.

${\displaystyle h(T)}$ is the UMVUE: Let ${\displaystyle S}$ be any square-integrable unbiased estimator and ${\displaystyle S^{\ast }={\mathbb{E}}_{\theta }[S\mid T]}$. By the factorization lemma there exists measurable function ${\displaystyle \tilde{h}}$ such that ${\displaystyle S^{\ast }=\tilde{h}(T)}$ and since ${\displaystyle S^{\ast }}$ is unbiased as well, by the above, it must hold ${\displaystyle S^{\ast }=\tilde{h}(T)=h(T)=T^{\ast }}$. Thus, by the Rao–Blackwell theorem, it follows$${\displaystyle {\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(T^{\ast })={\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(S^{\ast })={\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }({\mathbb{E}}_{\theta }[S\mid T])\le {\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(S),\mathrm{\forall }\theta \in \mathrm{\Theta }.}$$In other words, ${\displaystyle T^{\ast }=h(T)}$ is an UMVUE and according to the first part it is unique.

Application

The Lehmann–Scheffé theorem motivates two general methods to construct UMVU estimators for ${\displaystyle g(\theta )}$ in models which allow for a complete sufficient statistic ${\displaystyle T}$.

Method 1: Determining the function ${\displaystyle h}$

The UMVUE, if it exists, is the (unique) solution of the equation$${\displaystyle {\mathbb{E}}_{\theta }[h(T)]=g(\theta )}$$for all ${\displaystyle \theta \in \mathrm{\Theta }}$. If ${\displaystyle h}$ is, for example, a linear function, then solving this equation is fairly easy.

Method 2: Conditioning on an unbiased estimator

First, it suffices to find any unbiased estimator ${\displaystyle S}$ of ${\displaystyle g(\theta )}$, which is often easily feasible. The UMVUE can then be determined by evaluating the condition expectation ${\displaystyle {\mathbb{E}}_{\theta }[S\mid T]}$. Since the choice of ${\displaystyle S}$ is arbitrary, it is preferable to choose it such that the conditional expectation is as simple as possible.

Examples

Bernoulli distribution

Let ${\displaystyle X_1,\dots ,X_n}$ be Bernoulli-distributed with probability ${\displaystyle p\in (0,1)}$. The joint probability mass function ${\displaystyle f_p}$ is of the form$${\displaystyle f_p(x_1,\dots ,x_n)=\prod _{i=1}^n{p}^{x_i}(1-p{)}^{1-x_i}={\left(\frac{p}{1-p}\right)}^{\sum _{i=1}^n{x}_i}(1-p{)}^n,x_i\in \{0,1\},}$$thus, by the Fisher–Neyman factorization theorem, ${\displaystyle T(x)=\sum _{i=1}^n{x}_i}$ is a sufficient statistic. It is also complete: Let ${\displaystyle \phi }$ be any measurable function such that ${\displaystyle {\mathbb{E}}_{\theta }[\phi (T(X))]=0}$. Since ${\displaystyle T}$ is ${\displaystyle \mathrm{B}\mathrm{e}\mathrm{r}(n,p)}$-distributed, this means$${\displaystyle 0={\mathbb{E}}_{\theta }[\phi (T(X))]=\sum _{k=0}^n\phi (k)(\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}=(1-p{)}^n\sum _{k=0}^n\phi (k)(\genfrac{}{}{0ex}{}{n}{k})r^k,}$$with ${\displaystyle r=p/(1-p)}$. Since the right hand side is a polynomial in ${\displaystyle r>0}$ that is equal to zero, each coefficient must be zero as well, implying ${\displaystyle \phi (k)=0}$ for all ${\displaystyle k}$.

For the estimation of the parameter ${\displaystyle p}$, it is now easy to see that the UMVUE is the sample mean$${\displaystyle \overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{1}{n}T(X),}$$since it is unbiased and a function of ${\displaystyle T(X)}$.

Finding the UMVUE for the parameter ${\displaystyle p(1-p)}$ (that is, the variance of the distribution) is less obvious. According to method 1, we seek a function ${\displaystyle h}$ such that $${\displaystyle p(1-p)={\mathbb{E}}_p[h(T(X))]=\sum _{k=0}^{n}h(k)(\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}.}$$By defining ${\displaystyle \rho =p/(1-p)}$, the above rewrites as$${\displaystyle \sum _{k=0}^{n}h(k)(\genfrac{}{}{0ex}{}{n}{k}){\rho }^k=\rho (1+\rho {)}^{n-2}=\sum _{k=1}^{n-1}(\genfrac{}{}{0ex}{}{n-1}{k-1}){\rho }^k.}$$Comparing the coefficients shows that ${\displaystyle h(k)=k(n-k)/(n(n-1))}$, thus, the UMVUE is given by$${\displaystyle h(T(X))=\frac{n}{n-1}\overline{X}(1-\overline{X}).}$$Noting that in the Bernoulli model ${\displaystyle X_i=X_i^2}$ and that ${\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2=\sum _{i=1}^n{X}_i^2-n{\overline{X}}^2}$, the UMVUE is, in fact, the unbiased sample variance ${\displaystyle S_n^2}$.

Uniform distribution

Let ${\displaystyle X_1,\dots ,X_n}$ be uniformly distributed on the interval ${\displaystyle (0,\theta )}$ for some ${\displaystyle \theta }$ to be determined. The joint probability mass function ${\displaystyle f_p}$ is of the form$${\displaystyle f_p(x_1,\dots ,x_n)=\prod _{i=1}^n\frac{1}{\theta }{1}_{(0,\theta )}(x_i)=\frac{1}{{\theta }^n}{1}_{[0,\mathrm{\infty })}\left(\underset{i=1,\dots ,n}{min}{x}_i\right)1_{(-\mathrm{\infty },\theta ]}\left(\underset{i=1,\dots ,n}{max}{x}_i\right),x\in {\mathbb{R}}^n,}$$thus, by the Fisher–Neyman factorization theorem, ${\displaystyle T(x)=\underset{i=1,\dots ,n}{max}{x}_i}$ is a sufficient statistic.

To see the completeness of ${\displaystyle T}$, the distribution of ${\displaystyle T(X)}$ is

$${\displaystyle {\mathbb{P}}_{\theta }(T(X)\le t)={\mathbb{P}}_{\theta }(X_i\le t,\mathrm{\forall }i=1,\dots ,n)=\prod _{i=1}^n{\mathbb{P}}_{\theta }(X_i\le t)=\prod _{i=1}^n\frac{t}{\theta }=\frac{t^n}{{\theta }^n}.}$$

Thus, for any measurable function ${\displaystyle \phi }$ such that ${\displaystyle {\mathbb{E}}_{\theta }[\phi (T(X))]=0}$ it holds for any ${\displaystyle \theta >0}$

$${\displaystyle 0={\mathbb{E}}_{\theta }[\phi (T(X))]={\int }_0^{\theta }\phi (x)\frac{n}{{\theta }^n}{x}^{n-1}\mathrm{d}x⟹{\int }_0^{\theta }\phi (x)x^{n-1}\mathrm{d}x=0,}$$

and consequently for any ${\displaystyle b>a>0}$

$${\displaystyle {\int }_a^b\phi (x)x^{n-1}\mathrm{d}x={\int }_0^b\phi (x)x^{n-1}\mathrm{d}x-{\int }_0^a\phi (x)x^{n-1}\mathrm{d}x=0.}$$

This implies that ${\displaystyle \phi =0}$ (almost everywhere) on ${\displaystyle [0,\mathrm{\infty })}$. So ${\displaystyle T}$ is also complete. Its expectation is

$${\displaystyle {\mathbb{E}}_{\theta }[T(X)]={\int }_0^{\theta }x\frac{n}{{\theta }^n}{x}^{n-1}\mathrm{d}x=\frac{n}{{\theta }^n}\frac{{\theta }^{n+1}}{n+1}=\frac{n}{n+1}\theta .}$$

Thus, the rescaled estimator

$${\displaystyle T^{\ast }(X)=\frac{n+1}{n}T(X)=\frac{n+1}{n}\underset{i=1,\dots ,n}{max}{X}_i}$$

is unbiased and since it is a function of ${\displaystyle T}$, it is the UMVUE for ${\displaystyle \theta }$.

Exponential distribution

Let ${\displaystyle X_1,\dots ,X_n}$ be exponentially distributed with parameter ${\displaystyle \lambda }$ and suppose the parameter ${\displaystyle g(\lambda )={\mathbb{P}}_{\lambda }(X_1>t)}$ should be estimated for some fixed ${\displaystyle t>0}$. A straightforward unbiased estimator for ${\displaystyle g(\lambda )}$ would be ${\displaystyle \widehat{g(\lambda )}=\frac{1}{n}\sum _{i=1}^n{1}_{\{X_i>t\}}}$, however, it turns out to be sub-optimal.

Since the exponential distribution is an exponential family, it is known that its sufficient statistic ${\displaystyle \overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i}$ is also complete. Moreover, the estimator ${\displaystyle T=1_{\{X_1>t\}}}$ is unbiased. Thus, according to method 2, the estimator ${\displaystyle T^{\ast }=\mathbb{E}[T\mid \overline{X}]=\mathbb{P}(X_1>t\mid \overline{X})}$ is the UMVUE. It remains to evaluate the conditional expectation.

First of all, ${\displaystyle X_1/(n\overline{X})\sim \mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}(1,n-1)}$, which is independent of ${\displaystyle \lambda }$ (also called ancillary). In fact, since ${\displaystyle X_1}$ and ${\displaystyle S=\sum _{i=2}^n{X}_i}$ are independent and ${\displaystyle S\sim \mathrm{\Gamma }(n-1,\lambda )}$, the cumulative distribution function at ${\displaystyle y\in (0,1)}$ reads as

$${\displaystyle \begin{array}{cc}\mathbb{P}\left(\frac{X_1}{n\overline{X}}\le y\right)& =\mathbb{P}\left(\frac{X_1}{X_1+S}\le y\right)\\ & ={\int }_0^{\mathrm{\infty }}{\int }_0^{\mathrm{\infty }}{1}_{\{x/(x+s)\le y\}}\lambda e^{-\lambda x}\frac{{\lambda }^{n-1}}{(n-2)!}{s}^{n-2}{e}^{-\lambda s}\mathrm{d}x\mathrm{d}s\\ & ={\int }_0^1{\int }_0^{\mathrm{\infty }}{1}_{\{u\le y\}}\frac{{\lambda }^n}{(n-2)!}{r}^{n-2}(1-u{)}^{n-2}{e}^{-\lambda r}r\mathrm{d}r\mathrm{d}u\\ & ={\int }_0^1{1}_{\{u\le y\}}(1-u{)}^{n-2}\frac{1}{(n-2)!}\underset{=(n-1)!}{\underbrace{{\int }_0^{\mathrm{\infty }}{\lambda }^n{r}^{n-1}{e}^{-\lambda r}\mathrm{d}r}}\mathrm{d}u\\ & ={\int }_0^y(n-1)(1-u{)}^{n-2}\mathrm{d}u,\end{array}}$$

with the substitution ${\displaystyle x=ru}$ and ${\displaystyle s=r(1-u)}$. Thus, by Basu's theorem, ${\displaystyle X_1/(n\overline{X})}$ and ${\displaystyle \overline{X}}$ are independent and the conditional expectation simplifies to

$${\displaystyle \mathbb{P}(X_1>t\mid \overline{X}=x)=\mathbb{P}\left(\frac{X_1}{n\overline{X}}>\frac{t}{n\overline{X}}\mid \overline{X}=x\right)=\mathbb{P}\left(\frac{X_1}{n\overline{X}}>\frac{t}{nx}\right)={\left(1-\frac{t}{nx}\right)}^{n-1}.}$$

due to the above derivation. The constructed estimator

$${\displaystyle T^{\ast }={\left(1-\frac{t}{n\overline{X}}\right)}^{n-1}}$$

is unbiased and has a smaller variance than the naive estimator ${\displaystyle \widehat{g(\lambda )}}$ for all possible ${\displaystyle \lambda >0}$.

Counterexample with incomplete statistics

An example of an improvable Rao–Blackwell improvement, when using a minimal sufficient statistic that is not complete, was provided by Galili and Meilijson in 2016. Let ${\displaystyle X_1,\dots ,X_n}$ be a random sample from a scale-uniform distribution ${\displaystyle X\sim U((1-k)\theta ,(1+k)\theta ),}$ with unknown mean ${\displaystyle \mathrm{E}[X]=\theta }$ and known design parameter ${\displaystyle k\in (0,1)}$. In the search for "best" possible unbiased estimators for ${\displaystyle \theta }$, it is natural to consider ${\displaystyle X_1}$ as an initial (crude) unbiased estimator for ${\displaystyle \theta }$ and then try to improve it. Since ${\displaystyle X_1}$ is not a function of ${\displaystyle T=\left(X_{(1)},X_{(n)}\right)}$, the minimal sufficient statistic for ${\displaystyle \theta }$ (where ${\displaystyle X_{(1)}=\underset{i}{min}{X}_i}$ and ${\displaystyle X_{(n)}=\underset{i}{max}{X}_i}$), it may be improved using the Rao–Blackwell theorem as follows:

${\displaystyle {\hat{\theta }}_{RB}={\mathrm{E}}_{\theta }[X_1\mid X_{(1)},X_{(n)}]=\frac{X_{(1)}+X_{(n)}}{2}.}$

However, the following unbiased estimator can be shown to have lower variance:

${\displaystyle {\hat{\theta }}_{LV}=\frac{1}{k^2\frac{n-1}{n+1}+1}\cdot \frac{(1-k)X_{(1)}+(1+k)X_{(n)}}{2}.}$

And in fact, it could be even further improved when using the following estimator:

${\displaystyle {\hat{\theta }}_{\text{BAYES}}=\frac{n+1}{n}\left[1-\frac{\frac{X_{(1)}(1+k)}{X_{(n)}(1-k)}-1}{{\left(\frac{X_{(1)}(1+k)}{X_{(n)}(1-k)}\right)}^{n+1}-1}\right]\frac{X_{(n)}}{1+k}}$

The model is a scale model. Optimal equivariant estimators can then be derived for loss functions that are invariant.