Almost sure hypothesis testing

Description

For simplicity, assume we have a sequence of independent and identically distributed normal random variables, $x_{i} \sim N (μ, 1)$ , with mean $μ$ , and unit variance. Suppose that nature or simulation has chosen the true mean to be $μ_{0}$ , then the probability distribution function of the mean, $μ$ , is given by

Pr (μ \leq t) = [t \in [μ_{0}, + \infty]]

where an Iverson bracket has been used. A naïve approach to estimating this distribution function would be to replace true mean on the right hand side with an estimate such as the sample mean, $\hat{μ}$ , but

\begin{matrix} E [t \in [\hat{μ}, + \infty]] = Pr (\hat{μ} \leq t) \\ = & Φ (\sqrt{n} (t - μ_{0})) \to Pr (μ \leq t) - 0.5 [μ_{0} = t] \end{matrix}

which means the approximation to the true distribution function will be off by 0.5 at the true mean. However, $[\hat{μ}, + \infty]$ is nothing more than a one-sided 50% confidence interval; more generally, let $Z_{α_{n}}$ be the critical value used in a one-sided $1 - α_{n}$ confidence interval, then

E [t \in [\hat{μ} - \frac{Z_{α_{n}}}{\sqrt{n}}, + \infty]] \to Pr (μ \leq t) - lim_{n \to + \infty} α_{n} [μ_{0} = t]

If we set $α_{n} = 0.05$ , then the error of the approximation is reduced from 0.5 to 0.05, which is a factor of 10. Of course, if we let $α_{n} \to 0$ , then

E [t \in [\hat{μ} - \frac{Z_{α_{n}}}{\sqrt{n}}, + \infty]] \to Pr (μ \leq t)

However, this only shows that the expectation is close to the limiting value. Naaman (2016) showed that setting the significance level at $α_{n} = n^{- p}$ with $p > 1$ results in a finite number of type I and type II errors w.p.1 under fairly mild regularity conditions. This means that for each $t$ , there exists an $N (t)$ , such that for all $n > N (t)$ ,

[t \in [\hat{μ} - \frac{Z_{α_{n}}}{\sqrt{n}}, + \infty]] = Pr (μ \leq t)

where the equality holds w.p. 1. So the indicator function of a one-sided a.s. confidence interval is a good approximation to the true distribution function.

One more chapter is free with a Lably account.

Applications

Optional stopping

For example, suppose a researcher performed an experiment with a sample size of 10 and found no statistically significant result. Then suppose she decided to add one more observation, and retest continuing this process until a significant result was found. Under this scenario, given the initial batch of 10 observations resulted in an insignificant result, the probability that the experiment will be stopped at some finite sample size, $N_{s}$ , can be bounded using Boole's inequality

Pr (N_{s} < + \infty) < \sum_{n = 11}^{+ \infty} α_{n} < 0.0952

where $α_{n} = n^{- 2}$ . This compares favorably with fixed significance level testing which has a finite stopping time with probability one; however, this bound will not be meaningful for all sequences of significance level, as the above sum can be greater than one (setting $α_{n} = n^{- 1.2}$ would be one example). But even using that bandwidth, if the testing was done in batches of 10, then

Pr (N_{s} < + \infty) < \sum_{i = 2}^{\infty} {(10 i)}^{- 1.2} < 0.3

which results in a relatively large probability that the process will never end.

Publication bias

As another example of the power of this approach, if an academic journal only accepts papers with p-values less than 0.05, then roughly 1 in 20 independent studies of the same effect would find a significant result when there was none. However, if the journal required a minimum sample size of 100 and a maximum significance level is given by $α_{n} < n^{- 1.2}$ , then one would expect roughly 1 in 250 studies would find an effect when there was none (if the minimum sample size was 30, it would still be 1 in 60). If the maximum significance level was given by $α_{n} < n^{- 2}$ (which will have better small sample performance with regard to type I error when multiple comparisons are a concern), one would expect roughly 1 in 10000 studies would find an effect when there was none (if the minimum sample size was 30, it would be 1 in 900). Additionally, A.S. hypothesis testing is robust to the multiple comparisons.

Jeffreys–Lindley paradox

Lindley's paradox occurs when

The result is "significant" by a frequentist test, at, for example, the 5% level, indicating sufficient evidence to reject the null hypothesis, and
The posterior probability of the null hypothesis is high, indicating strong evidence that the null hypothesis is in better agreement with the data than is the alternative hypothesis.

However, the paradox does not apply to a.s. hypothesis tests. The Bayesian and the frequentist will eventually reach the same conclusion.