CDF-based nonparametric confidence interval

Properties of the bounds

Unlike approaches that make asymptotic assumptions, including bootstrap approaches and those that rely on the central limit theorem, CDF-based bounds are valid for finite sample sizes. And unlike bounds based on inequalities such as Hoeffding's and McDiarmid's inequalities, CDF-based bounds use properties of the entire sample and thus often produce significantly tighter bounds.

CDF bounds

When producing bounds on the CDF, we must differentiate between pointwise and simultaneous bands.

Pointwise band

A pointwise CDF bound is one which only guarantees their coverage probability of ${\displaystyle 1-\alpha }$ percent on any individual point of the empirical CDF. Because of the relaxed guarantees, these intervals can be much smaller.

One method of generating them is based on the binomial distribution. Considering a single point of a CDF of value ${\displaystyle F(x_i)}$, then the empirical distribution at that point will be distributed proportional to the binomial distribution with ${\displaystyle p=F(x_i)}$ and ${\displaystyle n}$ set equal to the number of samples in the empirical distribution. Thus, any of the methods available for generating a binomial proportion confidence interval can be used to generate a CDF bound as well.

Simultaneous band

CDF-based confidence intervals require a probabilistic bound on the CDF of the distribution from which the sample were generated. A variety of methods exist for generating confidence intervals for the CDF of a distribution, ${\displaystyle F}$, given an i.i.d. sample drawn from the distribution. These methods are all based on the empirical distribution function (empirical CDF). Given an i.i.d. sample of size n, ${\displaystyle x_1,\dots ,x_n\sim F}$, the empirical CDF is defined to be

${\displaystyle {\hat{F}}_n(t)=\frac{1}{n}\sum _{i=1}^{n}1\{x_i\le t\},}$

where ${\displaystyle 1\{A\}}$ is the indicator of event A. The Dvoretzky–Kiefer–Wolfowitz inequality, whose tight constant was determined by Massart, places a confidence interval around the Kolmogorov–Smirnov statistic between the CDF and the empirical CDF. Given an i.i.d. sample of size n from ${\displaystyle F}$, the bound states

${\displaystyle P(\underset{x}{sup}|F(x)-F_n(x)|>\epsilon )\le 2e^{-2n{\epsilon }^2}.}$

This can be viewed as a confidence envelope that runs parallel to, and is equally above and below, the empirical CDF.

The interval that contains the true CDF, ${\displaystyle F(x)}$, with probability ${\displaystyle 1-\alpha }$ is often specified as

${\displaystyle F_n(x)-\epsilon \le F(x)\le F_n(x)+\epsilon \text{ where }\epsilon =\sqrt{\frac{\ln \frac{2}{\alpha }}{2n}}}$

The equally spaced confidence interval around the empirical CDF allows for different rates of violations across the support of the distribution. In particular, it is more common for a CDF to be outside of the CDF bound estimated using the Dvoretzky–Kiefer–Wolfowitz inequality near the median of the distribution than near the endpoints of the distribution. In contrast, the order statistics-based bound introduced by Learned-Miller and DeStefano allows for an equal rate of violation across all of the order statistics. This in turn results in a bound that is tighter near the ends of the support of the distribution and looser in the middle of the support. Other types of bounds can be generated by varying the rate of violation for the order statistics. For example, if a tighter bound on the distribution is desired on the upper portion of the support, a higher rate of violation can be allowed at the upper portion of the support at the expense of having a lower rate of violation, and thus a looser bound, for the lower portion of the support.

A nonparametric bound on the mean

Assume without loss of generality that the support of the distribution is contained in ${\displaystyle [0,1].}$ Given a confidence envelope for the CDF of ${\displaystyle F}$ it is easy to derive a corresponding confidence interval for the mean of ${\displaystyle F}$. It can be shown that the CDF that maximizes the mean is the one that runs along the lower confidence envelope, ${\displaystyle L(x)}$, and the CDF that minimizes the mean is the one that runs along the upper envelope, ${\displaystyle U(x)}$. Using the identity

${\displaystyle E(X)={\int }_0^1(1-F(x))dx,}$

the confidence interval for the mean can be computed as

${\displaystyle \left[{\int }_0^1(1-U(x))dx,{\int }_0^1(1-L(x))dx\right].}$

A nonparametric bound on the variance

Assume without loss of generality that the support of the distribution of interest, ${\displaystyle F}$, is contained in ${\displaystyle [0,1]}$. Given a confidence envelope for ${\displaystyle F}$, it can be shown that the CDF within the envelope that minimizes the variance begins on the lower envelope, has a jump discontinuity to the upper envelope, and then continues along the upper envelope. Further, it can be shown that this variance-minimizing CDF, F', must satisfy the constraint that the jump discontinuity occurs at ${\displaystyle E[F^{\prime }]}$. The variance maximizing CDF begins on the upper envelope, horizontally transitions to the lower envelope, then continues along the lower envelope. Explicit algorithms for calculating these variance-maximizing and minimizing CDFs are given by Romano and Wolf.

Bounds on other statistical functionals

The CDF-based framework for generating confidence intervals is very general and can be applied to a variety of other statistical functionals including

• Entropy
• Mutual information
• Arbitrary percentiles