False discovery rate

Definitions

Based on definitions below we can define Q as the proportion of false discoveries among the discoveries (rejections of the null hypothesis): $${\displaystyle Q=\frac{V}{R}=\frac{V}{V+S}.}$$ where ${\displaystyle V}$ is the number of false discoveries and ${\displaystyle S}$ is the number of true discoveries.

The false discovery rate (FDR) is then simply the following: $${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}=Q_e=\mathrm{E}[Q],}$$ where ${\displaystyle \mathrm{E}[Q]}$ is the expected value of ${\displaystyle Q}$. The goal is to keep FDR below a given threshold q. To avoid division by zero, ${\displaystyle Q}$ is defined to be 0 when ${\displaystyle R=0}$. Formally, ${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}=\mathrm{E}[V/R\mid R>0]\cdot \mathrm{P}(R>0)}$.

Classification of multiple hypothesis tests

The following table defines the possible outcomes when testing multiple null hypotheses. Suppose we have a number m of null hypotheses, denoted by: H₁, H₂, ..., H_m. Using a statistical test, we reject the null hypothesis if the test is declared significant. We do not reject the null hypothesis if the test is non-significant. Summing each type of outcome over all H_i yields the following random variables:

• m is the total number hypotheses tested
• ${\displaystyle m_0}$ is the number of true null hypotheses, an unknown parameter
• ${\displaystyle m-m_0}$ is the number of true alternative hypotheses
• V is the number of false positives (Type I error) (also called "false discoveries")
• S is the number of true positives (also called "true discoveries")
• T is the number of false negatives (Type II error)
• U is the number of true negatives
• ${\displaystyle R=V+S}$ is the number of rejected null hypotheses (also called "discoveries", either true or false)

In m hypothesis tests of which ${\displaystyle m_0}$ are true null hypotheses, R is an observable random variable, and S, T, U, and V are unobservable random variables.

Controlling procedures

The settings for many procedures is such that we have ${\displaystyle H_1,\dots ,H_m}$ null hypotheses tested and ${\displaystyle P_1,\dots ,P_m}$ their corresponding p-values. We list these p-values in ascending order and denote them by ${\displaystyle P_{(1)},\dots ,P_{(m)}}$. A procedure that goes from a small test-statistic to a large one will be called a step-up procedure. In a similar way, in a "step-down" procedure we move from a large corresponding test statistic to a smaller one.

Benjamini–Hochberg procedure

The Benjamini–Hochberg procedure (BH step-up procedure) controls the FDR at level ${\displaystyle \alpha }$ under certain conditions, such as independent test statistics. It works as follows:

1. For a given ${\displaystyle \alpha }$, find the largest k for which ${\displaystyle P_{(k)}\le \frac{k}{m}\alpha }$
2. Reject the null hypothesis (i.e., declare discoveries) for all ${\displaystyle H_{(i)}}$ for ${\displaystyle i=1,\dots ,k}$

Geometrically, this corresponds to plotting ${\displaystyle P_{(k)}}$ vs. k (on the y and x axes respectively), drawing the line through the origin with slope ${\displaystyle \frac{\alpha }{m}}$ , and declaring discoveries for all points on the left, up to, and including the last point that is not above the line.

The BH procedure is valid when the m tests are independent, and also in various scenarios of dependence, but is not universally valid. It also satisfies the inequality: $${\displaystyle E(Q)\le \frac{m_0}{m}\alpha \le \alpha }$$ If an estimator of ${\displaystyle m_0}$ is inserted into the BH procedure, it is no longer guaranteed to achieve FDR control at the desired level. Adjustments may be needed in the estimator and several modifications have been proposed.

Note that the mean ${\displaystyle \alpha }$ for these m tests is ${\displaystyle \frac{\alpha (m+1)}{2m}}$, the Mean(FDR ${\displaystyle \alpha }$) or MFDR, ${\displaystyle \alpha }$ adjusted for m independent or positively correlated tests (see AFDR below). The MFDR expression here is for a single recomputed value of ${\displaystyle \alpha }$ and is not part of the Benjamini and Hochberg method.

Benjamini–Yekutieli procedure

The Benjamini–Yekutieli procedure controls the false discovery rate under arbitrary dependence assumptions. This refinement modifies the threshold and finds the largest k for which $${\displaystyle P_{(k)}\le \frac{k}{m\cdot c(m)}\alpha .}$$

• If the tests are independent or positively correlated (as in Benjamini–Hochberg procedure): ${\displaystyle c(m)=1}$
• Under arbitrary dependence (including the case of negative correlation), c(m) is the harmonic number: ${\displaystyle c(m)=\sum _{i=1}^m\frac{1}{i}}$. Note that ${\displaystyle c(m)}$ can be approximated by using the Taylor series expansion and the Euler–Mascheroni constant (${\displaystyle \gamma =0.57721\dots }$): $${\displaystyle \sum _{i=1}^m\frac{1}{i}\approx \ln (m)+\gamma +\frac{1}{2m}.}$$

Using MFDR and formulas above, an adjusted MFDR (or AFDR) is the minimum of the mean ${\displaystyle \alpha }$ for m dependent tests, i.e.,

${\displaystyle \frac{\mathrm{M}\mathrm{F}\mathrm{D}\mathrm{R}}{c(m)}=\frac{\alpha (m+1)}{2m[\ln (m)+\gamma ]+1}.}$

Another way to address dependence is by bootstrapping and rerandomization.

Storey–Tibshirani procedure

In the Storey–Tibshirani procedure, q-values are used for controlling the FDR.

Properties

Adaptive and scalable

Using a multiplicity procedure that controls the FDR criterion is adaptive and scalable. Meaning that controlling the FDR can be very permissive (if the data justify it), or conservative (acting close to control of FWER for sparse problem) - all depending on the number of hypotheses tested and the level of significance.

The FDR criterion adapts so that the same number of false discoveries (V) will have different implications, depending on the total number of discoveries (R). This contrasts with the family-wise error rate criterion. For example, if inspecting 100 hypotheses (say, 100 genetic mutations or SNPs for association with some phenotype in some population):

• If we make 4 discoveries (R), having 2 of them be false discoveries (V) is often very costly. Whereas,
• If we make 50 discoveries (R), having 2 of them be false discoveries (V) is often not very costly.

The FDR criterion is scalable in that the same proportion of false discoveries out of the total number of discoveries (Q), remains sensible for different number of total discoveries (R). For example:

• If we make 100 discoveries (R), having 5 of them be false discoveries (${\displaystyle q=5\mathrm{\%}}$) may not be very costly.
• Similarly, if we make 1000 discoveries (R), having 50 of them be false discoveries (as before, ${\displaystyle q=5\mathrm{\%}}$) may still not be very costly.

Dependency among the test statistics

Controlling the FDR using the linear step-up BH procedure, at level q, has several properties related to the dependency structure between the test statistics of the m null hypotheses that are being corrected for. If the test statistics are:

• Independent: $${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}\le \frac{m_0}{m}q}$$
• Independent and continuous: $${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}=\frac{m_0}{m}q}$$
• Positive dependent: $${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}\le \frac{m_0}{m}q}$$
• In the general case: $${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}\le \frac{m_0}{m}\frac{q}{1+\frac{1}{2}+\frac{1}{3}+\cdots +\frac{1}{m}}\approx \frac{m_0}{m}\frac{q}{\ln (m)+\gamma +\frac{1}{2m}},}$$ where ${\displaystyle \gamma }$ is the Euler–Mascheroni constant.

Proportion of true hypotheses

If all of the null hypotheses are true (${\displaystyle m_0=m}$), then controlling the FDR at level q guarantees control over the FWER (this is also called "weak control of the FWER"): ${\displaystyle \mathrm{F}\mathrm{W}\mathrm{E}\mathrm{R}=P\left(V\ge 1\right)=E\left(\frac{V}{R}\right)=\mathrm{F}\mathrm{D}\mathrm{R}\le q}$, simply because the event of rejecting at least one true null hypothesis ${\displaystyle \{V\ge 1\}}$ is exactly the event ${\displaystyle \{V/R=1\}}$, and the event ${\displaystyle \{V=0\}}$ is exactly the event ${\displaystyle \{V/R=0\}}$ (when ${\displaystyle V=R=0}$, ${\displaystyle V/R=0}$ by definition). But if there are some true discoveries to be made (${\displaystyle m_0<m}$) then FWER ≥ FDR. In that case there will be room for improving detection power. It also means that any procedure that controls the FWER will also control the FDR.

Average power

The average power of the Benjamini–Hochberg procedure can be computed analytically

Related concepts

The discovery of the FDR was preceded and followed by many other types of error rates. These include:

• PCER (per-comparison error rate) is defined as: ${\displaystyle \mathrm{P}\mathrm{C}\mathrm{E}\mathrm{R}=E\left[\frac{V}{m}\right]}$. Testing individually each hypothesis at level α guarantees that ${\displaystyle \mathrm{P}\mathrm{C}\mathrm{E}\mathrm{R}\le \alpha }$ (this is testing without any correction for multiplicity)
• FWER (the family-wise error rate) is defined as: ${\displaystyle \mathrm{F}\mathrm{W}\mathrm{E}\mathrm{R}=P(V\ge 1)}$. There are numerous procedures that control the FWER.
• ${\displaystyle k\text{-FWER}}$ (The tail probability of the False Discovery Proportion), suggested by Lehmann and Romano, van der Laan at al, is defined as: ${\displaystyle k\text{-FWER}=P(V\ge k)\le q}$.
• ${\displaystyle k\text{-FDR}}$ (also called the generalized FDR by Sarkar in 2007) is defined as: ${\displaystyle k\text{-FDR}=E\left(\frac{V}{R}{I}_{(V>k)}\right)\le q}$.
• ${\displaystyle Q^{\prime }}$ is the proportion of false discoveries among the discoveries", suggested by Soric in 1989, and is defined as: ${\displaystyle Q^{\prime }=\frac{E[V]}{R}}$. This is a mixture of expectations and realizations, and has the problem of control for ${\displaystyle m_0=m}$.
• ${\displaystyle {\mathrm{F}\mathrm{D}\mathrm{R}}_{-1}}$(or Fdr) was used by Benjamini and Hochberg, and later called "Fdr" by Efron (2008) and earlier. It is defined as: ${\displaystyle {\mathrm{F}\mathrm{D}\mathrm{R}}_{-1}=Fdr=\frac{E[V]}{E[R]}}$. This error rate cannot be strictly controlled because it is 1 when ${\displaystyle m=m_0}$.
• ${\displaystyle {\mathrm{F}\mathrm{D}\mathrm{R}}_{+1}}$ was used by Benjamini and Hochberg, and later called "pFDR" by Storey (2002). It is defined as: ${\displaystyle {\mathrm{F}\mathrm{D}\mathrm{R}}_{+1}=p\mathrm{F}\mathrm{D}\mathrm{R}=E\left[\frac{V}{R}|R>0\right]}$. This error rate cannot be strictly controlled because it is 1 when ${\displaystyle m=m_0}$. JD Storey promoted the use of the pFDR (a close relative of the FDR), and the q-value, which can be viewed as the proportion of false discoveries that we expect in an ordered table of results, up to the current line. Storey also promoted the idea (also mentioned by BH) that the actual number of null hypotheses, ${\displaystyle m_0}$, can be estimated from the shape of the probability distribution curve. For example, in a set of data where all null hypotheses are true, 50% of results will yield probabilities between 0.5 and 1.0 (and the other 50% will yield probabilities between 0.0 and 0.5). We can therefore estimate ${\displaystyle m_0}$ by finding the number of results with ${\displaystyle P>0.5}$ and doubling it, and this permits refinement of our calculation of the pFDR at any particular cut-off in the data-set.
• False exceedance rate (the tail probability of FDP), defined as: ${\displaystyle \mathrm{P}\left(\frac{V}{R}>q\right)}$
• ${\displaystyle W\text{-FDR}}$ (Weighted FDR). Associated with each hypothesis i is a weight ${\displaystyle w_i\ge 0}$, the weights capture importance/price. The W-FDR is defined as: ${\displaystyle W\text{-FDR}=E\left(\frac{\sum w_i{V}_i}{\sum w_i{R}_i}\right)}$.
• FDCR (False Discovery Cost Rate). Stemming from statistical process control: associated with each hypothesis i is a cost ${\displaystyle {\mathrm{c}}_i}$ and with the intersection hypothesis ${\displaystyle H_{00}}$ a cost ${\displaystyle c_0}$. The motivation is that stopping a production process may incur a fixed cost. It is defined as:

${\displaystyle \mathrm{F}\mathrm{D}\mathrm{C}\mathrm{R}=E\left(c_0{V}_0+\frac{\sum c_i{V}_i}{c_0{R}_0+\sum c_i{R}_i}\right)}$

• PFER (per-family error rate) is defined as: ${\displaystyle \mathrm{P}\mathrm{F}\mathrm{E}\mathrm{R}=E(V)}$.
• FNR (False non-discovery rates) by Sarkar; Genovese and Wasserman is defined as

${\displaystyle \mathrm{F}\mathrm{N}\mathrm{R}=E\left(\frac{T}{m-R}\right)=E\left(\frac{m-m_0-(R-V)}{m-R}\right)}$

• ${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}(z)}$ is defined as: ${\displaystyle \mathrm{F}\mathrm{D}\mathrm{R}(z)=\frac{p_0{F}_0(z)}{F(z)}}$
• ${\displaystyle \mathrm{f}\mathrm{d}\mathrm{r}}$, local-fdr is defined as: ${\displaystyle \mathrm{f}\mathrm{d}\mathrm{r}=\frac{p_0{f}_0(z)}{f(z)}}$ in a local interval of ${\displaystyle z}$.

False coverage rate

The false coverage rate (FCR) is, in a sense, the FDR analog to the confidence interval. FCR indicates the average rate of false coverage, namely, not covering the true parameters, among the selected intervals. The FCR gives a simultaneous coverage at a ${\displaystyle 1-\alpha }$ level for all of the parameters considered in the problem. Intervals with simultaneous coverage probability 1 − q can control the FCR to be bounded by q. There are many FCR procedures such as: Bonferroni-Selected–Bonferroni-Adjusted, Adjusted BH-Selected CIs (Benjamini and Yekutieli (2005)), Bayes FCR (Zhao and Hwang (2012)), and other Bayes methods.

Bayesian approaches

Connections have been made between the FDR and Bayesian approaches (including empirical Bayes methods), thresholding wavelets coefficients and model selection, and generalizing the confidence interval into the false coverage statement rate (FCR).

Structural false discovery rate (sFDR)

The structural false discovery rate (sFDR) is a generalization of the classical false discovery rate (FDR) introduced by D. Meskaldji and collaborators in 2018.

The sFDR extends the FDR by replacing the linear denominator R in the expected ratio E[V/R] with a non-decreasing concave function s(R), yielding the criterion E[V/s(R)]. This approach allows the control of false discoveries to adapt to the scale of testing, so that prudence increases faster than linearly as the number of rejections grows.

When s(R) = R, the classical FDR is recovered, while specific choices of s(R) can interpolate between FDR control and family-wise error control (k-FWER). The sFDR provides a structural connection between classical, local, and generalized false discovery concepts, and has been extended to online and adaptive settings.

Empirical false discovery rate (eFDR)

Conventional p-value adjustment methods, like the Bonferroni and Benjamini-Hochberg, often overcorrect the p-values when the input datasets are not independent but interconnected, which is often the case in biological data, like functional enrichment analysis of differentially expressed genes. This overcorrection resulted in potentially missing biologically relevant terms with significant enrichment. Empirical false discovery rate address this problem with the so-called "plug-in" estimate of the false discovery rate (Algorithm 18.3 of Hastie et al.), which is implemented within the mulea R package. This method is an empirical, resampling-based approach to calculating the false discovery rate (FDR), which we abbreviate as ${\displaystyle eFDR}$.

The description of the eFDR algorithm applied for functional enrichment analysis

For each ontology entry $(j=1,2,\dots ,J)$ and the investigated target set (e.g., significantly differentially expressed genes), mulea calculates a p-value $(p_j)$ based on the hypergeometric test. To assess the unbiased statistical significance of each ontology entry, we compute the empirical false discovery rate $(eFD{R}_j)$using a resampling-based approach.

First, we determine the rank of each ontology entry's p-value relative to the p-values of all ontology entries. $R_j$ refers to the rank of the p-value of the $j^{th}$ ontology entry. Here, we do note the indicator function with Iverson brackets: $I()$:

${\displaystyle R_j=\sum _{i=1}^{J}I\left(p_i\le p_j\right),j=1,\dots ,J}$

To calculate the expected rank $\left({\overline{R}}_j\right)$ of the p-value of the $j^{th}$ ontology entry, a resampling strategy is applied, where resampling steps are indexed with $(s=1,2,\dots ,S)$. In each resampling step, a simulated target set with the same size as the original target set is generated, but with randomly selected elements from the background set. Then we recalculate the hypergeometric tests and the ranks of the p-values $\left(R_j^s\right)$ for each resampling step. Let ${\overline{R}}_j$ be the expectation of the $R_j^s$ values over $s$:

${\displaystyle {\overline{R}}_j=\frac{{\displaystyle \sum _{s=1}^S{R}_j^s}}{S}}$

The $eFDR$ of the $j^{th}$ ontology entry $(eFD{R}_j)$ is calculated as the ratio of the expected rank $\left({\overline{R}}_j\right)$ to the actual rank $\left(R_j\right)$. If the calculated $eFD{R}_j$ exceeds $1$, it is truncated to $1$.

${\displaystyle eFD{R}_j=min\left(\frac{R_j}{{\overline{R}}_j},1\right)}$

Software implementations

• False Discovery Rate Analysis in R – Lists links with popular R packages
• False Discovery Rate Analysis in Python – Python implementations of false discovery rate procedures
• Empirical False Discovery Rate Analysis in R using the mulea package.