Precision and recall

Definition

For classification tasks, the terms true positives, true negatives, false positives, and false negatives compare the results of the classifier under test with trusted external judgments. The terms positive and negative refer to the classifier's prediction (sometimes known as the expectation), and the terms true and false refer to whether that prediction corresponds to the external judgment (sometimes known as the observation).

Let us define an experiment from P positive instances and N negative instances for some condition. The four outcomes can be formulated in a 2×2 contingency table or confusion matrix, as follows:

Precision and recall are then defined as:

$${\displaystyle \begin{array}{cc}\text{Precision}& =\frac{tp}{tp+fp}\\ \text{Recall}& =\frac{tp}{tp+fn}\end{array}}$$

Recall in this context is also referred to as the true positive rate or sensitivity, and precision is also referred to as positive predictive value (PPV); other related measures used in classification include true negative rate and accuracy. True negative rate is also called specificity.

$${\displaystyle \text{True negative rate}=\frac{tn}{tn+fp}}$$

Precision vs. recall

Both precision and recall may be useful in cases where there is imbalanced data. However, it may be valuable to prioritize one metric over the other in cases where the outcome of a false positive or false negative is costly. For example, in medical diagnosis, a false positive test can lead to unnecessary treatment and expenses. In this situation, it is useful to value precision over recall. In other cases, the cost of a false negative is high, and recall may be a more valuable metric. For instance, the cost of a false negative in fraud detection is high, as failing to detect a fraudulent transaction can result in significant financial loss.

Probabilistic definition

Precision and recall can be interpreted as (estimated) conditional probabilities: Precision is given by ${\displaystyle \mathbb{P}(C=P|\hat{C}=P)}$ while recall is given by ${\displaystyle \mathbb{P}(\hat{C}=P|C=P)}$, where ${\displaystyle \hat{C}}$ is the predicted class and ${\displaystyle C}$ is the actual class (i.e. ${\displaystyle C=P}$ means the actual class is positive). Both quantities are, therefore, connected by Bayes' theorem.

No-skill classifiers

The probabilistic interpretation allows to easily derive how a no-skill classifier would perform. A no-skill classifier is defined by the property that the joint probability ${\displaystyle \mathbb{P}(C=P,\hat{C}=P)=\mathbb{P}(C=P)\mathbb{P}(\hat{C}=P)}$ is just the product of the unconditional probabilities since the classification and the presence of the class are independent.

For example, the precision of a no-skill classifier is simply a constant ${\displaystyle \mathbb{P}(C=P|\hat{C}=P)=\frac{\mathbb{P}(C=P,\hat{C}=P)}{\mathbb{P}(\hat{C}=P)}=\mathbb{P}(C=P),}$ i.e. determined by the probability/frequency with which the class P occurs.

A similar argument can be made for the recall: ${\displaystyle \mathbb{P}(\hat{C}=P|C=P)=\frac{\mathbb{P}(C=P,\hat{C}=P)}{\mathbb{P}(C=P)}=\mathbb{P}(\hat{C}=P)}$ which is the probability for a positive classification.

Accuracy, imbalanced data, and balanced accuracy

$${\displaystyle \text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}}$$

Accuracy can be expressed as the weighted average of how well a classifier identifies positive cases and negative cases, weighted by the condition's base prevalence ($P$). This formulation relies on sensitivity (the true positive rate) and specificity (the true negative rate):

$${\displaystyle \text{Accuracy}=(P\times \text{Sensitivity})+((1-P)\times \text{Specificity})}$$

In machine learning terminology, sensitivity is equivalent to the recall of the positive class (often termed $recal{l}_1$), while specificity is equivalent to the recall of the negative class (often termed $recal{l}_0$). Because this equation scales the true positive and true negative rates directly by the actual number of positive and negative cases available in the dataset, it remains mathematically strictly bounded across all prevalence levels.

Alternatively, Accuracy can be expressed to depend on both precision ($P_{recision}$), recall ($R_{ecall}$, i.e., $recal{l}_1$), and prevalence ($P$). By expressing accuracy as 1 minus the total error rate, the accuracy of a classifier is given by:

$${\displaystyle \text{Accuracy}=1-P\left[(1-R_{ecall})+R_{ecall}\left(\frac{1-P_{recision}}{P_{recision}}\right)\right]}$$

In this form, the equation isolates how accuracy is penalized by two distinct errors scaled by prevalence: the rate at which positive cases are missed ($\text{FN rate}=1-R_{ecall}$), and the rate of collateral false positives generated by catching them ($\text{FP rate}=R_{ecall}\left(\frac{1-P_{recision}}{P_{recision}}\right)$). This explicit relationship highlights how changes in precision and recall impact overall accuracy disproportionately depending on the underlying prevalence in an imbalanced dataset.

However, because prevalence, precision, and recall are not independent variables, this formula is strictly bound by a mathematical limit. The mathematically implied number of false positives cannot exceed the total number of actual negative cases in the dataset. Therefore, the accuracy formula above is only valid subject to the following constraint on minimum precision:

$${\displaystyle P_{recision}\ge \frac{P\cdot R_{ecall}}{1-P+P\cdot R_{ecall}}}$$

Because of the above, Accuracy can be a limited/misleading metric when comparing data sets with different base prevalence (i.e. differently imbalanced data sets). For example, a method with 90% precision and 50% recall would yield an accuracy of either 83% or 50%, depending if the prevalence in the dataset is 30% or 90%. Also, if a sample has 95 negative and 5 positive values, then classifying all values as negative (100% recall and 0% precision) would give 0.95 accuracy score.

There are many metrics that don't suffer from this problem. For example, balanced accuracy (bACC) normalizes true positive and true negative predictions by the number of positive and negative samples, respectively, and divides their sum by two:

$${\displaystyle \text{Balanced accuracy}=\frac{TPR+TNR}{2}}$$

For the previous example (95 negative and 5 positive samples), classifying all as negative gives 0.5 balanced accuracy score (the maximum bACC score is one), which is equivalent to the expected value of a random guess in a balanced data set. Balanced accuracy can serve as an overall performance metric for a model, whether or not the true labels are imbalanced in the data, assuming the cost of FN is the same as FP.

Balanced accuracy is a linear combination of Youden's J statistic. (${\displaystyle \text{Balanced Accuracy}=\frac{J+1}{2}}$)

The TPR and FPR are a property of a given classifier operating at a specific threshold. However, the overall number of TPs, FPs etc. depend on the class imbalance in the data via the class ratio $r=P/N$. As the recall (or TPR) depends only on positive cases, it is not affected by $r$, but the precision is. We have that

${\displaystyle \text{Precision}=\frac{TP}{TP+FP}=\frac{P\cdot TPR}{P\cdot TPR+N\cdot FPR}=\frac{TPR}{TPR+\frac{1}{r}FPR}.}$

Thus the precision has an explicit dependence on $r$. Starting with balanced classes at $r=1$ and gradually decreasing $r$, the corresponding precision will decrease, because the denominator increases.

Another metric is the predicted positive condition rate (PPCR), which identifies the percentage of the total population that is flagged. For example, for a search engine that returns 30 results (retrieved documents) out of 1,000,000 documents, the PPCR is 0.003%.

$${\displaystyle \text{Predicted positive condition rate}=\frac{TP+FP}{TP+FP+TN+FN}}$$

According to Saito and Rehmsmeier, precision-recall plots are more informative than ROC plots when evaluating binary classifiers on imbalanced data. In such scenarios, ROC plots may be visually deceptive with respect to conclusions about the reliability of classification performance.

Different from the above approaches, if an imbalance scaling is applied directly by weighting the confusion matrix elements, the standard metrics definitions still apply even in the case of imbalanced datasets. The weighting procedure relates the confusion matrix elements to the support set of each considered class.

F-measure

A measure that combines precision and recall is the harmonic mean of precision and recall, the traditional F-measure or balanced F-score:

$${\displaystyle F=2\cdot \frac{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\cdot \mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

This measure is approximately the average of the two when they are close, and is more generally the harmonic mean, which, for the case of two numbers, coincides with the square of the geometric mean divided by the arithmetic mean. There are several reasons that the F-score can be criticized, in particular circumstances, due to its bias as an evaluation metric. This is also known as the ${\displaystyle F_1}$ measure, because recall and precision are evenly weighted.

It is a special case of the general ${\displaystyle F_{\beta }}$ measure (for non-negative real values of ${\displaystyle \beta }$):

$${\displaystyle F_{\beta }=(1+{\beta }^2)\cdot \frac{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\cdot \mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{{\beta }^2\cdot \mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

Two other commonly used ${\displaystyle F}$ measures are the ${\displaystyle F_2}$ measure, which weights recall higher than precision, and the ${\displaystyle F_{0.5}}$ measure, which puts more emphasis on precision than recall.

The F-measure was derived by van Rijsbergen (1979) so that ${\displaystyle F_{\beta }}$ "measures the effectiveness of retrieval with respect to a user who attaches ${\displaystyle \beta }$ times as much importance to recall as precision". It is based on van Rijsbergen's effectiveness measure ${\displaystyle E_{\alpha }=1-\frac{1}{\frac{\alpha }{P}+\frac{1-\alpha }{R}}}$, the second term being the weighted harmonic mean of precision and recall with weights ${\displaystyle (\alpha ,1-\alpha )}$. Their relationship is ${\displaystyle F_{\beta }=1-E_{\alpha }}$ where ${\displaystyle \alpha =\frac{1}{1+{\beta }^2}}$.

Limitations as goals

There are other parameters and strategies for performance metric of information retrieval system, such as the area under the ROC curve (AUC) or pseudo-R-squared.

Multi-class evaluation

Precision and recall values can also be calculated for classification problems with more than two classes. To obtain the precision for a given class, we divide the number of true positives by the classifier bias towards this class (number of times that the classifier has predicted the class). To calculate the recall for a given class, we divide the number of true positives by the prevalence of this class (number of times that the class occurs in the data sample).

The class-wise precision and recall values can then be combined into an overall multi-class evaluation score, e.g., using the macro F1 metric.

Generalisation to continuous distributions

The classical definitions of precision, recall, and F-score are formulated for binary classification, where each instance is either true or false. Many practical domains, however, produce inherently continuous signals rather than discrete labels. Generalising these metrics to continuous distributions enables their direct application to such data without the information loss introduced by thresholding.

Two motivating examples illustrate the limitation of binary frameworks:

• Quantitative finance. A model may output a continuous buy/sell signal proportional to its conviction for each asset at a given moment. Applying a fixed threshold (e.g. signal > 0.1 = buy) discards conviction magnitude and conflates strong and mild signals, limiting the model's ability to correctly distinguish a strong buy from a mild one.
• DNA methylation analysis. Sequencing tools detect differentially methylated regions whose underlying signal is continuous, typically ranging from −1 to +1. Forcing this into a binary framework introduces similar distortions by treating all positive deviations as equivalent regardless of magnitude.

Definitions

Let ${\displaystyle x_i}$ denote the expected (ground-truth) signal and ${\displaystyle y_i}$ the observed (predicted) signal at each data point ${\displaystyle i}$. Then, precision, recall, and F-score can be calculated as:

${\displaystyle \text{Precision}=\frac{\sum \frac{2x_i{y}_i}{x_i^2+y_i^2}{y}_i^2}{\sum y_i^2}}$

${\displaystyle \text{Recall}=\frac{\sum \frac{2x_i{y}_i}{x_i^2+y_i^2}{x}_i^2}{\sum x_i^2}}$

${\displaystyle F=\frac{\sum 2x_i{y}_i}{\sum x_i^2+y_i^2}}$

Note, sums may be replaced with integrals for continuous distributions depending on other continuous variables (i.e. ${\displaystyle \sum \to \int d\xi }$, ${\displaystyle x_i\to x(\xi )}$, ${\displaystyle y_i\to y(\xi )}$).

These definitions reduce to the classical binary versions when ${\displaystyle x_i,y_i\in \{0,1\}}$. However, continuous metrics reveal a deeper mathematical elegance. Consider a completely wrong ML model with random signal. Traditional binary metrics, blind to magnitude and sign, can still award such a model positive precision, recall and/or F-score (even 25% or higher for dense signal). The continuous formulas, by contrast, return exactly 0 for precision, recall, and F-score in this situation – because when expected and observed signals do not correlate, the numerator terms cancel to zero.