Statistical distance
Distance between two statistical objects
- 2 min read
- 4 chapters
Related on Lably
Through Statistics, Distance and Random variable
A distance between populations can be interpreted as measuring the distance between two probability distributions and hence they are essentially measures of distances between probability measures. Where statistical distance measures relate to the differences between random variables, these may have statistical dependence, and hence these distances are not directly related to measures of distances between probability measures. Again, a measure of distance between random variables may relate to the extent of dependence between them, rather than to their individual values.
Many statistical distance measures are not metrics, and some are not symmetric. Some types of distance measures, which generalize squared distance, are referred to as (statistical) divergences.
Terminology
Many terms are used to refer to various notions of distance; these are often confusingly similar, and may be used inconsistently between authors and over time, either loosely or with precise technical meaning. In addition to "distance", similar terms include deviance, deviation, discrepancy, discrimination, and divergence, as well as others such as contrast function and metric. Terms from information theory include cross entropy, relative entropy, discrimination information, and information gain.
Sign in to keep reading
3 more chapters are free with a Lably account.
Distances as metrics
Metrics
A metric on a set X is a function (called the distance function or simply distance) d : X × X → R+ (where R+ is the set of non-negative real numbers). For all x, y, z in X, this function is required to satisfy the following conditions:
- d(x, y) ≥ 0 (non-negativity)
- d(x, y) = 0 if and only if x = y (identity of indiscernibles. Note that condition 1 and 2 together produce positive definiteness)
- d(x, y) = d(y, x) (symmetry)
- d(x, z) ≤ d(x, y) + d(y, z) (subadditivity / triangle inequality).
Generalized metrics
Many statistical distances are not metrics, because they lack one or more properties of proper metrics. For example, pseudometrics violate property (2), identity of indiscernibles; quasimetrics violate property (3), symmetry; and semimetrics violate property (4), the triangle inequality. Statistical distances that satisfy (1) and (2) are referred to as divergences.
Statistically close
The total variation distance of two distributions and over a finite domain , (often referred to as statistical difference or statistical distance in cryptography) is defined as
.
We say that two probability ensembles and are statistically close if is a negligible function in .
Examples
Metrics
- Total variation distance (sometimes just called "the" statistical distance)
- Hellinger distance
- Lévy–Prokhorov metric
- Wasserstein metric: also known as the Kantorovich metric, or earth mover's distance
- Mahalanobis distance
- Integral probability metrics generalize several metrics or pseudometrics on distributions
Divergences
- Kullback–Leibler divergence
- Rényi divergence
- Jensen–Shannon divergence
- Ball divergence
- Bhattacharyya distance (despite its name it is not a distance, as it violates the triangle inequality)
- f-divergence: generalizes several distances and divergences
- Discriminability index, specifically the Bayes discriminability index, is a positive-definite symmetric measure of the overlap of two distributions.