Convergence of measures

Total variation convergence of measures

This is the strongest notion of convergence shown on this page and is defined as follows. Let ${\displaystyle (X,\mathcal{F})}$ be a measurable space. The total variation distance between two (positive) measures μ and ν is then given by

${\displaystyle {\Vert \mu -\nu \Vert }_{\text{TV}}=\underset{f}{sup}\left\{{\int }_{X}fd\mu -{\int }_{X}fd\nu \right\}.}$

Here the supremum is taken over f ranging over the set of all measurable functions from X to [−1, 1]. This is in contrast, for example, to the Wasserstein metric, where the definition is of the same form, but the supremum is taken over f ranging over the set of those measurable functions from X to [−1, 1] which have Lipschitz constant at most 1; and also in contrast to the Radon metric, where the supremum is taken over f ranging over the set of continuous functions from X to [−1, 1]. In the case where X is a Polish space, the total variation metric coincides with the Radon metric.

If μ and ν are both probability measures, then the total variation distance is also given by

${\displaystyle {\Vert \mu -\nu \Vert }_{\text{TV}}=2\cdot \underset{A\in \mathcal{F}}{sup}|\mu (A)-\nu (A)|.}$

The equivalence between these two definitions can be seen as a particular case of the Monge–Kantorovich duality. From the two definitions above, it is clear that the total variation distance between probability measures is always between 0 and 2.

To illustrate the meaning of the total variation distance, consider the following thought experiment. Assume that we are given two probability measures μ and ν, as well as a random variable X. We know that X has law either μ or ν but we do not know which one of the two. Assume that these two measures have prior probabilities 0.5 each of being the true law of X. Assume now that we are given one single sample distributed according to the law of X and that we are then asked to guess which one of the two distributions describes that law. The quantity

${\displaystyle \frac{2+\Vert \mu -\nu {\Vert }_{\text{TV}}}{4}}$

then provides a sharp upper bound on the prior probability that our guess will be correct.

Given the above definition of total variation distance, a sequence μ_n of measures defined on the same measure space is said to converge to a measure μ in total variation distance if for every ε > 0, there exists an N such that for all n > N, one has that

${\displaystyle \Vert {\mu }_n-\mu {\Vert }_{\text{TV}}<\epsilon .}$

Setwise convergence of measures

For ${\displaystyle (X,\mathcal{F})}$ a measurable space, a sequence μ_n is said to converge setwise to a limit μ if

${\displaystyle \underset{n\to \mathrm{\infty }}{lim}{\mu }_n(A)=\mu (A)}$

for every set ${\displaystyle A\in \mathcal{F}}$.

Typical arrow notations are ${\displaystyle {\mu }_n\stackrel{sw}{\to }\mu }$ and ${\displaystyle {\mu }_n\stackrel{s}{\to }\mu }$.

For example, as a consequence of the Riemann–Lebesgue lemma, the sequence μ_n of measures on the interval [−1, 1] given by μ_n(dx) = (1 + sin(nx))dx converges setwise to Lebesgue measure, but it does not converge in total variation.

In a measure theoretical or probabilistic context setwise convergence is often referred to as strong convergence (as opposed to weak convergence). This can lead to some ambiguity because in functional analysis, strong convergence usually refers to convergence with respect to a norm.

Weak convergence of measures

In mathematics and statistics, weak convergence is one of many types of convergence relating to the convergence of measures. It depends on a topology on the underlying space and thus is not a purely measure-theoretic notion.

There are several equivalent definitions of weak convergence of a sequence of measures, some of which are (apparently) more general than others. The equivalence of these conditions is sometimes known as the Portmanteau theorem.

Definition. Let ${\displaystyle S}$ be a metric space with its Borel ${\displaystyle \sigma }$-algebra ${\displaystyle \mathrm{\Sigma }}$. A bounded sequence of positive probability measures ${\displaystyle P_n(n=1,2,\dots )}$ on ${\displaystyle (S,\mathrm{\Sigma })}$ is said to converge weakly to a probability measure ${\displaystyle P}$ (denoted ${\displaystyle P_n\Rightarrow P}$) if any of the following equivalent conditions is true (here ${\displaystyle {\mathrm{E}}_n}$ denotes expectation or the integral with respect to ${\displaystyle P_n}$, while ${\displaystyle \mathrm{E}}$ denotes expectation or the integral with respect to ${\displaystyle P}$):

• ${\displaystyle {\mathrm{E}}_n[f]\to \mathrm{E}[f]}$ for all bounded, continuous functions ${\displaystyle f}$;
• ${\displaystyle {\mathrm{E}}_n[f]\to \mathrm{E}[f]}$ for all bounded and Lipschitz functions ${\displaystyle f}$;
• ${\displaystyle lim sup{\mathrm{E}}_n[f]\le \mathrm{E}[f]}$ for every upper semi-continuous function ${\displaystyle f}$ bounded from above;
• ${\displaystyle lim inf{\mathrm{E}}_n[f]\ge \mathrm{E}[f]}$ for every lower semi-continuous function ${\displaystyle f}$ bounded from below;
• ${\displaystyle lim sup{P}_n(C)\le P(C)}$ for all closed sets ${\displaystyle C}$ of space ${\displaystyle S}$;
• ${\displaystyle lim inf{P}_n(U)\ge P(U)}$ for all open sets ${\displaystyle U}$ of space ${\displaystyle S}$;
• ${\displaystyle lim{P}_n(A)=P(A)}$ for all continuity sets ${\displaystyle A}$ of measure ${\displaystyle P}$.

In the case ${\displaystyle S}$ and ${\displaystyle \mathbb{R}}$ (with its usual topology) are homeomorphic , if ${\displaystyle F_n}$ and ${\displaystyle F}$ denote the cumulative distribution functions of the measures ${\displaystyle P_n}$ and ${\displaystyle P}$, respectively, then ${\displaystyle P_n}$ converges weakly to ${\displaystyle P}$ if and only if ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}{F}_n(x)=F(x)}$ for all points ${\displaystyle x\in \mathbb{R}}$ at which ${\displaystyle F}$ is continuous.

For example, the sequence where ${\displaystyle P_n}$ is the Dirac measure located at ${\displaystyle 1/n}$ converges weakly to the Dirac measure located at 0 (if we view these as measures on ${\displaystyle \mathbb{R}}$ with the usual topology), but it does not converge setwise. This is intuitively clear: we only know that ${\displaystyle 1/n}$ is "close" to ${\displaystyle 0}$ because of the topology of ${\displaystyle \mathbb{R}}$.

This definition of weak convergence can be extended for ${\displaystyle S}$ any metrizable topological space. It also defines a weak topology on ${\displaystyle \mathcal{P}(S)}$, the set of all probability measures defined on ${\displaystyle (S,\mathrm{\Sigma })}$. The weak topology is generated by the following basis of open sets:

${\displaystyle \left\{U_{\phi ,x,\delta }|\phi :S\to \mathbb{R}\text{ is bounded and continuous, }x\in \mathbb{R}\text{ and }\delta >0\right\},}$

where

${\displaystyle U_{\phi ,x,\delta }:=\left\{\mu \in \mathcal{P}(S)|\left|{\int }_S\phi \mathrm{d}\mu -x\right|<\delta \right\}.}$

If ${\displaystyle S}$ is also separable, then ${\displaystyle \mathcal{P}(S)}$ is metrizable and separable, for example by the Lévy–Prokhorov metric. If ${\displaystyle S}$ is also compact or Polish, so is ${\displaystyle \mathcal{P}(S)}$.

If ${\displaystyle S}$ is separable, it naturally embeds into ${\displaystyle \mathcal{P}(S)}$ as the (closed) set of Dirac measures, and its convex hull is dense.

There are many "arrow notations" for this kind of convergence: the most frequently used are ${\displaystyle P_n\Rightarrow P}$, ${\displaystyle P_n\rightharpoonup P}$, ${\displaystyle P_n\stackrel{w}{\to }P}$ and ${\displaystyle P_n\stackrel{\mathcal{D}}{\to }P}$.

Weak convergence of random variables

Let ${\displaystyle (\mathrm{\Omega },\mathcal{F},\mathbb{P})}$ be a probability space and X be a metric space. If X_n: Ω → X is a sequence of random variables then X_n is said to converge weakly (or in distribution or in law) to the random variable X: Ω → X as n → ∞ if the sequence of pushforward measures (X_n)_∗(P) converges weakly to X_∗(P) in the sense of weak convergence of measures on X, as defined above.

Comparison with vague convergence

Let ${\displaystyle X}$ be a metric space (for example ${\displaystyle \mathbb{R}}$ or ${\displaystyle [0,1]}$). The following spaces of test functions are commonly used in the convergence of probability measures.

• ${\displaystyle C_c(X)}$ the class of continuous functions ${\displaystyle f}$ each vanishing outside a compact set.
• ${\displaystyle C_0(X)}$ the class of continuous functions ${\displaystyle f}$ such that ${\displaystyle \underset{|x|\to \mathrm{\infty }}{lim}f(x)=0}$
• ${\displaystyle C_B(X)}$ the class of continuous bounded functions

We have ${\displaystyle C_c\subset C_0\subset C_B\subset C}$. Moreover, ${\displaystyle C_0}$ is the closure of ${\displaystyle C_c}$ with respect to uniform convergence.

Vague Convergence

A sequence of measures ${\displaystyle {\left({\mu }_n\right)}_{n\in \mathbb{N}}}$ converges vaguely to a measure ${\displaystyle \mu }$ if for all ${\displaystyle f\in C_c(X)}$, ${\displaystyle {\int }_{X}fd{\mu }_n\to {\int }_{X}fd\mu }$.

Weak Convergence

A sequence of measures ${\displaystyle {\left({\mu }_n\right)}_{n\in \mathbb{N}}}$ converges weakly to a measure ${\displaystyle \mu }$ if for all ${\displaystyle f\in C_B(X)}$, ${\displaystyle {\int }_{X}fd{\mu }_n\to {\int }_{X}fd\mu }$.

In general, these two convergence notions are not equivalent.

In a probability setting, vague convergence and weak convergence of probability measures are equivalent assuming tightness. That is, a tight sequence of probability measures ${\displaystyle ({\mu }_n{)}_{n\in \mathbb{N}}}$ converges vaguely to a probability measure ${\displaystyle \mu }$ if and only if ${\displaystyle ({\mu }_n{)}_{n\in \mathbb{N}}}$ converges weakly to ${\displaystyle \mu }$.

The weak limit of a sequence of probability measures, provided it exists, is a probability measure. In general, if tightness is not assumed, a sequence of probability (or sub-probability) measures may not necessarily converge vaguely to a true probability measure, but rather to a sub-probability measure (a measure such that ${\displaystyle \mu (X)\le 1}$). Thus, a sequence of probability measures ${\displaystyle ({\mu }_n{)}_{n\in \mathbb{N}}}$ such that ${\displaystyle {\mu }_n\stackrel{v}{\to }\mu }$ where ${\displaystyle \mu }$ is not specified to be a probability measure is not guaranteed to imply weak convergence.

Weak convergence of measures as an example of weak-* convergence

Despite having the same name as weak convergence in the context of functional analysis, weak convergence of measures is actually an example of weak-* convergence. The definitions of weak and weak-* convergences used in functional analysis are as follows:

Let ${\displaystyle V}$ be a topological vector space or Banach space.

1. A sequence ${\displaystyle x_n}$ in ${\displaystyle V}$ converges weakly to ${\displaystyle x}$ if ${\displaystyle \phi \left(x_n\right)\to \phi (x)}$ as ${\displaystyle n\to \mathrm{\infty }}$ for all ${\displaystyle \phi \in V^{\ast }}$. One writes ${\displaystyle x_n\stackrel{w}{\to }x}$ as ${\displaystyle n\to \mathrm{\infty }}$.
2. A sequence of ${\displaystyle {\phi }_n\in V^{\ast }}$converges in the weak-* topology to ${\displaystyle \phi }$ provided that ${\displaystyle {\phi }_n(x)\to \phi (x)}$ for all ${\displaystyle x\in V}$. That is, convergence occurs in the point-wise sense. In this case, one writes ${\displaystyle {\phi }_n\stackrel{w^{\ast }}{\to }\phi }$ as ${\displaystyle n\to \mathrm{\infty }}$.

To illustrate how weak convergence of measures is an example of weak-* convergence, we give an example in terms of vague convergence (see above). Let ${\displaystyle X}$ be a locally compact Hausdorff space. By the Riesz Representation theorem, the space ${\displaystyle M(X)}$ of finite Radon measures is isometrically isomorphic to the space of continuous linear functionals on ${\displaystyle C_0(X)}$. One direction of the isomorphism is given by the following: for each measure ${\displaystyle {\mu }_n\in M(X)}$, we define the linear functional ${\displaystyle {\phi }_{\mu }\in C_0(X{)}^{\ast }}$ by

${\displaystyle {\phi }_{\mu }(f)={\int }_{X}fd\mu .}$

for all ${\displaystyle f\in C_0(X)}$. Applying the definition of weak-* convergence in terms of linear functionals, the characterization of vague convergence of measures is obtained.

For compact ${\displaystyle X}$, ${\displaystyle C_0(X)=C_B(X)}$, so in this case weak convergence of measures is a special case of weak-* convergence.