Differential entropy

Properties of differential entropy

• For probability densities ${\displaystyle f}$ and ${\displaystyle g}$, the Kullback–Leibler divergence ${\displaystyle D_{KL}(f\parallel g)}$ is greater than or equal to 0 with equality only if ${\displaystyle f=g}$ almost everywhere. Similarly, for two random variables ${\displaystyle X}$ and ${\displaystyle Y}$, ${\displaystyle I(X;Y)\ge 0}$ and ${\displaystyle h(X\mid Y)\le h(X)}$ with equality if and only if ${\displaystyle X}$ and ${\displaystyle Y}$ are independent.
• The chain rule for differential entropy holds as in the discrete case $${\displaystyle h(X_1,\dots ,X_n)=\sum _{i=1}^{n}h(X_i\mid X_1,\dots ,X_{i-1})\le \sum _{i=1}^{n}h(X_i).}$$
• Differential entropy is translation invariant, i.e. for a constant ${\displaystyle c}$. $${\displaystyle h(X+c)=h(X)}$$
• Differential entropy is in general not invariant under arbitrary invertible maps.In particular, for a constant ${\displaystyle a}$, $${\displaystyle h(aX)=h(X)+\log |a|}$$For a vector valued random variable ${\displaystyle \mathbf{X}}$ and an invertible (square) matrix ${\displaystyle \mathbf{A}}$ $${\displaystyle h(\mathbf{A}\mathbf{X})=h(\mathbf{X})+\log \left(\left|det\mathbf{A}\right|\right)}$$
• In general, for a transformation from a random vector to another random vector with same dimension ${\displaystyle \mathbf{Y}=m\left(\mathbf{X}\right)}$, the corresponding entropies are related via $${\displaystyle h(\mathbf{Y})\le h(\mathbf{X})+\int f(x)\log \left|\frac{\mathrm{\partial }m}{\mathrm{\partial }x}\right|dx}$$ where ${\displaystyle \left|\frac{\mathrm{\partial }m}{\mathrm{\partial }x}\right|}$ is the Jacobian of the transformation ${\displaystyle m}$. The above inequality becomes an equality if the transform is a bijection. Furthermore, when ${\displaystyle m}$ is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and ${\displaystyle h(Y)=h(X)}$.
• If a random vector ${\displaystyle X\in {\mathbb{R}}^n}$ has mean zero and covariance matrix ${\displaystyle K}$, $h(\mathbf{X})\le \frac{1}{2}\log (det2\pi eK)=\frac{1}{2}\log [(2\pi e{)}^{n}detK]$ with equality if and only if ${\displaystyle X}$ is jointly gaussian (see below).

However, differential entropy does not have other desirable properties:

• It is not invariant under change of variables, and is therefore most useful with dimensionless variables.
• It can be negative.

A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

Maximization in the normal distribution

Theorem

With a normal distribution, differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.

Proof

Let ${\displaystyle g(x)}$ be a Gaussian PDF with mean μ and variance ${\displaystyle {\sigma }^2}$ and ${\displaystyle f(x)}$ an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that ${\displaystyle f(x)}$ has the same mean of ${\displaystyle \mu }$ as ${\displaystyle g(x)}$.

Consider the Kullback–Leibler divergence between the two distributions $${\displaystyle 0\le D_{KL}(f\parallel g)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)\log \left(\frac{f(x)}{g(x)}\right)dx=-h(f)-{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)\log (g(x))dx.}$$ Now note that $${\displaystyle \begin{array}{cc}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)\log (g(x))dx& ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)\log \left(\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}\right)dx\\ & ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)\log \frac{1}{\sqrt{2\pi {\sigma }^2}}dx+\log (e){\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)\left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)dx\\ & =-\frac{1}{2}\log (2\pi {\sigma }^2)-\log (e)\frac{{\sigma }^2}{2{\sigma }^2}\\ & =-\frac{1}{2}\left(\log (2\pi {\sigma }^2)+\log (e)\right)\\ & =-\frac{1}{2}\log (2\pi e{\sigma }^2)\\ & =-h(g)\end{array}}$$ because the result does not depend on ${\displaystyle f(x)}$ other than through the variance. Combining the two results yields $${\displaystyle h(g)-h(f)\ge 0}$$ with equality when ${\displaystyle f(x)=g(x)}$ following from the properties of Kullback–Leibler divergence.

Alternative proof

This result may also be demonstrated using the calculus of variations. A Lagrangian function with two Lagrangian multipliers may be defined as:

$${\displaystyle L={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x)\log (g(x))dx-{\lambda }_0\left(1-{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x)dx\right)-\lambda \left({\sigma }^2-{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x)(x-\mu {)}^{2}dx\right)}$$

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition ${\displaystyle \left(1={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x)dx\right)}$ and the requirement of fixed variance ${\displaystyle \left({\sigma }^2={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x)(x-\mu {)}^{2}dx\right)}$, are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

$${\displaystyle 0=\delta L={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\delta g(x)\left[\log (g(x))+1+{\lambda }_0+\lambda (x-\mu {)}^2\right]dx}$$

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:

$${\displaystyle g(x)=e^{-{\lambda }_0-1-\lambda (x-\mu {)}^2}}$$

Using the constraint equations to solve for λ₀ and λ yields the normal distribution:

$${\displaystyle g(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}}$$

Example: Exponential distribution

Let ${\displaystyle X}$ be an exponentially distributed random variable with parameter ${\displaystyle \lambda }$, that is, with probability density function

$${\displaystyle f(x)=\lambda e^{-\lambda x}\text{ for }x\ge 0.}$$

Its differential entropy is then $${\displaystyle \begin{array}{cc}{h}_e(X)& =-{\int }_0^{\mathrm{\infty }}\lambda e^{-\lambda x}\log \left(\lambda e^{-\lambda x}\right)dx\\ & =-\left({\int }_0^{\mathrm{\infty }}(\log \lambda )\lambda e^{-\lambda x}dx+{\int }_0^{\mathrm{\infty }}(-\lambda x)\lambda e^{-\lambda x}dx\right)\\ & =-\log \lambda {\int }_0^{\mathrm{\infty }}f(x)dx+\lambda \mathrm{E}[X]\\ & =-\log \lambda +1.\end{array}}$$

Here, ${\displaystyle h_e(X)}$ was used rather than ${\displaystyle h(X)}$ to make it explicit that the logarithm was taken to base e, to simplify the calculation.

Relation to estimator error

The differential entropy yields a lower bound on the expected squared error of an estimator. For any random variable ${\displaystyle X}$ and estimator ${\displaystyle \hat{X}}$ the following holds: $${\displaystyle \mathrm{E}[(X-\hat{X}{)}^2]\ge \frac{1}{2\pi e}{e}^{2h(X)}}$$ with equality if and only if ${\displaystyle X}$ is a Gaussian random variable and ${\displaystyle \hat{X}}$ is the mean of ${\displaystyle X}$.

Differential entropies for various distributions

In the table below ${\displaystyle \mathrm{\Gamma }(x)={\int }_0^{\mathrm{\infty }}{e}^{-t}{t}^{x-1}dt}$ is the gamma function, ${\displaystyle \psi (x)=\frac{d}{dx}\log \mathrm{\Gamma }(x)=\frac{{\mathrm{\Gamma }}^{\prime }(x)}{\mathrm{\Gamma }(x)}}$ is the digamma function, ${\displaystyle B(p,q)=\frac{\mathrm{\Gamma }(p)\mathrm{\Gamma }(q)}{\mathrm{\Gamma }(p+q)}}$ is the beta function, and γ_E is Euler's constant.

Many of the differential entropies are from.

Variants

Differential entropy is also not invariant under continuous coordinate transformations. It also does not share all properties of discrete entropy. For example, it may be negative. Instead of differential entropy, one may deduce the limit of discrete entropy as calculated on an increasingly dense set of points. The expression thus attained will be non-negative, and allow for coordinate-free descriptions of entropy.

This modification of differential entropy differs by requiring a measure ${\displaystyle m(x)}$. The expression is as follows:

$${\displaystyle D(p\parallel m)=\int p(x)\log \frac{p(x)}{m(x)}dx.}$$

If ${\displaystyle m(x)}$ is a probability density, this is the Kullback–Leibler divergence and is when non-negativity holds. When performing coordinate transformations, both ${\displaystyle p(x)}$ and ${\displaystyle m(x)}$ are transformed. The value calculated will then remain constant.

The definition of differential entropy above can be obtained by partitioning the range of ${\displaystyle X}$ into bins of length ${\displaystyle h}$ with associated sample points ${\displaystyle ih}$ within the bins, for ${\displaystyle X}$ Riemann integrable. This gives a quantized version of ${\displaystyle X}$, defined by ${\displaystyle X_h=ih}$ if ${\displaystyle ih\le X\le (i+1)h}$. Then the entropy of ${\displaystyle X_h=ih}$ is

$${\displaystyle H_h=-\sum _{i}hf(ih)\log (f(ih))-\sum hf(ih)\log (h).}$$

The first term on the right approximates the differential entropy, while the second term is approximately ${\displaystyle -\log (h)}$. Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be ${\displaystyle \mathrm{\infty }}$.