Bernoulli distribution

Mean

The expected value of a Bernoulli random variable ${\displaystyle X}$ is

$${\displaystyle \mathrm{E}[X]=p}$$

This is because for a Bernoulli distributed random variable ${\displaystyle X}$ with ${\displaystyle Pr(X=1)=p}$ and $Pr(X=0)=q$ we find

$${\displaystyle \begin{array}{cc}\mathrm{E}[X]& =Pr(X=1)\cdot 1+Pr(X=0)\cdot 0\\ & =p\cdot 1+q\cdot 0\\ & =p.\end{array}}$$

Variance

The variance of a Bernoulli distributed ${\displaystyle X}$ is

$${\displaystyle \mathrm{Var}[X]=pq=p(1-p)}$$

We first find

$${\displaystyle \begin{array}{cc}\mathrm{E}[X^2]& =Pr(X=1)\cdot 1^2+Pr(X=0)\cdot 0^2\\ & =p\cdot 1^2+q\cdot 0^2\\ & =p=\mathrm{E}[X]\end{array}}$$

From this follows

$${\displaystyle \begin{array}{cc}\mathrm{Var}[X]& =\mathrm{E}[X^2]-\mathrm{E}[X{]}^2=\mathrm{E}[X]-\mathrm{E}[X{]}^2\\ & =p-p^2=p(1-p)=pq\end{array}}$$

With this result it is easy to prove that, for any Bernoulli distribution, its variance will have a value inside ${\displaystyle [0,1/4]}$.

Skewness

The skewness is ${\displaystyle \frac{q-p}{\sqrt{pq}}=\frac{1-2p}{\sqrt{pq}}}$. When we take the standardized Bernoulli distributed random variable ${\displaystyle \frac{X-\mathrm{E}[X]}{\sqrt{\mathrm{Var}[X]}}}$ we find that this random variable attains ${\displaystyle \frac{q}{\sqrt{pq}}}$ with probability ${\displaystyle p}$ and attains ${\displaystyle -\frac{p}{\sqrt{pq}}}$ with probability ${\displaystyle q}$. Thus we get

$${\displaystyle \begin{array}{cc}{\gamma }_1& =\mathrm{E}\left[{\left(\frac{X-\mathrm{E}[X]}{\sqrt{\mathrm{Var}[X]}}\right)}^3\right]\\ & =p\cdot {\left(\frac{q}{\sqrt{pq}}\right)}^3+q\cdot {\left(-\frac{p}{\sqrt{pq}}\right)}^3\\ & =\frac{1}{{\sqrt{pq}}^3}\left(p{q}^3-q{p}^3\right)\\ & =\frac{pq}{{\sqrt{pq}}^3}(q^2-p^2)\\ & =\frac{(1-p{)}^2-p^2}{\sqrt{pq}}\\ & =\frac{1-2p}{\sqrt{pq}}=\frac{q-p}{\sqrt{pq}}.\end{array}}$$

Higher moments and cumulants

The raw moments are all equal because ${\displaystyle 1^k=1}$ and ${\displaystyle 0^k=0}$.

$${\displaystyle \mathrm{E}[X^k]=Pr(X=1)\cdot 1^k+Pr(X=0)\cdot 0^k=p\cdot 1+q\cdot 0=p=\mathrm{E}[X].}$$

The central moment of order ${\displaystyle k}$ is given by $${\displaystyle {\mu }_k=(1-p)(-p{)}^k+p(1-p{)}^k.}$$ The first six central moments are $${\displaystyle \begin{array}{cc}{\mu }_1& =0,\\ {\mu }_2& =p(1-p),\\ {\mu }_3& =p(1-p)(1-2p),\\ {\mu }_4& =p(1-p)(1-3p(1-p)),\\ {\mu }_5& =p(1-p)(1-2p)(1-2p(1-p)),\\ {\mu }_6& =p(1-p)(1-5p(1-p)(1-p(1-p))).\end{array}}$$ The higher central moments can be expressed more compactly in terms of ${\displaystyle {\mu }_2}$ and ${\displaystyle {\mu }_3}$ $${\displaystyle \begin{array}{cc}{\mu }_4& ={\mu }_2(1-3{\mu }_2),\\ {\mu }_5& ={\mu }_3(1-2{\mu }_2),\\ {\mu }_6& ={\mu }_2(1-5{\mu }_2(1-{\mu }_2)).\end{array}}$$ The first six cumulants are $${\displaystyle \begin{array}{cc}{\kappa }_1& =p,\\ {\kappa }_2& ={\mu }_2,\\ {\kappa }_3& ={\mu }_3,\\ {\kappa }_4& ={\mu }_2(1-6{\mu }_2),\\ {\kappa }_5& ={\mu }_3(1-12{\mu }_2),\\ {\kappa }_6& ={\mu }_2(1-30{\mu }_2(1-4{\mu }_2)).\end{array}}$$

Entropy and Fisher's Information

Entropy

Entropy is a measure of uncertainty or randomness in a probability distribution. For a Bernoulli random variable ${\displaystyle X}$ with success probability ${\displaystyle p}$ and failure probability ${\displaystyle q=1-p}$, the entropy ${\displaystyle H(X)}$ is defined as:

$${\displaystyle \begin{array}{cc}H(X)& ={\mathbb{E}}_p\ln \frac{1}{Pr(X)}\\ & =-Pr(X=0)\ln Pr(X=0)-Pr(X=1)\ln Pr(X=1)\\ & =-(q\ln q+p\ln p).\end{array}}$$

The entropy is maximized when ${\displaystyle p=0.5}$, indicating the highest level of uncertainty when both outcomes are equally likely. The entropy is zero when ${\displaystyle p=0}$ or ${\displaystyle p=1}$, where one outcome is certain.

Fisher's Information

Fisher information measures the amount of information that an observable random variable ${\displaystyle X}$ carries about an unknown parameter ${\displaystyle p}$ upon which the probability of ${\displaystyle X}$ depends. For the Bernoulli distribution, the Fisher information with respect to the parameter ${\displaystyle p}$ is given by:

$${\displaystyle I(p)=\frac{1}{pq}}$$

Proof:

• The Likelihood Function for a Bernoulli random variable${\displaystyle X}$ is: $${\displaystyle L(p;X)=p^X(1-p{)}^{1-X}}$$ This represents the probability of observing ${\displaystyle X}$ given the parameter ${\displaystyle p}$.
• The Log-Likelihood Function is: $${\displaystyle \ln L(p;X)=X\ln p+(1-X)\ln (1-p)}$$
• The Score Function (the first derivative of the log-likelihood with respect to ${\displaystyle p}$ is: $${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }p}\ln L(p;X)=\frac{X}{p}-\frac{1-X}{1-p}}$$
• The second derivative of the log-likelihood function is: $${\displaystyle \frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{p}^2}\ln L(p;X)=-\frac{X}{p^2}-\frac{1-X}{(1-p{)}^2}}$$
• Fisher information is calculated as the negative expected value of the second derivative of the log-likelihood:$${\displaystyle \begin{array}{c}I(p)=-E\left[\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{p}^2}\ln L(p;X)\right]=-\left(-\frac{p}{p^2}-\frac{1-p}{(1-p{)}^2}\right)=\frac{1}{p(1-p)}=\frac{1}{pq}\end{array}}$$

It is maximized when ${\displaystyle p=0.5}$, reflecting maximum uncertainty and thus maximum information about the parameter ${\displaystyle p}$.

Related distributions

• If ${\displaystyle X_1,\dots ,X_n}$ are independent, identically distributed (i.i.d.) random variables, all Bernoulli trials with success probability p, then their sum is distributed according to a binomial distribution with parameters n and p:

  ${\displaystyle \sum _{k=1}^n{X}_k\sim \mathrm{B}(n,p)}$ (binomial distribution).

The Bernoulli distribution is simply ${\displaystyle \mathrm{B}(1,p)}$, also written as $\mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}(p).$

• The categorical distribution is the generalization of the Bernoulli distribution for variables with any constant number of discrete values.
• The Beta distribution is the conjugate prior of the Bernoulli distribution.
• The geometric distribution models the number of independent and identical Bernoulli trials needed to get one success.
• If $Y\sim \mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}\left(\frac{1}{2}\right)$, then $2Y-1$ has a Rademacher distribution.

Author's mention

• Abhirath, dwivedi; Kotz, Samuel; Kemp, Adrienne W. (1993). Univariate Discrete Distributions (2nd ed.). Wiley. ISBN 0-471-54897-9.
• Peatman, John G. (1963). Introduction to Applied Statistics. New York: Harper & Row. pp. 162–171.