Margin of error

Standard deviation and standard error

We would expect the average of normally distributed values $p_1,p_2,\dots$ to have a standard deviation which somehow varies with $n$. The smaller $n$, the wider the margin. This is called the standard error ${\sigma }_{\overline{p}}$.

For the single result from our survey, we assume that $p=\overline{p}$, and that all subsequent results $p_1,p_2,\dots$ together would have a variance ${\sigma }_P^2=P(1-P)$:

${\displaystyle \text{Standard error}={\sigma }_{\overline{p}}\approx \sqrt{\frac{{\sigma }_P^2}{n}}\approx \sqrt{\frac{p(1-p)}{n}}}$.

Note that $p(1-p)$ corresponds to the variance of a Bernoulli distribution.

Maximum margin of error at different confidence levels

For a confidence level ${\displaystyle 1-2\alpha }$, there is a corresponding confidence interval about the mean ${\displaystyle \mu \pm z_{1-\alpha }\sigma }$, that is, the interval ${\displaystyle [\mu -z_{1-\alpha }\sigma ,\mu +z_{1-\alpha }\sigma ]}$ within which values of ${\displaystyle P}$ should fall with probability ${\displaystyle 1-2\alpha }$. Precise values of ${\displaystyle z_{1-\alpha }}$ are given by the quantile function of the normal distribution (which the 68–95–99.7 rule approximates).

Note that ${\displaystyle z_{\gamma }}$ is undefined for ${\displaystyle |\gamma |\ge 1}$, that is, ${\displaystyle z_{1.00}}$ is undefined, as is ${\displaystyle z_{1.10}}$.

Since ${\displaystyle max{\sigma }_P^2=maxP(1-P)=0.25}$ at ${\displaystyle p=0.5}$, we can arbitrarily set ${\displaystyle p=\overline{p}=0.5}$, calculate ${\displaystyle {\sigma }_P}$, ${\displaystyle {\sigma }_{\overline{p}}}$, and ${\displaystyle z_{\gamma }{\sigma }_{\overline{p}}}$ to obtain the maximum margin of error for ${\displaystyle P}$ at a given confidence level ${\displaystyle \gamma }$ and sample size ${\displaystyle n}$, even before having actual results. With ${\displaystyle p=0.5,n=1013}$

${\displaystyle {\text{MOE}}_{95}(0.5)=z_{0.95}{\sigma }_{\overline{p}}\approx z_{0.95}\sqrt{\frac{{\sigma }_P^2}{n}}=1.96\sqrt{\frac{.25}{n}}=0.98/\sqrt{n}=\pm 3.1\mathrm{\%}}$

${\displaystyle {\text{MOE}}_{99}(0.5)=z_{0.99}{\sigma }_{\overline{p}}\approx z_{0.99}\sqrt{\frac{{\sigma }_P^2}{n}}=2.58\sqrt{\frac{.25}{n}}=1.29/\sqrt{n}=\pm 4.1\mathrm{\%}}$

Also, usefully, for any reported ${\displaystyle {\text{MOE}}_{95}}$

${\displaystyle {\text{MOE}}_{99}=\frac{z_{0.99}}{z_{0.95}}{\text{MOE}}_{95}\approx 1.3\times {\text{MOE}}_{95}}$

Specific margins of error

If a poll has multiple percentage results (for example, a poll measuring a single multiple-choice preference), the result closest to 50% will have the highest margin of error. Typically, it is this number that is reported as the margin of error for the entire poll. Imagine poll ${\displaystyle P}$ reports ${\displaystyle p_a,p_b,p_c}$ as ${\displaystyle 71\mathrm{\%},27\mathrm{\%},2\mathrm{\%},n=1013}$

${\displaystyle {\text{MOE}}_{95}(P_a)=z_{0.95}{\sigma }_{\overline{p_a}}\approx 1.96\sqrt{\frac{p_a(1-p_a)}{n}}=0.89/\sqrt{n}=\pm 2.8\mathrm{\%}}$ (as in the figure above)

${\displaystyle {\text{MOE}}_{95}(P_b)=z_{0.95}{\sigma }_{\overline{p_b}}\approx 1.96\sqrt{\frac{p_b(1-p_b)}{n}}=0.87/\sqrt{n}=\pm 2.7\mathrm{\%}}$

${\displaystyle {\text{MOE}}_{95}(P_c)=z_{0.95}{\sigma }_{\overline{p_c}}\approx 1.96\sqrt{\frac{p_c(1-p_c)}{n}}=0.27/\sqrt{n}=\pm 0.8\mathrm{\%}}$

As a given percentage approaches the extremes of 0% or 100%, its margin of error approaches ±0%.

Comparing percentages

Imagine multiple-choice poll ${\displaystyle P}$ reports ${\displaystyle p_a,p_b,p_c}$ as ${\displaystyle 46\mathrm{\%},42\mathrm{\%},12\mathrm{\%},n=1013}$. As described above, the margin of error reported for the poll would typically be ${\displaystyle {\text{MOE}}_{95}(P_a)}$, as ${\displaystyle p_a}$ is closest to 50%. The popular notion of statistical tie or statistical dead heat, however, concerns itself not with the accuracy of the individual results, but with that of the ranking of the results. Which is in first?

If, hypothetically, we were to conduct a poll ${\displaystyle P}$ over subsequent samples of ${\displaystyle n}$ respondents (newly drawn from ${\displaystyle N}$), and report the result ${\displaystyle p_w=p_a-p_b}$, we could use the standard error of difference to understand how ${\displaystyle p_{w_1},p_{w_2},p_{w_3},\dots }$ is expected to fall about ${\displaystyle \overline{p_w}}$. For this, we need to apply the sum of variances to obtain a new variance, ${\displaystyle {\sigma }_{P_w}^2}$,

${\displaystyle {\sigma }_{P_w}^2={\sigma }_{P_a-P_b}^2={\sigma }_{P_a}^2+{\sigma }_{P_b}^2-2{\sigma }_{P_a,P_b}=p_a(1-p_a)+p_b(1-p_b)+2p_a{p}_b}$

where ${\displaystyle {\sigma }_{P_a,P_b}=-P_a{P}_b}$ is the covariance of ${\displaystyle P_a}$ and ${\displaystyle P_b}$.

Thus (after simplifying),

${\displaystyle \text{Standard error of difference}={\sigma }_{\overline{w}}\approx \sqrt{\frac{{\sigma }_{P_w}^2}{n}}=\sqrt{\frac{p_a+p_b-(p_a-p_b{)}^2}{n}}=0.029,P_w=P_a-P_b}$

${\displaystyle {\text{MOE}}_{95}(P_a)=z_{0.95}{\sigma }_{\overline{p_a}}\approx \pm 3.1\mathrm{\%}}$

${\displaystyle {\text{MOE}}_{95}(P_w)=z_{0.95}{\sigma }_{\overline{w}}\approx \pm 5.8\mathrm{\%}}$

Note that this assumes that ${\displaystyle P_c}$ is close to constant, that is, respondents choosing either A or B would almost never choose C (making ${\displaystyle P_a}$ and ${\displaystyle P_b}$ close to perfectly negatively correlated). With three or more choices in closer contention, choosing a correct formula for ${\displaystyle {\sigma }_{P_w}^2}$ becomes more complicated.

Effect of finite population size

The formulae above for the margin of error assume that there is an infinitely large population and thus do not depend on the size of population ${\displaystyle N}$, but only on the sample size ${\displaystyle n}$. According to sampling theory, this assumption is reasonable when the sampling fraction is small. The margin of error for a particular sampling method is essentially the same regardless of whether the population of interest is the size of a school, city, state, or country, as long as the sampling fraction is small.

In cases where the sampling fraction is larger (in practice, greater than 5%), analysts might adjust the margin of error using a finite population correction to account for the added precision gained by sampling a much larger percentage of the population. FPC can be calculated using the formula

${\displaystyle \text{FPC}=\sqrt{\frac{N-n}{N-1}}}$

...and so, if poll ${\displaystyle P}$ were conducted over 24% of, say, an electorate of 300,000 voters,

${\displaystyle {\text{MOE}}_{95}(0.5)=z_{0.95}{\sigma }_{\overline{p}}\approx \frac{0.98}{\sqrt{72,000}}=\pm 0.4\mathrm{\%}}$

${\displaystyle {\text{MOE}}_{{95}_{FPC}}(0.5)=z_{0.95}{\sigma }_{\overline{p}}\sqrt{\frac{N-n}{N-1}}\approx \frac{0.98}{\sqrt{72,000}}\sqrt{\frac{300,000-72,000}{300,000-1}}=\pm 0.3\mathrm{\%}}$

Intuitively, for appropriately large ${\displaystyle N}$,

${\displaystyle \underset{n\to 0}{lim}\sqrt{\frac{N-n}{N-1}}\approx 1}$

${\displaystyle \underset{n\to N}{lim}\sqrt{\frac{N-n}{N-1}}=0}$

In the former case, ${\displaystyle n}$ is so small as to require no correction. In the latter case, the poll effectively becomes a census and sampling error becomes moot.