Paired difference test

Specific methods for carrying out paired difference tests include the paired-samples t-test, the paired Z-test, the Wilcoxon signed-rank test and others.

Use in reducing variance

Paired difference tests for reducing variance are a specific type of blocking. To illustrate the idea, suppose we are assessing the performance of a drug for treating high cholesterol. Under the design of our study, we enroll 100 subjects, and measure each subject's cholesterol level. Then all the subjects are treated with the drug for six months, after which their cholesterol levels are measured again. Our interest is in whether the drug has any effect on mean cholesterol levels, which can be inferred through a comparison of the post-treatment to pre-treatment measurements.

The key issue that motivates the paired difference test is that unless the study has very strict entry criteria, it is likely that the subjects will differ substantially from each other before the treatment begins. Important baseline differences among the subjects may be due to their gender, age, smoking status, activity level, and diet.

There are two natural approaches to analyzing these data:

• In an "unpaired analysis", the data are treated as if the study design had actually been to enroll 200 subjects, followed by random assignment of 100 subjects to each of the treatment and control groups. The treatment group in the unpaired design would be viewed as analogous to the post-treatment measurements in the paired design, and the control group would be viewed as analogous to the pre-treatment measurements. We could then calculate the sample means within the treated and untreated groups of subjects, and compare these means to each other.
• In a "paired difference analysis", we would first subtract the pre-treatment value from the post-treatment value for each subject, then compare these differences to zero.

If we only consider the means, the paired and unpaired approaches give the same result. To see this, let Y_i1, Y_i2 be the observed data for the i^th pair, and let D_i = Y_i2 − Y_i1. Also let D, Y₁, and Y₂ denote, respectively, the sample means of the D_i, the Y_i1, and the Y_i2. By rearranging terms we can see that

${\displaystyle \overline{D}=\frac{1}{n}\sum _i(Y_{i2}-Y_{i1})=\frac{1}{n}\sum _i{Y}_{i2}-\frac{1}{n}\sum _i{Y}_{i1}={\overline{Y}}_2-{\overline{Y}}_1,}$

where n is the number of pairs. Thus the mean difference between the groups does not depend on whether we organize the data as pairs.

Although the mean difference is the same for the paired and unpaired statistics, their statistical significance levels can be very different, because it is easy to overstate the variance of the unpaired statistic. Through Bienaymé's identity, the variance of D is

${\displaystyle \begin{array}{cc}\mathrm{v}\mathrm{a}\mathrm{r}(\overline{D})& =\mathrm{var}({\overline{Y}}_2-{\overline{Y}}_1)\\ & =\mathrm{var}({\overline{Y}}_2)+\mathrm{var}({\overline{Y}}_1)-2\mathrm{cov}({\overline{Y}}_1,{\overline{Y}}_2)\\ & ={\sigma }_1^2/n+{\sigma }_2^2/n-2{\sigma }_1{\sigma }_2\mathrm{corr}(Y_{i1},Y_{i2})/n,\end{array}}$

where σ₁ and σ₂ are the population standard deviations of the Y_i1 and Y_i2 data, respectively. Thus the variance of D is lower if there is positive correlation within each pair. Such correlation is very common in the repeated measures setting, since many factors influencing the value being compared are unaffected by the treatment. For example, if cholesterol levels are associated with age, the effect of age will lead to positive correlations between the cholesterol levels measured within subjects, as long as the duration of the study is small relative to the variation in ages in the sample.

Power of the paired Z-test

Suppose we are using a Z-test to analyze the data, where the variances of the pre-treatment and post-treatment data σ₁² and σ₂² are known (the situation with a t-test is similar). The unpaired Z-test statistic is

${\displaystyle \frac{{\overline{Y}}_2-{\overline{Y}}_1}{\sqrt{{\sigma }_1^2/n+{\sigma }_2^2/n}},}$

The power of the unpaired, one-sided test carried out at level α = 0.05 can be calculated as follows:

${\displaystyle \begin{array}{cc}P\left(\frac{{\overline{Y}}_2-{\overline{Y}}_1}{\sqrt{{\sigma }_1^2/n+{\sigma }_2^2/n}}>1.645\right)& =P\left(\frac{{\overline{Y}}_2-{\overline{Y}}_1}{S}>1.645\sqrt{{\sigma }_1^2/n+{\sigma }_2^2/n}/S\right)\\ & =P\left(\frac{{\overline{Y}}_2-{\overline{Y}}_1-\delta +\delta }{S}>1.645\sqrt{{\sigma }_1^2/n+{\sigma }_2^2/n}/S\right)\\ & =P\left(\frac{{\overline{Y}}_2-{\overline{Y}}_1-\delta }{S}>1.645\sqrt{{\sigma }_1^2/n+{\sigma }_2^2/n}/S-\delta /S\right)\\ & =1-\mathrm{\Phi }(1.645\sqrt{{\sigma }_1^2/n+{\sigma }_2^2/n}/S-\delta /S),\end{array}}$

where S is the standard deviation of D, Φ is the standard normal cumulative distribution function, and δ = EY₂ − EY₁ is the true effect of the treatment. The constant 1.645 is the 95th percentile of the standard normal distribution, which defines the rejection region of the test.

By a similar calculation, the power of the paired Z-test is

${\displaystyle 1-\mathrm{\Phi }(1.645-\delta /S).}$

By comparing the expressions for power of the paired and unpaired tests, one can see that the paired test has more power as long as

${\displaystyle \sqrt{{\sigma }_1^2/n+{\sigma }_2^2/n}/S=\sqrt{\frac{{\sigma }_1^2+{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2-2{\sigma }_1{\sigma }_2\rho }}>1\text{ where }\rho :=\mathrm{corr}(Y_{i1},Y_{i2}).}$

This condition is met whenever ${\displaystyle \rho }$, the within-pairs correlation, is positive.

A random effects model for paired testing

The following statistical model is useful for understanding the paired difference test

${\displaystyle Y_{ij}={\mu }_j+{\alpha }_i+{\epsilon }_{ij}}$

where α_i is a random effect that is shared between the two values in the pair, and ε_ij is a random noise term that is independent across all data points. The constant values μ₁, μ₂ are the expected values of the two measurements being compared, and our interest is in δ = μ₂ − μ₁.

In this model, the α_i capture "stable confounders" that have the same effect on the pre-treatment and post-treatment measurements. When we subtract to form D_i, the α_i cancel out, so do not contribute to the variance. The within-pairs covariance is

${\displaystyle \mathrm{cov}(Y_{i1},Y_{i2})=\mathrm{var}({\alpha }_i).}$

This is non-negative, so it leads to better performance for the paired difference test compared to the unpaired test, unless the α_i are constant over i, in which case the paired and unpaired tests are equivalent.

In less mathematical terms, the unpaired test assumes that the data in the two groups being compared are independent. This assumption determines the form for the variance of D. However, when two measurements are made for each subject, it is unlikely that the two measurements are independent. If the two measurements within a subject are positively correlated, the unpaired test overstates the variance of D, making it a conservative test in the sense that its actual type I error probability will be lower than the nominal level, with a corresponding loss of statistical power. In rare cases, the data may be negatively correlated within subjects, in which case the unpaired test becomes anti-conservative. The paired test is generally used when repeated measurements are made on the same subjects, since it has the correct level regardless of the correlation of the measurements within pairs.