Post hoc analysis

Because each post hoc analysis is effectively a statistical test, conducting multiple post hoc comparisons introduces a family-wise error rate problem, which is a type of multiple testing problem. This increases the likelihood of false positives unless corrected.

Post hoc tests are follow-up tests performed after a significant ANOVA result to identify where the differences lie (which specific groups differ). To compensate, multiple post hoc testing procedures are sometimes used, but that is often difficult or impossible to do precisely. Post hoc analysis that is conducted and interpreted without adequate consideration of this problem is sometimes called data dredging (p-hacking) by critics because the statistical associations that it finds are often spurious. In other words, findings from data dredging are invalid or not trustworthy.

Post hoc analyses are acceptable when transparently reported as exploratory. In other words, post hoc analyses are not inherently unethical. The main requirement for their ethical use is simply that their results not be mispresented as the original hypothesis. Modern editions of scientific manuals have clarified this point; for example, APA style now specifies that "hypotheses should now be stated in three groupings: preplanned–primary, preplanned–secondary, and exploratory (post hoc). Exploratory hypotheses are allowable, and there should be no pressure to disguise them as if they were preplanned."

Types of post hoc analysis

Types or categories of post hoc analyses include:

• Pairwise comparisons: Tests all possible pairs
• Trend analysis: Tests for linear or quadratic trends across ordered groups
• Simple effects analysis: Examines effects within factorial ANOVA
• Interaction probing: Analyzes interaction constraints within factorial ANOVA

• Restricted Sets of Contrasts: Testing smaller families of comparisons

In addition, a subgroup analysis examines whether findings differ between discrete categories of subjects in the sample. This approach is common in clinical and observational studies.