When (not) to trust the overlap in confidence intervals: A practical guide

Researchers often aim to compare estimates across groups. For an intuitive and compact presentation of empirical results, many practitioners prefer reporting group-specific estimates instead of pairwise differences, and subsequently seek to infer the statistical significance of pairwise differences from the confidence intervals of the group-specific estimates. Following the insight that an overlap in the 95% confidence intervals of two estimates does not imply the statistical insignificance of their pairwise difference, practitioners now increasingly turn to 83.4% confidence intervals. However, the use of 83.4% confidence intervals rests on stringent assumptions which fail to hold in most social science applications. This not only opens the door to fallacious inferences but also to the deliberate manipulation of statistical evidence. This article proposes a viable alternative: Lower triangular visualizations, which allow for compact combinations of group-specific estimates with accurate information on the statistical significance of pairwise differences even when the number of groups is large.