Search and Filter

Tip: use "quotes" for an exact phrase, like "teacher turnover".
Browse institutions A–Z

How Not to Fool Ourselves About Heterogeneity of Treatment Effects

722 views · 152 downloads

Abstract: Researchers across many fields have called for greater attention to heterogeneity of treatment effects—shifting focus from the average effect to variation in effects between different treatments, studies, or subgroups. True heterogeneity is important, but many reports of heterogeneity have proved to be false, non-replicable, or inflated. In this review, we catalog ways that past researchers fooled themselves about heterogeneity, and recommend steps to stop fooling ourselves about heterogeneity in the future.

We make 18 specific recommendations, which we illustrate with examples from education research. These are the most common themes: (1) seek  heterogeneity only when the causal mechanism offers clear motivation and the data offer adequate power; (2) shy away from seeking “no-but” heterogeneity when there is no main effect; (3) separate the noise of estimation error from the signal of true heterogeneity; (4) shrink variation in estimates toward zero; (5) increase p values and widen confidence intervals when conducting multiple tests; (6) estimate interactions rather than subgroup effects; and (7) check whether findings of heterogeneity are sensitive to changes in model or measurement. We also resolve two longstanding debates: one about centering interactions in linear models and one about estimating and interpreting interactions in nonlinear models such as logistic, ordinal, and interval regression. 

Following these recommendations will screen out many false, confusing, and non-replicable findings. Claims of heterogeneity that pass the screens should be more plausible, better calibrated, and more replicable.

Research Categories

Education level
Topics
Tags

Citation

Document Object Identifier (DOI)
10.26300/4qtt-0w17
EdWorkingPaper suggested citation:
von Hippel, Paul, and Brendan A. Schuetze. (). How Not to Fool Ourselves About Heterogeneity of Treatment Effects. (EdWorkingPaper: -1116). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/4qtt-0w17

Machine-readable bibliographic record: RIS, BibTeX

In the News

  • An article weighing the research evidence on Head Start's effectiveness. It links to the EdWorkingPaper by Paul von Hippel and Brendan Schuetze to argue that subgroup findings are usually misleading.