Disentangling Person-Dependent and Item-Dependent Causal Effects: Applications of Item Response Theory To The Estimation of Treatment Effect Heterogeneity
Disentangling Person-Dependent and Item-Dependent Causal Effects: Applications of Item Response Theory to the Estimation of Treatment Effect Heterogeneity
Analyzing heterogeneous treatment effects (HTE) plays a crucial role in understanding
the impacts of educational interventions. A standard practice for HTE analysis is to
examine interactions between treatment status and pre-intervention participant characteristics,
such as pretest scores, to identify how different groups respond to treatment.
This study demonstrates that identical patterns of HTE on test score outcomes can
emerge either from variation in treatment effects due to a pre-intervention participant
characteristic or from correlations between treatment effects and item easiness parameters.
We demonstrate analytically and through simulation that these two scenarios
cannot be distinguished if analysis is based on summary scores alone. We then describe
a novel approach that identifies the relevant data-generating process by leveraging
item-level data. We apply our approach to a randomized trial of a reading intervention
in second grade, and show that any apparent HTE by pretest ability is driven by the
correlation between treatment effect size and item easiness. Our results highlight the
potential of employing measurement principles in causal analysis, beyond their common
use in test construction.
Gilbert, Joshua B., Luke W. Miratrix, Mridul Joshi, and Benjamin W. Domingue. (). Disentangling Person-Dependent and Item-Dependent Causal Effects: Applications of Item Response Theory to the Estimation of Treatment Effect Heterogeneity. (EdWorkingPaper:
-881). Retrieved from
Annenberg Institute at Brown University: https://doi.org/10.26300/6b7w-vp07
Value added models (VAMs) attempt to estimate the causal effects of teachers and schools on student test scores. We apply Generalizability Theory to show how estimated VA effects depend upon the selection of test items.
Does student-teacher match quality exist? While prior research documents disparities in teachers' impacts across student types, it has not distinguished between sorting and causal effects as the drivers of these disparities.