College Mathematics Beliefs and Belonging Survey
Category: Student Well-Being
Randomized controlled trials (RCTs) are the reference method for causal inference. To conduct field experiments in educational settings, study design must balance statistical power with the risk of treatment-control contamination. This study investigates both crossover and spillover contamination in a large-enrollment, in-person college course in which we tested an AI-enabled chatbot intervention. We compare two randomization approaches, individual-level and laboratory-level, to assess contamination risks. Contrary to expectations, no crossover occurred under student-level randomization. However, survey data indicate evidence of spillover, with treatment-group students reporting that they shared chatbot messages with peers. Using estimated contamination levels, we assess changes in minimum detectable effect size (MDES) and show that individual-level randomization remains preferable. Our findings offer practical guidance for balancing contamination risk and statistical power when designing RCTs in interactive educational settings.
Keywords
randomization, causal inference, behavioral economics, nudge, higher education
Education level
Topics
Document Object Identifier (DOI)
10.26300/rq74-c249
