College Readiness Assessment
Category: Pathways to and Through Postsecondary
This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross over into the treatment group, thus biasing treatment effect estimates. This study provides empirical evidence from a pilot intervention in two sections of a college-level introductory chemistry course, where a course-specific chatbot was introduced. We tested two randomization strategies: simple student-level randomization and laboratory-level randomization. We hypothesized that the greatest risk for crossover would have occurred under the simple individual randomization approach, however, no crossover occurred in either condition. Survey responses and system usage data indicate that this was not due to a lack of interaction among students or disinterest in the chatbot. These findings suggest that student-level randomization, even in an in-person course setting, can proceed with minimal risk of contamination for testing our focal intervention.
Keywords
randomization, causal inference, behavioral economics, nudge, higher education
Education level
Topics
Document Object Identifier (DOI)
10.26300/rq74-c249
