Search and Filter
Joshua B. Gilbert
Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes
Topics: MethodsTags: Assessment, EfficacyValue added models (VAMs) attempt to estimate the causal effects of teachers and schools on student test scores. We apply Generalizability Theory to show how estimated VA effects depend upon the selection of test items. Standard VAMs estimate causal effects on the items that are included on the… more →
Effectiveness of Structured Teacher Adaptations to an Online Content Literacy Intervention for Third Graders: A Randomized Controlled Trial During COVID-19
Jackie E. Relyea, Joshua B. Gilbert, Mary A. Burkhauser, Ethan Scherer, Douglas M. Mosher, Zhongyu Wei, Johanna N. Tvedt, James S. Kim.Topics: Student LearningScaling up evidence-based educational interventions to improve student outcomes presents challenges, especially when adapting to new contexts while maintaining fidelity. Structured teacher adaptations that integrate the strengths of experimental science (high fidelity) and improvement science (… more →
Mechanisms of Effect Size Differences Between Researcher Developed and Independently Developed Outcomes: A Meta-Analysis of Item-Level Data
Topics: MethodsDifferences in effect sizes between researcher developed (RD) and independently developed (ID) outcome measures are widely documented but poorly understood in education research. We conduct a meta-analysis using item-level outcome data to test potential mechanisms that explain differences in… more →
Leveraging Item Parameter Drift to Assess Transfer Effects in Vocabulary Learning
Topics: MethodsLongitudinal models of individual growth typically emphasize between-person predictors of change but ignore how growth may vary within persons because each person contributes only one point at each time to the model. In contrast, modeling growth with multi-item assessments allows evaluation of… more →
How Measurement Affects Causal Inference: Attenuation Bias is (Usually) More Important Than Scoring Weights
Topics: MethodsTags: AssessmentWhen analyzing treatment effects on test scores, researchers face many choices and competing guidance for scoring tests and modeling results. This study examines the impact of scoring choices through simulation and an empirical application. Results show that estimates from multiple methods applied… more →
Disentangling Person-Dependent and Item-Dependent Causal Effects: Applications of Item Response Theory to the Estimation of Treatment Effect Heterogeneity
Topics: MethodsAnalyzing heterogeneous treatment effects (HTE) plays a crucial role in understanding the impacts of educational interventions.
Time to Transfer: Long-Term Effects of a Sustained and Spiraled Content Literacy Intervention in the Elementary Grades
James S. Kim, Joshua B. Gilbert, Jackie E. Relyea, Patrick Rich, Ethan Scherer, Mary A. Burkhauser, Johanna N. Tvedt.Topics: Student LearningWe investigated the effectiveness of a sustained and spiraled content literacy intervention that emphasizes building domain and topic knowledge schemas and vocabulary for elementary-grade students. The Model of Reading Engagement (MORE) intervention underscores thematic lessons that provide an… more →
Estimating Treatment Effects with the Explanatory Item Response Model
Topics: MethodsThis simulation study examines the characteristics of the Explanatory Item Response Model (EIRM) when estimating treatment effects when compared to classical test theory (CTT) sum and mean scores and item response theory (IRT)-based theta scores. Results show that the EIRM and IRT theta scores… more →
The COVID-19 Impact on Reading Achievement Growth of Grade 3-5 Students in a U.S. Urban School District: Variation across Student Characteristics and Instructional Modalities
Topics: Student LearningThe current study aimed to explore the COVID-19 impact on the reading achievement growth of Grade 3-5 students in a large urban school district in the U.S. and whether the impact differed by students’ demographic characteristics and instructional modality. Specifically, using administrative data… more →
Modeling Item-Level Heterogeneous Treatment Effects with the Explanatory Item Response Model: Leveraging Online Formative Assessments to Pinpoint the Impact of Educational Interventions
Topics: MethodsTags: AssessmentAnalyses that reveal how treatment effects vary allow researchers, practitioners, and policymakers to better understand the efficacy of educational interventions. In practice, however, standard statistical methods for addressing Heterogeneous Treatment Effects (HTE) fail to address the HTE that… more →