Search EdWorkingPapers
Methodology, measurement and data
Displaying 1 - 10 of 182
Inequality in college has both structural and psychological causes; these include the presence of self-defeating beliefs about the potential for growth and belonging. Such beliefs can be addressed through large-scale interventions in the college transition (Walton & Cohen, 2011; Walton et al., 2023) but are hard to measure. In our pre-registered study, we provide the strongest evidence to date that the belief that belonging challenges are common and tend to improve with time (“a process-oriented perspective”), the primary target of social-belonging interventions, is critical. We did so by developing and applying computational language measures to 25,000 essays written during a randomized trial of this intervention across 22 broadly representative US colleges and universities (Walton et al., 2023). We compare the hypothesized mediator to one of simple optimism, which includes positive expectations without recognizing that challenges are common. Examining the active control condition, we find that socially disadvantaged students are, indeed, significantly less likely to express a process-oriented perspective spontaneously, and more likely to express simple optimism. This matters: Students who convey a process-oriented perspective, both in control and treatment conditions, are significantly more likely to complete their first year of college full-time enrolled and have higher first-year GPAs, while simple optimism predicts worse academic progress. The social-belonging intervention helped distribute a process-oriented perspective more equitably, though disparities remained. These computational methods enable the scalable and unobtrusive assessment of subtle student beliefs that help or hinder college success.
This study provides the first large-scale quantitative exploration of mathematical language use in upper elementary U.S. classrooms. Our approach employs natural language processing techniques to describe variation in teachers’ and students’ use of mathematical language in 1,657 fourth and fifth grade lessons in 317 classrooms in four districts over three years. Students’ exposure to mathematical language varies substantially across lessons and between teachers. Results suggest that teacher modeling, defined as the density of mathematical terms in teacher talk, does not substantially cause students to uptake mathematical language, but that teachers may encourage student use of mathematical vocabulary by means other than mere modeling or exposure. However, we also find that teachers who use more mathematical language are more effective at raising student test scores. These findings reveal that teachers who use more mathematical vocabulary are more effective math teachers.
Despite well-designed curriculum materials, teachers often face challenges in their implementation due to diverse classroom needs. This paper investigates whether Large Language Models (LLMs) can support middle-school math teachers by helping create high-quality curriculum scaffolds, which we define as the adaptations and supplements teachers employ to ensure all students can access and engage with the curriculum. Through Cognitive Task Analysis with expert teachers, we identify a three-stage process for curriculum scaffolding: observation, strategy formulation, and implementation. We incorporate these insights into three LLM approaches to create warmup tasks that activate background knowledge. The best-performing approach, which provides the model with the original curriculum materials and an expert-informed prompt, generates warmups that are rated significantly higher than warmups created by expert teachers in terms of alignment to learning objectives, accessibility to students working below grade level, and teacher preference. This research demonstrates the potential of LLMs to support teachers in creating effective scaffolds and provides a methodology for developing AI-driven educational tools.
Researchers and policymakers aspire for educational interventions to change children’s long-run developmental trajectories. However, intervention impacts on cognitive and achievement measures commonly fade over time. Less is known, although much is theorized, about socialemotional skill persistence. The current meta-analysis investigated whether intervention impacts on social-emotional skills demonstrated greater persistence than impacts on cognitive skills. We drew studies from eight pre-existing meta-analyses, generating a sample of 86 educational RCTs targeting children from infancy through adolescence, together involving 56,662 participants and 450 outcomes measured at post-test and at least one follow-up. Relying on a meta-regression approach for modeling persistence rates, we tested the extent to which post-test impact magnitudes predicted follow-up impact magnitudes. We found that post-test impacts were equally predictive of follow-up impacts for cognitive and social-emotional skills at 6- to 12- months follow-up, indicating similar conditional persistence rates across skill types. At 1- to 2- years follow-up, rates were lower and, if anything, cognitive skills showed greater conditional persistence than social-emotional skills. A small positive follow-up effect was observed, on average, beyond what was directly predicted by the post-test impact, indicating that interventions may have long-term effects that are not fully mediated by post-test effects. This pattern of results implied that smaller post-test impacts produced more persistent effects than larger post-test impacts, and social-emotional skill impacts were smaller, on average, than cognitive skill impacts. Considered as a whole, intervention impacts on both social-emotional and cognitive skills demonstrated fadeout, especially for interventions that produced larger initial effects. Implications for theory and future directions are discussed.
This study examines the effects of universal public pre-kindergarten for 3-year-olds (Pre-K3) on later public education outcomes, including enrollment, school mobility, special education status, and in-grade retention from kindergarten through second grade. While universal pre-kindergarten programs typically target 4-year-olds, interest in expanding to 3-year-olds is growing. Using the centralized assignment lottery in the District of Columbia as the basis for a quasi-experimental design, we find that Pre-K3 students are more likely to persist in the public system and remain in the same school. These effects are strongest for residents of low-income neighborhoods and communities of color and for students enrolled in dual language programs. Overall, public Pre-K3 appears to stabilize children’s early educational experiences, especially those starting furthest from opportunity.
This paper documents several facts about graduate program graduation rates using administrative data covering public and nonprofit graduate students in Texas. Despite conventional wisdom that most graduate students complete their programs, only 58 percent of who started their program in 2004 graduated within 6 years. Between the 2004 and 2013 entering cohorts, graduate student completion rates grew by 10 percentage points. Graduation rates vary widely by field of study--ranging from an average of 81 percent for law programs to 53 percent for education programs. We also find large differences in graduation rates across institutions. On average, 72 percent of students who entered programs in flagship public universities graduated in 6 years compared to only 57 percent of those who entered programs in non-research intensive (non-R1) institutions. Graduate students who do not complete may face negative consequences due to lower average earnings and substantial levels of student debt.
Teachers’ professional identities are the foundation of their practice. Previous scholarship has largely overlooked the extent to which the broader reform culture shapes teachers’ professional identities. In this study, I draw on survey data from 950 teachers across four US states (California, New York, Florida, and Texas) to examine the extent to which teachers’ professional identities are associated with what I term “institutionalized conceptions” of their roles. Across diverse state policy contexts, I find that teachers draw upon a shared set of institutionalized conceptions of their roles, which are associated with their professional identities. The findings suggest that the taken-for-granted ways society frames teaching may be associated with dimensions of teachers’ professional identity, such as self-efficacy and professional commitment.
As states incorporate measures of college readiness into their accountability systems, school and district leaders need effective strategies to identify and support students at risk of not enrolling in college. Although there is an abundant literature on early warning indicators for high school dropout, fewer studies focus on indicators for college enrollment, especially those that are simple to calculate and easy for practitioners to use. This study explores three potential indicators of college readiness that educational leaders may consider using as part of an early warning system for college enrollment. Using district administrative data, our analysis shows that an indicator based on attendance, grades, and advanced course-taking is slightly more effective at predicting college enrollment than indicators based on course failures or standardized test scores. However, the performance of these indicators varies across different student demographic and socioeconomic subgroups, highlighting the limitations of these measures and pointing to areas where they may need to be supplemented with contextual information. Through event history analysis, we demonstrate that the ninth grade is a particularly challenging year for students, especially those who are male, Black, Hispanic, or economically disadvantaged. These results suggest that educational leaders ought to consider identifying and targeting students at risk of not attending college with additional resources and support during the freshman year of high school.
Colleges have increasingly turned to data science applications to improve student outcomes. One prominent application is to predict students’ risk of failing a course. In this paper, we investigate whether incorporating data from learning management systems (LMS)--which captures detailed information on students’ engagement in course activities--increases the accuracy of predicting student success beyond using just administrative data alone. We use data from the Virginia Community College System to build random forest models based on student type (new versus returning) and data source (administrative-only, LMS-only, or full data). We find that among returning college students, models that use administrative-only outperform models that use LMSonly. Combining the two types of data results in minimal increased accuracy. Among new students, LMS-only models outperform administrative-only models, and accuracy is significantly higher when both types of predictors are used. This pattern of results reflects the fact that community college administrative data contains little information about new students. Within the LMS data, we find that LMS data pertaining to students’ engagement during the first part of the course has the most predictive value.
Teachers are the most important school-specific factor in student learning. Yet, little evidence exists linking teacher professional development programs and the strategies or activities that comprise them to student achievement. In this paper, we examine a fellowship model for professional development designed and implemented by Leading Educators, a national nonprofit organization that aims to bridge research and practice to improve instructional quality and accelerate learning across school systems. During the 2015-16 and 2016-17 school years, Leading Educators conducted its fellowship program for two cohorts of instructional leaders, such as department chairs, mentor teachers, instructional coaches, and assistant principals, to provide these educators ongoing, collaborative, job-embedded professional development and to improve student achievement. Relying on quasi-experimental methods, we find that a school’s participation in the fellowship program significantly increased student proficiency rates in English language arts and math on state achievement exams. The positive impact was concentrated in the first cohort and in just one of three regions, and approximately 80 percent of treated schools were charters. Student achievement benefitted from a more sustained duration of participation in the fellowship program, varied depending on the share of a school’s educators who participated in the fellowship, and differed based on whether fellows independently selected into the program or were appointed to participate by their school leaders. Taken together, findings from this paper should inform professional learning organizations, schools, and policymakers on the design, implementation, and impact of educator professional development.