Search EdWorkingPapers

EdWorkingPapers

Brian Holzman, Horace Duffy.

As states incorporate measures of college readiness into their accountability systems, school and district leaders need effective strategies to identify and support students at risk of not enrolling in college. Although there is an abundant literature on early warning indicators for high school dropout, fewer studies focus on indicators for college enrollment, especially those that are simple to calculate and easy for practitioners to use. This study explores three potential indicators of college readiness that educational leaders may consider using as part of an early warning system for college enrollment. Using district administrative data, our analysis shows that an indicator based on attendance, grades, and advanced course-taking is slightly more effective at predicting college enrollment than indicators based on course failures or standardized test scores. However, the performance of these indicators varies across different student demographic and socioeconomic subgroups, highlighting the limitations of these measures and pointing to areas where they may need to be supplemented with contextual information. Through event history analysis, we demonstrate that the ninth grade is a particularly challenging year for students, especially those who are male, Black, Hispanic, or economically disadvantaged. These results suggest that educational leaders ought to consider identifying and targeting students at risk of not attending college with additional resources and support during the freshman year of high school.

More →


Jeonghyeok Kim.

Each year, over a thousand public schools in the US close due to declining enrollments and chronic low performance, displacing hundreds of thousands of students. Using Texas administrative data and empirical strategies that use within-student across-time and within-school across-cohort variation, I explore the impact of school closures on students' educational and labor market outcomes. The findings indicate that experiencing school closures results in disruptions in both test scores and behavior. While the drop in test scores is recovered within three years, behavioral issues persist. This study further finds decreases in post-secondary education attainment, employment, and earnings at ages 25–27. These impacts are particularly pronounced among students in secondary education, Hispanic students, and those from originally low-performing schools and economically disadvantaged families.

More →


Kelli A. Bird, Benjamin L. Castleman, Yifeng Song, Renzhe Yu.

Colleges have increasingly turned to data science applications to improve student outcomes. One prominent application is to predict students’ risk of failing a course. In this paper, we investigate whether incorporating data from learning management systems (LMS)--which captures detailed information on students’ engagement in course activities--increases the accuracy of predicting student success beyond using just administrative data alone. We use data from the Virginia Community College System to build random forest models based on student type (new versus returning) and data source (administrative-only, LMS-only, or full data). We find that among returning college students, models that use administrative-only outperform models that use LMSonly. Combining the two types of data results in minimal increased accuracy. Among new students, LMS-only models outperform administrative-only models, and accuracy is significantly higher when both types of predictors are used. This pattern of results reflects the fact that community college administrative data contains little information about new students. Within the LMS data, we find that LMS data pertaining to students’ engagement during the first part of the course has the most predictive value.

More →


Ariana Audisio, Rebecca Taylor-Perryman, Tim Tasker, Matthew P. Steinberg.

Teachers are the most important school-specific factor in student learning. Yet, little evidence exists linking teacher professional development programs and the strategies or activities that comprise them to student achievement. In this paper, we examine a fellowship model for professional development designed and implemented by Leading Educators, a national nonprofit organization that aims to bridge research and practice to improve instructional quality and accelerate learning across school systems. During the 2015-16 and 2016-17 school years, Leading Educators conducted its fellowship program for two cohorts of instructional leaders, such as department chairs, mentor teachers, instructional coaches, and assistant principals, to provide these educators ongoing, collaborative, job-embedded professional development and to improve student achievement. Relying on quasi-experimental methods, we find that a school’s participation in the fellowship program significantly increased student proficiency rates in English language arts and math on state achievement exams. The positive impact was concentrated in the first cohort and in just one of three regions, and approximately 80 percent of treated schools were charters. Student achievement benefitted from a more sustained duration of participation in the fellowship program, varied depending on the share of a school’s educators who participated in the fellowship, and differed based on whether fellows independently selected into the program or were appointed to participate by their school leaders. Taken together, findings from this paper should inform professional learning organizations, schools, and policymakers on the design, implementation, and impact of educator professional development.

More →


Carly D. Robinson, Cynthia Pollard, Sarah Novicoff, Sara White, Susanna Loeb.

In-person tutoring has been shown to improve academic achievement. Though less well-researched, virtual tutoring has also shown a positive effect on achievement but has only been studied in grade five or above. We present findings from the first randomized controlled trial of virtual tutoring for young children (grades K-2). Students were assigned to 1:1 tutoring, 2:1 tutoring, or a control group. Assignment to any virtual tutoring increased early literacy skills by 0.05-0.08 SD with the largest effects for 1:1 tutoring (0.07-0.12 SD). Students initially scoring well below benchmark and first graders experienced the largest gains from 1:1 tutoring (0.15 and 0.20 SD, respectively). Effects are smaller than typically seen from in-person early literacy tutoring programs but still positive and statistically significant, suggesting promise particularly in communities with in-person staffing challenges.

More →


Joshua B. Gilbert, James S. Kim, Luke W. Miratrix.

Longitudinal models of individual growth typically emphasize between-person predictors of change but ignore how growth may vary within persons because each person contributes only one point at each time to the model. In contrast, modeling growth with multi-item assessments allows evaluation of how relative item performance may shift over time. While traditionally viewed as a nuisance under the label of “item parameter drift” (IPD) in the Item Response Theory literature, we argue that IPD may be of substantive interest if it reflects how learning manifests on different items or subscales at different rates. In this study, we present a novel application of the Explanatory Item Response Model (EIRM) to assess IPD in a causal inference context. Simulation results show that when IPD is not accounted for, both parameter estimates and their standard errors can be affected. We illustrate with an empirical application to the persistence of transfer effects from a content literacy intervention on vocabulary knowledge, revealing how researchers can leverage IPD to achieve a more fine-grained understanding of how vocabulary learning develops over time.

More →


Carly D. Robinson, Katharine Meyer, Chasity Bailey-Fakhoury, Amirpasha Zandieh, Susanna Loeb.

College students make job decisions without complete information. As a result, they may rely on misleading heuristics (“interesting jobs pay badly”) and pursue options misaligned with their goals. We test whether highlighting job characteristics changes decision making. We find increasing the salience of a job’s monetary benefits increases the likelihood college students apply by 196%. In contrast, emphasizing prosocial, career, or social benefits has no effect, despite students identifying these benefits as primary motivators for applying. The study highlights the detrimental incongruencies in students’ decision making alongside a simple strategy for recruiting college students to jobs that offer enriching experiences.

More →


Kristen Shure, Zach Weingarten.

Many decentralized matching markets experience high rates of instability due to information frictions. This paper analyzes these frictions in a particularly unstable U.S. market, the labor market for first-year school teachers. We develop and estimate a dynamic, partial equilibrium model of labor mobility that incorporates non-pecuniary information frictions for school climate and teacher workload. In terms of reducing turnover, a policy that improves information outperforms each alternative considered, including targeted wage premiums at hard-to-staff schools, large retention bonuses, and relaxed tenure requirements. Replicating the gains made through information revelation requires retention bonuses valued at 35% of teachers’ current salaries.

More →


Steven Michael Carlo.

The National Assessment of Educational Progress (NAEP) has tested the civic, or citizenship knowledge of students across the nation at irregular intervals since its very inception. Despite advancements in reading and mathematics, evidenced by results from the National Assessment of Educational Progress (NAEP), civics proficiency has remained consistently low, which raises concerns among educators and policymakers. This study attempts to provide those educators and policymakers with state-level predictions, not currently provided for the civics assessment. This research addresses this gap in state-level civics education data by applying multilevel regression with poststratification (MRP) to NAEP's nationally representative civics scores, yielding state-specific estimates that account for student demographics. A historical analysis of NAEP's development underscores its significance in national education and highlights the challenges of transitioning to state-level reporting, particularly for civics, which lacks state-level generalizability. Furthermore, this paper evaluates NAEP's frameworks, questioning their alignment with civics education's evolving needs, and investigates the presence of opportunity gaps in civics knowledge across gender and racial/ethnic lines. By comparing MRP estimates with published NAEP results, the study validates the method's credibility and emphasizes the potential of MRP in educational research. The findings reveal persistent racial/ethnic disparities in civic knowledge, with profound implications for civics instruction and policy. The research concludes by stressing the necessity for state-specific data to inform education policy and practice, advocating for teaching methods that enhance civic understanding and engagement, and suggesting future research directions to address the uncovered disparities.

More →


Brian Heseung Kim, Julie J. Park, Pearl Lo, Dominique J. Baker, Nancy Wong, Stephanie Breen, Huong Truong, Jia Zheng, Kelly Ochs Rosinger, OiYan Poon.

Letters of recommendation from school counselors are required to apply to many selective colleges and universities. Still, relatively little is known about how this non-standardized component may affect equity in admissions. We use cutting-edge natural language processing techniques to algorithmically analyze a national dataset of over 600,000 student applications and counselor recommendation letters submitted via the Common App platform. We examine how the length and topical content of letters (e.g., sentences about Personal Qualities, Athletics, Intellectual Promise, etc.) relate to student self-identified race/ethnicity, sex, and proxies for socioeconomic status. Paired with regression analyses, we explore whether demographic differences in letter characteristics persist when accounting for additional student, school, and counselor characteristics, as well as among letters written by the same counselor and among students with comparably competitive standardized test scores. We ultimately find large and noteworthy naïve differences in letter length and content across nearly all demographic groups, many in alignment with known inequities (e.g., many more sentences about Athletics among White and higher-SES students, longer letters and more sentences on Personal Qualities for private school students). However, these differences vary drastically based on the exact controls and comparison groups included – demonstrating that the ultimate implications of these letter differences for equity hinges on exactly how and when letters are used in admissions processes (e.g., are letters evaluated at face value across all students, or are they mostly compared to other letters from the same high school or counselor?). Findings do not point to a clear recommendation whether institutions should keep or discard letter requirements, but reflect the importance of reading letters and overall applications in the context of structural opportunity. We discuss additional implications and possible recommendations for college access and admissions policy/practice.

More →