Browse by Topics
- Covid-19 Education Research for Recovery
- Early childhood
- K-12 Education
- Post-secondary education
- Access and admissions
- Education outside of school (after school, summer…)
- Educator labor markets
- Educator preparation, professional development, performance and evaluation
- Finance
- Inequality
- Markets (vouchers, choice, for-profits, vendors)
- Methodology, measurement and data
- Multiple outcomes of education
- Parents and communities
- Politics, governance, philanthropy, and organizations
- Program and policy effects
- Race, ethnicity and culture
- Standards, accountability, assessment, and curriculum
- Students with Learning Differences
Breadcrumb
Search EdWorkingPapers
Beth E. Schueler
What Impacts Should We Expect from Tutoring at Scale? Exploring Meta-Analytic Generalizability
U.S. public schools are engaged in an unprecedented effort to expand tutoring in the wake of the COVID-19 pandemic. Broad-based support for scaling tutoring emerged, in part, because of the large effects on student achievement found in prior meta-analyses. We conduct an expanded meta-analysis of 265 randomized control trials and explore how estimates change when we better align our sample with a policy-relevant target of inference: large-scale tutoring programs in the U.S. aiming to improve standardized test performance. Pooled effect sizes from studies with stronger target-equivalence remain meaningful but are only a third to a half as large as those from our full sample. This result is driven by stark declines in pooled effect sizes as program scale increases. We explore four hypotheses for this pattern and document how a bundled package of recommended design features serves to partially inoculate programs from these attenuated effects at scale.
How State Takeovers of School Districts Affect Education Finance, 1990 to 2019
State takeover of school districts—a form of political centralization that shifts decision-making power from locally elected leaders to the state—has increased in recent years, often with the purported goal of improving district financial condition. Takeover has affected millions of students throughout the U.S. since the first takeover in 1988 and is most common in larger districts and communities serving large shares of low-income students and students of color. While previous research finds takeovers do not benefit student academic achievement on average, we investigate whether takeovers achieve their goal of improving financial outcomes. Using an event study approach, we find takeovers from 1990 to 2019 increased annual school spending by roughly $2,000 per pupil after five years, on average, leading to improvements in financial condition. Increased funding came primarily from state sources and funded districts’ legacy costs. However, takeover did not affect spending for districts with majority-Black student populations—which are disproportionately targeted for takeover—adding to a growing literature suggesting that takeover unequally affects majority-Black communities.
Partisanship, Race, Markets, and Public Health: The Politics of Pandemic School Operations for Reopening and Beyond
Partisanship influenced learning modality after the pandemic’s onset, but it is unknown whether partisanship predicted other aspects of educational operations. We study the role of partisanship, race, markets, and public health in predicting a range of operations—from modality to family engagement to social-emotional support to teacher PD—throughout 2020-21 in the context of Virginia. Districts’ partisan makeup and racial composition were similarly predictive of in-person offerings throughout 2020-21 but partisanship was less predictive over time. District characteristics explained limited variation in other aspects of operations, though districts with larger private school sectors provided more supports. Results emphasize the role of partisanship, race, and markets in reopening but also suggest school operational decisions were less politicized than choice of modality.
Interscholastic Policy Debate Promotes Critical Thinking and College-going: Evidence From Boston Public Schools
Few interventions reduce inequality in reading achievement, let alone higher order thinking skills, among adolescents. We study “policy debate”—an extracurricular activity focused on improving middle and high schoolers’ critical thinking, argumentation, and policy analysis skills—in Boston schools serving large concentrations of economically-disadvantaged students of color. Student fixed effects estimates show debate had positive impacts on ELA test scores of 0.13 SD, equivalent to 68% of a full year of average 9th grade learning. Gains were concentrated on analytical more than rote subskills. We find no harm to math, attendance, or disciplinary records, and evidence of positive effects on high school graduation and postsecondary enrollment. Impacts were largest among students who were lowest achieving prior to joining debate.
Assessors influence results: Evidence on enumerator effects and educational impact evaluations
A significant share of education and development research uses data collected by workers called “enumerators.” It is well-documented that “enumerator effects”—or inconsistent practices between the individual people who administer measurement tools— can be a key source of error in survey data collection. However, it is less understood whether this is a problem for academic assessments or performance tasks. We leverage a remote phone-based mathematics assessment of primary school students and survey of their parents in Kenya. Enumerators were randomized to students to study the presence of enumerator effects. We find that both the academic assessment and survey was prone to enumerator effects and use simulation to show that these effects were large enough to lead to spurious results at a troubling rate in the context of impact evaluation. We therefore recommend assessment administrators randomize enumerators at the student level and focus on training enumerators to minimize bias.
Can learning be measured by phone? Evidence from Kenya
School closures induced by COVID-19 placed heightened emphasis on alternative ways to measure student learning besides in-person exams. We leverage the administration of phone-based assessments (PBAs) measuring numeracy and literacy for primary school children in Kenya, along with in-person standardized tests administered to the same students prior to school shutdowns, to assess the validity of PBAs. Compared to repeated in-person assessments, PBAs did not severely misclassify students’ relative performance, but PBA scores did tend to be further from baseline in-person scores than repeated in-person assessments from each other. As such, PBAs performed well at measuring aggregate but not individual learning levels. Administrators can therefore use these tools for aggregate measurement, such as in the context of impact evaluation, but be wary of PBAs for individual-level tracking or high-stakes decisions. Results also reveal the importance of making deliberate efforts to reach a representative sample and selecting items that provide discriminating power.
A Cautionary Tale of Tutoring Hard-to-Reach Students in Kenya
Covid-19-induced school closures generated great interest in tutoring as a strategy to make up for lost learning time. Tutoring is backed by a rigorous body of research, but it is unclear whether it can be delivered effectively remotely. We study the effect of teacher-student phone call interventions in Kenya when schools were closed. Schools (n=105) were randomly assigned for their 3rd, 5th and 6th graders (n=8,319) to receive one of two versions of a 7-week weekly math-focused intervention—5-minute accountability checks or 15-minute mini-tutoring sessions—or to the control group. Although calls increased student perceptions that teachers cared, accountability checks had no effect on math performance up to four months after the intervention and tutoring decreased math achievement among students who returned to their schools after reopening. This was, in part, because the relatively low-achieving students most likely to benefit from calls were least likely to return and take in-person assessments. Tutoring substituted away from more productive uses of time, at least among returning students. Neither intervention affected enrollment. Tutoring remains a valuable tool but to avoid unintended consequences, careful attention should be paid to aligning tutoring interventions with best practices and targeting interventions to those who will benefit most.
Evaluating Education Governance: Does State Takeover of School Districts Affect Student Achievement?
Local school boards have primary authority for running educational systems in the U.S. but little is known empirically about the merits of this arrangement. State takeovers of struggling districts represent a rare alternative form of educational governance and have become an increasingly common response to low performance. However, limited research explores whether this effectively improves student outcomes. We track all takeovers nationwide from the late 1980s, when the first takeovers occurred, through 2016 and describe takeover districts. While these districts are low performing, we find academic performance plays less of a role in predicting takeover for districts serving larger concentrations of African American students. We then use a new data source allowing for cross-state comparisons of student outcomes to estimate the effect of takeovers that occurred between 2011 and 2016. On average, we find no evidence that takeover generates academic benefits. Takeover appears to be disruptive in the early years of takeover, particularly to English Language Arts achievement, although the longer-term effects are less clear. We also observe considerable heterogeneity of effects across districts. Takeovers were least effective in districts with higher baseline achievement and least harmful in majority Latinx communities. Leaders should be cautious about using takeover without considering local context and a better understanding of why some takeovers are more effective than others.
Improving Low-Performing Schools: A Meta-Analysis of Impact Evaluation Studies
The public narrative surrounding efforts to improve low-performing K-12 schools in the U.S. has been notably gloomy. Observers argue that either nothing works or we don’t know what works. At the same time, the federal government is asking localities to implement evidence-based interventions. But what is known empirically about whether school improvement works, how long it takes, which policies are most effective, and which contexts respond best to intervention? We meta-analyze 141 estimates from 67 studies of turnaround policies implemented post-NCLB. On average, these policies have had a moderate positive effect on math but no effect on ELA achievement as measured by high-stakes exams. We find evidence of positive impacts on low-stakes exams in STEM and humanities subjects and no evidence of harm on non-test outcomes. Some elements of reform, namely extended learning time and teacher replacements, predict greater effects. Contexts serving majority-Latinx populations have seen the largest improvements.