Search EdWorkingPapers
Matthew A. Kraft
What Impacts Should We Expect from Tutoring at Scale? Exploring Meta-Analytic Generalizability
U.S. public schools are engaged in an unprecedented effort to expand tutoring in the wake of the COVID-19 pandemic. Broad-based support for scaling tutoring emerged, in part, because of the large effects on student achievement found in prior meta-analyses. We conduct an expanded meta-analysis of 265 randomized control trials and explore how estimates change when we better align our sample with a policy-relevant target of inference: large-scale tutoring programs in the U.S. aiming to improve standardized test performance. Pooled effect sizes from studies with stronger target-equivalence remain meaningful but are only a third to a half as large as those from our full sample. This result is driven by stark declines in pooled effect sizes as program scale increases. We explore four hypotheses for this pattern and document how a bundled package of recommended design features serves to partially inoculate programs from these attenuated effects at scale.
The Causes and Consequences of U.S. Teacher Strikes
The U.S. has witnessed a resurgence of labor activism, with teachers at the forefront. We examine how teacher strikes affect compensation, working conditions, and productivity with an original dataset of 772 teacher strikes generating 48 million student days idle between 2007 and 2023. Using an event study framework, we find that, on average, strikes increase compensation by 8% and lower pupil-teacher ratios by 0.5 students, driven by new state revenues. We find little evidence of sizable impacts on student achievement up to five years post-strike, though strikes lasting 10 or more days decrease math achievement in the short-term.
JEL: I22, J30, J45, J52.
The Scaling Dynamics and Causal Effects of a District-Operated Tutoring Program
Public school systems across the U.S. have made major investments in tutoring to support students’ academic recovery in the wake of the COVID-19 pandemic. We evaluate a large urban district’s efforts to design, implement, and scale a district-operated, standards-based tutoring program across three years. We draw on extensive interviews and survey data to document the dynamic changes in the program as Metro Nashville Public Schools integrated core operations into its leadership and school structures, expanded tutor supply by pivoting from a volunteer to a teacher-based staffing model, and addressed scheduling constraints by offering tutoring immediately before and after school in addition to during the school day. The district steadily scaled the program across two years, delivering over 125,000 total hours of tutoring to more than 6,800 students while also increasing dosage each semester. Using a collection of experimental and quasi-experimental designs, we find consistent evidence of a small to medium average positive effect on students’ reading test scores (0.04 to 0.09 standard deviations), but no average effects on math test scores or course grades in either subject. We discuss four possible explanations for these results, including a limited treatment-control contrast, modest program duration, heterogeneous effects, and miscalibrated expectations of tutoring effects at scale.
The Effect of Student-Tutor Ratios: Experimental Evidence from a Pilot Online Math Tutoring Program
Budget constraints and limited supplies of local tutors have caused many K-12 school districts to pivot from individual tutoring in-person toward small-group tutoring online to expand access to personalized instruction. We conduct a field experiment to explore the effect of increasing student-tutor ratios on middle school students’ math achievement and growth during an online tutoring program. We leverage a novel feature of the program where tutors often taught individual and small-group tutoring sessions, allowing them to directly compare their experiences across these settings. Both experimental estimates and tutor survey responses suggest 1:1 tutoring is more effective than 3:1 tutoring online. Tutoring small groups in an online format presents additional challenges for personalizing instruction, developing relationships, fostering participation, and managing student behavior.
The Rise and Fall of the Teaching Profession: Prestige, Interest, Preparation, and Satisfaction over the Last Half Century
We examine the state of the U.S. K-12 teaching profession over the last half century by compiling nationally representative time-series data on four interrelated constructs: occupational prestige, interest among students, the number of individuals preparing for entry, and on-the-job satisfaction. We find a consistent and dynamic pattern across every measure: a rapid decline in the 1970s, a swift rise in the 1980s extending into the mid 1990s, relative stability, and then a sustained decline beginning around 2010. The current state of the teaching profession is at or near its lowest levels in 50 years. We identify and explore a range of hypotheses that might explain these historical patterns including economic and sociopolitical factors, education policies, and school environments.
Time in School: A Conceptual Framework, Synthesis of the Causal Research, and Empirical Exploration
We examine the fundamental and complex role that time plays in the learning process. We begin by developing a conceptual framework to elucidate the multiple obstacles schools face in converting total time in school into active learning time. We then synthesize the causal research and document a clear positive effect of additional time on student achievement typically of small to medium magnitude depending on dosage, use, and context. Further descriptive analyses reveal how large differences in the length of the school day and year across public schools are an underappreciated dimension of educational inequality in the United States. Finally, our case study of time loss in one urban district demonstrates the potential to substantially increase instructional time within existing constraints.
Taking Teacher Evaluation to Scale: The Effect of State Reforms on Achievement and Attainment
Federal incentives and requirements under the Obama administration spurred states to adopt major reforms to their teacher evaluation systems. We examine the effects of these reforms on student achievement and attainment at a national scale by exploiting their staggered implementation across states. We find precisely estimated null effects, on average, that rule out impacts as small as 0.017 standard deviations for achievement and 1.2 percentage points for high school graduation and college enrollment. We highlight five factors that likely limited the efficacy of teacher evaluation at scale: political opposition, decentralization, capacity constraints, limited generalizability, and the absence of compensating wages.
Grow Your Own: An Umbrella Term for Very Different Localized Teacher Pipeline Programs
“Grow Your Own” (GYO) programs have recently emerged as a promising approach to expand teacher supply, address localized teacher shortages, and diversify the profession. However, little is known about the scale and design of GYO programs, which recruit and support individuals from the local community to become teachers. We conduct a quantitative content analysis to describe 94 GYO initiatives. We find that GYO is used broadly as an umbrella term to describe teacher pipeline programs with very different purposes, participants, and program features. Our results suggest that misalignment between some GYOs’ purposes and program features may inhibit their effectiveness. Finally, we propose a new typology to facilitate more precise discussions of GYO programs.
Teacher Shortages: A Framework for Understanding and Predicting Vacancies
We develop a unifying conceptual framework for understanding and predicting teacher shortages at the state, region, district, and school levels. We then generate and test hypotheses about geographic and subject variation in teacher shortages using data on unfilled teaching positions in Tennessee during the fall of 2019. We find that teacher staffing challenges are highly localized, causing shortages and surpluses to coexist. Aggregate descriptions of staffing challenges mask considerable variation between schools and subjects within districts. Schools with fewer local early-career teachers, smaller district salary increases, worse working conditions, and higher historical attrition rates have higher vacancy rates. Our findings illustrate why viewpoints about, and solutions to, shortages depend critically on whether one takes an aggregate or local perspective.
Preferences, Inequities, and Incentives in the Substitute Teacher Labor Market
We examine the labor supply decisions of substitute teachers – a large, on-demand market with broad shortages and inequitable supply. In 2018, Chicago Public Schools implemented a targeted bonus program designed to reduce unfilled teacher absences in largely segregated Black schools with historically low substitute coverage rates. Using a regression discontinuity design, we find that incentive pay substantially improved coverage equity and raised student achievement. Changes in labor supply were concentrated among Black and Hispanic substitutes from nearby neighborhoods with experience in incentive schools. Wage elasticity estimates suggest incentives would need to be 50% of daily wages to close fill-rate gaps.