Search for EdWorkingPapers here by author, title, or keywords.
Program and policy effects
Effect sizes in the Cohen’s d family are often used in education to compare estimates across studies, measures, and sample sizes. For example, effect sizes are used to compare gains in achievement students make over time, either in pre- and post-treatment studies or in the absence of intervention, such as when estimating achievement gaps. However, despite extensive research dating back to the paired t-test literature showing that such growth effect sizes should account for within-person correlations of scores over time, such achievement gains are often standardized relative to the standard deviation from a single timepoint or two timepoints pooled. Such a tendency likely occurs in part because there are not many large datasets from which a distribution of student- or school-level gains can be derived. In this study, we present a novel model for estimating student growth in conjunction with a national dataset to show that effect size estimates for student and school growth are often quite different when standardized relative to a distribution of gains rather than static achievement. In particular, we provide nationally representative empirical benchmarks for student achievement and gains, including for male-female gaps in those gains, and examine the sensitivity of those effect sizes to how they are standardized. Our results suggest that effect sizes scaled relative to a distribution of gains are less likely to understate the effects of interventions over time, and that resultant effect sizes often more closely match the estimand of interest for most practice, policy, and evaluation questions.
Field trips to see theater performances are a long-standing educational practice, however, there is little systematic evidence demonstrating educational benefits. This article describes the results of five random assignment experiments spanning two years where school groups were assigned by lottery to attend a live theater performance, or for some groups, watch a movie-version of the same story. We find significant educational benefits from seeing live theater, including higher levels of tolerance, social perspective taking, and stronger command of the plot and vocabulary of those plays. Students randomly assigned to watch a movie did not experience these benefits. Our findings also suggest that theater field trips may cultivate the desire among students to frequent the theater in the future.
Despite broad public interest in Veterans' education, there is relatively little evidence documenting the postsecondary trajectories of military service members after they return to civilian life. In the current report we investigate how U.S. Army service member college enrollment and progression trends compare to a similar population of civilians, using Army administrative personnel data merged with administrative records from the National Student Clearinghouse and the Educational Longitudinal Study (ELS) of 2002. Civilians were nearly three times as likely to enroll in college within one year of high school graduation (or one year of separation). Civilians were also much more likely to earn a bachelor’s degree within the period of study than either of the Army samples. While members of minority race/ethnicity groups in both military samples enroll at higher rates than their white counterparts, racial/ethnic minorities do not graduate at higher rates than their white counterparts. We discuss policy implications of our analyses in the final section of our paper.
Effective teacher hiring is fundamental to improving schools and yet few studies investigate this process. In this exploratory study of six successful, high-poverty schools (three charter, three district) in one Massachusetts city, we analyze the policy contexts that influenced hiring and examine the schools’ hiring practices. Through interviews with 142 teachers and administrators, we learned that, despite significant differences, these schools’ approaches were strikingly similar. Each used a two-way, information-rich hiring process that provided schools and candidates with opportunities to exchange information and assess one another before making an offer or signing a contract. Participants viewed their investment in hiring as an essential part of their school’s success. Based on our findings, we provide recommendations for policymakers, practitioners, and researchers.
The College Board sought to reduce barriers in the college application process by minimizing information aggregation costs, encouraging a broad application portfolio, and providing an impetus to start the search process. Some students were offered additional encouragements, such as text message reminders or college application fee waivers. In a randomized control trial with 785,000 low- and middle-income students in the top 50% of the PSAT and SAT distributions, we find no changes in college enrollment patterns, with the exception of a 0.02σ increase in college quality measures for African-American and Hispanic students.
A growing body of research evaluates the effects of turnaround on chronically low-performing schools. We extend this research to formally test factors that either mediate or suppress the effects of two turnaround initiatives in Tennessee: the Achievement School District (ASD) and local Innovation Zones (iZones). Using difference-in-differences models within a mediational framework, we find that hiring highly effective teachers and employing effective principals partially explain positive effects of iZone interventions. In the ASD, high levels of teacher turnover suppress potential positive effects after the first year. In iZone schools, several factors suppress even larger positive effects: hiring more novice teachers; hiring more principals with less experience; and high levels of student chronic absenteeism and student in-migration.
Social programs and mandates are usually studied in isolation even though they often interact closely with each other. Given the immense recent changes to health insurance systems, there is much potential for spillover effects to other systems in which health plays a large role. In this study, we examine how health insurance interacts with education, specifically the education of students with disabilities. We present the first analysis in the literature of how a mandate for health insurers to cover therapy for Autism Spectrum Disorder (ASD) up to age 18 affects educational services received by, and test scores of, students with ASD. A key aspect of the mandate is that children covered by Medicaid aged out of benefits quickly (by age 6), leaving them with a far weaker benefit than children covered by private insurance. Since we do not observe insurance status directly, we proxy for private insurance coverage using ineligibility for free/reduced-price lunch (FRPL) and estimate impacts on identification with ASD, special education services, and achievement through a series of difference-in-differences and triple difference models. We find little evidence of an overall shift in ASD identification, but we do find substantial crowd-out of special education services for students with ASD from the mandate. The stronger mandate led to increased mainstreaming of students in general education classrooms and a reduction in special education support services like teacher consultants. Girls in particular are more likely to be mainstreamed. There is little evidence of changes in achievement, which supports our interpretation of the service reductions as crowd-out.
Using an additional decade of CNLSY data, this study replicated and extended Deming’s (2009) evaluation of Head Start’s life-cycle skill formation impacts in three ways. Extending the measurement interval for Deming’s adulthood outcomes, we found no statistically significant impacts on earnings and mixed evidence of impacts on other adult outcomes. Applying Deming’s sibling comparison framework to more recent birth cohorts born to CNLSY mothers revealed mostly negative Head Start impacts. Combining all cohorts shows generally null impacts on school-age and early adulthood outcomes.
We present a reanalysis of the Tennessee Voluntary Prekindergarten Program (TNVPK), a state-funded program designed to promote the school readiness of 4-year-olds from low-income families. Oversubscribed programs used a lottery to randomly assign prospective enrollees a chance to attend TNVPK. We found that assignment to the program had largely null effects on measures of behavior, attendance, and retention collected during elementary school. TNVPK increased enrollment in special education by 4% between kindergarten and grade 3, and generated negative but generally statistically insignificant impacts on third-grade state test scores. We explore reasons for fadeout as well as threats to internal validity.