Search EdWorkingPapers

Search for EdWorkingPapers here by author, title, or keywords.

Program and policy effects

Matthew Kraft.

Researchers commonly interpret effect sizes by applying benchmarks proposed by Cohen over a half century ago. However, effects that are small by Cohen’s standards are large relative to the impacts of most field-based interventions. These benchmarks also fail to consider important differences in study features, program costs, and scalability. In this paper, I present five broad guidelines for interpreting effect sizes that are applicable across the social sciences. I then propose a more structured schema with new empirical benchmarks for interpreting a specific class of studies: causal research on education interventions with standardized achievement outcomes. Together, these tools provide a practical approach for incorporating study features, cost, and scalability into the process of interpreting the policy importance of effect sizes.

More →

Download PDF808.62 KB

Tolani Britton, Birunda Chelliah, Millie Symns, Vandeen Campbell.

Research suggests that earning college credits in high school increases the likelihood of postsecondary progress and graduation. In this study, we measure the impact of dual enrollment in high school and college courses through the College Now (CN) program on college enrollment for students in New York City. We use a regression discontinuity design (RDD) that estimates the causal local average effect of the treatment — eligibility for dual enrollment in college classes while in high school — on college enrollment. We find that being eligible for CN leads to a 7% point increase in the likelihood of college enrollment and an 8.6% point increase in the likelihood of enrollment in a four-year college. Students who were eligible for CN and enrolled in CN were 20% points more likely to enroll in college.

More →

Patrick J. Wolf, John F. Witte, Brian Kisida.

The Milwaukee Parental Choice Program (MPCP), the first modern private school choice program in the United States, has grown from 341 students attending 7 private schools in 1990 to 27,857 students attending 126 private schools in 2019. The MPCP has been subject to extensive study focused largely on student performance on standardized tests. This study presents new data on the college enrollment, persistence, and graduation of MPCP and MPS students who were tracked over 12 years beginning in 2006. MPCP participants are compared with a matched sample of MPS students who lived in the same neighborhood and had similar demographic characteristics and test scores at the beginning of the study. The collective evidence in this paper indicates that students in the MPCP program have greater educational attainment than the comparison group, as measured by college experience and outcomes. Most of the college attainment benefits of the MPCP are clear for both students who were in ninth grade at the beginning of the study, for whom positive attainment effects have previously been reported, and students who were initially enrolled in grades three through eight, who we examine here for the first time. As of 2018, MPCP students have spent more total years in a four-year college than their MPS peers. The MPCP students in the grade three through eight sample attained college degrees at rates that are statistically significantly higher than their matched MPS peers.

More →

Elizabeth Setren, Kyle Greenberg, Oliver Moore, Michael Yankovich.

In a flipped classroom, an increasingly popular pedagogical model, students view a video lecture at home and work on exercises with the instructor during class time. Advocates of the flipped classroom claim the practice not only improves student achievement, but also ameliorates the achievement gap. We conduct a randomized controlled trial at West Point and find that the flipped classroom produced short term gains in Math and no effect in Economics, but that the flipped model broadened the achievement gap: effects are driven by white, male, and higher achieving students. We find no long term average effects on student learning, but the widened achievement gap persists. Our findings demonstrate feasibility for the flipped classroom to induce short term gains in student learning; however, the exacerbation of the achievement gap, the effect fade-out, and the null effects in Economics suggest that educators should exercise caution when considering the model.

More →

Eric Brunner, Shaun Dougherty, Stephen Ross .

We examine the effect of admission to 16 stand-alone technical high schools within the Connecticut Technical High School System (CTHSS) on student educational and labor market outcomes. To identify the causal effect of admission on student outcomes, we exploit the fact that CTHSS utilizes a score-based admissions system and identify the effect of admission using a regression discontinuity approach. We find that male students attending one of the technical high schools are approximately 10 percentage points more likely to graduate from high school and 8 percentage points less likely to attend college, although there is some evidence that the negative effects on college attendance fade over time. We also find that male students attending a technical high school have quarterly earnings that are approximately 31% higher. Analyses of potential mechanisms behind these results reveal that male students that attend a technical high school have higher 9th grade attendance rates and higher 10th grade test scores. We find little evidence that attending a technical high school affects the educational or labor outcomes of women. These effects appear relatively broad based across different types of students in that we find little evidence of heterogeneity in these effects over student attributes like race and ethnicity, free lunch eligibility or residence in a poor, central city school district. However, when distinguishing between students based on the Career and Technical Education (CTE) offerings of the high school that these students likely would have attended, we find that the effects of admission to a CTHSS school are noticeably larger when the counterfactual high school has less CTE offerings.   

More →

Zachary Mabel, CJ Libassi, Michael Hurwitz.

Policymakers are increasingly including early-career earnings data in consumer-facing college search tools to help students and families make more informed post-secondary education decisions. We offer new evidence on the degree to which existing college-specific earnings data equips consumers with useful information by documenting the level of selection bias in the earnings metrics reported in the U.S. Department of Education’s College Scorecard. Given growing interest in reporting earnings by college and major, we focus on the degree to which earnings differences across four-year colleges and universities can be explained by differences in major composition across institutions. We estimate that more than three-quarters of the variation in median earnings across institutions is explained by observable factors, and accounting for differences in major composition explains over 30 percent of the residual variation in earnings after controlling for institutional selectivity, student composition, and local cost of living differences. We also identify large variations in the distribution of earnings within colleges; as a result, comparisons of early-career earnings can be extremely sensitive to whether the median, 25th, or 75th percentiles are presented. Taken together, our findings indicate that consumers can easily draw misleading conclusions about institutional quality when using publicly available earnings data to compare institutions.

More →

Elizabeth Setren.

Using novel variation in special education and English Language Learner classification from admissions lotteries, I find that students can achieve large academic gains without specialized services. Enrolling in a Boston charter school doubles the likelihood that students lose their special education or English Language Learner status, but exposes students to a high-performing general education program that includes high intensity tutoring, data driven instruction, and increased instructional time. The positive effects extend to college: charters nearly double the likelihood that English Language Learners enroll in four-year colleges and quadruple the likelihood that special education students graduate from two-year college. A multiple instrument strategy suggests that high quality general education practices drive the gains and finds no detrimental effect from lower classification rates.

More →

Anjali Adukia.

Programs that provide lower-skill employment are a popular anti-poverty strategy in developing countries, with India's employment-guarantee program (MGNREGA) employing adults in 23% of Indian households.  A potential concern is that guaranteeing lower-skill employment opportunities may discourage investment in human capital and long-run income growth.  Using large-scale administrative data and household survey data, I estimate precise spillover impacts on education that reject substantive declines in children's education from the government's rollout of MGNREGA.  I estimate that these small negative impacts are inexpensive to counteract, particularly compared to MGNREGA expenditures on rural employment and poverty alleviation.

More →

Benjamin L. Castleman, Francis X. Murphy, Richard W. Patterson, William L. Skimmyhorn.

The Post-9/11 GI Bill allows service members to transfer generous education benefits to a dependent. We run a large-scale experiment to test whether active choice framing impacts US Army service members’ decision to transfer benefits.  Individuals who received email messages framing GI Bill use as an active choice between own use and transfer to a family member are more likely to pursue information about the benefit than individuals receiving outreach that does not frame the decision as an active choice.  While we find no overall effect of framing on transfer, active choice increases transfer among service members with graduate degrees.

More →

Steven W. Hemelt, Nathaniel L. Schwartz, Susan M. Dynarski.

Dual-credit courses expose high school students to college-level content and provide the opportunity to earn college credits, in part to smooth the transition to college. With the Tennessee Department of Education, we conduct the first randomized controlled trial of the effects of dual-credit math coursework on a range of high school and college outcomes. We find that the dual-credit advanced algebra course alters students’ subsequent high school math course-taking, reducing enrollment in remedial math and boosting enrollment in precalculus and Advanced Placement math courses. We fail to detect an effect of the dual-credit math course on overall rates of college enrollment. However, the course induces some students to choose four-year universities instead of two-year colleges, particularly for those in the middle of the math achievement distribution and those first exposed to the opportunity to take the course in 11th rather than 12th grade. We see limited evidence of improvements in early math performance during college.

More →