Search EdWorkingPapers

Search EdWorkingPapers by author, title, or keywords.

Methodology, measurement and data

Kelli A. Bird, Benjamin L. Castleman, Yifeng Song.

Predictive analytics are increasingly pervasive in higher education. However, algorithmic bias has the potential to reinforce racial inequities in postsecondary success. We provide a comprehensive and translational investigation of algorithmic bias in two separate prediction models -- one predicting course completion, the second predicting degree completion. Our results show that algorithmic bias in both models could result in at-risk Black students receiving fewer success resources than White students at comparatively lower-risk of failure. We also find the magnitude of algorithmic bias to vary within the distribution of predicted success. With the degree completion model, the amount of bias is nearly four times higher when we define at-risk using the bottom decile than when we focus on students in the bottom half of predicted scores. Between the two models, the magnitude and pattern of bias and the efficacy of basic bias mitigation strategies differ meaningfully, emphasizing the contextual nature of algorithmic bias and attempts to mitigate it. Our results moreover suggest that algorithmic bias is due in part to currently-available administrative data being less useful at predicting Black student success compared with White student success, particularly for new students; this suggests that additional data collection efforts have the potential to mitigate bias.

More →


Valerie Bostwick, Christopher Severen.

We provide evidence that graduated driver licensing (GDL) laws, originally intended to improve public safety, impact human capital accumulation. Many teens use automobiles to access both school and employment. Because school and work decisions are interrelated, the effects of automobile-specific mobility restrictions are ambiguous. Using a novel triple-difference research design, we find that restricting mobility significantly reduces high school dropout rates and teen employment. We develop a multiple discrete choice model that rationalizes unintended consequences and reveals that school and work are weak complements. Thus, improved educational outcomes reflect decreased access to leisure activities rather than reduced labor market access.

More →


Courtney Bell, Jessalynn James, Eric S. Taylor, James H. Wyckoff.

We study the returns to experience in teaching, estimated using supervisor ratings from classroom observations. We describe the assumptions required to interpret changes in observation ratings over time as the causal effect of experience on performance. We compare two difference-in-differences strategies: the two-way fixed effects estimator common in the literature, and an alternative which avoids potential bias arising from effect heterogeneity. Using data from Tennessee and Washington, DC, we show empirical tests relevant to assessing the identifying assumptions and substantive threats—e.g., leniency bias, manipulation, changes in incentives or job assignments—and find our estimates are robust to several threats.

More →


Maria Marta Ferreyra, Carlos Garriga, Juan David Martin-Ocampo, Angelica Maria Sanchez-Diaz.

This paper estimates a dynamic model of college enrollment, progression, and graduation. A central feature of the model is student effort, which has a direct effect on class completion and an indirect effect mitigating risks on class completion and college persistence. The estimated model matches rich administrative data for a representative cohort of college students in Colombia. Estimates indicate that effort has a much greater impact than ability on class completion. Failing to consider effort as an input to class completion leads to overestimating ability’s role by a factor of two or three. It also promotes tuition discounts based on a pre-determined student trait—ability—rather than effort, which can be affected through policy, thus limiting higher education’s potential for social mobility.

More →


Lelys Dinarte-Diaz, Maria Marta Ferreyra, Tatiana Melguizo, Angelica Maria Sanchez-Diaz.

Short-cycle higher education programs (SCPs), lasting two or three years, capture about a quarter of higher education enrollment in the world and can play a key role enhancing workforce skills. In this paper, we estimate the program-level contribution of SCPs to student academic and labor market outcomes, and study how and why these contributions vary across programs. We exploit unique administrative data from Colombia on the universe of students, institutions, and programs to control for a rich set of student, peer, and local choice set characteristics. We find that program-level contributions account for about 60-70 percent of the variation in student-level graduation and labor market outcomes. Our estimates show that programs vary greatly in their contributions, across and especially within fields of study. Moreover, the estimated contributions are strongly correlated with program outcomes but not with other commonly used quality measures. Programs contribute more to formal employment and wages when they are longer, have been provided for a longer time, are taught by more specialized institutions, and are offered in larger cities.

More →


Maria Marta Ferreyra, Camila Galindo, Sergio Urzua.

This paper estimates the heterogeneous labor market effects of enrolling in higher education short-cycle (SC) programs. Expanding access to these programs might affect the behavior of some students (compliers) in two margins: the expansion margin (students who would not have enrolled in higher education otherwise) and the diversion margin (students who would have enrolled in bachelor’s programs otherwise). To quantify these responses, we exploit local exogenous variation in the supply of higher education institutions (HEIs) facing Colombian high school graduates in an empirical multinomial choice model with several instruments. According to our findings, the presence of at least one HEI specialized in SC programs in the vicinity of the student’s high school municipality increases SC enrollment by 3.7-4.5 percentage points (40-50% of the SC enrollment rate). The diversion margin largely drives this effect. For female compliers, enrollment in SC programs increases formal employment relative to the next-best alternative. For male compliers, in contrast, it lowers formal employment and wages. These results should alert policymakers of the unexpected consequences of higher education expansionary policies.

More →


Lelys Dinarte-Diaz, Maria Marta Ferreyra, Sergio Urzua, Marina Bassi.

Short-cycle higher education programs (SCPs) can play a central role in skill development and higher education expansion, yet their quality varies greatly within and among countries. In this paper we explore the relationship between programs’ practices and inputs (quality determinants) and student academic and labor market outcomes. We design and conduct a novel survey to collect program-level information on quality determinants and average outcomes for Brazil, Colombia, Dominican Republic, Ecuador, and Peru. Categories of quality determinants include training and curriculum, infrastructure, faculty, link with productive sector, costs and funding, and practices on student admission and institutional governance. We also collect administrative, student-level data on higher education and formal employment for SCP students in Brazil and Ecuador and match it to survey data. Using machine learning methods, we select the quality determinants that predict outcomes at the program and student levels. Estimates indicate that some quality determinants may favor academic and labor market outcomes while others may hinder them. Two practices predict improvements in all labor market outcomes in Brazil and Ecuador—teaching numerical competencies and providing job market information—and one practice—teaching numerical competencies—additionally predicts improvements in labor market outcomes for all survey countries. Since quality determinants account for 20-40 percent of the explained variation in student-level outcomes, quality determinants might have a role shrinking program quality gaps. Findings have implications for the design and replication of high-quality SCPs, their regulation, and the development of information systems.

More →


Michael S. Hayes, Jing Liu, Seth Gershenson.

Teachers affect a wide range of students’ educational and social outcomes, but how they contribute to students’ involvement in school discipline is less understood. We estimate the impact of teacher demographics and other observed qualifications on students’ likelihood of receiving a disciplinary referral. Using data that track all disciplinary referrals and the identity of both the referred and referring individuals from a large and diverse urban school district in California, we find students are about 0.2 to 0.5 percentage points (7% to 18%) less likely to receive a disciplinary referral from teachers of the same race or gender than from teachers of different demographic backgrounds. Students are also less likely to be referred by more experienced teachers and by teachers who hold either an English language learners or special education credential. These results are mostly driven by referrals for defiance and violence infractions, Black and Hispanic male students, and middle school students. While it is unclear whether these findings are due to variation in teachers’ effects on actual student behavior, variation in teachers’ proclivities to make disciplinary referrals, or a combination of the two, these results nonetheless suggest that teachers play a central role in the prevalence of, and inequities in, office referrals and subsequent student discipline.

More →


Danielle Victoria Handel, Eric A. Hanushek.

The impact of school resources on student outcomes was first raised in the 1960s and has been controversial since then. This issue enters into the decision making on school finance in both legislatures and the courts. The historical research found little consistent or systematic relationship of spending and achievement, but this research frequently suffers from significant concerns about the underlying estimation strategies. More recent work has re-opened the fundamental resource-achievement relationship with more compelling analyses that offer stronger identification of resource impacts. A thorough review of existing studies, however, leads to similar conclusions as the historical work: how resources are used is key to the outcomes. At the same time, the research has not been successful at identifying mechanisms underlying successful use of resources or for ascertaining when added school investments are likely to be well-used. Direct investigations of alternative input policies (capital spending, reducing class size, or salary incentives for teachers) do not provide clear support for such specific policy initiatives.

More →


Michael Dinerstein, Isaac M. Opper.

What happens when employers screen their employees but only observe a subset of output? We specify a model with heterogeneous employees and show that their response to the screening affects output in both the probationary period and the post-probationary period. The post-probationary impact is due to their heterogeneous responses affecting which individuals are retained and hence the screening efficiency. We show that the impact of the endogenous response on both the unobserved outcome and screening efficiency depends on whether increased effort on one task increases or decreases the marginal cost of effort on the other task. If the response decreases unobserved output in the probationary period then it increases the screening efficiency, and vice versa. We then assess these predictions empirically by studying a change to teacher tenure policy in New York City, which increased the role that a single measure -- test score value-added -- played in tenure decisions. We show that in response to the policy teachers increased test score value-added and decreased output that did not enter the tenure decision. The increase in test score value-added was largest for the teachers with more ability to improve students' untargeted outcomes, increasing their likelihood of getting tenure. We estimate that the endogenous response to the policy announcement reduced the screening efficiency gap -- defined as the reduction of screening efficiency stemming from the partial observability of output -- by 28%, effectively shifting some of the cost of partial observability from the post-tenure period to the pre-tenure period.

More →