Search EdWorkingPapers

Search EdWorkingPapers by author, title, or keywords.

Methodology, measurement and data

Brian Heseung Kim, Katharine Meyer, Alice Choe.

Interactive, text message-based advising programs have become an increasingly common strategy to support college access and success for underrepresented student populations. Despite the proliferation of these programs, we know relatively little about how students engage in these text-based advising opportunities and whether that relates to stronger student outcomes – factors that could help explain why we’ve seen relatively mixed evidence about their efficacy to date. In this paper, we use data from a large-scale, two-way text advising experiment focused on improving college completion to explore variation in student engagement using nuanced interaction metrics and automated text analysis techniques (i.e., natural language processing). We then explore whether student engagement patterns are associated with key outcomes including persistence, GPA, credit accumulation, and degree completion. Our results reveal substantial variation in engagement measures across students, indicating the importance of analyzing engagement as a multi-dimensional construct. We moreover find that many of these nuanced engagement measures have strong correlations with student outcomes, even after controlling for student baseline characteristics and academic performance. Especially as virtual advising interventions proliferate across higher education institutions, we show the value of applying a more codified, comprehensive lens for examining student engagement in these programs and chart a path to potentially improving the efficacy of these programs in the future.

Open source code on GitHub.

More →

Christine Mulhern, Isaac M. Opper.

There is an emerging consensus that teachers impact multiple student outcomes, but it remains unclear how to measure and summarize the multiple dimensions of teacher effectiveness into simple metrics for research or personnel decisions. We present a multidimensional empirical Bayes framework and illustrate how to use noisy estimates of teacher effectiveness to assess the dimensionality and predictive power of teachers' true effects. We find that it is possible to efficiently summarize many dimensions of effectiveness and most summary measures lead to similar teacher rankings; however, focusing on any one specific measure alone misses important dimensions of teacher quality.

More →

Michael F. Lovenheim, Jonathan Smith.

Early research on the returns to higher education treated the postsecondary system as a monolith.  In reality, postsecondary education in the United States and around the world is highly differentiated, with a variety of options that differ by credential (associates degree, bachelor’s degree, diploma, certificate, graduate degree), the control of the institution (public, private not-for-profit, private for-profit), the quality/resources of the institution, field of study, and exposure to remedial education. In this Chapter, we review the literature on the returns to these different types of higher education investments, which has received increasing attention in recent decades. We first provide an overview of the structure of higher education in the U.S. and around the world, followed by a model that helps clarify and articulate the assumptions employed by different estimators used in the literature. We then discuss the research on the return to institution type, focusing on the return to two-year, four-year, and for-profit institutions as well as the return to college quality within and across these institution types. We also present the research on the return to different educational programs, including vocational credentials, remedial education, field of study, and graduate school. The wide variation in the returns to different postsecondary investments that we document leads to the question of how students from different backgrounds sort into these different institutions and programs. We discuss the emerging research showing that lower-SES students, especially in the U.S., are more likely to sort into colleges and programs with lower returns as well as results from recent U.S.-based interventions and policies designed to support success among students from disadvantaged backgrounds. The Chapter concludes with some broad directions for future research.

More →

Brendan Bartanen, Aliza N. Husain.

A growing literature uses value-added (VA) models to quantify principals' contributions to improving student outcomes. Principal VA is typically estimated using a connected networks model that includes both principal and school fixed effects (FE) to isolate principal effectiveness from fixed school factors that principals cannot control. While conceptually appealing, high-dimensional FE regression models require sufficient variation to produce accurate VA estimates. Using simulation methods applied to administrative data from Tennessee and New York City, we show that limited mobility of principals among schools yields connected networks that are extremely sparse, where VA estimates are either highly localized or statistically unreliable. Employing a random effects shrinkage estimator, however, can alleviate estimation error to increase the reliability of principal VA.

More →

Wendy Castillo, David Gillborn.

‘QuantCrit’ (Quantitative Critical Race Theory) is a rapidly developing approach that seeks to challenge and improve the use of statistical data in social research by applying the insights of Critical Race Theory. As originally formulated, QuantCrit rests on five principles; 1) the centrality of racism; 2) numbers are not neutral; 3) categories are not natural; 4) voice and insight (data cannot ‘speak for itself); and 5) a social justice/equity orientation (Gillborn et al, 2018). The approach has quickly developed an international and interdisciplinary character, including applications in medicine (Gerido, 2020) and literature (Hammond, 2019). Simultaneously, there has been ferocious criticism from detractors outraged by the suggestion that numbers are anything other than objective and scientific (Airaksinen, 2018). In this context it is vital that the approach establishes some common understandings about good practice; in order to sustain rigor, make QuantCrit accessible to academics, practioners, and policymakers alike, and resist widespread attempts to over-simplify and pillory. This paper is intended to advance an iterative process of expanding and clarifying how to ‘QuantCrit’.

More →

Susha Roy.

In this paper, I study the effect of winning the public school choice lottery on public school enrollment. In particular, I look at how different outside options affect how sensitive students are to receiving their first choice in the public school lottery, focusing on three measures of outside options: ability to afford private schools, geographic convenience of private schools, and zoned-school quality. Using rich administrative data from applications submitted through a centralized enrollment system in Tulsa Public Schools (TPS), I find that overall, students who do not get assigned to their top choice school in a public school choice program are 15 percentage points more likely to leave the public school system entirely than those who do get an offer at their top choice. This effect is driven by higher-income students: these students, who are more likely to be able to afford private schools, are 33 percentage points more likely to leave the public school system if they do not get an offer at the public school they rank first than those who do get a spot. Geographic convenience of private schools and zoned-school quality do not differentially affect students’ enrollment decisions once they receive a school assignment. These effects are important to understand as districts undergo efforts to increase participation in school choice programs, while seeking to maintain district enrollment. They also provide useful insights about how attrition may affect estimates of the impact of choice schools on student outcomes.

More →

Stephen B. Billings, Hugh Macartney, Geunyong Park, John D. Singleton.

In this paper, we show that the election of a new school board member causes home values in their neighborhood to rise. This increase is identified using narrowly-decided contests and is driven by non-Democratic members, whose neighborhoods appreciate about 4% on average relative to those of losing candidates. We find that student test scores in the neighborhood public schools of non-Democratic winners also relatively increase, but this effect is driven by changing student composition, including via the manipulation of attendance zones, rather than improvements in school quality (as measured by test score value-added). Notably, we detect no differential changes when comparing neighborhood or scholastic outcomes between winning and losing Democratic school board candidates. These results suggest that partisan affiliation is correlated with private motivations for seeking public office.

More →

Zid Mancenido.

Many teacher education researchers have expressed concerns with the lack of rigorous impact evaluations of teacher preparation practices. I summarize these various concerns as they relate to issues of internal validity, external validity, and measurement. I then assess the prevalence of these issues by reviewing 166 impact evaluations of teacher preparation practices published in peer-reviewed journals between 2002-2019. Although I find that very few studies address issues of internal validity, external validity and measurement, I highlight some innovative approaches and present a checklist of considerations to assist future researchers in designing more rigorous impact evaluations.

More →

Matt S. Giani, Colton E. Krawietz, Tiffany A. Whittaker.

Dual-enrollment courses are theorized to promote students' preparedness for college in part by bolstering their beneficial beliefs, such as academic self-efficacy, educational expectations, and sense of college belonging. These beliefs may also shape students' experiences and outcomes in dual-enrollment courses, yet few if any studies have examined this possibility. We study a large dual-enrollment program created by a university in the Southwest to examine these patterns. We find that mathematics self-efficacy and educational expectations predict performance in dual-enrollment courses, even when controlling for students' academic preparedness, while factors such as high school belonging, college belonging, and self-efficacy in other academic domains are unrelated to academic performance. However, we also find that students of color and first-generation students tend to have lower self-efficacy and educational expectations before enrolling in dual-enrollment courses, in addition to having lower levels of academic preparation. These findings suggest that students from historically marginalized populations may benefit from social-psychological as well as academic supports in order to receive maximum benefits from early postsecondary opportunities such as dual-enrollment. Our findings have implications for how states and dual-enrollment programs determine eligibility for dual-enrollment as well as how dual-enrollment programs should be designed and delivered in order to promote equity in college preparedness.

More →

Daniel Rodriguez-Segura, Beth E. Schueler.

School closures induced by COVID-19 placed heightened emphasis on alternative ways to measure student learning besides in-person exams. We leverage the administration of phone-based assessments (PBAs) measuring numeracy and literacy for primary school children in Kenya, along with in-person standardized tests administered to the same students prior to school shutdowns, to assess the validity of PBAs. Compared to repeated in-person assessments, PBAs did not severely misclassify students’ relative performance, but PBA scores did tend to be further from baseline in-person scores than repeated in-person assessments from each other. As such, PBAs performed well at measuring aggregate but not individual learning levels. Administrators can therefore use these tools for aggregate measurement, such as in the context of impact evaluation, but be wary of PBAs for individual-level tracking or high-stakes decisions. Results also reveal the importance of making deliberate efforts to reach a representative sample and selecting items that provide discriminating power.

More →