Methodology, measurement and data

Teachers’ professional identities are the foundation of their practice. Previous scholarship has largely overlooked the extent to which the broader institutional environment shapes teachers’ professional identities. In this study, I bridge institutional logics with theory on teacher professional identity to empirically examine the deeply institutionalized, taken-for-granted ways American society has come to think of teaching (e.g., as a moral calling, as a profession, as labor) are internalized by K-12 teachers. I draw on survey data from 950 teachers across four US states (California, New York, Florida, and Texas), and develop an original survey measure to capture what I term teachers’ “institutionalized conceptions of teaching.” Across diverse state policy contexts, I find that teachers’ conceptions of teaching are guided by three underling logics: (1) an accountability logic, (2) a democratic logic, and (3) a moral calling logic. I then surface a typology of teacher professional and examine the relationship between these logics and teachers’ professional identities. I find that the taken-for-granted ways society frames teaching may be associated with dimensions of teachers’ professional identity, such as self-efficacy and professional commitment. Together, the findings suggest that supporting the professional well-being of K-12 teaching may demand shifting the deeply institutionalized norms of the profession to be more aligned with teachers’ democratic and moral aims—rather than our system's deep norms around external accountability. The study offers methodological contributions to the study of logics, as well as practical implications for the field of teaching.

More →

U.S. public schools are engaged in an unprecedented effort to expand tutoring in the wake of the COVID-19 pandemic. Broad-based support for scaling tutoring emerged, in part, because of the large effects on student achievement found in prior meta-analyses. We conduct an expanded meta-analysis of 265 randomized control trials and explore how estimates change when we better align our sample with a policy-relevant target of inference: large-scale tutoring programs in the U.S. aiming to improve standardized test performance. Pooled effect sizes from studies with stronger target-equivalence remain meaningful but are only a third to a half as large as those from our full sample. This result is driven by stark declines in pooled effect sizes as program scale increases. We explore four hypotheses for this pattern and document how a bundled package of recommended design features serves to partially inoculate programs from these attenuated effects at scale. 

More →

We examine the efficiency of traditional school districts versus charter schools in providing students with teachers who meet their demographic and education needs. Using panel data from the state of Michigan, we estimate the relationship between enrollment of Black, Hispanic, special education, and English learner students and the presence of Black, Hispanic, Special Education, and ESL teachers, and test whether this relationship differs at charter and traditional district-run schools. Because charter schools typically have less market power in hiring than large districts, we compare charter school employment practices to traditional public schools in districts of comparable size. Our results suggest that charter schools are more likely to employ same race teachers for Black students but not Hispanic students, and districts schools are slightly better at providing ESL and SPED teachers. We conclude that charter autonomy does not necessary generate better student-teacher matches, but Michigan charters may occupy a market niche by serving Black students and staffing Black teachers.

More →

Sometimes a treatment, such as receiving a high school diploma, is assigned to students if their scores on two inputs (e.g., math and English test scores) are above established cutoffs. This forms a multidimensional regression discontinuity design (RDD) to analyze the effect of the educational treatment where there are two running variables instead of one. Present methods for estimating such designs either collapse the two running variables into a single running variable, estimate two separate one-dimensional RDDs, or jointly model the entire response surface. The first two approaches may lose valuable information, while the third approach can be very sensitive to model misspecification. We examine an alternative approach, developed in the context of geographic RDDs, which uses Gaussian processes to flexibly model the response surfaces and estimate the impact of treatment along the full range of students that were on the margin of receiving treatment. We demonstrate theoretically, in simulation, and in an applied example, that this approach has several advantages over current approaches, including over another nonparametric surface response method. In particular, using Gaussian process regression in two-dimensional RDDs shows strong coverage and standard error estimation, and allows for easy examination of treatment effect variation for students with different patterns of running variables and outcomes. As these nonparametric approaches are new in education-specific RDDs, we also provide an R package for users to estimate treatment effects using Gaussian process regression.

More →

Students’ postsecondary course-taking is of interest to researchers, yet has been difficult to study at large scale because administrative transcript data are rarely standardized across institutions or state systems. This paper uses machine learning and natural language processing to standardize college transcripts at scale. We demonstrate the approach’s utility by showing how the disciplinary orientation of students’ courses and majors align and diverge at 18 diverse four-year institutions in the College and Beyond II dataset. Our findings complicate narratives that student participation in the liberal arts is in great decline. Both professional and liberal arts majors enroll in a large amount of liberal arts coursework, and in three of the four core liberal arts disciplines, the share of course-taking in those fields is meaningfully higher than the share of majors in those fields. To advance the study of student postsecondary pathways, we release the classification models for public use.

More →

Inequality in college has both structural and psychological causes; these include the presence of self-defeating beliefs about the potential for growth and belonging. Such beliefs can be addressed through large-scale interventions in the college transition (Walton & Cohen, 2011; Walton et al., 2023) but are hard to measure. In our pre-registered study, we provide the strongest evidence to date that the belief that belonging challenges are common and tend to improve with time (“a process-oriented perspective”), the primary target of social-belonging interventions, is critical. We did so by developing and applying computational language measures to 25,000 essays written during a randomized trial of this intervention across 22 broadly representative US colleges and universities (Walton et al., 2023). We compare the hypothesized mediator to one of simple optimism, which includes positive expectations without recognizing that challenges are common. Examining the active control condition, we find that socially disadvantaged students are, indeed, significantly less likely to express a process-oriented perspective spontaneously, and more likely to express simple optimism. This matters: Students who convey a process-oriented perspective, both in control and treatment conditions, are significantly more likely to complete their first year of college full-time enrolled and have higher first-year GPAs, while simple optimism predicts worse academic progress. The social-belonging intervention helped distribute a process-oriented perspective more equitably, though disparities remained. These computational methods enable the scalable and unobtrusive assessment of subtle student beliefs that help or hinder college success.

More →

This study provides the first large-scale quantitative exploration of mathematical language use in upper elementary U.S. classrooms. Our approach employs natural language processing techniques to describe variation in teachers’ and students’ use of mathematical language in 1,657 fourth and fifth grade lessons in 317 classrooms in four districts over three years. Students’ exposure to mathematical language varies substantially across lessons and between teachers. Results suggest that teacher modeling, defined as the density of mathematical terms in teacher talk, does not substantially cause students to uptake mathematical language, but that teachers may encourage student use of mathematical vocabulary by means other than mere modeling or exposure. However, we also find that teachers who use more mathematical language are more effective at raising student test scores. These findings reveal that teachers who use more mathematical vocabulary are more effective math teachers.

More →

Despite well-designed curriculum materials, teachers often face challenges in their implementation due to diverse classroom needs. This paper investigates whether Large Language Models (LLMs) can support middle-school math teachers by helping create high-quality curriculum scaffolds, which we define as the adaptations and supplements teachers employ to ensure all students can access and engage with the curriculum. Through Cognitive Task Analysis with expert teachers, we identify a three-stage process for curriculum scaffolding: observation, strategy formulation, and implementation. We incorporate these insights into three LLM approaches to create warmup tasks that activate background knowledge. The best-performing approach, which provides the model with the original curriculum materials and an expert-informed prompt, generates warmups that are rated significantly higher than warmups created by expert teachers in terms of alignment to learning objectives, accessibility to students working below grade level, and teacher preference. This research demonstrates the potential of LLMs to support teachers in creating effective scaffolds and provides a methodology for developing AI-driven educational tools.

More →

Researchers and policymakers aspire for educational interventions to change children’s long-run developmental trajectories. However, intervention impacts on cognitive and achievement measures commonly fade over time. Less is known, although much is theorized, about socialemotional skill persistence. The current meta-analysis investigated whether intervention impacts on social-emotional skills demonstrated greater persistence than impacts on cognitive skills. We drew studies from eight pre-existing meta-analyses, generating a sample of 86 educational RCTs targeting children from infancy through adolescence, together involving 56,662 participants and 450 outcomes measured at post-test and at least one follow-up. Relying on a meta-regression approach for modeling persistence rates, we tested the extent to which post-test impact magnitudes predicted follow-up impact magnitudes. We found that post-test impacts were equally predictive of follow-up impacts for cognitive and social-emotional skills at 6- to 12- months follow-up, indicating similar conditional persistence rates across skill types. At 1- to 2- years follow-up, rates were lower and, if anything, cognitive skills showed greater conditional persistence than social-emotional skills. A small positive follow-up effect was observed, on average, beyond what was directly predicted by the post-test impact, indicating that interventions may have long-term effects that are not fully mediated by post-test effects. This pattern of results implied that smaller post-test impacts produced more persistent effects than larger post-test impacts, and social-emotional skill impacts were smaller, on average, than cognitive skill impacts. Considered as a whole, intervention impacts on both social-emotional and cognitive skills demonstrated fadeout, especially for interventions that produced larger initial effects. Implications for theory and future directions are discussed.

More →

This study examines the effects of universal public pre-kindergarten for 3-year-olds (Pre-K3) on later public education outcomes, including enrollment, school mobility, special education status, and in-grade retention from kindergarten through second grade. While universal pre-kindergarten programs typically target 4-year-olds, interest in expanding to 3-year-olds is growing. Using the centralized assignment lottery in the District of Columbia as the basis for a quasi-experimental design, we find that Pre-K3 students are more likely to persist in the public system and remain in the same school. These effects are strongest for residents of low-income neighborhoods and communities of color and for students enrolled in dual language programs. Overall, public Pre-K3 appears to stabilize children’s early educational experiences, especially those starting furthest from opportunity.

More →