Search for EdWorkingPapers here by author, title, or keywords.
Methodology, measurement and data
A growing literature uses value-added (VA) models to quantify principals' contributions to improving student outcomes. Principal VA is typically estimated using a connected networks model that includes both principal and school fixed effects (FE) to isolate principal effectiveness from fixed school factors that principals cannot control. While conceptually appealing, high-dimensional FE regression models require sufficient variation to produce accurate VA estimates. Using simulation methods applied to administrative data from Tennessee and New York City, we show that limited mobility of principals among schools yields connected networks that are extremely sparse, where VA estimates are either highly localized or statistically unreliable. Employing a random effects shrinkage estimator, however, can alleviate estimation error to increase the reliability of principal VA.
‘QuantCrit’ (Quantitative Critical Race Theory) is a rapidly developing approach that seeks to challenge and improve the use of statistical data in social research by applying the insights of Critical Race Theory. As originally formulated, QuantCrit rests on five principles; 1) the centrality of racism; 2) numbers are not neutral; 3) categories are not natural; 4) voice and insight (data cannot ‘speak for itself); and 5) a social justice/equity orientation (Gillborn et al, 2018). The approach has quickly developed an international and interdisciplinary character, including applications in medicine (Gerido, 2020) and literature (Hammond, 2019). Simultaneously, there has been ferocious criticism from detractors outraged by the suggestion that numbers are anything other than objective and scientific (Airaksinen, 2018). In this context it is vital that the approach establishes some common understandings about good practice; in order to sustain rigor, make QuantCrit accessible to academics, practioners, and policymakers alike, and resist widespread attempts to over-simplify and pillory. This paper is intended to advance an iterative process of expanding and clarifying how to ‘QuantCrit’.
In very low-income settings, how much does family demand matter for child learning? In rural Gambia, caregivers with high aspirations for their children’s future education and career, measured before children start school, invest substantially more than other families in their children’s education. Despite this, essentially no children are literate or numerate three years later. In contrast, in villages receiving a highly impactful, teacher-focused supply-side intervention, children of high-aspirations caregivers are 25 percent more likely to achieve literacy and numeracy than other children. In such settings, greater demand can map onto developmentally meaningful learning differences, but only with adequate complementary inputs.
In this paper, I study the effect of winning the public school choice lottery on public school enrollment. In particular, I look at how different outside options affect how sensitive students are to receiving their first choice in the public school lottery, focusing on three measures of outside options: ability to afford private schools, geographic convenience of private schools, and zoned-school quality. Using rich administrative data from applications submitted through a centralized enrollment system in Tulsa Public Schools (TPS), I find that overall, students who do not get assigned to their top choice school in a public school choice program are 15 percentage points more likely to leave the public school system entirely than those who do get an offer at their top choice. This effect is driven by higher-income students: these students, who are more likely to be able to afford private schools, are 33 percentage points more likely to leave the public school system if they do not get an offer at the public school they rank first than those who do get a spot. Geographic convenience of private schools and zoned-school quality do not differentially affect students’ enrollment decisions once they receive a school assignment. These effects are important to understand as districts undergo efforts to increase participation in school choice programs, while seeking to maintain district enrollment. They also provide useful insights about how attrition may affect estimates of the impact of choice schools on student outcomes.
In this paper, we show that the election of a new school board member causes home values in their neighborhood to rise. This increase is identified using narrowly-decided contests and is driven by non-Democratic members, whose neighborhoods appreciate about 4% on average relative to those of losing candidates. We find that student test scores in the neighborhood public schools of non-Democratic winners also relatively increase, but this effect is driven by changing student composition, including via the manipulation of attendance zones, rather than improvements in school quality (as measured by test score value-added). Notably, we detect no differential changes when comparing neighborhood or scholastic outcomes between winning and losing Democratic school board candidates. These results suggest that partisan affiliation is correlated with private motivations for seeking public office.
Many teacher education researchers have expressed concerns with the lack of rigorous impact evaluations of teacher preparation practices. I summarize these various concerns as they relate to issues of internal validity, external validity, and measurement. I then assess the prevalence of these issues by reviewing 166 impact evaluations of teacher preparation practices published in peer-reviewed journals between 2002-2019. Although I find that very few studies address issues of internal validity, external validity and measurement, I highlight some innovative approaches and present a checklist of considerations to assist future researchers in designing more rigorous impact evaluations.
Dual-enrollment courses are theorized to promote students' preparedness for college in part by bolstering their beneficial beliefs, such as academic self-efficacy, educational expectations, and sense of college belonging. These beliefs may also shape students' experiences and outcomes in dual-enrollment courses, yet few if any studies have examined this possibility. We study a large dual-enrollment program created by a university in the Southwest to examine these patterns. We find that mathematics self-efficacy and educational expectations predict performance in dual-enrollment courses, even when controlling for students' academic preparedness, while factors such as high school belonging, college belonging, and self-efficacy in other academic domains are unrelated to academic performance. However, we also find that students of color and first-generation students tend to have lower self-efficacy and educational expectations before enrolling in dual-enrollment courses, in addition to having lower levels of academic preparation. These findings suggest that students from historically marginalized populations may benefit from social-psychological as well as academic supports in order to receive maximum benefits from early postsecondary opportunities such as dual-enrollment. Our findings have implications for how states and dual-enrollment programs determine eligibility for dual-enrollment as well as how dual-enrollment programs should be designed and delivered in order to promote equity in college preparedness.
School closures induced by COVID-19 placed heightened emphasis on alternative ways to measure student learning besides in-person exams. We leverage the administration of phone-based assessments (PBAs) measuring numeracy and literacy for primary school children in Kenya, along with in-person standardized tests administered to the same students prior to school shutdowns, to assess the validity of PBAs. Compared to repeated in-person assessments, PBAs did not severely misclassify students’ relative performance, but PBA scores did tend to be further from baseline in-person scores than repeated in-person assessments from each other. As such, PBAs performed well at measuring aggregate but not individual learning levels. Administrators can therefore use these tools for aggregate measurement, such as in the context of impact evaluation, but be wary of PBAs for individual-level tracking or high-stakes decisions. Results also reveal the importance of making deliberate efforts to reach a representative sample and selecting items that provide discriminating power.
This paper compares and contrasts two required building level school violence measures under NCLB, arrests and incidents of well-defined school misconduct acts, across 20 years of Pennsylvania’s approximately 3,000 public school buildings. Generally, both arrests for school violence and incidents of school violence are rare events. Over 20 years, the third quartile arrest rate was zero and, the third quartile incident rate was 3.3%. Relatively few, 4.1% overall, of Pennsylvania’s school buildings were persistently dangerous as defined and reported pursuant to Pennsylvania’s state plan to the US Department of Education; however, these buildings represented about 7.8% of the student population statewide. When we measure whether or not a school building is dangerous based on reported school violence incidents, that is without an arrest requirement, fully 36.9% of Pennsylvania’school buildings were dangerous, and they represented 46.7% of the students statewide. Both Philadelphia and Pittsburgh public school buildings were disproportionately unsafe and among the top 20 districts in the state which were unsafe over the 20 year study period.
Exploratory regression analysis of mean building scale scores for math and language arts explained about 58% of the variation in such learning outcome measures. As expected, household poverty, holding all else constant, has very strong, negative effects on learning outcomes. A school building composed entirely of low income students will score about 240 scale points lower, about 1.24 standard deviations lower, than a school building without any low income students. A school building at the 90th percentile in terms of student misconduct and poverty rates, would have lower student test scores by about 1 to 1.28 standard deviations. Were a school administrator to reduce student misconduct rates from the 90th percentile to the 50th percentile, our regression coefficients predict learning gains on the order of (100-43) = 2/3 of a standard deviation in mean scale scores.