Search EdWorkingPapers

Search for EdWorkingPapers here by author, title, or keywords.

Jing Liu

Jing Liu, Julie Cohen.

Valid and reliable measurements of teaching quality facilitate school-level decision-making and policies pertaining to teachers, but conventional classroom observations are costly, prone to rater bias, and hard to implement at scale. Using nearly 1,000 word-to-word transcriptions of 4th- and 5th-grade English language arts classes, we apply novel text-as-data methods to develop automated, objective measures of teaching to complement classroom observations. This approach is free of rater bias and enables the detection of three instructional factors that are well aligned with commonly used observation protocols: classroom management, interactive instruction, and teacher-centered instruction. The teacher-centered instruction factor is a consistent negative predictor of value-added scores, even after controlling for teachers’ average classroom observation scores. The interactive instruction factor predicts positive value-added scores.

More →

Megan Kuhfeld, James Soland, Beth Tarasawa, Angela Johnson, Erik Ruzek, Jing Liu.

With 55 million students in the United States out of school due to the COVID-19 pandemic, education systems are scrambling to meet the needs of schools and families, including planning how best to approach instruction in the fall given students may be farther behind than in a typical year. Yet, education leaders have little data on how much learning has been impacted by school closures. While the COVID-19 learning interruptions are unprecedented in modern times, existing research on the impacts of missing school (due to absenteeism, regular summer breaks, and school closures) on learning can nonetheless inform projections of potential learning loss due to the pandemic. In this study, we produce a series of projections of COVID-19-related learning loss and its potential effect on test scores in the 2020-21 school year based on (a) estimates from prior literature and (b) analyses of typical summer learning patterns of five million students. Under these projections, students are likely to return in fall 2020 with approximately 63-68% of the learning gains in reading relative to a typical school year and with 37-50% of the learning gains in math. However, we estimate that losing ground during the COVID-19 school closures would not be universal, with the top third of students potentially making gains in reading. Thus, in preparing for fall 2020, educators will likely need to consider ways to support students who are academically behind and further differentiate instruction.

More →

Jing Liu, Susanna Loeb, Ying Shi.

Classroom teachers in the US are absent on average approximately six percent of a school year. Despite the prevalence of teacher absences, surprisingly little research has assessed the key source of replacement instruction: substitute teachers. Using detailed administrative and survey data from a large urban school district, we document the prevalence, predictors, and variation of substitute coverage across schools. Less advantaged schools systematically exhibit lower rates of substitute coverage compared with peer institutions. Observed school, teacher, and absence characteristics account for only part of this school variation. In contrast, substitute teachers’ preferences for specific schools, mainly driven by student behavior and support from teachers and school administrators, explain a sizable share of the unequal distribution of coverage rates above and beyond standard measures in administrative data.

More →

Jing Liu, Monica Lee, Seth Gershenson.

We provide novel evidence on the causal impact of student absences in middle and high school on state test scores, course grades, and educational attainment using a rich administrative dataset that includes the date and class period of each absence. Our identification strategy addresses potential endogeneity due to time-varying student-level shocks by exploiting the fact that in a given year, there exists within-student, between-class variation in absences. We also leverage information on the timing of absences to show that absences that occur after the annual window for state standardized testing do not appear to affect test scores, which provides a further check of our identification strategy. We find that absences in middle and high school harm contemporaneous student achievement and longer-term educational attainment: On average, missing 10 math classes reduces math test scores by 7% of a standard deviation, math course grades by 19% of a standard deviation, the probability of on-time graduation by 8%, and the probability of immediate college enrollment by 7%. Similar results hold for absences in English Language Arts classes. These results suggest that absences in middle school and high school are just as harmful, if not more so, than absences in elementary school. Moreover, the timing of absences during the school year matters, as both the occurrence and the impact of absences are dynamic phenomena.

More →

Min Sun, Jing Liu, Junmeng Zhu, Zachary LeClair.

Although program evaluations using rigorous quasi-experimental or experimental designs can inform decisions about whether to continue or terminate a given program, they often have limited ability to reveal the mechanisms by which complex interventions achieve their effects. To illuminate these mechanisms, this paper analyzes novel text data from thousands of school improvement planning and implementation reports from Washington State, deploying computer-assisted techniques to extract measures of school improvement processes. Our analysis identified 15 coherent reform strategies that varied greatly across schools and over time. The prevalence of identified reform strategies was largely consistent with school leaders’ own perceptions of reform priorities via interviews. Several reform strategies measures were significantly associated with reductions in student chronic absenteeism and improvements in student achievement. We lastly discuss the opportunities and pitfalls of using novel text data to study reform processes.

More →

Jing Liu, Susanna Loeb.

Teachers’ impact on student long-run success is only partially explained by their contributions to students’ short-run academic performance. For this study, we explore a second dimension of teacher effectiveness by creating measures of teachers’ contributions to student class-attendance. We find systematic variation in teacher effectiveness at reducing unexcused class absences at the middle and high school level. These differences across teachers are as stable as those for student achievement, but teacher effectiveness on attendance only weakly correlates with their effects on achievement. We link these measures of teacher effectiveness to students’ long-run outcomes. A high value-added to attendance teacher has a stronger impact on students’ likelihood of finishing high school than does a high value-added to achievement teacher. Moreover, high value-added to attendance teachers can motivate students to pursue higher academic goals as measured by Advanced Placement course taking. These positive effects are particularly salient for low-achieving and low-attendance students.

More →