Under the Individuals with Disabilities Education Act, schools are required to provide a free and appropriate public education to students with disabilities and show that the students are making academic progress. This study compares within- and across-years academic growth from kindergarten to 4th grade for students who were ever in special education (ever-SPED) to students who were never in special education (never-SPED). We follow one cohort of about 4,200 students for five years and assess the students up to three times per year. Although ever-SPED students started kindergarten with lower math and reading test scores and grew less in both subjects than never-SPED students during the kindergarten school year, ever-SPED students grew more than never-SPED students during the 1st and 2nd grade school years in math and 1st, 3rd, and 4th grade school years in reading. However, ever-SPED students lost more learning during every summer than never-SPED students. This led the test score disparities between the two to grow from under 0.5 standard deviations in kindergarten to 1.0 standard deviation in 4th grade. These findings suggest that summer learning opportunities are crucial for improving educational outcomes for students with disabilities.