The summer after the nationwide move to remote learning in March 2020, Impact Florida brought together academic leaders from four Florida districts to ask what they were most concerned about, heading back into a school year none of us had envisioned.
Before the COVID pandemic, each of these districts had been involved in one of Impact Florida’s Districts for Impact learning cadres, yearlong cross-district learning experiences where leaders learned from each other and national experts how to strengthen their work in one of Impact Florida’s Five Conditions that Support Great Teaching.
At that moment, however, each of these leaders was facing more uncertainty than they ever had in their careers. What would happen to students who were already struggling due to longstanding opportunity gaps?
And so from that first conversation, a concept began to emerge. Impact Florida proposed a new COVID Recovery Cadre, partnering these districts with Harvard’s Proving Ground, an expert in continuous improvement methodologies, to help districts respond to the crisis and strengthen their long-term capacity to address gaps affecting Black and Latino students and students of all races experiencing poverty. The foundation of the work was the trust and mutual vulnerability developed through Impact Florida’s previous learning cadres.
The four districts identified root causes of inequities in pre-Algebra and Algebra 1 during the critical transition year from eighth to ninth grade, widely viewed as a critical gateway for future academic success. Then, they selected a to support their students, all while committing to significant measurement (a randomized control trial or program evaluation) to be able to analyze the impact of those interventions.
What were the results, and what did we learn? There were three main takeaways:
First, we learned that the interventions indeed showed significant promise to improve student learning in math at this critical juncture in their K-12 education, even in the midst of a pandemic.
In Highlands County, supporting students with peer tutoring through Spring Math resulted in an improvement of about ⅓ a year of additional learning for students who completed the tutoring as compared to those who did not. The randomized controlled trial showed an improvement of .135 standard deviation on end-of-year exams (the Florida Standards Assessment and Algebra 1 end-of-course exams). Increases were also apparent on the district’s formative assessment tool, iReady, particularly within their learning of numbers of operations and algebra strands.
In Collier County, for teachers was a transformative experience for the 12 math teachers who participated in a pilot of the video coaching program. Teachers appreciated that the coach provided a safe, non-evaluative space to reflect on their practice and improve. Over the course of the video coaching, teachers developed a growth mindset: instead of focusing on their weaknesses, they learned to recognize and build upon their strengths. Most importantly, teachers shifted their practice in the classroom, allowing more wait time, asking students to explain their thinking more often, and allowing students more time to discuss math. As a result, coaches observed that 80 percent of teachers improved the cognitive demand of their instruction in the classroom – putting more of the thinking work on students instead of the teacher. Finally, more than 80 percent of the teachers who participated said they would do the coaching again.
Second, the impact on students could be significantly better if districts worked through implementation problems.
Palm Beach and Pasco counties decided to implement PERTS Online Modules for 9th Graders, which consisted of two free, half-hour online modules to increase student motivation, completed at home and in school. National studies showed that even this lighter-lift intervention could have modest, positive results for students. The results in Palm Beach and Pasco showed a tiny, less than .01 gain in GPA during Quarter 3 of SY 2020-2021, and the improvements didn’t last in the fourth quarter of the school year. These results weren’t surprising given significant implementation challenges. In Palm Beach, only 33 percent of targeted students actually completed both modules and there was only 50 percent of completion in Pasco. More worrying, students of color, low-income students, and low-performing students were less likely than their peers to have taken the modules – perhaps due to student absences, implementation issues in certain schools, or the disproportionate burden of COVID on those families. The fact that some positive results were achieved even given this context led us to wonder – what would be possible for students if these districts kept working and improved their implementation?
Third, a continuous improvement approach to implementation is powerful and essential.
In early April, Impact Florida hosted our Education Summit, bringing together leaders from 28 districts across Florida, and convening the COVID Recovery Cadre once more to look at results and talk about how the work continues in their districts. It was clear that the Cadre’s work continues to influence the way districts approached inequities in eighth and ninth grade math. In the case of Palm Beach, for example, the district continued its use of the PERTS modules in the 2021-2022 school year, and achieved much better implementation, with more than 90 percent of intended students taking the modules. In Collier, the powerful effect of video coaching has led district leaders to explore how to bring coaching to more teachers.
At this closed-door session, we heard loud and clear that the continuous improvement practices learned in the Cadre have made a lasting impact on the way these districts approach their work. Most importantly, the conversation reminded us that we shouldn’t expect implementation in any context – much less the context of a pandemic – to go perfectly on the first try. Rather, we should expect failures, lessons learned, and surprise successes, and plan to iterate after our first attempt – that is the true spirit of continuous improvement.
To that end, Impact Florida has recently released a set of resources – Workouts for Continuous Improvement – designed to help teams strengthen their continuous improvement muscles. The workouts take 90 minutes or less to complete, and are designed to stand alone or work as an iterative process from beginning to end of an implementation effort. We encourage district and school teams to try out these resources – and in the spirit of continuous improvement – to share their feedback with us via a short survey.
Looking back, as we hope to exit the world of acute COVID focus, the trusting relationships that Impact Florida has built across the state with leading educators continues to pay dividends for Florida’s diverse student population.
The headlines about Florida aren’t always sunny, but we see a very different picture behind the scenes. We have the privilege of supporting Florida educators and district leaders who are focused on serving their students, no matter the challenge.