North Lawndale Reads

Evaluation, Capacity Building, Strategic Planning

This case study demonstrates PIE’s ability to conduct quasi-experimental, multi-site evaluations in dynamic school settings. It highlights our ability to make sense out of complex data from multiple sources, and how those insights can support new strategic priorities to support sustained impact over time.

The Challenge:

The Steans Family Foundation is a place-based foundation focused on Chicago’s North Lawndale community. For two years, they funded a comprehensive literacy program in four public elementary schools in the community with the goal to support 3rd grade reading. The literacy program targeted all K-3 reading achievement and offered a host of student, parent, and teacher supports to boost reading: teacher professional development and coaching, principal coaching, reading intervention and tutoring for students who were reading below grade level, counseling for students experiencing trauma, case management for chronically absent students, and parent engagement events. But with such a complex intervention, and with many data points over two years, the foundation needed an evaluation partner capable of rigorous and detailed data and implementation analysis. Key questions from the foundation included: How can we measure reading achievement across the myriad assessments used across the schools? How well is the program working? What parts of the program were the most impactful and why? And how can we improve?

PIE’s Solution

1

2

3

Prepare

As a starting point, getting familiar with what already exists helps PIE orient to the project. Our first step here was to do a deep dive into the data. We reviewed all grant reports and publicly available school data, and engaged in a listening tour with the grantee partners, school personnel, parents, and program officers about how their work supports K-3 literacy. Through these conversations and reviews, we built out a theory of change for the program and identified critical questions that the evaluation needed to answer, aligned to the needs of the community, schools, partners, and foundation. We also conducted a literature review to identify critical impact levers. We convened all stakeholders at a large group meeting to discuss our plans for the evaluation and got feedback on ways to improve it.

1

2

3

Develop

The result of this work was a focus on 3rd grade literacy outcomes using a shared assessment across all schools, linking students to community partner services over time (i.e., counseling, literacy intervention), and understanding the challenges teachers, parents, and partners face implementing integrated school-based programming. We identified comparison schools, based on community demographics and previous reading achievement, and also looked at the program schools’ historical reading achievement levels to help benchmark success. Protocols were put in place to consent students to track them over time in the programs. Finally, we identified consensus around specific outcomes across all schools and community partners (e.g., reduced chronic absenteeism, improved student growth and achievement on NWEA MAP assessments, improved teacher practices in balanced literacy) which we reviewed consistently throughout the year for formative learning and summative insights.

1

2

3

Learn

As a result of these efforts, our foundation partner and their NL Reads community partners, were able to understand their impact. The result of PIE’s intentional evaluation process provided significant insights. For example, the program supported 3rd grade reading achievement, out-performed comparison schools, and closed the gap between schools and the district. Tutoring support needed to meet specific dosage thresholds to be effective, and reading interventions and chronic absentee case management both needed two years of services to see significant improvements. While these findings were exciting, we also found that sustained student success seemed tenuous over the long-term. Key challenges such as student mobility, teacher turnover, and a lack of data-informed, individualized instruction were barriers to sustained success. A new strategic plan for the program was needed.

4

Build Capacity

As we do in every project, PIE sought opportunities to help our clients build the capacity to carry this important work forward. Throughout the initiative, PIE met 1:1 with each community partner and asked “What supports do you need to collect better data? How can we help?” We were able to build logic models, create data systems, and provide coaching to community partners and school leaders, such that they were able to identify their data priorities, and collect, analyze, and reflect on the data that matters. For example, we worked with schools to help them understand efficient ways to pull, clean, and review data with teachers and their leadership teams. Similarly, we helped a parent engagement organization link their data collection tools to research best practices, which helped them tell their impact story in a more compelling way to funders and schools. Across all partners, we supported capacity building efforts that helped them serve all students, across all their programs, in more targeted ways.

Results

As a result of this work, the program was continued at the current suite of schools and expanded to other schools; students in need of more reading services received them. In addition, due to the challenges identified with the program, PIE was able to support a new strategic plan for the initiative, which is currently being piloted, to ensure broader and continued success.