National Museum of Mexican Art

Capacity Building, Evaluation Coaching

This case study is an example of how evaluation capacity building develops sustained evaluation practice for nonprofits. This is also an example of how organizations can conduct useful, internal evaluations on a shoestring budget. You will read how PIE’s evaluation coaching creates a culture of learning within a community-based organization, and helps them identify areas of impact and ways to improve in the future.

The Challenge:

Routine evaluation is elusive to many small to medium size non-profit organizations due to their limited time, money, and expertise to consistently collect, analyze, and review data. The National Museum of Mexican Art was one such organization, where their staff was comprised of artists, teaching artists, curators, and fundraisers who were all expected to use information for decision making, and were required to report data as an external funding requirement. Nobody on staff was trained in evaluation, which made the process of collecting and reporting data stressful and time consuming. In addition, staff felt that their data did not really capture their organization’s impact nor effectively “told their story.”

PIE’s Solution

1

2

3

Prepare

Due to the museum’s positive experience working with PIE as an external evaluator for a state-funded education project, they wanted to apply the same evaluation strategy to all of their educational programs and projects, but had a limited budget available. The museum pooled their limited monetary resources and continued funding PIE to provide evaluation coaching for program staff, such that staff would be able to internally evaluate their programs with the support of a trusted evaluation expert. PIE’s coaching supported nine program staff, who were responsible for implementing evaluation across 15 different programs.

1

2

3

Develop

This model of coaching was developed organically on a shoestring budget because staff members needed to learn to do evaluation themselves in order to meet their internal and external information needs within their budget. PIE moved from the role of external evaluator to the role of evaluation coach. PIE’s role as the external evaluator was to provide all of the tools and database development, data collection, entry, analysis, and reporting with the funds provided using our own external staff. As an evaluation coach, PIE’s role was to train and support NMMA staff members to collaboratively develop their own evaluation skills and knowledge, including: developing their own logic models, tools, and databases, and supporting them to conduct their own data analysis and reporting. PIE met with the museum staff in large group settings and in 1:1 coaching sessions to help all staff develop the outcomes, tools, and reporting processes needed to routinely conduct internal evaluation. At the beginning of the coaching process, some staff members were unfamiliar with evaluation or the evaluation cycle. In these instances, PIE’s evaluation coach provided baseline information to build the knowledge of the staff members and ensure that all staff had the same base of knowledge. After this initial step, the evaluator introduced the concept of developing a logic model. Some staff need the evaluation coach to help them turn their narratives of program theory into succinct concepts that logically connected to each other, whereas others were easily able to construct the model on their own. The next steps of evaluation coaching quite naturally followed the cycle of evaluation. Program staff met with the evaluator and received support on whatever stage of the evaluation process they were currently working. The ongoing support and individual coaching continued throughout the cycle and repeated again during the next evaluation cycle, with staff gradually relying less on the coach.

1

2

3

Learn

By working with each program staff member at their own pace in both large group settings, and individually, PIE’s evaluation coach developed relationships with each of them over time and was able to effectively build the capacity of all staff and programs. At the end of the coaching process, all programs had logic models, databases, data collection tools, and reporting processes, such that all staff could routinely collect and use data across their programs.

4

Build Capacity

The purpose of this work was to intentionally build the capacity of staff to support their own routine collection and use of data. As a result of PIE’s capacity building and evaluation coaching efforts, all staff received formal training in evaluation, developed program logic models, and created protocols and processes for evaluation across all their programs.

Results

To this day, the museum still uses these tools and processes to report their impact to their funders, and reflect on their data internally for program improvements. The evaluation capacity built many years ago was sustained within the organization, as the museum consistently engages staff in conversations around evaluation, program impact, and formative learning. The results of PIE’s coaching process, and the sustained impact it had on the museum, were written up in two peer-reviewed articles here, and here.