Building a Developmental Evaluation Framework for the Community Innovation Labs

Developmental evaluation mixes the critical thinking typical of evaluation with the creative thinking necessary for innovation.  Previously, Jamie Gamble of ImprintConsulting introduced this concept on ArtsFwd’s piece, Evaluating Innovation: An Introduction to Developmental Evaluation and, as a follow up, we at Archipelago Consultants would like to explore what developmental evaluation looks like in practice.  We will describe how we created a framework for the developmental evaluation of EmcArts’ Community Innovation Labs and provide an easy tool to help you start to develop your own developmental evaluation framework. 

Combining creativity and critical thinking: Objectives as hypotheses

The Community Innovation Labs (CILs) are an ongoing EmcArts program, in which diverse stakeholders come together to make progress on deeply entrenched community challenges.  The CIL process holds space for open-ended outcomes and each Lab is born out of a unique, complex issue defined by a particular local community. Therefore, any evaluation must be site-specific and cannot anticipate pre-determined outcomes.

Whenever the rubber hits the road on an innovative project or program like CIL, the facilitation team very quickly plunges into the small details and minute-to-minute decisions that are vital to keeping the program running smoothly.  In this environment, it is easy to lose sight of the long-term objectives and it quickly becomes hard to tell whether you are really making progress.  Evaluation can help to anchor the frenzy of delivering because it requires teams to articulate their goals and to explain the logic of how their short-term actions are helping to bring these goals closer.

In a traditional evaluation, this logic might be laid out in a series of linear steps to be followed like a recipe.  In a developmental evaluation, however, both the small steps and the final objective(s) can be thought of as a series of hypotheses.  This is a crucial difference.  Instead of using the evaluation framework primarily to judge whether a project has been successful (i.e. whether it followed the plan and achieved the predicted result), developmental evaluation instead primarily uses the framework to help the team learn about their program and tests assumptions about both the plan and the end destination.

How do you establish these hypotheses?  A simple, and perhaps familiar, tool to help you set up a developmental evaluation framework is to ask the why, what, how and who questions (and sometimes where, though not in the context of CIL).  By answering these questions about your project, you begin by setting up the broad objectives and gradually get more specific, until you get down to measurable indicators and a plan for how you are going to track and capture your learnings.  This post explains how this tool was used in the Labs.

The Why, What, How and Who of the CILs

Why: Why do the work at all? What does progress look like?

The why question is used to establish the ultimate goals of your project.  Why do the work at all?  The goal of the CILs are to help communities make progress on the complex challenges they face.  What then, does “progress” mean? Early on in the process, we agreed that the kinds of change processes involved in Labs are complex and that any long-term impacts in the community might be felt only years later and direct attributions of each Lab’s impact would be virtually impossible.  However, there were more specific ways that EmcArts wanted to impact the communities after the Labs that could be tracked.

First and foremost, they hoped to build the capacities of Lab participants to engage in social innovation.

Second, the Labs hoped to build new networks, connecting participants to actors in the system they would not normally engage with.

Third, and finally, through some of the prototypes and ideas that emerged from the process, the Labs hoped to produce ‘moments of leverage’: i.e., occasions where a small amount of effort produces a disproportionate impact on the system.

We were testing EmcArts’ hypothesis that by building capacities and networks, and by creating moments of leverage, EmcArts would be giving communities the tools to deal with complex challenges themselves.

What: What are the capacities needed for social innovation?

If the Why question gets at the ultimate long-term goals of the lab, the What question focuses on the immediate impacts.  What do you intend to achieve in the short-term and how will you know if you are having the kind of impact you want?

In asking the why question we uncovered the importance of capacities, networks and moments of leverage.  Now we needed to elaborate on those in detail so that we could measure them.  Of the three, the most diffuse was the notion of capacities.  What are the capacities that individuals and groups need for social innovation?

This was not a question that could be answered right away.  Drawing upon their rich experience with systems change and innovation work in organizations, EmcArts was able to articulate a set of capacities that seemed to be central to their work and could be re-contextualized for the Community Innovation Labs. The capacities are posed as statements, each specific enough that they could be measured through observation and interviewing, while participants could also use to reflect upon themselves.  By collecting data about these capacities, we could gauge in real time the impacts from the Labs, and we could begin to test whether an expansion of these capacities actually produced innovation and systemic change.

How: How do you facilitate innovation in communities through the arts?

The how question is about methodology, tools and techniques.  How is the project going to produce the impacts that it hopes to have?  What tools and techniques are most successful and appropriate?  If answering the previous why and what questions can provide objectives and measurement criteria, answering the how question is often more about developing an evaluation work plan.

In many innovative projects, these questions are impossible to answer at the outset because what is being done has never been tried before, so instead the focus must be on learning and improving as the project develops.  In each Lab, we were able to build upon EmcArts’ experience in

using facilitation techniques as a baseline of knowledge to determine which of these techniques would work best in the new context of Community Labs.

We knew ahead of time that, as evaluators, we needed to regularly capture feedback from participants about how facilitation activities were landing and feed this back to facilitators.  To that end, local evaluators were hired for each Lab to observe all of the workshops in real time and gather data using the measures we had developed.  The local evaluators also provide feedback by sharing the data directly with EmcArts’ own in-house evaluator who in turn relays these observations during regular team debriefs to ensure the facilitation team can respond quickly and learn what was working and what was not.

Who: Who should participate and host the labs?

The Who question is both about who your project involves and who it serves.  Who do you need to engage to be successful, and whose lives will be changed if you are?

For the Labs, we needed to decide if the true end users of the Lab were the participants or the people in the communities they inhabited, and who was suitable for participating in the Labs.  Once again, we worked from a basis of EmcArts’ experience, and from their research into other innovation labs such as those advanced by REOS.  By design, EmcArts sought a broad diversity in participants, for example in their levels of experience and authority, as well as their sectoral background and their viewpoints and interests in the community.  Furthermore, EmcArts identified individuals with an outward attitude of open-mindedness, a willingness to experiment, and an aptitude for collaboration as suitable personalities for the structure of the Lab.

Working from these criteria set forth by EmcArts, we are in the process of gathering data about who has been benefiting most from the Labs in order to continually refine our understanding of how to build successful relationships within each Lab.

Final thoughts

The actual developmental evaluation framework for the Community Innovation Labs is a fairly lengthy document that specifies methods, timings and types of reporting, and includes more detailed assessment criteria.  That said, a lot of the core evaluative framework was uncovered by asking the Why, What, How and Who questions.  Even without a developmental evaluator on your team, asking these questions and answering them in a deep way can be immensely helpful: they can help you understand more about what you are trying to achieve, give you assessment criteria, and reveal gaps in your knowledge where you need to do testing.  In this way, you can maximize your learning when trying to break new ground.

About

Ola Tjornbo is a researcher, writer, educator and systems thinker. He is one half of Archipelago Consultants, and worked with EmcArts as the Lead Developmental Evaluation Consultant for the first two rounds of Community Innovations Labs in Winston-Salem, NC, Providence, RI, and Dallas, TX.