Introducing Developmental Evaluation:
Innovation is familiar territory for the artist. Novelty, creativity, and unique combinations of ideas fuel artists’ work. EmcArts has shown us that building skills for innovation is also important for arts organizations: investing in R&D, building processes for experimentation, and nurturing cultures that foster an open-ness to change.
Evaluation, on the other hand, has historically been about prediction and control. Evaluation asks questions such as: Was a program implemented as it was supposed to be? Did an intervention achieve a pre-determined result? Were benchmarks met? These are excellent questions, but evaluation’s standardization, linearity and compliance-driven nature are inappropriate to the complexity of innovation. Adaptive and innovative work is organic, non-linear, and involves a steady stream of ups and downs.
How can you judge if something was done as expected when you are in the midst of creating it? If you are doing something for the first time, there is no benchmark, so what do you measure against? Evaluation in its traditional form was a perennial frustration for social innovators. Evaluation couldn’t keep up to the usual pace of adaptation, and often had the effect of constraining the innovation process.
Developmental evaluation is a field of practice that’s emerged as a response to this paradox. Developmental evaluation asks: What if we could blend the creative thinking of innovation with the critical thinking of evaluation? There are elements of evaluation that are really valuable in open-ended nature of adaptability: being systematic and thorough in our thought process, collecting data, drawing upon evidence, and testing assumptions, to name a few. Developmental evaluation is evaluation in support of social innovation and tackling complex issues.
It’s helpful to understand how development evaluation is fundamentally distinct from other evaluation purposes. A formative or summative evaluation is more appropriate to a program or initiative that is more clearly defined, less likely to undergo major adaptations, and is probably at a later stage in of development. For instance:
- A formative evaluation helps to improve a program or initiative that is fairly well developed. Evaluation in this context is about enhancing and fine-tuning: How can we deliver with more efficiency? How can we refine the program to increase effects, reduce costs or make implementation easier? What can we confirm about our initiative that will help us to deliver with more consistency in the future?
- A summative evaluation is meant to be a definitive test of effectiveness. In order to do this, a program or intervention needs to be fully defined and implemented to a set of standards. This evaluation is often best applied to mature programs where ‘all the bugs have been worked out’, and there is an appetite (and resources) for more intensive research.
- A developmental evaluation supports social innovators to explore possibilities for addressing complex challenges, identify and develop innovative approaches and solutions, and inform adaptation in uncertain and dynamic conditions.
Situation recognition is an essential leadership skill. Organizations can make effective use of their evaluation resources and energy when they distinguish between summative, formative and developmental situations. Just as a traditional approach to evaluation is unhelpful for innovation, a complexity-oriented, developmental evaluation is not suited to proving the efficacy of an established model, assessing if a best practice is implemented properly or fine-tuning a well-developed intervention.
Are you wondering if developmental evaluation may be the right fit for something you are working on?
The best way to learn about developmental evaluation is to do it. Start small, and begin to integrate it into your organization’s practice. The four ideas below are simple ways you can start.
- Assess the suitability for developmental evaluation: If you are wondering if a developmental evaluation is right for your organization, this assessment tool can help. This resource guides you through a series of questions to assess whether a situation is appropriate for a developmental evaluation and gauge your organization’s readiness for supporting a developmental evaluation. The tool can be done individually; I recommend doing it as group, with the questions providing a prompt for a team discussion.
- Do a prototype: A prototype is an experiment with the primary purpose of learning something. Prototypes help us to generate rapid feedback. In taking action we probe the complexity of the situation. What emerges? Do our assumptions play out? What criteria develop to tell the difference between “working” and “not working”? We also do prototypes to clarify our thinking. Expectations, priorities, and key concepts become clearer when we move from the theoretical to action. The key is not simply to do something, but to do something with a specific learning agenda with sufficient attention, data, and critical thinking to understand what emerges and inform what is next.
- Put a team member in the developmental evaluation role: When your organization is engaged in the strategy and decision-making relating to the more innovative edge of your work, task someone with playing a developmental evaluation role. By embedding a team member and empowering them to ask evaluation questions you create space for a critical perspective to be at the table as the innovation develops. As people put forward ideas, make decisions and argue preferences, their implicit theory of change becomes more explicit. The developmental evaluator can organize and frame this thinking to help the group surface alignment, uncertainty and underlying assumptions. If you have already engaged in a prototype, your developmental evaluator can lead an after action review as an analytical exercise.
- Do a retrospective developmental evaluation: Looking back at your pattern of activity in responding to a complex challenge can reveal lessons about the nature of the challenge, the potential (and limitations) of your strategies, and the processes that are helpful to support innovation in your organization. In a retrospective evaluation, you can ask your team: What were the major decision points? What were the implications of these choices? What did you learn about the nature of change in the system in which you operate? What processes helped to stimulate or sustain innovation? This can be done as a simple group reflection, or can be more comprehensive if time and energy permits. One organization I worked with did this as a peer-interview exercise. Staff, board members, and some trusted outsiders interviewed one another over the period of a couple weeks using a common interview guide. Everyone shared their summary notes and then convened for a team workshop to dive deeper into the material.
When we hear of evaluation, we often think about accountability. As social innovators, the highest form of accountability is to ourselves. We are accountable for learning, for finding novel solutions to persistent challenges, and for effectively adapting in highly emergent situations. Developmental evaluation brings evaluation discipline and techniques to better equip us to meet this challenge.
Here are some additional resources on developmental evaluation to dig into:
- Developmental Evaluation: Applying complexity concepts to enhance innovation and use by Michael Quinn Patton (Guilford Press, 2010).
- Developmental Evaluation Exemplars: Principles in practice by Michael Quinn Patton, Kate McKegg and Nan Wehipeihana. (Guilford Press, 2015).
- A Developmental Evaluation Primer. Jamie Gamble. (2008). Montréal: The J.W. McConnell Family Foundation.
- Evaluating Social Innovation. Hallie Preskill and Tanya Beer. (2012). FSG.