Webinar Recording available here.
Webinar FAQ responses available here.
PowerPoint: The presentation is a work in progress, so the PowerPoint is a work in progress. The webinar recording is available but the PP is not ready for dissemination. I’m working on an article that will be published later this year. That will have the full, polished presentation. But that will take some time and these matters are urgent, so the webinar is an immediate effort to generate dialogue and discussion about transformation at this critical time of global crisis.
WEBINAR SUMMARY
Opening Context
“We are the first generation to know that we are destroying the planet,
and the last generation that can do anything about it.”
Tanya Steele, CEO, World Wildlife Fund (2018)
Beyond the current Coronavirus pandemic emergency, we are experiencing a longer-term and larger-scale climate emergency. UN Secretary-General Antonio Guterres recently spoke of not letting the climate emergency be eclipsed by the current health emergency. We have moved from the Holocene Epoch of the last 12,000 years into the Anthropocene, the current epoch where humans are having more impact on Earth than natural phenomena. Research shows that we are overrunning critical planetary boundaries in arenas ranging from biodiversity loss to the nitrogen cycle. We have arrived at a place that Martin Luther King, Jr., foresaw: meets this moment-“Every nation must develop an overriding loyalty to mankind as a whole in order to preserve their individual society.”
For more on the climate emergency and its implications for evaluation see two recent blogs on the Blue Marble Evaluation website:
- Climate Emergency from a Blue Marble Perspective
- The Interconnection between the Global and the Personal: A Blue Marble Perspective
Beyond a project mentality
The field of evaluation grew up with a project mentality. Collectively, evaluators have become accomplished at evaluating project outcomes, goal implementation, and generating recommendations. Evaluators are now being asked to evaluate community impacts, sustainability, and global systems change. All of that is a part of recognizing that we as humanity are in a different place now, and transformation is the clarion call of our times.
In the Anthropocene, we need to think beyond autonomous projects and programs to major systems transformation. A key framework for doing these will be the criteria we use, the core of evaluation.
Criteria Matter
- Criteria constitute the nuclear core of evaluation’s energy function: rendering judgment. Without criteria there can be no judgment. Without judgment there can be no evaluation.
- Criteria express, manifest, encompass, make explicit, and operationalize what is valued. Criteria mediate the conversion of values into judgments.
- Criteria prioritize what is important.
- Criteria direct what questions to ask, data to collect, and results to highlight.
- Criteria focus evaluation reporting and conclusions: To what extent and in what ways have criteria been met?
DAC criteria
The Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC), based on work begun in 1984, developed, endorsed, and disseminated five evaluation criteria in 1991: Relevance, Effectiveness, Efficiency, Sustainability, Impact. These criteria became widely adopted as defining evaluation quality for evaluating all international development and humanitarian projects, programs and policies. Niels Dabelstein, a distinguished thought leader in international development evaluation, wrote a detailed history of 30 years of the work of the DAC Network on Development Evaluation (OECD, 2013). He concluded the DAC criteria have “had a profound impact on development evaluation. These have become the widely accepted criteria upon which every development evaluation will base its assessments—or make excuses for why not” (OECD, 2013, p. 33).
Robert Picciotto, former director of the World Bank’s Independent Evaluation Group believes that the DAC criteria contributed significantly to improving the quality of development evaluations, especially in formulating multiple criteria that went beyond the singularly dominant criteria of goal attainment and return on investment. “Historically, the addition of relevance, sustainability and impact to the list made a very big difference. It was nothing short of revolutionary” (personal interaction). He goes on to caution that “criteria are a floor rather than a ceiling. The DAC criteria are necessary but far from sufficient.” Criteria are not enough to guarantee quality. As Megan Kennedy-Chouane, Head of Evaluation, OECD, put it in a webinar introducing the revised DAC criteria: “Good evaluations can be conducted with bad criteria and bad evaluations can be conducted with good criteria.” Still, the criteria matter. The quality of the “floor” affects the solidity of the above-floor structure.
Revising the DAC Criteria
In light of the Global Agenda for 2030 and the Sustainable Development Goals (SDGs), DAC began reviewing whether the criteria needed revision. A consultation process from March to October 2018 included interviews with key stakeholders, a consultation workshop, discussions at international meetings/seminars throughout the world, an OECD DAC Network member survey, and a public survey with stakeholders. The survey received 691 survey responses with over 700 pages of qualitative comments. In addition, eleven development agencies submitted formal commentaries. The concluding survey item asked whether the current DAC criteria should be retained, adapted or removed. Over 89% recommended retaining the existing criteria.
Building on this feedback, the revised criteria were published in late 2019 in a report entitled Better Criteria for Better Evaluation. The document lays out adapted definitions for relevance, effectiveness, efficiency, impact and sustainability, and for one new criterion, coherence.
REVISED DAC CRITERIA
The revised criteria are useful for those who want to continue designing and evaluating projects and programs in familiar, comfortable, well-known, and well-traveled ways. The revision amounts to some fine-tuning and tweaking, but is basically business-as-usual. What was once innovative and even “revolutionary” has become staid, conservative, and common place. The criteria apply primarily to summative and accountability purposes of evaluation, important to be sure, but hardly relevant to the great panorama of evaluation approaches and purposes that have emerged in the last three decades. The unchanged labels of the five original DAC criteria – relevance, effectiveness, efficiency, impact, and sustainability – carry the message that things can go on as before. You have to study the revised definitions carefully and scrutinize them diligently to detect any meaningful substantive changes. I suspect that few who use the current criteria, whether in commissioning evaluations, conducting them, or judging their quality (meta-evaluation) will notice any differences or study the accompanying guidance document meant to explicate adaptations.
The addition of “coherence” as a new criterion addresses some of the limitations of the original five, meant to address climate change, poverty reduction, equity, and other goals at whatever level the evaluation is occurring, based on the goals of the intervention. But “coherence” doesn’t evoke or connote, at least for me, the nature of complexity or the magnitude and urgency of transformation. You have to dig deep and interpret broadly to arrive at that understanding. The very label “coherence” evokes orderly stability Indeed, the overall message of the DAC criteria revision is business as usual. Carry on. Continuity reigns supreme.
Alternative pathways forward
It seems to me that there are two options for determining criteria for evaluating transformation. One approach would be to interpret the revised DAC criteria in ways that make them relevant to transformation. That is what the guidance document recommends by including transformation under the Impact criterion.
Impact addresses the ultimate significance and potentially transformative effects of the intervention. It seeks to identify social, environmental and economic effects of the intervention that are longer term or broader in scope than those already captured under the effectiveness criterion. (OECD/DAC Network on Development Evaluation, 2019b, p. 11)
The alternative is to elevate attention to transformation by developing criteria that specifically highlight the nature, scope, and breadth of changes connoted by the term TRANSFORMATION. Responding to the systemic threats of the climate emergency requires audacity: emergency responses, by definition, disrupt business-as-usual mindsets, modalities, and methods. Yet, policy makers have yet to grasp the nettle and evaluators are mostly going about their evaluations in a business-as-usual mode. In what follows I offer examples of alternative criteria to suggest what transformation-specific criteria might communicate. The criteria offered here result from two years of reflection, consultation, workshopping, and feedback about criteria for transformation with others. In sharing them here, I mean for them to illustrate possibilities and stimulate further contextual adaptation not to be treated as universal, standardized, and/or mandated criteria.
EVALUATION CRITERIA FOR EVALUATING TRANSFORMATION
1. Transformation Fidelity. Assess the extent to which the realities of transformational change initiatives match transformational aspirations and rhetoric.
- Ensure that what is called transformation constitutes transformation.
- Evaluate whether and how what is called transformational engagement constitutes a trajectory toward transformation.
There is a lot of hype around transformation as the term has become widely used and taken on a trendy cachet. Claims of transformation abound. Ensuring that such claims are meaningful and consistent with the face validity of the construct becomes a transformational evaluation priority under this criterion. Thus, the fidelity criterion aims to bring some rigor to the very notion of transformation.
Examples: Dramatically increasing the food available through food shelves involves increased impact but is not transformational. Changing the system so the food shelves are unnecessary and everyone has adequate food is transformational.
Increasing the number of beds in shelters for the homeless constitutes increased impact. Changing the housing system so that no one is homeless is transformational.
2. Complex systems framing. Assess systems transformation using systems thinking principles and complexity concepts.
- Ensure that transforming systems is the transformational focus.
- Apply complex systems understandings and frameworks in evaluating transformation.
Transformation is not a project or program. Transformational initiatives are not targeted to achieving SMART goals, which is the traditional criterion of effectiveness. Transformation means changing systems, which means dealing with complexity dynamics in a world characterized by turbulence, uncertainty, unpredictability, and uncontrollability. The focus of evaluation, the evaluand in our jargon, is transformed systems.
Evaluating systems transformation will involve mapping system relationships, boundaries, perspectives, interrelationships, and dynamics. See Guidance from the AEA Systems in Evaluation Topical Interest Group.
Evaluating complex dynamic systems means dealing with nonlinearities, interconnectedness, emergence, co-evolution, adaptation, and path dependence.
Resources:
Dealing with Complexity in Development Evaluation
3. Resilient Ecological Sustainability. Evaluate transformational sustainability as manifesting ecosystem resilience and adaptability at the nexus between humans and the environment.
- Employ a dynamic view of sustainability.
- Make the ecosystem viability the focus of sustainability not a program, project, or intervention.
Climate change, water shortages and other environmental crises are telling us that our use of the the Earth’s finite resources is not sustainable. Ecological sustainability has emerged as a priority criterion for evaluation. Neither the original nor the revised DAC criteria address ecological sustainability as a priority. The DAC “sustainability” criterion focuses on continuity: Will benefits attained last? This criterion is quite understandable from a funder perspective. Funders want to see change and want those changes to be maintained. Evaluators are commissioned to determine both whether the desired and intended changes occurred, and if so, whether they can be continued and sustained. This is fundamentally an accountability perspective imposed from the perspective of funders who must demonstrate that they have made good use of assets entrusted to them. This conceptualization of sustainability as continuation is linear, mechanistic, and static in formulation and evaluation.
A note on language: I have called this criterion “Resilient Ecological Sustainability” to distinguish it from the DAC criterion called “Sustainability.” Because the DAC criterion is really about continuity not resilient ecological sustainability, I felt a need to include the longer terminology to be clear about the orientation and substance of this criterion. In a different context simply referring to sustainability might suffice, or other adjectives might be used (e.g., adaptable sustainability).
Sustainability as a universal evaluation criterion
The theme of the 2019 conference of the International Development Evaluation Association (IDEAs) in Prague was Evaluation for Transformative Change supported by a publication with that title (Berg, Magro, & Mulder, 2019). At the conclusion of the conference, participants from around the world adopted a “Declaration on Evaluation for Transformational Change.” The sixth item states:
Focus on sustainability
In all our evaluations, we commit to evaluating for social, environmental and economic sustainability and transformation, including by assessing contextual factors and systemic changes. We commit to assessing and highlighting, in all evaluations, unintended negative social, economic and environmental effects. (Item 6 of 10 in the Declaration. For the full declaration, see IDEAS, 2019).
All evaluations, emphasis on ALL, are mandated to include attention to sustainability, that is, resilient ecosystem sustainability. The global emergency requires action and engagement by everyone everywhere. This isn’t about whether intervention outcomes continue. This is about whether the world is transformed. Transformation involves multiple, interdependent dimensions of sustainability. Andy Rowe has sounded a “call to action” aimed at evaluators to incorporate a two-system framework connecting human and natural systems to design “sustainability-ready evaluations”.
Resource:
Evaluating Sustainability: Evaluative Support for Managing Processes in the Public Interest
4. Eco-efficient Full Cost Accounting
- Document and assess the full costs and benefits of systems transformations, including economic, social, and environmental dimensions.
- Compare the full costs and benefits of baseline versus transformed systems. Evaluate whether, how, and to what extent transformational engagement generates net eco-efficient benefits.
Eco-efficient full cost accounting offers a framework for examining transformation from unsustainable development to sustainability. This means looking beyond the traditional DAC efficiency criterion of examining the comparative costs (inputs) and benefits (outcomes) of an intervention within the boundaries of the intervention, essentially a closed system analysis. Eco-efficiency opens and expands the analysis to examine the effects of creating goods and offering services on the use of environmental resources, effects on ecosystems, human health, community well-being, and possible contributions to climate change, waste, and pollution.
For details about full cost accounting see The Economics of Ecosystems and Biodiversity’s publication, Measuring what Matters in Agriculture and Food Systems and my book review, Measuring what matters in agriculture and food systems: A synthesis of the results and recommendations of TEEB for Agriculture and Food’s Scientific and Economic Foundations report and TEEB for agriculture & food: Scientific and economic foundations. American Journal of Evaluation, Vol. 40(3) 459-464
5. Diversity/Equity/Inclusion (DEI)
- Evaluate how transformational engagement manifests the values of diversity, equity, and inclusion.
- Evaluate whether, how, and to what extent transformational engagement enhances systems level diversity, equity, and inclusion.
The 21st century has witnessed a growing concentration of wealth and increasing economic inequality. The wealth of 62 people is equal to the wealth of the poorest 3.5 billion people, and the richest 1% have more wealth than the other 99%. Humanity as a whole has had much less impact on Earth than the 1% who acquires more than 80% of the world’s wealth generated in a year (Oxfam, 2019; Quackenbush, 2019).
Transformation as a values-based vision flows from the hopes expressed in the Universal Declaration of Human Rights (adopted in 1948) and subsequently in the Declaration of the Rights of the Child (adopted in 1959). Global DEI norms and values are expressed and codified in the Declaration on the Rights of Indigenous Peoples and the International Women’s Bill of Rights. All people, all of humankind, young and old, have the right to food, water, sanitation, security, shelter, respect, and dignity. As expressed in the ambitious Sustainable Development Goals (SDGs) adopted in 2015, entitled Transforming Our World (UN, 2015), transformation means No One Left Behind. Thus, in this vision, sustainability and equity, combined, are the foundation for transformation. This links sustainability to equity and transformation. For example, Amnesty International established as its top priority for 2020 tackling the climate crisis by supporting a “human rights-centered transition to a green economy.”
This vision for evaluation’s role in the world was articulated in the theme of the 2014 annual conference of the American Evaluation Association (AEA): Visionary Evaluation for a Sustainable, Equitable Future, which was followed by a book of the same name by Beverly Parsons, Lovely Dhillon and Matt Keene. Two important evaluation thought leaders, Stewart Donaldson and Robert Picciotto, R. (2016) also edited a book on Evaluation for an Equitable Society.
The Equitable Evaluation Initiative (EEI) promotes the use of evaluation as a tool for advancing equity (e.g., TCC Group, 2019) and posits that all evaluations should address equity. This makes equity, like sustainability, a universal criterion.
6. Interconnectedness momentum. Identify, understand, and evaluate the interconnections that are essential and integral to transformation.
- Evaluate whether, how, and to what extent interconnections among people, networks, institutions, ideas, and movements are deepened and enhanced to support, nurture, catalyze, and accelerate transformational trajectories.
- Evaluate whether, how, and to what extent dysfunctional and constraining interconnections are disrupted and broken to liberate positive transformational energy and momentum.
Identifying interconnectedness momentum as a criterion for designing and evaluating transformation is based on the nature of transformation. Systems are defined by interconnections among elements in the system. Transforming systems means changing interconnections within and between systems. To evaluate interconnections momentum is to evaluate movement toward critical mass and tipping points in transformation theory. Interconnectedness could be subsumed under the criterion of complex systems framing, but mapping, tracking, understanding, and evaluating interconnections is so essential to evaluating transformational trajectories that I believe it deserves elevation to a priority criterion for evaluating transformation. The evaluation question for this criterion is: To what extent and in what ways are intensifying interconnections generating momentum toward systems transformation?
Summary: Six Criteria for Evaluating Transformation
- Transformation fidelity involves examining the connection between transformation rhetoric and reality – evaluating the scale, scope, and pace of actual transformational engagement which means reality-testing the vision against what is being done and accomplished.
- Complex systems framing connects system thinking and complexity concepts to define the processes, nature, and results of transformation.
- Eco-efficient full cost accounting makes transparent the interconnections between an intervention’s direct costs and benefits in relation to broader environmental and human/societal systems costs and benefits (economic externalities).
- Resilient ecosystem sustainability invites evaluation of adaptive capacity interconnections between ecosystem and human systems sustainability over time.
- Diversity/equity/inclusion focuses attention on the interconnections between who is engaged in and affected by processes of inclusion and diversity toward transformational aspirations of greater equity.
- Interconnectedness momentum calls attention to the transformational implications of integrating across divisions, silos, differing perspectives, historical divisions, and competing interests toward a vision of a more sustainable and equitable future.
I want to reiterate that the criteria I’ve offered for evaluating transformation are meant to be illustrative of what is possible.
Let me close by offering three conclusions about criteria generally:
- Any standardized universal set of criteria cannot possibly cover the rich variety of kinds of interventions and evaluations in this marvelously diverse world.
- There is substantial value in having the primary intended users of any evaluation, engage in the process of deliberating what criteria are appropriate for a given evaluation.
- Articulating criteria is a necessary but insufficient action. The criterion identified must then be applied diligently, implemented systematically, and used appropriately.
Criteria in the context of the global climate emergency
We stand at a critical juncture in our collective efforts to limit dangerous global heating. By the end of the coming decade we will be on one of two paths.
One is the path of surrender, where we have sleepwalked past the point of no return, jeopardizing the health and safety of everyone on this planet. Do we really want to be remembered as the generation that buried its head in the sand, that fiddled while the planet burned?
The other option is the path of hope. A path of resolve, of sustainable solutions.
-UN Secretary-General António Guterres (2019)
Let me conclude with some reflections about the importance of having explicit criteria for transformation. Evaluators need to engage and take seriously the global emergency — and do so with a sense of urgency and transformational scale. The DAC criteria are intentionally technical and neutral. Relevance, effectiveness, efficiency, impact, continuity (sustainability), and coherence communicate evaluation priorities. They are framed as applicable to any and all interventions. That is both their strength and weakness, their strength in that they posit universal applicability, their weaknesses in that they are correspondingly milquetoast and bland. They are old news. They fail to inspire. In contrast, if evaluators are to become part of the solution to the global crisis, we must engage with the scale, scope, and urgency of transformation. The criteria offered here as examples of what is possible are values-based. They make it clear that evaluators have a stake in transformation and manifest our stake in the future of humanity by addressing transformation explicitly and directly. Their values-based nature will be a source of controversy, even disdain, for those who prefer to treat evaluation as an independent technical activity. Transformation-focused criteria treat evaluation as part of the transformation process and evaluators as having skin in the game, the game being the future of humanity on Earth.
Distinguished management consultant Peter Drucker asserted that:
“The greatest danger in times of turbulence is not the turbulence—it is to act with yesterday’s logic” (quoted by Carrigan, 2010, p. 99).
Applied to evaluation this becomes:
The greatest danger for evaluators in times of turbulence is not the turbulence – it is to act with yesterday’s logic and criteria.
Thanks to Charmagne Campbell-Patton, Glenn Page, and Samuel Matey for assistance with this review.
