Webinar Recording available here.
Click here to read a summary blog about webinar with resource links.
CHAT QUESTIONS and FOLLOW-UP QUESTIONS (in red) and PARTICIPANTS’ COMMENTS, WITH MQP RESPONSES
Questions (in red) grouped together. There are many questions so I’ll try to keep responses short. This is an ongoing dialogue.
Will the PowerPoint be available?
MQP: The presentation is a work in progress, the PowerPoint is therefore a work in progress. The webinar recording is available but the PP is not ready for dissemination. I’m working on an article that will be published later this year. That will have the full, polished presentation. But that will take some time and these matters are urgent, so the webinar is an immediate effort to generate dialogue and discussion about transformation at this critical time of global crisis.
OECD criteria rule the evaluation of development programmes. Has the transformational vision been presented and discussed with OECD/DAC evaluation people, donors etc.? Unless they buy in, evaluators of international programmes can do little. Evaluations in that sphere are dealing with a highly political context.
MQP: A major theme in the feedback presented to DAC during the revision consultation process was that use and application of the DAC criteria had become mechanistic and routine. There was lively discussion of alternative criteria during the revision process. For an example, see Zenda Ofir’s set of blog posts on the criteria and the many responses to her posts. In the end, the revised criteria largely ignored these comments and concerns and took the path of minimizing change because that was the dominant perspective among those who have used and grown comfortable with the existing criteria. It therefore falls to others to suggest alternative criteria appropriate to evaluating transformation since, in my judgment, DAC has failed to do so. Zenda Ofir’s blog and here.
By criticizing the DAC criteria and proposing alternative criteria, aren’t you creating conflict and division within the evaluation community? At this time of crisis, shouldn’t we attempt to be united and present a united front to the world?
MQP: Officially, the guidance offered in introducing the revised DAC criteria invite thoughtful consideration of their appropriate use and additional or alternative criteria as needed. So, posing alternative criteria is not, in and of itself, a hostile act. Criticizing the DAC criteria may be perceived that way. The DAC criteria have the backing of the powerful members of the OECD. We have a long-standing commitment as evaluators to speak truth to power. My critique honors that commitment.
But having the truth spoken to them is seldom well received by the powerful. I believe the truth is that the DAC criteria communicate a conservative, business-as-usual message that is out of touch with the global climate emergency we face. All evaluators will have to decide for themselves whether that is the case. I present alternatives to stimulate that discussion and dialogue.
Let me add that the presentation of the DAC criteria acknowledge that, while more radical alternatives were proposed and considered, the dominant consensus was to minimize changes and communicate a steady as she goes message.
It is perhaps worth noting that the survey soliciting feedback about whether the DAC criteria needed revision provided no context by way of mentioning the global climate emergency or calls for transformation. Might there have been more support for significant reform of the DAC criteria if a transformational context had been provided? We’ll never know. Such a context was not provided, so what emerged was reaffirmation of normal evaluation in support of the status quo and a modest openness to marginal, incremental change.
As for creating conflict and division within the evaluation community, my 45-year evaluation career has witnessed a great variety of perspectives, diversity of approaches, and debates about everything from how evaluation is defined to what methods are preferred. Our united front is that evaluation is important, ought to occur, and ought to be used. How to do so generates great differences of opinion. I don’t expect that to change. And how boring would it be if we all agreed. I miss the old days when we actually had plenary debates at AEA.
To be clear, are you proposing that everyone adopt your alternative criteria instead of the DAC criteria?
MQP: As I said in the webinar, it seems to me that there are two options for determining criteria for evaluating transformation. One approach would be to interpret the revised DAC criteria in ways that make them relevant to transformation. That is what the DAC guidance document recommends by including transformation under the Impact criterion.
Impact addresses the ultimate significance and potentially transformative effects of the intervention. It seeks to identify social, environmental and economic effects of the intervention that are longer term or broader in scope than those already captured under the effectiveness criterion. (OECD/DAC Network on Development Evaluation, 2019b, p. 11)
The DAC approach, then, is to subsume transformation as one possible focus under the broader criterion of IMPACT.
The alternative is to elevate attention to transformation by developing criteria that specifically highlight the nature, scope, and breadth of changes connoted by the term TRANSFORMATION. Responding to the systemic threats of the climate emergency requires audacity: emergency responses, by definition, disrupt business-as-usual mindsets, modalities, and methods. Yet, policy makers have yet to grasp the nettle and evaluators are mostly going about their evaluations in a business-as-usual mode. In offering examples of alternative criteria to suggest what transformation-specific criteria might communicate, I aim to stimulate dialogue and debate. The criteria offered in the webinar result from two years of reflection, consultation, workshopping, and feedback about criteria for transformation with others. In sharing them, I mean for them to illustrate possibilities and stimulate further contextual adaptation not to be treated as universal, standardized, and/or mandated criteria.
I am preparing a paper for publication later this year that will go into much more detail about the origins and applications of the criteria. For now, I’m simply hoping to slow the juggernaut implementation and adoption of the revised DAC criteria and get people to stop and think about the situation the world is in and ask how evaluators ought to respond, including our role in supporting transformation, and the criteria we ought to use in evaluating transformation.
While DAC criteria are relevant for Projects evaluation. They are not relevant for evaluating transformation of systems. So how we should proceed?
MQP: That’s the point of this webinar, to encourage evaluators to develop with stakeholders criteria that are appropriate to the nature of the intervention, especially transformation, and not just mechanically adopt and apply the DAC criteria. The point is to be thoughtful about the criteria and match them to the situation
What data collection methodologies and sampling approaches would be used differently in evaluating transformation and systems vs evaluating a project?
MQP: Evaluating systems transformation will involve mapping system relationships, boundaries, perspectives, interrelationships, and dynamics. See Guidance from the AEA Systems in Evaluation Topical Interest Group.
There were several questions about how to get started, whether evaluating transformation is practical at a local level. For example: “How could this evaluation approach apply to small Project in practice with limited Resources and time frame? Until donor programming structure changes, will this work?”
MQP: As an all change processes, you start where you are and do what you can. At every level and in every evaluation ask questions about what criteria are appropriate given the global climate emergency that affects everyone. At least put transformational criteria on the table for consideration, to raise consciousness about our collective responsibility, and to address the global emergency at every level.
I reckon “catalytic function” as an evaluation criterion almost covers everything. We evaluate whether the intervention changes systems at scales and specialties around it and way beyond the proportion of the intervention.
MQP: Jeph, Thanks for this. I like the “catalytic” language and think it captures the role that evaluation can play in supporting transformation. The “Interconnections Momentum” criterion can be elaborated as tracking catalytic convergence.
Is there value in advocating for a both/and approach to using established evaluation criteria for project and program use, and also advocating for substantial increase in additional attention to evaluating transformation with a focus on use at a systems level?
MQP: Certainly the point is to do what is appropriate, meaningful, and useful in a given context. I don’t mean to suggest an orthodoxy or rigidity about criteria. The history of attempting to integrate approaches, in my experience, is that the both/and approaches, while inclusive and ideal, in principle, seldom work in practice. One approach dominates the other. Evaluations that support both learning and accountability often end up being more about accountability than learning. Evaluations that track project accountability with DAC criteria may have a hard time, with limited resources, addressing systems transformation at the same time. But all of this is contextual and situational, so some combination and integration of criteria may well be appropriate. I’ll look forward to hearing how that works out.
How do we move beyond project thinking when funders continue to design programmes and also commission evals according to this logic?
and
Great idea but how/when the conventional development approach respect with increasing scope and time to fit with this approach and how to bring this concept / theory into practice (i.e. intervention at local level) are my major concerns once I hear this webinar!!
and
How can the quick uptake of this mindset change (i.e. transformation evaluation through the blue marble perspective) be feasibly mainstreamed?
and
How are you advocating for intervention implementers to have this approach, especially in smaller and shorter interventions in in development countries?
MQP: Blue Marble Evaluation advocates that evaluators be involved in conceptualizing, designing, and framing interventions so as to bring Blue Marble transformational principles into the discussion at the beginning. But wherever you are in the process and however you are able to engage, the point is to offer alternative criteria for dialogue and negotiation, not just accept traditional and dominant criteria because they are traditional and dominant.
When should we use evaluation for transformation? Who will be the stakeholders? How often should it be used? What about the budget required for this ? Is this a state or government initiative?
MQP: Use it wherever people are concerned with interventions aimed at dealing with the future sustainability and equity of humanity. Hopefully, that becomes everyone, everywhere. The point is for evaluators to be generating momentum in that direction by the criteria we offer, the questions we ask, and our interactions with stakeholders, wherever and whenever.
When we use the term “environmental sustainability” are we reducing what are complex dynamics among environmental, social, and economic systems to something that gives the appearance that it focuses only on the environment? I worry that when many of my evaluation colleagues see the term “environmental sustainability,” they think that this is not their evaluation “gig” . Thoughts?
and
Evaluation” equate to ecological analysis? Like, hire an ecologist as part of your team? Or is it something further, beyond ecology? (Where are the values in ecological analysis and are those explicit enough to support evaluation?)
and
I question if the ecosystem is always our scale of responsibility in every evaluation.
MQP: Transformation Fidelity includes a multidimensional, cross-silos, local/global approach to sustainability. Finding the right level of intervention, appropriate and meaningful language and criteria to communicate this multidimensional nature of sustainability is one of our challenges. It is everyone’s gig hopefully.
Every intervention occurs within some ecosystem, which involves interactions between humans and nature. That is the sense in which I’m referring to ecosystem.
How is criterion #6, Interconnections Momentum, different from the new DAC Coherence criteria.
MQP: The addition of “coherence” as a new criterion addresses some of the limitations of the original five, and is meant to address climate change, poverty reduction, equity, and other goals at whatever level the evaluation is occurring, based on the goals of the intervention, according to the DAC guidance. But “coherence” doesn’t evoke or connote, at least for me, the nature of complexity or the magnitude and urgency of transformation. You have to dig deep and interpret broadly to arrive at that understanding. The very label “coherence” evokes orderly stability. As I’ve emphasized throughout the webinar, the overall message of the DAC criteria revision is business as usual. Carry on. Continuity reigns supreme. Complex dynamic transformation is messy, uncertain, unpredictable, and uncontrollable.
The proposed alternative criterion of Interconnections Momentum is based on a theory of transformation articulated in the Blue Marble Evaluation book about how transformation occurs from the intersection, interdependence, and interactions of multiple interventions (see chapters 3, 13-15). Tracking the momentum of interconnections does not assume coherence, but assumes interaction effects, which are subject to inquiry and evaluation.
If we educate people how our choices both in the past and in the present resulted in these ailments of capitalist systems and then to open up the imagination to what could alter the emergent complex systems it does help. If people fail to see the connections between culture and ailments, why would they change? How could they change?
MQP. The Blue Marble Evaluation theory of transformation posits that multiple actors taking multiple actions at multiple levels is required to create the critical mass for tipping points to transform systems. No single intervention or strategy will suffice. Tracking the momentum of interconnections and supporting alignment is a contribution evaluation can make to transformation.
Seeing the value of say interconnectedness, how would you suggest to an evaluator whose TORS indicate the use of a set of criteria, How you practically go about it?
MQP. You negotiate within the context of the evaluation what specific criteria are appropriate and how to make them meaningful and useful within that context. There isn’t a generalizable, universal answer, nor a standardized way of proceeding. It begins by bringing alternative criteria to the table, discussing their meaningfulness, and negotiating what is possible and useful.
Is the “efficiency” criteria making noise in the group of criteria? (Cost/value thinking). Shouldn’t it be part of Audits instead of development evaluations?
MQP: True cost accounting is a systems level evaluation approach, far different from any notion of auditing I’ve seen. Here are details about the True Cost Accounting criterion I discussed.
What do you think is the role of trust (as the social fabric) for evaluating transformation?
and
Ditto question regarding the role of trust… does the quality of relationships fall under one or more of the criteria?
MQP: Building trust is a fundamental part of Utilization-Focused Evaluation which remains an overarching umbrella and framework for Blue Marble Evaluation. Here is a blog I wrote blog on trust and transformation.
I began the webinar by noting that we are in the midst of a global health emergency that is transformative. Transformation itself is neutral. It is about scale, scope, depth, and breath of systems change. We are seeing and experiencing global systems transformation in dealing with the coronavirus. It will be sometime before we know the full scope and nature of this transformation, and its cumulative and aggregate effects into the future. However it unfolds, it will be a cautionary tale for evaluation carrying lessons for evaluating transformation.
Comments from webinar participants
- I don’t think that donors are the main issue but one of the issues, I think that one of the issues is us evaluators not taking transformational action but accepting the status quo as is. Once we begin thinking in the transformational change mindset, then everyone may follow.
- I agree with Michael: pose the questions – this is great!
- Exactly what is needed is more courage, boldness. Why should someone else’s perspective be more important than ours. Wisdom has authority. Foolishness doesn’t, even if the fool is the king.
- There’s a lack of courageous vision.
- It does not help to make long lists of ailments of the capitalist systems. What are opportunities today to change how the political systems work; how politicians think; how different classes in America think?…..
- Think that including ecologists in teams could be important but more important is to understand the complexity of interactions and systems. Systems evaluation has built off of the work by many ecologists and those working in the resilience space.
- Resource from Jonny Morell: I’m a big believer in the biotic hierarchy. Do we really want to focus on ecosystem when we can think in terms of other levels as well – organism, population, community, below the ecosystem level system. I’m not so sure the goal should be to tolerate all tradeoffs below the ecosystem level. If people want to see it let me know. I have a lot of goodies on applying ecology and evolutionary biology to evaluation here.
- Invitation from Jonny Morell: I’m collecting models of transformation. If you have any ship them to me. Thanks. I need examples because I’m working of a metatheory of transformation based on complexity and applicable to all theories of transformation. Click here to email him
Fore more details, check out my book: Blue Marble Evaluation, Guilford Press
