On April 21 ,2020, the Asia Pacific Evaluation Association hosted a webinar where they asked Michael Quinn Patton to explore the implications of the novel coronavirus for evaluation.
The recording of the webinar can be found here.
The webinar generated lively discussion among participants. Below are some of the questions and comments shared in the chat during the webinar.
Evaluations need to be done online these days. How to do sampling ? And reach those who do not have Tech-connect. Does data collection through Google form work?
Google forms - yes it works. but response rate is low and you have to worry about GDPR as well.
Rather than google forms KoBo Collect is ideal for our evaluations. We can use it online, off-line and as mobile app too (PASS Asia). If any one interested to know more on such mobile data collection and IVR methods for remote data collection Samantha of PASSAsia can be of support.
Our concern regarding data collection are 1) risk of not only catching COVID but also of bringing it into vulnerable area and 2) everyone is on panic mode, they dont want to be “distracted” with non-essential processes… how to know when is the right time to come in?
I am very interested in getting a toolbox with remote data and information collection approaches and methods.
Happy to share links to several resources on evaluation in Hard to Reach Areas - from an initiative we organised last year for the European Commission / DG DEVCO. Send an email to marco.lorenzoni@
Am currently designing an evaluation to study how vulnerable people are coping in refugee camps. my worry is not all of them have phone so that they can interviewed. how best can I ensure that my sample is accessible?
I wonder whether is it possible to conduct focus group discussion using remote approach in times like today?
I think that we need to re-define some of the guidelines in data collections under social distancing. Just an idea
How can we unravel what we do not know if we cannot get to and communicate with the people whose perspective matters in light of this pandemic, movement restrictions and weariness of some donors about remote evaluation/reliance on local consultants?
How to advocate elites/national policy maker to evaluate differently in time of covid-19. Is collective action matter? if so in what way?
To Michael’s point about “showing the value of evaluations in a post-COVID19 world”, what are some example of how evaluators can help programme managers and influencers make the “right” decisions in an unfamiliar operating environment?
How do we help people understand the deeper values that are shaping the crisis and how to amplify the desired values for the future direction?
Evaluation Approaches and Rigor
I would be interested in hearing about Michael (and others') recent lessons learned on real time evaluation. The "good enough standard of rigor" makes so much sense to me, but I don't see it implemented effectively in my organization. ...though I know all my colleagues would all like to be able to move in this direction.
Is there a possibility of a common approach to do real-time evaluation of the response to the COVID-19 pandemic. This will help us to bring out the lessons in preparedness for future pandemics.
What do you see as the implications for localisation of the evaluation process? DO you see the role of the international evaluator changing?
Is there are any methodology to conduct system thinking as this will become an important part of evaluation as we go forward?
I would be interested in the "good enough data" approach: what areas can we compromise on in terms of quality data without losing its veracity? For example, how about reducing confidence levels or increasing margins of error?
Thanks for this list! This is a great contribution. I would like to add a caution about us being fact-checkers and about "good enough" data. I have seen many people creating graphs and projections without any epidemiology background. We can be knowledgeable about statistics, but we are not epidemiologists, and we need to also be humble about what we know.
How can we effectively ensure that causal modelling and casual strands result in desired development outcomes amidst such turmoil and uncertainty?
When you say working to differentiate the right from the wrong information when most of the information coming is from the government and is also sometimes made less accurate to avoid panic? What do you suggest to do in such condition?
Do you see HIPPA medical science / public health discipline in competition or superseding human subjects protections in social science under COVID?
I trust evaluators are challenged by all that is happening now….methodology, timing, emergency, sense making, talking to policy and decision makers, involving stakeholders…ethics
within the think global act local, how dynamic should be the ethical protocols to evaluation processes esepcially working with vulnerable populations during this Pandemic?
Are our traditional ethical protocols / rules as evaluators still valid in times of pandemics? what elements should we integrate in our way of working to better address these new challenges?
How can we as a community put out a statement or best practices around Eval ethics, Governace, policy, and data protections or data sovereignty? especially given tribal/First Nations (only sovereign nations-states) so we are not recreating harm and legal/ethical violations?
Governments are unfortunately the entities that reinforce inequality in many cases, even when trying their best to follow an Equity strategy. How can we support each other to identify these counter-productive effects?
What is loop learning?
If you have been following the medical controversy around therapy, research…I would like some comments on learning from that and how our community could help.
EvalIndigenous is working globally on these issues. please remember to include us: https://www.evalpartners.org/
What is the significance difference between Environment day June 5 and Earth Day April 22?
Earth Day may be more of a US-based event, though it is also recognized by the UN: https://www.un.org/en/
Call to Action
If you are interested in getting more involved in these issues, please visit https://www.eval4action.org/.