Hostname: page-component-78c5997874-t5tsf Total loading time: 0 Render date: 2024-11-12T22:18:36.921Z Has data issue: false hasContentIssue false
  • English
  • Français

Peer-Reviewed Validation of a Comprehensive Framework for Disaster Evaluation Typologies

Published online by Cambridge University Press:  17 June 2019

Diana F. Wong ,
Caroline Spencer ,
Leanne Boyd ,
Frederick M. Burkle Jr.  and
Frank Archer
Diana F. Wong*
Affiliation:
Monash University Disaster Resilience Initiative (MUDRI), Monash University, Melbourne, Australia New South Wales (NSW) Health, Sydney, New South Wales, Australia
Caroline Spencer
Affiliation:
Monash University Disaster Resilience Initiative (MUDRI), Monash University, Melbourne, Australia
Leanne Boyd
Affiliation:
School of Nursing and Midwifery, Monash University, Melbourne, Australia Cabrini Health, Malvern, Victoria, Australia
Frederick M. Burkle Jr.
Affiliation:
Monash University Disaster Resilience Initiative (MUDRI), Monash University, Melbourne, Australia Harvard Humanitarian Initiative, Harvard University, Cambridge, MassachusettsUSA Woodrow Wilson International Center for Scholars, Washington, DCUSA
Frank Archer
Affiliation:
Monash University Disaster Resilience Initiative (MUDRI), Monash University, Melbourne, Australia
*
Correspondence: Diana Wong, PhD, MCP Nsg Monash University Disaster Resilience Initiative (MUDRI), Monash University, Melbourne, Australia E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Introduction:

The Comprehensive Framework for Disaster Evaluation Typologies, developed in 2017 (CFDET 2017), aims to unify and facilitate agreement regarding the identification, structure, and relationships between various evaluation typologies found in the disaster setting. A peer-reviewed validation process sought input from international experts in the fields of disaster medicine, disaster/emergency management, humanitarian/development, and evaluation. This paper discusses the validation process, its results, and outcomes.

Research Problem:

Previous frameworks, identified in the literature, lack validation and consistent terminology. To gain credibility and utility, this unique framework needed to be validated by international experts in the disaster setting.

Methods:

A mixed methods approach was designed to validate the framework. An initial iterative process informed an online survey which used a combination of a five-point Likert scale and open-ended questions. Pre-determined consensus thresholds, informed by a targeted literature review, provided the validation criteria.

Results:

A sample of 33 experts from 11 countries responded to the validation process. Quantitative measures largely supported the elements and relationships of the framework, and strongly supported its value and usefulness for supporting, promoting, and undertaking evaluations, as well as its usefulness for teaching evaluation in the disaster setting. Qualitative input suggested opportunities to strengthen and enhance the framework. There were limited responses to better understand the barriers and enablers of undertaking disaster evaluations. A potential for self-selection bias of respondents may be a limitation of this study. The attainment of high consensus thresholds, however, provides confidence in the validity of the results.

Conclusion:

For the first time, a framework of this nature has undergone a rigorous validation process by experts in three related disciplines at an international level. The modified framework, CFDET 2018, provides a unifying framework within which existing evaluation typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting and facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.

Keywords

disastersevaluationframeworkvalidation
Type
Original Research
Information
Prehospital and Disaster Medicine , Volume 34 , Issue 3 , June 2019 , pp. 230 - 240
Copyright
© World Association for Disaster and Emergency Medicine 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Wong, DF, Spencer, C, Boyd, L, Burkle, FM Jr, Archer, F. Disaster metrics: a comprehensive framework for disaster evaluation typologies. Prehosp Disaster Med. 2017;32(5):501–514.CrossRefGoogle ScholarPubMed
Task Force on Quality Control of Disaster Management (TFQCDM); World Association for Disaster and Emergency Medicine (WADEM); Nordic Society for Disaster Medicine. Health disaster management guidelines for evaluation and research in the Utstein style. Prehosp Disaster Med. 2003;17(Supplement 3):1–177.Google Scholar
Kulling, P, Birnbaum, M, Murray, V, Rockenschaub, G. Guidelines for reports on health crises and critical health events. Prehosp Disaster Med. 2010;25(4):377–382.CrossRefGoogle ScholarPubMed
Powers, R, Daily, E (eds). International Disaster Nursing. Cambridge UK: Cambridge University Press and World Association for Disaster and Emergency Medicine; 2010.CrossRefGoogle Scholar
Stephenson, C. Impacts Framework for Natural Disasters and Fire Emergencies. Melbourne, Australia: RMIT University and Bushfire CRC; 2010.Google Scholar
Debacker, M, Hubloue, I, Dhondt, E, et al. Utstein-style template for uniform data reporting of acute medical response in disasters. PLoS Curr. 2012;4.Google ScholarPubMed
Sundnes, KO. Health disaster management: guidelines for evaluation and research in the “Utstein Style.” Structural framework and preparedness. Scand J Public Health. 2014;42(14):3–195.Google Scholar
Glassey, S. Preventing “lessons lost:” is evidence-based dynamic doctrine the answer? Australian J Emerg Manag. 2015;30(3):11–14.Google Scholar
Birnbaum, ML, Daily, EK, O’Rourke, AP, Loretti, A. Disaster research/evaluation frameworks, Part 1: an overview. Prehosp Disaster Med. 2014;29(2):1–12.CrossRefGoogle Scholar
Birnbaum, ML, Daily, EK, O’Rourke, AP, Loretti, A. Research and evaluations of the health aspects of disasters, Part I: an overview. Prehosp Disaster Med. 2015;30(5): 512–522.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Daily, EK, O’Rourke, AP, Loretti, A. Research and evaluations of the health aspects of disasters, Part II: the disaster health conceptual framework revisited. Prehosp Disaster Med. 2015;30(5):523–538.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Daily, EK, O’Rourke, AP. Research and evaluations of the health aspects of disasters, Part III: framework for the temporal phases of disasters. Prehosp Disaster Med. 2015;30(6):628–632.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Daily, EK, O’Rourke, AP. Research and evaluations of the health aspects of disasters, Part IV: framework for societal structures: the social systems. Prehosp Disaster Med. 2015;30(6):633–647.CrossRefGoogle Scholar
Birnbaum, ML, Daily, EK, O’Rourke, AP. Research and evaluations of the health aspects of disasters, Part V: epidemiological disaster research. Prehosp Disaster Med. 2015;30(6):648–656.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Daily, EK, O’Rourke, AP, Kushner, J. Research and evaluations of the health aspects of disasters, Part VI: interventional research and the disaster logic model. Prehosp Disaster Med. 2016;31(2):181–194.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Daily, EK, O’Rourke, AP. Research and evaluations of the health aspects of disasters, Part VII: the relief/recovery framework. Prehosp Disaster Med. 2016;31(2):195–210.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Loretti, A, Daily, EK, O’Rourke, AP. Research and evaluations of the health aspects of disasters, Part VIII: risk, risk reduction, risk management, and capacity building. Prehosp Disaster Med. 2016;31(3):195–210.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Daily, EK, O’Rourke, AP, Loretti, A. Research and evaluations of the health aspects of disasters, Part IX: risk-reduction framework. Prehosp Disaster Med. 2016;31(3):309–325.CrossRefGoogle ScholarPubMed
Stratton, SJ. Disaster research and evaluation frameworks (editorial). Prehosp Disaster Med. 2014;29(2):1–2.CrossRefGoogle Scholar
Stratton, S. Is there a scientific basis for disaster health and medicine? (editorial). Prehosp Disaster Med. 2014;29(3):221–222.CrossRefGoogle Scholar
Cardoni, P. Validating internal models within the Solvency II Directive: Solvency 2 Experts Group; 2017. http://solvency2experts.net/blog/?page_id=105. Accessed October 1, 2018.Google Scholar
Government, Q. Disaster Management Phases Brisbane: Queensland Government; 2016. http://www.disaster.qld.gov.au/About_disaster_management/Pages/Disaster-management-phases.aspx. Accessed October 1, 2018.Google Scholar
Twigg, J. Disaster Risk Reduction: Good Practice Review 9. London UK: Humanitarian Policy Group, Overseas Development Institute (ODI); 2015.Google Scholar
Coetzee, C, Van Niekerk, D. Tracking the evolution of the disaster management cycle: a general system theory approach. JAMBA. 2012;4(1):a54.Google Scholar
Lindell, MK. Disaster studies. SociopediaISA. 2011.Google Scholar
Baird, ME. The “Phases” of Emergency Management; 2010:50. http://www.memphis.edu/ifti/pdfs/cait_phases_of_emergency_mngt.pdf. Accessed October 1, 2018.Google Scholar
Neal, DM. Reconsidering the phases of disaster. IJMED. 1997;15(2):239–264.Google Scholar
CHS Alliance. Humanitarian Accountability Report; On the Road to Istanbul, How Can the World Humanitarian Summit Make Humanitarian Response More Effective? CHS Alliance; 2015.Google Scholar
Sphere Project. The Core Humanitarian Standard and the Sphere Core Standards: Analysis and Comparison. Geneva, Switzerland: Sphere Project; 2015.Google Scholar
Creswell, JW, Miller, DL. Determining validity in qualitative inquiry. Theory Pract. 2000;39(3):124–30.CrossRefGoogle Scholar
DeVon, HA, Block, ME, Moyle-Wright, P, et al. Clinical scholarship: a psychometric toolbox for testing validity and reliability. J Nurs Scholarsh. 2007;39(2):155–164.CrossRefGoogle Scholar
Keeney, S, Hasson, F, McKenna, H. The Delphi Technique in Nursing and Health Research. Oxford, UK: Blackwell Publishing; 2011.CrossRefGoogle Scholar
Fattah, S, Rehn, M, Reierth, E, Wisborg, T. Systematic literature review of templates for reporting prehospital major incident medical management. BMJ Open. 2013;3:1–8.CrossRefGoogle ScholarPubMed
Argyrous, G. A Monitoring and evaluation framework for disaster recovery programs. Version 2 - May 2018. Sydney, Australia: Australian and New Zealand School of Government (ANZOG); 2018.Google Scholar
Davis-Stober, CP. When is a crowd wise? American Psychological Association. 2014;1(2):79–101.Google Scholar
Jorm, AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49(10):1–11.CrossRefGoogle ScholarPubMed
Surowiecki, J. The Wisdom of Crowds: Why the Many Are Smarter Than the Few. London, UK: Abacus; 2004:296.Google Scholar
Jeon, Y-H, Conway, J, Chenoweth, L, Weise, J, Thomas, THT, Williams, A. Validation of a clinical leadership qualities framework for managers in aged care: a Delphi study. J Clin Nurs. 2014;24(7–8):999–1010.CrossRefGoogle ScholarPubMed
Ivankova, N, Wingo, N. Applying mixed methods in action research: Methodological potentials and advantages. Am Behav Sci. 2018;62(7):978–997.CrossRefGoogle Scholar
Qualtrics. Qualtrics Insight Platform (intermediate surveys). 2017. https://www.monash.edu/esolutions/software/qualtrics-insight-platform. Accessed October 1, 2018.Google Scholar
Bamberger, M, Rugh, J, Mabry, L. Real-World Evaluation: Working Under Budget, Time, Data, and Political Constraints. Second edition. Thousand Oaks, CA, USA: Sage; 2012.Google Scholar
Markiewicz, A, Patrick, I. Developing Monitoring and Evaluation Frameworks. Los Angeles, CA, USA: Sage Publications, Inc.; 2016:293.Google Scholar
WHO. WHO Regional Offices. Geneva: World Health Organization (WHO); 2018. http://www.who.int/about/regions/en/. Accessed October 1, 2018.Google Scholar
Statistics Canada. Measures of central tendency: Statistics Canada; 2017. https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch11/5214867-eng.htm. Accessed October 1, 2018.Google Scholar
United Nations General Assembly. Sendai Framework for Disaster Risk Reduction 2015–2030. http://www.wcdrr.org/uploads/Sendai_Framework_for_Disaster_Risk_Reduction_2015-2030.pdf. Accessed October 1, 2018.Google Scholar
GFDRR. Resilient Recovery: An Imperative for Sustainable Development. Washington, DC, USA: International Bank for Reconstruction and Development; International Development Association for The World Bank; 2015.Google Scholar
Coppola, DP. Introduction to International Disaster Management. Third Edition. Amsterdam, The Netherlands: Butterworth-Heinemann; 2015.Google Scholar
WHO. International Health Regulations (2005): Second Edition. Geneva, Switzerland: World Health Organization; 2008. Report No.: ISBN: 9789241580410.Google Scholar
CDC. Spotlight: International Heath Regulations (2005). Atlanta, Georgia USA: Centers for Disease Control and Prevention (CDC); 2015. https://www.cdc.gov/globalhealth/ihr/index.html. Accessed October 1, 2018.Google Scholar
Agenda for Humanity. World Humanitarian Summit 2016. New York USA: United Nations Office for the Coordination of Humanitarian Affairs (UNOCHA); 2016. https://agendaforhumanity.org/summit. Accessed October 1, 2018.Google Scholar
Habitat III; New York USA: The United Nations Conference on Housing and Sustainable Urban Development; 2016. http://habitat3.org. Accessed October 1, 2018.Google Scholar
O’Connell, D, Wise, R, Doerr, V, et al. Approach, methods, and results for co-producing a systems understanding of disaster. Technical report supporting the development of the Australian Vulnerability Profile. CSIRO; 2018:218.Google Scholar
Attorney-General’s Department. “Understanding the drivers of disaster. The case for developing an Australian Vulnerability Profile.” In: Emergency Management Australia. Canberra, Australia: Attorney-General’s Department; 2017:59.Google Scholar
UN Women. What is Baseline Assessment? New York USA: United Nations Entity for Gender Equality and the Empowerment of Women; 2012. http://www.endvawnow.org/en/articles/1323-what-is-a-baseline-assessment.html?next=1324. Accessed October 1, 2018.Google Scholar
IASC. IASC Guidelines: Common Operational Datasets (CODs) in Disaster Preparedness and Response. Geneva, Switzerland: Inter-Agency Standing Committee; 2010.Google Scholar
Handbook, UE. Common operational datasets (CODs) and fundamental operational datasets (FODs); New York USA: The UN Refugee Agency (UNHCR); 2018. https://emergency.unhcr.org/entry/50307. Accessed October 1, 2018.Google Scholar
Humanitarian Response. Common Operational Datasets; New York USA: UNOCHA; 2018. https://www.humanitarianresponse.info/en/applications/tools/category/operational-datasets. Accessed October 1, 2018.Google Scholar
UNISDR. Sendai Framework Indicators; Geneva Switzerland: UNISDR; 2018. https://www.preventionweb.net/sendai-framework/sendai-framework-monitor/indicators. Accessed October 1, 2018.Google Scholar
World Health Organization; Public Health England; UNISDR. Health Emergency and Disaster Risk Management fact sheets. Health Emergency and Disaster Risk Management, Overview. Geneva, Switzerland: WHO; 2017:8.Google Scholar
Inter-Agency Standing Committee (IASC). Multi-Sector Initial Rapid Assessment Guidance: Revision July 2015; Geneva, Switzerland: Inter-Agency Standing Committee (IASC); 2015. https://interagencystandingcommittee.org/. Accessed October 1, 2018.Google Scholar
Global Cluster for Early Recovery. Early recovery: global cluster for early recovery (GCER); Geneva, Switzerland: Global Cluster for Early Recovery (GCER); 2018. http://www.earlyrecovery.global/about. Accessed October 1, 2018.Google Scholar
Australian Government - Department of Home Affairs. Community Recovery, Handbook 2. Canberra, Australia: Australian Government, Department of Home Affairs; 2018:1–148.Google Scholar
Veenema, TG, (editor). Disaster Nursing and Emergency Preparedness for Chemical, Biological, and Radiological Terrorism and Other Hazards (Second Edition). New York USA: Springer Publishing Company; 2007.Google Scholar
Bamberger, M, Independent Evaluation Group (IEG). Institutionalizing Impact Evaluation Within the Framework of a Monitoring and Evaluation System. Washington, DC USA: World Bank; 2009.Google Scholar
Puri, J, Aladysheva, A, Iversen, V, Ghorpade, Y, Bruck, T. What Methods May Be Used in Impact Evaluations of Humanitarian Assistance? 2015. http://ftp.iza.org/dp8755.pdf. Accessed October 1, 2018.CrossRefGoogle Scholar
White House. Some Reflections on Current Debates in Impact Evaluation. New Delhi, India: International Initiative for Impact Evaluation (3ie); 2009.Google Scholar
Puri, J, Aladysheva, A, Iverson, V, Ghorpade, Y, Bruck, T. Can rigorous impact evaluations improve humanitarian assistance? J Dev Effectiveness. 2017;9(4):519–542.CrossRefGoogle Scholar
3ie. Impact Evaluation Glossary. New Delhi, India: International Initiative for Impact Evaluation (3ie); 2012. Contract No.: Version No. 7.Google Scholar
Tan, YSA, von Schreeb, J. Humanitarian assistance and accountability: what are we really talking about? Prehosp Disaster Med. 2015;30(3):264–270.CrossRefGoogle Scholar
Cosgrave, J, Buchanan-Smith, M. Evaluation of Humanitarian Action Guide. London, UK: Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP); 2016.Google Scholar
Core Humanitarian Standard on Quality and Accountability. HAP International People in Aid Sphere Project; 2014.Google Scholar
Hilhorst, D. Being Good at Doing Good? Quality and Accountability of Humanitarian NGOs. Disasters. 2002;26(3):193–212.CrossRefGoogle ScholarPubMed
Blagescu, M, de Las Casas, L, Lloyd, R. Pathways to Accountability: The GAP Framework. London, UK: One World Trust; 2005.Google Scholar
Turoff, M. Past and present emergency response information systems. Commun ACM. 2002;45(4):29–32.CrossRefGoogle Scholar
Raman, M, Kuppusamy, MV, Dorasamy, M, Nair, S. Knowledge management systems and disaster management in Malaysia: an action research approach. J Inf Knowl Manag. 2014;13(01):1–15.Google Scholar
Moore, GF, Audrey, S, Barker, M, et al. Process evaluation of complex interventions: Medical Research Council guidance. Br Med J. 2015;350:1–7.CrossRefGoogle ScholarPubMed
Christoplos, I, Knox Clarke, P, Cosgrave, J, Bonino, F, Alexander, J. Strengthening the Quality of Evidence in Humanitarian Evaluations. London, UK: ALNAP ODI; 2017.Google Scholar
Christoplos, I, Dillon, N, Bonino, F. Evaluation of Protection in Humanitarian Action. London, UK: ALNAP; 2018.Google Scholar
Blanchet, K, Allen, C, Breckon, J, et al. Using Research Evidence in the Humanitarian Sector: A Practice Guide. London, UK: Evidence Aid London School of Hygiene and Tropical Medicine Nesta; 2018.Google Scholar
Taylor-Butts, A. Emergency Preparedness in Canada, 2014. Ottowa, Canada: Statistics Canada; 2016.Google Scholar
Security USDoH. 2017 National Preparedness Report. Washington, DC USA: Department of Homeland Security; 2017.Google Scholar
Government Statistician Queensland Treasury and Trade. Queensland Community Preparedness Survey November 2013– Household Preparedness for Natural Disasters. Brisbane, Australia: State of Queensland (Queensland Treasury and Trade); 2014.Google Scholar
Arup. City Resilience Index. London, UK: The Rockefeller Foundation Arup; 2015.Google Scholar
Parsons, M. The Australian Natural Disaster Resilience Index: a system for assessing the resilience of Australian communities to natural hazards: Bushfire and Natural Hazards CRC; 2018. https://www.bnhcrc.com.au/research/hazard-resilience/251. Accessed October 1, 2018.Google Scholar
Parsons, M, Morley, P. The Australian Natural Disaster Resilience Index. Australian J Emerg Manag. 2017;32(2):20–22.Google Scholar
Parsons, M, Morley, P, McGregor, J, et al. Overview of Indicators: The Australian Natural Disaster Resilience Index. Melbourne, Australia: Bushfire & Natural Hazards CRC; 2016.Google Scholar
Horney, J, Dwyer, C, Chirra, B, McCarthy, K, Shafer, J, Smith, G. Measuring successful disaster recovery. IJMED. 2018;36(1):1–22.Google Scholar
Supplementary material: PDF

Wong et al. supplementary material

Wong et al. supplementary material 1

Download Wong et al. supplementary material(PDF)
PDF 2.8 MB
Supplementary material: PDF

Wong et al. supplementary material

Wong et al. supplementary material 2

Download Wong et al. supplementary material(PDF)
PDF 952.2 KB