Disaster Metrics: A Comprehensive Framework for Disaster Evaluation Typologies

Abstract

Introduction

The frequency of disasters is increasing around the world with more people being at risk. There is a moral imperative to improve the way in which disaster evaluations are undertaken and reported with the aim of reducing preventable mortality and morbidity in future events. Disasters are complex events and undertaking disaster evaluations is a specialized area of study at an international level.

Hypothesis/Problem

While some frameworks have been developed to support consistent disaster research and evaluation, they lack validation, consistent terminology, and standards for reporting across the different phases of a disaster. There is yet to be an agreed, comprehensive framework to structure disaster evaluation typologies.

The aim of this paper is to outline an evolving comprehensive framework for disaster evaluation typologies. It is anticipated that this new framework will facilitate an agreement on identifying, structuring, and relating the various evaluations found in the disaster setting with a view to better understand the process, outcomes, and impacts of the effectiveness and efficiency of interventions.

Methods

Research was undertaken in two phases: (1) a scoping literature review (peer-reviewed and “grey literature”) was undertaken to identify current evaluation frameworks and typologies used in the disaster setting; and (2) a structure was developed that included the range of typologies identified in Phase One and suggests possible relationships in the disaster setting.

Results

No core, unifying framework to structure disaster evaluation and research was identified in the literature. The authors propose a “Comprehensive Framework for Disaster Evaluation Typologies” that identifies, structures, and suggests relationships for the various typologies detected.

Conclusion

The proposed Comprehensive Framework for Disaster Evaluation Typologies outlines the different typologies of disaster evaluations that were identified in this study and brings them together into a single framework. This unique, unifying framework has relevance at an international level and is expected to benefit the disaster, humanitarian, and development sectors. The next step is to undertake a validation process that will include international leaders with experience in evaluation, in general, and disasters specifically. This work promotes an environment for constructive dialogue on evaluations in the disaster setting to strengthen the evidence base for interventions across the disaster spectrum. It remains a work in progress.

WongDF, SpencerC, BoydL, BurkleFMJr., ArcherF. Disaster Metrics: A Comprehensive Framework for Disaster Evaluation Typologies. Prehosp Disaster Med. 2017;32(5):501–514.

Introduction

The frequency of disasters is increasing around the world with more people being at risk. ^{1 - 6} They can be a complex mix of natural hazards and human action. ⁷ There is a moral imperative to improve the approach to undertaking and reporting disaster evaluations, with the aim of reducing preventable mortality and morbidity in future events. ^{8 , 9} Improving the quality of disaster evaluations and strengthening accountability is urgently required. ^{10 - 12} While some frameworks have been developed to support consistent disaster research and evaluation, ^{13 , 14} they are fragmented and uni-focused. There is yet to be an agreed, comprehensive framework to structure disaster evaluation typologies. Such a framework could provide consistency in terminology and standards for reporting across the different phases of a disaster, with a view to providing comparability to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Sharing methodological experiences would contribute to the further development of these standards and guidelines to systematically build disaster science.

Undertaking disaster evaluations is a specialized area of study at an international level. Different approaches to evaluation over the years have led to a variety of definitions being offered when describing the term “evaluation.” ¹⁵ An earlier definition of evaluation put forward by the Joint Committee on Standards for Educational Evaluation (JCSEE) in 1994 states that “evaluation is the systematic assessment of the worth or merit of an object.” ¹⁶ The “object” in this case is the program, project, or intervention under review. ¹⁶ Other recent definitions focus more on active purposes such as accountability assessment, decision making, program improvement, judgement, and organizational learning. ¹⁷ Regardless of the definition used, evaluations are largely conducted to find areas for improvement and to generate an assessment of overall quality and value, usually for reporting or decision making purposes. ¹⁸

The aim of this paper is to outline an evolving Comprehensive Framework for Disaster Evaluation Typologies. It is anticipated that this new framework will facilitate an agreement on organizing and describing the various evaluations found in the disaster setting. While continuing to be a work a progress, it is intended that this work will add structure to the current understanding and help to underpin the diversity of disaster evaluation typologies that currently exist.

When considering the title of the framework and how best to describe this body of work, the authors considered two words: methodology and typology. The word “method” or “methodology” is defined as “a particular procedure for accomplishing or approaching something.” ¹⁹ The preferred term for the framework was “typology,” which refers to “a structure of different types,” ²⁰ and is a closer match to describing the classification of the variety of disaster evaluation styles that are currently available.

Methodology

This research was undertaken in two phases. Phase One was designed to identify current evaluation frameworks and typologies used in the disaster setting. A scoping literature review ²¹ was undertaken in two parts. Firstly, the peer-reviewed literature was searched using major electronic databases, including: PubMed/Medline (US National Library of Medicine, National Institutes of Health; Bethesda, Maryland USA); CINAHL (EBSCO Information Services; Ipswich, Massachusetts USA); EMBASE (Elsevier; Amsterdam, Netherlands); ProQuest (Ann Arbor, Michigan USA); Science Web (Thomson Reuters; New York, New York USA); Scopus (Elsevier; Amsterdam, Netherlands); and Web of Knowledge (Thomson Reuters; Philadelphia, Pennsylvania USA). These databases were searched to identify contributions to the history and development of disaster/disaster health evaluation frameworks/models/repositories. The key search words used included “disaster OR emergency,” AND “health,” AND “guidelines OR frameworks OR models OR repositories OR evaluation OR typology.” Inclusion criteria consisted of articles in English, published after 2003, and included frameworks, models, or methodologies rather than exemplars of specific evaluations. Additional references were identified through examination of bibliographies from the most recent publications (snowballing) and through scrutiny of the contents pages of highly relevant journals. ²² This scoping review was supplemented by a convenience sample of international colleagues who commented on the evolving framework to identify additional relevant typologies.

Secondly, a review of the “grey literature” also was undertaken, including similar key words, using Google and Google Scholar (Google Inc.; Mountain View, California USA) and supplemented by “ReliefWeb,” a resource maintained by the United Nations Office for the Coordination of Humanitarian Affairs (UNOCHA; New York, USA and Geneva, Switzerland) ²³ and the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP; London, United Kingdom). The ALNAP is an organization that is dedicated to improving humanitarian performance through learning and accountability. ²⁴

The scoping review provided examples of a wide range of typologies used under the general label of “evaluation.” The following Comprehensive Framework lists at least one example of each evaluation type identified as an exemplar; however, it does not intend to list all evaluations identified. One hundred twenty-two papers were used in developing this Comprehensive Framework.

In Phase Two, all co-authors contributed to developing a structure that included the range of typologies identified in Phase One, and suggested possible relationships in the disaster setting.

The resultant “Comprehensive Framework for Disaster Evaluation Typologies” not only identifies and structures different disaster evaluation typologies, but it also suggests relationships between these typologies and all phases of the disaster cycle. Various disaster evaluation typologies are mapped across the disaster timeline, demonstrating their inter-relationships. It is not the intent of this paper to describe perceived strengths or weaknesses of any particular evaluation typology. It is important to note that Baselines, Consequences, and Outcomes evaluation typologies are related but not hierarchical; that is, one is not more important that the other and are to be interpreted within the context of a specific disaster.

Results

The literature review revealed that more information can be found in the “grey literature” and humanitarian arena than in peer-reviewed literature. There were very few evaluations of health interventions during disasters reported in the literature. Most evaluation reports were descriptive, process-focused, and lacked a core conceptual framework. ²⁵ Recent research undertaken by Stratton in 2014 identified that the majority of papers submitted to Prehospital and Disaster Medicine (PDM) were surveys or descriptive in nature. ²⁶ The published reports did not demonstrate a consistent and structured approach to evaluations of interventions, and the impact of interventions on the affected population was rarely mentioned. ²⁷ Many nongovernment organizations (NGOs), such as the International Federation of Red Cross and Red Crescent Societies (IFRC; Geneva, Switzerland) and various United Nations (UN) agencies, have their own internal standards for evaluations. Attempts are being made to consolidate standards and guidelines across the sector, as evidenced by the work of ALNAP ²⁴ and the Inter-Agency Standing Committee (IASC). ²⁸ National government disaster organizations, however, were noticeably absent in this activity.

One influential guideline identified during the literature review was Health Disaster Management Guidelines for Evaluation and Research (hereafter referred to as “The Guidelines”). The Guidelines were co-authored by the Task Force on Quality Control and Disaster Management (TFQCDM), the World Association of Disaster and Emergency Medicine (WADEM; Madison, Wisconsin USA), and the Nordic Society for Disaster Medicine. ¹³ It provided a conceptual framework for undertaking research and evaluation in the disaster setting. The core of this conceptual framework was frequently referenced in peer-reviewed papers, ^{29 , 30} scholarly journals, and in higher degree research theses, ^{31 , 32} but it was rarely used as the methodological framework for undertaking disaster evaluations and research. ³³ The literature review revealed one journal article ³⁴ and two books ^{35 , 36} that utilized the “Conceptual Framework” and terminology used in The Guidelines. The three articles/books were based on the Sumatra-Andaman Earthquake and subsequent Asian Tsunami that occurred in December 2004.

Consideration of The Guidelines to underpin the “Comprehensive Framework” included a validation step. In-depth interviews of 18 experts in the fields of disaster and emergency health and medicine undertaken by the lead author in 2014 and 2015 revealed that the core framework of The Guidelines was deemed to be valuable ³³ and was being referenced. It was not, however, being used to structure research and evaluation in the disaster setting. ³²

In an attempt to test the validity of the core “Guidelines,” the authors undertook a thematic analysis of seven Australian disaster reports/inquiries dating from 2006-2014 to see if the core elements of the conceptual framework in The Guidelines were present in all reports/inquiries. The disasters occurred in four different Australian States, covered four different types of events, included four different types of reports, and were chaired by six different Chairpersons. Results from the thematic analysis were reviewed by two researchers and identified that all elements of the “Conceptual Framework” were present in each of the seven Australian disaster reports/inquiries. ³⁷

Given this support for The Guidelines from both the international experts and the thematic review of Australian reports, it was decided to use its core structure, with some modifications, to underpin this Framework for Disaster Evaluation Typologies.

Other frameworks or guidelines that were identified included work by Stephenson; ³⁸ Powers and Daily; ³⁹ Kulling et al; ¹⁴ Debacker; ⁴⁰ Fattah; ^{41 - 43} Sundnes; ⁹ and Birnbaum et al. ^{27 , 44 - 51} While Fattah identified more than 10 frameworks, she also discovered that none had been validated and they were not commonly used to structure evaluations and research in the disaster setting. ⁴²

It became evident that a core unifying framework did not exist to structure disaster evaluation and research. In an attempt to create a tool and consolidate the diverse non-validated frameworks together, the authors utilized key components from The Guidelines ¹³ and “The Impacts Framework” (comprising of event, event characteristics, object, harm, and impacts) from Stephenson ³⁸ into their framework. Disaster Evaluation Typologies was created by linking and integrating various typologies into a unifying core structure that aimed to inform and support a Comprehensive Framework for Disaster Evaluation Typologies.

The Comprehensive Framework for Disaster Evaluation Typologies is presented using the following headings:

1. 1. Figure One: Core Structure;

2. 2. Figure Two: Baselines;

3. 3. Figure Three: Consequences;

4. 4. Figure Four: Outcomes;

5. 5. Figure Five: Impact Evaluations;

6. 6. Figure Six: Accountability;

7. 7. Figure Seven: Evaluation Standards and Evidence; and

8. 8. Figure Eight: Disaster Evaluation Typologies: Comprehensive Framework.

Core Structure

The Core Structure outlines the fundamental framework of Disaster Evaluation Typologies to which all other entries will be related (Figure 1: Core Structure).

Figure 1 Core Structure.

The Core Structure consists of three important layers. The first layer is found at the bottom of the diagram and provides a preliminary and simplistic view of the disaster continuum or timeline. At its most basic level, this layer has three core elements or phases that are represented by a pre-event phase, an event phase, and the post-event phase of an emergency or disaster. While each phase can be identified individually, their timing is not necessarily sequential and the phases can overlap. Post-event phases will influence the pre-event phase of subsequent events.

The second layer is represented by an expansion of the earlier pre-event, event, and post-event phases and is based on a modified representation of The Guidelines, ¹³ as already noted. Additional detail is evident and the relationships of key disaster phases are demonstrated, including:

• ∙ Pre-event status (of the community);

• ∙ Hazard;

• ∙ Risk reduction;

• ∙ Event;

• ∙ Damage;

• ∙ Disruption, changes in functions, consequences of damage;

• ∙ Response (respond and adapt), rescue, surge, and relief;

• ∙ Recovery (recover, grow/thrive/transform), reconstruction, development, renewal, and regeneration (early recovery and persistent recovery); and

• ∙ Post-event status (of the community). ^{9 , 13 , 14 , 27 , 38 , 39}

The third layer introduces the concept of “Strengthening Resilience” as an overarching theme. It enhances and enriches the Core Structure of Disaster Evaluation Typologies and is an emerging, international imperative embraced within the Sendai Framework for Disaster Risk Reduction 2015-2030 (SFDRR). ⁵² Key elements of “Strengthening Resilience” include:

• ∙ System structure, governance, coordination, and leadership; ^{53 - 55}

• ∙ “Culture of Safety” with consideration to: risk reduction, prevention, and mitigation; ^{52 , 56 , 57}

• ∙ Reduce exposures and vulnerabilities; build anticipative, absorbing, and adaptive capacities; and promote community development; ⁵⁸ and

• ∙ Elements are identified in five contemporary and influential international frameworks:

1. 1. Sendai Framework for Disaster Risk Reduction 2015-2030 (SFDRR); ⁵²

2. 2. Sustainable Development Goals (SDGs); ⁵⁹

3. 3. Climate Change Conference (COP21); ⁶⁰

4. 4. Global Facility for Disaster Reduction and Recovery (GFDRR) Recovery Framework, also known as Post-Disaster Needs Assessment (PDNA) Recovery Framework; ⁶¹ and

5. 5. Rockefeller Foundation 100 Resilient Cities (RC) Framework. ⁶²

The Core Structure for the Comprehensive Framework is illustrated in Figure 1.

Baselines

Baselines are a series of evaluations or assessments that occur during the pre-event phase of disasters and provide information about the current state of the community (Figure 2: Baselines).

Figure 2 Baselines.

Baseline evaluations include any information or data that have been collected prior to an event or disaster occurring. It covers both the pre-event status of a community and the actual hazard itself. Obtaining Baseline information in the pre-event phase is critical in understanding the state of the community, how it has been affected by a disaster, ^{2 , 63} and the subsequent damage that has occurred. This information assists in identifying community strengths, weaknesses, and vulnerabilities to disasters. Additionally, this information will assist in developing appropriate disaster management and disaster risk-reduction strategies.

Innovations in science and technology have made it easier in recent years to collect information that helps reduce disaster risk, and therefore, plan for the future. ⁶⁴ The SFDRR, ⁵² ratified in Sendai, Japan in March 2015 by 187 UN Member States, acknowledges that there is a growing demand for science and technology to play a more prominent and effective role in providing evidence for policy and decision making. Knowledge is essential to the process. A strengthened evidence base to support the implementation of disaster risk-reduction strategies also is required. ^{65 , 66} Furthermore, Priority 4, paragraph 34(b) of the SFDRR supports the “further development and dissemination of instruments, such as standards, codes, operational guides, and other guidance instruments to support coordinated action in disaster preparedness and response to facilitate information sharing on lessons learned and best practices for policy practice and post-disaster reconstruction programmes.” ⁵²

Examples of Baseline evaluations include, but are not limited to:

• ∙ Baseline evaluations ^{67 , 68} and evaluability; ⁶⁹

• ∙ Demographics and infrastructure (such as Geographic Information Systems/GIS); ^{70 , 71}

• ∙ Epidemiology and emerging threats; ^{72 , 73}

• ∙ Define minimum standards/criteria: baselines, targets, and indicators; ^{74 , 75}

• ∙ Preparedness, resilience capacity with a special focus on the SFDRR; ⁵²

• ∙ Risk management, surveillance, and early warning; ⁷⁶ and

• ∙ Health Impact Assessment (HIA) Predictive. ⁷⁷

Baseline information and evaluations are illustrated in Figure 2.

Consequences

Consequences are a series of evaluations and/or assessments that occur after the event or disaster has occurred and include assessment of damage and changes in function (Figure 3: Consequences).

Figure 3 Consequences.

Consequence evaluations include any information or assessments that have been collected after an event or disaster has occurred. It covers both the event and post-event phase of the disaster timeline. Systematic data collection and assessment is required in order to inform disaster needs analysis after an event. It is used in monitoring the effectiveness of response and recovery interventions and to aid decision making.

Examples of Consequence evaluations include, but are not limited to:

• ∙ Rapid needs assessments (damage), usually occur on Day 1 after the event; ^{78 , 79}

• ∙ Detailed needs assessments (functional), usually occur on Days 2-3 and may include PDNA; ⁸⁰

• ∙ Continual assessments that include monitoring and surveillance, usually occur on multiple occasions after the event; ⁷⁸ and

• ∙ Independent real-time evaluations ⁸¹ and collaborative joint evaluations are contemporary evaluation types. ⁸²

The information received from these evaluations will ideally be compared with previous Baseline studies and incorporated into helping to plan response and recovery for the current event, provide feedback into planning and preparing for subsequent events, and assist in disaster risk reduction. ⁸³ Currently, damage and loss trends are difficult to monitor over time, partly due to inconsistent methodologies and the fact that very few countries keep national disaster databases. Even then, only one in five countries will have consistently recorded economic losses using validated tools and data collection methods. ⁸⁴ The PDNAs aim to provide a common approach to post-crisis needs assessments and recovery planning. ⁸⁵

The Centre for Research on the Epidemiology of Disasters (CRED; Brussels, Belgium) promotes research, training, and information dissemination on disasters. ⁸⁶ In the Australian context, the Australian Business Roundtable for Disaster Resilience and Community Safety provides a first-time overview of disaster data with the aim of making Australian communities safer and more resilient to natural disasters. ⁸⁷

In an attempt to reduce disaster risk and strengthen resilience, a feedback loop is present in Figure 8 from Consequences to Baselines. Consequence evaluations are illustrated in Figure 3.

Outcomes

Outcomes are a series of evaluations and/or assessments that occur towards the end of the post-event phase of a disaster (Figure 4: Outcomes).

Figure 4 Outcomes.

Outcome evaluations reflect information or data that have been collected after an event or disaster has occurred. These evaluations include summative reviews of processes used in managing the event and outcomes related to the post-event status of the community. This information will ideally be incorporated into planning and preparing for the next event or disaster.

Examples of Outcome evaluations include, but are not limited to:

• ∙ Operational/Strategic/Institutional Reviews that are Internal Reviews and include: Debriefs, After Action Reviews, ^{88 , 89} and Lessons Learned; ⁹⁰

• ∙ Quality Reviews that are External Reviews and include: Audit, Key Performance Indicators, and Quality Improvement; ⁹¹

• ∙ Government Inquiries; ^{92 , 93}

• ∙ Multi-disciplinary event reports/reviews (for example, Kulling et al ¹⁴ and Fattah); ^{41 - 43}

• ∙ Process and outcome evaluations that include: ^{94 , 95} logic maps/models, theories of change, and causal links/attribution are used to guide process evaluations. Each intervention and evaluation requires its own logic map that describes the sequence of actions to be undertaken and that communicates what the program is and will do; ⁹⁶ and

• ∙ HIA Evaluations. ⁷⁷

In an attempt to reduce disaster risk and strengthen resilience, a feedback loop is present in Figure 8 from Outcomes to Baselines. Outcome evaluations are illustrated in Figure 4.

Impact Evaluations

Impact Evaluations of programs, projects, and interventions are evaluations that include a measure of causality or attribution ⁹⁷ and can occur during any phase of the disaster timeline (Figure 5: Impact Evaluations).

Figure 5 Impact Evaluations.

In the disaster setting, Impact Evaluations have gained popularity for identifying causal links between specific interventions and outcomes. This is a result of the international community demanding accountability and improved evidence-based interventions. ^{10 , 12} Although there remains ongoing debate about the exact definition of Impact Evaluations, ⁶ they are particularly well-suited to answer important questions, such as: whether interventions do or do not work; whether interventions make a positive or negative impact; whether there are intended or unintended consequences; and how cost effective they are. ^{98 , 99} It is believed they will greatly improve the effectiveness of interventions delivered in the disaster setting by identifying what works for whom, and why. ¹⁰⁰

Examples of Impact Evaluations include, but are not limited to:

• ∙ Buttenheim Impact Evaluation in the Post-Disaster Setting: A Case Study of the 2005 Pakistan Earthquake; ¹⁰¹

• ∙ Rogers Introduction to Impact Evaluation; ¹⁰²

• ∙ Puri et al What Methods May Be Used in Impact Evaluations of Humanitarian Assistance? ⁹⁹ and

• ∙ Chambers et al Designing Impact Evaluations: Different Perspectives. ¹⁰³

Impact Evaluations are illustrated in Figure 5.

Accountability

Accountability to donors, stakeholders, and beneficiaries is a cross-cutting theme across all phases of the disaster timeline and is applicable to every evaluation undertaken in the disaster setting ¹⁰⁴ (Figure 6: Accountability).

Figure 6 Accountability.

Over the last 20 years, there has been a call for greater Accountability in disaster and humanitarian settings. More recently, at an international level, there has been mounting pressure to strengthen quality, accountability, and learning practices, while also ensuring transparency. ^{105 - 108} The lack of an accepted definition for “accountability” in the humanitarian context remains a challenge. The term “accountability” seems to represent a whole range of concepts and principles. ¹⁰⁵ The ALNAP is an example of an international organization dedicated to improving humanitarian performance through accountability and increased learning. ¹⁰⁹

For the purpose of this Typology, the term “accountability” will be defined as the means in which power is used responsibly. This includes consideration of the views of all interested parties (including donors, stakeholders, and beneficiaries). ⁷⁵

Examples of Accountability evaluations include, but are not limited to:

• ∙ 2013 Humanitarian Accountability Report; ¹¹⁰ and

• ∙ Catholic Relief Services Monitoring, Evaluability, Accountability, and Learning in Emergencies: A Resource Pack for Simple and Strong MEAL. ¹¹¹

Accountability is illustrated in Figure 6.

Evaluation Standards and Evidence

Evaluation Standards and Guidelines, Evidence-Based Reviews and Registries, and Knowledge Management are important cross-cutting themes that are relevant throughout the entire disaster timeline (Figure 7: Evaluation Standards and Evidence).

Figure 7 Evaluation Standards and Evidence.

Evaluation Standards and Guidelines include generic Evaluation Standards, such as:

• ∙ General evaluation theorists, including but not limited to Patton, ¹¹² Scriven, ¹¹³ and Stufflebeam; ¹⁵

• ∙ United Nations Evaluation Group (UNEG) Standards for Evaluation in the UN System; ¹¹⁴

• ∙ American Evaluation Association (AEA) Program Evaluation Standards, ¹¹⁵ which are based on earlier work by Yarbrough et al; ¹⁷

• ∙ Organization for Economic Co-Operation and Development (OECD), Development Assistance Committee (DAC) Principles for Evaluation of Development Assistance, originally published in 1991 ¹¹⁶ and updated in 2008; ¹¹⁷ and

• ∙ Hawe et al Evaluating Health Promotion: A Health Worker’s Guide. ¹¹⁸

Additionally, there are disaster-specific Evaluation Standards and Guidelines published by many NGOs that include, but are not limited to:

• ∙ IFRC Project/Programme Monitoring and Evaluation (M & E) Guide; ¹¹⁹

• ∙ Save the Children Evaluation Handbook; ¹²⁰

• ∙ ALNAP Evaluation of Humanitarian Action Guide; ¹²¹

• ∙ International Initiative for Impact Evaluation (3ie) Principles for Impact Evaluation; ¹²²

• ∙ World Bank Group Impact Evaluations: Relevance and Effectiveness; ¹²³

• ∙ Centers for Disease Control and Prevention (CDC) Program Performance and Evaluation Office – Program Evaluation; ¹²⁴ and

• ∙ Australian Agency for International Development (AusAID) Monitoring, Evaluation, and Learning Framework. ¹²⁵

Evaluation Standards and Guidelines also include guidelines for responsible and ethical conduct in undertaking evaluations that include, but are not limited to:

• ∙ United Nations Evaluation Group (UNEG) Ethical Guidelines for Evaluation; ¹²⁶

• ∙ Australian Council for International Development (ACFID) Guidelines for Ethical Research and Evaluation in Development; ¹²⁷ and

• ∙ Australasian Evaluation Society (AES) Guidelines for the Ethical Conduct of Evaluations. ¹²⁸

Evidence-Based Reviews and Registries include meta-evaluations, systematic reviews, other types of literature review typologies, and registries of evaluation reports.

The level and quality of evidence in this setting has recently been reviewed by Clarke and Darcy in Insufficient Evidence? The Quality and Use of Evidence in Humanitarian Action – ALNAP Study. ¹⁰ Despite improvements over the last 20 years, they identified that there remains room for further development in the quality and use of evidence in the humanitarian setting. The authors also suggest that “evidence matters: the use of good quality evidence improves the effectiveness and accountability of humanitarian action, and is in accordance with humanitarian ethics and principles.” ¹⁰

Systematic Reviews

Systematic Reviews are structured, comprehensive literature reviews that utilize a rigorous and published search strategy, with the aim of minimizing selection bias. ^{129 - 131}

Examples of Systematic Reviews in this discipline include, but are not limited to:

• ∙ Blanchet et al An Evidence Review of Research on Health Interventions in Humanitarian Crises; ¹³²

• ∙ Clarke et al What Evidence is Available and What is Required, in Humanitarian Assistance? 3ie Scoping Paper 1; ¹²

• ∙ Gallardo et al Core Competencies in Disaster Management and Humanitarian Assistance: A Systematic Review; ¹³³ and

• ∙ Moslehi et al Characteristics of an Effective International Humanitarian Assistance: A Systematic Review. ¹³⁴

Other literature review typologies include: scoping reviews, ^{21 , 135} gap analyses, ¹² and priority settings. ¹³⁶

Meta-Evaluations

Meta-Evaluations are systematic and formal evaluations of evaluations ¹³⁷ and are a high-level of evidence; however, they are uncommon in the disaster setting.

Examples of Meta-Evaluations include, but are not limited to:

• ∙ ALNAP Review of Humanitarian Action in 2003: Improving Monitoring to Enhance Accountability and Learning, Chapter 4 Meta-Evaluation; ¹³⁸ and

• ∙ Groverman and Hartmans Meta-Evaluation and Synthesis of the 2010 Pakistan Floods Response by SHO Participants: A Synthesis of Conclusions, Report Phase 2. ¹³⁹

Registries

For the purpose of this paper, Registries (sometimes called repositories) are defined as publicly available, free-access collations of evaluation studies that have been undertaken in the disaster setting. Registries aim to help build capacity and strengthen disaster risk reduction and resilience. ¹⁴⁰ A separate review of such Registries undertaken by the lead author suggests that these are not well-known in the disaster sector but contain a large number of evaluation reports that might be of use to aid decision making and improve practice. ¹²

Examples of disaster evaluation Registries include, but are not limited to:

• ∙ Independent Evaluation Group (IEG) hosted by the World Bank; ¹⁴¹

• ∙ Humanitarian Evaluation and Learning Portal (HELP) hosted by ALNAP; ¹⁴²

• ∙ IFRC; ¹⁴³

• ∙ Evaluation and Research Database (ERD) hosted by United Nations Children’s Emergency Fund (UNICEF); ¹⁴⁴ and

• ∙ Impact Evaluations hosted by 3ie. ¹⁴⁵

Knowledge management includes cross-sectoral research, collaboration, and dissemination of information to improve the evidence base of disaster science and to improve practice. Sharing knowledge enables informed decision making regarding disaster risk reduction and management. ¹⁴⁶

Evaluation Standards and Guidelines, Evidence-Based Reviews and Registries, and Knowledge Management are relevant in advancing the science of disaster evaluations by providing scientific rigor, common terminology, and the ability to replicate various methodologies.

Evaluation Standards and Guidelines are illustrated in Figure 7.

Disaster Evaluation Typologies: Comprehensive Framework

Disaster Evaluation Typologies: Comprehensive Framework identifies the different typologies of disaster evaluations and demonstrates key relationships in a single diagram. It suggests the interdependencies and relationships between various evaluation typologies along the disaster timeline and within the disaster setting. This consolidates the previous Figures 1–7 (Figure 8: Disaster Evaluation Typologies: Comprehensive Framework).

Figure 8 Disaster Evaluation Typologies: Comprehensive Framework.

A strong evaluation framework for disaster settings is extremely important given the increasing frequency and scale of disasters. It would need to utilize agreed definitions and be able to measure the impact and effectiveness of interventions. It is anticipated that Disaster Evaluation Typologies: Comprehensive Framework will create a useful and usable framework and promote an environment for constructive dialogue on an international level. ¹⁴⁷

Figure 8 also includes feedback loops from Consequences and Outcomes to improve Baselines, reduce disaster risk, and strengthen resilience. The framework is not limited to any one phase of the disaster timeline and can be used for responding to disasters, humanitarian crises, or in the development sector.

Disaster Evaluation Typologies: Comprehensive Framework are illustrated in Figure 8.

Discussion

Natural disasters of themselves are complex events ^{148 - 150} and undertaking structured evaluations in this setting is also a complex activity. The authors intended to create a classification of disaster evaluation typologies that would provide structure, encourage common terminology, and advance the evidence base of disaster science. The role of the framework is to support the ability to measure and evaluate the effectiveness of interventions provided in the disaster setting and thereby reduce the increasing human and economic costs associated with disasters.

The Comprehensive Framework outlined in this paper is the first framework of this type and thus makes a unique contribution to current knowledge. No previous reference has been located that identifies such a wide range of evaluation typologies used in the disaster setting and further provides conceptual relationships in a single comprehensive framework. The aim of the authors is consistent with that of James J. James in his recent Editorial where he concludes “A common Disaster Medicine and Global Health taxonomy will form the foundation of a safer, more resilient world, through more effective preparedness and response; but we must first come together for the public good.” ¹⁵¹

The Comprehensive Framework will undergo further research to validate the typologies and their relationships through structured interviews with targeted international experts in both general evaluation and disaster evaluations. Subsequently, additional work is needed to identify and develop toolkits of standards and guidelines for each of the evaluation typologies identified, as well as any methods that come to light as a result of the validation process. A recent example includes the frameworks for Disaster Research and Evaluation published by Birnbaum, Daily, O’Rourke, Loretti, and Kushner. ^{27 , 44 - 51}

Limitations

Limitations of the evolving Comprehensive Framework include difficulty in searching the “grey literature” and identifying all evaluation typologies used in this sector. There may be evaluation typologies that have not been identified. Secondly, the suggested relationships are framed through the eyes of the authors and there may be alternative perspectives to frame these relationships. Thirdly, there is a lack of a unifying theory for disaster evaluation. Finally, the authors have not considered specific research methods that might be utilized in the disaster setting. These can be found in any standard textbook on research methods in epidemiology, social sciences, or kindred disciplines. It is hoped that the validation process will address these limitations.

Conclusion

Disaster Evaluation Typologies: Comprehensive Framework identifies the different typologies of disaster evaluations that were identified in this study and brings them together in a single framework. It suggests interdependencies and relationships that exist between various evaluation typologies within the disaster setting. This unique unifying framework has relevance at an international level and is expected to benefit the disaster, humanitarian, and development sectors. This work promotes an environment for constructive dialogue on evaluation in the disaster setting and adds to the evidence base of disaster evaluation and research.