The Challenge of Mass Casualty Incident Response Simulation Exercise Design and Creation: A Modified Delphi Study

Abstract

Background:

A Mass Casualty Incident response (MCI) full scale exercise (FSEx) assures MCI first responder (FR) competencies. Simulation and serious gaming platforms (Simulation) have been considered to achieve and maintain FR competencies. The translational science (TS) T0 question was asked: how can FRs achieve similar MCI competencies as a FSEx through the use of MCI simulation exercises?

Methods:

T1 stage (Scoping Review): PRISMA-ScR was conducted to develop statements for the T2 stage modified Delphi (mD) study. 1320 reference titles and abstracts were reviewed with 215 full articles progressing for full review leading to 97 undergoing data extraction.

T2 stage (mD study): Selected experts were presented with 27 statements derived from T1 data with instruction to rank each statement on a 7-point linear numeric scale, where 1 = disagree and 7 = agree. Consensus amongst experts was defined as a standard deviation ≤ 1.0.

Results:

After 3 mD rounds, 19 statements attained consensus and 8 did not attain consensus.

Conclusions:

MCI simulation exercises can be developed to achieve similar competencies as FSEx by incorporating the 19 statements that attained consensus through the TS stages of a scoping review (T1) and mD study (T2), and continuing to T3 implementation, and then T4 evaluation stages.

Introduction

The complexity of an individual first responder participating in a sudden onset disaster mass casualty incident response (MCI), ¹ involves their patient care or duty skills, and the layering of their participation in their agency’s Incident Management System (IMS). ² The first responder’s usual, daily standard operation command and control, changes once the MCI is declared. The first responder must have the skills required for MCI patient care or duty station, but they must also operate in the rarely utilized agency MCI-plan IMS. A live full-scale exercise (FSEx) examines these relationships; the communication between the first responder and the patient, and the responder’s IMS to prepare the first responder and system for an actual MCI response (Figure 1).

Figure 1. Relationships of the first responder in an MCI.

IC = Incident Command

IMS = Incident Management System

Agency = The pre-hospital or hospital, government or non-government, agency, organization, group, hospital, or health care delivery system of the individual first responder

Jurisdiction = The lead command and control health authority of the MCI response

Direct 2-way interaction between the first responder with the patient and simultaneously with their agency incident command

Jurisdiction provides command and control to the First Responder

First Responder may provide information to the Jurisdiction Incident Command

An FSEx presents obstacles and challenges for designers, developers, planners, and exercise conductors to fulfill the mission of ensuring first responder competencies and MCI plan effectiveness (Supplemental digital content Figure 1s). ³ Staff, stuff, and structures (SSS), ⁴ may be stretched thin to cover the daily need for medical services, let alone devote staff for training to gather appropriate observers, evaluators, volunteers to recruit/ moulage patient actors, and the technical staff to conduct an FSEx. Furthermore, government permits may be needed to alter road, highway, plane, and train transportation routes and/ or ensure that ecological and environmental concerns are considered in the FSEx – all time-consuming activities.

Travel to a conference that features an FSEx adds a budgetary constraint and creates a void in the staffing schedule, and there may be no capable replacement for the first responder. Education and training budgets must include resources to devote to training for the rare MCI, when emergency medical services (EMS) and health care facility administrators must also assure appropriate education and training for staff to achieve competencies for daily operations. Even allocating assets for the planning and execution of a valid FSEx may be difficult, regardless of regulatory expectations (e.g., nuclear, airplane, train, manufacturing, etc.). ^5,6 Financial and time constraints may limit the discussion and agreement process by health authorities, regulators, first response agency, and health care facility administrators. (Supplemental Digital Content Figure 2)

Figure 2. PRISMA flow diagram.⁹

Included References.^13–110

Specifically discussing competencies.^{16–19,24,29–39,41–44,46–55,71,84,90,91,95,96,99,100,103–110}

This dilemma was made vivid with the Coronavirus disease-2019 (COVID-19) response worldwide. Staff and facilities struggled to acquire and maintain SSS to mount a safe COVID-19 response. There was no way to gather responsible actors, create or revise an MCI plan, or develop and execute an FSEx. Face-to-face classroom learning and travel to conferences became a public health casualty to prevent the spread of COVID-19.

Simulation and serious gaming (Simulation) for skills-based medical education has become an effective adjunct to, or has supplanted conventional methodologies, and was considered to replace face-to-face MCI response education, and training to achieve similar or equal first responder competencies. The challenge of the realism that an FSEx creates for first responders’ critical decisions has been postulated to be similar using other simulation methodologies but was not ready to be ‘taken off the shelf’ to incorporate into an MCI simulation exercise of any scale.

This study is designed to address the scope of the NO-FEAR Project (Network Of practitioners For Emergency medicAl systems and cRitical care) under work package number 5⁷: education and training of personnel and volunteers, regarding technical and non-technical skills, teamwork, critical thinking, clinical care, incident management, and psychological support. NO-FEAR asks for new simulation tools for education and training to achieve MCI response competencies (e.g., high-fidelity and live simulation, 2-dimensional enhanced and immersive simulation software, tools to provide quantitative, and qualitative evaluation of responders’ performance during exercises). ⁷ The present research is aimed at describing the design needs for future MCI response training simulations through a combination of literature review and expert assessment that achieves equal or similar competencies as a FSEx. A competency component of the MCI response training simulation design is to approximate the realism of the FSEx to better prepare the first responder for an actual dynamic MCI response.

Methods

This translational science (TS) study begins with the TS question (T0) ⁸ : How can first responders achieve similar MCI competencies as an FSEx using MCI simulation exercises?

The objective of this study produced the first stage (T1) scoping review and second stage (T2) modified Delphi study (mD) consensus statements that can be offered to formulate MCI simulation exercise guidelines in the future TS third (T3) MCI simulation exercise creation stage. The fourth TS (T4) stage that follows will study these MCI simulation exercises to determine if similar or equal FSEx competencies were obtained.

T1: Scoping review

A systematic review following the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR), ⁹ was conducted from August 2021 through October 2021. Scoping reviews synthesize knowledge following a defined scientific process to identify sources that can be interrogated using a defined data extraction tool to determine concepts, theories, and knowledge gaps. ¹⁰ The study of MCI education, training, and exercises that lead to competencies is multi-disciplinary, creating a body of knowledge that is heterogeneous, and thus apropos for PRISMA-ScR methodology. ¹¹

Literature search criteria

A T0 research question was developed using the Patient, Intervention, Control/ Comparison, Outcome (PICO) standard to frame the search strategy. ¹²

1. 1) Population: MCI, pre-hospital and hospital providers, simulation training exercise, or drill.

2. 2) Intervention: Not MCI simulation training exercise or drill, individual duty station competencies.

3. 3) Comparison: Individual intra-agency and inter-agency IMS competencies.

4. 4) Outcomes: Intra-agency and inter-agency IMS competencies.

Literature search methods

Inclusion criteria

The search strategy included only terms relating to or describing the intervention (Table 1). The review included English-language papers published from January 1, 1990, to July 1, 2021, in these databases:

1. 1) PubMed (National Center for Biotechnology Information, National Institutes of Health, Bethesda, Maryland, USA)

2. 2) SCOPUS (The largest and most comprehensive abstract and citation database of peer-reviewed literature from Elsevier, Netherlands)

3. 3) CINAHL (Cumulative Index to Nursing and Allied Health Literature, EBSCO, Elton B Stephens Company, Ipswich, Massachusetts, USA)

4. • DTIC (Defense Technical Information Center, United States Department of Defense) database for reports and other government publications

5. • ECRI (Emergency Care Research Institute, Plymouth Meeting, Pennsylvania, USA) Trust for published guidelines

6. • PsycInfo (American Psychological Association, Washington District of Columbia, USA) validating surveys/questionnaires on disaster training databases.

Table 1. T1 scoping review search terms

Finally, an ancestry search was also performed to identify additional references from the bibliography of references when appropriate.

Exclusion criteria

References from the databases that did not meet the inclusion criteria, specifically did not study, or report an MCI/ MCI exercise, were excluded.

PRISMA ScR Figure 2

2 review authors independently screened 1320 reference titles and abstracts to determine if inclusion criteria were met; any disagreement was resolved by discussion. Then each of the remaining 215 full articles was read by both authors to determine if inclusion criteria were met, and again any disagreement was resolved by discussion.

Of the remaining 97 included articles, ^13–110 47 discussed competencies, ^{16–19,24,29–39,41–44,46–55,71,84,90,91,95,96,99,100,103–110} and were split between 2 authors to extract data into an Excel database (Microsoft Corp., Redmond, Washington, USA) that was developed using themes and subthemes from MCI exercise publications to derive statements for the mD. ^111–115 (Supplemental Digital Content link to Excel database)

T2: Modified Delphi study

The mD method permits experts from various locations to independently review statements to attain consensus when no consensus existed previously. ⁹ The first stage of the mD began after the final database was analyzed by lead authors who created initial draft statements based on the most relevant datapoints. Then an internal focus group of authors discussed and edited these statements to meet the format of Delphi statements for the Stat59 statistical analysis platform (Stat59 Services Limited, Alberta, Canada). ¹¹⁶ An external focus group comprised of content experts in the field of MCI simulation exercises and NO-FEAR partners was established to further discuss and edit the draft statements to be clear and concise. After a video conference moderated by the lead authors, these experts discussed the draft statements. External focus group participants performed asynchronous editing via a shared online document producing the final 27 statements for the second stage of the mD.

A list of content experts, derived from the authors of included references, academicians, and researchers studying MCI simulation exercises was created to establish the mD expert panel. Introductory emails were sent explaining the project objectives and the mD to these mD experts.

mD experts that agreed were sent an email from the Stat59 (Stat59 Services Limited, Alberta, Canada) mD organizational program on the day that the mD began with a link to the Stat59 (Stat59 Services Limited, Alberta, Canada) website consent page. Each mD expert registered an account, validated it, and were sent a new email to log into their secure webpage to begin the first mD expert consensus round. 3 days later, mD experts that had not logged into the system were asked to verify their access and log in and asked to notify the author if they had not received the introductory email, with instructions on how to ensure future emails were received.

Once the mD experts logged in, they were provided with a formal explanation of the mD methodology and informed consent was obtained. For informed consent (Supplemental Digital Content link to Stat59 Consent Page), participants were notified that they were anonymous volunteers who could withdraw at any time, that participation or withdrawal would not impact their employment, and that their data was secure (Supplemental Digital Content link to Stat59 Security Page).

The next page was the list of 27 statements that were finalized in the T2 external focus group with instruction to rank each statement on a 7-point linear numeric scale, where 1 = disagree and 7 = agree. With this initial set of statements, the mD expert was asked to answer 4 demographic questions. Consensus amongst mD experts was defined as a standard deviation ≤ 1.0.

Statements that attained consensus after this first mD expert round were included in the final report. Each mD expert received an email from the Stat59 (Stat59 Services Limited, Alberta, Canada) program after the first mD expert round and a reminder email from the author shortly afterwards to log back into their Stat59 page that showed the mean response of all the mD experts for each statement that did not attain consensus, their own response for that specific remaining statement, and were asked to reconsider their 7-point linear numeric scale for these remaining statements.

This process was repeated after the second mD expert round with statements that attained consensus included in the final report. The statements that did not attain consensus were advanced to the third and final round with the mD experts asked to reconsider these statements. This third mD expert round produced the final statements that attained consensus to add to the first and second round consensus statements in the final report. Remaining statements after this third round were the final statements that did not attain consensus.

The McLeod Health Institutional Review Board Office (Florence, South Carolina USA) has determined that this study does meet the exemption criteria found at 45 CFR 46.104(d)(2). ¹¹⁷

Results

As summarized in Figure 3, 35 mD experts confirmed their participation and established a unique account on the Stat59 website (Table 2). 31 completed the first mD expert round that was open from January 17, 2022, until January 30, 2022. 5 statements attained statistical significance with a standard deviation ≤ 1.0 after this first mD expert round, and achieved consensus (Table 3, first round, first section in bold). The 22 statements that did not attain statistical significance, with standard deviation > 1.0, were advanced to the second mD expert round.

Figure 3. Modified Delphi statement creation.

Table 2. Modified Delphi expert panel demographics

Table 3. Statements that attained consensus

Bold T2 mDE Rounds 1 and 3

Not Bold T2 mDE Round 2

29 mD experts completed the second mD expert round that was open from January 31, 2022, to February 19, 2022. 12 of the 22 statements that advanced to the second mD expert round achieved consensus (Table 3, second round middle section). The remaining 10 statements were unable to attain consensus and advanced to the third mD expert round.

29 mD experts completed the third and final mD expert round that was open from February 23, 2022, to March 9, 2022. 2 of the remaining 10 statements achieved consensus, so a total of 19 statements achieving consensus. (Table 3 third round, last section in bold). The remaining 8 statements were unable to attain consensus after 3 T2 mD expert rounds and were not recommended for T3 consideration (Table 4).

Table 4. Statements that did not attain consensus

Limitations

The PRISMA-ScR produced a qualitative analysis of published studies and reports. Though the search followed this process, there may have been references that were not discovered.

The Delphi method seeks to arrive at group consensus by the aggregate of a panel of experts who rate a statement on a linear numeric scale. Internal validity is largely unknown; therefore, stability of response is more accurate to determine consensus or lack of consensus.

The objective of the distribution of mD experts was to represent MCI simulation exercise designers, developers, and those that would execute an exercise. The distribution of mD experts favor resource rich countries but all experts are involved in MCI exercises in a way.

An essential component of the Central Limit Theorem is that the average of sample means will be the population mean, or if 1 finds the average of all of the standard deviations in the sample, then 1 will find the actual standard deviation for the population. ¹¹⁸ This will hold true regardless of whether the source population is normal or skewed, provided the sample size is sufficiently large (usually ≥ 30, the number of mD experts in this study per round averaged 29.67). ¹¹⁹ The application of the Central Limit Theorem to this study infers that the 19 statements that attained consensus can be recommended to assist the T3 development of guidelines for MCI simulation exercises.

Discussion

Providing MCI training can be challenging for several reasons: (1) provider schedules are often erratic and involve long hours; (2) there is a temporal dissociation between disaster response training and the application of the skills, leading to cognitive skill decay at the time the skill needs to be performed; and (3) traditional learning methods, such as didactic presentations, tabletop simulations, and FSExs require the physical presence of learners and educators at a certain place and time (synchronous learning). ³⁶ In response, mD experts agreed, educators should design periodic simulation training to maintain competencies of rarely used skills that will deteriorate over time.

Briggs et al. showed that heterogeneous organizations with different command structures and missions participate in the response to a disaster, and therefore a clear objective of an MCI plan is to define the IMS of a region. ¹²⁰ An example of a 2017 FSEx objective of Regional Operability in Ohio (United States) as reported by McElroy is that ‘Participants shall identify the management structure to support effective operational coordination between all agencies and entities.’ ⁸³ The mD experts agreed that the simultaneous integration of patient care or duty skills (e.g., triage, utilization of resources, communication) and IMS skills should be incorporated into the exercise design to achieve the competency for the individual to recognize and assume their position in their agency MCI plan through situation awareness and critical decision-making.

As the layers of FSEx objectives are incorporated into the FSEx design, mD experts agreed that patient care or duty competencies can be evaluated using distinct separate modalities or as part of an exercise designed to include these skills with IMS skills. mD experts agreed that multiple exercise modalities should be considered to minimize time, cost, and impact to non-participants affected by the exercise. Conceptually a simulation exercise can be created to achieve one of the many IMS competency objectives achieved by an FSEx as required by health authorities, regulators, IMS partners, and stakeholders:

1. 1) Utilize effective means of inter- and intra-agency communication through redundant modalities;

2. 2) Achieve the appropriate continuum of patient triage and treatment from the initial evaluation to definitive care;

3. 3) Deliver the right patient to the right alternate care facility or definitive health care facility capable of attending to the injuries of that patient;

4. 4) Maintain accurate patient tracking leading to expedient hospital registration;

5. 5) Manage resources utilizing the supply chain in a resource scarce environment; and

6. 6) Manage information and media through intelligence acquisition, vetting, and transmission.

MCI simulation education and training is multimodal: lectures, readings, and individual hands-on skill sessions to meet the requirements of the individual’s MCI patient care duties. To do that, education must be active, interactive, and experiential, as well as participatory. ¹²¹ The additional layers in MCI education and training acknowledge that the individual is part of the IMS. The evolution of MCI simulation exercises will appreciate these complex simultaneous actions of the individual as they manage patient care and their role in the IMS, to achieve the same competencies that the FSEx will produce. To approximate the success of the FSEx, MCI exercise simulation must depend on high physical fidelity to develop the individual’s manual abilities, as well as high conceptual fidelity to develop clinical reasoning and problem-solving skills.

Lastly, high emotional or experiential fidelity, ¹²² where the learner is emotionally invested in the simulation to a degree that memories of the experience are believable, will develop the individual’s retention of the material. ⁴⁴ There must be emotional learning in which positive emotions under stress facilitate greater retention of data. This success is not completely based on the realism of the simulation, but on the commitment of the participants in their roles; that there is an adequate connection between those involved; and that the student manages to actively link the social, psychological and clinical experiences lived. ¹²³ The mD experts agreed that the realism created through moulage of simulated patients and environmental special effects in any simulation setting should focus on the participant’s situation awareness and not deter from the overall exercise objectives. In 2019, Saunders et al. ⁹⁷ published a study that demonstrated that virtual-reality-based law enforcement trainings, either by themselves or in combination with traditional hands-on training, can be as effective as highly resource-intensive practical training sessions. ¹²³

Individuals suspend disbelief during the FSEx to approximate the physical and emotional stress of an actual MCI (e.g., the demand of an unknown number of patients with unknown injuries, an unknown supply of SSS in a resource-scarce environment with sensory overload, and fear for their own safety, as well as a failure to accomplish their assignments, or letting their team/ agency down). A successful MCI simulation exercise would match the individual’s pressure for realism through their critical decisions suspending disbelief in the simulation environment without the sensory elements inherent in an FSEx. This can be achieved following design principles explained by Alharthi et al., along with the understanding that the designers must simulate the actual experience making the exercise practical through highly cognitive intense work and physical exertion immersing the individual in the simulation. ¹²⁴ (Supplemental Digital Content Table 1)

To accomplish the objectives of an FSEx, controllers plan and manage exercise play, set up/ operate the exercise site, and act in the roles of organizations, agencies, or individuals that are not playing in the exercise. ¹²⁵ Controllers direct the pace of the exercise, provide key data to players, and may prompt or initiate certain player actions to ensure exercise continuity. Simulators or facilitators provide feedback and cues based on predetermined expected actions of players as well as injects in response to player’s actions that are not expected to maintain the flow of the exercise and to instruct. In addition, they issue exercise material to players as required, monitor the exercise timeline, and supervise the safety of all exercise participants and the surrounding environment.

mD experts agreed that an exercise controller should build realism based on a prepared action script to anticipate and deliver injects based on player’s actions and responses to evaluate non-technical communication skills. MCI simulation exercise creators have the challenge to integrate live or reflex injects based on player’s responses to the scenario, changes in patient’s clinical conditions or other player’s actions. An FSEx occurs in real time with all the agencies simultaneously responding; mD experts agreed that the challenge of MCI simulation exercises is the appreciation of the passage of time to accomplish an action as if in the real time of an FSEx. To achieve this level of realism, mD experts agreed that the exercise design should have modality technicians supporting the exercise to be able to recognize and address any participant struggling with the modality to promote the overall objectives.

A crucial component of any MCI simulation is the debriefing process following the activity; this provides a structured reflection for participants to analyze and self-correct their behavior, decisions, and thought processes to promote cognitive accommodation and assimilation of their learning experience into future professional practice. ⁵² mD experts agreed that observers and evaluators should be specific content experts external to the exercise and use validated template scoring tools to evaluate competencies of players, the exercise itself, and any documentation that is required of regulators or the health authority. These specific content experts can utilize ‘debriefing through meaningful learning,’ ¹²⁶ to provide exercise objectives education. mD experts agreed that this debrief can discover latent safety threats through a frank non-punitive discussion to uncover potential actions that may lead to medical errors. ³⁰ mD experts further agreed that a semi-structured debrief based on validated formats should include all stakeholders to improve the exercise, using open-ended questions to determine MCI simulation exercise areas of improvement.

Conclusion

The modified Delphi experts agreed that the simultaneous integration of individual duty and incident management skills should be incorporated into simulation MCI exercise design to achieve competencies depending on high physical fidelity to develop the individual’s manual abilities, as well as high conceptual fidelity, to develop the individual’s clinical reasoning and problem-solving skills.

MCI simulation exercises can be developed to achieve similar competencies as FSExs incorporating the 19 statements that attained consensus through the TS stages of a scoping review (T1) and mD (T2). The TS process should continue with development of these exercises in the T3 implementation stage and then evaluated in the T4 stage.