Operation based exercises represent simulation activities, which are of great importance for emergency preparedness, as they simulate real experiences in a guided manner. Whereas their primary purpose is to address the organizational emergency preparedness, little is known about the personal benefits of involved participants and whether these positive changes endure over time.

Immediate and medium term assessment of the effectiveness on individual preparedness and benefits of participants, based on self-perception, after participating in a set of 4 interdisciplinary field exercises organized as part of the MSc in Global Health-Disaster Medicine of the Medical School of the National and Kapodistrian University of Athens, Greece. The field exercises were carried out yearly, from 2016 to 2019. Data were collected via questionnaires pre- and post-exercise (1 week and 10 months after participation). The sample size was 228 trainees, with a response rate of 88%.

The majority (95%) stated that Mass Casualty Incident (MCI) exercises are appropriate for disaster management training in terms of comprehending theory, and for team-building training. In the case of a real MCI, 22% of the participants declared themselves to be ready to respond prior to MCI exercises. Upon completion, the overall perception of readiness among the participants increased to 77%. Trainee feedback indicated enhancement of both technical and non-technical skills (87%), which were persistent over time, and revealed a high level of satisfaction with the training.

This study shows a positive immediate and medium-term impact of operation-based exercises on technical, non-technical skills, and self-perception of participants.

The evaluation tool was first derived from the formerly Consortium of British Humanitarian Agencies’ (CBHA; United Kingdom), now “Start Network’s,” Core Humanitarian Competency Framework and formatted in an electronic data capture tool that allowed for offline evaluation. During a 3-day humanitarian simulation event, participants in teams of eight to 10 were evaluated individually at multiple injects by trained evaluators. Participants were assessed on five competencies and a global rating scale. Participants evaluated both themselves and their team members using the same tool at the end of the simulation exercise (SimEx).

All participants (63) were evaluated. A total of 1,008 individual evaluations were completed. There were 90 (9.0%) missing evaluations. All 63 participants also evaluated themselves and each of their teammates using the same tool. Self-evaluation scores were significantly lower than peer-evaluations, which were significantly lower than evaluators’ assessments. Participants with a medical degree, and those with humanitarian work experience of one month or more, scored significantly higher on all competencies assessed by evaluators compared to other participants. Participants with prior humanitarian experience scored higher on competencies regarding operating safely and working effectively as a team member.

This study presents a novel electronic evaluation tool to assess individual performance in five of six globally recognized humanitarian competency domains in a 3-day humanitarian SimEx. The evaluation tool provides a standardized approach to the assessment of humanitarian competencies that cannot be evaluated through knowledge-based testing in a classroom setting. When combined with testing knowledge-based competencies, this presents an approach to a comprehensive competency-based assessment that provides an objective measurement of competency with respect to the competencies listed in the Framework. There is an opportunity to advance the use of this tool in future humanitarian training exercises and potentially in real time, in the field. This could impact the efficiency and effectiveness of humanitarian operations.

EvansAB, HulmeJM, NugusP, CranmerHH, CoutuM, JohnsonK. An Electronic Competency-Based Evaluation Tool for Assessing Humanitarian Competencies in a Simulated Exercise. Prehosp Disaster Med. 2017;32(3):253–260.

In resource-constrained environments, appropriately employing triage in disaster situations is crucial. Although both case-based learning (CBL) and simulation exercises (SEs) commonly are utilized in teaching disaster preparedness to adult learners, there is no substantial evidence supporting one as a more efficacious methodology. This randomized controlled trial (RCT) evaluated the effectiveness of CBL versus SEs in addition to standard didactic instruction in knowledge attainment pertaining to disaster triage preparedness.

This RCT was performed during a one-day disaster preparedness course in Lucknow, India during October 2014. Following provision of informed consent, nursing trainees were randomized to knowledge assessment after didactic teaching (control group); didactic plus CBL (Intervention Group 1); or didactic plus SE (Intervention Group 2). The educational curriculum used the topical focus of triage processes during disaster situations. Cases for the educational intervention sessions were scripted, identical between modalities, and employed structured debriefing. Trained live actors were used for SEs. After primary assessment, the groups underwent crossover to take part in the alternative educational modality and were re-assessed. Two standardized multiple-choice question batteries, encompassing key core content, were used for assessments. A sample size of 48 participants was calculated to detect a ≥20% change in mean knowledge score (α=0.05; power=80%). Robustness of randomization was evaluated using X2, anova, and t-tests. Mean knowledge attainment scores were compared using one- and two-sample t-tests for intergroup and intragroup analyses, respectively.

Among 60 enrolled participants, 88.3% completed follow-up. No significant differences in participant characteristics existed between randomization arms. Mean baseline knowledge score in the control group was 43.8% (standard deviation=11.0%). Case-based learning training resulted in a significant increase in relative knowledge scores at 20.8% (P=0.003) and 10.3% (P=.033) in intergroup and intragroup analyses, respectively. As compared to control, SEs did not significantly alter knowledge attainment scores with an average score increase of 6.6% (P=.396). In crossover intra-arm analysis, SEs were found to result in a 26.0% decrement in mean assessment score (P < .001).

Among nursing trainees assessed in this RCT, the CBL modality was superior to SEs in short-term disaster preparedness educational translation. Simulation exercises resulted in no detectable improvement in knowledge attainment in this population, suggesting that CBL may be utilized preferentially for adult learners in similar disaster training settings.

AluisioAR, DanielP, GrockA, FreedmanJ, SinghA, PapanagnouD, ArquillaB. Case-based Learning Outperformed Simulation Exercises in Disaster Preparedness Education Among Nursing Trainees in India: A Randomized Controlled Trial. Prehosp Disaster Med. 2016;31(5):516–523.

The objective of this study was to highlight 2 models, the Hospital Incident Command System (HICS) and the Disaster Management Indicator model (DiMI), for evaluating the in-hospital management of a disaster situation through simulation exercises.

Two disaster exercises, A and B, with similar scenarios were performed. Both exercises were evaluated with regard to actions, processes, and structures. After the exercises, the results were calculated and compared.

In exercise A the HICS model indicated that 32% of the required positions for the immediate phase were taken under consideration with an average performance of 70%. For exercise B, the corresponding scores were 42% and 68%, respectively. According to the DiMI model, the results for exercise A were a score of 68% for management processes and 63% for management structure (staff skills). In B the results were 77% and 86%, respectively.

Both models demonstrated acceptable results in relation to previous studies. More research in this area is needed to validate which of these methods best evaluates disaster preparedness based on simulation exercises or whether the methods are complementary and should therefore be used together. (Disaster Med Public Health Preparedness. 2016;10:544–548).

Good staff procedure skills in a management group during incidents and disasters are believed to be a prerequisite for good management of the situation. However, this has not been demonstrated scientifically. Templates for evaluation results from performance indicators during simulation exercises have previously been tested. The aim of this study was to demonstrate the possibility that these indicators can be used as a tool for studying the relationship between good management skills and good staff procedure skills.

Good and structured work (staff procedure skills) in a hospital management group during simulation exercises in disaster medicine is related to good and timely decisions (good management skills).

Results from 29 consecutive simulation exercises in which staff procedure skills and management skills were evaluated using quantitative measurements were included. The statistical analysis method used was simple linear regression with staff procedure skills as the response variable and management skills as the predictor variable.

An overall significant relationship was identified between staff pro-cedure skills and management skills (p ≤0.05).

This study suggests that there is a relationship between staff procedure skills and management skills in the educational setting used. Future studies are needed to demonstrate if this also can be observed during actual incidents.