Ultrasound during Critical Care Simulation: A Randomized Crossover Study

Abstract

Objectives

We sought to compare two ultrasound simulation interventions used during critical care simulation. The primary outcome was trainee and instructor preference for either intervention. Secondary outcomes included the identification of strengths and weaknesses of each intervention as well as overall merits of ultrasound simulation during high-fidelity, critical care simulation. The populations of interest included emergency medicine trainees and physicians.

Methods

This was a randomized crossover study with two ultrasound simulation interventions. 25 trainees and eight emergency physician instructors participated in critical-care simulation sessions. Instructors were involved in session debriefing and feedback. Pre- and post-intervention responses were analyzed for statistically significant differences using t test analyses. Qualitative data underwent thematic analysis and triangulation.

Results

Both trainees and instructors deemed ultrasound simulation valuable by allowing trainees to demonstrate knowledge of indications, correct image interpretation, and clinical integration (p<0.05). Trainees described increased motivation to develop and use ultrasound skills. The edus2 was the preferred intervention, as it enabled functional fidelity and the integration of ultrasound into resuscitation choreography. Instructors preferred the edus2, as it facilitated better assessment of trainees’ skills, thus influencing feedback.

Conclusions

These findings support the use of ultrasound simulation during critical care simulations. The increased functional fidelity associated with edus2 suggests that it is the preferred intervention. Further study of the impact on clinical performance is warranted.

Résumé

Objectifs

L’étude visait à comparer deux interventions échographiques simulées, dans le cadre de séances de formation en soins intensifs. Le principal critère d’évaluation consistait à déterminer la préférence des stagiaires et celle des instructeurs pour l’une ou l’autre des interventions. Les critères d’évaluation secondaires comprenaient les points forts et les points faibles de chacune des interventions ainsi que les avantages généraux de la simulation d’échographie dans le cadre de séances de formation réaliste en soins intensifs. La population concernée se composait de stagiaires en médecine d’urgence et d’urgentologues.

Méthode

Il s’agit d’une étude croisée, à répartition aléatoire, qui comptait deux interventions échographiques simulées. Vingt-cinq stagiaires et huit urgentologues instructeurs ont participé aux séances de simulation en soins intensifs. Les instructeurs étaient chargés des séances de compte rendu et de la rétroaction des stagiaires. Il y a eu analyse des réponses recueillies avant et après les interventions afin de dégager des écarts statistiquement significatifs à l’aide d’analyses fondées sur le test t. Quant aux données qualitatives, elles ont été soumises à une analyse thématique, puis à la triangulation.

Résultats

Tant les stagiaires que les instructeurs ont jugé intéressante la simulation d’échographie, les stagiaires ayant la possibilité de montrer leur connaissance des indications, d’interpréter correctement les images et d’intégrer la matière sur le plan clinique (p<0,05). Les stagiaires ont aussi fait état d’une motivation accrue pour acquérir des compétences en échographie et les appliquer. En outre, la deuxième intervention s’est révélée la formule préférée des stagiaires parce qu’elle avait un caractère fonctionnel réaliste et qu’elle permettait l’intégration de l’échographie dans l’enchaînement des gestes de réanimation. De leur côté, les instructeurs ont également indiqué leur préférence pour la deuxième intervention parce qu’elle facilitait l’évaluation des compétences des stagiaires et qu’elle influait, par le fait même, sur la rétroaction.

Conclusions

Les résultats de l’étude confirment la pertinence des simulations d’échographies dans le cadre de séances de formation en soins intensifs. C’est le caractère fonctionnel réaliste de la deuxième intervention qui en aurait fait la formule préférée. Il faudrait mener d’autres études afin de déterminer l’incidence de ce type d’enseignement sur la compétence clinique.

Introduction

Critically ill patients presenting to emergency departments are assessed urgently through a variety of approaches. Increasingly, emergency physicians are using point-of-care ultrasound (PoCUS) to enhance their assessment of such patients. ^{1 - 3}

Performing resuscitative PoCUS (ultrasonography directly related to an acute resuscitation in critical care) ¹ is made more complex by the patient’s precarious clinical state and resulting surroundings, creating logistical challenges in terms of timing and access to the patient. Effective resuscitative PoCUS relies on situational awareness, understanding of the rationale for performing a scan at a given time, and the ability to accurately interpret and act on the findings. ⁴ Generally, individual aspects of critical care should be employed in a “simultaneous, choreographed approach.” ⁵ Such choreography strives to minimize interruptions in critical actions while ensuring that the patient undergoes all appropriate assessments and receives required therapies.

As such, emergency medicine (EM) trainees must not only master the skill of generating ultrasound images, but also become efficient at incorporating such scanning into their resuscitation. Competence in resuscitative PoCUS can thus be divided into three broad components: awareness of indications, with associated rationale; the technical skill of image generation, with simultaneous interpretation of findings; and, lastly, the appropriate integration of findings into resuscitation. The role of simulation for ultrasound skill development and competence has only recently gained attention, with the majority of current literature focusing on ultrasound-guided procedures and image generation/interpretation. ^{6 , 7} The incorporation of PoCUS simulation into high-fidelity simulation (HFS), through what has been described as hybrid simulation, ⁸ may contribute to the development of overall resuscitative PoCUS competence. Such hybrid simulations, or simulated patient encounters, offer an intermediary step whereby the skills learned outside the critical-care context (i.e., at courses, through scanning volunteers and patients who are otherwise well) can be re-integrated into a clinical environment in a way that is developmentally appropriate. ⁹

Many training programs have already incorporated resuscitative PoCUS into HFS to varying degrees. Some centers may use printed images or offer verbal descriptions of findings (representing the simplest method for integrating ultrasound into critical care simulation). In contrast, Kobayashi et al. (2010) published an ultrasound simulation teaching package that incorporated a laptop (placed on an audiovisual cart) as a basic ultrasound simulation device. ¹⁰ Expanding upon such integration, Kulyk et al. incorporated a simulated ultrasound probe to the laptop concept (known as the edus2), thus providing trainees with an opportunity to perform scans in real time during HFS. ¹¹

The purpose of this study was to evaluate two comparable ultrasound simulation interventions as used during HFS. Comparing two somewhat similar but distinct interventions allowed the study team to assess and isolate for the potential value of basic probe handling and other logistical aspects associated with the use of either intervention.

Methods

Study design

This was a prospective, randomized, crossover trial involving post-graduate specialist trainees in EM (trainees) and EM physicians involved in PoCUS teaching (instructors). Trainees and instructors participated in a day-long simulation-based critical care ultrasound course. This research project was approved by the Behavioural Research Ethics Board of both the University of Saskatchewan Research Ethics Board (BEH #12-292) and by Barts Health Trust, NHS (ReDA #9017).

Study setting and population

This study was conducted at Medical Education Training Suite, Whipps Cross Hospital, Whipps Cross Road, Leytonstone, London, UK. The trainee population was recruited from the London Specialist School of Emergency Medicine. A total of 25 EM trainees participated in the study over four separate course dates. EM instructors were selected by the local study supervisor (T.M.) based on expertise in PoCUS and overall teaching skill-set. Of the eight faculty physicians in the study, five were full-time consultants in EM. The remaining three were senior trainees (chosen by T.H.) deemed sufficiently experienced in EM and PoCUS to serve as instructors for the course.

Study interventions

The edus2 is a PoCUS simulator that is comprised of a modified ultrasound probe and laptop. ¹¹ When used within a high-fidelity simulation (HFS) setting, the edus2 plays pre-recorded video clips of areas of interest through the coupling of those videos to specific Radio Frequency Identification Device (RFID) cards placed under the skin of any commercially available HFS mannequin. Passing the edus2 probe over a RFID card located beneath a mannequin’s skin initiates a video clip of the corresponding anatomic area (on the HFS mannequin) to be viewed by the trainee on the simulator screen. Multiple scans are possible during any given scenario (by placing several cards under the skin of the mannequin); however, no image manipulation can take place (clips play to completion once the probe passes over the RFID tag under the skin).

The second ultrasound simulation intervention was a laptop placed on an audiovisual cart. During a given scenario, trainees can bring the laptop to the mannequin’s bedside and choose the desired clip as labeled within the case folder. ¹⁰ While this intervention lacks probe manipulation, it is an established method of integrating some cognitive aspects of resuscitative PoCUS into critical care simulation (what to scan, image interpretation, and integration). Both interventions require minimal financial investment (beyond an existing HFS infrastructure), as both can run on used laptops. In addition, both interventions could be used with real standardized patients.

Study protocol

Trainees were randomized based on order of arrival to the simulation centre into two groups (A or B) with two to four trainees per group. Each trainee group was assigned to one of two study arms that involved both the use of the edus2 as well as the simple laptop for a total of four HFS scenarios. Group A trainees completed their first two cases with the edus2 followed by two cases with the use of the laptop. Group B was assigned the same cases but with the PoCUS interventions presented in reverse order. All trainees completed an entrance MCQ (multiple choice question) exam (based on the American College of Emergency Physician’s EMSONO online exam, ¹² permission obtained) as a means of establishing level of knowledge as well as the success of group randomization (Figure 1).

Figure 1 Study Design.

Each HFS scenario was followed by a standardized debriefing session led by an instructor and a course facilitator. In order to capture differences in the debriefing experiences that followed each intervention, the EM physician instructors were surveyed. These surveys focused on the performance of the PoCUS interventions as well as the debriefing experiences that followed each case.

Pre- and post-intervention responses (10-point Likert scale) were compared and analyzed in order to determine the strengths of either intervention, as well as for direct comparison.

Scenario development

The four cases were chosen from the ultrasound case simulation package prepared by Kobayashi et al. (2010). They represent the shock or peri-arrest states associated with the following conditions: ruptured aortic aneurysm; blunt abdominal trauma with hemoperitoneum; cardiac tamponade; and cardiac arrest secondary to massive pulmonary embolism. ¹⁰

Outcome measures

The primary outcome was trainee and instructor preference for either intervention. Secondary outcomes included identification of strengths and weaknesses of each intervention, as well as overall merits of PoCUS simulation during critical care simulation.

Data analysis

Quantitative data included test and survey scores (Likert scale). The significance (alpha) level for all analysis was set at p<.05. t test analyses were used for comparisons of pre- and post-intervention scores within interventions, as well as comparisons between the two interventions at different time-points.

Qualitative data analysis included thematic and emergent analysis from written responses in self-reports and online survey feedback. The author and a second evaluator (K.T.) independently completed thematic analysis of the qualitative data. The thematic analysis then underwent triangulation, where only themes agreed to by two assessors were included in the final analysis.

Results

Average pre-intervention MCQ score for all trainees was 71.5%, which suggests familiarity with PoCUS, as would be expected for middle-level trainees. Most of the trainees (21/25) had had previous HFS experience, with the majority of trainees having had 3-5 previous HFS experiences. Nearly all had attended a level I PoCUS course or equivalent (21/25). Trainees responded that to date previous HFS experiences had only poorly, if at all, integrated resuscitative PoCUS into HFS, with a mean score of 3.26 out of 10 (SD 2.73).

Primary study question

The primary study outcome was trainee and instructor preference for either intervention. Trainees preferred the edus2 intervention to the simple laptop. Initial survey responses demonstrated that trainees rated the two interventions similarly (see Figure 2). Both interventions were deemed to be an improvement from previous experiences in all of the evaluated aspects of the interventions. No statistically significant difference was identified between the edus2 and laptop intervention. Upon completion of the qualitative survey (a few weeks after the course), all respondents (20/25 study participants) indicated a preference for the edus2. As one trainee put it, “holding the probe makes the simulation closer to reality and real time.” The themes of “real-time” and “hands-on use” dominated the survey responses. This was contrasted with the artificial nature of the videos found on the laptop. “The laptop meant you were trying to find the correct [clip], which detracted from the simulation.”

Figure 2 Trainees were asked how well the simulation scenario (previous, laptop, or edus2) addressed the following aspects of Resuscitative PoCUS. All comparisons to previous experiences were statistically significant (p<0.05). No statistically significant difference was observed between the laptop and edus2.

Instructors rated the edus2 as the superior training intervention, as compared to both previous experiences and the laptop intervention (see Figures 3 and 4). Upon review of qualitative data, features identified with the edus2 included the ability to assess trainee use of PoCUS in real time, as well as basic probe handling: “I was able to assess their use of [ultrasound] in cardiac arrest and the timing of echo with CPR.” This theme relates to the previously described concept of resuscitation choreography.

Figure 3 Instructors were asked how well the simulation scenario (previous, laptop, or edus2) addressed the following aspects of Resuscitative PoCUS. *Laptop intervention scores did not meet statistical significance (p<0.05).

Figure 4 Instructors were asked how well the simulation scenario (previous, laptop, or edus2) addressed the following training aspects of Resuscitative PoCUS. *Laptop intervention scores did not meet statistical significance (p<0.05).

Secondary outcomes

No statistically significant differences in MCQ scores were generated after completion of the first two cases.

In terms of impact of the session on their clinical work, the majority of trainees expressed an increased awareness of indications for use of PoCUS. As one trainee commented, the course increased “the number of situations in which I would consider using ultrasound.” This theme was coupled with what trainees described as an increased “enthusiasm for practicing [ultrasound] scanning [and] an appreciation of the value of [ultrasound] scanning in management of critically ill patients.” The day-long session impacted training by encouraging most trainees “to seek more training in [ultrasound] scanning and to reach Level I sign off and beyond.” More advanced trainees described a lesser impact, as they “already use [ultrasound] scanning as a part of [their] examination” in many patient encounters, but they also noted that the session inspired them to “get involved in [ultrasound] teaching.”

Both interventions were rated favorably in terms of integrating PoCUS into HFS when compared to previous experiences (see Figures 2 and 4). Faculty members commented that both interventions allowed for “a reasonable assessment of [knowledge] of indications” and noted that the interventions clearly “added PoCUS into the decision making process.”

The next series of questions posed to instructors related to the impact the PoCUS interventions had on the debriefing sessions (see Figure 4). One faculty participant explained that the edus2 allowed for “feedback on everything from positioning of equipment, to communication with patient, to documentation and medico-legal issues.” On the other hand, faculty felt the laptop intervention offered them very little insight into trainee skills and as such made it hard to bring up PoCUS skills during debrief. This “lack of ownership of the skill made the feedback less applicable to the trainees.”

Discussion

Our conceptual framework (Figure 5) illustrates key concepts supporting the use of PoCUS simulation in HFS. Learning takes place according to Bloom’s three domains. ¹² Development is paired through interplay between the trainee and the instructor/preceptor. As per Cognitive Load Theory (CLT), ¹³ as trainees become more proficient with PoCUS (relying less on short-term memory and more on both long-term and motor memory), they become increasingly capable of focusing on the clinical picture before them. Instructors can identify aspects of trainee PoCUS use that may require further development, and subsequently can create opportunities for deliberate practice. Simultaneously, clinical competence can be assessed using Miller’s framework, ¹⁴ while recognizing the challenges inherent to the assessment of critical care skills (namely the infrequency and non-standardizability of such cases). Lastly, Kirkpatrick’s Hierarchy of Evidence ¹⁵ allows one to evaluate whether transfer of learning has taken place, and may help with determining if the intervention will have any impact on actual patient care. ¹⁶

Figure 5 Conceptual Framework PoCUS – Point of Care Ultrasound, ZPD - Zone of Proximal Development.

In terms of PoCUS integration into HFS, trainees rated both interventions favorably. This may be related to the fact that both interventions, though not identical, offered trainees a simplified form of the more complex task of resuscitative PoCUS. As Kneebone et al. stated, one of the strengths of simulation is that it “offers controllable levels of challenge that can be adjusted according to individual need.” ⁹

Thematic analysis of the qualitative data revealed that all trainees preferred the edus2 to the laptop, as it offered “a more hands on experience in real time.” Despite recreating only the basics of image generation, use of the edus2 challenges the trainee to consider the logistical challenges of appropriate use (for example, the proper sequence of PoCUS during cardiac arrest has been questioned). ¹⁷ Furthermore, the established resuscitation sequences of trauma, cardiac arrest, and undifferentiated shock are not identical; ^{18 , 19} neither is the integration of PoCUS into these scenarios. Based on the findings of this study and related simulation literature, the next step should be to teach PoCUS choreography in HFS. Existing evidence of improved team performance ^{20 , 21} and resuscitation ^{22 , 23} supports the use of simulation interventions for the integration of resuscitative PoCUS.

Simultaneously, probe driving skills can be taught outside the clinical context. As such, resuscitative PoCUS may be best taught through a combination of training environments. This type of “progressive fidelity” ²⁴ may prove to be an efficient way to teach as complex and multidimensional a skill as resuscitative PoCUS.

The edus2 and the laptop represent two ways of integrating PoCUS into HFS. Other options include placing a diagnostic trainer (i.e., CAE Vimedix) beside the HFS mannequin. This trainer can then be turned to and used when indicated during a scenario. ²⁵ Each of these approaches represent varying degrees of engineered and psychological fidelity, and as such also represent the spectrum of cost (where increasing engineered fidelity is generally associated with increasing costs). ²⁴

Hamstra et al. (2014) recently addressed the challenges associated with fidelity, given that increased engineered fidelity doesn’t always result in increased educational value. ²⁶ They coined the term “functional fidelity,” and moreover, the concept of “functional task alignment,” where a given simulation intervention is defined by the learning objectives and tasks associated with the clinical skill, rather than the intervention's physical resemblance. This approach frames the evaluation of our two PoCUS simulation interventions not by their semblance to the real equipment, but rather by each intervention’s ability to address learning objectives and simulate the desired set of tasks. When introducing resuscitative PoCUS into critical care simulation, it can be argued that it is clinical integration (knowing when to scan, what to scan, and how to interpret findings within the case) that is of greatest interest. Specific image generation, on the other hand, can be taught outside the critical care context. While both interventions met several of the objectives as related to resuscitative PoCUS skills, the edus2 was found to meet more of the desired tasks, and therefore showed better functional task alignment.

Limitations

This study has several limitations. Unlike this study, most simulation intervention studies are designed to be longitudinal and as such harness the fullest potential of key constructs like deliberate practice. ²⁷ Furthermore, trainees were asked about the perceived impact of the single session on their future clinical work and education, but we did not follow/verify those reports, nor does the study inform us of the impact of repeated exposure to the intervention.

Despite significant efforts to reduce personal bias, it is possible that the lead author’s involvement in the sessions (as the voice of the simulated patients and presenter of the two interventions), while also being co-developer of the edus2, may have biased some of the trainee and faculty responses. Several steps were taken during the design and implementation of the study to mitigate bias. The decision to carry out the study at another institution served to minimize personal biases and conflicts of interest that may exist between the lead author and students/instructors at his institution. Secondly, the use of previously designed cases by Kobayashi et al. ensured that the HFS scenarios were not biased toward one type of intervention or the other. Thirdly, the lead author holds no patents or commercial interests pertaining to the edus2 or related ultrasound simulation products. The edus2 is under a creative commons copyright and as such is to be shared and further developed in an open and non-commercial fashion.

Future Research

Developing a repertoire for use of PoCUS during resuscitation and emergent care will require deliberate practice. Future studies could be directed at determining whether repeated exposures to this training intervention results in better use of resuscitative PoCUS during real resuscitation.

Conclusion

The integration of resuscitative PoCUS into critical care HFS represents an evolution of PoCUS training in residency. This progression is driven by a desire by both EM trainees and faculty to safely integrate PoCUS into the care of critically ill patients. The findings of the study support the integration of PoCUS into HFS. Such integration was found to be of value to both trainees and instructors in terms of allowing trainees to demonstrate knowledge of indications as well as correct image interpretation. The edus2 was identified as being a superior teaching intervention, as it allowed for greater functional integration of PoCUS into critical care (real time and hands-on), significantly better assessment of trainee skills, and subsequently had greater impact on session debrief and formative feedback.