Sustained improvement in fellows’ echocardiographic completeness through the coronavirus pandemic with a standardised imaging protocol

Abstract

First-year cardiology fellows must quickly learn basic competency in echocardiography during fellowship orientation. This educational process was disrupted in 2020 due to the coronavirus pandemic, as our hands-on echocardiography teaching transitioned from practice on paediatric volunteers to simulation-based training. We previously described an improvement in echocardiographic completeness after implementation of a standardised imaging protocol for the performance of acute assessments of ventricular function. Herein, we assessed whether this improvement could be sustained over the two subsequent years, including the fellowship year affected by the pandemic. Echocardiograms performed by first-year paediatric cardiology fellows to assess ventricular function were reviewed for completeness. The frequency with which each requested component was included was measured. A total demographic score (out of 7) and total imaging score (out of 23) were calculated. The pre-protocol years (2015–2017) were compared to the post-protocol years (2018–2020), and the pre-COVID years (2018–2019) were compared to the year affected by COVID (2020). There was a sustained improvement in completeness after protocol implementation with improvement in the demographic score (median increasing from 6 to 7, p < 0.001) and imaging score (median increasing from 13 to 16, p < 0.001). More individual components showed a statistically significant increase in frequency compared to our prior publication. The COVID pandemic resulted in very few differences in completeness. Demographic reporting improved modestly (p = 0.04); the imaging score was unchanged (p = 0.59). The only view obtained less frequently was the apical two-chamber view. A standardised imaging protocol allowed sustained improvements in echocardiographic completeness despite the disruption of fellowship orientation by COVID-19.

At the start of paediatric cardiology fellowship, fellows must quickly become facile with performing basic echocardiograms, as this essential skill is not taught during general paediatrics residency. Thus, many fellowship programmes have designed and implemented “boot camps” during fellowship orientation. ^1–5 For the fellowship class that started in summer 2020, these established orientations were thrown into turmoil by the COVID-19 pandemic. ⁶ Many didactic lectures were moved to virtual conferences, and hands-on instruction with the echocardiography machine was curtailed. Our main goal in the present study was to determine whether these changes caused any regression in first-year fellows’ performance on metrics of echocardiographic completeness.

Prior to COVID-19, our institution’s echocardiography orientation consisted of didactic lectures and hands-on practice performing echocardiograms on healthy volunteers. In addition to learning cardiac anatomy and the basics of imaging CHD, the focus of these practice sessions was a competency-based programme ⁷ with the goal that all new fellows should be able to complete an acute assessment of ventricular function (a “function check”) by the end of orientation. To this end, we developed a standardised reporting and imaging protocol as part of a quality improvement initiative. We have previously reported how the implementation of this protocol improved completeness of fellow-performed echocardiograms. ⁸ In 2020, due to institution-wide COVID-19 safety precautions, our orientation changed to online lectures with in-person sessions using echocardiography simulators instead of with healthy volunteers. The goal of competency with the function check protocol by the end of orientation was unchanged.

This study is a follow-up analysis to our prior study, with the addition of two further fellowship years of data, in which we address two research questions. First, were the gains in completeness sustainable across multiple fellowship classes? Second, did the fellowship year affected by the COVID-19 pandemic have a significant difference in completeness relative to the 2 prior years? Our initial hypothesis was that while overall completeness would continue to be higher with the standardised protocol than pre-protocol implementation, the most recent class whose orientation was during the COVID-19 pandemic would score lower on our metrics when performing focused echocardiograms to assess ventricular function.

Materials and methods

This study is a report from an on-going quality improvement project. The study was reviewed by the institutional review board and judged to be quality improvement and standard educational practice, not meeting criteria for human patients research. Informed consent was waived for both fellows and the patients whose echocardiograms were reviewed. The study was conducted in accordance with ethical guidelines and patient privacy protections.

Echocardiography orientation

Our paediatric cardiology orientation is organised by the fourth-year advanced imaging fellows under the guidance of faculty in charge of echocardiography education. Prior to the COVID-19 pandemic, this orientation had included one 1-hour session on the basics of the echocardiography machine followed by six 2-hour sessions where the first-year fellows (six-to-seven per year) practised obtaining standard echocardiographic views on healthy paediatric volunteers. Volunteers were recruited by asking staff and faculty if their children would like to participate. These sessions were primarily focused on obtaining the basic views and measurements necessary to perform the acute function check protocol. ⁸ Additional time during orientation was spent on didactic lectures, which included topics such as segmental anatomy, standard echocardiographic views, and image optimisation.

In July, 2020, due to COVID-19-related restrictions and the inability to have healthy volunteers on-site for orientation, the hands-on training transitioned to two sessions (each 2 hours in length) with echocardiographic simulators. The sessions were conducted in accordance with institutional social distancing and proper sanitising procedures. Three transthoracic echocardiography simulators were utilised: CAE Vimedix (CAE Healthcare, Tampa, FL), Echocom:Neo (Echocom, Nieheim, Germany), and Sheehan (Sheehan LLC, Mercer Island, WA). Fellows rotated between simulators and practised obtaining standard views, with structured simulator- and session-specific goals and objectives. An echocardiography machine (Epiq, Philips, Cambridge, MA) was also incorporated into the simulation sessions in order to give the fellows experience reviewing images and making measurements on the machine interface. Similar to previous sessions with human volunteers, while all standard views were practised, there was a primary focus on the basic views utilised in the function check protocol. The didactic lecture series was moved to a virtual environment.

Imaging protocol

Our standardised protocol for assessing ventricular function in the acute setting has been previously described. ⁸ Briefly, the protocol included two sections: key demographic information to be included in the header of the echocardiogram’s report; and the images to be obtained and the standard calculations of systolic function to be performed. Demographic information included the patient’s height, weight, blood pressure, original cardiac diagnosis, any prior surgeries or procedures, the immediate clinical concern prompting the need for the study during off-hours (proper indication for the study), and the ordering cardiologist. Fellows were asked to specifically document if a study was terminated due to patient instability and to list the imaging views not performed. Image requirements included subcostal, apical, parasternal long, and parasternal short views and sweeps; calculation of ejection fraction using Simpson’s method from apical four- and two-chamber views; M-mode (for tricuspid plane annular systolic excursion and shortening fraction); colour Doppler assessment of inflows, outflows, and the atrial septum; and spectral Doppler of the atrioventricular inflows and the abdominal descending aorta.

Study echocardiograms

Transthoracic echocardiograms were eligible for inclusion if they were assessments of ventricular function or pericardial effusion and performed by first-year fellows without sonographer supervision between October 1 and January 31 of academic years 2015–2020. Exclusion criteria were inability to complete a study due to patient instability (as documented by the fellow); mechanical circulatory support (extracorporeal circulatory membrane oxygenation or ventricular assist device); or use of a different standardised imaging protocol (e.g., pulmonary hypertension or an initial assessment for possible CHD). All studies which met the inclusion criteria were reviewed for demographic reporting completeness. A review of imaging completeness was only performed on studies where all requested images could have been obtained; thus, additional exclusion criteria were applied: single-ventricle heart disease; systemic right ventricles; and unrepaired complex CHD.

Each study echocardiogram was reviewed by one rater (BRW) for both demographic and imaging completeness. Echocardiograms were graded with a yes/no score as to whether they included each requested demographic and imaging component. Total demographic and imaging completeness scores were calculated; the maximum possible demographic score was 7, and the maximum possible imaging completeness score was 23.

Statistical analysis

For each demographic and imaging metric, the percent of compliant echocardiograms was calculated. To compare two groups of fellows, percentages were compared using Pearson’s chi-squared test. Total demographic and imaging scores (presented as median and interquartile ranges) were compared between groups using the Wilcoxon rank-sum test. The threshold for significance was assumed to be p ≤ 0.05.

First, we asked whether the gains seen in our prior manuscript were sustained. As the primary method for this analysis, the three pre-protocol years (fellowship years beginning in 2015–2017) were compared to the three post-protocol years (fellowship years beginning 2018–2020). As a secondary analysis, the first post-protocol year (2018–2019, as previously published) was compared to the subsequent 2 years (2019–2020 and 2020–2021). Our second goal was to ask whether the changes in orientation and clinical practice due to COVID-19 affected imaging completeness. For this analysis, we only considered post-protocol fellowship years. The two pre-COVID years (2018–2019 and 2019–2020) were compared to the post-COVID fellowship year (2020–2021).

Results

Study echocardiograms

A total of 226 pre- and 217 post-protocol echocardiograms met inclusion criteria and had their demographic information reviewed. Of the 217 echocardiograms performed after the function check protocol was introduced, 66 were performed by the 2018–2019 fellowship year and were included in the prior manuscript, 80 were performed by the 2019–2020 fellowship year, and 72 were performed by the 2020–2021 fellowship year. Images were reviewed for 186 pre-protocol and 164 post-protocol echocardiograms. Of these, 49 echocardiograms were from 2018 to 2019 and included in the prior analysis, 60 were from 2019 to 2020, and 55 were from 2020 to 2021.

Sustained completeness improvements after protocol implementation

There was a statistically significant increase in both demographic and imaging completeness scores after the creation of the protocol (Fig 1). These results were similar in magnitude to those in the previously published analysis ⁸ (the demographic reporting score increased from a median of 6 [interquartile range: 5–7] to 7 [interquartile range: 6–7], p < 0.001, and the imaging completeness score increased from a median of 13 [interquartile range: 9–17] to 16 [interquartile range: 13–18], p < 0.001).

Figure 1. Run charts demonstrating compliance with the demographic ( a ) and imaging ( b ) protocols across all of the studied fellowship years. Each blue dot represents the median score for each month. The red lines represent the median score for each fellowship year.

With the increased power available in this follow-up analysis, more widespread improvements were seen in the completeness of individual imaging components (Table 1). With regard to demographics, in addition to increased frequency of reporting blood pressure, the ordering clinician, and the cardiac diagnosis (all seen in our prior manuscript), there were now also statistically significant improvements in the reporting of a patient’s height and prior surgeries. When examining the inclusion of the requested echocardiographic views, all improvements seen in the prior study (spectral Doppler of the descending aorta, the apical two-chamber view, Simpson’s method from the apical four-chamber view, tricuspid annular plane systolic excursion, and the parasternal short axis view of the base of the heart) were sustained with the exception of the calculation of ejection fraction using Simpson’s method from the apical two-chamber view. Additional improvements were now seen in the frequency of which the following views were performed: subcostal sweeps for function, colour Doppler assessment of the atrial septum, the apical four-chamber view, pulse wave Doppler of the mitral valve inflow, colour Doppler of the left ventricular outflow tract, and parasternal long axis views (Table 1).

Table 1. Frequency with which components of the ventricular function protocol were obtained by first-year paediatric cardiology fellows both before and after the introduction of the standardised function check protocol. TAPSE: tricuspid annular plane systolic excursion; LAO: left anterior oblique

We performed a secondary analysis asking whether the previously published post-protocol year differed from the two newly analysed years. There were no differences in overall demographic (median 7, interquartile range 6–7 for both groups, p = 0.09) or imaging (median 17, interquartile range 12.75–19 versus median 16, interquartile range 13–18, p = 0.22) scores. The two more recent years were more likely to report a height (p = 0.04) and the ordering physician’s name (p = 0.02, Supplemental Table 1). Over the subsequent 2 years, some apical views were less likely to be obtained (four-chamber sweep, p = 0.46; two-chamber, p = 0.16; and two-chamber Simpson’s method, p < 0.001, Supplemental Table 1). We note that these results are confounded by the COVID pandemic (see the next analysis).

Effects of the COVID-19 pandemic and changes in orientation

Despite the disruptions in echocardiography orientation due to the COVID-19 pandemic, there were very few differences in demographic or imaging compliance between the 2018–2019 fellowship years and the 2020 fellowship year (Table 2). The 2020 fellowship year did annotate the weight less frequently (100% versus 97%, p = 0.04). However, their demographic reporting score overall improved relative to the prior years (increased from a median of 7, interquartile range 6–7 to a median of 7, interquartile range 7–7, p = 0.04). The only echocardiographic view which the 2020 year performed significantly less frequently was the apical two-chamber view (compliance decreased from 61 to 38%, p = 0.007). Overall imaging scores were similar (median 16, interquartile range 13–18 versus median 16, interquartile range: 12.25–18, p = 0.59).

Table 2. Frequency with which components of the ventricular function protocol were obtained by first-year paediatric cardiology fellows after the introduction of the standardised function check protocol before and during the COVID-19 pandemic. TAPSE: tricuspid annular plane systolic excursion; LAO: left anterior oblique

Discussion

These results extend our earlier investigation of the effects of implementing a standardised reporting and imaging protocol for first-year cardiology fellows evaluating cardiac function in the acute care setting. The improvements seen with the use of the protocol were sustained across two additional fellowship years, and the increased power allowed us to demonstrate more wide-ranging improvements than seen in the initial study. Most interestingly, and contrary to expectations, we saw that there were only minimal differences in performance by the fellows who had a simulation-based echocardiography orientation due to the COVID-19 disruption. However, our results do demonstrate some areas where compliance with the protocol remains low.

With regard to demographic completeness, the immediate indication for the echocardiogram was included only 80% of the time. Future education could emphasise the importance of communicating with the ordering team to understand why a particular patient needs an echocardiogram at a particular time. From an imaging standpoint, some views are still obtained less than 50% of the time. Often this is due to fellows performing one of a “set” of images (e.g., a parasternal long axis view but not a sweep); education could focus more on why multiple views of structures are helpful. Fellows also often perform imaging not requested by the protocol (e.g., spectral Doppler of the LVOT/RVOT or tissue Doppler), whereas limited time spent imaging a critically ill child might better be spent on the images designated in the protocol. The apical two-chamber view remained inconsistently obtained and Simpson’s method from this view was performed infrequently. The two-chamber view is notoriously difficult to perform correctly, ⁹ and the use of simulators may not adequately prepare fellows for how to obtain this view in clinical practice. Of note, only the CAE simulator included the ability to perform two-chamber views. Fellows’ continued difficulties performing this view provide additional motivation to emphasise the use of the 5/6 area-length method for quantification of ejection fraction instead of Simpson’s method. ^10,11

Despite these areas for improvement, these results are overall encouraging that echocardiographic education could remain strong despite the COVID-19 pandemic. The pandemic resulted in massive disruptions to the typical educational schedule. ^6,12 Many programmes increased the amount of online didactic content used in their orientations, bootcamps, and educational programmes. ¹³ Online learning has been shown to be an effective method to learn knowledge related to paediatric cardiology ^14,15 and echocardiography. ¹⁶ However, echocardiography is clearly not a skill that can be obtained in a purely virtual or remote learning environment. ¹⁷ Our results demonstrate that a combination of remote learning and simulator sessions could provide the fundamental knowledge and hands-on skills necessary to complete a focused assessment of ventricular function in the acute setting.

Multiple echocardiography simulators have been developed to aid training ¹⁸ and have been reported to be helpful for paediatric cardiology education. ^2,19 One notable limitation of simulator training, as compared to the use of volunteers, is the lack of imaging artefact secondary to lung tissue or bone. While certain simulators have been designed to attempt to emulate this artefact, anecdotally our fellows reported this feature was not very realistic. While, in theory, there is no reason why fellows could not memorise the protocol’s views regardless of hands-on training method, in practice, when on overnight or weekend call, fellow’s performance is affected by distractions, frustrations at imaging difficulty, and the need to work efficiently. The use of human volunteers may better prepare fellows for real-world imaging, so that more mental effort can be spent on remembering the protocol requirements. A second significant limitation of echocardiography simulation in paediatric cardiology is that many simulators focus on adult as opposed to paediatric echocardiography. Simulator views differ from basic paediatric views and they generally lack models of CHD. Given that our study focused on the assessment of ventricular function in patients with normal cardiac anatomy or following repair to a biventricular circulation, the echocardiograms in this study were likely as close to simulators and healthy paediatric volunteers as would be possible.

Our study has limitations typical of quality improvement studies. Fellows were not randomised to receive any educational interventions. The entirety of the fellowship orientation is refined yearly by each new group of 4th-year fellows. For example, the time spent with human volunteers increased over the first 5 years of this study. Thus, the changes seen may not be solely due to a standardised protocol or COVID-19. From a demographic standpoint, the components that we assessed can also be changed by the attending within the echocardiography reporting software; thus, some improvement may be due to increasing attending familiarity with the reporting requirements. Also, we only assessed whether certain images were included in the study, and we did not judge imaging quality or the ability to answer a specific clinical question. It is possible that some studies, while “complete”, were of poor quality. We also continue to only assess first-year fellows; we have not examined whether improvements in completeness were sustained across fellowship years.

Our study does not attempt to capture fellows’ performance in the diagnosis or assessment of CHD, where a comparison between these fellowship classes may have demonstrated differences. As CHD represents a broad spectrum of pathology, increased availability of a simulators with classic congenital cardiac pathologies could be extremely helpful in giving trainees more practice performing echocardiograms of CHD, where the ability to image from many views and in the inclusion of off-axis views may be particularly important. A similar imaging completeness metric to ours could be developed for CHD-specific imaging protocols to allow a more complete assessment of first-year fellow competence.

The continuing results of this study demonstrate the success of a standardised imaging protocol and focused, competency-based education to improve the completeness of first-year paediatric cardiology fellow-performed echocardiograms. Having a standardised protocol allowed these gains to be sustained across multiple academic years despite changes in educators. In particular, we saw only a minimal effect due to the COVID-19 pandemic. Lower protocol compliance remains in some areas, and future educational interventions as part of this quality improvement initiative should target those deficiencies.