The diagnostic accuracy of a point-of-care ultrasound protocol for shock etiology: A systematic review and meta-analysis

Abstract

Objective

The aim of this study was to perform a systematic review and meta-analysis of the diagnostic accuracy of a point-of-care ultrasound exam for undifferentiated shock in patients presenting to the emergency department.

Methods

Ovid MEDLINE, Scopus, Cochrane Central Register of Controlled Trials, and research meeting abstracts were searched from 1966 to June 2018 for relevant studies. QUADAS-2 was used to assess study quality, and meta-analysis was conducted to pool performance data of individual categories of shock.

Results

A total of 5,097 non-duplicated studies were identified, of which 58 underwent full-text review; 4 were included for analysis. Study quality by QUADAS-2 was considered overall a low risk of bias. Pooled positive likelihood ratio values ranged from 8.25 (95% CI 3.29 to 20.69) for hypovolemic shock to 40.54 (95% CI 12.06 to 136.28) for obstructive shock. Pooled negative likelihood ratio values ranged from 0.13 (95% CI 0.04 to 0.48) for obstructive shock to 0.32 (95% CI 0.16 to 0.62) for mixed-etiology shock.

Conclusion

The rapid ultrasound for shock and hypotension (RUSH) exam performs better when used to rule in causes of shock, rather than to definitively exclude specific etiologies. The negative likelihood ratios of the exam by subtype suggest that it most accurately rules out obstructive shock.

Résumé

Objectif

L’étude visait à réaliser une revue systématique et une méta-analyse de l'exactitude diagnostique de l’échographie au point de service dans le diagnostic des états de choc non différenciés au service des urgences.

Méthode

Une recherche documentaire a été entreprise dans les bases de données Ovid, MEDLINE, Scopus et le registre Cochrane Central Register of Controlled Trials ainsi que dans des résumés de congrès sur la recherche dans le but de relever des études pertinentes, de 1966 à juin 2018. La qualité des études a été évaluée à l'aide de QUADAS-2, et une méta-analyse, effectuée afin de réunir des données sur la performance de l'examen dans différentes catégories d’état de choc.

Résultats

Ont été relevées en tout 5097 études nouvelles, dont 58 ont fait l'objet d'un examen en version intégrale; sur ce dernier nombre, 4 ont été retenues aux fins de l'analyse. La qualité des études comportait, dans l'ensemble, un faible risque de bais, d'après l'instrument QUADAS-2. Les valeurs réunies des rapports de vraisemblance positifs variaient de 8,25 (IC à 95% : 3,29 à 20,69) pour les chocs d'origine hypovolémique à 40,54 (IC à 95% : 12,06 à 136,28) pour les chocs d'origine obstructive. Les valeurs réunies des rapports de vraisemblance négatifs variaient de 0,13 (IC à 95% : 0,04 à 0,48) pour les chocs d'origine obstructive à 0,32 (IC à 95% : 0,16 à 0,62) pour les chocs d'origine mixte.

Conclusion

L'EPS pratiquée d'urgence dans les cas de choc et d'hypotension donne de meilleurs résultats lorsqu'il s'agit d'inclure des causes d’état de choc que lorsqu'il s'agit d’écarter définitivement des causes particulières. D'après les rapports de vraisemblance négatifs répartis en sous-catégorie, l'examen permettrait d’écarter avec une grande exactitude les états de choc d'origine obstructive.

CLINICIAN'S CAPSULE

What is known about the topic?

The rapid ultrasound for shock and hypotension (RUSH) exam has been suggested to help diagnose the etiology of undifferentiated shock.

What did this study ask?

What does the literature say regarding the diagnostic accuracy of the RUSH exam for shock etiology by subtype?

What did this study find?

The RUSH exam is better able to “rule in” than “rule out” most shock subtypes.

Why does this study matter to clinicians?

Diagnostic accuracy for shock evaluation can be improved with use of an ultrasound protocol, but additional clinical information should still be used.

INTRODUCTION

Shock is a state of severe metabolic and circulatory dysfunction resulting in inadequate tissue oxygenation and perfusion.¹ Shock etiology can be categorized into four categories: hypovolemic/hemorrhagic, distributive, cardiogenic, and obstructive. “Undifferentiated shock” denotes a shock state of an unclear source, a common initial presentation of shock in the emergency department (ED).¹

Point-of-care ultrasound (POCUS) can diagnose various potential causes of shock.^2–8 In one study, early multi-organ POCUS correctly classified the final diagnosis in 80% of cases of shock, offering support that multi-organ POCUS can help diagnose the cause of shock, and therefore, expedite treatment decisions.⁹

Numerous multi-organ POCUS protocols exist, the rapid ultrasound for shock and hypotension (RUSH) exam being amongst the comprehensive.^10–21 The RUSH exam involves assessments of the heart, lungs, inferior vena cava (IVC), peritoneum, abdominal aorta, and possibly lower extremity veins.^{16, 19} However, the diagnostic accuracy of the RUSH exam to distinguish different shock subtypes remains undefined. The purpose of this systematic review and meta-analysis is to evaluate the ability of the RUSH exam to diagnose the etiology of shock among patients with different shock subtypes presenting to the ED.

METHODS

Study design

This systematic review was registered at the Prospero registry of systematic reviews (CRD42016036033). The design and manuscript structure conforms to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement.²²

Search strategy

A trained medical librarian performed the literature search using Ovid MEDLINE (1966–June 13, 2018), Cochrane Central Register of Controlled Trials (1991–June 13, 2018), and Scopus (1996–June 13, 2018). The following major terms were searched in combination (full search strategy in Appendix Document 1): “rapid ultrasound in shock,” “rush protocol,” “rush exam,” “undifferentiated shock,” “undifferentiated hypotension,” “shock,” “hypotension,” “ultrasound,” “ultrasonography,” “echocardiography.” Results from all languages were included, with Google Translate (Mountain View, CA) used to translate non-English manuscripts.²³ Two reviewers (SPS and RG) searched abstracts accepted for presentation at national conferences and published in Annals of Emergency Medicine, Academic Emergency Medicine, Journal of Emergency Medicine, Journal of Ultrasound in Medicine, Critical Ultrasound Journal, CJEM, and Critical Care Medicine from 2000 through June 13, 2018. Authors of any potentially relevant articles were contacted for further information. Clinicaltrials.gov was searched for ongoing and completed trials.

Study selection

Two authors (SPS and RG) independently reviewed titles and abstracts generated by the literature search and selected relevant manuscripts for a full-text review. Diagnostic studies using the RUSH exam or an identical protocol for the evaluation of shock/hypotension in ED patients were considered for inclusion. An identical protocol was considered one that included POCUS evaluation of the heart (for size, strain, contractility, and tamponade), lungs (for pneumothorax, pulmonary edema, and pleural effusion), IVC (for size and collapsibility), peritoneum (for free fluid), and aorta (for aneurysm or dissection) for the evaluation shock/hypotension etiology. Evaluation of lower extremity veins for deep venous thrombosis (DVT) was not considered necessary for inclusion due to variable inclusion in the original RUSH exam publications.^{16, 19}

Table 1 defines the RUSH exam criteria for each shock subtype and associated clinical diagnosis (see also supplemental videos).^{16, 24} “Mixed-etiology” shock was defined by the authors. Any disagreements between reviewers for study inclusion were resolved by discussion.

Table 1. RUSH exam findings in shock subtypes

IVC = inferior vena cava.

Individual evidence quality appraisal

Two reviewers (SS and CK) independently used the Quality Assessment Tool for Diagnostic Accuracy Studies (QUADAS-2) for systematic reviews to evaluate the quality of evidence for identified studies.²⁵ The reviewers used several a priori conditions to evaluate risk of bias for individual studies and applicability to the PICO question:

Patient selection

Reviewers assessed whether a study's inclusion and exclusion criteria led investigators to evaluate patients with shock who were more or less acutely ill than those typically evaluated in ED settings, or those in whom an ultrasound evaluation may be easier or more challenging to acquire adequate images (e.g., body habitus). Such exclusions could introduce spectrum bias.²⁶

Index test

Reviewers assessed whether the criteria for shock subtype according to the RUSH exam were uniform and consistent with criteria stated in the source articles (see Table 1).

Reference standard

Because no single test is accepted to diagnose every etiology of shock, reviewers accepted a reference standard of chart review or blinded expert opinion/consensus.

Disagreements were discussed between SS and CK to achieve consensus.

Data collection

Two reviewers (SS and GU) independently abstracted data from included studies, including the title, publication year, enrolment period, sample sizes, sampling methods, inclusion and exclusion criteria, operators and equipment used, ultrasound protocol description, study test characteristics, and reference standard. Authors of included papers were contacted for additional data if all relevant data were not included in the original manuscript.

Data analysis

One reviewer (CRC) computed meta-analysis estimates when >1 study assessed the same findings on POCUS and were compared to a similar criterion standard. We generated combined estimates for diagnostic accuracy using a random-effects model (Meta-DiSc Hospital Universitario Ramón y Cajal, Madrid, Spain).^{27, 28} Interstudy heterogeneity was assessed using the Der-Simonian-Laird random effects model and the Index of Inconsistency (I²).^{29, 30} Pooled estimates of dichotomous positive (LR+) and negative (LR-) likelihood ratios were also reported from the random effects model. Publication bias was not assessed because this is not an accepted approach in diagnostic meta-analyses due to the small number of studies generally identified.³¹

Additionally, one study reported indeterminate data regarding POCUS results. Although not planned prior to commencing our review, we conducted a separate analysis to evaluate the potential statistical impact of these indeterminate cases (Appendix Document 2).

RESULTS

Search results

Our search identified 6,462 citations; 5,097 studies remained after the removal of duplicates (Figure 1). Nine studies met inclusion criteria. Four were included in our final analysis after determining that one study included data presented in a later publication,³² one study included a subset of data contained in a previous publication,³³ and another provided inadequate reference standard data.³⁴ Additionally, two studies were excluded following a QUADAS-2 assessment: one considered high risk for incorporation bias and imperfect gold standard bias, and another considered high risk for incorporation bias and double gold standard bias, both following author correspondence.³⁵ We excluded two conference abstracts due to insufficient data following unanswered requests for further information.^{36, 37} One trial was identified on clinicialtrials.gov for possible inclusion, but early results were not available.³⁸ All included studies were deemed to be of low overall risk of bias (Table 2). Although all included studies cited an original RUSH exam article in the description of the protocol performed,¹⁶ we received clarification from only one author group regarding what specific POCUS criteria were used to define each shock subtype. The reference standard was a medical chart review.

Figure 1. Search results flowchart.

Table 2. Included studies characteristics

ACES = abdominal and cardiac evaluation with sonography for shock; POCUS = point-of-care ultrasound;

RUSH, rapid ultrasound in shock and hypotension; SBP = systolic blood pressure.

Study characteristics

Table 3 displays study characteristics for the included studies. All were prospective trials involving convenience samples of ED patients with acute undifferentiated shock.

Table 3. Individual results of included studies by shock category

*The study by Ghane et al. included eight patients who did not have a final clinical diagnosis. These patients were removed from analysis to avoid differential verification bias.

#For the study by Shokoohi et al, obstructive shock only included cases attributed to pulmonary embolism. Cardiac tamponade was not included.

US+ Positive diagnosis by ultrasound.

US- Negative diagnosis by ultrasound.

US+/- Indeterminate diagnosis by ultrasound

Dx+ Positive diagnosis by gold standard

Dx- Negative diagnosis by gold standard

Risk of bias within studies

Table 4 displays the QUADAS-2 assessment results. All included studies were deemed to be of overall low risk for bias; we felt that the issue regarding criteria for RUSH exam interpretation was a theoretical concern, rather than a clear high risk of bias.

Table 4. QUADAS-2 results

Low risk High risk Unclear risk

Meta-analysis results

The graphical results of our meta-analysis are included in Appendix Figures 1–5. Table 5 shows the pooled sensitivity, specificity, LR+ and LR- by shock subtype. All of the meta-analysis results are based on four studies of 357 patients, except for obstructive, which was based on three studies of 239 patients, and mixed-etiology, which was based on three studies of 332 patients.

Table 5. Pooled sensitivities, specificities, and likelihood ratios by shock subtype

Hypovolemic

Pooled LR+ and LR- were 8.25 (95% CI 3.29–20.69) and 0.19 (95% CI 0.07–0.50), respectively. Statistical heterogeneity (I²) was high (near or above 70%) for all test characteristics calculated.

Cardiogenic

Pooled LR+ and LR- were 24.14 (95% CI 12.43–46.86) and 0.24 (95% CI 0.12–0.49), respectively. For sensitivity, I² was high (near or above 70%). For specificity and positive likelihood ratio, I² was 0%.

Obstructive

Data from the Bagheri-Hairiri et al. study were not included in the meta-analysis of obstructive shock because there were no cases of obstructive shock in their study.³⁹ Additionally, only cases of massive pulmonary embolism as causes of obstructive shock were noted in the Shokoohi et al. study.⁴⁰ Pooled LR+ and LR- were 40.54 (95% CI 12.06–136.28) and 0.13 (95% CI 0.04–0.48), respectively. I² for specificity and LR+ were moderate at 59.8% and 25.6%, respectively. I² for sensitivity and LR- were 0%.

Distributive

Pooled LR+ and LR- were 17.56 (95% CI 3.46–86.19) and 0.30 (95% CI 0.11–0.79), respectively. I² for all test characteristics were high (above 70%).

Mixed

Data from the study by Shokoohi et al. were not included in the meta-analysis of mixed etiology shock because there were no specifically documented cases of that category.⁴⁰ Pooled LR+ and LR- were 12.91 (95% CI 0.84–198.84) and 0.32 (95% CI 0.16–0.62), respectively. I² for specificity and LR+ were high (both above 85%), whereas I² was 0% for sensitivity and LR-.

Indeterminate studies

One study was unique in that it included data on indeterminate RUSH exams.⁴¹ We sought to evaluate the impact that this may have on the results. There were eight cases (10%) where no final clinical diagnosis was reported and five cases where the ultrasound diagnosis was indeterminate. The eight cases without reference standard data were removed from our analysis. Table 3 includes our 3 x 2 cell matrix; Appendix Document 2 includes all calculated data when factoring the potential impact of indeterminate results. Overall, worst case LR+ ranged from 7.4 (mixed) to 11 (cardiogenic), and best case LR+ ranged from 26.5 (hypovolemic) to 50.9 (obstructive). Worst case LR- ranged from 0.4 (mixed) to 0 (hypovolemic), and best case LR- ranged from 0.28 (mixed) to 0 (hypovolemic and cardiogenic).

DISCUSSION

Summary of evidence

With this systematic review, we sought to evaluate the accuracy with which the RUSH exam performed by ED providers was able to diagnose the etiology of shock by subtype.

A comparison of LR+ across shock subtypes reveals that the RUSH exam is most accurate for obstructive shock and least accurate for mixed-etiology shock. However , there were very few cases of obstructive shock overall, and no cases of cardiac tamponade in the included studies. Also, the wide confidence intervals for hypovolemic, distributive, and mixed-etiology shock suggest that further data are needed to understand the true value of the RUSH exam for these etiologies. The high specificity values in our analysis of the RUSH exam support that a positive finding when performing the protocol may yield clinically useful information for all shock subtypes except mixed.

The pooled LR- for obstructive shock was also useful to rule out these etiologies. On the other hand, the pooled LR- for cardiogenic, distributive, hypovolemic, and mixed-etiology shock is potentially inadequate when used alone to exclude these etiologies, suggesting that additional clinical judgment is necessary.

Our search found few studies directly evaluating the diagnostic accuracy of the RUSH exam. The paucity of RUSH exam diagnostic accuracy research may reflect a lack of awareness of the protocol or lack of experience with components of the protocol amongst clinicians and researchers. Lack of buy-in and poor integration of POCUS in clinical practice may be another barrier, as suggested by a recent analysis of billing data in the United States noting only 0.7% of emergency medicine practitioners received reimbursement for ED POCUS exams.⁴² Whether this reflects simply low billing rates or low utilization is unclear, but it seems plausible that poor clinical integration may affect research endeavours. Additionally, because a relatively large number of multi-organ POCUS protocols have been proposed fairly recently (Appendix Table 1), a criticism may be in order that the POCUS community has been more focused on proposing protocols rather than testing and refining those that already exist.^{10, 13, 15–17, 19–21}

Our analysis found that in the single, small study that included indeterminate data, inclusion or exclusion of indeterminate studies can have a marked effect on the results – for example, the worst case LR+ for mixed etiology shock was 7.4, but the best case was 42.8, which changes the impact of the study from being strongly suggestive to clinching the diagnosis. We are unaware of studies evaluating the frequency with which indeterminate POCUS exams occur. POCUS researchers should include data on indeterminate results to better characterize their analyses.

Our review also highlights the need for refined POCUS research for shock. Existing and future multi-organ POCUS protocols should be compared head-to-head to determine which provides the most critical information in the most pragmatic and expeditious fashion when used by typical (not ultrasound expert) ED providers.^{10, 13, 15–17, 19–21} One group has attempted to devise an optimal multi-organ POCUS protocol for shock via a preliminary study and subsequent consensus conference statement, but this has not been compared with existing protocols to prove superiority.^{43, 44} Additionally, although clinicians and researchers may equate diagnostic accuracy with improved patient care, high test accuracy does not necessarily lead to better patient outcomes. Atkinson et al. sought to evaluate the impact of multi-organ POCUS on 30-day mortality or discharge for patients presenting to the ED with undifferentiated shock.⁴⁵ Their analysis noted no significant difference in the primary end point, with a mortality rate of around 24% in the POCUS and control groups. Further research into the patient-centric value that POCUS brings to shock evaluation should be considered, particularly regarding any potential clinical decision rule derivation and validation studies, so that the additive value of the imaging beyond other bedside tests (e.g., history, physical exam, and labs) can be fully evaluated.⁴⁶

This review demonstrates that the RUSH exam can aid in the identification of the category of undifferentiated shock subtypes, which can help guide further decision-making. However, used in isolation, the RUSH exam is imperfect, particularly in excluding any subtype besides obstructive, potentially leading to an incomplete or incorrect diagnosis. The RUSH exam is best used as one component in a complete evaluation of a hypotensive patient, rather than the sole determinant for decision-making.

Limitations

There are several limitations to our review. In this study, we accepted a diagnosis of “mixed” as its own group, without explicit identification of which comprised the diagnosis. Therefore, our data may be interpreted as a best-case scenario for the ability of the RUSH exam to diagnose mixed-etiology shock. A less biased method to quantify diagnostic accuracy would be for researchers to define what shock types contribute to the “mixed” category and compare those with an agreed upon reference standard.

The sample sizes for several studies were small, and the overall number of patients was low. Only one of the included studies reported an a priori sample size estimate, and only one followed Standards for Reporting Diagnostic Accuracy (STARD) criteria,⁴⁷ which limits future investigators’ ability to reproduce researchers’ methods.^{48, 49} Methods for estimating appropriate diagnostic sample sizes exist and should prevent underpowered studies.⁵⁰ Additionally, not all causes of shock were equally represented. In particular, obstructive causes of shock were the least represented in the analysis, with no cases of tamponade. While this may reflect the relatively low incidence of this category of shock, it may also impair the ability to make judgments on the ability of the RUSH exam to evaluate its representative pathologies equally.

We accepted the medical chart review as an adequate reference standard to compare with RUSH exam diagnoses. While this would seem to hold some face (clinical) validity, it is difficult to determine whether medical chart reviewers were using the same criteria for each subtype diagnosis. Because no suitable comparative diagnostic tool exists across etiologies, we felt this was a reasonable reference standard to use, but readers should keep this limitation in mind.

Heterogeneity (I²) was fairly high across several statistical parameters for most subtypes. This may be due in part to low sample sizes, differences in RUSH exam interpretation or performance (e.g., in one study clinical, data may have played a role in RUSH exam interpretation; for all studies it is unknown what the providers’ previous experience was with RUSH exam interpretation beyond subjective descriptions), or differences in study populations (e.g., no mixed category in one study, few obstructive cases overall). Further studies would be aided by appropriately powered sample sizes and rigorous standards for RUSH exam interpretation.

Although all studies reported those performing the RUSH exams to have been familiar with the protocol, none included data on inter-rater reliability or intra-rater reliability. This limits the ability to evaluate procedural competence and its effect on the reported results. Whether the reported results reflect the value of the RUSH exam when performed perfectly, by an “average” clinical ultrasonographer, or by a novice, is unclear.

Few authors fulfilled our requests for the data sheets used to determine RUSH exam diagnoses. We were unable to compare whether the same criteria for index test diagnosis were used across studies, raising concern for possible differential verification bias, which may artificially improve statistical measures of diagnostic accuracy.²⁶

We did not include DVT evaluation as necessary for inclusion. This was decided upon because the original description of the RUSH exam by Weingart et al. did not include DVT evaluation, although subsequent versions varied.^{16, 19, 51} Rather than appear to favour one version over the other, we chose to be as comprehensive as possible and viewed the inclusion of DVT evaluation as optional. Our results should therefore be interpreted with this in mind, although we feel that obstructive shock due to pulmonary embolism would most likely have other ultrasonographic findings of obstructive shock besides DVT.³ Therefore, we do not feel strongly that this should affect the impact of our analysis.

CONCLUSION

Our review suggests that the RUSH exam performs better when used to confirm suspected causes of shock, rather than to definitively exclude specific etiologies. The LR- values of the RUSH exam by subtype suggest that it most accurately rules out obstructive shock. Future research should evaluate the comparative accuracy and real-world acceptability of the various multi-organ POCUS protocols for the evaluation of undifferentiated shock, as well as evaluate patient-oriented outcomes in regard to POCUS evaluation for shock patients.