Development and validation of a simple risk score for diagnosing COVID-19 in the emergency room

Abstract

As the COVID-19 pandemic continues to escalate and place pressure on hospital system resources, a proper screening and risk stratification score is essential. We aimed to develop a risk score to identify patients with increased risk of COVID-19, allowing proper identification and allocation of limited resources. A retrospective study was conducted of 338 patients who were admitted to the hospital from the emergency room to regular floors and tested for COVID-19 at an acute care hospital in the Metropolitan Washington D.C. area. The dataset was split into development and validation sets with a ratio of 6:4. Demographics, presenting symptoms, sick contact, triage vital signs, initial laboratory and chest X-ray results were analysed to develop a prediction model for COVID-19 diagnosis. Multivariable logistic regression was performed in a stepwise fashion to develop a prediction model, and a scoring system was created based on the coefficients of the final model. Among 338 patients admitted to the hospital from the emergency room, 136 (40.2%) patients tested positive for COVID-19 and 202 (59.8%) patients tested negative. Sick contact with suspected or confirmed COVID-19 case (3 points), nursing facility residence (3 points), constitutional symptom (1 point), respiratory symptom (1 point), gastrointestinal symptom (1 point), obesity (1 point), hypoxia at triage (1 point) and leucocytosis (−1 point) were included in the prediction score. A risk score for COVID-19 diagnosis achieved area under the receiver operating characteristic curve of 0.87 (95% confidence interval (CI) 0.82–0.92) in the development dataset and 0.85 (95% CI 0.78–0.92) in the validation dataset. A risk prediction score for COVID-19 can be used as a supplemental tool to assist clinical decision to triage, test and quarantine patients admitted to the hospital from the emergency room.

Introduction

In December of 2019, an outbreak of a novel coronavirus disease was reported in the Hubei Province of China. Caused by the emerging virus Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), the disease has quickly spread across the world. On 30 January 2020, the World Health Organization (WHO) declared the outbreak a Public Health Emergency of International Concern, and on 11 March 2020, declared it a pandemic [1]. As of 11 October 2020, there were over 37.0 million confirmed cases worldwide, with 7.7 million being in the USA [2].

The pandemic has caused significant adverse impacts throughout the USA and the world. Since the early days of the COVID-19 pandemic in the USA, hospital systems have found themselves overwhelmed and with limitations in the capacity to triage, diagnose and treat patients afflicted by COVID-19. As the pandemic continues unabated and as it spreads through the USA, it is necessary to improve hospital screening and stratification of at-risk populations to enable timely and appropriate quarantine, treatment and use of limited resources.

At present, no validated risk score or stratification system is readily available to aid the clinical decision-making process of hospital-based staff in determining when testing for COVID-19 is appropriate [3]. The availability of testing for COVID-19 continues to be an ongoing limitation throughout the USA. A system of clinical risk stratification can help to identify patients that present a higher risk and warrant COVID-19 testing in a resource-limited setting.

In this retrospective study, we reviewed the records of patients presenting to an emergency department in an acute care hospital in the Metropolitan Washington D.C. area who were tested for SARS-CoV-2 and admitted. We reviewed the clinical characteristics, radiographic findings and laboratory findings between those who tested positive and negative, then developed a simple bedside risk prediction scoring system.

Methods

Cohort design and subjects

A retrospective review was performed for patients tested for COVID-19 and admitted to MedStar Southern Maryland Hospital, a 262-bed acute care hospital located in a suburb of Washington D.C. between 1 April 2020 and 30 April 2020. During this time, the hospital and surrounding region experienced a surge of COVID-19 admissions, but universal testing for COVID-19 was not performed for hospitalised patients. Patients were included in the study if they presented to the emergency room and were admitted to the hospital with laboratory-confirmed COVID-19 (cases) or tested negative for COVID-19 within 24 h of hospital admission (controls). All COVID-19 diagnosis was made by nasopharyngeal swab and reverse transcription polymerase chain reaction for SARS-CoV-2. Patients who were admitted from the emergency room directly to the intensive care unit (ICU) were excluded from the study. These patients were excluded as a majority of them were not in a condition to describe their symptoms at the time of the presentation, and the severity of their illnesses often necessitated COVID-19 testing during empiric work up.

Data collection

For all eligible patients, records from initial hospital encounter were reviewed. Demographics (age, sex, race and smoking status), past medical history (diabetes, hypertension, chronic obstructive pulmonary disease (COPD), asthma, coronary artery disease (CAD), congestive heart failure (CHF), atrial fibrillation, chronic kidney disease (CKD) and end-stage renal disease (ESRD)), sick contact with suspected or confirmed COVID-19 case, presenting symptoms (fever, chills, myalgia, cough, shortness of breath, nausea, vomiting or diarrhoea), triage vital signs (temperature, heart rate, systolic and diastolic blood pressure, respiratory rate and oxygen saturation), initial basic laboratory test results (complete blood count and creatinine) and chest X-ray results were collected.

Development of prediction model

The dataset was randomly split into a development cohort and a validation cohort with a 6:4 ratio. Baseline characteristics were compared between cases and controls within each cohort. Categorical variables were compared using the χ ² test and continuous variables were compared using the Student t test. Univariable logistic regressions were performed in a development cohort to identify potential predictors of COVID-19 status. A multivariable logistic regression model was built in a stepwise fashion by a parsimonious approach. Variables associated with COVID-19 status in univariable analysis (P-value < 0.1) entered the model one by one in the order of their strength of association with COVID-19 diagnosis. Variables retained in the model if the addition of the variable improved the fit of the model (P-value < 0.05). Then we developed risk score models using the coefficients from the final logistic regression model. A total of three risk score models were created. The first model rounded the coefficient from the regression model and used them as a score for each variable. The second and third models simplified the coefficients to make it easy and practical to use. The risk scores were validated in the testing cohort. The area under the receiver operating characteristic curve (AUROC) was calculated for each risk score. The analyses were performed using STATA version 15.1 (STATA Corp., Texas, USA).

Ethical consideration

This study was approved by the Institutional Review Board (IRB) of the MedStar Health Research Institute with a waiver of individual consents. (IRB ID: MOD00004296)

Results

Study population

A total of 656 patients were admitted to the hospital during the study period. Of them, 79 patients who were admitted to the ICU directly from the emergency room were excluded from the study. Among 577 patients admitted to the medical floor, 338 patients received testing for COVID-19 and were included in this study. Of those included, 136 (40.2%) patients tested positive for SARS-CoV-2, and 202 (59.8%) patients tested negative. The demographic characteristics of the patients were described in Table 1. In the entire cohort, the median age was 65 years old (interquartile range (IQR) 54–76 years old), 52.7% were males, 82.8% were African Americans, 10.1% were Hispanics and 5.6% were Caucasians, 30.5% were current or former smokers, 14.2% were from skilled nursing facilities, 42.9% were obese, 64.5% had hypertension, 38.5% had diabetes, 21.0% had CKD, 16.0% had CHF and 12.7% had COPD.

Table 1. Baseline characteristics of the patients admitted to the hospital in a development and a validation cohort

^aData are presented as median (IQR) for a continuous variable, and n (%) for categorical variables.

The dataset was split into development and validation datasets. In total, 203 patients were assigned to the development cohort, and among them, 87 (42.9%) patients were tested positive for COVID-19. In total, 135 patients were assigned to the validation cohort, and among them, 49 (36.3%) were tested positive. Baseline demographics and clinical characteristics of each cohort are summarised in Table 1 and Table 2. Nursing facility residence were more common in COVID-19 admissions as compared to non-COVID-19 admissions in both development cohort (P-value = 0.093) and validation cohort (P-value = 0.021). Diabetes, hypertension and COPD were more common in non-COVID-19 admissions as compared to COVID-19 admissions in both cohorts, but these differences were not statistically significant (P-value > 0.05).

Table 2. Clinical presentation and initial work-up results in the emergency room

Eight variables were included in the final multivariable model. Sick contact and nursing facility residence were the two biggest risk factors for COVID-19, followed by respiratory symptom (cough or shortness of breath), gastrointestinal symptom (nausea, vomiting or diarrhoea), hypoxia at triage, obesity and constitutional symptom (fever, chills or myalgia). Leucocytosis was negatively associated with COVID-19 (Table 3). Based on the coefficients from the final model, three risk scores were created (Table 4). The first model used the rounded value of the coefficients as a score for each variable. Sick contact and nursing facility residence were assigned 10 points each, and obesity was assigned 3 points. The constitutional symptom was assigned 2 points, and respiratory symptom, gastrointestinal symptom and hypoxia on triage were assigned 4 points each. Leucocytosis was assigned minus 3 points. This risk score achieved AUROC of 0.87 (95% CI 0.83–0.92) in the development cohort and 0.85 (95% CI 0.78–0.92) in the validation cohort. Simplified risk scores were created for practical use (models 2 and 3). For the second model, our simplest model, sick contact and nursing facility residence were assigned 2 points each; obesity, respiratory symptom, gastrointestinal symptom, constitutional symptom and hypoxia at triage were assigned 1 point each; and leucocytosis was assigned minus 1 point. The second model achieved AUROC of 0.87 (95% CI 0.83–0.92) in the development dataset and 0.83 (95% CI 0.76–0.90) in the validation dataset. Our final model, model 3, modified model 2 by assigning 3 points each to sick contact and nursing facility residence, and it achieved AUROC of 0.87 (95% CI 0.82–0.92) in the development dataset and 0.85 (95% CI 0.78–0.92) in the validation dataset. The receiver operating characteristic curve for the final model, model 3, is presented in Figure 1. Using the final model, a risk score of ⩾3 achieved a sensitivity of 85.1% and a specificity of 75.0% in the development cohort, and a sensitivity of 79.6% and a specificity of 70.9% in the validation cohort. Positive and negative predictive values were 71.8% and 87.0% in the development cohort and 60.9% and 85.9% in the validation cohort, respectively. Sensitivities, specificities and positive and negative predictive values for different cut-off values are presented in Table 5.

Fig. 1. Receiver operating characteristic curves of the COVID-19 risk score (model 3) among patients admitted to the hospital from the emergency room in a development cohort (left) and a validation cohort (right).

Table 3. Results of univariable analysis and multivariable analysis

Table 4. COVID-19 risk scores and corresponding AUROCs

Table 5. Sensitivities, specificities, positive predictive values (PPVs) and negative predictive value (NPVs) of the final risk score system for each prediction score cut-off

Discussion

Early and proper identification and isolation of suspected patients with COVID-19 are essential to allow timely treatment, conserve resources, protect patients and healthcare staff, and to avoid the spread of COVID-19 in healthcare facilities [4]. In a resource-limited setting, it is critically important to risk-stratify patients who need testing. In this study, we present a novel bedside score developed to aid the diagnosis of COVID-19. Prior studies suggested a few logistic regression models predicting COVID-19 diagnosis [3, 5, 6]. However, logistic models are not practical to implement in clinical practice due to complex mathematical calculations needed to perform the prediction. We simplified our prediction model by creating a scoring system that is simple and practical for use and internally validated its utility. Our simplified final risk score (model 3) achieved AUROC comparable to the first model that used rounded values of coefficients from the regression model as scores, highlighting the utility of a simple bedside score. We only included variables that are readily available at the initial hospital encounter to enable practical implementation.

In our analysis, sick contact with suspected or confirmed COVID-19 cases and nursing facility residence were found to be two major risk factors for COVID-19 among newly admitted patients, and they were assigned 3 points each in the final model. Nursing facility residents are a high-risk group for COVID-19. They reside in a healthcare setting with an increased risk of contracting the virus, and they are often elderly with multiple underlying medical conditions and therefore at increased risk of worse outcomes from COVID-19. Thus, active screening of this high-risk group is critical. Using a cut-off of three in our final model will allow screening all patients admitted to the hospital from nursing facilities and those who had sick contact with presumed COVID-19 cases. Male sex and chronic medical conditions such as diabetes and hypertension were known to be associated with the worse outcome from COVID-19 [7–10]. However, in this study, we did not find significant differences in the proportion of males, hypertensives and diabetics between COVID-19-positive admissions and negative admissions. Interestingly, hypertension and diabetes were more common in non-COVID-19 admissions than in COVID-19 admissions in both development and validation cohorts. This is likely because our control group was patients who were admitted to the hospital with non-COVID-19 causes, and hypertension and diabetes are important risk factors for hospitalisations from non-COVID-19 reasons such as cardiovascular conditions. Obesity is another risk factor for worse clinical outcomes in COVID-19 [9, 10]. In this study, newly admitted patients with obesity were more likely to have COVID-19. In addition to demographic risk factors and symptom scores, we also identified that patients with leucocytosis were less likely to have COVID-19. While COVID-19 is known to be associated with leukopenia and thrombocytopenia, they were not independent predictors of COVID-19 diagnosis in our cohort [11, 12].

In summary, the indicators used in this risk score stratification are readily available at the time of admission into hospitals throughout the nation. The ability to quickly and appropriately risk stratify and identify suspected patients requiring quarantine and testing may allow physicians to make appropriate decisions in terms of early diagnosis and management.

Nonetheless, our study has limitations. Our study is limited by small cohort size. In this study, we did not find a significant association between chest X-ray findings and COVID-19 status after adjusting for the effects of confounders. This may be due to our small sample size; however, as our study excluded patients who were directly admitted to the ICU from the emergency room, chest X-ray may not be a very strong predictor of COVID-19 among newly hospitalised patients who are not critically ill. Inflammatory markers such as d-dimer, C-reactive protein and ferritin were reported to be often elevated in COVID-19 but these lab values were not available for many study patients and therefore not included in the model [13, 14]. As our scoring system was developed in a cohort of patients admitted to the regular medical floor from the emergency room, and our study result cannot be generalised to other settings, such as outpatient practices, urgent cares or ICUs. SARS-CoV-2 is also known to cause asymptomatic infection, and our score system is designed to risk-stratify newly admitted patients with symptoms concerning for COVID-19 infection, therefore cannot be used to identify asymptomatic patients. Given the above limitations and single-centre study design, the risk score should be further validated in larger and/or multicentre studies.

Conclusion

In summary, we developed a simple, easy-to-implement bedside scoring system for COVID-19 risk stratification among patients who are being admitted to the hospital. The risk score system achieved AUROC of 0.85 in validation and can be used as a supplemental tool to assist clinical decision in the triage, quarantine and testing of patients admitted to the hospital with suspicion of COVID-19 infection.