Validation of clinical risk tools for recurrent Clostridioides difficile infection

Abstract

Objective:

We sought to validate available tools for predicting recurrent C. difficile infection (CDI) including recurrence risk scores (by Larrainzar-Coghen, Reveles, D’Agostino, Cobo, and Eyre et al) alongside consensus guidelines risk criteria, the leading severity score (ATLAS), and PCR cycle threshold (as marker of fecal organism burden) using electronic medical records.

Design:

Retrospective cohort study validating previously described tools.

Setting:

Tertiary care academic hospital.

Patients:

Hospitalized adult patients with CDI at University of Virginia Medical Center.

Methods:

Risk scores were calculated within ±48 hours of index CDI diagnosis using a large retrospective cohort of 1,519 inpatient infections spanning 7 years and compared using area under the receiver operating characteristic curve (AUROC) and the DeLong test. Recurrent CDI events (defined as a repeat positive test or symptom relapse within 60 days requiring retreatment) were confirmed by clinician chart review.

Results:

Reveles et al tool achieved the highest AUROC of 0.523 (and 0.537 among a subcohort of 1,230 patients with their first occurrence of CDI), which was not substantially better than other tools including the current IDSA/SHEA C. difficile guidelines or PCR cycle threshold (AUROC: 0.564), regardless of prior infection history.

Conclusions:

All tools performed poorly for predicting recurrent C. difficile infection (AUROC range: 0.488–0.564), especially among patients with a prior history of infection (AUROC range: 0.436–0.591). Future studies may benefit from considering novel biomarkers and/or higher-dimensional models that could augment or replace existing tools that underperform.

Introduction

Clostridioides difficile is a Gram positive, spore forming, toxin producing bacterium which can colonize and infect the human intestinal track causing a robust immune response. ¹ Unlike most other infectious diseases, C. difficile infection (CDI) tends to relapse and re-infect despite antibiotic treatment and remission of symptoms. C. difficile spores are capable of surviving treatment, resulting in persistent carriage, serving as a nidus for recurrent infection. Most recurrent CDI episodes occur within 2–8 weeks of the initial infection and are due to the previous strain. ² Newer anti-CDI therapies (ie, fidaxomicin, ³ bezlotoxumab, ⁴ and fecal microbiota transplant ⁵ ) effectively prevent recurrent CDI and as of April 2023, ⁶ the first orally administered fecal microbiota product was approved for the prevention of recurrent CDI by the US Food and Drug Administration. However, newer therapies to prevent recurrent CDI have historically been underutilized, ⁷ despite adoption by recent CDI consensus guidelines. ⁸ This is likely due to two major issues with CDI treatments: first, these treatments have a high cost and logistical challenges, limiting their use to patients believed to be most at risk of recurrence, ^9,10 ; second, determining which patients are likely to recur is difficult without a widely accepted risk prediction model for recurrent C. difficile infection.

Several published outcome models (Table 1) for risk stratifying patients with CDI for developing future recurrent infection have been developed retrospectively or by using previous clinical trial data. Few recurrent CDI models have been externally validated and limited evidence suggest that they generalize poorly between centers. In addition, recently updated clinical management guidelines for C. difficile infection mention specific risk factors for recurrence, however, the performance of these features together as a risk stratification tool has not been externally evaluated. ⁸ We were also interested in comparing C. difficile PCR cycle threshold (as an inverse measure of fecal organism burden), which we recently showed may be a potentially useful biomarker for recurrent infection ²¹ alongside existing clinical models. Here, we validate and compare the relative performance of the leading recurrence risk models available for hospitalized cases of C. difficile infection.

Table 1. Clinical risk models for predicting recurrent C. difficile infection

C. difficile infection (CDI), methicillin-resistant S. aureus (MRSA), proton pump inhibitor (PPI), white blood cell (WBC)

Methods

Study population

A previously described ^22,23 retrospective cohort of hospitalized adult patients with C. difficile infection at University of Virginia Medical Center (a 645-bed, tertiary care academic hospital) was used for this analysis. Hospitalized CDI cases were enrolled with a positive C. difficile polymerase chain reaction (PCR; GeneXpert®; Cepheid, Sunnyvale, CA) result and anti-C. difficile treatment. Children <18 years and episodes that were not treated with anti-C. difficile therapy (oral vancomycin, fidaxomicin, and/or IV/oral metronidazole) were excluded. Recurrent C. difficile infection was defined as the presence of a repeat positive PCR test available within 11–60 days post-infection or symptom relapse requiring retreatment/initial treatment extension for CDI within 60 days of index diagnosis. Patients without electronically available follow-up positive testing underwent additional clinician chart review (E.L., including clinician notes) to determine reports of positive C. difficile testing from outside laboratories and/or CDI symptom recurrence with retreatment/initial treatment extension. This study received approval from the University of Virginia Institutional Review Board (#20082).

Data collection/risk score calculation

To identify risk models for recurrent CDI, a literature search using medical literature databases (ie, PubMed, Google Scholar) and the bibliographies of relevant manuscripts was conducted. Manuscripts were evaluated for relevance; specifically, studies which generated clinically relevant tools for predicting recurrent CDI were of interest. So that each model could be compared in later analyses, the models had to include parameters that could be reliably gathered or imputed from the electronic medical record, and >3 ordinal scores to fit to an ROC curve. Criteria used to exclude studies from further analysis included highly specific study populations (ie, children, transplant patients), single binary predictors of recurrence, multivariable logistic regression models without a clinically applicable prediction tool (ie, integer-based scoring or similar systems by which a clinician could easily estimate relative risk), use of non-clinically accessible features, and meta-analysis and reviews. 7 models were identified which fulfilled the criteria. ^11,13–18 Of these, 2 models were excluded due to inability to reproduce with our cohort (ie, Hu et al model included Horn Index, a subjective measure which was not among the cohort’s available features) ¹⁷ or a lack of sufficient details for a scoring system that could be easily imputed from the electronic medical record (ie, Zilberberg et al). ¹⁸ Two models, Cobo and D’Agostino, utilized definitions of recurrence (4 months and 28 days, respectively) than other models. In this study, 60 days was utilized for all models to remain consistent in analysis. In addition to the identified models, the previously established ATLAS score for predicting severe CDI was included as a control for models specific to recurrence vs general disease severity (Table 1). ²⁰

Clinical data were gathered using the University of Virginia Clinical Data Warehouse, a database containing billing/coding, clinical, pharmacy, and laboratory data. Baseline features were defined as the closest available measurement within ±48 hours of C. difficile infection diagnosis (defined by the date/time of index positive C. difficile PCR specimen collection). For cases with multiple available laboratory measurements, maximum white blood cell count (WBC), creatinine, and minimum albumin measurements were used. For the Eyre model, ¹⁸ all patients received 1 point for ≥3 unformed stools/day as this was a criterion for testing based on UVA diagnostic stewardship practices. ²⁴ Past gastroenterology admission and C-reactive protein measurements (Eyre score) and presence/absence ≥10 unformed bowel movements/day (D’Agostino) could not be reliably collected. These were imputed as 0 added points when missing. For the IDSA/SHEA Recurrence risk stratification, one point was empirically assigned for each risk factor for recurrent CDI from the Updated 2017 IDSA/SHEA Guidelines. IDSA/SHEA Guidelines include age ≥65 years, a recurrent CDI episode within the last 6 months, immunocompromised host, and severe CDI on presentation which included the measures of white blood cell count ≥15,000 cells/ml, serum creatinine level >1.5mg/dl, or signs of fulminancy (hypotension, shock, ileus, or megacolon). ^8,25

An immunocompromised host was defined as active (administered during hospitalization) receipt of immunosuppressant medications (≥60 mg oral daily prednisone or equivalent systemic corticosteroid, azathioprine, rapamycin derivatives, cyclosporine, tacrolimus, or mycophenolate) and/or chemotherapy (Supplementary Table 1). ¹⁹ Antimotility medications were defined as loperamide, diphenoxylate, oral opium, or bismuth subsalicylate (receipt within 7 days preceding CDI diagnosis).

Data analysis

The area under the receiver operating characteristic curve (AUROC) for each model were calculated from these score-specific diagnostic test summary indices. The Youden Index (sensitivity + specificity -1) was calculated as an overall measure of diagnostic effectiveness and as one method to identify optimal cutoffs that balance sensitivity and specificity.

DeLong’s test of variance was used to calculate two-sided statistical comparisons of the highest performing model AUROC against each of the others. This was done using correlated AUROC curves. The significance of the AUROC was calculated using only using the patients with complete data for both models being compared. Analyses were performed using statistical software R, version 4.2.3 (R Core Team, Vienna, Austria) and R packages: dpylr, ²⁶ comorbidity, ²⁷ ROCit, ²⁸ pROC, ²⁹ and PRROC. ³⁰

Results

Study population

1,519 hospitalized cases of C. difficile infection among 1,302 individuals were identified between January 2014 and April 2021 with available PCR cycle threshold, white blood cell count, and creatinine measurements to calculate risk scores. Baseline characteristics are shown in Table 2.

Table 2. Study population baseline characteristics

Recurrent infection

157 recurrent CDI (rCDI) events were identified based on a repeat positive C. difficile test alone, using the UVA Health laboratory records. Clinical chart review performed for the remaining 1,362 cases found an additional 50 rCDI events with repeat positive C. difficile tests (either from the Care Everywhere^TM electronic medical record-sharing feature in Epic^TM or textually reported from an outside laboratory within clinician notes), and 138 rCDI events identified based on symptom relapse following anti-CDI treatment requiring retreatment without retesting. No rCDI cases were identified with negative PCR testing. In total, 345/1,519 (22.7%) patients were found to have recurrent CDI within 60 days.

1,211/1,519 (79.7%) of patients had at least one IDSA-defined risk factor for recurrent infection, all of which (recurrent CDI episode within the last 6 months, immunocompromised host, severe CDI on presentation) except one (age ≥65) was more likely among cases with subsequent recurrence. PCR Cycle threshold (Ct) among patients who went on to develop recurrent infection was lower (ie, higher stool organism burden) compared to patients without recurrence (mean PCR Ct 25.7 recurrence vs 26.8 no recurrence; t-test P = .0003). A frequency table of score distributions is shown in Table 3 and Supplementary Table 2.

Table 3. Frequency table for recurrent CDI score distributions

Top: The number of patients were stratified by either numerical, integer-based scores, probability of recurrence according to the D’Agostino Model (in bold), or cycles for each of the recurrence models. Bottom: The mean of value for each model in the populations with no recurrence, recurrence, or overall population. Supplementary Table 2 includes the median.

Clinical prediction tools

AUROCs predicting recurrence of all tools are shown in Figure 1 for all patients (1A), patients with no previous recurrence (1B), and patients with 1 or more previous recurrent episodes (1C). An AUROC of 0.500 represents the prediction ability of random guessing. Each model performed poorly in this validation with the best performing model, Reveles et al, having an AUROC of 0.523, 95% CI [0.490, 0.555] when all occurrences were considered. The IDSA guideline recurrence risk criteria (AUROC of 0.520) performed as well as the trained models and 75/345 (21.7%) of patients who developed recurrent CDI had no IDSA-defined recurrence risk factors. The ATLAS score, designed to predict severe outcomes and not recurrence, had a low AUROC of 0.494. The two models which had been previously validated, Cobo et al and Larrainzar-Coghen et al, performed worse in our validation than in the derivation cohort but similar to previous validation study (Table 1).

Figure 1. Receiver operator curves with area under the receiver operating characteristic curve (AUROC) for C. difficile risk models with the full cohort (1,519 cases) (A), patients with no previous recurrence (1,230 cases) (B), and patients with 1 or more previous recurrent episodes (289 cases) (C). Note: Data was unavailable for some patients. See Table 3 and Supplementary Table 2 for the stratification of the data including the number of missing data points for each model.

PCR Ct had a higher AUROC (0.564) compared to any of the clinical tools, including Reveles et al (DeLong’s test of AUROC difference P = .082), and IDSA (DeLong’s test of AUROC difference P = .052), but AUROC differences were not statistically significant. Median PCR Ct was 25.3 and the positive/negative predictive values at the 1st, 2nd, and 3rd quartile cutoffs (<22.1, <25.3, <30.8) were 0.29/0.80, 0.26/0.82 and 0.23/0.82, respectively. Most tools performed better when only the likelihood of the first recurrence was evaluated (Figure 1B). However, all tools except PCR Ct (AUROC 0.591) preformed worse when only patients with at least one previous recurrence were considered (Figure 1C). Youden index (which maximizes specificity plus sensitivity) curves generated for all cases for each score are shown in Figure 2.

Figure 2. Youden Indices for C. difficile Severity Score Cut-offs. Youden Index is equal to 0 for tests with poor diagnostic accuracy, equal to 1 for a perfect test, and assigns equal weight to sensitivity and specificity. The Youden Index and the ideal cutoff may not apply to other patient populations. ³¹

Discussion

Accurately anticipating risk is a crucial step towards improving access and stewardship of effective new therapies to prevent recurrent CDI, however, predicting future recurrence is a difficult problem. We demonstrate that existing clinical risk scores generalize poorly among a large cohort of hospitalized patients with C. difficile infection, especially early in the disease course. The rudimentary methods of generating a scoring systems used by these tools (ie, rounding logistic regression coefficients to integers) may have been inaccurate due to model bias, however, Escobar et al strove to generate more robust models using machine learning techniques and concluded that neither existing models nor their own model could be trained to accurately predict recurrent C. difficile infection in test patients (maximum achievable out-of-sample validation AUC 0.605). ¹⁹ Other factors to explain poor generalizability include significant C. difficile strain variation ³² or population-level differences (ie, location, socio-economic status, cohort recruitment, etc) between institutions that may influence recurrence risk or unmeasured putative features such as the delayed anti-toxin humoral immune response. ³³

Established risk factors for recurrent infection per recent consensus management guidelines are of marginal value for predicting recurrent C. difficile infection, with similar performance to other curated models. Discrepancy between reports of advanced age ≥65 as a positive risk factor for recurrent infection may be due to the competing risk of death within 60 days, which was two times higher among older patients ≥65 years in our cohort (139/685 (20%) patients ≥65 years vs 81/834 (9.7%) <65 years).

A significant hurdle in developing a generalizable and robust prediction tool for clinicians is the inability for the generated models to accurately predict outside of the data on which they were trained. All tools evaluated in this study performed more poorly than in the study in which they were derived. Similarly, when Rossen et al ¹² validated the Cobo and Larrainzar-Coghen models, they observed lower AUROCs, with each model performing worse than random chance. (Table 1) Escobar et al ¹⁹ observed a similar phenomenon in the models which they generated and in a previously generated model. This suggests that prediction tools have been overfitted to the data on which they were trained. A deep neural network with regularization may help to optimize the bias-variance tradeoff while incorporating a higher dimension of inputs to improve prediction. However, it is also possible that the task of accurately predicting future recurrence using clinical observations at the time of index CDI diagnosis is too complex or simply not feasible. Simple measures, such PCR Ct, may be more universally predictive throughout patient populations.

This analysis was intended as a practical assessment of existing tools implemented electronically to predict rCDI and there are important limitations to this analysis. rCDI tools were not validated using the exact same rCDI time frame definitions to which they were trained but, for feasibility reasons, a single rCDI definition was used. For example, the Eyre tool (trained to predict rCDI by 4 months) may perform better predicting late rCDI or relapse. In addition, several potentially important clinical features were omitted from some models because they could not be collected electronically (eg, stool frequency). C. difficile misdiagnosis (ie, colonization versus true infection) is a substantial issue in CDI, especially in the context of PCR-based testing and may have confounded both existing tools and our validations. ³⁴ Recurrent C. difficile infection is typically confirmed by repeat testing, which may not be reported in the same electronic medical record or may not be feasible in favor of expediting re-initiation of anti-C. difficile treatment (eg, in outpatient settings). To avoid undercapturing outcome events, we defined recurrent CDI broadly to include cases without retesting if symptoms relapsed requiring a new round of anti-C. difficile therapy. While our rCDI may have over captured some cases, we believe a symptom-based definition is a pragmatic approach to identify clinically relevant rCDI retrospectively, especially considering that PCR testing often remains positive regardless of rCDI. ³⁵ Our resulting rate of recurrence (22.7%) approximates rates from prospective trials (∼25%). ²⁵ 40% of retrospective rCDI events occurred without retesting at the same health system laboratory, underscoring potential flaws in retrospective studies that use test-based rCDI definitions (and thus likely artifactually low rCDI rates) and which could help explain the poor generalizability and performance of existing rCDI tools.

A better, validated prediction tool for recurrence would be helpful to individualize therapeutic approaches to prevent recurrent infection at the time of diagnosis (eg, prioritize fidaxomicin treatment and referral for bezlotoxumab in patients with highest risk for recurrence). Interestingly, of all tools evaluated in this study, PCR Ct had the highest AUROC, similar to at least 4 other studies demonstrating that low CT and/or stool toxin concentration predict rCDI. ^36–39 However, it should be noted that AUROC comparisons between categorical and quasi-continuous measures may not be valid given that scale discretization (of categorical risk scores) may reduce the precision of ROC measurements. ⁴⁰ While not a very useful standalone predictor, PCR Ct in a more complex model could increase the reliability of predicting recurrent CDI. Unlike the other biomarkers identified, PCR Ct is readily accessible (PCR is now used by >70% of US hospitals to diagnose CDI), ⁴¹ for any positive C. difficile PCR ³⁹ but Ct data are not traditionally reported to clinicians.

Future studies should carefully plan analysis to avoid common issues (eg, overfitting) and may benefit from considering novel biomarkers and/or higher-dimensional models that could augment or replace existing tools that underperform.