External validation of the Michigan PICC catheter–associated bloodstream infections score (MPC score) for predicting the risk of peripherally inserted central catheter–associated bloodstream infections: A single-center study in Japan

Abstract

The Michigan peripherally inserted central catheter–associated bloodstream infection score (MPC score) had been developed for hospitalized medical patients but had not been externally validated. A retrospective analysis of a clinically heterogeneous case-mix in a university hospital cohort in Japan failed to validate its originally reported good performance.

Peripherally inserted central venous catheters (PICCs) are a common type of central venous catheters (CVCs) used for hospitalized patients. ^1,2 However, PICC-associated bloodstream infections (PBSI) have an incidence rate of up to 7.71 per 1,000 catheter days, which prolongs hospital stay and increases costs and mortality. ¹

The Michigan PICC catheter-associated bloodstream infections score (MPC score) is a risk prediction model to estimate an individual’s risk of developing PBSI. ³ Although the US multicenter derivation cohort involving 23,088 adult medical patients reported a promisingly good discriminative performance (C statistic, 0.67–0.77), ³ its external validation had yet to be performed. Therefore, in this research, we aimed to validate its discrimination and calibration performance in a Japanese university hospital setting involving all hospitalized patients regardless of the departments to which they were admitted.

Methods

Source of data and participants

This study was a retrospective analysis using routinely collected data from Fujita Health University Hospital, a 1,435-bed, tertiary-care, academic center in Japan. The institutional review board of Fujita Health University approved this study (approval no. HM19-479) and waived the requirement for informed consent.

Between October 2015 and January 2020, all adult (≥18 years old) hospitalized patients, regardless of the departments in which they were admitted, who had PICCs inserted by the dedicated PICC nursing team were prospectively and consecutively registered with the hospital PICC database to survey CVC-related complications. Patients were operationally subcategorized into 4 groups based on the departments to which they were admitted (ie, medicine, surgery, ICU, and others).

Outcome

Following the National Healthcare Safety Network Laboratory Confirmed Bloodstream Infection (LCBI) Checklist, PBSIs were defined as bloodstream infections that satisfied the LCBI-2 criterion. ⁴

Predictors

Predictors of PBSI (points assigned by the score ³ ) included (1) past and present history of hematological malignancy (3 points); (2) history of central-line–associated bloodstream infection (CLABSI) within 3 months of PICC insertion (2 points); (3) active cancer with receipt of chemotherapy (2 points); (4) multilumen PICC (2 points); (5) presence of another CVC upon PICC placement (1 point); and (6) receipt of total parenteral nutrition through the PICC (1 point).

Statistical analysis

In this study, we followed the recommended framework for assessing discrimination and calibration in external validation of a Cox prognostic model. ⁵ Full details of the methods are available as Supplementary Material (online). To compare a clinical “net benefit” among strategies based on the original or updated MPC score and among default strategies performing some interventions for all or no patients, a decision curve analysis was performed. ⁶

In our main analysis, we used the original MPC score and grouped patients into 7 risk categories according to the total points (0, 1, 2, 3, 4, 5, and ≥6 points). ³ In sensitivity analyses, we used the linear predictor with the originally reported coefficients to calculate the prognostic index.

Results

Patient characteristics

Our case-mix variation explicitly differed from the derivation cohort in the inclusion of patients hospitalized in nonmedical departments, higher patient age, and higher prevalence of malignancy and history of CLBSI (Table 1).

Table 1. Baseline Characteristics of Derivation and Validation Cohorts

Note. CLABSI, central-line–associated bloodstream infection; CVC, central venous catheterization; PICC, peripherally inserted central catheter; PBSI, peripherally inserted central catheter–associated bloodstream infection.

^a Results were for patients who did not develop CLABSI (n = 22,839). The median dwell time for patients who developed CLABSI (n = 249) was 15 days (IQR, 9–26).

Outcomes

Over a median (per catheter) follow-up duration of 15 days (IQR, 12–31 days; total, 29,812 days), 89 PBSIs occurred (incidence, 6.1%; incidence rate, 2.99 per 1,000 catheter days). In our cohort, the median PICC dwell time in patients who did develop a PBSI was 17 days and in those who did not develop a PBSI it was 15 days. These dwell times were respectively longer than those in the derivation cohort, which were 15 and 11 days, respectively. Of the 89 causative organisms in our PBSI cases, 60 (67.4%) included gram-positive cocci, and coagulase-negative Staphylococcus were the most prevalent (n = 44, 49.4%) (Supplementary Table 1 online).

External validation of the MPC score

The median values of the MPC score and prognostic index calculated based on the linear predictor were 1 (IQR, 1–2) and 0.60 (IQR, 0.60–1.33), respectively; the distributions of both the values were skewed to the right (Supplementary Fig. 1 online).

Discrimination

Visual assessment of the Kaplan–Meier curves did not demonstrate discrimination between the risk groups (log-rank χ ² = 11.66; P = .11) (Fig. 1A). The discrimination measures were generally poor (eg, C statistic = 0.608 and K statistic = 0.544) (Supplementary Table 2 online).

Fig. 1. Observed and predicted PBSI even curves. Kaplan–Meier plots for the observed PBSI events (A, left panel) and predicted mean event curves for the original (B, center panel) and updated (C, right panel) MPC scores. The curves are color-coded based on the assigned scores in dark green (0 points), orange-red (1 point), navy (2 points), maroon (3 points), teal (4 points), sienna (5 points), and orange (≥ 6 points). Note. MPC score, Michigan peripherally inserted central catheter–associated bloodstream infections score; PBSI, peripherally inserted central catheter–associated bloodstream infection.

Calibration

The predicted risks according to the MPC score were systematically too low and had insufficient variety (calibration slope, 1.16; 95% CI, 1.02–1.32; P = .024; χ ² of model misspecification, 23.74; P < .001). The observed event rates for the lower-risk groups (≤3 scores) were generally higher than the corresponding predicted mean curves (Fig. 1A and B). The calibration plots at any observed timing consistently showed underestimation (Supplementary Fig. 2 online).

Update of the MPC score

Updating the baseline survival function increased the predicted mean event rates that were higher than the original score (Fig. 1B and 1C), which improved the poor calibration of the original score (updated calibration slope, 1.02; 95% CI, 0.04–1.99; P = .051). See Supplementary Fig. 3 (online) for the calibration plots.

Decision curve analysis

The updated MPC score outperformed the other strategies when the threshold probability ranged from 4% to 13%, and the cumulative PBSIs up to 28 days were concerning (Supplementary Fig. 4 online). However, the maximal net benefit was only 0.016 (ie, 16 more cases appropriately managed per 1,000 cases) compared to the “intervention for all” strategy without model when the threshold probability was 8.7% and the cumulative PBSI at 28 days was the outcome of interest. The “intervention for all” strategy outperformed the original MPC score up to 28 days regardless of the threshold probability.

Sensitivity analysis

Although the prognostic index based on the original linear predictor slightly improved the discrimination of the MPC score (log-rank χ ² = 14.5; P = .006), the discriminative and calibration performance measures were still unsatisfactory (Supplementary Table 1 and Supplementary Figs. 5 and 6 online). The maximal net benefit of the updated MPC score was similar to that of the main analysis (Supplementary Fig. 7 online).

Discussion

In this study, we failed to replicate the good performance of the original MPC score. Our cohort had a clinically more heterogeneous case mix with a higher PBSI incidence compared to the more uniform derivation cohort consisting of only hospitalized medical patients. Specifically, the discriminative performance was poor, and the model largely underestimated the risk of PBSI. However, with a recalibrated baseline hazard function, the updated model had improved calibration. A decision curve analysis showed that the updated MPC score marginally outperformed reference baseline strategies without the model, although the increase in the net benefit was not substantial.

Our study had several strengths. It was the first external validation of the MPC score of its kind. We included consecutive, prospectively registered patients who underwent PICC catheterizations uniformly performed by a dedicated PICC team with no missing information. We used the standard approach for updating and adapting an existing model to a local setting. ^7,8 This study also had limitations. It was a retrospective, single-center design with only 89 PBSI events, which did not satisfy the minimum target events to ensure precise estimates of model performance ⁹ and did not allow subgroup analysis on hospitalized medical patients or a full model update.

The case mix and baseline risk of our cohort differed from that of the derivation cohort. ³ Therefore, the observed poor calibration of the original score is not surprising. Recalibrating the baseline hazard function is only the first step in locally updating the model, ^8,10 and the sample size was small. Thus, implementing a multicenter collaborative study, re-estimating the coefficients for the individual predictors, and incorporating new, additional parameters, ¹⁰ would be the next steps to making better use of the MPC score.