External validation and updating of a prediction model for nosocomial pneumonia after coronary artery bypass graft surgery

Summary

The generalizability of a prediction model from North America for incident nosocomial pneumonia following coronary artery bypass graft surgery was assessed for 23247 patients on the Australian and New Zealand Society of Cardiac and Thoracic Surgeons (ANZSCTS) registry. The performance of the North American model was evaluated using measures of calibration and discrimination. The model had reasonable discrimination (area under the receiver-operating characteristic curve, AUC = 0·69), but unsatisfactory calibration (Hosmer–Lemeshow test, P < 0·001) in the ANZSCTS patients. An update of the model coefficients yielded a model with AUC = 0·71 and good calibration (P = 0·46).

Pneumonia is a leading cause of increased length of stay in the intensive-care unit, and is associated with increased mortality rates by as much as 14-fold [1].

Many studies have explored factors that increase the risk of post-operative pulmonary complications after coronary artery bypass graft (CABG) surgery [2]. While a few different models have been developed to predict incident pneumonia after CABG surgery [3, 4] the sample sizes used in these studies have been small [2–4] with one exception from North America which used 17143 CABG patients from Tenet Healthcare's Quality and Resource Management System [3]. While this North American tool shows some promise, external validation of it is lacking and assurance is needed that it maintains good predictive ability in different patients [5].

As has been discussed previously [3], strategies to reduce the incidence of hospital-acquired pneumonia may need to be restricted to high-risk patients in order to achieve an appropriate balance of benefit with substantial economic cost [6]. Identification of high-risk patients might be achieved using the published prediction model [3]; however, it is necessary to confirm that it has appropriate calibration of high risk in new settings as well as that it continues to demonstrate reasonable ability to discriminate between higher- and lower-risk individuals. We present the first external validation study of Kinlin's model [3] by using the multi-centre Australian and New Zealand Society of Cardiac and Thoracic Surgeons (ANZSCTS) registry of cardiac surgery patients.

For our study, the cohort consisted of 23247 patients from the ANZSCTS registry who underwent CABG between 2001 and 2009. The ANZSCTS is responsible for this registry of cardiac and thoracic surgery in Australia and New Zealand and description of the registry has been published previously [7]. Our validation cohort consisted of patients from 18 hospitals across four states of Australia and three (17%) of these hospitals were non-teaching.

The ANZSCTS registry recommends that data recording for post-operative complications including pneumonia is completed at each participating hospital by a registrar and that the data manager checks for completeness and amends as required. However, this process can vary from hospital to hospital.

Pneumonia was recorded in the ANZSCTS registry according to the definition of diagnosis by one of the following: positive cultures of sputum or trans-tracheal aspirate and consistent with clinical findings of pneumonia (including radiological changes). By comparison, in development of the model [3] outcome was defined on the basis of (1) new onset of pneumonia during current hospitalization, as documented by a treating physician; (2) positive results of sputum, blood, pleural, empyema, trans-tracheal, or trans-thoracic fluid cultures, compatible with the diagnosis and clinical findings of pneumonia; or (3) chest radiograph diagnostic of pulmonary infiltrates. Further, aspiration pneumonia was coded separately from nosocomial pneumonia and omitted from the outcomes used for model development.

There were 13 risk factors in the published model and information on nine of these was collected in the ANZSCTS registry (Table 1). For the model development, patients who underwent isolated CABG surgery or CABG with a concomitant valve procedure were included. For the published model concomitant valve replacement was considered for inclusion but was not significant in univariate analysis nor was it required in the final model. We applied the same inclusion criteria to patients on the ANZSCTS registry.

Table 1. Demographic and clinical characteristics of two populations: the North American population on which the published (Kinlin et al. [3]) risk prediction model for nosocomial pneumonia following cardiac surgery was developed and the Australian and New Zealand Society of Cardio-Thoracic Surgeons (ANZSCTS), population used for external validation of the model

CABG, Coronary artery bypass graft; CCS class, Canadian Cardiovascular Society classification of angina; PTCA, percutaneous transluminal coronary angioplasty; BMI, body mass index; n.a., not available.

* Coefficients are log odds ratios from a logistic regression model. The exponential of the coefficient is the odds ratio for nosocomial pneumonia. Intercept coefficients for the published and updated tools are −5·70 and −4·02, respectively.

† Underweight is defined as BMI <18·5, where BMI is calculated as weight in kilograms divided by the square of height in metres.

This research project was undertaken following approval from the ANZSCTS Research Committee which governs access to data from the registry. Ethical approval for the use of de-identified registry data for secondary research purposes such as this project had previously been provided by each participating institution's ethics review committee.

To assess the performance of the model in the ANZSCTS patients, we investigated discrimination by calculation of the area under the maximum likelihood receiver-operating characteristic curve (AUC). This AUC gives the likelihood that a randomly selected infected patient would have a higher model-predicted probability of pneumonia infection than a randomly selected non-infected patient. An AUC between 0·7 and 0·8 is considered acceptable, between 0·8 and 0·9 excellent, and >0·9 outstanding discrimination [8].

Calibration of the model predictions in the ANZSCTS patients was assessed graphically and with the Hosmer–Lemeshow goodness-of-fit test. In a perfect fit, for a group of patients with similar predicted probability of pneumonia this prediction should be similar to the observed proportion of these patients having pneumonia. A good fit would be demonstrated graphically by a diagonal line on a calibration plot of observed proportion vs. predicted probability of pneumonia. The Hosmer–Lemeshow goodness-of-fit test investigates (under the null hypothesis that there is no difference) the difference between the model predictions and the actual observations using declines of predicted probability to group patients. A P value >0·05 indicates a good fit [8]. The published model employs a logistic regression formula to relate its 13 predictive factors to pneumonia risk after CABG surgery [3]. The regression coefficients are listed in Table 1. Updating of this model was performed to recalibrate its performance for the Australian context. The updating involved re-estimation of the model intercept and odds ratios (ORs) for the nine risk factors available in the ANZSCTS registry and assumed ORs of one for the four risk factors not in the registry [9].

Two approaches to missing data were considered; first, analysis of all patients following imputation of the ‘lowest risk’ value for any risk factor with missing values (n = 23247) (this followed the model development approach [3]), and second, analysis of 21723 complete cases who had no missing data on risk factors. Stata version 11 (Stata Corp, USA) was used for all statistical analyses.

Following CABG, the new-onset pneumonia rate in 23247 ANZSCTS patients was 5·0%. The incidence of nosocomial pneumonia in the North American population used for model development was 2·0%. The comparability of the two patient populations is shown in Table 1. More ANZSCTS patients had a history of smoking although these patients had lower chronic obstructive pulmonary disease prevalence. Canadian Cardiovascular Society classification of angina (CCS class), creatinine level and intraoperative blood transfusion levels were similar in the two populations. Emergency surgery, underweight and percutaneous transluminal coronary angioplasty use were less common in the ANZSCTS patients but these variables had relatively low prevalence overall. Of ANZSCTS patients, 86% underwent CABG alone (4·6% with pneumonia) and 14% underwent CABG plus a valve procedure (pneumonia incidence rate 7·2%). The annual incidence of pneumonia had a slight decrease between 2001 and 2009 [OR 0·976 per year, 95% confidence interval (CI) 0·953–1·000].

The published model had an AUC of 0·75 from internal validation in North American patients. The model's discrimination in the ANZSCTS patients was summarized by an AUC of 0·69 (95% CI 0·68–0·71). The Hosmer–Lemeshow statistic (P < 0·001) suggested unsatisfactory calibration in ANZSCTS patients.

Using complete cases to deal with missing data yielded almost identical results (AUC = 0·69, 95% CI 0·68–0·71, Hosmer–Lemeshow P < 0·001).

Regression coefficients for the updated model are shown in Table 1. The updated model had an AUC = 0·71 (95% CI 0·69–0·72) and good calibration (Hosmer–Lemeshow P = 0·46). The performance characteristics of the updated model are also shown in Figure 1 from which it can be seen that the updated model predicted probabilities correspond well with the observed proportion of patients with pneumonia for all risk factor combinations up to the maximum predicted probability of 28%. This updated model identified as high risk 881 patients with predicted probability ⩾20%, and 173 patients with predicted probability of ⩾25%.

Fig. 1. Calibration plot for the updated model in 23247 Australian and New Zealand patients. Each point represents a unique risk factor combination and plots the pneumonia-predicted probability for that combination (x axis) against the observed proportion of patients with pneumonia for the same combination (y axis). Perfect prediction corresponds to a slope of 1 (diagonal, 45° straight line).

To summarize these findings, the published model [3] offered discriminatory power in the ANZSCTS dataset (AUC = 0·69) that was of borderline acceptability [9], albeit not greatly reduced from its internal validation performance in North America summarized by an AUC of 0·75. A drop in performance is anticipated in external validation exercises [5]. Nevertheless, the existing model may be acceptable for discriminating among patients for the development of new-onset pneumonia following CABG although its miscalibration will probably limit its clinical applicability. Our updated model had similar discrimination ability and successfully fixed the miscalibration and may therefore be of more use to clinicians in the Australian context, especially for the purpose of screening for high-risk subgroups.

Some differences between the two populations, their healthcare systems, the methods and definitions of the two studies, differences in risk-factor prevalence and a lack of information on four model risk factors may all have contributed to the deterioration in model performance in the ANZSCTS population (Table 1). The four variables missing in the ANZSCTS registry data present a problem due to their unknown prevalence and association with outcome in this registry. Among them, prior CABG with internal mammary artery graft had a relatively large coefficient in the published model but was not common in that population (prevalence 2·5%) so would have contributed only modestly to the model's original discriminatory ability. The other three variables (admission from a non-residential setting, cancer history, and vancomycin administration pre-operatively) had smaller coefficients but were more common. The outcome definition in the ANZSCTS registry was more sensitive and less specific compared to the North American study, but no perfect definition for clear clinical delineation of nosocomial pneumonia exists [10]. While registrars, with back up from data managers, recorded post-operative data for ANZSCTS, in the North American study, trained nurse case managers recorded the occurrence of complications after surgery, including the development of nosocomial pneumonia.

In conclusion, we have presented the first external validation of a North American prediction model for nosocomial pneumonia following cardiac surgery and in so doing assessed whether the model generalizes sufficiently well to be of use in an Australian context. The results have demonstrated discrimination ability of borderline acceptability for the existing model. An updated version of the prediction model addressed a problem with miscalibration of the original model in the Australian context. The updated model may be useful in identifying high-risk patients for intensive preventive measures.