Challenging the assertion of comparability of surveillance and administrative data

To the Editor—In the April 2017 issue of Infection Control and Hospital Epidemiology, we read with interest the study by Ramirez Mendoza et al ¹ that compared administrative versus surveillance data for capturing hospital-associated methicillin-resistant Staphylococcus aureus (MRSA) infections in Canadian hospitals. However, due to the lack of a clear objective and some methodological concerns, we disagree with their study conclusion that there is good evidence for comparability between administrative and surveillance data.

The primary objective stated in the abstract was to assess the accuracy of administrative data concerning in-hospital bloodstream infections (BSIs) and all-body-site infections due to MRSA. However, in the body of the paper, the last paragraph of the background mentions that the primary objective of the study was to assess the feasibility of using administrative data to monitor these MRSA infections. A secondary objective was to determine whether there was a correlation between MRSA BSIs and all-body-site MRSA infections within the administrative data only. This lack of clarity around the primary objective is a concern because the methodological approach to determine feasibility differs from establishing accuracy.

If the author’s intention was to assess accuracy, the high correlation coefficient (r) reported in the paper does not necessarily mean high agreement between 2 datasets, nor accuracy of the data. For example, if the number of cases counted is exactly the same between 2 datasets, the cases themselves may not represent the same patients. Thus, a strong correlation between datasets is not meaningful unless the comparison of the datasets was conducted using patient-level data, which this study did not. Moreover, if the intention was to assess reliability of the datasets, we believe it would have been more appropriate to perform Lin’s concordance correlation coefficient. ² Lin’s coefficient adjusts the Pearson correlation coefficient by assessing both how close the data are about the line of best fit and the precision of agreement.

The administrative data from Alberta identified 113 MRSA infections, for a rate of 0.43 per 10,000 patient days, compared to 229 cases and a rate of 0.90 MRSA infections identified by surveillance data. Although the correlation coefficient indicated comparability (r=0.92; 95% CI, 0.88–0.94; P<.0001), the rate was twice the amount, meaning that for every 1 case in the administrative dataset there were 2 cases in the surveillance dataset. Furthermore, there may be intrinsic differences in the cases identified that impact the ability to use administrative data in place of surveillance data. Finally, conducting a comparison of this data using different infection sources (ie, MRSA bacteremia BSI in the Ontario dataset and all-body-site MRSA infections in the Alberta dataset) is of vital concern considering the differences between these datasets and the noted challenges with using administrative data to identify infections.

We challenge the assertion of comparability of surveillance and administrative data based on how the data are generated. Surveillance data employ comprehensive definitions and case finding methods, which includes a review of positive laboratory results and clinical data (ie, chart reviews, in-person interviews, etc) to capture hospital infections. Administrative data are based on applying International Classification of Disease, Tenth Revision, Canada (ICD-10-CA) diagnostic codes against information documented in the patient chart. The quality and completeness of clinical documentation are recognized limitations of administrative data; they impact coders’ ability to accurately code cases, potentially leading to an overestimate or underestimate of cases. Correctly identifying a patient with an infection based on coding alone is challenging because it can be difficult to differentiate an infection from colonization for nonsterile sites. In a study involving professional coders in Alberta, Tang et al ⁴ identified multiple barriers to producing high-quality administrative data, including documentation that is incomplete and nonspecific and often contains errors and discrepancies. ³ Furthermore, Nicholls et al ⁴ emphasized the importance of validating coding accuracy to reduce the potential of misclassification bias.

This study used data from Alberta and Ontario, and we do not think it is appropriate to generalize the conclusion for Canadian hospitals. Two unpublished analyses from 2 health regions in British Columbia were not in agreement with this study conclusion. An analysis conducted by Interior Health compared MRSA in-hospital infections from the Discharge Abstracts Database (DAD) to new healthcare-associated MRSA infection cases from the surveillance database of 22 acute-care facilities. The Lin’s concordance coefficient was 0.31. ⁵ For the same period, a patient-level record-linkage analysis involving 9 acute-care facilities conducted by Vancouver Coastal Health found that the DAD data only captured 30% of the 180 healthcare-associated MRSA infection cases identified through surveillance. Both analyses demonstrate the poor concordance between administrative and surveillance data for identifying in-hospital MRSA infections.

In conclusion, we cannot support the authors claim that there is good evidence of comparability of administrative and surveillance data in Canadian hospitals based on problematic data and methods used.