A systemwide approach to validation of electronic device denominator data

Abstract

A multidisciplinary team collaborated to develop and validate a process to electronically capture patient and device denominator data at 6 hospitals in the same healthcare system. Validation was completed within 4–16 months. Manual count errors were identified as the main driver of electronic versus manual discrepancies.

Accurately capturing patient, central-line, indwelling urinary catheter, and ventilator denominator data is essential for calculation of healthcare-associated infection (HAI) metrics. National Healthcare Safety Network (NHSN) denominator data collection methods include manual daily capture, manual weekly sampling, and electronic capture. ^1–4 Manual data collection is burdensome, leaving less time for essential healthcare duties. Wright et al ⁵ established that accurate automation of device data was achievable, and others have reported on the importance of validation. ^7,8 To use electronically captured denominator data, the NHSN requires that it is validated to match manual data within 5% for 3 consecutive months. ¹ Validation is required when facilities first implement electronic capture and when transitioning from one electronic method to another. ¹ Denominator validation is an operationally challenging task that healthcare facilities often struggle to successfully accomplish.

At our expanding health system, validation of electronic denominator data was previously completed at each hospital using disparate approaches and electronic health record (EHR) systems. The process was inefficient, unstandardized, and fraught with challenges. However, a planned integration of all hospitals into the same EHR necessitated revalidation. The goal of this project was to apply knowledge gained from past challenges to develop a standardized, systemwide, scalable approach to electronic denominator data validation.

Methods

In March 2018, 6 hospitals (5 acute-care hospitals and 1 critical-access hospital) in a large Illinois health system integrated onto 1 EHR platform and infection control module (Epic, Verona, WI), rendering past methods of electronic data capture nonfunctional. Consequently, infection prevention and analytics teams collaborated to develop and validate a new electronic data-capture process (Fig. 1).

Fig. 1. Overview of the process for the validation of electronic patient and device denominator data, indicating who was responsible for each item. Note. IP, infection prevention.

The analytics team developed custom structured query language (SQL) code to run daily at 11:59 p.m., to extract EHR data and store it in a database. The query was standardized at all hospitals; it adhered to NHSN specifications ^1–4,6 and it used EHR flowsheet data to capture device information (device type, insertion and removal data, first and last assessment date and time). All facilities had the same documentation standards, but logic was designed to best approximate device presence despite imperfect nursing documentation (eg, if device removal date/time was missing, value was imputed based on other available data).

The infection prevention team established a new manual data collection process, created educational materials, and trained designated nursing staff on accurate data collection, using NHSN definitions. ^1–4,6 The staff printed a time-stamped census list from the EHR each night (∼11:59 p.m.) and indicated device(s) present next to each patient name. Infection preventionists retrieved completed lists weekly, then entered the data into a form on a shared website (SharePoint, Microsoft, Redman, WA). Infection preventionists at all hospitals could simultaneously enter data, and dropdown selections ensured consistency in naming and formatting conventions.

The analytics team then developed a process to extract manual data from the shared website and then incorporate it into the same database storing electronic denominator data. An automated report (Appendix online) accessed through an internal web portal summarized validation results and allowed users to view detail-level electronic data (eg, list of patients and devices by date). If manual data were missing for a particular nursing unit and date, electronic data for that unit and date were also excluded from comparisons.

When denominator counts did not match within 5%, infection preventionists compared electronic and manual detail-level data then performed chart review (EHR flowsheets, notes, radiology reports) to investigate discrepancies and ascertain device presence. Data were collected on a sample of discrepant charts and were recorded on the shared website. Documentation of discrepancies was not mandatory, but it served as a helpful tool to gain insight, provide feedback, and drive improvement. If programmatic issues were identified, the analytics team altered the query. When there was indisputable evidence from multiple sources in the EHR that a manual count was incorrect, it was corrected, and then was reflected in the validation report upon subsequent data extraction.

Manual counts from the validation report were used for monthly reporting to the NHSN and were adjusted to account for missing data according to NHSN guidelines. ⁶ Once a unit completed validation, staff stopped manual data collection and transitioned to electronic denominator reporting.

Results

Validation of electronic denominator data was completed at all 6 hospitals, within 4–16 months (Table 1). Among 67 participating nursing units, 40 (59.7%) completed validation in 4 months, 10 (14.9%) in 5–8 months, and 17 (25.4%) in ≥9 months.

Table 1. Denominator Validation Results by Hospital

Note. EHR, electronic health record.

^a Inpatient nursing units and licensed beds, as of March 2018.

^b Epic software (Verona, WI).

Among a sample of 639 manual versus electronic count discrepancies recorded at 4 hospitals, the following reasons were noted (categories not mutually exclusive): 486 manual count errors (76.1%), 140 nursing documentation deficiencies (21.9%) (ie, inaccurate or missing device removal date and time, missing assessments, missing documentation of nonaccessed central lines), 33 electronic report issues (5.2%) (ie, patient inclusion criteria, missing device types), and 14 other (2.2%). Among 486 manual count errors identified, common reasons included omitting nonaccessed ports or long-term central venous catheters; omitting ventilators; inappropriately counting excluded devices (ie, midline catheters, peripheral intravenous devices, intra-aortic balloon pumps, suprapubic catheters, condom catheters or ureteral stents); and capturing manual counts well outside the recommended 11:59 p.m. count time.

Discussion

Our systemwide approach to electronic denominator validation streamlined data collection and reporting to improve efficiency. With dedicated analytics support and automated tools, validation was completed at all hospitals with various timelines. Specific factors that affected time to completion included prior experience documenting in Epic software, facility size, device prevalence (ie, difficulty matching within 5% with small numbers), and the amount of time infection preventionists dedicated to discrepancy review on assigned units. Devoting time to regularly review discrepancies and provide feedback to nursing early in the process were essential to completing validation and saving future time and effort.

Manual counting errors accounted for >70% of the discrepancies identified. Infection preventionists provided feedback and re-education on data manual collection, and they corrected data when there was clear evidence of errors. The high prevalence of manual count errors calls into question whether manual counts should be considered the source of truth in device data capture. Although validation is essential to ensuring data accuracy, manual data are imperfect, prone to human error, and rely on continual education to maintain accuracy. Additionally, in our situation of revalidation due to a transition between electronic methods, staff may have benefitted from extra time to adjust to documenting in a new EHR and collecting manual data before initiating validation efforts.

We chose to develop a custom method of capturing electronic denominator data due to limitations noted in vendor solutions at the time of project initiation. In particular, the vendor’s electronic capture method lacked flexibility to account for incomplete documentation, and changing predefined logic was difficult or impossible. Although monthly aggregated electronic data were readily available in an EHR report, we could not easily obtain and store detail-level electronic data to compare with manual data stored outside the EHR system. Moving forward, EHR and IP software vendors have the opportunity to further advance capability to facilitate validation of electronic data validation.

With advancing technology, expanding health systems, and increasing need to reduce burden on underresourced teams, supporting efforts to efficiently validate and implement electronic denominator data is crucial. Accurate denominator data are not only necessary for correct calculation of NHSN infection metrics for each individual hospital, but they also have a collective impact on national and state benchmarks. This report has summarized a multidisciplinary approach to successfully scale and optimize electronic denominator data validation across a large healthcare system.