Antibiotics are often prescribed for inappropriate indications and excessive durations. Reference Barlam, Cosgrove and Abbo1,Reference Fleming-Dutra, Hersh and Shapiro2 Inappropriate antibiotic use can contribute to the development of antibiotic resistance, which is a well-recognized public health threat with a substantial impact. Reference Goff3,Reference Spellberg, Guidos and Gilbert4 One-third of hospital antibiotic orders involve potential problems such as prescribing an antibiotic without proper diagnostic testing or evaluation, prescribing an antibiotic unnecessarily, or prescribing excessive antibiotic durations for common infections. 5 Thus, antibiotic stewardship initiatives are essential to improve antibiotic prescribing in the inpatient setting. Reference Barlam, Cosgrove and Abbo1,Reference Goff, Kullar, Bauer and File6
The Centers for Disease Control and Prevention (CDC) Core Elements for Hospital Antibiotic Stewardship Programs suggest implementing antibiotic order sets (AOSs) to help improve antibiotic use. 5 Order sets have been shown to improve the management of various disease states by promoting evidence-based care, thereby reducing inappropriate prescribing. However, most of these studies are limited to emergency departments, outpatient settings, or sepsis-specific protocols. Reference Hauck, Adler and Mulla7–Reference Vuong, Kenney and Thomson16 Furthermore, many order sets have used clinical pathways that had to be accessed outside the electronic medical record (EMR), which may have deterred providers from readily accessing these resources. Incorporation of AOSs into the EMR may facilitate their use.
After the development and implementation of AOSs for common infections into our EMR, we evaluated antibiotic stewardship metrics including the standardized antimicrobial administration ratio (SAAR), antibiotic days of therapy (DOT) per 1,000 patient days (PD), use of targeted antibiotics, and Clostridioides difficile infection (CDI) cases. We also assessed clinician perceptions of the AOSs.
Methods
Institutional antibiotic stewardship
The State University of New York (SUNY) Upstate University Hospital is a 752-bed, tertiary-care, academic medical center located in Syracuse, New York. The antibiotic stewardship program (ASP) was established in 2014 and is co-led by an infectious diseases (ID) physician and an ID pharmacist. The ASP subcommittee reports to the medical executive committee within the health system and has representation from ID physicians (adult and pediatric), ID pharmacists (adult and pediatric), emergency medicine physicians, microbiology, and infection control. In 2018, the ASP was recognized as a Center of Excellence by the Infectious Diseases Society of America (IDSA). 17
The ID physician has 0.25 full-time equivalents for antibiotic stewardship. The ID pharmacist has 1.0 FTE for antibiotic stewardship with the support of a pharmacy school-funded ID faculty member and a postgraduate year 2 ID pharmacy resident. The antibiotic stewardship service is staffed by ID pharmacist(s) with ID physician support and input weekdays from 08:00 a.m. to16:30 p.m., excluding weekends and holidays. Daily antibiotic stewardship activities include, but are not limited to, prospective audit and feedback, antibiotic preauthorization, discharge antibiotic prescription review, pharmacokinetic monitoring, renal dose adjustments, intravenous-to-oral conversions for highly bioavailable antibiotics, and drug information. Communication with the primary services occurs mostly through a chat function within the EMR. Chart notes from an ID physician and/or ID pharmacist can also be used as needed.
AOS implementation
Prior to the development of the AOS, no clinical pathways or institutional ID management guidelines existed at our institution. In September 2018, our ASP implemented empiric AOS for pneumonia (community-acquired and hospital-acquired/ventilator-associated), urinary tract infection (UTI), intra-abdominal or gastrointestinal infection, skin and soft-tissue infection, bone and joint infection, meningitis and encephalitis, neutropenic fever, endovascular infection, pelvic infection, and oral cavity or neck infection. Antibiotic selection was based on guidance from IDSA when available, our antibiogram, and hospital formulary. All AOSs were directly incorporated into Epic software (Epic, Verona, WI). To increase visibility and awareness, AOSs were accessible by entering numerous keywords including any antibiotic order (eg, piperacillin-tazobactam) or by keying the word ‘antibiotic’ into the order queue, which would create a ‘pop-up’ for the AOS to be selected. Clinicians then selected the infection that they were managing, navigated through subheadings within the infection type, and applied patient-specific factors as appropriate. The AOS included drug names, dose, route, frequency, and suggested duration (ie, not limited to a single dose). Durations could later be truncated or extended as appropriate by the treating clinician. The AOS also incorporated patient-specific factors as appropriate. For example, the community-acquired pneumonia (CAP) AOS is subcategorized into those with or without risk factors for multidrug-resistant bacteria, alternative antibiotics for patients with a severe penicillin allergy, and dose adjustments for patients with renal dysfunction (Fig. 1). Antibiotic therapy could subsequently be de-escalated and/or adjusted as needed based on the patient’s clinical response and evolving microbiology and ID data.
Figure 1. Example of community-acquired pneumonia empiric antibiotic order set embedded within the electronic medical record. Note. CVVH, continuous venovenous hemofiltration; CrCl, creatinine clearance; MDRO, multidrug resistant organism; MRSA, methicillin-resistant Staphylococcus aureus.
Education regarding the AOS was provided during the initial implementation phase via pharmacy and therapeutics committee distribution, electronic newsletters to clinical staff, and educational seminars (eg, grand rounds, trainee physician orientation, antibiotic stewardship presentations). Trainee physicians were educated on the purpose, logistics, and use of the AOS annually in July by 2 investigators (S.J.T. and W.D.K.).
Study outcomes
The primary outcome was to evaluate the impact of empiric AOS on the SAAR. Secondary outcomes included the following: antibiotic DOT per 1,000 PD; targeted use of fluoroquinolones, carbapenems, clindamycin, and vancomycin; AOS utilization; CDI cases; and clinicians’ perceptions of the AOS.
Study design
This 5-year, quasi-experimental study comprised 5 phases. For each phase, data were collected over a 10-month period from August 1 to May 31, excluding June and July to minimize the influence of trainee physician cycling. Phase 1 spanned August 1, 2017, through May 31, 2018; phase 2 spanned August 2018 through May 31, 2019; phase 3 spanned August 2019 through May 31, 2020; phase 4 spanned August 2020 through May 31, 2021, and phase 5 spanned August 2021 through May 31, 2022. Phase 1 was before the AOSs were implemented (ie, before the intervention or control cohort). Phase 2 was the intervention period when the AOSs were implemented (September 2018). Phases 3, 4, and 5 were the first, second, and third postintervention periods, respectively. Phases 3–5 were all affected by the coronavirus 2019 (COVID-19) pandemic; however, COVID-19 was declared a pandemic during phase 3. 18 The ASP did not introduce any novel antimicrobial stewardship initiatives and/or practices aside from the AOS. ASP physician and pharmacist staff numbers as well as daily ASP activities were consistent throughout all phases to better evaluate the impact of the AOS.
Unique to phase 5, the admitting trainee and attending physicians, which were the same throughout the entire week, were contacted every Monday via EPIC chat to provide a reminder on how to access the AOS and to strongly encourage their use. This group was targeted because the investigators deemed them to be the most likely clinicians to use the AOS when admitting a patient. We evaluated this targeted educational initiative during phase 5 only. Data related to adult patients aged ≥18 years were included, whereas all pediatric-related data were excluded. This study was reviewed and deemed exempt from review by the Institutional Review Board at SUNY Upstate Medical University (IRB no. 1545645-1).
A 20-question survey was also developed to evaluate clinician’s perceptions of the AOS after phase 5 (ie, June 2022). This survey was distributed to clinicians at SUNY Upstate University Hospital who staffed primary inpatient admission services including attending physicians, trainee physicians, and advanced practice providers (n = 247). Clinicians of consulting services were excluded from the survey because they do not actually place antibiotic orders but rather provide recommendations in their chart notes for antibiotic recommendations for the primary services to order. Incomplete survey responses were also excluded. A survey invitation letter with appropriate informed consent information was attached to the survey instrument and distributed electronically via email. By entering the survey, the respondent agreed to participate.
A Research Electronic Data Capture (REDCap) platform was used to design and collect survey responses. Survey questions were developed using the expert opinion of the ASP leadership of ID physicians and ID pharmacists. Participation in the survey was voluntary, but participation was encouraged. The survey was left open for 6 weeks with weekly reminder emails to nonresponders.
Definitions
The SAAR and antibiotic DOT per 1,000 PD metrics were calculated using our institution’s data reported to the National Healthcare Safety Network. Targeted antibiotic orders were those that our ASP identified as ‘broad spectrum’ or ‘high risk’ based on their adverse effect profile. These included fluoroquinolones (intravenous or oral formulations of ciprofloxacin, levofloxacin, and moxifloxacin), carbapenems (meropenem, ertapenem, and imipenem-cilastatin), clindamycin (intravenous and oral formulations), and vancomycin (intravenous formulation). Targeted antibiotic orders were extracted from EPIC software during each period to represent total use. AOS utilization data were extracted from EPIC as the total signed antibiotic orders from the adult AOS order panel during phases 2–5. A CDI case was defined as a positive polymerase chain reaction (PCR) assay with positive enzyme immunoassay (EIA) as reported by our institution to potentially minimize identification of patients who may have colonization rather than have infection (ie, PCR positive, EIA negative) despite the lower sensitivity of the EIA. Notably, our institution converted to this PCR/EIA algorithm in February 2019, during phase 2 of our study. For the survey, clinicians included attending physicians, trainee physicians, and advanced practice providers.
Statistical analysis
All statistical analyses including descriptive statistics were performed using R software (R Foundation for Statistical Computing, Vienna, Austria). Locally estimated scatterplot smoothing (LOESS) curves were utilized to display trend variables of interest between different periods (ie, phases 1–5) in an exploratory manner. To test the difference between periods, linear regression models were fitted using the generalized least squares (GLS) method. The ARMA (autoregressive moving average) correlation structure of order (p, q) was applied to model the corrections between monthly observations within each period. The Akaike Information Criterion (AIC) was used to select the appropriate values of p and q. In the model, period was considered the discrete variable. The overall significance of period was tested using the likelihood ratio test, and the comparisons of each period against the first were tested using the Wald test. A 2-tailed significance of P < .05 was considered statistically significant. For the survey analysis, continuous data were presented using median and interquartile range (IQR) whereas categorical data were presented using number and percentage. Comparisons of categorical data were performed using the χ2 test or the Fisher exact test, as appropriate. Comparisons of continuous data were performed using the Student t test or ANOVA, as appropriate.
Results
The LOESS curves to describe study outcomes associated with the AOS during phases 1–5 are displayed in Figures 2–5. Figure 2 describes the primary outcome of the SAAR during phases 1–5. A statistically significant decrease in the SAAR was observed from phase 1 to phase 5: 1.0 (95% confidence interval [CI], 1.0–1.1) versus 0.90 (95% CI, 0.8–1.0; P< .001). This difference was largely driven by a reduction in the SAAR for broad-spectrum antibacterial agents primarily used for hospital-onset infections for patients on general medicine wards. Secondary outcomes of antibiotic DOT per 1,000 PD, targeted antibiotic orders, and CDI cases displayed in Figures 3, 4, and 5, respectively. From phase 1 to phase 5, a statistically significant decrease in the DOT per 1,000 PD was detected: 4,884 (95% CI, 4,491–5,277) versus 3,939 (95% CI, 3383–4,495; P = .001).
Figure 2. Standardized antimicrobial administration ratio evaluation. Solid blue line with grey shading is the LOESS curve. Dashed grey line indicates phase 1 (August 1, 2017–May 31, 2018). Dashed yellow line indicates phase 2 (August 1, 2018–May 31, 2019). Dashed red line indicates phase 3 (August 1, 2019–May 31, 2020). Dashed blue line indicates phase 4 (August 1, 2020–May 31, 2021). Dashed green line indicates phase 5 (August 1, 2021–May 31, 2022). Note. GLS, generalized least squares; LOESS, locally estimated scatterplot smoothing; SAAR, standardized antimicrobial administration ratio.
Figure 3. Antibiotic days of therapy per 1,000 patient days evaluation. Solid blue line with grey shading is the LOESS curve. Dashed grey line indicates phase 1 (August 1, 2017–May 31, 2018). Dashed yellow line indicates phase 2 (August 1, 2018–May 31, 2019). Dashed red line indicates phase 3 (August 1, 2019–May 31, 2020). Dashed blue line indicates phase 4 (August 1, 2020–May 31, 2021). Dashed green line indicates phase 5 (August 1, 2021–May 31, 2022). Note. GLS, generalized least squares; LOESS, locally estimated scatterplot smoothing.
Figure 4. Targeted antibiotic order evaluation. (A) Fluoroquinolones. (B) Carbapenems. (C) Clindamycin. (D) Vancomycin. Solid blue line with grey shading is the LOESS curve. Dashed grey line indicates phase 1 (August 1, 2017–May 31, 2018). Dashed yellow line indicates phase 2 (August 1, 2018–May 31, 2019). Dashed red line indicates phase 3 (August 1, 2019–May 31, 2020). Dashed blue line indicates phase 4 (August 1, 2020–May 31, 2021). Dashed green line indicates phase 5 (August 1, 2021–May 31, 2022). Note. GLS, generalized least squares; LOESS, locally estimated scatterplot smoothing.
Figure 5. Clostridioides difficile infection cases. Solid blue line with grey shading is the LOESS curve. Dashed grey line indicates phase 1 (August 1, 2017–May 31, 2018). Dashed yellow line indicates phase 2 (August 1, 2018–May 31, 2019). Dashed red line indicates phase 3 (August 1, 2019-May 31, 2020). Dashed blue line indicates phase 4 (August 1, 2020–May 31, 2021). Dashed green line indicates phase 5 (August 1, 2021–May 31, 2022). Note. GLS, generalized least squares; LOESS, locally estimated scatterplot smoothing.
Figure 4A–D describes targeted antibiotic orders during phases 1–5 with fluoroquinolones, carbapenems, clindamycin, and vancomycin representing Figure 4A, 4B, 4C, and 4D, respectively. From phase 1 to phase 5 in Figure 4A, a statistically significant reduction in fluoroquinolone orders was observed: 407 (95% CI, 350–464) versus 175 (95% CI, 94–256; P < .001). In Figure 4B, a statistically significant reduction in carbapenem orders was observed from phase 1 to phase 5: 147 (95% CI, 122–172) versus 106 (95% CI, 71–141; P = .024). In Figure 4C, a statistically significant reduction in clindamycin orders was observed from phase 1 to phase 5: 113 (95% CI, 92–135 versus 73 (95% CI, 42–103; P = .01). In Figure 4D, no statistically significant change was detected in vancomycin orders from phase 1 to phase 5: 991 (95% CI, 889–1,093 versus 902 (95% CI, 759–1,046; P = .221). The total signed antibiotic orders from the AOS for phases 2 (implementation phase), 3, 4, and 5 were 6,203, 10,800, 10,467, and 10,026, respectively. Figure 5 displays CDI cases during phase 1 through phase 5 where a statistically significant decrease in CDI cases was observed: 7.8 (95% CI, 10.1) versus 2.4 (95% CI, 5.8; P = .002). As mentioned previously, there was a change in CDI case diagnosis in February 2019 with PCR/EIA implementation. When comparing phase 3 to phase 5, no statistically significant difference in CDI cases were detected: 1.8 (95% CI, 1.6–5.1) versus 2.4 (95% CI, 1–5.8; P > .05).
In total, 58 of 247 clinicians completed the survey, for a response rate of 23.5%. Table 1 displays clinicians’ demographics and aggregated survey responses regarding perceptions of the AOS. Most respondents were trainee physicians (51.7%), had a practice area of internal medicine (44.8%), and encountered ≤20 patients per week with an infection (75.9%). On a 1-to-5 Likert scale, the median agreement was 4 (agree) or 5 (strongly agree) for all confidence and perception statements regarding the AOS. Clinicians strongly agreed that they were familiar with the AOS, that they were aware of the infections included, that they were confident in AOS location, that the AOS were easy to use, that the AOS helped them select the appropriate antibiotics including dosing and duration, and that they were well-informed about them. Furthermore, medicine and intensive care physicians indicated they were more familiar with the AOSs [median, 5.0 (IQR, 5.0–5.0) versus median, 4.0 (IQR, 4.0–5.0); P < .001] and were more likely to encourage colleagues to use the AOSs [median, 5.0 (IQR, 4.0–5.0) vs median 4.0 (IQR, 3.0–5.0; P = .013)] compared to surgery, emergency medicine, and other types of physicians, respectively. Clinicians agreed that their utilization of the AOSs increased throughout the previous year and that they encouraged their colleagues to utilize them.
Table 1. Clinician Demographics and Survey Responses Regarding Perceptions of the Empiric Antibiotic Order Sets (AOSs) (N = 58)
Note. CDC, Centers for Disease Control and Prevention; ICU, intensive care unit; IQR, interquartile range.
a Units unless otherwise specified.
b Based on a likert scale of 1–5: 1, strongly disagree; 2, disagree; 3, neither disagree or agree; 4, agree; 5, strong agree.
On a Likert scale of 1 to 5, the median agreement was 4 (agree) for statements regarding antimicrobial stewardship and antibiotic prescribing. Despite clinicians agreeing that they received adequate education or training to prescribe antibiotic appropriately and had a good understanding of antimicrobial stewardship. They also agreed that they were interested in receiving more antimicrobial stewardship education. Trainee physicians strongly agreed that they were aware of receiving a reminder via EPIC chat to encourage AOS use and found these reminders useful when admitting patients. Most clinicians (44.8%) indicated that they did not perceive any identifiable barriers to using the AOS. Of the potential barriers listed, the primary barriers selected by respondents were the differential diagnosis including >1 type of infection (13.8%) or attending physician preference is often not in alignment with the AOS (13.8%).
Discussion
To our knowledge, this is the largest, quasi-experimental study to evaluate the impact of AOS on antibiotic use metrics among hospitalized, adult patients. Implementation of AOS was associated with a statistically significant reduction in the SAAR and antibiotic DOT per 1,000 PD from the preimplementation period (phase 1) to 3 years after implementation (phase 5). We identified a nearly 50% reduction in the SAAR for broad-spectrum antibacterial agents predominately used for hospital-onset infections for patients on general medicine wards. This finding was not unexpected because the AOS for community-acquired infections (eg, CAP) steered clinicians away from broad-spectrum agents like cefepime and piperacillin-tazobactam in favor of ceftriaxone. There were also a statistically significant reductions in targeted antibiotics including fluoroquinolones, carbapenems, and clindamycin from the preimplementation period (phase 1) to 3 years after implementation (phase 5), but this reduction was not observed with vancomycin. It is unclear why there was no significant difference with vancomycin when reductions were demonstrated with other targeted antibiotics. The most significant reduction in DOT per 1,000 PD was observed for fluoroquinolones, whereas lesser reductions were seen for carbapenems and clindamycin. In addition to reductions in targeted antibiotic use, particularly with high-risk CDI antibiotics (eg, clindamycin, fluoroquinolones, carbapenems), a statistically significant difference in CDI cases was reported. However, this finding may be attributed to the change in our CDI testing algorithm. AOS utilization was generally consistent throughout phases 3–5, demonstrating that sustained usage supporting the changes in antibiotic use metrics were associated with the AOSs. Regarding clinician perceptions of the AOS, clinicians indicated that they were overall familiar with and found the AOS useful. Most indicated no perceived barriers to use.
Previous studies have evaluated AOS, but these have generally been limited to emergency departments, outpatient settings, or sepsis-specific protocols. Reference Hauck, Adler and Mulla7–Reference Vuong, Kenney and Thomson16,Reference Nichols, Petschke, Webber and Knoderer19–Reference Buehrle, Shively, Wagener, Clancy and Decker24 Few studies describe interventions for multiple infections among adult, hospitalized patients. Seitz et al Reference Seitz, Wiley, Francois, Moran, Rupp and Sexton25 implemented AOSs for cystitis, pyelonephritis, cellulitis, and chronic obstructive pulmonary disease based on clinical practice guidelines, their antibiogram, and desire to avoid antibiotics with higher CDI risk. They demonstrated improved antibiotic selection, first-dose timing, and prescription duration with nearly all providers indicating the orders sets were easy to use. Reference Seitz, Wiley, Francois, Moran, Rupp and Sexton25 However, these AOSs were limited to the emergency department and represented fewer infections than in our study. Colmerauer et al Reference Colmerauer, Linder, Dempsey, Kuti, Nicolau and Bilinskaya26 evaluated AOS implementation on broad-spectrum antibiotic DOT per 1,000 PD for CAP. The AOSs were associated with a statistically significant reduction in median broad-spectrum antibiotic DOT per 1,000 PD (2 days vs 0 days; P < .001). However, this was limited to only one infection, and the study was conducted over 2 months in pre- and postintervention periods. Reference Colmerauer, Linder, Dempsey, Kuti, Nicolau and Bilinskaya26 Chan et al Reference Chan, Kapur, Langford and Downing27 evaluated EMR-embedded AOS for cellulitis, UTI, and CAP. They reported statistically significant reductions in ciprofloxacin (mean, 16.6 DOT per 1,000 PD vs 13.6 DOT per 1,000 PD; P = .026) and moxifloxacin usage (mean, 9.3 DOT per 1,000 PD vs 5.2 DOT per 1,000 PD) during the study, which were consistent with the statistically significant reduction in fluoroquinolone use that we reported. Reference Chan, Kapur, Langford and Downing27
Demonstrations of the impact of AOS on antimicrobial stewardship metrics are needed. Our findings indicate that AOSs can indeed reduce antibiotic consumption. As such, our experience and data provide a model for other institutions depending on their resources and analysis availability. Our study had several strengths. First, the study period was over 5 years with a 3-year postimplementation period. Thus, we evaluated not only the initial impact but also the sustainability of this AOS intervention. Second, we evaluated specific antibiotic use metrics such as SAAR and antibiotic DOT per 1,000 PD, which are important for ASP monitoring and reporting. Both demonstrated statistically significant reductions. We also evaluated the impact of an order set that encompassed multiple infection types rather than one with a singular infection type.
This study had several limitations. First, this study was performed at a single center with a specific AOS design. Our AOS evaluation was limited to hospitalized adult patients; thus, results may not be generalizable to other institutions, the outpatient setting, or pediatric patients. Second, although we reported a reduction in several antibiotic use metrics, unknown confounding variables could have affected these changes outside the AOSs. However, the ASP did not introduce any novel antimicrobial stewardship initiatives and/or practices, and no changes in ASP staffing models occurred throughout the postintervention period. Third, although we included the AOS utilization numerical data, we did not directly measure AOS usage relative to the total number of antibiotic orders because changes to the orders in the form of dose adjustments, renewals, and intravenous to oral conversion could potentially have undermined the results. Furthermore, providers could have also ordered the appropriate antibiotic regimen correctly through education and awareness of the AOS, even if it was not directly ordered from the AOS; thus, the AOS utilization uptake could potentially appear falsely lower. Lastly, the survey responses could be at risk for response bias by clinicians as well as nonresponse bias because the response rate was relatively low.
In conclusion, our findings have demonstrated that implementing AOS into the EMR as an ASP initiative was associated with a statistically significant reduction in SAAR, antibiotic DOT per 1,000 PD, selected antibiotic orders (ie, fluoroquinolones, carbapenems, and clindamycin), and CDI cases among hospitalized adult patients. Clinicians indicated that the AOSs were easy to use overall and that they helped them select the appropriate antibiotics.
Acknowledgments
Financial support
This study was funded in part by a research grant from Investigator-Initiated Studies Program of Merck Sharp & Dohme. The opinions expressed in this paper are those of the authors and do not necessarily represent those of Merck Sharp & Dohme.
Competing interests
Wesley D. Kufel resports research grants from Merck, Sharp, & Dohme and Melinta Therapeutics, He serves on an advisory board for Theratechnologies. Stephen J. Thomas reports a research grant from Merck Sharp & Dohme. Jeffrey M. Steele serves on an advisory board for Paratek Pharmaceuticals. All other authors have nothing to disclose.
Open access
