Pandemic hits: Evaluation of an antimicrobial stewardship program website for hospital communication during the coronavirus disease 2019 (COVID-19) pandemic

Antimicrobial stewardship programs (ASPs) took on key roles in response to the coronavirus disease 2019 (COVID-19) pandemic, including development of local treatment guidelines, access to COVID-19 therapeutics, and ensuring appropriate use. ^1–3 Although prior research has explored implementation of ASP guidance documents, few ASPs review utilization of their educational resources or measure efforts required for maintenance. ⁵ Understanding which tools are most utilized may facilitate efficient deployment of limited stewardship resources. Web analytics can evaluate the use of online resources and may be helpful in guiding ASP efforts. ^6,7

Our ASP utilizes a web-based application (CustomID) as the primary means of efficient communication with clinicians in our hospital. ⁸ Duke CustomID includes institution-specific guidelines for treatment of various infections, local antibiograms, policies for restricted antimicrobials, and more. The resource is maintained directly by our ASP pharmacists and physicians, allowing for rapid updates. The application is readily accessible from all standard web browsers and is optimized for use on mobile devices. Users are primarily prescribers [ie, medical doctors (MDs and DOs), physician assistants, and nurse practitioners] and pharmacists who must have access to the local network to access the site.

We sought to quantify longitudinal trends in user engagement with CustomID in the context of the COVID-19 pandemic and to estimate engagement relative to effort required to maintain COVID-19–specific content. We hypothesized that the increased COVID-19 caseload would increase user engagement.

Methods

This retrospective, descriptive time-series analysis characterized engagement with CustomID during a 1-year prepandemic period through the severe acute respiratory coronavirus virus 2 (SARS-CoV-2) omicron surge, from January 2019 to March 2022. The analysis included data from Duke University Hospital, a 1,000-bed academic medical center served by a large ASP. To evaluate engagement relative to the COVID-19 caseload, we collected weekly data on total hospitalizations, COVID-19 hospitalizations, and positive SARS-CoV-2 tests at our hospital and affiliated outpatient sites. Google Analytics (analytics.google.com) was used to measure page-level views or “hits” across the platform and for COVID-19–specific pages on a weekly basis. Given fluctuations in hospitalization rates, we defined the primary outcome as the rate of hits divided by total hospitalizations.

COVID-19–specific pages were categorized into management pages, which consisted of treatment guidelines and drug pages that included information on COVID-19–specific therapeutics. To estimate the effort required to maintain these resources, we recorded the number of days each COVID-19–specific page required editing and looked at hit-to-edit ratios for each COVID-19 page.

Weekly data were assessed graphically with page hits, positive SARS-CoV-2 tests, and COVID-19 hospitalizations. We used negative binomial regression to quantify the association between COVID-19 hospitalizations and hit rates and to measure the trend engagement over time, adjusted for seasonality. Hits, edits, and hit-to-edit ratios for each COVID-19 page were evaluated using descriptive statistics.

Results

During the prepandemic period, COVID-19–specific page views had a median of 1,707 hits per week (range, 1,165–2,354) and hit rate median of 1.95 per hospitalization (range, 1.40–2.86). CustomID engagement increased over time; spikes were observed at the beginning of the pandemic and during subsequent COVID-19 surges (Fig. 1). The highest peaks occurred in March 2020 and January 2022, with hit rates of 4.59 and 3.87, respectively. Engagement increased over the study period by 15% (relative rate of week 1 vs week 170, 1.15; 95% confidence interval [CI], 1.02–1.28; P = .02). On average, for every 100 COVID-19 hospitalizations, the hit rate increased by 0.08 (95% CI, 0.004–0.16; P = .04).

Fig. 1. Duke CustomID hits and maintenance efforts over the COVID-19 pandemic. Top: COVID-19-specific CustomID hits per week (black), positive COVID-19 tests per week (gray) over time. Middle: Total CustomID page hits relative to total hospitalizations per week (black), COVID-19 hospitalizations (gray). Bottom: Number of edits to COVID-19–specific CustomID pages per week, stratified by management pages and drug pages. Several dates of significance are highlighted including the emergency use authorizations (EUAs) for remdesivir (Veklury), the COVID-19 vaccines, and oral antivirals (ie, molnupiravir [Lagevrio] and nirmatrelvir/ritonavir [Paxlovid]).

Page maintenance for COVID-19–specific pages over the pandemic period included 329 revisions with a range of 0–12 per week (median, 2) (Fig. 1). Increased page edits occurred during COVID-19 surges, national guideline updates, and novel therapeutic authorizations. Specific pages had high variability in the number of page edits required. Research protocol and monoclonal antibodies pages required frequent updates (133 and 50, respectively), but COVID-19 prophylaxis and molnupiravir pages required <5 edits each. Guideline pages generally received higher page views but were often more labor intensive to maintain than drug pages (Supplementary Table 1 online).

Discussion

Our ASP successfully utilized a pre-existing web application to rapidly distribute emerging recommendations and increase utilization of resources created by our ASP. Engagement was significantly associated with COVID-19 caseload. We believe that this communication tool was essential in our institution’s pandemic response and that overall efforts to maintain it were justified.

Previous evaluations of web-based tools for antimicrobial stewardship have shown that clinician utilization may be associated with improved adherence to guideline recommendations. ⁵ Another ASP reported launching guidelines via a smartphone “app,” and hits were graphically associated with COVID-19 activity. ³ Our experience adds information about longitudinal trends in overall engagement and maintenance needs at the page-content level. Our study also demonstrates the potential for web analytics to be incorporated into assessments of stewardship educational content. Prioritizing high-impact interventions and communication strategies is important, especially given the limited resources for ASPs and limited attention from clinicians. ^9,10 If data had been monitored in real time, our ASP may have chosen to discontinue certain pages earlier, to redirect users, or to adjust content for more efficient delivery.

Our analysis had several limitations. Our experience is from a single, large, academic center. We evaluated user engagement, but we did not assess guideline adherence or appropriateness. Additionally, page edits were a limited metric for the total time invested per edit. Nevertheless, we felt the speed and accessibility of these simple metrics provided utility for application in our stewardship program.

Hospital ASPs are already providing customized clinical guidance for frontline clinicians and are uniquely suited to support institutional pandemic response. Future opportunities to use web analytics in ongoing evaluation and reassessment of ASP educational content may help optimize the deployment of ASP resources.