Mixed-methods process evaluation of a respiratory-culture diagnostic stewardship intervention

Abstract

Objective:

To conduct a process evaluation of a respiratory culture diagnostic stewardship intervention.

Design:

Mixed-methods study.

Setting:

Tertiary-care pediatric intensive care unit (PICU).

Participants:

Critical care, infectious diseases, and pulmonary attending physicians and fellows; PICU nurse practitioners and hospitalist physicians; pediatric residents; and PICU nurses and respiratory therapists.

Methods:

This mixed-methods study was conducted concurrently with a diagnostic stewardship intervention to reduce the inappropriate collection of respiratory cultures in mechanically ventilated children. We quantified baseline respiratory culture utilization and indications for ordering using quantitative methods. Semistructured interviews informed by these data and the Consolidated Framework for Implementation Research (CFIR) were then performed, recorded, transcribed, and coded to identify salient themes. Finally, themes identified in these interviews were used to create a cross-sectional survey.

Results:

The number of cultures collected per day of service varied between attending physicians (range, 2.2–27 cultures per 100 days). In total, 14 interviews were performed, and 87 clinicians completed the survey (response rate, 47%) and 77 nurses or respiratory therapists completed the survey (response rate, 17%). Clinicians varied in their stated practices regarding culture ordering, and these differences both clustered by specialty and were associated with perceived utility of the respiratory culture. Furthermore, group “default” practices, fear, and hierarchy were drivers of culture orders. Barriers to standardization included fear of a missed diagnosis and tension between practice standardization and individual decision making.

Conclusions:

We identified significant variation in utilization and perceptions of respiratory cultures as well as several key barriers to implementation of this diagnostic test stewardship intervention.

Ventilator-associated infections (VAIs), including ventilator-associated tracheitis (VAT) and pneumonia (VAP), are among the most common indications for antibiotic use in the pediatric intensive care unit (PICU). ^1,2 Although some of this antibiotic use is warranted, up to half of antibiotics prescribed for VAIs in children are inappropriate. ² One driver of antibiotic overuse is the imprecision of respiratory cultures in differentiating bacterial colonization from infection; endotracheal and tracheostomy tubes are nearly universally colonized with potentially pathogenic bacteria soon after placement. ^3–6 Therefore, a “positive” respiratory culture indicating colonization may be misinterpreted as evidence of infection, contributing to inappropriate antibiotic use in patients who do not have a VAI.

Microbiologic diagnostic test stewardship, in which the practice of ordering cultures is modified to reduce the number of cultures that are ordered absent evidence of infection, have consistently reduced culture utilization, with inconsistent reductions in antimicrobial use. ^7–10 Several studies have evaluated the determinants of uptake of antimicrobial stewardship interventions, but few have focused on diagnostic test stewardship interventions, particularly in the PICU setting. ^11–16 These data are fundamental to optimizing the implementation of diagnostic test stewardship interventions. ^17,18 Therefore, we performed a mixed-methods process evaluation concurrent with a diagnostic-test stewardship intervention focused on reducing inappropriate respiratory-culture orders in our tertiary-care PICU.

Methods

Study design, sample, and recruitment

Our diagnostic-test stewardship intervention utilized a guideline defining indications for collecting a respiratory culture in our tertiary-care PICU, which was created using a multidisciplinary consensus-based process (Supplementary Fig. 1 online). In the first phase of the process evaluation, we used quantitative methods to characterize the indications for respiratory culture orders as well as variability in culture utilization across clinicians between September 2019 and August 2020. In the second phase, we conducted semistructured interviews of PICU clinicians between March and July 2021, including attending physicians, fellows, and nurse practitioners and hospitalists (Fig. 1). We utilized a purposive sampling strategy to sample PICU attending physicians in the highest and lowest quartile of respiratory culture utilization (Supplementary Fig. 2 online). Because we were not objectively able to quantify utilization among non–attending physicians, we randomly sampled these groups through a series of 3–4 e-mails. Recruitment stopped when thematic saturation was achieved. ¹⁹

Fig. 1. Process evaluation timeline relative to diagnostic test stewardship intervention.

In the third phase of this study, we conducted a survey including PICU clinicians (attending physicians, fellows, nurse practitioners, and hospitalist physicians), infectious diseases (ID) clinicians (attending physicians and fellows), pulmonary clinicians (attending physicians and fellows), pediatric residents who completed their PICU rotation in the 4 months prior to conducting the survey, and PICU nurses and respiratory therapists. Completion of the survey was voluntary, and respondents were invited to participate via a series of 2 e-mails. The survey was administered using Research Electronic Database Capture (REDCap) software in October 2021 (Fig. 1). This study was classified as exempt research by the Chidren’s Hospital of Philadelphia (CHOP) Institutional Review Board.

Data collection and instruments

In the first phase of our process evaluation, we quantified baseline variability in indications for respiratory culture collection by reviewing all respiratory cultures ordered in patients mechanically ventilated for >48 hours in the 1 year prior to the intervention. Presence of fever, hypothermia, change in secretion quality or quantity, chest radiograph infiltrate (determined by radiologist’s interpretation), any change in positive end expiratory pressure (PEEP) or fraction of inspired oxygen (FiO₂), or combination of these findings were assessed in the 48 hours prior to culture collection by chart review. Variation in culture collection across attending physicians was assessed by measuring the number of cultures collected per day of clinical service worked.

We developed the interview guide for the semistructured interviews using a combination of our baseline quantitative data, questions derived from the Consolidated Framework for Implementation Research (CFIR), and themes from the published literature related to diagnostic test stewardship. The CFIR is a pragmatic meta-theoretical framework consisting of 5 domains, each with several constructs that influence effective implementation. Key CFIR domains include the following: characteristics of the individual (knowledge or beliefs and self-efficacy), intervention characteristics (evidence strength and quality and relative advantage), and inner setting (culture and implementation climate). ²⁰ Interviews were conducted during the second phase of the process evaluation by trained study-team members experienced in conducting qualitative interviews (J.W. and D.M.). The interview guide was refined for clarity after a pilot interview. All interviews were recorded and transcribed prior to analysis with consent from participants.

Closed-ended survey questions were developed based on themes identified in the semistructured interviews and in the published literature. In addition, to classify individuals as higher or lower utilizers of respiratory cultures, we asked respondents to rate the likelihood of sending a respiratory culture in 3 controversial clinical scenarios. Respondents were then ranked into quartiles, with the lowest quartile including clinicians least likely to order respiratory cultures and the highest quartile including clinicians most likely to order respiratory cultures. The survey instrument was piloted by a group of 4 physicians, 4 nurses, and a respiratory therapist. Modifications were made after this pilot testing, and the final instrument, distributed in the third part of the process evaluation, contained 27 closed-ended questions with 4- or 5-point Likert scale responses and 2 open-ended questions (Supplementary Table 1 online).

Table 1. Indications for Respiratory Culture Orders

Data analysis

Interview data and open-ended survey questions were analyzed using an inductive approach to thematic analysis. Beginning with familiarization, team members reviewed the interview (J.W. and D.M.) and survey data (K.C.), identified and applied codes emerging from data, and lastly, the full team (J.W., D.M., K.K., and K.C.) generated and refined the resultant, triangulated themes. Quantitative data were analyzed using descriptive statistics, including frequencies and percentages, using Stata statistical software (StataCorp, College Station, TX). For the purposes of analyzing data by groups, physician and nurse practitioner survey respondents were classified by primary specialty and by role, including attending physicians, trainees (including residents and fellows), and nurse practitioners and PICU hospitalists.

Results

Quantitative process evaluation

In total, 625 respiratory cultures were ordered in the 1 year prior to guideline implementation. Indications included the following: isolated fever or hypothermia (124 cultures, 20%), fever and any change in PEEP or FiO₂ (71 cultures, 11%), and isolated change in PEEP or FiO2 (67 cultures, 11%) (Table 1). The frequency of respiratory culture orders varied across critical-care attending physicians between 2.2 and 27 respiratory cultures per 100 clinical days (Supplementary Fig. 2 online).

Semistructured interviews

In total, 14 interviews were performed: 7 with attending physicians, 4 with fellows, and 3 with PICU nurse practitioners or hospitalists. Themes that emerged from these interviews included individual knowledge and beliefs about respiratory cultures, decision making about respiratory culture ordering, standardization of practices in the PICU, and the culture of implementation and impact of the intervention (Table 2).

Table 2. Themes Identified in Semistructured Interviews and Sample Quotes

Survey

In total, 87 clinicians (response rate, 47%) and 77 PICU nurses and respiratory therapists (response rate, 17%) completed the survey. Response rates were highest among critical care and infectious diseases attending physicians, followed by critical care and infectious diseases fellows (Table 3). Respondents were ranked into quartiles based on their stated likelihood of ordering a respiratory culture in response to 3 hypothetical scenarios according to a 4-point Likert scale (Supplementary Table 1 online). Most infectious diseases clinicians (76%) fell into the 2 quartiles least likely to order a respiratory culture, whereas most pulmonary clinicians fell into the 2 quartiles most likely to order a respiratory culture (91%).

Table 3. Survey Response Rate by Specialty and Role

Salient themes

Findings related to the key themes identified in the semistructured interviews and further explored in the survey (Supplementary Table 1 online), including individual knowledge and beliefs about respiratory cultures, decision making around culture ordering, standardization around respiratory culture ordering practices, and the culture of implementation, are summarized below.

Knowledge and beliefs about respiratory cultures (CFIR domain: characteristics of individuals)

Interview respondents noted significant variation in clinician practices regarding ordering and interpreting respiratory cultures. For example, fever alone was noted to be a sufficient trigger for ordering a culture for some clinicians whereas others questioned the value of sending a culture in this particular scenario. Uncertainty as to whether a positive culture should be interpreted as evidence of infection warranting antibiotic treatment was also noted as a challenge in utilizing the respiratory culture as a diagnostic test.

Moreover, 75% of attending physicians, 76% of trainees, and 80% of PICU nurse practitioners and hospitalists who responded to the survey felt that respiratory cultures were overutilized, whereas 20% of nurses and 29% of respiratory therapists felt that respiratory cultures were overutilized. Also, 80% of PICU clinicians and 100% of ID clinicians felt that respiratory cultures were overused, which was a much greater proportion than pulmonary clinicians (Table 4). Finally, clinicians who ranked in the quartile least likely to order a respiratory culture more often agreed that respiratory cultures were overutilized compared to those most likely to order a respiratory culture (96% vs 55%).

Table 4. Knowledge and Beliefs About Respiratory Culture Ordering and Standardization

Note. PICU, pediatric intensive care unit.

^a “Strongly agree” and “agree” were collapsed into “agree,” and “disagree” and “strongly disagree” were conmbined into “disagree.”

Consistent with the noted variability in interpretation of respiratory cultures, clinicians were divided as to whether a Gram-stain positive for moderate or many white blood cells indicated a bacterial infection: 41% strongly agreed or agreed, 59% disagreed or strongly disagreed. Similarly, opinions varied as to whether a culture positive for Pseudomonas aeruginosa in a patient with increased and thick respiratory secretions was suggestive of bacterial infection: 47% strongly agreed or agreed and 53% disagreed or strongly disagreed. When stratified across quartiles of utilization, individuals more likely to order a respiratory culture more often interpreted both gram stains and cultures as suggestive of infection and were more likely to endorse that respiratory cultures had greater utility in the diagnosis and management of ventilator-associated infection (Table 5).

Table 5. Knowledge and Beliefs About Interpretation of Respiratory Cultures by Quartiles of Utilization

Note. WBC, white blood cell.

^a Quartile 1 includes clinicians least likely to order a respiratory culture based on responses to hypothetical scenarios included in the survey.

^b Quartile 4 includes clinicians most likely to order a respiratory culture based on responses to hypothetical scenarios included in the survey.

Decision making around ordering respiratory cultures (CFIR domains: characteristics of individuals, inner setting). Ordering respiratory cultures as a “default” practice was noted by several interview respondents, particularly in response to fever. In addition, a culture of “fear of missing something” was cited as influencing all practices in the PICU, including respiratory culture ordering. Finally, the actual or perceived opinions of the PICU attending physicians influenced the decision making of the PICU hospitalists, nurse practitioners, and fellows. Many respondents felt overruled by attending physicians in situations in which they did not feel that a respiratory culture was indicated. Responses from attending physicians regarding the role of hierarchy acknowledged that while decisions around whether a culture is ordered ultimately rest with the attending, most often this decision is made by non-attending clinicians. Several attending physicians also reported that cultures were collected in scenarios in which justification for ordering a culture was insufficient.

Using this survey, we explored drivers of individual decision making. Personal views of the value of a respiratory culture in a given scenario were most influential for attending physicians, trainees, nurses, and respiratory therapists. In contrast, expectations of attending physicians of one’s own specialty was the most cited influence on culture ordering for nurse practitioners and hospitalists. Institutional guidelines, the focus of the concurrent diagnostic test stewardship project, were consistently influential across provider types, particularly trainees. Finally, clinicians were less often influenced by parental concerns regarding testing, whereas nurses and respiratory therapists were more likely to be influenced by parental concerns (Table 6).

Table 6. Drivers of Respiratory Culture Ordering by Role

Note. NP, nurse practitioner; NA, not applicable.

^a Strongly and moderately influences my decision was categorized as “influential.”

^b Mildly or no influence was classified as “not influential.”

Standardization of practices within the PICU (CFIR domain: inner setting, intervention characteristics). Many interview respondents reported a concern that standardization may remove clinician autonomy necessary to care for a medically complex patient population. Therefore, the need to recognize scenarios in which a clinician should deviate from a guideline recommendation was highlighted by many respondents. In addition, a concern that overreliance on standardized guidelines would lead to missed diagnoses was highlighted as a potential risk.

Across specialties and roles, physicians and nurse practitioners agreed that standardization of respiratory cultures was a priority and of benefit to both clinicians and patients, though few endorsed that standardization would be easy. In contrast, nurses were less likely to feel that standardization would be beneficial (Table 4). Salient themes cited as advantages to standardization included reducing inappropriate antibiotic use, consistency across members of the treatment team, reducing cost and/or resource utilization, improving efficiency around decision making, and improving the diagnosis of VAI. Disadvantages to standardization included limited individual decision making, fear of missing an infection, complexity of individual patients in the PICU, and concerns that standardizing practices may increase antibiotic use and/or prompt more cultures to be collected.

Culture of implementation and impact (CFIR constructs: inner setting, intervention characteristics). The local culture was generally felt to be receptive to changes in practice and implementation of this guideline, though several barriers were noted. First, multiple other guidelines and quality improvement projects were being implemented simultaneously. Coupled with the baseline high workload in the PICU, prioritizing this intervention was a challenge. Second, given the practice variation regarding ordering respiratory cultures across clinicians at baseline, uptake of the guideline was felt to be variable. Finally, as several clinicians discussed, although the specifics of the guideline were complex and may not be memorable, having the guideline in place prompted them to be more judicious and to consider how respiratory culture testing would change management. Clinicians also noted that fewer cultures to interpret might contribute to greater efficiency given the challenges in determining whether a respiratory culture reflected infection or colonization.

Discussion

We conducted a mixed-methods process evaluation concurrent with a diagnostic test stewardship intervention focused on reducing the overuse of respiratory cultures in a tertiary-care PICU. Quantitative data demonstrated variable perceptions of the utility of respiratory cultures as well as drivers of culture orders across clinicians, factors that may cluster by specialty or role. Qualitative data further highlighted that practice variation was perceived by individuals; that group “defaults,” hierarchy, and fear influenced decision making; and that patient complexity and fear of missed diagnoses were challenges to standardization, despite broad agreement that standardizing respiratory-culture ordering practices would be beneficial for patients and clinicians. These findings support several key conclusions and expand upon the limited literature published to date related to antibiotic and diagnostic test stewardship in the PICU setting.

First, variable respiratory-culture ordering practices appear to be driven by both individual-level variation as well as variation across specialties. Absent evidence-based guidelines defining when a respiratory culture should or should not be sent, individual beliefs and specialty-specific culture may therefore be primary drivers of clinical practice. ²¹ Given that such guidelines are unlikely to be forthcoming, diagnostic-test stewardship interventions must acknowledge and define these multilevel influences on culture ordering practices. This acknowledgment is particularly important in the PICU practice setting, where medically complex patients are often cared for by a multispecialty and interprofessional teams. ²²

Second, our semistructured interviews demonstrated that cultural factors, including perceived “norms” or “default practices,” as well as hierarchy within clinician group influence culture ordering practices. For example, ordering a culture in response to isolated fever was a common “default” practice, although several interview respondents acknowledged that this practice was often low yield. Similar findings were demonstrated in a study of blood-culture ordering practices in the PICU. This element of “testing etiquette” may influence PICU clinician behavior, similar to the more familiar phenomenon of antibiotic “prescribing etiquette,” in which local culture defines expected practice. ^12–14 However, and a novel finding of this study, is that there may be a mismatch between actual and perceived attending expectations, suggesting that attending support of diagnostic test stewardship interventions may facilitate uptake among non–attending physicians.

Third, fear of a missed diagnosis, both related to individual decision making and standardizing unit-wide practices, was a prominent theme, consistent with the limited published literature related to antibiotic and diagnostic test stewardship in the ICU setting. ^13–15 However, in the case of respiratory cultures, this fear may be misplaced given that respiratory cultures perform poorly as a diagnostic test. Furthermore, actionable results from respiratory cultures are not available for 24–72 hours after the culture is ordered, such that clinicians must often make an initial diagnosis and take therapeutic action based on other clinical data. Future work should therefore explore the unique influences of fear on decision making regarding diagnostic tests, which are likely distinct from those driving treatment decisions.

Finally, while clinicians agreed that standardization of practices regarding ordering a respiratory culture is beneficial, barriers included concerns around a perceived loss of clinician autonomy and fear of missing a diagnosis. These findings are aligned with a qualitative study demonstrating unique barriers and facilitators to implementation of practice changes in the PICU, including the tension between standardization and clinician autonomy. ²² This issue may be exacerbated in the case of respiratory cultures, where the evidence base informing optimal criteria for testing is limited.

Our study had several limitations. First, the single-center design may limit generalizability and transferability, given that many local cultural and contextual factors were highlighted during this process evaluation. However, the consistency of our findings with related work strengthens our conclusions, which may generalize best to tertiary-care PICUs with multispecialty teams and trainees. Second, our semistructured interviews were limited to a small number of respondents, and only critical-care clinicians were included. We sought to overcome this limitation by including a multispecialty and multiprofessional population in our survey. Furthermore, we achieved thematic saturation despite our relatively small number of interviews, an observation supported by published data. ¹⁹ Third, assessments of individual culturing practices were based on stated, rather than observed, practices. This was unavoidable given our study design and the fact that only critical-care clinicians place orders in our PICU. Finally, our process evaluation took place several months after guideline implementation, so it is possible that the guideline itself influenced reported clinician views of the respiratory culture diagnostic test. However, because our goal was to simultaneously evaluate attitudes toward respiratory cultures as well as the guideline itself, this study timeline was necessary.

Overall, this process evaluation provides novel insights into clinician perceptions of respiratory cultures as well as a diagnostic test stewardship intervention to reduce inappropriate ordering of respiratory cultures. Based on our findings, attending physician support for stewardship interventions, engagement from subspecialty stakeholders, and implementation strategies focused on standardizing practice may facilitate uptake by promoting a culture of stewardship. Future studies should explore differences in determinants of ordering respiratory cultures across specialties and roles, the influence of fear and emotions on culture practices, and optimal strategies for promoting uptake of evidence-based practices related to ordering respiratory cultures.