A Randomized Controlled Trial of the Effectiveness of Traditional and Mobile Public Health Communications With Health Care Providers

Abstract

Objectives

Health care providers play an essential role in public health emergency preparedness and response. We conducted a 4-year randomized controlled trial to systematically compare the effectiveness of traditional and mobile communication strategies for sending time-sensitive public health messages to providers.

Methods

Subjects (N=848) included providers who might be leveraged to assist with emergency preparedness and response activities, such as physicians, pharmacists, nurse practitioners, physician’s assistants, and veterinarians. Providers were randomly assigned to a group that received time-sensitive quarterly messages via e-mail, fax, or cell phone text messaging (SMS) or to a no-message control group. Follow-up phone interviews elicited information about message receipt, topic recall, and perceived credibility and trustworthiness of message and source.

Results

Our main outcome measures were awareness and recall of message content, which was compared across delivery methods. Per-protocol analysis revealed that e-mail messages were recalled at a higher rate than were messaged delivered by fax or SMS, whereas the as-treated analysis found that e-mail and fax groups had similar recall rates and both had higher recall rates than the SMS group.

Conclusions

This is the first study to systematically evaluate the relative effectiveness of public health message delivery systems. Our findings provide guidance to improve public health agency communications with providers before, during, and after a public health emergency. (Disaster Med Public Health Preparedness. 2016;10:98–107)

Critical partners in the public health emergency preparedness and response system infrastructure are health care providers (HCPs) who, before, during, and after a suspected or confirmed public health emergency, can be called upon to treat the injured, prevent excess deaths, and mitigate suffering. ^{1 - 3} Important objectives of communication between public health agencies and HCPs exist at every stage of a public health emergency, including emergency planning, early event detection, situational awareness, emergency response capacity, and communication with the public during an emergency. HCPs serve as “eyes on the ground” for public health surveillance, situational awareness, and case detection ^{4 , 5} and are considered trusted and preferred communicators of health information to the public during emergencies. ^{6 , 7}

The importance of HCPs in emergency preparedness and response has become evident with public health emergencies such as the 1999 West Nile virus outbreak ⁸ and 2001 inhalation anthrax cases. ⁹ While “medical preparedness requires better communication among physicians and between public health communities,” ^{10 (p12)_} this objective requires systems that ensure public health communications to HCPs are delivered, received, deemed credible, and actionable. ^{11 , 12} Public health after-action and “lessons learned” reports consistently identify communication system breakdowns as the most common source of failure in most emergency events, disaster response, and emergency and preparedness exercises, ^{13 , 14} and a lack of communication and coordination between public health and key partners, such as health care organizations and HCPs, continues to be documented. ^{13 , 15 - 21}

The emergency and disaster communication infrastructure in the United States has improved with systems such as the Epidemic Information Exchange ²² and Health Alert Network, ²³ which aim to enhance health tracking, facilitate sharing of health surveillance information, and support communications during emergencies. ²⁴ However, an increasing assortment of diverse systems can inundate HCPs with multiple, redundant, and conflicting messages disseminated through national, state, local, and professional communication channels that can engender “alert overload.” ^{25 , 26} To reduce, rather than increase, communication challenges between public health and HCPs requires undertaking a systematic approach to updating communication strategies.

The most recent 2013 National Association of County and City Health Officials report profiling local health departments (LHDs) noted that “LHDs rely heavily on traditional channels for communication, such as e-mail alert systems (70%), broadcast fax (57%), and automated phone calling (52%).” ²⁷ These traditional modalities can be problematic communication strategies: public health agencies may be unable to document whether a message has been delivered, fax machines can jam, phone lines are disrupted, and blasting information by e-mail or fax may depend on staff noticing a message and distributing it appropriately. Specific to e-mail are the issues that most public health e-mail distribution lists require opting in to receive messages, undelivered or bounced e-mail may not be easily resent, and maintaining correct contact information is time-consuming for public health agency staff.

The range of communication strategies and solutions for getting time-sensitive information to the right person at the right time is expanding with mobile modalities such as cell phone text messaging (SMS). Currently, US cell phone ownership exceeds 91% among US adults, with 58% owning a smartphone. ²⁸ Eighty percent of cell phone owners use SMS to communicate, ²⁹ sending or receiving an average of 10 messages daily. ³⁰ Specific to HCPs, a 2011 survey of US physicians estimated smartphone ownership at 85% with 63% of respondents reporting use of mobile applications in their clinical work. Residents and fellows had higher rates of smartphone and mobile application use than more experienced attending physicians, which suggests that younger physicians are more likely to integrate mobile technologies into their work. ³¹ A 2012 study of medical interns reported that over 94% of survey respondents owned a smartphone, with over 83% of the smartphone owners reporting they made or received phone calls, over 87% sent or received text messages, and over 41% sent or received e-mails on their device for work purposes only. ³² Adoption of smartphones among HCPs is projected to increase, ^{33 , 34} and some medical schools are now requiring or distributing smartphones for their students. ³⁵

However, SMS is not a common public health strategy for engaging with health care and it is unknown how to maximize the likelihood that a time-sensitive SMS message is not only received but deemed credible, and, when appropriate, acted upon by its target audience. In 2008, the Institute of Medicine proposed that, to meet present-day and future information needs for emergency preparedness, more attention needs to be given to evaluating the effectiveness of public health systems in a scientifically rigorous manner. ³⁶

Few studies to date have used comparative methods to determine the best techniques for working with clinicians on issues pertinent to public health emergencies. The REACH (Rapid Emergency Alert Communications in Health) study is a multi-site, randomized controlled trial (RCT) to systematically evaluate and compare the effectiveness of traditional (e-mail and fax) and novel (SMS) communication strategies for sending public health messages to HCPs. The primary aim was to investigate which communication modalities are most effective in real-world situations. Our approach was informed by McGuire’s communication/persuasion matrix model, ^{37 , 38} which stipulates that the effectiveness of a communication strategy depends in part on (1) whether the disseminated communications raise the awareness of the recipients in the desired direction, and (2) whether the modified awareness in turn influences recipients’ behaviors (detailed in Figure 1).

Figure 1 Communication/Persuasion Matrix Model Sequence. Adapted from McGuire, 2013. ³⁷

Traditionally, the HCP contact lists utilized by public health agencies to disseminate messages are created by an “opt-in” process, ie, the HCP provides phone, fax, and e-mail information upon enrollment and public health agencies rely on the HCP to update this contact information when needed. An additional real-world condition is that there is no action on the part of public health agencies when a message bounces or cannot be delivered, other than resending the information. Given our primary aim to investigate which communication modalities are most effective in real-world situations, we sought to replicate these conditions as much as possible in our study design. Thus, enrolled HCPs were included in the study and sample size regardless of ability to receive messages through the communication channel to which they were randomly assigned.

Under normal conditions, retention of a message will decay over time, thus influencing the persuasiveness and recall of the message. We sought to identify whether communication channel impacted recalling the message topic after a short period of time (5 to 10 days) and whether a novel communication channel (SMS) impacted retention of the message. Because public health agencies in general and our participating study sites specifically did not utilize SMS, the additional communication channel provided the opportunity to study how the target HCP audience responded to a novel form of public health communication.

Although an RCT is the gold standard for assessing efficacy, a pure intention-to-treat—the “once randomized, always analyzed”—analysis approach can introduce interpretative problems owing to the inclusion of any deviations in protocol after randomization, such as noncompliance, withdrawals, and missing outcomes data. ³⁹ A recommendation for overcoming these issues is to modify the intention-to-treat approach by conducting a per-protocol (PP) analysis in which the subset of the intention-to-treat population who completed the study without any major protocol violations are included, thus more accurately reflecting actual treatment differences among groups. ^{40 , 41}

In this article, we report the primary outcomes of REACH, the first RCT to systematically evaluate the effectiveness of SMS, e-mail, and fax for delivery of time-sensitive public health messages to HCPs.

Methods

Overall Study Design

The REACH study was a multi-site RCT with a 1:1:1:1 allocation ratio conducted between 2009 and 2012 in a partnership between the University of Washington, Seattle, and 3 public health agencies in the US Pacific Northwest. Providers who might contribute to emergency preparedness and response activities and were potential first points of public contact during an emergency for information or care were invited to participate in each study site: primary care physicians, including family medicine, pediatrics, internal medicine, general practice, infection control, and emergency medicine specialties; nurse practitioners; physician’s assistants; pharmacists; and veterinarians. The trial had 4 intervention arms: e-mail, fax, SMS, and a no-message control group. Providers were blindly and randomly assigned to communication group regardless of whether they were capable of receiving messages through their assigned modality; ie, a participant who did not provide a cell number when enrolling was as likely to be assigned to the SMS group as one who did provide a cell number.

For 6 to 12 months, depending on site, 3 to 4 time-sensitive messages based on real events of public health interest were sent on behalf of the site’s public health agency to HCPs through their allocated delivery method. All messages, regardless of format, included a link to a web page with additional information on the message topic. All participants, including HCPs in the control group, were contacted 5 to 10 days after the message delivery date for follow-up telephone interviews regarding the message. As stated above, if a provider’s contact information changed during the course of the study, messages continued to be sent to the original number. This allowed us to conduct a PP analysis of recall as described above, in which study participants were maintained in the groups to which they were randomized regardless of known message delivery failures or inability to receive messages. A secondary as-treated (AT) analysis in which study participants were included only if it was known that they were able to receive messages through their assigned communication group method and there were no known delivery failures was conducted to evaluate correct message topic recall only among participants that received messages.

Ethics

The study protocol received approval (Minimal Risk Category 7) from the University of Washington Institutional Review Board. The REACH Trial is not registered with clinicaltrials.gov because the protocols were determined to be outside the scope for RCT registration.

Site Selection

The trial was conducted in partnership with Public Health Seattle & King County, the Montana Department of Public Health & Human Services, and the Spokane County Health Department. The sites were chosen to represent a diverse range of population densities and demographics, health care workforces, potential natural hazards, and agency organizational structures. Sites were required to have a sufficient census of HCPs within their jurisdictions to meet study recruitment targets. Site participation was staggered.

Sample Size

Sample size was calculated to ensure detection of any differences between randomization groups using pairwise comparisons over repeated observations. Calculations were based on the following assumptions: 35% baseline success rate for intervention groups, 44% within-group standard deviation, possible 20% follow-up failure rate, and repeated observations (3 to 4 samples from each HCP over 3 study sites). Assuming 80% power at a 0.05 significance level to detect at least a 15% difference in frequency of receipt of messages, the study required a target sample size of 207 individuals per randomization group (minimum total sample size N=828).

Recruitment and Enrollment

Listings of active HCPs were acquired from state licensing databases. Participants were excluded if they practiced outside of the study site geographic area, were involved with our study, worked for a study partner agency, were inactive or retired, or had a suspended state practice license at the time of recruitment. Eligible HCPs were recruited by postal mail. Upon enrollment, providers were asked to provide the following: provider type and specialty, practice setting, current address, fax number, landline and cell phone number, and e-mail address. Physicians were over-sampled given their historically low response rates to mailed study invitations.

Randomization and Blinding

Enrolled HCPs were blindly allocated into intervention groups by using a simple randomization without replacement protocol. As stated previously, message group assignment was not contingent on the HCP’s ability to receive a message through a given delivery method. Data analysts and investigators were blinded to randomization groups for the primary analysis. The study enrollment and allocation flow as per CONSORT protocols are detailed in Figure 2. ⁴²

Figure 2 REACH (Rapid Emergency Alert Communications in Health) CONSORT Flow Diagram. Abbreviations: ARNP, nurse practitioner; KC, Public Health - Seattle & King County; MD, primary care physician; MT, Montana Department of Public Health & Human Services; PA, physician’s assistant; PHRM, pharmacist; SC, Spokane County Health Department; VET, veterinarian.

Messaging Intervention

A messaging system was created in-house to deliver study messages, log delivery failures, and track click-through rates for links or URLs embedded in the messages. E-mail and fax messages were designed to resemble those routinely disseminated by the partner public health agency at each site, conforming to each site’s message layout and including health officer contacts. SMS messages were limited to 160 characters. Embedded links were “tinified” by a system program that shortened the link to a maximum of 24 characters. See Supplemental Figure 1 in the online data supplement for example e-mail, fax, and SMS messages and website to which the “tinified” URL pointed in the message.

Intervention Assessment

Five to 10 days after each message was sent, each HCP was contacted by phone. The interview protocol has been described previously, ²⁵ but in brief, surveys included questions regarding recall of study message receipt, message topic, and clicking on the embedded hyperlink. The questions and flow of the follow-up phone interview are detailed in Figure 3. Phone interviewers were allowed to make up to 10 attempts to call HCPs; HCPs who were not contacted for interview represented a missed outcome measure. HCPs who did not participate in at least 1 phone survey or refused to be interviewed over the course of the study at their site were considered nonresponsive (see follow-up totals in Figure 2). These missing outcome data were excluded from the PP and AT analyses.

Figure 3 Flow Chart of Follow-Up Interview.

Data Analysis Approach

The primary outcome of interest was correct recall of the study message. Three primary outcome variables were examined: correct recall of study topic, recall of receipt of any public health message and correct recall of study topic given recall of receipt of any public health message (see bolded boxes in Figure 3). Primary data analysis was designed to measure differences between the 3 communication groups by using PP analysis. A set of secondary AT analyses examined the outcomes excluding providers for whom it was known that a message could not have been received by the method randomized to (owing to incorrect contact information, known technical failures, etc). AT analyses excluded the control group because there were no situations in which providers in the control group were known to have received study messages.

All analyses were conducted by using R v2.13.0. ⁴³ Analyses were conducted by repeated-measures logistic regression, using the geeglm and geese functions in the geepack R package. ⁴⁴ Statistical significance was assessed by the Wald statistic ⁴⁵ using the function anova.geeglm. Contrasts between categories were computed using the contrast.geese function in the contrast library. Because HCPs were sent multiple messages over the course of the study, the within-participant primary response variables have a propensity to be correlated. Within-participant residuals from a naïve generalized linear model with logistic link function and binomial family were observed to be correlated (r=0.175, 0.171, and 0.130 for Public Health Seattle & King County, the Montana Department of Public Health & Human Services, and the Spokane County Health Department, respectively). A permutation test for nonzero correlation for each site showed statistical significance at the P<0.001 level. Generalized estimating equations were fit to each primary outcome variable by using a generalized linear model with binomial error family and logistic link function, with randomization group a covariate and HCP defining correlated responses. An exchangeable correlation structure was assumed. A sensitivity analysis to the choice of correlation structure was performed and results were not sensitive to choice of correlation structure. A generalized linear mixed effects model was also considered and produced similar results for the estimated differences in population-level probabilities. Age, sex, and activation of the embedded hyperlink were evaluated as potential effect modifiers of the relationship between communication method and topic recall. Pairwise comparisons between exposure groups across sites were also made to determine the differences in frequency of outcomes between exposure groups.

Results

A total of 826 HCPs were included in the analysis. Study message topic and survey response rates per message by site are shown in Table 1.

Table 1 Study Message Topics and Intervention Assessment Response Rates ^a

^a Abbreviations: KC, Public Health - Seattle & King County; MT, Montana Department of Public Health & Human Services; SC, Spokane County Health Department.

^b Completion Rate=(completed interviews)/(providers enrolled).

^c Cooperation Rate=(completed+partially completed interviews)/[(completed+partially completed interviews)+refusals].

Analysis

The PP analysis revealed a higher rate of recall of study message for messages sent by e-mail than for those sent by fax or SMS (Table 2). No statistically significant differences in the recall rate were observed between messages sent by fax and those sent by SMS. Compared with HCPs in the fax and SMS groups, HCPs in the e-mail group were significantly more likely to have correctly remembered the message topic (P<0.001). There were no significant differences in message recall between the SMS and fax groups (P=0.801).

Table 2 Per-Protocol Analyses of Correct Recall of Message Topic by Group and Between-Group Comparison of Correct Recall of Message Topic ^a

^a Abbreviations: CI, confidence interval; OR, odds ratio; SMS, cell phone text messaging.

^b Controls excluded as no message receipt.

The AT analysis examined the rate of recall of study messages among those deliveries that did not fail (Table 3). The rates of recall for the e-mail and fax groups were similar (48.3%) and higher than for the SMS group (38.9%). Message group was significantly associated with correct recall (P<0.001): the SMS group had a lower recall rate than either the fax (P=0.045) or e-mail (P=0.042) group, whereas no statistically significant differences in recall rates were observed between the fax and e-mail groups (P=0.935).

Table 3 As-Treated Analyses of Correct Recall of Message Topic by Group and Between-Group Comparison of Correct Recall of Message Topic ^a

^a Abbreviations: CI, confidence interval; OR, odds ratio; SMS, cell phone text messaging.

Modifiers of Recall

Factors associated with rate of recall included age, length of time between message delivery date and interview, and accessing the embedded hyperlink. Every 10-year increase in provider age was associated with a 16.4% reduced likelihood of message recall. Each additional day that lapsed between the message delivery date and interview resulted in a 6.5% reduced likelihood of message recall. Rates for accessing the link varied by group and were statistically significant (P≤0.001) between fax and SMS link access and e-mail and SMS link access. Group differences between fax and e-mail link access were not statistically significant (P=0.157). Link access impacted rate of recall: the odds of correct recall of study topic were significantly higher among those who accessed the link than among those who did not (OR=3.9; P<0.001).

Summary

In the PP analysis, the e-mail group had a higher recall rate than did either the fax or SMS group, whereas in the AT analysis the e-mail and fax groups had similar recall rates and both had higher recall rates than did the SMS group. Together these analyses suggest that the rate of recall of messages that were successfully delivered was about the same for e-mail and fax messages and lower for SMS messages. Sex, provider type, organization type, and study site did not have significant effects on message recall (data not shown).

The AT analysis excluded known message delivery failures, either due to missing or incorrect contact information (provided either at enrollment or because of contact information changing) or refused or undeliverable messages (for example, firewall or cell carrier blockage or fax number busy). Rates of missing contact information did not differ by site (P=0.827) but did differ by message group (P< 0.001), with higher rates for the fax group than for the SMS or e-mail group. Not surprisingly, then, the rate of message delivery failure for the fax group was higher than that for either the SMS or e-mail group. If the ability to detect delivery failure was the same for each group, this suggests that the difference in recall rate between the fax and e-mail group in the PP analysis was due to the higher delivery failure rate for fax and that, conditional on receiving the study message, the fax and e-mail groups had similar recall rates.

Discussion

In summary, our study confirms that all of these communication methods—SMS, fax, and e-mail—are more effective than no communication, that HCPs do pay attention to messages sent by public health agencies, and that HCPs often explore the content of the message by clicking on embedded links to access additional information sources. Our findings contribute needed evidence for public health agencies regarding communicating with HCPs and should encourage public health agencies that rely solely on fax to consider adding e-mail and possibly SMS to their communications strategy. However, this change should be undertaken with caution. As stated, most public health e-mail distribution lists require HCPs to opt in to receive messages, list maintenance may not be feasible for many public health jurisdictions, and additional message delivery channels may increase the potential alert fatigue in HCPs who consequently disregard critical public health messages. ^{25 , 26}

Mobile devices are increasingly being used to deliver clinically focused treatment and prevention, to enhance diagnostics, and to provide consumer health access to information and services. ⁴⁶ Research regarding the use, utility, feasibility, acceptability, and effectiveness of mobile devices is expanding. ⁴⁷ As SMS increasingly penetrates the clinical environment, the acceptability and effectiveness of communication by mobile device may increase in parallel. Already, smartphones are being incorporated into the health care environment; the HCPs of the future may expect public health to communicate with them through mobile devices. Although the findings from our study suggest that SMS content was less likely to be recalled than content delivered through more traditional communication methods of e-mail and fax, the positive impact of link click-through on message recall observed in our study is noteworthy. Click-through rates were higher in the SMS group than in the other groups. Encouraging providers to access additional information provided in hyperlinks may be a useful strategy for improving awareness of time-sensitive public health message content.

Recent assessments of emergency planning, preparedness, and response efforts have noted that new technologies and social media need to be explored for their effectiveness in disaster response. ^{22 , 24 , 48} Social media and crowdsourcing tools and SMS have been used to collect and disseminate information during natural disasters ⁴⁹ and for disease monitoring. ⁵⁰ To inform best practices in public health emergency preparedness and response communications and to provide needed information for public health agencies to make decisions regarding adoption of new technologies, more systematic studies such as ours need to be conducted.

Limitations

There were several limitations to our study. Study messages, although time-sensitive, were not emergency alerts and so may have been perceived as less important or memorable. For some providers, such as veterinarians, the messages may have been perceived as not relevant to their practice. We did not include a question regarding relevance in the intervention assessment. We also could not control other factors that may have impacted recall, such as the number and topics of other messages received by HCPs, membership in professional e-mail distribution lists, and internal communications at the HCP’s workplace.

Previously, we reported a sub-analysis of “alert fatigue” on recall by HCPs, noting that the number of local public health e-mail messages HCPs received through subscribed lists significantly decreased the odds of them correctly remembering the receipt and content of our study message. This finding suggests that alert fatigue and information overload might inhibit the ability of HCPs to respond effectively to messages during a public health emergency when the volume of messages may be high. ²⁵

Another item to note is that preference for a specific communication channel might also impact recall. We recently reported a substudy regarding preferences for receiving public health alerts and advisories in which we found that HCPs had an overall preference for e-mail messages, as compared to fax or SMS. However, in that study we also found that prior exposure to communication channel was associated with an increased preference for that channel; ie, greater familiarity, and possibly comfort, with SMS raised the likelihood of preferring SMS. ⁵¹ Thus, another limitation was that the modality to which the HCP was randomized may have been unfamiliar or undesirable.

Although we met our target sample size to power the study, the study sample was not large. However, we believe our findings are generalizable beyond this study for several reasons. First, we sought to include HCPs who represented the wide variety of providers that will be included in all levels of emergency preparedness and response communications from public health agencies (local, state, territorial, and national). We included HCPs working within both urban and rural contexts and the settings of each site represented a diverse range of population densities and demographics, potential natural hazards, and varying public health agency organizational structures. We believe this combination of varied HCP roles and metro and nonmetro settings supports generalizing our findings beyond the Pacific Northwest.

Conclusions

Ensuring that public health information reaches its target audience through a modality that maximizes the likelihood the message is delivered, received, deemed credible, and actionable is critical to averting communication breakdowns that are the most common source of failure in emergency events and disaster response. There is limited evidence-based research to improve communications between public health agencies and health care providers. By systematically evaluating the relative effectiveness of mobile and traditional message delivery systems for emergency preparedness and response communications, our study contributes to the evidence base for improving approaches to emergency communications. More research is needed to understand the impacts of the method of message delivery, the source of the message, the type of information being disseminated, and the health care organizational context to guide or improve the practice of communication between public health agencies and HCPs before, during, and after a public health emergency.