A scoping review to explore the suitability of interactive voice response to conduct automated performance measurement of the patient’s experience in primary care

Abstract

Introduction

Practice-based performance measurement is fundamental for improvement and accountability in primary care. Traditional performance measurement of the patient’s experience is often too costly and cumbersome for most practices.

Objective/Methods

This scoping review explores the literature on the use of interactive voice response (IVR) telephone surveys to identify lessons for its use for collecting data on patient-reported outcome measures at the primary care practice level.

Results

The literature suggests IVR could potentially increase the capacity to reach more representative patient samples and those traditionally most difficult to engage. There is potential for long-term cost effectiveness and significant decrease of the burden on practices involved in collecting patient survey data. Challenges such as low response rates, mode effects, high initial set-up costs and maintenance fees, are also reported and require careful attention.

Conclusion

This review suggests IVR may be a feasible alternative to traditional patient data collection methods, which should be further explored.

Introduction

There are growing demands for accountability and a recognized need for constant quality improvement (QI) in primary care practices (Government of Ontario, 2012). Performance measurement is often used to identify and address strengths and weaknesses through data collection and the provision of feedback (Simpson and Birdshell, 2006; WHO, 2008). However traditional methods of data collection, particularly those measuring patient-reported outcomes, can be laborious and costly and many primary care practices lack the time and resources to implement them (Stroebel et al., 2005; Parker et al., 2007; Ivers et al., 2012; Creswell, 2013). Thus for tools to be applicable to primary care settings they must cater to practice capabilities and concerns and also be efficient, minimally disruptive, and cost effective (Stroebel et al., 2005; Parker et al., 2007; Government of Ontario, 2012; Ivers et al., 2012). In looking for such methods, we conducted a scoping review of the literature on automated performance measurement (APM) through the use of interactive voice response technology (IVR), in order to assess lessons for its potential use in primary care health services research. For this review APM is defined, using elements from two sources, as: the use of telephones to administer computer-driven surveys, utilizing computerized or pre-recorded voices and touch tone response or speech recognition, to gather performance data (Abu-Hasaballah et al., 2007; Shaw and Verma, 2007). We specifically sought to identify the benefits and challenges in using an IVR approach (that includes any type of IVR system using a telephone only) to APM to determine feasibility for primary care practices’ quality improvement efforts and practice-based research studies.

Methods

In order to broadly explore this technology, we adopted a scoping review method outlined by Arksey and O’Malley (2005) and Levac et al. (2010). Our goal was to identify relevant lessons from published literature on the potential of an emerging technology, automated interactive voice response surveys, for primary care health services research and practice-based quality improvement. We sought all relevant literature regardless of study design, to record the approach used, and main findings, both broad and specific. We searched the MEDLINE database from the year 2000 to 2013 using keywords related to interactive voice response, performance measurement, and primary care. A list of the search terms and combined searches may be found in Appendix. International articles reporting the use of IVR for any reason in any health care practice setting were included if they were research studies or literature reviews. Articles were excluded if they were not published in English, contained a non-health care-related research question, the abstract was not available through Medline, or their interactive voice response systems were not conducted through the telephone.

M.F. ran the search and screened the titles and abstracts for meeting the inclusion criteria. Included articles at this stage underwent full text review by M.F. Their references were also reviewed for relevant sources. Articles were assessed for relevance to use of IVR for health care performance measurement such as applicable methods or population, as only a limited number addressed performance measurement specifically. Articles discussing interactive voice response survey technology in other health care contexts outside of primary care were also included to broaden the search. Included articles underwent in depth review and data extraction, the rest were excluded following initial review because they were not relevant to IVR and automated performance measurement.

Studies using IVR for a broad range of functions such as health intervention, follow-up calls, disease management, health education, and health databases were analysed alongside performance measurement articles due to a lack of data specifically on performance measurement. We extracted data from included studies on study goals, population/context, method of IVR implementation, type of IVR administration (call in/call out), survey questions, response rates, number of follow-up calls required, data quality, cost information, and factors reported as facilitating or hindering integration into a clinical practice. Descriptive data from each article was analysed by M.F., S.J., and W.H. to identify lessons for the use of interactive voice response technology for performance measurement in primary care. We identified common findings across studies, and paid attention to disconfirming results and areas requiring further investigation. To promote the rigour of the study, our analysis team was multidisciplinary including two clinician primary care health services researchers and one non clinician health systems researcher. Further we presented our early analysis findings to several health services researchers and an information technology expert in the field of IVR incorporating feedback into our ongoing assessment of relevance to primary care. We synthesized the main findings relevant for primary care, based on recurrence between studies and/or the impact or potential impact for performance measurement. These are presented under categories in the result section.

Results

The initial search retrieved 1987 articles, 123 of which were selected for full text review. 10 additional sources were added after searching the references of included articles. In all, 33 articles were included for in depth review and data extraction. A summarized figure of the search results is shown in Figure 1.

Figure 1 Scoping review

A total of 33 articles were included in our search and addressed interactive voice response technology use in health care but very few specifically addressed primary care and performance measurement. Five studies utilized IVR for collection of patient satisfaction data while 14 studies used IVR alongside patient services such as monitoring and education. Multiple study types were reviewed, including 14 randomized controlled studies, other experimental designs, observational studies, and systematic reviews. Five studies were literature reviews of IVR or related technologies. The studies used diverse methods, sample populations, implementations of IVR, and outcomes measures. In all, 27 studies had sample populations ranging from 36 to 58 440, with a total of 117 692 participants across all studies reviewed. An additional study included three practices as their sample population and the remainder of studies were literature reviews. All studies were set in the United States except for one that was conducted in Canada by Reid et al. (2007).

This section presents the results most pertinent to the topic of interactive voice response for automated performance measurement. The following tables present brief summaries of each source grouped by study focus. Table 1 summarizes the methods and relevant findings of studies specifically evaluating IVR survey methods. Table 2 summarizes the findings of studies which combined IVR surveys with another intervention. The articles detailing technological specifications or implementation issues are summarized in Table 3. The studies investigating mode effect are described in Table 4, and Table 5 presents the literature reviews on this topic. Highly relevant or commonly reported aspects of IVR surveys are presented below.

Table 1 IVR methods and findings from included studies relevant to potential use for primary care patient-reported outcome measures

IVR=interactive voice response.

Table 2 IVR methods and relevant findings from included studies which paired IVR with another intervention

IVR=interactive voice response.

Table 3 Studies detailing IVR technological specifications and/or implementation issues

IVR=interactive voice response; EMR=electronic medical record.

Table 4 Studies investigating mode effects

IVR=interactive voice response.

Table 5 Literature reviews on IVR

IVR=interactive voice response.

IVR features

Several sources discussed the multiple features that interactive voice response technology provides. IVR can reach multiple simultaneous participants, across a wide geographic spread, in multiple languages and accents, and most people own and know how to use a phone (Stuart et al., 2003; Brodey et al., 2005; Abu-Hasaballah et al., 2007; Estabrooks et al., 2009; Oake et al., 2009; Dalal et al., 2010; Lundy and Coons, 2012; Skolarus et al., 2012). IVR may use computerized voice or pre-recorded messages, can ask multiple choice and open-ended questions, and can randomize participants to receive different messages or survey questions (Abu-Hasaballah et al., 2007; Welker, 2007; Willig et al., 2013). IVR allows skipping and branching of irrelevant questions to save participant time and can call-out to patients during specific time intervals or allow patients to call-in at their own convenience (Lee et al., 2003; Abu-Hasaballah et al., 2007; Shaw and Verma, 2007; Naylor et al., 2008; Shea et al., 2008). Finally IVR technology can be integrated with databases and enter information directly into tables and graphs (Aharonovich et al., 2012).

Data quality

IVR produced data quality comparable to other modes in multiple studies (Weiler et al., 2004; Shea et al., 2008; Dalal et al., 2010; Skolarus et al., 2012; Houser et al., 2013). Mode effects due to the telephone delivery method and automated nature of IVR were experienced in some studies, as respondents were found more likely to choose the extreme categories (Rodriguez et al., 2006; Dillman et al., 2009). Some participants became frustrated by the survey pace causing them to ‘satisfice’, selecting the first available answer to move on to the next question, which biased results (Rodriguez et al., 2006). In contrast, recency effects may result when the participant remembers the last answer choice read out, due to it being heard most recently, and is more prone to pick it as a result (Dillman et al., 2009).

IVR was used effectively to reach diverse populations and hard to reach patient samples such as the visually impaired, illiterate, lower education, and non-English speaking patients. However, studies noted that those with hearing impairment may have difficulty participating in a phone survey and some populations may not trust reporting information to a machine (Stuart et al., 2003; Brodey et al., 2005; Goldman et al., 2008; Naylor et al., 2008; Dalal et al., 2010; Rose et al., 2010a; 2010b; Graham et al., 2012; Skolarus et al., 2012).

Sources suggested that to achieve good data quality installers need to carefully consider variables of speech recognition such as sensitivity of speech, risks of words being misrecognized, poor audio quality such as echoes, poor telephone service, and that phones encourage the use of shorter rating scales and simplified wording (Welker, 2007; Dillman et al., 2009; Skolarus et al., 2012; Willig et al., 2013).

Costs

Multiple sources cited IVR technology as a feasible tool for practices and though there is a substantial initial expense, it is balanced against low incremental costs and long term savings (Janda et al., 2001; Abu-Hasaballah et al., 2007; Piette et al., 2008; Oake et al., 2009; Rose et al., 2010a; Houser et al., 2013; Willig et al., 2013). Initial expenses included paying for: the system, programming, script creation, technical support staff, and hardware for data storage (Stuart et al., 2003; Ariza et al., 2004; Abu-Hasaballah et al., 2007). Janda et al. (2001) listed two programs that required programming knowledge to use, EASE (2800$) and Artisoft (1400$). Another package by VOS ($3300) did not require programming knowledge. Furthermore an outsourced company could be used for 40$ per respondent (Janda et al., 2001); Graham et al. (2012) reported that IVR calls were 25$ per patient. Shaw and Verma (2007) found cost savings of 100$ per IVR call compared with a live interviewer. However, Rodriguez et al. (2006) reported that a mailed survey was cheaper than IVR in their study and IVR costs increased when low response rates required crossover mailing. Lower response rates in some studies created a need to supplement IVR with other formats to improve response rates, which can incur additional expenses (Stuart et al., 2003; Rodriguez et al., 2006; Shaw and Verma, 2007; Dillman et al., 2009).

Some studies used the technology for service delivery or clinical care including for monitoring health outcomes, reminders, interventions and disease management (Reid et al., 2007; Goldman et al., 2008; Naylor et al., 2008; Sikorskii et al., 2009; Bender et al., 2010; Kempe et al., 2012; Greaney et al., 2012; Houser et al., 2013). The cost was part of health care delivery, often eliminating the need for a member of the health care team to carry out the task.

Response rate

Low response rates (around 30%) were observed when studies focused solely on survey administration (Isenberg et al., 2001; Tourangeau et al., 2003; Rodriguez et al., 2006; Shea et al., 2008; Dillman et al., 2009). Aside from two sources, much higher response rates were usually observed and sustained over longer periods when IVR questions were paired with interventions (Table 2). For example, a 79% response rate was observed in an intervention to reduce non-injection drug use (Aharonovich et al., 2012) and there was a 74% response rate at 12 months in an IVR study that aided in a weight loss intervention (Estabrooks et al., 2009).

Many studies sought survey recruitment and patient consent using a different format such as mail, a live phone call, or were recruited during a visit to the practice and expected to call into the system (Stuart et al., 2003; Shea et al., 2008; Aharonovich et al., 2012). Patient thoughts on such approaches throughout the process indicated annoyance with having to call into the IVR system and noted that they would rather receive calls (Stuart et al., 2003; Rodriguez et al., 2006; Aharonovich et al., 2012); it is possible to combine both call in and call out methods to further accessibility (Estabrooks et al., 2009).

Several studies reported strategies to improve response rates including: brief calls, call-in lines for additional time, pre-recorded voices, money incentives, calling cards, draw prizes, medical information, instructions, reminders, useful feedback, physician endorsements, indicators of practice improvement, and use of caller ID and statements to distinguish from telemarketing calls (Isenberg et al., 2001; Lee et al., 2003; Brodey et al., 2005; Abu-Hasaballah et al., 2007; Shaw and Verma, 2007; Welker, 2007; Goldman et al., 2008; Aharonovich et al., 2012; Skolarus, 2012; Willig et al., 2013).

Ease of integration of IVR with clinical practice

Once installed, IVR caused little disruption to normal practice functioning in several circumstances (Stuart et al., 2003; Bender et al., 2010; Rose et al., 2010b; Willig et al., 2013). Studies also noted that the integration of IVR into a database allowed for timely, accurate information, and decreased staff hours (Lee et al., 2003; Shaw and Verma, 2007; Goldman et al., 2008; Naylor et al., 2008; Rose et al., 2010a; Graham et al., 2012; Skolarus et al., 2012; Aharonovich et al., 2012).

Discussion

The review of studies reporting on interactive voice response technology found some important considerations for practices and research teams considering IVR as a tool to better understand the patient experience in primary care. Several features of IVR survey capacity are particularly noteworthy as they might address limitations in our current approaches to collecting data on patient-reported outcome measures. The capacity to reach diverse populations through IVR technology addresses a significant limitation of the common practice-based waiting-room administration of surveys which reach only those who attend the practice, potentially creating biased samples (Green et al., 2012). IVR surveys offer the flexibility to programme for a random sample from a defined roster of patients, perhaps the entire list, or a sub-set of highest interest for the study, and use different languages. As the telephone is almost universal, IVR surveys could facilitate access to a more representative sample of the group being studied. This is especially important for those patients who are traditionally more difficult to reach with paper-based waiting room surveys or even online surveys such as the visually impaired, illiterate, lower education, and non-English speaking patients (Stuart et al., 2003; Brodey et al., 2005; Abu-Hasaballah et al., 2007; Estabrooks et al., 2009; Oake et al., 2009; Dalal et al., 2010; Lundy and Coons, 2012; Skolarus et al., 2012).

A major barrier to practices participating in research or collecting data on patient reported outcomes of care is the perceived lack of time and concern of added burden on already stretched practices (Fitzpatrick, 2009; Johnston et al., 2010). Once installed IVR could have a limited burden, if any at all, on a practice and if integrated with an electronic medical record (EMR) efficiently create outreach to patients without involving practice staff (Stuart et al., 2003; Shaw and Verma, 2007; Bender et al., 2010; Rose et al., 2010b; Aharonovich et al., 2012; Willig et al., 2013). It could be programmed to generate and provide data at meaningful intervals to match study or practice needs.

The range in response rates and the relatively high upfront costs of investing in IVR technology require attention to maximize response rates. A significant feature of IVR for primary care is the ability to have the technology multi-task (Isenberg et al., 2001; Janda et al., 2001; Tourangeau et al., 2003; Rodriguez et al., 2006; Abu-Hasaballah et al., 2007; Shea et al., 2008; Dillman et al., 2009; Estabrooks et al., 2009; Aharonovich et al., 2012). The capacity to customise IVR and integrate it with EMR (Lee et al., 2003; Shaw and Verma, 2007; Goldman et al., 2008; Naylor et al., 2008; Rose et al., 2010a; Graham et al., 2012; Skolarus et al., 2012; Aharonovich et al., 2012; Willig et al., 2013) opens possibilities for combining patient services such as appointment reminders or information on clinical services with patient survey questions. Services such as appointment reminders are often welcome by patients and might help overcome reluctance to complete general survey questions (Dillman et al., 2009). Additionally, several practices could share one IVR system, which would help diminish the burden of cost on an individual practice (Rose et al., 2010a).

Several studies suggested patients prefer a simpler process of a direct call-out to them rather than a call-in option (Stuart et al., 2003; Rodriguez et al., 2006; Aharonovich et al., 2012). However, using a simple call-out survey will raise challenges in securing patient consent when IVR is used for research purposes and there have been variable approaches by institutional ethics review boards on whether prior consent is required (Nelson et al., 2002). The potential harm to patients from the responses they might give would need to be very low to make a short consent process possible via automated recorded message ethically acceptable. Securing patient informed consent beforehand, such as during a practice visit, or through a mail-out to potential participants requesting permission to contact them by phone could significantly decrease participation rates or limit the sample to those who attend a practice (Nelson et al., 2002; Angus et al., 2003; Krousel-Wood et al., 2006). However the use of IVR for practice-based quality improvement patient surveys might not require the same degree of information as patient informed consent for research purposes.

An automated IVR phone survey is a different mode from the ones used for many previously validated surveys (Weiler et al., 2004; Sikorskii et al., 2009; Dalal et al., 2010). The potential for mode effects such as satisficing and recency require attention such as changing the order of questions asked (Rodriguez et al., 2006; Dillman et al., 2009). Survey instruments used in other modes would have to be validated for IVR (Abu-Hasaballah et al., 2007). Designing surveys which take advantage of the strengths of IVR would maximise the value of this approach. Shorter surveys spread over a larger population, customized by patient input on language, would be best suited for IVR (Weiler et al., 2004; Abu-Hasaballah et al., 2007; Goldman et al., 2008; Naylor et al., 2008; Shea et al., 2008; Dalal et al., 2010; Rose et al., 2010b; Greaney et al., 2012; Willig et al., 2013). Survey questions benefitting from specific follow up timing would also be suited for IVR (Houser et al., 2013; Willig et al., 2013).

The various studies highlight diverse strategies to improve response rates, which could be a potential weakness of IVR used for research or quality improvement purposes. Using the strengths of flexibility to optimise timing for patients and the capacity to integrate the IVR survey with patient services are attractive ways to enhance the functionality of IVR and possibly increase response rates (Reid et al., 2007; Goldman et al., 2008; Naylor et al., 2008; Sikorskii et al., 2009; Bender et al., 2010; Kempe et al., 2012; Greaney et al., 2012; Houser et al., 2013). Further exploration of response rates and reach capacity with IVR surveys linked to patient services would significantly advance the understanding of this technology to improve primary health care data collection capacity.

Conclusion

As primary care reform continues, the need for understanding the patient experience and improving data collection capacity on patient-reported outcome measures, especially from traditionally difficult to reach populations, is essential. This review of interactive voice response technology, for potential use in automated performance measurement of the patient experience in primary care practices, suggests it may be a feasible alternative to traditional patient data collection methods, which should be further explored. The lessons from the use of IVR in many other health care sectors can guide implementation efforts within the primary care sector to maximise the potential of this approach.