Association between expectations and clinical outcomes in online v. face-to-face therapy – an individual participant data meta-analysis

Abstract

Background

Online treatments are increasing in number and are currently available for a wide range of clinical problems. To date little is known about the role of treatment expectations and other placebo-like mechanisms in online settings compared to traditional face-to-face treatment. To address this knowledge gap, we analyzed individual participant data from randomized clinical trials that compared online and face-to-face psychological interventions.

Methods

MEDLINE (Ovid) and PsycINFO (Ovid) were last searched on 2 February 2021. Randomized clinical trials of therapist guided online v. face-to-face psychological interventions for psychiatric or somatic conditions using a randomized controlled design were included. Titles, abstracts, and full texts of studies were independently screened by multiple observers. The Preferred Reporting Items for Systematic Reviews and Meta-analyses guideline was followed. Authors of the matching trials were contacted for individual participant data. Ratings from the Credibility and Expectancy Questionnaire and the primary outcome measure from each trial were used to estimate the association between expectation ratings and treatment outcomes in online v. face-to-face interventions, using a mixed-effects model.

Results

Of 7045 screened studies, 62 full-text articles were retrieved whereof six studies fulfilled the criteria and provided individual participant data (n = 491). Overall, CEQ ratings predicted clinical outcomes (β = 0.27) at end of treatment with no moderating effect of treatment modality (online v. face-to-face).

Conclusions

Online treatment appears to be equally susceptible to expectancy effects as face-to-face therapy. This furthers our understanding of the importance of placebo-like factors in online treatment and may aid the improvement of healthcare in online settings.

Introduction

Psychological interventions are increasingly delivered over the internet and evidence suggests comparable effects of online v. traditional face-to-face therapy for a range of clinical problems (Hedman-Lagerlöf et al., 2023; Titov et al., 2019). Most online treatments include cognitive-behavioral interventions and there is support for better outcomes in therapist-guided online treatments, compared to unguided (Andersson, Titov, Dear, Rozental, & Carlbring, 2019; Baumeister, Reichler, Munzinger, & Lin, 2014). Online treatments can either be synchronous (i.e. the therapist and the patient are present at the same time in therapy sessions, e.g. videoconference) or asynchronous (i.e. learning materials are provided in a digital format and the therapist and patient interacts via asynchronous messages). The focus of the present study is on the latter form of online treatment.

In any clinical intervention, there are factors other than the active ingredient that may shape the clinical outcome. These are oftentimes referred to as non-specific treatment factors or common factors (Cuijpers, Reijnders, & Huibers, 2019), and are part of the placebo response (Finniss & Benedetti, 2005). For example, patients' treatment expectations (patients' prognostic beliefs about how they will respond to treatment (Constantino, Visla, Coyne, & Boswell, 2018)) have been shown to shape symptoms and to be a robust predictor in face-to-face psychotherapy research (Constantino, Arnkoff, Glass, Ametrano, & Smith, 2011). This has also been suggested for online treatments (El Alaoui et al., 2015; El Alaoui et al., 2016). However, there has been no formal comparison of the effect of treatment expectations between studies in which patients have been randomized to either internet or face-to-face treatment. It is therefore unclear if expectations play the same role in symptom reduction in online therapeutic settings as seen in face-to-face treatment.

There are many obvious differences between online and face-to-face treatment contexts such as the clinical environment (i.e. being at the hospital v. being at home) and the clinician-patient interaction (i.e. physical meeting v. reading and writing messages). Online interventions often involve asynchronous communication and lack of access to non-verbal behaviors. This apparent difference between modalities has raised concerns if online interventions can deliver the same important ingredients involved in therapy (Rochlen, Zack, & Speyer, 2004; Wells, Mitchell, Finkelhor, & Becker-Blease, 2007). In addition, large patient surveys have found low acceptability for online treatments (Handley, Perkins, Kay-Lambkin, Lewin, & Kelly, 2015; Mohr et al., 2010) and high dropout rates (Waller & Gilbody, 2009). This might, however, be more common for certain subgroups of patients (Karyotaki et al., 2015). Finally, lay people and health professionals show a preference for face-to-face treatment (Gun, Titov, & Andrews, 2011).

One of the most used measurements for treatment expectation in clinical trials is the Credibility and expectancy questionnaire by Devilly and Borkovec (Devilly & Borkovec, 2000) as it is in the public domain and easily adapted to different treatments and populations (Constantino et al., 2018). It consists of six questions where three items are suggested to load on the credibility factor and the other half on the expectancy factor, although the factors strongly correlate (Haanstra et al., 2015). Credibility is defined as ‘how believable, convincing and logical the treatment is’ and expectancy as ‘improvements that clients believe will be achieved’ (Devilly & Borkovec, 2000). The scale has been used to investigate treatment expectations, by using the expectancy factor separately but also together by using a sum score of the scale (Constantino et al., 2018).

In the present study, we performed an Individual Participant Data Meta-Analysis (IPDMA) based on data from existing clinical trials in which patients had been randomized to face-to-face or therapist-guided online behavioral therapy with asynchronous communication. The aim was to determine if patients' treatment expectations predict clinical outcomes at study end, and differential effects of expectations in online v. face-to-face treatment. Three research questions were investigated: (1) Are there differences in patients' treatment expectations of online v. face-to-face treatment? (2) Can treatment expectations predict the clinical outcomes at study end? (3) Does type of treatment modality moderate the effect of treatment expectations on clinical outcomes? We hypothesized that higher treatment expectations would predict better treatment outcome in both treatment modalities but with a larger effect in face-to-face treatment compared to online treatment.

Method

Study sample and procedures

The study was performed according to the PRISMA guidelines and Cochrane Collaboration recommendations. Each study included in the IPDMA was approved by its own local ethical review board, and an additional ethical approval was granted before onset of the present IPDMA (Swedish Ethical Review Authority # 2021-03833). The study and its research questions were preregistered in PROSPERO (CRD42021245299) and the protocol was followed without deviations.

Searches of MEDLINE (Ovid) and PsycINFO (Ovid) were last conducted 2 February 2021 in collaboration with staff at the Karolinska Institutet University Library. The study selection process is shown in a PRISMA flow diagram (see online eAppendix in the Supplement). We searched for randomized clinical trials (RCTs) involving internet-delivered and face-to-face psychological interventions for all types of diagnoses and outcome measures.

Inclusion criteria were: comparison between therapist-guided Internet-delivered therapy with asynchronous communication and face-to-face therapy using a randomized controlled design; interventions aimed at treatment of psychiatric or somatic conditions (e.g. not merely prevention or psychoeducation); comparison of treatments that were similar in content in both treatment conditions; Internet-delivered behavioral therapy where the Internet treatment was the main component and not a secondary complement to other interventions; outcome data from an adult patient sample; outcomes in terms of assessment of symptoms of the target problem. Exclusion criteria were: other types of studies than randomized controlled studies, articles in non-English. The search strategy and preregistered methods can be found in the online eAppendix in the Supplement.

Two pairs of reviewers (M.P, M.J, E.M, V.V) independently screened the titles and abstracts of all studies to identify those that potentially met the inclusion criteria. Any disagreement within the pair was resolved through discussion with a third researcher. The full texts of the potentially eligible studies were retrieved and, independently, assessed for eligibility by two researchers (M.P and M.J). Two independent reviewers (E.M, V.V) extracted data to describe the included studies (see Table 1).

Table 1. Study and patient characteristics

^a Main outcome used in the analysis.

ICBT, internet-based cognitive behavioral therapy; ITT, Intention to treat; INT, Internet-based treatment; FTF, face-to-face treatment; SIAS, Social Interaction Anxiety Scale; SPS, Social Phobia Scale; HAI, 18-item Health Anxiety Inventory; MADRS-S, Montgomery Åsberg Depression Rating Scale; GAD-7, Generalized Anxiety Disorder 7-item scale; MPI, Multidimensional Pain Inventory; LSAS, Liebowitz Social Anxiety Scale; THI, The Tinnitus Handicap Inventory; Mini-TQ, Mini-Tinnitus Questionnaire; TRQ, Tinnitus Reaction Questionnaire.

Methodological quality and risk of bias

Methodological quality and risk of bias of the included studies were independently assessed by two researchers (E.M, V.V) using the Psychotherapy Outcome Study Methodology Rating Form (POMRF) and the Jadad Scale. POMRF is a 22-item assessment of methodological quality (e.g. representativeness of sample, use of blind evaluators) (Ost, 2008). The Jadad scale independently assesses the methodological quality of RCTs and takes the following factors into account: randomization, blinding and withdrawal/dropouts. The maximum score of 5 is representative of a good quality study (Jadad et al., 1996).

Data collection

Authors of studies that met the inclusion criteria were contacted according to the standardized schedule described by Veroniki and colleagues (Veroniki et al., 2019), where authors were contacted by email and then sent reminders up to four times. In addition, a letter was sent if there was no answer. Credibility and Expectancy Questionnaire (CEQ) ratings and pre- and post-treatment primary outcome data were requested as well as basic clinical characteristics and demographics.

Statistical analysis

All included studies used different primary outcomes, and the CEQ sum score was calculated. Two studies used a six item CEQ, three studies used a five-item questionnaire, and one study used a three item questionnaire. For the CEQ sum score, the pre-treatment outcome score and the post-treatment outcome score, harmonization was performed by first subtracting the minimum value within each study and then dividing by the maximum value within each study. After this normalization, all aforementioned variables received a range of 0–1, with 0 standing for the lowest score observed in the study, and 1 standing for the highest score observed in the study.

$${\rm Scor}{\rm e}_{ij} = \displaystyle{{{\rm Scor}{\rm e}_{ij}-{\rm min}( {{\rm Scor}{\rm e}_j} ) } \over {{\rm max}( {{\rm Scor}{\rm e}_j} ) -{\rm min}( {{\rm Scor}{\rm e}_j} ) }}, \;\;\;i = {\rm subject}, \;\;j = {\rm study}$$

A one-stage method was employed (Tierney, Stewart, & Clarke, 2023). For our primary analysis, a linear mixed model was used (as assuming normality of residuals in a mixed model is less restrictive and reasonably robust to deviations from normality). We modeled the effect of the CEQ sum on treatment outcomes with the individual participant at level 1, and study at level 2. The post-treatment outcome score was regressed on the CEQ sum, the treatment condition (online v. face-to-face), and the pre-treatment outcome score. The model included a study-level random CEQ slope but no random intercept due to data limitations. Test of overall difference on CEQ sum between online v. face-to-face was tested using a linear mixed model with treatment condition as independent variable. p values for the mixed model were obtained using the Satterthwaite method (for more information about the model see online eAppendix in the Supplement). Sensitivity analyses were conducted to assess whether country of the included studies or disorder were related to outcome.

The primary analysis was performed on complete case basis. For completeness, we also provide results from an imputed model that includes all participants (see online eAppendix in the Supplement). Imputation of the missing variables post-treatment and CEQ sum was performed using multiple imputation with 20 imputed datasets, the factors age, sex, education, and pre-treatment outcome were included as well as a categorical variable for Study to take into account between-study variation. In addition, we specified the imputation to be computed by treatment group.

All analyses were performed in R version 4.1.2. All tests were two-sided and p values below 0.05 were considered significant.

Results

Study selection

We identified 7045 unique records of which 62 full-text articles were retrieved. Twenty-six studies met the inclusion criteria, and the corresponding authors were approached and asked whether the CEQ had been administered and if they would be willing to share individual participant data. Thirteen studies did not administer the CEQ and three research groups were not able to send data, citing limited resources (Carlbring et al., 2005; Zerwas et al., 2017) or unavailability of data (Kiropoulos et al., 2008). Authors of four studies did not respond to any emails or mail. Six studies were able to provide such data and were thus included in our analyses (Andrews, Davies, & Titov, 2011; Axelsson et al., 2020; Boersma et al., 2019; Hedman et al., 2011; Jasper et al., 2014; Kaldo et al., 2008).

Study characteristics

Study characteristics are listed in Table 1. The following diagnoses were included (clinical trials, k): social anxiety (k = 2), tinnitus (k = 2), pathological health anxiety (somatic symptom disorder or illness anxiety disorder) (k = 1), chronic pain with comorbid depression and anxiety (k = 1). Four studies diagnosed the participants at the beginning of the trial (Axelsson et al., 2020; Boersma et al., 2019; Jasper et al., 2014; Kaldo et al., 2008) and in two studies participants were diagnosed before referral (Andrews et al., 2011; Hedman et al., 2011). Participants were recruited from routine care referral (Andrews et al., 2011), self-referral (Boersma et al., 2019) or a combination of the two (Axelsson et al., 2020; Hedman et al., 2011; Jasper et al., 2014; Kaldo et al., 2008). Four studies were carried out in Sweden (Axelsson et al., 2020; Boersma et al., 2019; Hedman et al., 2011; Kaldo et al., 2008), one study in Australia (Andrews et al., 2011) and one in Germany (Jasper et al., 2014). The CEQ was administered before randomization (Jasper et al., 2014; Kaldo et al., 2008), after randomization (Andrews et al., 2011) at week two (Axelsson et al., 2020; Hedman et al., 2011) and mid treatment (Boersma et al., 2019).

Participant characteristics

Among the 491 participants, the mean (s.d.) age was 41 (13) years and 274 (56%) were female. Across the included studies, 20% of subjects (124/491) were removed due to missing data for CEQ and/or the outcome measure (19% in the internet treatment and 21% in the face-to-face groups). The following number of subjects was removed for each study: 10 (29%) from Andrews et al. 2011, 14 (7%) from Axelsson et al. 2020, 53 (46%) from Boersma et al. 2019, 28 (22%) from Hedman et al. 2011, 8 (10%) from Jasper et al. 2014, and 11 (22%) from Kaldo et al. 2008.

Results of the IPDMA

The overall CEQ ratings indicated no significant difference in treatment expectations among patients randomized to online or face-to-face treatment [β (s.e.), −0.01 (0.038); p = 0.81], as the median (IQR) was similar for online treatment 0.72 (0.56; 0.81) and face-to-face 0.73 (0.61–0.83), see Table 2. Treatment expectations (the CEQ sum) was predictive of the post-treatment clinical outcome across groups [β (s.e.), −0.27 (0.057); p < 0.01], where higher expectations were associated with greater reductions of symptoms. When testing the interaction between CEQ sum and treatment group on clinical outcomes there was no significant difference between online v. face-to-face treatments [β (s.e.), 0.01 (0.08); p = 0.93], see Fig. 1. The statistical significance was stable after sensitivity analyses controlling for studies' country of origin and disorder.

Table 2. Treatment expectations for online v. face-to-face treatment

Median values for the CEQ score for each treatment type (TRT) in each study (Internet = 1; face-to-face = 0).

Figure 1. Visualization of the association between treatment expectations and clinical improvement for each study. A significant association between expectations (CEQ score) and clinical outcomes was seen in all studies, demonstrating that higher expectations are associated with a decrease in symptoms from pre to post treatment. However, there was no significant interaction between CEQ scores and treatment type (Blue = face to face; Red = internet). The standard deviation random effect for treatment expectation was 0.08, meaning that 95% of studies were expected to have a CEQ sum effect between −0.27±0.16.

Risk of bias

Out of a possible total of 44 points on the POMRF scale, the mean score for the reviewed studies was 24.3 (s.d. = 5.9), median 24 (IQR = 3.7) with a range of 14 to 31. Out of a possible 5 points on the Jadad scale the mean score was 3 (s.d. = 0), median 3 (IQR = 0). Total scores for each study can be found in the online eAppendix in the Supplement.

Discussion

Internet-based treatments have been available for almost 20 years and are becoming more common for delivering evidence-based psychological treatment. While the efficacy of online treatment programs is now well-established (Hedman-Lagerlöf et al., 2023) the relative importance of expectancy effects in the online v. face-to-face setting is largely unknown. The purpose of this study was to investigate the differential effects of treatment expectations in online v. face-to-face psychological treatments.

Based on research from face-to-face studies, treatment expectations are closely linked to the patient-clinician relationship (Connolly Gibbons et al., 2003; Visla, Constantino, Newkirk, Ogrodniczuk, & Sochting, 2018) where positive clinical encounters have been associated with better treatment outcome (Fluckiger, Del Re, Wampold, Symonds, & Horvath, 2012) higher placebo responses (Kaptchuk et al., 2008) and self-estimated ability to return to work, among sick-listed patients (Lynoe, Wessel, Olsson, Alexanderson, & Helgesson, 2011). Despite fundamental differences in treatment contexts and patient-clinician interactions, the present study found equivalent expectations in online and face-to-face treatment and no significant or clinically meaningful difference in the effect of expectations on the treatment outcome between the two modalities.

With regards to research question number one, the comparable expectation ratings between face-to-face and online treatment may be an indication of a high acceptability of these interventions among patients. This is in contrast with previous surveys showing low acceptability for online interventions in the general population (Handley et al., 2015; Mohr et al., 2010) and that Internet-based cognitive behavioral therapy is heavily underutilized (Carper, McHugh, & Barlow, 2013; Hennemann, Beutel, & Zwerenz, 2017) [although showing comparable efficacy with face-to-face (Carlbring, Andersson, Cuijpers, Riper, & Hedman-Lagerlof, 2018)]. The present analysis suggests that once patients are randomized to online therapy, they hold a more positive view of the treatment. This is in line with previous reports of acceptability in various randomized controlled trials, showing high acceptability for the internet treatment (Hedman, Ljótsson, & Lindefors, 2012; van Ballegooijen et al., 2014). This underscores the importance of acceptability-facilitating interventions for online treatments (Molloy, Ellis, Su, & Anderson, 2021), as the online intervention seems to be appreciated once the patients commence treatment.

There is also a possibility that individuals who choose to participate in a randomized trial that includes online treatment have a more positive view of this type of treatment from the beginning. Results should be interpreted while taking this possible bias into consideration. On the other hand, a large proportion of the patients were recruited through the regular health care system, which might mean a more naturalistic setting, and is a strength in the current material. Also, it is still important to compare differences in expectancies among the group of help-seeking patients that could accept both face-to-face and internet-based treatment. The overall preference for different forms of treatment modalities in a target population, for example everyone with a certain diagnosis, should be further explored in epidemiological studies.

With regards to research question number two, we found that expectation ratings predicted clinical outcome at post-treatment, which is consistent with the previous literature (Constantino et al., 2011, 2018), Patients' expectation ratings have shown relatively robust correlations with treatment outcome across different diagnoses and treatments (Greenberg, Constantino, & Bruce, 2006; Noble, Douglas, & Newman, 2001) even though a recent review suggested that constructs other than expectation may be more predictive of treatment outcomes among patients with chronic conditions who have treatment failures in their medical history (Kaptchuk, Hemond, & Miller, 2020). Future studies could perhaps benefit from adding other constructs, such as hope or motivation, as predictors in psychological interventions for chronic conditions.

With regards to research question number three, there was no interaction between online v. face-to-face treatments regarding the predictive role of expectations. This suggests that treatment expectations have a similar impact on therapeutic outcomes in online and face-to-face treatment. This adds to previous research that has shown similar effect of patient clinician relationship on treatment outcome in online and face-to-face treatments (Fluckiger, Del Re, Wampold, & Horvath, 2018).

It is important to note that all online studies in the present study were guided, i.e., included therapist contact (asynchronous). Most online treatments involve a clinician who guides the client through the program, provides feedback and general support. Guided treatment programs tend to be more effective than self-guided (Baumeister et al., 2014; Spek et al., 2007). It is generally believed that the relationship between a patient and clinician (the so-called therapeutic alliance) is central for expectancy effects (Jensen & Kelley, 2016) and equivalent expectations might indicate that patients do develop a positive relationship with their online therapist. However, it is also possible that the whole online program is rated and not just to the interactions between the client and the therapist (Andersson et al., 2019). It would be of interest to investigate if online treatment without guidance also induces equally high expectations.

Online settings seem to evoke similar expectations as face-to-face treatment. An early indication of this was demonstrated in a recent double-blind experiment (Ponten, Ljotsson, & Jensen, 2019) where we tested the novel hypothesis that expectations about pain relief, and placebo analgesia, can be induced via online patient-clinician communication. With no face-to-face communication prior to pain testing, participants displayed significant placebo responses with an effect size comparable to ordinary face-to-face experiments. Our placebo experiment also indicated that participants' perception of the online communication (how positive it felt) transferred to the perceived alliance during the subsequent face-to-face pain testing. This suggests that online and face-to-face communication may be interchangeable in some clinical contexts, as the online interaction was highly associated with the perception of face-to-face interaction, although not tested in a clinical context. The present IPDMA confirms that online expectations are relevant in a clinical context, demonstrated in a large number of patients, with several different clinical problems.

The role of expectations in online treatment settings has been poorly investigated. To our knowledge, this was the first systematic review and IPDMA to consider the association between expectation ratings and treatment outcomes in online v. face-to-face settings. To include as many studies as possible, different diagnoses and treatment outcomes were combined in this IPDMA, which is both a strength (large dataset) and limitation (heterogeneity). One limitation that may have introduced heterogeneity in our data was that the expectation measure (CEQ) was administered at different time-points between the studies, however exploratory analysis shows that the timepoint when CEQ was administered had no impact on the main results. If the CEQ is administered early, we suspect that scores are more likely to mirror widespread presumptions about the treatment format, and if the CEQ is administered at a later stage, scores are more likely to be a function of actual experiences with the treatment including its effects. In addition, the selected studies used different versions of the CEQ (two studies used a six-item CEQ, three studies used a five-item CEQ, one study used a three-item CEQ). The bias this could introduce was reduced by normalizing the scores. We only included studies that randomized participants between internet and face-to-face treatment. This limited the number of studies that met our inclusion criteria but had the advantage of ensuring that patients had not selectively chosen internet or face-to-face treatment. Finally, we chose to perform our analyses on cases with complete data, which reduced the number of participants, however we provide the analyses performed on imputed data in the online supplementary material and the results are comparable. Importantly, the number of participants with complete data was comparable between internet and face-to-face and has not led to any systematic bias.

In conclusion, we found no difference in how expectation ratings predict treatment response in guided Internet-delivered v. face-to-face behavioral therapy. Treatment expectation thus seems to play just as an important role in online as in face-to-face treatment. Our results have clinical implications as they imply that therapists need to attend to the patient-clinician relationship and foster positive expectations to the same extent in online as face-to-face treatment. This may be done by repeatedly assessing patients' treatment expectations and deliberately verify and validate the patient's belief in order to shape expectancy. The efficacy of digital healthcare interventions would be greatly aided by the further investigation of the placebo components of online therapeutic relationships and interventions, for example by using expectation ratings in predictive models in clinical practice (Forsell et al., 2019).