Brief scales for the measurement of target variables and processes of change in cognitive behaviour therapy for major depression, panic disorder and social anxiety disorder

Abstract

Background:

The measurement of process variables derived from cognitive behavioural theory can aid treatment development and support the clinician in following treatment progress. Self-report process measures are ideally brief, which reduces the burden on patients and facilitates the implementation of repeated measurements.

Aims:

To develop 13 brief versions (3–6 items) of existing cognitive behavioural process scales for three common mental disorders: major depression, panic disorder, and social anxiety disorder.

Method:

Using data from a real-world teaching clinic offering internet-delivered cognitive behavior therapy (n=370), we drafted brief process scales and then validated these scales in later cohorts (n=293).

Results:

In the validation data, change in the brief process scales significantly mediated change in the corresponding domain outcomes, with standardized coefficient point estimates in the range of –0.53 to –0.21. Correlations with the original process scales were substantial (r=.83–.96), internal consistency was mostly adequate (α=0.65–0.86), and change scores were moderate to large (|d|=0.51–1.18). For depression, the brief Behavioral Activation for Depression Scale-Activation subscale was especially promising. For panic disorder, the brief Agoraphobic Cognitions Questionnaire-Physical Consequences subscale was especially promising. For social anxiety disorder, the Social Cognitions Questionnaire, the Social Probability and Cost Questionnaire, and the Social Behavior Questionnaire-Avoidance and Impression Management subscales were all promising.

Conclusions:

Several brief process scales showed promise as measures of treatment processes in cognitive behaviour therapy. There is a need for replication and further evaluation using experimental designs, in other clinical settings, and preferably in larger samples.

Introduction

Depression and the common anxiety disorders such as panic disorder and social anxiety disorder are among the worldwide leading causes of disease burden (Bandelow and Michaelis, 2022; Liu et al., 2020). Cognitive behaviour therapy (CBT) is a widely researched and recommended treatment for these conditions, with typical remission rates around 43–59% (Loerinc et al., 2015; Santoft et al., 2019a). Certain processes are widely believed to maintain depression and anxiety over time, and are therefore targeted in mainstream CBT. Cognitive theory tends to emphasize how the formation of dysfunctional schemas and assumptions about the self and the world can influence information processing and result in problematic cognitive, emotional and behavioural responses (Beck and Haigh, 2014; Bennett-Levy et al., 2004). Behavioural theory instead focuses on the functional analysis of behavioural responses to the environment, and tends to emphasize the role of conditioning and the beneficial effects of increasing positively reinforced behaviours while reducing negatively reinforced safety and avoidance behaviours (Mowrer, 1960; Sturmey, 2007). Mainstream CBT aims to systematically (a) modify dysfunctional assumptions, (b) achieve beneficial changes in behaviour and environmental factors in accordance with functional analysis, or (c) a combination of the two. The measurement of process variables derived from cognitive behavioural theory can aid treatment development, open new avenues for research into processes, and support the clinician in following treatment progress.

In both research and clinical practice, several self-report questionnaires have been developed to measure key processes in therapy in accordance with cognitive behavioural theory. Here follows a list of relatively widespread self-report process measures of relevance for the present study. In depression, one existing process scale is the Automatic Thoughts Questionnaire (ATQ) which measures ‘the covert self-statements reported by depressives as being representative of the kinds of cognitions they experience’ (Hollon and Kendall, 1980), i.e. cognitions indicative of dysfunctional schema and core assumptions targeted with cognitive interventions (Clark and Beck, 2010). The ATQ was originally developed as a 30-item instrument, but has been shortened to 15- and 8-item versions with adequate psychometric properties (Netemeyer et al., 2016). Variations on the ATQ have been found to change with depression symptoms during CBT, and to mediate the effect of CBT on depressive symptoms (i.e. the ATQ appears to be a variable that changes contingent on the treatment, and as part of the same process as the outcome; Garratt et al., 2007; Stice et al., 2010). Also relevant for depression, the Behavioral Activation for Depression Scale (BADS; Kanter et al., 2006; Kanter et al., 2008) is an existing measure of behavioural activation, behavioural avoidance, and rumination, which are key constructs of the mainstream learning theory account of depression (Kanter et al., 2010; Martell et al., 2013; Ramnerö et al., 2016; Spasojevic and Alloy, 2001). The conventional BADS has 25 items, although an abbreviated 9-item version has also been developed with psychometric evaluation rendering mixed results (Fuhr et al., 2016; Manos et al., 2011). There is some evidence indicating that an improvement in the BADS is typically associated with beneficial effects of CBT on depression symptoms (Manos et al., 2010), and that behavioural activation as measured using the BADS can be regarded a mediator of the treatment effect at least under some operationalizations (Seeley et al., 2019; van Luenen et al., 2019). Also based on learning theory, the Reward Probability Index (RPI) measures the degree to which individuals are exposed to rewarding environmental events (first subfactor) and environmental suppressors (second subfactor); the former usually regarded a proxy measure of response-contingent positive reinforcement, and the latter referring to properties of the environment that discourage or prevent an individual from engaging in activities that result in response-contingent positive reinforcement (Carvalho et al., 2011; Lewinsohn and Graf, 1973; Martell et al., 2013). The RPI is a 20-item instrument, and to our knowledge no abbreviated version has yet been developed. The RPI and its subfactors has been found to mediate the effect of behavioural activation and avoidance on mood in healthy adults (Carvalho and Hopko, 2011; Gill et al., 2019; Hill et al., 2017). However, we are only aware of one adequately powered clinical trial where the RPI was analysed as a potential mediator of the beneficial effects of treatment for depression. This study focused on pregnant women and reported promising effects, but warrants replication (Dimidjian et al., 2017).

In panic disorder, the Agoraphobic Cognitions Questionnaire (ACQ) incorporates 14 items pertaining to catastrophic thoughts relating to the ‘fear of fear’ (Goldstein and Chambless, 1978) in terms of physical consequences (e.g. having a heart attack or a stroke) and in terms of the loss of control (e.g. acting strange or going crazy) (Chambless et al., 1984) that are indicative of dysfunctional assumptions of the kind typically targeted in cognitive interventions (Bennett-Levy et al., 2004). Supporting the idea of agoraphobic cognitions as a key target in treatment, a 2021 meta-analysis found that CBT for the anxiety disorders produces large average improvements in threat reappraisal that are especially pronounced in panic disorder (Draheim and Anderson, 2021). The ACQ has been evaluated as a process measure in CBT, and has been found to reduce over the course of treatment and mediate treatment effects (Gloster et al., 2014; Vogele et al., 2010). Also relevant for panic disorder, the Behaviors Questionnaire (BQ) measures the existence of panic disorder-related avoidance and safety behaviours. These are behaviours targeted in mainstream CBT (Craske and Barlow, 2014) that occur in response to panic-related anxiety or fear, and which are commonly believed to persist due to the resulting short-term reduction in anxiety, the lack of opportunities for learning new behaviours, and the lack of opportunities for disproving dysfunctional assumptions (Centre for Anxiety Disorders & Trauma (CADAT) at King’s College London, 2020; Marks and Mathews, 1979). Although we have been unable to identify a primary peer-reviewed publication, the most widespread version of the BQ appears to have 31 items grounded in cognitive behavioural theory, and the avoidance subscale appears to have been developed on the basis of the Fear Questionnaire by Marks and Mathews (1979), which according to its primary publication has been found to change with treatment. Highlighting the potential benefits of the BQ as a behavioural process measure, a 2015 systematic review identified a higher baseline level of behavioural avoidance as the most consistent predictor of a smaller symptom improvement in CBT for panic disorder (Porter and Chambless, 2015). To our knowledge, no abbreviated version of the ACQ or BQ has ever been developed.

In social anxiety disorder, the Social Cognitions Questionnaire (SCQ) surveys the frequency and believability of catastrophic thoughts about social situations (Clark, 2001; Oxford Centre for Anxiety Disorders and Trauma (OxCADAT), 2022). Although we are not aware of a primary peer-reviewed publication detailing the development of the SCQ, the instrument consists of 22 items grounded in cognitive behavioural theory, appears to be relatively widespread which could be argued to speak for its face validity, and its internal consistency has been found to be excellent (Thew et al., 2020) including with regard to an adapted version for adolescents (Chiu et al., 2021). The SCQ has been found to change with CBT, and to mediate the treatment’s effect on overall social phobia (Thew et al., 2020; Thew et al., 2022). At least one longitudinal cohort study has also found a variation on the SCQ to be predictive of increased social anxiety as studied outside of treatment (Chiu et al., 2021). Also relevant for social anxiety, the Social Probability and Cost Questionnaire (SPCQ) probes the perceived probability and cost of adverse social outcomes, indicative of dysfunctional assumptions that can be targeted in therapy (Foa et al., 1996; Hoffart et al., 2009; Hoffart et al., 2016; McManus et al., 2000). Although the most widespread version of the SPCQ has 33 items (McManus et al., 2000), an 8-item version has also been used (Hoffart et al., 2009; Santoft et al., 2019b). Probability and cost biases have been found to respond to CBT for social anxiety disorder (Benbow and Anderson, 2018), and at least one study has been indicative of a reciprocal relationship between the SPCQ and social anxiety over the course of CBT (Santoft et al., 2019b). Another process scale is the 28-item Social Behavior Questionnaire (SBQ) which measures the presence of avoidance and safety behaviours common in social anxiety disorder, including strategies for impression management (Gray et al., 2019). These behaviours are at the core of most cognitive behavioural conceptualizations of the maintenance of social anxiety disorder, and are believed to maintain or perhaps even exacerbate dysfunctional patterns in cognition and emotion (Lervik et al., 2022; Wong and Rapee, 2016). In line with this view, avoidance behaviours including as measured using the SBQ have been found to mediate the effect of CBT on social anxiety disorder (Santoft et al., 2019b; Thew et al., 2022). In summary, there are several existing self-report measures of targets of CBT for clinical depression, panic disorder, and social anxiety disorder.

For these scales to be more useful in clinical work and research, it would be preferable to develop shorter and yet psychometrically sound versions. Brief versions place less burden on the respondent, can be more easily combined with other scales, and administered on a repeated basis. This makes it easier to monitor progress and provide personalized feedback in the clinic, and more accurate conclusions can be drawn in research (Ziegler et al., 2014). The use of repeated measurement is also important for mediational analyses and investigations into processes of change (Kazdin, 2007). In this study, we aimed to develop brief versions of the aforementioned CBT process scales, as based on data from a real-world teaching clinic offering internet-delivered CBT for depression, panic disorder, and social anxiety disorder. In order to establish a frame of reference and ensure that the beneficial properties of the original full scales were maintained, we also systematically investigated the psychometric properties of the original process scales including as potential mediators of the treatment effects in CBT.

Method

Design

We developed 13 brief CBT process scales at an ICBT teaching clinic run by master-level psychologist students and their clinical supervisors at Karolinska Institutet, Stockholm, Sweden, as described previously (Niles et al., 2021). This study was based on longitudinal data from all 674 patients attending the clinic between 2014 and 2021, with the exception of 11 patients who did not give informed consent for participating in research. Thus, we analysed data from 663 patients.

Participants

Advertisements for the ICBT teaching clinic were posted on social media and in newspapers, under the heading ‘Depression? Social phobia? Panic attacks?’. Patients that showed interest were instructed to complete an online self-report screening battery after which those who reported subclinical symptoms, suicidal ideation, severe depression, or an alcohol use disorder were referred to routine care. Those remaining completed an eligibility telephone interview with a psychologist student working under the supervision of a licensed psychologist. In order to be offered treatment, patients were required to meet full criteria for a primary diagnosis of depression, panic disorder, or social anxiety disorder as assessed using the Mini International Neuropsychiatric Interview (MINI; Sheehan et al., 1998). Patients were also required not to report manic or psychotic symptoms, meet full criteria for post-traumatic stress disorder, or to have an alcohol, substance use, or eating disorder. Although no reliability checks were conducted, previous studies have indicated that the MINI is a reliable instrument (Sheehan et al., 1998) that is appreciated by clinicians (Pettersson et al., 2018), and the diagnoses were validated by a licensed psychologist. The recruitment flow is illustrated in Fig. 1, and patient characteristics are listed in Table 1. In total, 663 patients were included, of whom 218 had received ICBT for depression, 183 ICBT for panic disorder, and 262 ICBT for social anxiety disorder. We divided the sample into a selection/training sample (n=370) of patients treated in 2014–2017 and a validation sample (n=293) of patients treated in 2018–2021.

Figure 1. Flowchart of the recruitment procedure at the Internet-delivered cognitive behavior therapy teaching clinic. AUDIT = Alcohol Use Disorders Identification Test; CBT = cognitive behavior therapy; MADRS-S = Montgomery-Åsberg Depression Rating Scale – Self report version; MDD = major depressive disorder; PD = panic disorder; SAD = social anxiety disorder; ITT = intention to treat.

Table 1. Sociodemographic and clinical characteristics of the two samples

ICBT, therapist-guided internet-delivered cognitive behavior therapy; MADRS-S, Montgomery-Åsberg Depression Rating Scale-Self-report version; LSAS-SR, Liebowitz Social Anxiety Scale-Self-report version; PDSS-SR, Panic Disorder Severity Scale-Self-report version; USS, upper secondary school, equivalent to International Standard Classification of Education (ISCED) 1997 level 3.

^a Data on age missing for 19 patients (18%) in the validation depression group, 20 (26%) in the validation panic disorder group, and 13 (12%) in the validation social anxiety disorder group.

^b Valid percentage, i.e. proportion of non-missing. Data on education missing for 5 patients (4%) in the selection/testing depression group, 1 (1%) in the selection/testing panic disorder group, 1 (1%) in the validation depression group, 2 (3%) in the validation panic disorder group, and 4 (4%) in the validation social anxiety disorder group.

Treatments

Patients were enrolled in 10 weeks of internet-delivered cognitive behaviour therapy (ICBT) for their primary diagnosis (depression, panic disorder, or social anxiety disorder) based on mainstream cognitive behavioural theory and guided by a masters-level psychologist student under the supervision of a clinical psychologist. An advantage of ICBT for the study of treatment processes is the highly structured treatment format which ensures that all patients are exposed to the same content, and reduces the risk that treatment components are omitted. It is also often argued that the supportive role of the therapist, who may devote as little as 10–20 minutes per patient and week, may reduce the influence of unintended therapist behaviours (so called ‘therapist drift’) (Andersson, 2016). The treatment was text-based and delivered via a secure web platform where each patient had a personal account, and logged in regularly to complete eight modules that resembled book chapters with bundled homework exercises. Access to new modules was given contingent on progress, meaning that the completion of module 1 was necessary to reach module 2, the completion of module 2 was necessary to reach module 3, and so on. Patients and therapists could also communicate via an asynchronous email-like messaging system where a typical message consisted of 100–400 words. The aim was to convey the same educational content, and to have the patient engage in the same strategies and behaviour changes, as in conventional face-to-face CBT even though the therapist was expected to devote less time per patient and week.

ICBT for depression was primarily based on behavioural activation. This protocol had not previously been empirically validated, but followed best practice learning theory principles found to be effective in randomized controlled trials (Cuijpers et al., 2007; Dimidjian et al., 2011; Santoft et al., 2019a). Educational components emphasized the role of functional analysis and the importance of reducing behavioural avoidance. Patients were also encouraged to practice functional analysis of negative thoughts, to assess whether ruminating thoughts were helpful, and to engage in structured problem solving. ICBT for panic disorder adhered to a published manual (Carlbring et al., 2001) and emphasized exposure and response prevention including interoceptive exposure (Pompoli et al., 2018), while also incorporating the identification and systematic exploration of negative automatic thoughts, breathing retraining, mindfulness, and acceptance strategies. ICBT for social anxiety disorder was based on another published manual (Andersson et al., 2006) and emphasized the use of behavioural experiments and exposure as a means of challenging dysfunctional patterns in cognition. This treatment also highlighted the importance of reducing self-focus as a type of safety behaviour (e.g. McManus et al., 2008), and promoted strategies for enhancing social skills, primarily in terms of active listening and the use of open-ended questions to maintain a conversation. For a more extensive description of the ICBTs, see the supplementary material of Niles et al. (2021).

Measures

The patients completed self-report questionnaires via the same web platform used for delivering ICBT. Patients completed the domain symptom scale and all process scales that corresponded to their primary diagnosis, at the beginning of each week over the 10-week treatment, thus resulting in 10 assessment points. The first and last assessment were more exhaustive, as all patients completed all domain symptom scales. Each intermediate weekly assessment consisted of approximately 45–52 items. All scales were administered in Swedish, and simple (non-weighted) sum scoring was used for all questionnaires throughout the study.

Domain-specific symptom outcomes

We measured depression symptoms using the Montgomery-Åsberg Depression Rating Scale-Self-report version (MADRS-S; Svanborg and Åsberg, 1994) which is well documented to have adequate psychometric properties (Fantino and Moore, 2009; Thorndike et al., 2009). The MADRS-S consists of nine items, each scored 0–6, resulting in a 0–54 sum score. In the selection/training data, at baseline, the MADRS-S was unifactorial and exhibited adequate internal consistency (α=0.85). We measured panic disorder symptoms using the Panic Disorder Severity Scale-Self-report (PDSS-SR; Houck et al., 2002) which is typically found to have adequate psychometric properties (Houck et al., 2002; Shear et al., 2001). The PDSS–SR consists of seven items, each scored 0–4, resulting in a 0–28 sum score. In the selection/training data, at baseline, the PDSS-SR was unifactorial and exhibited excellent internal consistency (α=0.91). We measured social anxiety disorder symptoms using the Liebowitz Social Anxiety Scale-Self-report (LSAS-SR; Fresco et al., 2001) which is typically found to have adequate psychometric properties (Baker et al., 2002; Fresco et al., 2001; Hedman et al., 2010). The LSAS-SR consists of 24 items, each with one fear and one avoidance subitem each scored 0–3, resulting in a 0–144 sum score. In the selection/training data, at baseline, the LSAS-SR was unifactorial and exhibited excellent internal consistency (α=0.97).

Depression process scales

This study involved three process scales focusing on the treatment of depression. We administered an 8-item version of the Automatic Thoughts Questionnaire (ATQ) which is usually found to be unifactorial (Hollon and Kendall, 1980; Netemeyer et al., 2016). From the Behavioral Activation for Depression Scale (BADS), we administered two established subfactors: the 7-item Activation subscale (BADS-AC) and the 8-item Avoidance/Rumination subscale (BADS-AV/R) (Kanter et al., 2006; Kanter et al., 2008). The Reward Probability Index (RPI) is commonly found to have two subfactors: Reward Probability (RPI-RP) and Environmental Suppressors (RPI-ES) (Carvalho et al., 2011). We administered a subset of 13 items from the original RPI (seven items from the RPI-RP and six from the RPI-ES), as originating from an earlier project (ClinicalTrials.gov: NCT01636791, NCT01667822) where these items were chosen by clinical psychologists with expert knowledge in CBT. This was done on the basis of face validity and coherence relative to theory including coverage of the construct and its facets, phrasings suitable for repeated measurements, and high factor loadings.

Panic disorder process scales

This study involved two process scales focusing on the treatment of panic disorder. The Agoraphobic Cognitions Questionnaire (ACQ) has 14 items, and is commonly found to have two subfactors: the 7-item Physical Consequences subscale (ACQ-PC) and the 7-item Loss of Control subscale (ACQ-LOC) (Chambless et al., 1984). The Behaviors Questionnaire (BQ) is usually divided into the Avoidance (BQ-A) and Safety Behaviors (BQ-SB) subscales (Centre for Anxiety Disorders & Trauma (CADAT) at King’s College London, 2020; Marks and Mathews, 1979). We administered a subset of 10 items from the conventional BQ-A and 15 items from the conventional BQ-SB, as originating from an earlier project (ClinicalTrials.gov: NCT01636791, NCT01667822; see above).

Social anxiety disorder process scales

This study involved three process scales focusing on the treatment of social anxiety disorder. We administered 10 items from the Social Cognitions Questionnaire (SCQ); five relating to the believability, and five to the frequency, of cognitions common in social anxiety (Oxford Centre for Anxiety Disorders and Trauma (OxCADAT), 2022), as originating from an earlier project (ClinicalTrials.gov: NCT01636791, NCT01667822; see above). In accordance with Hoffart et al. (2009), we also administered eight items from the Social Probability and Cost Questionnaire (SPCQ) which concerns the perceived probability and valence of adverse social outcomes (Foa et al., 1996; Hoffart et al., 2016; McManus et al., 2000). The Social Behavior Questionnaire (SBQ) is usually found to have two subfactors: Avoidance (SBQ-A) and Impression Management (SBQ-IM) (Gray et al., 2019). Originating from the same study as the items of the SCQ, we also administered a subset of five items from the SBQ-A and five from the SBQ-IM.

Statistical analyses

Effectiveness of ICBT

We conducted the statistical analyses using Stata 15 and R 4.1.0 (R Core Team, 2016) with lavaan 0.6-8 (Rosseel, 2012). In the assessment of adherence to the protocol, patients were classified as probable treatment drop-outs if they either (i) missed at least three assessment points in a row and did not complete subsequent assessment points or (ii) completed less than four assessment points in total. Prior to the main analyses pertaining to the development of brief process scales, we assessed change in depression symptoms, panic disorder symptoms, and social anxiety disorder symptoms over the course of ICBT to ensure that there were beneficial effects on each domain outcome. For this purpose, all patients with at least one measurement point were included in linear mixed effects regression models with a random intercept and slope, unstructured covariance over these, and an autoregressive residual covariance structure. We calculated Cohen’s d standardized effects as the model-implied change score (intention to treat, i.e. using data from all patients regardless of adherence), divided by the standard deviation of change as based on observed data. This type of effect size which weighs in both the mean change and the variance in change (the pre- to post-treatment correlation) can be thought of as a ‘signal to noise ratio’ that is similar to a traditional test statistic (Morris and DeShon, 2002).

Main phase 1: Item selection and drafting of the brief process scales

In accordance with recommendations, in reducing the number of process scale items, we strived to marry theoretical considerations with empirical considerations (Ziegler et al., 2014) and revised our selection of items in an iterative manner based on the psychometric evidence (Furr, 2011). We first formulated each target construct, and then decided on which psychometric properties that were desirable for a brief CBT process scale, before assessing these in the selection/training data (n=370; Kleka and Soroko, 2018). Whenever a full process scale had 10 or fewer items, we attempted to draft a brief version of this scale as a whole (the ATQ, SCQ, SPCQ, and initially also the SBQ). If the full process scale had more than 10 items, we instead drafted a brief version of each subscale, and, treated these as full scales to be shortened (the BADS-AC separate from the BADS-AV/R, the RPI-RP from the RP-ES, the ACQ-PC from the ACQ-LOC, and the BQ-A from the BQ-SB). Two assessors (E.A. and B.L.) independently reviewed the psychometric evidence derived from the selection/training data. Each assessor selected three to four items from each scale (or subscale) based on the following:

• We conducted a factor analysis with all scale items and the domain symptom scale. This was based on principal axis factoring with promax rotation, and the idea was to retain items that allowed us to capture all dimensions, ideally with standardized factor loadings ≥0.40. The Kaiser-Meyer-Olkin (KMO) statistic was at least 0.70 in conjunction with a significant Barlett’s test for all full scales (see Supplementary material, Table B3). All item distributions are tabulated in the Supplementary material, Table B1.

• We modelled item change over time using linear mixed effects models and standardized effect sizes. A quadratic time term (time×time) reflective of a curvilinear effect was added if this improved the model fit. The idea was to give priority to items that changed more over time in terms of a standardized within-group effect (d). The numerator for d was derived from the linear mixed model, i.e. either the coefficient for the simple effect of time whenever a curvilinear relationship was lacking, or the sum of the coefficients for time and time×time whenever inclusion of the latter had improved model fit.

• We explored the relationship between the fitted item slopes and the corresponding domain symptom slopes, as based on Pearson correlations and also the factor analysis of fitted slopes. That is, having derived the fitted item slopes from the linear mixed models, we did two things: first, for the slopes of each item, we calculated the correlation with the corresponding domain outcome slopes; second, we conducted a factor analysis of the item slopes of each scale. Ideally, we wanted to proceed with items that captured change in the full instrument and showed a clear relationship with the domain outcome in terms of change.

• We explored if change (fluctuations) in each item (e.g. of the ATQ) was uniquely predictive of subsequent change (fluctuations) in the domain outcome (e.g. the MADRS-S) on a week-by-week basis, using random-intercepts cross-lagged panel models (Hamaker et al., 2015) specified analogous to Axelsson et al. (2020). This effect (γ) was constrained to be equal from week to week (i.e. the unstandardized effect), and we preferred to see that change (fluctuations) in item scores were predictive of subsequent change (fluctuations) in the domain outcome.

• We assessed the face validity of each item in terms of its phrasing (Was it easy to understand? Could it easily be misunderstood?) and the relationship to cognitive behavioural theory (Does it measure what the scale is intended to measure? Does the selection of items cover the theoretical construct?). This was to ensure that each brief scale would measure what it was intended to measure, and that the brief scale sums would accurately capture the same latent traits, and pick up on the same processes of change, as the full scale sums.

A third assessor (F.S.) reviewed the suggestions for item selection by E.A. and B.L., and formulated suggestions for 3–4 item brief scales. This formed the basis for a subsequent discussion where preliminary brief scales were drafted by means of voting. Still using the selection/training data, these items sets were analysed as preliminary brief scales, primarily to ensure that the internal consistency appeared to be acceptable, and that change in the summed scale correlated with change in the domain outcome in the same manner as the individual items. This was an important step, considering that it is theoretically possible, although unlikely, for the psychometric properties of the sum scale not to mirror properties of the individual items. We employed unweighted item summation for all scales except the SCQ, where we first multiplied the a items (scored 1–5) by 2 to approximate the range of the b items (scored 0–10). This review of the brief scale sums resulted in a few changes to the preliminary versions (see Results). We then had a final list of brief scales to be validated.

Main phase 2: Validation in a new sample

We evaluated the psychometric properties of the brief scales in a new, separate, validation dataset. Five aspects were explored. First, the correlation with the corresponding full scale. Second, the internal consistency of the brief scale, compared with the full scale. Third, the standardized effect size pertaining to change in the brief scale, compared with the full scale. This was based on linear mixed models, as detailed above. Fourth, we tested if there was a week-by-week unique predictive effect of change (fluctuations) in the brief scale on subsequent change (fluctuations) in the domain outcome, and contrasted the standardized effect related to that of the full scale. This was based on random-intercepts cross-lagged panel models (Hamaker et al., 2015), as detailed above. Fifth, we tested if the brief scale mediated change in the domain outcome, and contrasted the standardized effect to that of the full scale. This test for mediation was based on separate linear mixed effects models for the a path from time to the potential mediator, and the b path from the potential mediator to the domain outcome. The models were fitted on all measurement points, and we constructed 95% confidence intervals for the a×b products using bias-corrected cluster bootstrapping (MacKinnon et al., 2004).

Results

Adherence to the protocol

Missing data rates are presented in Fig. 1. The average patient completed 8.4 out of 10 (SD=2.6) assessment points. The rate of probable treatment drop-outs was 13% (28/218) in ICBT for depression, 17% (31/183) in ICBT for panic disorder, and 16% (43/262) in ICBT for social anxiety disorder.

Effectiveness of ICBT

As was first reported by Niles et al. (2021), patients in the 2014–2017 selection/training sample (n=370) improved over ICBT for depression (MADRS-S: b=–7.9; 95% CI=–9.3 to –6.6; d=1.11), ICBT for panic disorder (PDSS-SR: b=–5.3; 95% CI=–6.2 to –4.3; d=1.07), and ICBT for social anxiety disorder (LSAS-SR: b=–25.5; 95% CI=–29.2 to –21.7; d=1.12). Similarly, the patients of the 2018–2021 validation sample (n=293) improved over ICBT for depression (MADRS-S: b=–8.7; 95% CI=–10.1 to –7.3; d=1.31), ICBT for panic disorder (PDSS-SR: b=–5.5; 95% CI=–6.6 to –4.4; d=1.23), and ICBT for social anxiety disorder (LSAS-SR: b=–24.5; 95% CI=–28.9 to –20.1; d=1.12). Results were similar, with slightly higher point estimates, in patients who completed all 10 weekly assessments (Supplementary material, supplement B).

Main phase 1: Item selection and drafting of brief process scales

Each of the two independent assessors selected 43 out of 103 items for the brief scales, and agreed on 29 of these (67%; κ=0.44). Absolute agreement was achieved for the RPI-ES and SBQ. After the discussion with assessor 3, the resulting preliminary brief scales were scored in the selection/training data. We found two preliminary scales to be problematic. The BADS–AV/R had been reduced to a 4-item scale where α was 0.62, which we did not find satisfactory. Because we could not draft an alternative, more satisfactory, 3- or 4-item version, we instead proceeded with a longer 6-item version. Also, the SBQ had been reduced to a 4–item scale where α was 0.59. We therefore split the SBQ into two brief subscales: the SBQ–A and SBQ-IM. For the final selection of items for the brief process scales, see the ‘Brief’ column of Table B1 in the Supplementary material.

Main phase 2: Validation in a new sample

The main outcomes of the validation phase are presented in Table 2. Non-adjusted correlations were substantial (≥.83). Internal consistency was adequate for all brief scales except the RPI-ES (α=0.65). Most brief scales changed to a moderate to large degree, similar to their full-scale equivalents. Week-by-week temporal precedence was found for brief versions of the BADS-AC, ACQ-PC, SCQ, SPCQ, SBQ-A and SBQ-IM. All brief scales mediated the effect of time on the domain outcome as indicated by significant a×b products. The final brief scales can be found in Swedish and English in the Supplementary material.

Table 2. Brief process scales: psychometric properties and relationship to the corresponding full scales in the validation data (n=293)

Gamma (γ) is derived from random-intercepts cross-lagged panel models, and stands for the median completely standardized unique effect of deviations in the brief process scale score of one week, on deviations in the domain outcome score of the following week. ACQ-LOC, Agoraphobic Cognitions Questionnaire-Loss of Control subscale; ACQ-PC, Agoraphobic Cognitions Questionnaire-Physical Consequences subscale; ATQ, Automatic Thoughts Questionnaire; BADS-AC, Behavioral Activation for Depression Scale-Activation subscale; BADS-AV/R, Behavioral Activation for Depression Scale-Avoidance/Rumination subscale; BQ-A, Behaviors Questionnaire-Avoidance; BQ-SB, Behaviors Questionnaire-Safety Behaviors; MDD, major depressive disorder; PD, panic disorder; RPI-ES, Reward Probability Index-Environmental Suppressors subscale; RPI-RP, Reward Probability Index-Reward Probability subscale; SAD, social anxiety disorder; SBQ-A, Social Behavior Questionnaire-Avoidance subscale; SBQ-IM, Social Behavior Questionnaire-Impression Management subscale; SCQ, Social Cognitions Questionnaire; SPCQ, Social Probability and Cost Questionnaire.

^a Non-adjusted Pearson correlation. Each full scale was rescored as the brief scale, meaning that all scores on shared items were identical.

^b Including the curvilinear effect of time on the brief process scale, which improved model fit when modelling the a path.

Discussion

In this study, we developed and validated brief versions of 13 questionnaires intended to capture target variables and core processes of change in CBT for depression, panic disorder, and social anxiety disorder. We found that several of the original scales showed promise in terms of their psychometric properties and also as mediators of the treatment effect. As is illustrated in Table 2, most of these beneficial properties appeared to be maintained also when the number of items per scale was roughly cut in half. We regard this to be an important finding, considering that around two times as many constructs can be measured using equal resources. The use of brief process scales can also facilitate repeated measurement regimens which could prove helpful for clinicians and researchers alike (Kazdin, 2007; Lutz et al., 2022; Ziegler et al., 2014).

Specific recommendations

For depression, we recommend use of the brief BADS-AC which showed promise as a process measure of behavioural activation. The brief ATQ, RPI-RP and BADS-AV/R are also worth studying further, although a week-by-week time-lagged effect on depression was not seen here. The brief RPI-ES which purports to measure environmental suppressors of rewards exhibited questionable internal consistency and face validity, and is probably not worth studying further. Notably, very few previous studies have been concerned with brief CBT process scales for depression. In one study, the BADS-AC was reduced to 5–6 items, i.e. twice as long as the 3-item version evaluated here (Manos et al., 2011). In two single-subject cases, that version appeared to be promising for tracking treatment progress but no more extensive longitudinal evaluation was ever conducted. Interestingly, the brief version developed here was not part of the 5- to 6-item version but yet exhibited adequate psychometric properties.

For panic disorder, we recommend use of the brief ACQ-PC which showed promise as a measure of catastrophic thoughts about the physical consequences for panic attacks. The brief ACQ-LOC, BQ-A and BQ-SB are also worth studying further. To our knowledge, no previous study has evaluated the psychometric properties of brief CBT process scales for panic disorder. There are, however, recent examples of longer alternatives; one being the Oxford Cognitions and Defences Questionnaire with subscales for threat cognitions, anxious avoidance, and safety behaviours each consisting of 8–14 items, i.e. each at least twice as many items as the brief scales evaluated here (Rosebrock et al., 2022).

For social anxiety disorder, all four brief process scales – i.e. the SCQ, SPCQ, SBQ-A, and SBQ-IM – exhibited promising psychometric properties and, tentatively, we recommend further use and evaluation of all of these. The authors of a recent systematic review of cognitive components in social anxiety concluded that ‘future studies would benefit from the inclusion of […] multiple well-validated measures within the same domain of social cognition […] outside of the laboratory’ (Alvi et al., 2022). Considering that we are not aware of previous attempts at developing brief CBT process scales for social anxiety disorder, the scales evaluated here could potentially be of use in this endeavour.

Caveats regarding temporal precedence

We wish to comment briefly on the random-intercepts cross-lagged panel models that we used to assess whether change in the brief scales was systematically predictive of subsequent change in the corresponding symptom domain on a week-by-week basis. Although establishing a timeline from change in the process scale to subsequent change in the presumed outcome is an important aspect of investigating causation (Kazdin, 2007), for many of the target variables probed by these brief scales it is not clear over what time frame (seconds, minutes, hours, days, weeks, months?) effects on the domain outcome are to be expected. Thus, even though we could not demonstrate a week-by-week temporal precedence for any of the depression target variables except behavioural activation (the brief BADS-AC), nor any of the panic disorder targets except catastrophic thoughts about the physical consequences of panic attacks (the brief ACQ-PC), it cannot be ruled out that the expected temporal ordering effects could be observed with another temporal resolution (e.g. minute to minute, or hour to hour) than in this study. There is also an ongoing debate over which statistical modelling techniques that are to be preferred for the study of time-lagged effects, and how such effects should be interpreted (Usami et al., 2019).

Strengths and limitations

An important strength of this study is that it was conducted in a structured setting where all patients were diagnosed with a primary psychiatric disorder corresponding to their treatment, and the ICBT format ensured that similar treatment content was conveyed to all participants, which speaks for reproducibility. Another strength is that all process measures were administered over 10 assessment points, which made analyses of change relatively robust. An important limitation is that there was no control group, which means that estimates of change may not mirror causal effects. We administered several scales in parallel which may have led to fatigue that affected estimates. Another interesting threat to the validity of this study is that, although we used widely recognized symptom scales, there is some degree of conceptual overlap between these and the process measures. For example, the ATQ measured ‘automatic negative thoughts’ while 1 out of 9 items of the MADRS-S established the level of depression by asking the patient about pessimism. This said, symptom scales of the kind used in this study are widely recognized as valid indicators of psychopathology and of primary interest in much of psychological treatment research. Another limitation is that while we drafted brief scales to tap into the processes of change that occurred naturally as part of CBT, our results say less about the usefulness of the brief process scales as clinical tools for the prospective adaptation of CBT during the course of a treatment. Importantly, the present study did not focus on retaining items in the sense that we wanted for example every type of avoidance behaviour to be surveyed for each psychiatric condition. Rather, we focused on whether each brief scale sum would be appropriate to use instead of the corresponding full-scale sum. Overall, results were promising in this regard, although we wish to remind the reader that the BQ and SCQ appear to lack primary peer-reviewed journal publications which means that even though their brief scales versions appeared to be promising in this study, there is little reference data pertaining to the psychometric properties of the full-scale equivalents beyond what was presented here. We also wish to bring to the reader’s attention that CBT protocols can differ in their components, and that we evaluated the role of these brief process scales in relation to three specific CBT protocols. It is conceivable that some of the scales evaluated in this study had seemed more promising, and some less promising, had we focused on CBT protocols with a slightly different emphasis. For example, we found the BADS-AC to be especially promising as a process variable in CBT for depression, but this may, at least to some degree, be because our CBT protocol for depression focused heavily on behavioural activation. The implication of this is that it is important to evaluate the brief scales also with other CBT protocols in order to determine their usefulness as process scales in that particular context. Another limitation is that the samples sizes were generally smaller than ideal, and power was limited for example in the study of time-lagged effects for panic disorder in the validation data (n=77), which means that these estimates need be interpreted with caution and warrant replication. Last, the validation phase of this study was based on further rescoring of the existing full process scales rather than actual administration of the brief scales. Considering the potential impact of ordering effects (e.g. Knowles, 1988), this further highlights the need for replication.

Conclusion

In this study, we have presented several brief scales that can be used to measure target variables in CBT for depression, panic disorder, and social anxiety disorder. These brief scales can potentially facilitate treatment development, research into processes, and the evaluation of treatment progress. Results were generally promising, but based on limited sample sizes, with data derived from one clinical setting only, and without a control group. We wish to emphasize that there is a need for replication and further psychometric evaluation of these brief scales, ideally using variations on CBT protocols, based on experimental designs, and in larger samples.