Development and application of criteria to evaluate written CBT self-help interventions adopted by Improving Access to Psychological Therapies services

Abstract

Guided CBT self-help represents a low-intensity intervention to deliver evidence-based psychological therapy within the Improving Access to Psychological Therapies (IAPT) programme. Best practice guidance highlighting characteristics associated with CBT self-help is available to help services reach decisions regarding which interventions to adopt. However, at present a single process to evaluate written CBT self-help interventions informed by guidance is lacking. This study reports on the development of a standardised criteria-driven process that can be used to determine the extent written CBT self-help interventions are consistent with guidance regarding the fundamental characteristics of low-intensity CBT and high-quality written patient information. Following development, the process was piloted on 51 IAPT services, with 23 interventions identified as representing free-to-use written CBT self-help interventions. Overall, inter-rater reliability was acceptable. Following application of the criteria framework, 14 (61%) were considered suitable to be recommended for use within the IAPT programme. This pilot supports the development and potential utility of an independent criteria-driven process to appraise the suitability of written workbook-based CBT self-help interventions for use within the IAPT programme.

Key learning aims

1. (1) To recognise the range of written low-intensity CBT self-help interventions currently used within IAPT services.

2. (2) To identify separate criteria associated with high-quality written CBT self-help interventions.

3. (3) To use identified criteria to develop a framework to evaluate written workbook based low-intensity CBT self-help interventions for use within the IAPT programme.

4. (4) To evaluate inter-rater reliability of the criteria framework to evaluate the quality and appropriateness of written workbook based low-intensity CBT self-help interventions used within IAPT services.

Introduction

The self-help genre represents a specific mode of reading that values books as tools to promote action, facilitate self-change and enhance well-being (Blum, 2020). The concept of self-help targeting ‘self-actualisation’ and offering general advice on an area leading to self-improvement has now become mainstream (Harwood and L’Abate, 2010). Although evidence regarding the effectiveness of specific titles remains limited (Harwood and L’Abate, 2010), written or computerised self-help is increasingly recognised as a legitimate way to provide psychological therapy (Delgadillo, 2018). The most well-established use of self-help interventions to serve as a platform for the delivery of psychological therapy is associated with cognitive behavioural therapy (CBT; Farrand, 2020).

Written CBT self-help became popularised through the publication of titles such as Living with Fear (Marks, 1978), The Feeling Good Handbook (Burns, 1999) and Mind Over Mood (Greenberger and Padesky, 1995). In general, these and subsequent written CBT self-help interventions follow a standard format providing psychoeducation about mental health problems and use a CBT model to help the reader appreciate ways they may experience the difficulty, followed by several specific CBT techniques to address the problem. Informed by a CBT approach (Beck, 2020), the user engages with the written self-help intervention through interactive worksheets setting out specific CBT techniques. In effect the written self-help intervention guides the reader through a CBT treatment protocol (e.g. CG113; National Institute for Health and Care Excellence, 2020). In the case of Mind Over Mood (Greenberger and Padesky, 1995) a Clinician’s Guide (Padesky and Greenberger, 1995) has also been developed to facilitate use of the written intervention as adjuvant to therapist-delivered CBT. Requiring the reader to engage with written CBT self-help interventions, the approach extends beyond standard psychoeducation (Goldman, 2006) and different from self-help books that provide advice rather than encouraging active engagement through worksheets (Schindler Zimmerman et al., 2004).

The effectiveness of CBT self-help for the treatment of common mental health difficulties has been demonstrated through an evidence base consisting of over 30 systematic reviews and 50 controlled trials (Delgadillo, 2018). Additionally, no significant differences in effectiveness or drop-out up to one-year post-assessment have been reported between CBT self-help and face-to-face psychotherapies (Cuijpers et al., 2010). Informed by this evidence base, guided CBT self-help is implemented within the Improving Access to Psychological Therapies (IAPT) programme (National Collaborating Centre for Mental Health, 2020). However, there is considerable heterogeneity regarding the characteristics of the CBT self-help interventions adopted within studies that form the basis of NICE recommendations (Farrand, 2020).

CBT self-help interventions vary in the specific CBT techniques and content they adopt (Lewis et al., 2012). Such heterogeneity is problematic given that CBT self-help interventions represent Step 2 of the stepped care service delivery model adopted by the IAPT programme (Richards, 2010). These interventions have several characteristics that distinguish them from therapist-delivered high-intensity CBT (HICBT) (Farrand, 2020). Whilst HICBT recognises the presenting difficulty in the ‘here and now’, emphasis is also placed upon factors that may have precipitated or maintain the difficulty, through a developmental formulation (Beck, 2020). This helps inform the range of CBT techniques that HICBT therapists may use to address cognitive distortions in the form of negative automatic thoughts in the ‘here and now’, alongside unhelpful assumptions and negative schemas that exist at different levels of cognition (Beck, 2020). In contrast, the focus of low-intensity CBT (LICBT) is solely on the ‘here and now’, with CBT self-help interventions comprising a ‘single-strand’, where during assessment a single CBT technique that will comprise the LICBT self-help intervention is identified (Turpin et al., 2010). This contrasts with HICBT, where evidence-based treatment protocols usually specify the delivery of several different CBT techniques.

As the characteristics of LICBT within the IAPT programme have become better established (Farrand, 2020), it is surprising that guidance to evaluate CBT self-help interventions (e.g. UCL, n.d.-a and n.d.-b) have largely focused on factors such as their evidence base, quality (Charnock, 1998) and engagement (Martinez et al., 2008). However, identifying fundamental characteristics that distinguish LICBT from HICBT is needed to ensure CBT self-help interventions demonstrate fidelity to the LICBT clinical method and wider fundamental assumptions of CBT (Beck, 2020). With respect to written CBT workbook based self-help interventions commonly adopted by IAPT services, this paper reports on the development and application of a single criteria framework to evaluate both quality and consistency with the fundamentals of LICBT informing the IAPT programme.

Method

Developing criteria for rating written workbook based CBT self-help interventions

Recruiting the criteria application panel

To recruit qualified psychological wellbeing practitioners (PWPs) still in practice or trainers on LICBT programmes onto the criteria application panel, advertisements were sent to all LICBT accredited training providers with a request to forward to all services commissioning training. There were nine respondents, with the first four from different IAPT services (located in London, the Midlands and the South-East) selected to form the criteria application panel, along with P.F. An Expert by Experience who had engaged with CBT self-help as a treatment (M.D.) was also recruited by a PWP panel member.

Criteria framework development

Following consultation between P.F. and A.R., a single criteria framework to rate written workbook based CBT self-help interventions and make recommendations for IAPT services was developed. Consultation was based on a review of the literature surrounding written CBT self-help interventions within the IAPT programme, recommendations regarding written CBT self-help interventions and criteria associated with good quality written patient information (Table 1).

Table 1. Literature informing criteria framework development

Literature review identified 26 criteria incorporated to create a single criteria framework. These criteria were organised into five domains – Scope Consistent with Low-Intensity CBT, Accurate and Clear Information, Engagement, Usability and Behavioural Principles. The criteria for demonstrating fidelity was set as a minimum average score of 4 for individual items within each domain (highlighted in grey in Appendix 1 of the Supplementary material) except all Behavioural Principles criterion (University College London, 2015) given that these were a recent addition to the curriculum (UCL, n.d.-b).

Following development of the pilot criteria (Appendix 1 of the Supplementary material), discussion was undertaken with panel members to ensure it appropriately captured the use of written CBT self-help interventions by PWPs and addressed areas of importance from an ‘Experts by Experience’ perspective. To reach consensus and inform development of the pilot criteria, adaptations were made to the name given to one of the domains (Current adapted to Accurate and Clear Information). Additionally, minimum average scores that were required to be met to meet criteria (highlighted in grey in Appendix 1 of the Supplementary material) was reduced from 4 to 3 on two specific criteria (Complementary, Collaborative), on the basis that support could compensate for these criteria if not fully addressed in the intervention directly. Following adaptation, it was ensured that the pilot criteria framework maintained consistency with the characteristics of LICBT stated in the IAPT Manual (National Collaborating Centre for Mental Health, 2020). At the end of this meeting, M.D. identified domains or individual criterion she did not feel comfortable rating due to lack of knowledge. Furthermore, it was agreed that P.F. would not be involved in rating any intervention he was associated with to avoid a conflict of interest. Following application, further discussions were planned with members of the criteria panel to discuss the process generally and to particularly focus on specific criterion if poor agreement was identified.

Identifying written workbook based CBT self-help interventions adopted by IAPT services

A member of the national IAPT programme team sent a request to IAPT services in the 156 Clinical Commissioning Groups asking them to identify a member of staff who could list the written CBT self-help interventions adopted by their service. On return, these were screened by P.F. to ensure these interventions met basic criteria – written, free to use, did not represent a single or collection of individual worksheets addressing different techniques and were accessible for use via a weblink.

Pilot application of the criteria framework

Members of the panel were sent a list of weblinks to the written workbook based CBT self-help interventions that met criteria (and for pragmatic reasons, were adopted by more than four services). Panel members were asked to independently apply the criteria (Appendix 1 of the Supplementary material) to each written LICBT intervention, rating each individual criteria using the 5-point Likert scale (1, not at all; 5, completely).

Inter-rater reliability

Each member of the criteria application panel submitted ratings for each criterion included within the criteria framework and were independently coded and analysed by a researcher not otherwise associated with the study. For each domain addressed in the criteria framework (see Table 3), ratings for individual criteria were combined and an average domain rating separately calculated for each panel member. Fleiss’ Kappa (κ) was used to examine the level of agreement between members for each domain. Reporting reliability results was informed by Kottner et al. (2011) with level of agreement interpreted according to guidelines (Landis and Koch, 1977).

Results

Intervention rating

Responses were received from 51 IAPT services, with at least one service from each of the seven NHS commissioning regions. Twenty-three free to use and accessible written workbook based CBT self-help interventions were identified, of which 16 (70%) met criteria for a written workbook based LICBT self-help intervention (Table 2).

Table 2. Recommendations for written CBT self-help interventions for common mental health difficulties treated by the IAPT programme and adopted by four or more services

On all occasions, interventions targeting a common mental health difficulty but not meeting criteria failed to achieve the minimum rating for the Single Strand criterion. One intervention (Depression; Centre for Clinical Interventions) was also rejected because it employed a HICBT technique to challenge core beliefs, which is outside of the LICBT clinical method. Goal Setting (Farrand and Woodford, 2013a, 2013b) failed to meet criteria as it was not considered to represent an IAPT LICBT intervention.

Sixteen (31%) of the 51 services reported only using CBT technique-specific worksheets that do not represent a written CBT workbook self-help intervention. Worksheets developed by the Centre for Clinical Interventions to address anxiety and sleep were reported as being used by 10 (20%) and seen (14%) of services, respectively. Six (12%) services also reported using Reach Out (Richards and Whyte, 2011) training resources for Cognitive Restructuring and Exposure. Furthermore, 21 (41%) services reported using worksheets freely available on several websites. Six (29%) of these services also indicated employing worksheets that were directed at a mental health problem or behaviour not treated with LICBT within the IAPT programme (e.g. binge-eating disorder, bipolar disorder, body dysmorphia, self-harm). A range of intervention-specific worksheets unrepresentative of the LICBT clinical method were also adopted (e.g. coping strategies, interoceptive exposure sheet, positive log, responsibility pie chart).

Inter-rater reliability

Level of agreement between panel member ratings varied across each of the domains addressed in the criteria (Table 3).

Table 3. Agreement (Fleiss’ kappa, κ) between panel members on average score for each domain

Substantial agreement was identified for Behavioural Principles (κ = 0.77) and Accurate and Clear Information (κ = 0.64). With respect to Scope Consistent with IAPT LICBT (κ = 0.58) and Engagement (κ = 0.51), moderate levels of agreement arose, with only fair agreement identified for Usability (κ = 0.39). Given that only a fair level of agreement was identified for the Usability domain, inter-rater reliability was calculated for individual criteria within this domain. Moderate levels of agreement were identified for each individual criterion except for Support Risk Management (κ = 0.34) and Inviting Presentation (κ = 0.32).

Discussion

This study describes the development and application of a single criteria framework to evaluate written workbook CBT self-help interventions adopted by the IAPT programme. The framework was practical and easy to use, with at least moderate levels of inter-rater reliability across four of the domains addressed. A fair level of agreement was identified for Usability, however, with lower inter-rater reliability on the Support Risk Management and Inviting Presentation domains.

Pilot application of the criteria identified that only a small number of written CBT self-help interventions targeting specific common mental health difficulties were consistent with LICBT adopted within the IAPT programme. Not adopting a single LICBT technique but continuing to employ multi-strand CBT techniques associated with HICBT (Farrand, 2020), was the most common reason for written workbook CBT self-help interventions failing to meet criteria. Perhaps unsurprisingly, the majority of recommended written workbook CBT self-help interventions were developed by training providers (e.g. CEDAR; University of Exeter, University of Manchester) or services (e.g. TalkPlus) with close connections to the IAPT programme.

Except for Usability, criteria demonstrated at least moderate levels of inter-rater reliability across domains. The minimum level of inter-rater reliability required to recommend criteria use is the subject of debate, although it is proposed that the situation to which criteria are applied should be considered when interpreting inter-rater reliability and inform adoption (O’Neill, 2017). As the aim of the criteria were solely to provide a tool to evaluate written CBT self-help interventions for recommendation, moderate inter-rater can potentially be argued to justify adoption. Within the Usability domain, inter-rater reliability was reduced to fair, dependent on lower levels of agreement on the specific criterion Support Risk Management and Inviting Presentation.

Given lower levels of agreement on these selected criteria, a consultation process was undertaken between members of the review panel to identify reasons and generate solutions. Differences on the risk management criterion arose from varied perspectives regarding the need for written CBT self-help interventions to have a dedicated section addressing action to take were the patient to consider themselves at risk. Adopting recommendation regarding the need for specific and clear information regarding risk (e.g. Duman and Farrell, 2000), it was decided to revise the criterion to state the need to have a specific section of the intervention directing patients to take appropriate action and providing signposting information (Appendix 1 of the Supplementary material). Lower levels of agreement on Inviting Presentation potentially highlights differences in general aesthetic preferences between people and when applied to product design (Hoegg, 2015). During consultation it was recognised that whilst differences will exist, reducing the number of guidance documents associated with good quality written information addressing aspects of presentation could serve to improve agreement. For future applications of the criteria framework therefore, for issues related to presentation it was decided to solely consult a review of the evidence regarding good patient information developed by the Patient Information Forum (Treadgold and Grant, 2014). This review also highlighted benefits of personalising and tailoring presentation to enhance acceptability for different groups. Involving Experts by Experience from groups with diversity to adapt the criteria framework for specific populations could help ensure greater inclusivity.

Encouragingly, the majority of interventions selected for review were consistent with many criteria surrounding the core assumptions of CBT such as interactivity (e.g. Richards and Farrand, 2010; University College London, n.d.-a) and guidance regarding good written patient information (e.g. EQIP: Moult et al., 2004; NHS Patient Information Toolkit: Department of Health, 2003). However, there were some notable exceptions regarding specific criteria that may be helpful for authors to include when planning to develop LICBT self-help interventions. For example, providing a brief description of author background and experience (rather than relying on the name of an organisation) is recommended to establish competency (e.g. DISCERN; Charnock et al., 1998). Doing so has potential to enhance a patient’s sense of treatment credibility and improve effectiveness (Constantino et al., 2018). Additionally, as LICBT interventions are intended to be supported by the IAPT programme (National Collaborating Centre for Mental Health, 2020), setting out the way in which this support is offered within the LICBT self-help intervention is important and may help foster and maintain engagement. For example, enabling the patient to spend similar amounts of time engaging with the self-help intervention between support sessions as a patient spends engaging with HICBT within treatment sessions (van Straten et al., 2015). Finally, interventions should include signposting so that users are directed to other sources of support if their mental health difficulty becomes worse between support sessions (e.g. Treadgold and Grant, 2014).

Although results from this pilot study do not represent all IAPT services, applying the criteria framework highlights that a wide range of written LICBT interventions varying across services are adopted. This highlights the utility of developing a criteria framework to evaluate interventions already being employed and new ones being considered for adoption. This is important given that use of the criteria framework highlights several areas of concern regarding the adoption of LICBT self-help interventions. Given the CBT self-help genre generally employs techniques associated with HICBT (e.g. Burns, 1999; Greenberger and Padesky, 1995), it is perhaps unsurprising that these CBT self-help interventions adopted multiple CBT techniques and therefore did not meet the single-strand criteria (Turpin et al., 2010). The addition of several techniques resulted in extending the CBT clinical method from a focus on the presenting difficulty in the ‘here and now’. This included techniques to address different levels of cognitive distortion at the level of core beliefs (Depression; Centre for Clinical Interventions). Whilst not being consistent with the characteristics of LICBT self-help interventions adopted in the IAPT programme (Farrand, 2020), such CBT self-help books continue to have utility when adopted as a HICBT intervention.

Many services reported using individual CBT worksheets rather than incorporated within a self-help intervention to be guided by PWPs (National Collaborating Centre for Mental Health, 2020). Whilst the criteria have only been developed to evaluate written CBT self-help workbooks and not single worksheets, caution would need to be exercised if worksheets are employed outside of a workbook format. Using a range of individual worksheets may result in PWPs applying competencies more commonly associated with HICBT and result in ‘therapeutic drift’ (Waller, 2009; Waller and Turner, 2016). This is problematic because many HICBT competencies go beyond those identified by the national PWP national training curriculum (University College London, 2015). Furthermore, several services reported using worksheets targeting mental health problems beyond those treated with LICBT and not supported by the evidence base (National Collaborating Centre for Mental Health, 2020). Adopting interventions or resources that are inconsistent with the LICBT clinical method threatens IAPT recovery rate targets.

Development of a criteria framework to evaluate written CBT self-help demonstrated moderate levels of agreement between raters. Although higher levels of agreement would enhance confidence, restricting the use of the adapted criteria framework to solely serve as the basis of recommendations for discussion by services could justify adoption (O’Neill, 2017). If adopted, the adapted criteria framework (Appendix 1 of the Supplementary material) could serve as the basis of a systematic process to review other interventions adopted by services that did not participate in this pilot or are under development. Given knowledge and competency guiding LICBT self-help interventions, the process should be led by practising PWPs, providing opportunities for national involvement in the IAPT programme. Implementing the process would address concerns regarding interventions currently adopted by IAPT services and direct services to written LICBT workbook-based self-help interventions appropriate for use within the IAPT programme (National Collaborating Centre for Mental Health, 2020).

Limitations

Given the number of services participating in this pilot study, findings may not be representative of IAPT services nationally. As such, caution should be exercised when considering concerns raised in this paper regarding the selection of interventions.

This paper is solely focused on written workbook based LICBT interventions that remain the format most adopted within IAPT services. However, other formats are available, and this may require criteria to be adapted.

Different definitions regarding written LICBT exist, therefore the application of criteria is directly relevant to the evaluation of written workbook based LICBT interventions based on a definition employed within the IAPT programme.

Although an Expert by Experience was fully involved in all stages developing and applying the criteria framework and recognised as an author, there was potential for greater Expert by Experience involvement.