A study of self-perceived change in CBT competence among Norwegian therapists undertaking a 2-year CBT training for working with young people

Abstract

The rising prevalence of mental illness among youth has underscored the critical need for proficient therapists trained in effective methods for treating prevalent mental disorders. As mental health services grapple with resource constraints, the demand for shorter training programs has increased. This study aimed to investigate whether therapist competence improves continuously over a two-year cognitive behavioural therapy (CBT) training course, whether specific CBT skills show greater improvement than generic skills, and whether self-perceived competence varies based on therapist characteristics. Data were gathered from seven different CBT training courses spanning the years 2013–2021 involving a total of 151 Norwegian therapists. Therapist self-assessed their CBT competence using the Cognitive Therapy and Adherence Scale (CTACS). The collected data were analysed using mixed effects models using the R program. The findings revealed a significant increase in self-reported competence throughout the two-year training period. Notably, skills related to case formulation and CBT techniques displayed the most substantial improvements, indicating a high level of competence attained by the conclusion of training. No significant interactions were identified between time and therapist characteristics. The outcomes of this study support the efficacy of in-depth training through an extended CBT course spanning multiple semesters. The observed progress in CBT skills suggests that investments in therapist training can yield high levels of competence. Furthermore, this investment appears warranted regardless of therapists’ individual characteristics.

Key learning aims

1. (1) To gain insight into the differential development of generic and CBT specific competence during CBT training.

2. (2) To reflect on the significance of therapist characteristics in the training process.

3. (3) To reflect on the possible importance of reaching a high level of CBT competence during CBT training.

Introduction

The escalating global prevalence of mental health disorders among youth, underscored by studies such as those conducted by Polanczyk et al. (2015) on a global scale and Bang (2018) in the Norwegian youth population, has heightened the urgency of equipping therapists with the proficiency to administer effective treatments for the most common mental disorders. Particularly, anxiety and depression have emerged as critical targets due to their pervasive impact on children and adolescents, as indicated by Barker et al. (2019). However, a disparity exists between the high incidence of these disorders and the limited application of evidence-based treatments, as observed in the work of Weisz et al. (2015) where few young people receive evidence-based treatments. In response, numerous initiatives have been launched to bridge this gap and ensure access to evidence-based interventions.

One such initiative, the Improving Access to Psychological Therapies (IAPT) program was initiated in the UK to enhance services for adults grappling with anxiety and depression, as noted by Wakefield et al. (2021). In 2011, this initiative was extended to encompass the youth population, aiming to augment the workforce and address gaps in the provision of evidence-based interventions, as highlighted by Shafran et al. (2014). Subsequently, practitioners have been trained in diverse settings, spanning from low-intensity school-based treatments to high-intensity interventions delivered in a hospital setting (Ludlow et al., 2020).

In parallel, the Norwegian government responded to the challenge by implementing a policy shift in 2015, mandating first-line services, staffed by various professions, to offer treatment for mild to moderate anxiety and depression (Nygaard and Kårikstad, 2015). Aligned with international and national guidelines, such as those set by the National Institute for Health & Clinical Excellence (NICE) (2019) and the Norwegian Audit Office (2020-2021) this directive highlights the need for evidence-based methods in mental health care. However, unlike the policy, no concerted effort was made to augment training in evidence-based methods.

Cognitive Behavioral Therapy (CBT) stands out as a promising evidence-based approach, as evidenced by numerous studies, including Weisz et al. (2017) and Fordham et al. (2021). Despite its potential, the integration of CBT and other evidence-based methods faces challenges such as time constraints for professional development. The Norwegian Audit Office (2020–2021) reveals that 70% of therapists report a lack of time for acquiring new therapeutic techniques. Furthermore, service providers grapple with constrained resources, prompting the quest for effective training methods, often leading to the preference for shorter training courses.

Within this intricate landscape, it becomes imperative to investigate whether competence evolution is a continuous process or whether significant changes resulting in high-quality competence in evidence-based methods manifest early or exclusively in the final stages. Moreover, an exploration of whether professionals with diverse backgrounds and characteristics acquire competence at a uniform rate during training adds an essential layer of understanding.

Past research has indicated that therapists advance during extended training courses such as weekly sessions for a year, or 20 hours per week for 30 weeks, exemplified by studies such as those by Barnfield et al. (2007), Bennett-Levy and Beedie (2007) and James et al. (2001). Studies have also unveiled variation across skills and individuals, emphasizing the individualized nature of competence acquisition (Bennett-Levy and Beedie, 2007). Experience and gender-related disparities were also noted by James et al. (2001) in a study including 20 participants as more experienced therapists were reported to be more competent and with males exhibiting a higher rate of improvement compared to females. Furthermore, Clark (2018) underscored the success of the IAPT program in training therapists from diverse professional backgrounds.

While it may be difficult finding a definition and agreed-upon measure for therapist competence (Kazantzis, 2003), Perepletchikova and Kazdin (2005) categorize therapeutic quality into therapist competence and adherence, an approach now commonly embraced. The former denotes the adept execution of interventions, while the latter signifies adherence to therapeutic approaches (Rapley and Loades, 2018).

In this study, we delve into the trajectory of competence change over a two-year CBT training course, aiming to unravel whether competence evolution is a continuous process or whether significant changes manifest early or exclusively in the final stages. We hypothesize that therapists will self-report an augmented competence in CBT, with improvement persisting throughout the two-year training period. Furthermore, we anticipate that specific CBT skills will undergo more pronounced enhancement compared with generic skills, with variations in self-perceived competence change linked to therapist characteristics, such as age, gender, experience, profession and caseload.

To unravel these intricacies, we gather therapists’ self-perceptions of their competence change during training. The Norwegian Centre for Research Data (61371) approved the study.

In the subsequent sections, we delve into the methods, results, and discussions that elucidate the dynamics of therapist competence acquisition within the context of a comprehensive two-year CBT training program.

Method

Design

This study employed a longitudinal design in a naturalistic setting to assess therapist competence using the Cognitive Therapy and Adherence Scale (CTACS) (see ‘Measures’ section below) at four time-points during the two-year CBT training course. Due to this design, the inclusion of a control group that did not receive training was not possible.

Recruitment and participants

In the period 2013–2021, data were collected four times from each of the seven different cohorts participating in the two-year CBT training course. ¹ Of the 170 participants starting the training course, two declined to participate in the study. Some participants quit before the course was completed (n=17) due to parental leave, job changes and health problems. Hence a total of n=151 therapists in training (hereafter therapists) participated in the study (82.1% women, mean age 41 years). Participants in the CBT training course per cohort were n ₁=22, n ₂=11, n ₃=27, n ₄=22, n ₅=29, n ₆=16 and n ₇=24.

To be eligible for the course, individuals must hold at least a bachelor’s degree in youth mental health and a minimum two years of relevant therapeutic work experience with youth prior to admission to the course. Participants are recruited from various professions and services, including first-line services, in- and out-patient clinics, and child welfare services. Educators and child welfare educators comprised the largest group (n=63, 41.7%), while school health nurses/nurses were n=32 (21.2%), doctors and psychologist were n=31 (20.5%), and social workers were n=17 (11.3%). Eight participants were placed in the ‘Other group’ (5.3%), which included professionals working in the school system. Mean experience reported was 12.9 years (ranging from 1 to 30 years) and the majority (n=70, 46.4%) reported having a small CBT caseload (between 1 and 4 cases), n=48 reported a medium CBT caseload (between 5 and 10 cases, 31.7 %), n=26 reported a large CBT caseload (17.2%), while n=7 (4.6%) failed to answer this question.

Course content and process of learning

The two-year CBT course is a comprehensive, well-established training program for CBT with children and adolescents, with consistent content across seven different cohorts. Previous qualitative interviews have indicated high levels of user satisfaction with the training (unpublished data). In-depth training in CBT is considered necessary, as recommended by the Norwegian Association for Cognitive Therapy. The course includes 240 teaching hours and 80 hours of group supervision which involves guided reading, writing short essays and clinical assessments, including group supervision of four patient videos per student led by a CBT supervisor with training in both CBT and CBT-supervision. The training course emphasizes experiential learning including skills training, role-play, and practical exercises along with theoretical lectures.

The specific content of the course is described in Appendix 1 (see Supplementary material). Themes and content are adapted to therapy with children and young people, and with a clear emphasis on the therapists using a developmentally sensitive approach.

Measures

To assess the participants’ self-perception of competence, a Norwegian version of the Cognitive Therapy Adherence and Competence scale (CTACS) with 25 items rated on a scale ranging from 0 (‘poor/low competence’) to 6 (‘excellent/high competence’). The CTACS contains four subscales (Structure, Alliance, Case-formulation, and CBT-techniques) that aggregate into a Total competence score. The 25 items are divided into five sections, see Appendix 2 (Supplementary material). The Norwegian version used in the current study is based on the original full version developed by Barber et al. (2003) but without the description guiding each item; see Appendix 3 (Supplementary material). Self-report is a cost-effective and straightforward method of measuring competence and is commonly used in routine practice settings (Muse and McManus, 2013). CTACS has acceptable levels of inter-rater reliability and criterion validity measuring adherence (α=0.92) and competence (α=0.93) (Barber et al., 2003).

The Norwegian translation of the CTACS was conducted by Nordahl et al. (2006) in agreement with the official authors. The scale is scored on a continuum ranging from 0 (poor) to 6 (excellent). While the scale can measure both adherence and competence, this study only collected competence scores. According to the psychometric evaluation of the Norwegian version of the CTACS by Lervik et al. (2021) there is no evidence for divergent validity between the adherence and competence scale. The study reported acceptable internal consistency for the full scale (α=0.91) and acceptable to low internal consistency for the subscales structure (α=0.76), therapeutic relationship (α=0.84), and CBT techniques (α=0.73) and conceptualization (α=0.61).

Internal consistency was calculated for all subscales at the four time-points; see Table 1. Internal consistency ranged from acceptable to excellent (α=0.75–0.92).

Table 1. Internal consistency of the CTACS* subscales

* CTACS, CBT Competence and Adherence scale (Barber et al., 2003)

Procedures and data collection

The therapists reported on competence four times (beginning of first (T1) and third (T3) semester, and the end of the second (T2) and fourth (T4) semester during the CBT course after having been explained the purpose of data collection and how to measure overall competence. All participating therapists signed informed consent and provided personal information. The therapists rated their competence based on an assessment of their average CBT competence across a range of cases, and not, as this measure usually is used, in connection with a specific case or session.

Statistical analysis

Mixed effects models give valid inference for missing at random in the dependent variables (CBT competence subscales) (Pinheiro and Bates, 2000). We included random effects for differences within and between participants and fixed effects for categorical time with adjustment for gender, age, profession, caseload, and experience. In addition, a model with interaction between time and the adjustment variables was estimated.

Separate models for the subscales of the dependent variable CTACS (Structure, Alliance, Case-formulation and CBT-techniques) were estimated.

Analysis used R (The R Foundation for Statistical Computing, Vienna, Austria) with the R package nlme (v3.1-152; Pinheiro et al., 2021) for mixed effects models.

Results

The results are shown in Tables 2 and 3, and in Fig. 1.

Table 2. Descriptives CTACS* subscales over four semesters

* CTACS, CBT Competence and Adherence scale (Barber et al., 2003). Range: 0 = poor to 6 = excellent.

Table 3. Model based estimates for development of competence on subscales of the CTACS*

* CTACS, CBT Competence and Adherence scale (Barber et al., 2003).

Figure 1. Change in competence on subscales of the CTACS over four semesters.

As can be seen in Table 2, Alliance was the subscale rated highest of the four subscales at T1 (mean score of 3.52 (SD=1.95), ending at 4.95 (SD=0.58) at T4. Use of Case-formulation and CBT-techniques were rated lowest of the subscales at T1, Case-formulation starting at 1.11 (SD=1.08), ending at 4.01 (SD=0.71) at T4, while CBT-techniques went from 1.24 (SD=1.22) to 4.07 (SD=0.77). The subscale Structure was rated from 2.35 (SD=0.98) to 4.50 (SD=0.72) from T1 to T4.

There was clear evidence of positive changes in self-reported competence over time for all subscales of the CTACS when adjusted for gender, age, profession, caseload and experience, p<0.001; see Table 2. Hence, as expected the therapists self-reported increase in competence during the course, although the magnitude of these changes varied between the subscales and the time-points. The largest change was for Case-formulation between T1 and T4, with Alliance showing the least change in the same time span.

Examining percentage change based on the mixed effects model without interaction, the largest percentage increase in mean competence from T1 to T2 was use of Case-formulation (129%), while CBT-techniques had an increase of 76%. The subscale Structure changed 51% from T1 to T2 while Alliance had the smallest change (24%) from T1 to T2. Examining the change from T1 to T4 the pattern still held with a change of 235%, 127%, 83% and 40%, respectively.

This percentage change was based on the group of female psychologists/physicians, with few patients (cases), 13 years’ experience, and age 40 as reference; see Fig. 1.

There was no evidence for interactions of time with gender, age, profession, caseload and experience; the lowest p-values for interactions were 0.224 for the subscale Structure, 0.240 for Alliance, 0.174 for Case-formulation and 0.139 for CBT-techniques.

Discussion

In the present study, the findings illuminate the trajectory of therapist competence development over a comprehensive two-year CBT training course. Through the examination of self-reported competence levels across different time-points, we found a steady improvement in competence during the two-year training course. We also found a significant change in competence of all skills from the third until the end of fourth semester, indicating continued learning throughout the whole course, although different patterns emerged in the generic versus the CBT-specific skills. This confirmed our expectations that specific CBT skills would improve more than generic relational skills, which is also expected as the focus in the training was on CBT-specific skills. We found no significant interaction between time and adjustment variables in self-perceived competence based on therapist characteristics, such as gender, experience, profession and caseload. Hence, our expectation was not supported.

The observed significant increase in self-reported competence over the two-year training course aligns with the study’s expectations and is consistent with previous research (Keen and Freeston, 2008; McManus et al., 2010). Notably, the data indicated a significant improvement in competence in all skills from the third semester onwards, suggesting a continuous learning curve throughout the entire training period. This reinforces the concept that the process of acquiring and refining competence extends beyond the initial stages, reflecting ongoing growth and adaptation.

An intriguing pattern emerged in the differentiation of skill-specific competence improvements. The Alliance skill, which started with the highest rating, exhibited the smallest increase in competence. This finding aligns with previous research by Bennett-Levy and Beedie (2007) underscoring the limited growth observed in interpersonal skills over time. Although Alliance demonstrated the smallest relative improvement, the change remained significant, echoing findings reported by Milne et al. (1999). The high initial rating in Alliance, possibly due to therapists’ previous experience (mean experience of 13 years), could have contributed to the more modest growth. Alliance would therefore most probably be a generic skill they may have learnt earlier in their career, having less room for improvement. Weck et al. (2015) have emphasized the role of therapeutic alliance as a pre-condition for competent implementation of cognitive behavioural therapy. Similarly, Haug et al. (2016) reported lower alliance rating later in therapy was associated with client drop-out. Our findings with high rating of alliance early in training may therefore be an important factor to prevent young people in therapy from dropping out of CBT treatment.

Conversely, Case-formulation and CBT-techniques displayed the most substantial improvement, validating the study’s hypothesis that specific CBT skills would exhibit greater enhancement compared with generic skills. While Structure reached an acceptable competence level even before the second year of training, CBT-techniques and Case conceptualization first reached this level during the last part of the second year of training. The delayed attainment of high competence levels in Case-formulation and CBT-techniques highlights their complexity and the necessity of mastering these skills to provide adaptable, tailored CBT interventions. Too strict a reliance on structure, may result in a rigid therapist not meeting the needs of the client, and the skills acquired during the latter part of the second year is therefore of utmost importance to provide flexible high quality CBT therapy.

Therefore, although brief CBT training (e.g. a 2-day workshop followed by 1 year of telephone consultations) has been reported to improve competence with lasting effects (Simons et al., 2010), the current study highlights the importance of providing longer training courses to achieve high competency levels. The training programs of the Norwegian Association of Cognitive Therapy emphasize that to successfully complete a CBT course, competence in the different skills should reach a score of 3 or above on at least 50% of the CTACS subscales (Norsk forening for kognitiv terapi, 2013). While more generic skills such as Structure and Alliance reached this level after 1 year, the competence continued to increase to level 4 and 5 in the second year of training and for CBT-specific skills the second year was important to reach such a level. In order to maximize the benefits and value of investing in therapists learning an evidence-based method, it is essential for organizations to recognize the significance of in-depth training where participants can reach a level of competence in CBT-specific skills that indicate mastery during the training period.

Contrary to initial expectations and existing literature, the study’s findings did not indicate any interaction between change in self-perceived competence and therapist characteristics. In contrast to studies that suggested differences in profession, gender, age and caseload might influence competence development (James et al., 2001; Liness et al., 2019; McManus et al., 2010), this study’s results suggest a different narrative.

According to Tyler et al. (2021) there has been little research on the impact of profession on the training and quality of therapy. They did, however, conclude that treatment with trained psychologists was more effective due to their competence and subject-specific knowledge (Tyler et al., 2021). Our results may, on the other hand, support that there is no obstacle mixing professions in a CBT training course as there was no significant difference in competence gains based on professional background. This may furthermore have important implications for the recruitment of CBT therapists. If different professions acquire CBT skills at the same rate it may be possible to make this required competence available for more youths, an important target for health authorities. It also appears that neither gender nor age influences the development of CBT competence, which is contrary to previous studies (James et al., 2001). However, it is important to exercise caution when generalizing the finding of James et al. (2001) due to their small sample size including only nine males.

Furthermore, experience, age and caseload were not significantly related to change in CBT-competence in the current study. The lack of significant relationship between age and competence was in accordance with the results published by James et al. (2001), but contrary to McManus et al. (2010) who found that older trainees performed worse. Furthermore, the absence of a significant relationship between caseload and competence contradicted the study’s initial expectations. The assumption that practising skills with actual patients would foster better competence development was challenged. Group activities, supervision, sharing clinical experience through video recordings of sessions. and role-playing during the training might have provided many opportunities for skill practice, compensating for the absence of direct patient encounters. The therapists were also motivated to learn, joining a two-year training course which may explain that both older and younger therapists improved. It could also be possible that the use of self-report may have influenced the results with inexperienced therapists over-rating their performance, and experienced therapists under-rating their performance. A study showed that whilst there was a significant correlation between self-ratings and expert-ratings of competence, therapists significantly over-rated their competence relative to the expert rater, and with the less competent therapists over-rating their competence to a greater degree than therapists who met criteria for competence (Brosan et al., 2008).

Strengths and limitations

The study’s strengths lie in its relatively large sample size, systematic data collection, and consistent evaluation procedure at multiple time-points. These factors provide support to the therapists being able to evaluate their own change in competence. Additionally, the CBT course maintained a stable content and structure throughout the years of data collection, ensuring therapists had equal opportunities to enhance their competence.

Although the use of self-report measures is a limitation in this study, it is supported by research suggesting that it encourages individuals to reflect on their experiences (Bennett-Levy and Beedie, 2007; Wampold et al., 2017). However, the literature also highlights that individuals tend to rate their own competence higher than a third-party observer (Dunning et al., 2004). As the current study relies solely on self-evaluation, this bias must be acknowledged. Nonetheless, as the focus of the study is on changes over time, this general bias is unlikely to significantly impact the results.

One additional limitation of this study is the lack of power calculation prior to data collection. Performing a power calculation prior to data collection is best practice, as calculating power after data collection is not recommended (Lydersen, 2019). On the other hand, the use of sound statistical methods to analyse the data is a strength of the study,

Nevertheless, the study’s naturalistic design without a control group restricts the ability to definitively attribute changes to training alone. Furthermore, caution should be exercised when applying these findings to other settings or training programs due to the study’s limited generalizability to the specific training course that was studied.

Conclusion

In conclusion, this study provides valuable insights into the development of therapist competence within the context of a comprehensive two-year CBT training course. The observed continuous competence enhancement highlights the significance of prolonged training for achieving high levels of proficiency. The differentiation in skill-specific improvements underscores the importance of mastering complex CBT-specific methods. Therapist characteristics such as gender, age, profession and caseload did not significantly impact competence gains, indicating the feasibility and effectiveness of cross-profession training. Ultimately, these findings contribute to the ongoing efforts to equip therapists with the skills necessary to effectively address the rising mental health challenges among youth.

Future research should explore the notion of competence from a variety of perspectives such as coding video sessions of therapy or assessing client outcome, in addition to therapist self-report. Conducting long-term follow-up to investigate whether the training produces lasting improvements in competence would also be valuable.