Development of a training programme in individual case formulation skills and a scale for assessing its effectiveness

Abstract

Background:

Little is known about the skills involved in clinical formulation. The individual case formulation (ICF) approach, based on functional analysis, employs clinical descriptions that are theory-free and depicts formulations constructed according to a set of basic conventions.

Aims:

We report a test of whether this method could be taught and if the quality of the resulting diagrams could be reliably rated.

Method:

Participants (n=40) participated in a training course in formulation. A draft rating scale was refined in the course of rating formulation diagrams and basic inter-rater reliability established.

Results:

Results of the study support further development of the ICF approach.

Introduction

Our research addresses the problem of teaching novice therapists to formulate a client’s presenting problems. Although disorder-specific models provide a guide, and manuals specify steps in therapy, it is still necessary to translate interview responses into psychologically meaningful conceptual units, particularly when a client’s problem does not conform to an available disorder-specific model. One indication that the training of therapists needs to be improved is the relatively low success rates achieved for post-traumatic stress disorder (PTSD) by Improving Access to Psychological Therapies (IAPT) services compared with outcomes of research studies (Murray, 2017), whereas this is less true for other problems. This study set out to teach conventions for classifying observations that are, as far as possible, theory-free, and to distinguish them from the therapist’s chosen hypotheses they use to explain the observations and their functional relationships. It is hoped that such a more individualised approach will encourage generalisable skills and improve clients’ outcomes. A case formulation is understood to be a unique evolving process. Competing hypotheses will prompt different therapy decisions, the consequences of which can be evaluated through a variety of methods to obtain feedback on progress.

Method

Development of a scale to assess formulation skills

The conventions for representing a case formulation symbolically in the form of a visual diagram are set out in the Supplementary material. Our study is an attempt to develop a scale for assessing a capacity to formulate in this manner. It focuses on nine dimensions of skill, as follows: (1) comprehensiveness of information pertaining to the presenting problem; (2) the relevance of information depicted in the diagram; (3) the reliable sourcing and accuracy of observations; (4) the extent to which inferences (hypotheses) are evidence-based and correctly interpreted; (5) a clear depiction of relationship and causal direction between observations (as antecedents, consequents, inhibitory, excitatory, etc.); (6) the provision of causal explanations for relationships (i.e. hypotheses); (7) the identification of contextual moderators of causal relationships; (8) the whole diagram as a coherent synthesis; and (9) the diagram prioritises aspects of the problem for intervention.

The 9-item scale (scoring range 0–18) was developed in earlier studies (Griffiths, 2017; Pettman, 2017; Turner, 2022) and was shown to have high internal consistency (alpha = 0.91) and moderate inter-rater reliability (0.57) when data were rated by skilled independent assessors. These are complex skills, and so we are attempting to refine the scale through further research. In this study, small modifications to the scale were agreed by the raters when working on the first four training diagrams (but not when the final test diagram was rated). Due to the guided nature of the training, the final item of the scale, synthesis and prioritisation, was not used to assess skills.

Training procedure

The training material consisted of written vignettes based on actual clients diagnosed with PTSD in a specialised NHS service. PTSD was selected for training purposes because a CBT model of this problem is widely known and informs practice (Ehlers and Clark, 2000). It was a convenient vehicle for training participants to select and encode relevant information, not an attempt to teach this particular theoretical model for formulating PTSD. As with any disorder-specific model, idiosyncratic observations have to be collected and fitted to the general categories the model employs. The participants in this study were relatively naive to formulation because the majority were beginning their training as clinical psychologists (n=22). Due to COVID restrictions, the workshop was carried out online.

Participants completed two tasks in which a detailed written description of a client who was presenting with disturbing post-trauma experiences and behaviour was the basis for drawing up the formulation diagram. The first task was used for training; the second (a new case vignette) was for the evaluation of acquired skills.

The initial training tasks consisted of four steps, each followed by a test. The steps were: (1) accurate description and distinguishing observations from potential explanations (i.e. hypotheses); (2) identifying functional equivalences between observations (for both cues and behaviours); (3) recognising and representing mediators of behaviour, whether cognitive or otherwise; and (4) identifying and labelling observations understood to be acting as moderators of functional relationships that were already depicted in the participant’s diagram.

Following teaching and illustrative examples for each step, participants had to translate the formulation concept into a diagram with appropriately labelled symbol conventions (see Supplementary material for details). They were then provided with a hyperlink to a diagrammatic formulation that used our chosen conventions, and they were encouraged to compare this version with their own. (The focus here was on correct use of descriptive conventions, not to mimic an ‘ideal formulation’ of the client devised by the authors.) Participants’ diagrams could vary considerably without being penalised when rated for quality. The training procedure was intended to assist participants to independently identify and learn from their errors, mimicking a process that might otherwise take place in supervision. On completion of the training steps, a verbal description of a second test case vignette was presented, and each participant had to construct a formulation diagram using the conventions they had learned. Participants then uploaded photographs or screenshots of their diagrams and these were later independently rated by G.B. and R.H. on the items of a 9-item dimensional scale constructed for this purpose (see above and Supplementary material online for details).

At this stage of development of the scale, only percentage agreement between raters was calculated. Overall agreement (within 4 points) on summed scores was 72.7%, and ranged between 68 and 100% for individual items.

Results

Any diagram which scored zero on two of the first three dimensions for rating was deemed unrateable overall and removed from the analysis. It was assumed that an inability to grasp these initial tools would render meaningless any diagram produced later, and so for the final test diagram, 17 participants were excluded (remaining n=22).

No participant was excluded from the training phase and scores were summed for all four steps (n=39). The mean score was 6.3 (SD=2.4, range 0–11, maximum possible score 14). There was a high correlation between step 1 and overall score (r=0.84, p<.01), suggesting that initial skill strongly determined final learning. The mean score for the final test phase was 10.21 (SD=1.87, range 5.5–10.5, maximum score 17). Cronbach’s alpha for the total scale was .80 based on 2000 bootstrapped samples.

Discussion

Although agreement between independent raters was not as high as we had hoped, it was sufficient to justify our belief that the scale can be developed further and can be used in studies to assess its external validity; for instance, by showing a relationship between formulation skills and therapy outcomes. Some participants produced diagrams with close to ceiling scores, demonstrating that the conventions can be understood and taught. However, we may have under-estimated the amount of teaching required.

Regarding the difficulty that some participants experienced, it is possible that they could understand the conceptual distinction between an observation and its interpretation, but could not put this knowledge into practice. Perhaps they were using cognitive heuristics, such as the availability bias, making it difficult for them to depart from, say, psychiatric or folk terms. Alternatively, prior academic learning might have interfered with an individual case formulation approach which focuses on idiographic information. If these difficulties are common amongst new trainees, it is likely to raise problems in clinical supervision. Highly skilled practitioners might have performed very differently on our tasks. It would be helpful in future studies to obtain data on self-reported confusion or attitudes towards the task. This should reveal the reason for difficulty in representing basic client information, and applying theory to understand causal relationships.

For instance, it was striking that a high proportion of participants were unable to draw a diagram that described observations in a sufficiently neutral and precise manner, free from an imposed interpretation. This was despite being provided with a straightforward verbal description of the client’s problems in everyday language. It is likely that an even greater level of failure could be expected with a verbatim transcript (or recording) of an actual assessment interview.

Just as the formulation of a problem has aspects unique to the individual, so does the choice of appropriate goals. Although a reduction in ‘psychopathology’, assessed by standardised measures, is likely to remain an important goal of therapy, a more nuanced and individualised approach to goal-setting is also needed. Another stage in our research will be to evaluate a therapist’s ability to use their formulation to devise an intervention strategy in the light of goals agreed between client and therapist.