Outcome measurement in child and adolescent mental health services has recently seen an upsurge in interest. In 1998 the Department of Health launched its approved outcome measure in mental health, the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA). Field trials suggested that HoNOSCA had satisfactory validity, interrater reliability and levels of acceptability in multi-disciplinary practice (Gowers et al, Reference Gowers, Harrington and Whitton1999a ,Reference Gowers, Harrington and Whitton b ). These findings have been replicated in Britain (Reference Yates, Garralda and HigginsonYates et al, 1999; Reference Garralda, Yates and HigginsonGarralda et al, 2000) and in Australia (Reference Brann, Coleman and LukBrann et al, 2001). The use of HoNOSCA in the Audit Commission's review of child and adolescent mental health services (Audit Commission, 1999) and the establishment of the HoNOSCA training base have led to its routine use in many mental health services. One criticism, however, is that HoNOSCA fails to incorporate a patient perspective. The current project aimed to introduce an adolescent-rated version of HoNOSCA, assess its properties, and explore its performance against other measures of adolescent difficulty.
METHOD
In May 1997, following training in the use of HoNOSCA, the multi-disciplinary team of the Mersey Regional Adolescent Mental Health Service began incorporating this measure into their routine clinical work. All adolescents admitted to the in-patient unit had a clinician-rated HoNOSCA assessment on admission, reviewed at 6-weekly intervals and at discharge. Those attending a regional out-patient eating disorders service were also rated at presentation and at 6-weekly intervals. In parallel with the clinician assessments, a self-rated version of the scale (HoNOSCA—SR; see Appendix) was administered as a guided questionnaire by an assistant psychologist not involved in the patient's care. The self-rated version was based on the items in the clinician-rated version rephrased as questions. The glossary used for HoNOSCA was judged to be inappropriate for adolescents, given the complexity of the language and the time it would take to master. The phrasing of the questions was kept as close as possible to the original HoNOSCA items in order to maximise the possibility of agreement between the two versions. Each question was read out and explained to the adolescent. Any problems reported were explored in terms of severity or frequency in order to help the patient assign a score (from 0=not at all, to 4=severely) before moving on to the next item.
The reliability of HoNOSCA—SR was considered in two ways. Split-half reliability comprised a correlation between scores on odd and even numbered items for a series of 65 patients. Test—retest reliability was rated for 24 in-patients tested on two occasions 1 week apart. Validity was tested by correlation with scores on self-reported and clinician-rated measures of psychopathology and general functioning, namely the Strengths and Difficulties Questionnaire (SDQ; Reference GoodmanGoodman, 1997) and, for the eating disorder group, the Morgan—Russell schedule (Reference Morgan and HaywardMorgan & Hayward, 1988) and the Eating Disorders Inventory (EDI—2; Reference GarnerGarner, 1991).
Correlations between clinician— and user-rated versions enabled assessment of agreement between raters. The complete data were analysed using SPSS (SPSS, 1998).
RESULTS
Recruitment
Over the period of study there were 96 admissions to the in-patient unit, of which 41 were emergencies and 55 were elective. The patients admitted had a mean age of 15.05 (range 13-18). There were 69 girls and 27 boys. The eating disorders service saw 54 new referrals for out-patient treatment for anorexia nervosa. All these patients were female, with a similar mean age (15.6 years). Clinician-rated HoNOSCAs were completed for all in-patients and out-patients at all rating points, comprising 361 scoring episodes.
The self-rated version of HoNOSCA was completed by all adolescents at admission and discharge from the in-patient unit, although two failed to record mid-treatment review ratings. All out-patients completed self-ratings at each time point. There were therefore 359 self-ratings, representing a completion rate of 99.4%.
The completion of the self-rated version took 10-15 minutes for the first rating and about 5 minutes for subsequent ratings.
Reliability of HoNOSCA—SR
Split-half reliability showed a correlation between the odd— and even-numbered scores of 0.73 based on a series of 65. Test—retest reliability of HoNOSCA—SR was explored in a subset of 24 patients to whom the instrument was administered on two occasions, a week apart. As these were in-patients with moderately severe disorder, it was assumed that there was unlikely to have been much change in severity of difficulties between the two rating points. The intraclass correlation coefficients varied from 0.32 for item 1 (antisocial and aggressive behaviour) to 0.88 for item 2 (attention and concentration). The total score (Pearson's) correlation between time 1 and time 2 was 0.806, P < 0.001 (Table 1).
Table 1 Test—retest reliability for the self-rated version of HoNOSCA (n=24)
| Scale item number | Kappa correlation coefficient |
|---|---|
| 1 | 0.318 |
| 2 | 0.882 |
| 3 | 0.565 |
| 4 | 0.357 |
| 5 | 0.636 |
| 6 | 0.731 |
| 7 | 0.412 |
| 8 | 0.739 |
| 9 | 0.357 |
| 10 | 0.565 |
| 11 | 0.560 |
| 12 | 0.578 |
| 13 | 0.399 |
Validity of HoNOSCA—SR
The validity of HoNOSCA—SR was explored by correlating it with the SDQ and (for the eating disorder series) the modified Morgan—Russell severity schedule, which provides a global severity measure covering physical, psychological and social adjustment (Reference Morgan and HaywardMorgan & Hayward, 1988). This measure has been widely used as a severity measure in anorexia nervosa (Reference Gowers, Weetman and ShoreGowers et al, 2000). The HoNOSCA—SR gave a correlation of 0.66 (P < 0.001) with the adolescent-rated SDQ (n=39 cases); this compares with a correlation of 0.397 between the clinician-rated version of HoNOSCA and the SDQ (Reference Yates, Garralda and HigginsonYates et al, 1999). For the patients with eating disorders, HoNOSCA—SR correlated moderately well with the total (self-report) EDI—2 score (r=0.629, n=53, P < 0.001), but only weakly (and inversely, given the direction of scoring) with the clinician-rated Morgan—Russell total score (r=-0.32, n=54,P=0.018).
Patient view of clinical severity and outcome
At admission to the in-patient unit, or at presentation to the out-patient service, the adolescent patients recorded comparable levels of difficulty to the clinician ratings. For the in-patients (n=96) the mean total admission score was 17.8, compared with the mean clinician rating of 19.6. For the out-patients the mean scores were 14.9 and 18.8, respectively. There were, however, notable differences between scale items (Fig. 1), with clinicians tending to give higher ratings to difficulties with peer relationships and school attendance, while the young people themselves drew attention to problems with attention and concentration (question 2) and scholastic and language skills (question 5).
Fig. 1 Comparison of clinician-rated and self-reported scores on the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) at admission and discharge for 96 adolescent in-patients. This scale scores for (1) aggression, (2) concentration, (3) self-injury, (4) substance misuse, (5) language, (6) physical illness, (7) hallucinations, (8) somatic disturbance, (9) emotion, (10) peer relationships, (11) independence, (12) family relationships and (13) school attendance; see Appendix for full details.
During the course of treatment both the clinician-rated and self-rated versions of HoNOSCA demonstrated sensitivity to change. The in-patients reported a significant change over time, but slightly less than was reported by the staff (mean admission and discharge scores for 57 patients admitted for longer than 6 weeks were 18.44 and 12.82 self-rated, compared with clinician ratings of 18.35 and 9.75). The out-patients with eating disorders showed changes comparable with the clinician ratings. The changes in ratings over time for a consecutive smaller series from both services who were treated for a minimum of 4 months is shown in Table 2.
Table 2 Mean clinician-rated and self-report HoNOSCA scores over time for in-patient and out-patient cases for all patients in treatment for 18 weeks
| In-patient (n=18) | Out-patient (n=17) | |||
|---|---|---|---|---|
| Self-report | Clinician | Self-report | Clinician | |
| First assessment | 16.22 | 20.17 | 15.00 | 18.00 |
| Week 6 | 12.11 | 14.89 | 13.65 | 13.71 |
| Week 12 | 10.72 | 11.94 | 10.53 | 11.06 |
| Discharge/week 18 | 9.56 | 10.11 | 8.88 | 9.76 |
Considering each scale item in turn, both raters reported improvement across the board, but young people were more positive about improvements in family relationships (item 12) and clinicians about behavioural items (items 1-3) (see Fig. 1).
Agreement between clinician and patient ratings
In individual cases agreement between clinician and patient ratings was poor, with a correlation (Pearson's r) between HoNOSCA and HoNOSCA—SR of 0.27 at presentation, rising to 0.58 at discharge. There were notable differences by diagnosis, for the in-patients. Those with psychoses tended to see themselves as having fewer problems than the clinician's assessment, while those with emotional (chiefly affective) disorders rated themselves as having more. A subgroup of patients with incipient borderline personality disorder with poor peer relationships and self-harm were the highest scorers on HoNOSCA—SR (mean admission score 24.6). The out-patient eating-disorder group showed much higher levels of agreement, with a correlation of 0.58 at presentation and quite good agreement (r=0.72) at discharge.
DISCUSSION
Strengths and limitations of HoNOSCA
Clinicians, patients and commissioners all require comprehensive outcome data in order to evaluate services and individual treatment response. The HoNOSCA assessment has become well established as a clinician-rated general outcome measure (Reference Garralda, Yates and HigginsonGarralda et al, 2000), balancing simplicity with a reasonable level of reliability and validity (Reference Brann, Coleman and LukBrann et al, 2001). It has been shown to correlate highly with the Children's Global Assessment Scale (Reference Shaffer, Gould and BrasicShaffer et al, 1983) and to complement the Paddington Complexity Scale (PCS) in terms of its coverage of mental health problems in children and adolescents (Reference Yates, Garralda and HigginsonYates et al, 1999).
A survey of 120 child and adolescent mental health services who had received training in the use of HoNOSCA suggested that half were using it 6 months later (further details available from the author upon request). Not all have been persuaded that the enthusiasm for HoNOSCA is fully justified, however (Reference JaffaJaffa, 2000), and other measures have since been developed. Alongside the PCS, the Strengths and Difficulties Questionnaire (Reference GoodmanGoodman, 1997) is a notable addition to the field. Incorporating both parent and child versions, the SDQ raises the question of who is the most relevant rater of a child's difficulties: clinician or child? One of the major reservations about HoNOSCA is that there is no patient perspective, and that clinicians might inevitably see progress when their interventions are being assessed.
Adolescent views of severity
Despite theoretical difficulties of insight and sometimes poor engagement between adolescents and mental health services, this study shows that adolescents are aware of their difficulties in a range of domains and will freely admit to them. They often do not see difficulties in the same areas as their treating clinicians, however. It is noteworthy that adolescents themselves rate the subjective items (such as difficulty with concentration and attention) as more problematic than staff do. This discrepancy highlights the importance of therapeutic engagement in identifying difficulties and hence common treatment aims.
Adolescent views on outcome
Despite the above reservations, in the two services studied adolescents reported significant improvements with treatment, which spanned the duration of intervention. This gave some support to the average length of treatment of 3-4 months. In the out-patient service, this view matched the progress seen by staff.
Agreement between clinician and patient ratings
The correlation between HoNOSCA and HoNOSCA—SR is weak, particularly for in-patients. These patients have been rated at the point of admission, and their attitude to admission may be reflected in the scores. Approximately one-third was suffering from an acute psychotic disorder, with predictable levels of suspicion and lack of insight. The greater level of agreement between clinicians and patients in the out-patient service requires explanation. This could be related to the diagnostic homogeneity of the out-patient group, but people with eating disorders are not noted for acknowledging difficulties. A more likely explanation is that in an out-patient service clinicians are more reliant on patient and informant report in completing ratings, whereas in-patient staff can observe rather than take the patient's word for performance in such areas as peer relationships. It may therefore be that despite the lower levels of agreement, the clinician ratings for the in-patients are more objective.
Therapeutic use of HoNOSCA—SR
The two services have used the clinician— and self-rated scores to monitor progress through treatment with the patients themselves. In the out-patient eating disorder service particularly, where patients are encouraged to change behaviour, the scores provide regular feedback in addition to physical measures and the scores on specific eating disorder self-report questionnaires. Thus, patients with anorexia nervosa might be encouraged to gain weight and judge for themselves whether this results in an improvement or deterioration in general functioning by their own account. Progress in the first 6 weeks can then be used to boost motivation for the next phase of treatment. The possibilities for the therapeutic use of HoNOSCA to provide feedback to patients look promising.
Clinical Implications and Limitations
CLINICAL IMPLICATIONS
-
• The self-rated version of the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) offers a useful perspective on adolescent mental health outcomes to support the clinically rated version.
-
• Adolescents with mental health problems report similar overall levels of difficulty to those noted by clinicians, but are often concerned about different areas of functioning.
-
• A combination of self-report and clinician-rated outcome measures can be used therapeutically to monitor progress and enhance motivation.
LIMITATIONS
-
• The self-rated HoNOSCA should be completed with the help of a member of staff who is not directly involved in treatment, to maximise patient understanding and quality of scoring.
-
• Correlations between clinician-rated and self-reported HoNOSCA total scores are poor.
-
• The self-reported HoNOSCA, like the clinician-rated version, may not address the mental health problems of all adolescents presenting to child and adolescent mental health services.
APPENDIX
Self-rated version of the Health of the Nation Outcome Scales for Children and Adolescents
The patient is asked to assess the following items, selecting answers from the categories ‘not at all’, ‘insignificantly’, ‘mild but definitely’, ‘moderately’ or ‘severely’.
In the last two weeks
-
1. Have you been troubled by your disruptive behaviour, physical or verbal aggression?
-
2. Have you suffered from lack of concentration or restlessness?
-
3. Have you done anything to injure or harm yourself on purpose?
-
4. Have you had problems as a result of your use of alcohol, drugs or solvents?
-
5. Have you experienced difficulties keeping up with your usual educational attainments and abilities?
-
6. Has any physical illness or disability restricted your activities?
-
7. Have you been troubled by hearing voices, seeing things, suspicious or abnormal thoughts?
-
8. Have you suffered from self-induced vomiting, head/stomachaches/bedwetting or soiling, or other symptoms with no physical cause?
-
9. Have you been feeling in a low or anxious mood, or troubled by fears, obsessions or rituals?
-
10. Have you been troubled by a lack of satisfactory friendships or bullying?
-
11. Have you found it difficult to look after yourself or take responsibility for your independence?
-
12. Have you been troubled by relationships in your family or substitute home?
-
13. Have you stopped attending school/college/education sessions?
eLetters
No eLetters have been published for this article.