Reliability and validity of the CANDID – a needs assessment instrument for adults with learning disabilities and mental health problems

Abstract

Background

People with learning disabilities and mental health problems have complex needs. Care should be provided according to need.

Aim

To develop a standardised needs-assessment instrument for adults with learning disabilities and mental health problems.

Method

The Camberwell Assessment of Need for Adults with Developmental and Intellectual Disabilities (CANDID) was developed by modifying the Camberwell Assessment of Need (CAN). Concurrent validity was tested using the Global Assessment of Functioning (GAF) and the Disability Assessment Schedule (DAS). Test–retest and interrater reliability were investigated using 40 adults with learning disabilities and mental health problems.

Results

CANDID scores were significantly correlated with both DAS (P<0.05) and GAF scores (P<0.01). Correlation coefficients for interrater reliability were 0.93 (user), 0.90 (carer), and 0.97 (staff ratings); for test–retest reliability they were 0.71, 0.69 and 0.86 respectively. Mean interview duration was less than 30 minutes.

Conclusions

The CANDID is a brief, valid and reliable needs assessment instrument for adults with learning disabilities and mental health problems.

Following the closure of mental handicap hospitals, most adults with learning disabilities receive care in various community settings at a substantial cost. The National Health Service (NHS) and local authorities spend approximately £1 000 000 000 on services for people with learning disabilities (Audit Commission, 1992). The NHS and Community Care Act (House of Commons, 1990) established a statutory duty on social services to assess the needs of people who may require community care. The Camberwell Assessment of Need (CAN; Phelan et al, 1995) is an established needs-assessment instrument for those with severe mental illness. Although people with learning disabilities are more likely to develop mental health problems than their non-disabled counterparts (Reiss, 1994), there is no widely accepted instrument for measuring needs in this group. This study aimed to develop the Camberwell Assessment of Need for Adults with Developmental and Intellectual Disabilities (CANDID) and investigate its validity and reliability.

METHOD

Development of the CANDID

The CANDID was developed by modifying the CAN to make its content relevant to adults with learning disabilities and mental health problems while retaining its format and structure. The CANDID shares with the CAN the criteria set prior to the latter's development, namely that it should:

1. (a) have acceptable validity and reliability;

2. (b) be brief and suitable for use by a range of professionals;

3. (c) require no formal training;

4. (d) record separately the views of the service users, their informal carers and staff;

5. (e) measure met and unmet need;

6. (f) assess the help provided by informal carers and local services;

7. (g) be suitable for clinical and research use.

The questions in each of the 25 need areas of the CANDID are divided into four sections: Section 1 assesses the absence or presence of need, and, if present, whether it is met or unmet; Section 2 rates the help received from informal carers; Section 3 asks how much help by local services is provided (3a) and needed (3b); Section 4 inquires about the respondent's satisfaction with the type (4a) and amount (4b) of help received from local services.

First draft

Focus groups of service users, informal carers and staff identified areas of needs relevant to people with learning disabilities and mental health problems. The users group (n=8) consisted of adults with mild or moderate learning disabilities who were attending a day centre or were living in local residential facilities. The carers group (n=7) consisted of informal carers of people using the above facilities. The staff group (n=9) consisted of staff from a variety of disciplines working with people with learning disabilities and mental health problems. The first draft of the instrument was developed using findings from these focus groups.

Second draft

The first draft was commented on by health and social services professionals (n=24) with expertise in working with adults with learning disabilities and mental health problems. These consultations were conducted on an individual basis and focused on the content and structure of the instrument and its usefulness in research and clinical settings. Case vignettes were used, taking into account the whole range of the target population. As a result of these consultations the second draft was developed.

Validity studies

Content validity

We designed a questionnaire that asked about the views of service users (n=45) and their informal carers on the list of need areas identified through the process described above. Users and carers were asked to score each need area item according to its relevance, and to suggest any additional items that should have been included. Adults with all levels of learning disabilities were included in this sample. For those with a level of learning disability severe enough to interfere significantly with the comprehension of the questionnaire, it was completed by carers alone.

Consensual validity

Fifty-five experts in the field of mental health in people with learning disabilities from a range of professional backgrounds and all parts of the UK were surveyed. Their opinion was sought on the content, language and structure of the CANDID by mailing them a copy of the second draft accompanied by a questionnaire inviting them to rate, on a five-point Likert scale, ‘helpfulness of anchor points’, ‘ease of use’ and ‘appropriateness of language’.

Criterion validity

No ‘gold standard’ needs-assessment instrument currently exists for people with learning disabilities and mental health problems. In order to establish the concurrent validity of the CANDID, two instruments were used: the Disability Assessment Schedule (DAS; Holmes et al, 1982); and the Global Assessment of Functioning (GAF; American Psychiatric Association, 1994). The DAS was developed in order to assess level of functioning in 12 life areas of people with learning disabilities; the GAF measures global level of psychiatric symptom severity and disability. Concurrent validity was calculated in two ways: first CANDID summary scores (total number of needs) rated by staff were compared with total DAS and GAF scores; second, a comparison was made between DAS scores in individual areas of need (behaviour, communication, mobility, social interaction and self-care) and corresponding areas of the CANDID. These areas were selected because they were the overlapping areas in the two instruments that could be meaningfully compared.

Predictive validity is relative to a needs-assessment instrument because of its capacity to predict future service utilisation and therefore assist with needs-led service planning. However, no attempt was made to establish the predictive validity of the CANDID because this would require a longitudinal study design, which was beyond the scope of this study.

Reliability studies

Sample acquisition

Two sampling frames were used: first, all adults (n=210) using a community-based specialist learning disabilities mental health service in an outer London borough (Bromley); second, all in-patients (n=12) of a national unit for adults with mild or moderate learning disabilities at a psychiatric hospital (Bethlem Royal Hospital). The community subsample (n=31), although not randomly selected, included people with a range of levels of intellectual ability and a variety of mental health and behavioural problems characteristic of users of specialist learning disability mental health services. The in-patient subsample comprised nine people, after three patients were judged by their consultant psychiatrist to be too disturbed to participate. Thus, 40 people in total were recruited for the reliability study. Only one of those approached refused to participate. An estimate based on the original CAN data had indicated that for interrater reliability a sample of this size would be adequate to estimate an intraclass correlation of 0.88 to within ±0.1 with approximately 95% confidence.

Reliability study design

Five interviewer/raters were used: a psychiatrist, an occupational therapist, a social worker and two nurses. A brief explanation of the scope of the instrument and the rating procedure, but no formal training, was given.

Forty subject trios, each consisting of a service user, their informal carer and a member of staff, were enrolled in the reliability study. Of these, nine users could not be interviewed owing to the severity of their learning disabilities, and 13 carers were unavailable. Hence for the investigation of interrater reliability, 31 users, 27 carers and 40 staff were interviewed at a given point of time (T₁ ). All interviews performed at T₁ were timed. With 29 of the 40 triplets the interviews were conducted ‘live’ by an interviewer in the presence of a silent second rater (all five raters rotated in their role as interviewer or second rater). The remaining 11 trios were interviewed by one interviewer alone (the same interviewer conducting all interviews), and the interviews were audio-taped. All four second raters rated the taped material at a later stage.

For the test—retest reliability exercise, the same interviewer who performed interviews at T₁ re-interviewed the respondents at a second point in time (T₂ ), this time alone. The interval between T₁ and T₂ was on average 11 days, and 77.5% of the subjects were re-interviewed at T₂ . For the taped interviews (where all four second raters rated the same material), reliability was estimated separately for each second rater, and the overall reliability was calculated.

Statistical analysis

For testing criterion validity, non-parametric correlation (Spearman's ρ) and Student's t-test were used. Interrater and test-retest reliability were examined for the total number of needs and for each need item individually. For the reliability of individual items, two measures of agreement were calculated: complete percentage agreement and unweighted κ coefficient. For the reliability of the total number of needs, variance component estimation was performed using the MINQUE (minimum norm quadratic unbiased estimation) method in the Statistical Package for the Social Sciences (SPSS) version 7.5 for Windows (SPSS, 1996). Variance components estimation is a flexible method of obtaining reliability coefficients if there are several sources of variation, and the MINQUE method is robust concerning moderate departure from normality (Dunn, 1992). For interrater estimates, both patient variation and rater variation were estimated as random effects. For test—retest estimates, time was included as a fixed effect. Each interclass correlation coefficient was estimated as the ratio of variation between subjects to total variation. Relative bias in T₂ estimates compared with T₁ estimates was tested by using a paired t-test. Fixed effects between raters were tested by using a fixed effect analysis of variance. Also, a Student's t-test was used to compare the mean differences in the ratings of users, carers and staff for the comparison of the individual DAS scores in the two CANDID groups.

RESULTS

Socio-demographic characteristics and needs profile of the study sample

The characteristics of the 40 service users recruited for the reliability study are shown in Table 1.

Table 1 Characteristics of patients enrolled in the reliability study (n=40)

Characteristic	Value
Age (years; mean (range))	37.5 (20-67)
Gender (n (%))
Male	27 (67.5%)
Female	13 (32.5%)
Ethnic origin (n (%))
Caucasian	38 (95%)
African-Caribbean	1 (2.5%)
Asian	1 (2.5%)
Patient status (n (%))
Outpatient	15 (37.5%)
Inpatient	9 (2.5%)
Residential	16 (40%)
Level of learning disability (n (%))
Mild	26 (65%)
Moderate	19 (22.5%)
Severe/profound	5 (12.5%)
Living situation (n (%))
Alone	2 (5%)
With partner	2 (5%)
With parents	9 (22.5%)
With others	27 (67.5%)
Marital status (n (%))
Single	37 (92.5%)
Married	3 (7.5%)
Clinical conditions (n (%))
Psychotic illness	20 (50%)
Autism	10 (25%)
Epilepsy	11 (27.5%)

The mean total number of needs per user identified at T₁ by the users themselves (n=31) was 11.55 (s.d.=2.51, 95% CI 10.63-12.47), while informal carers (n=27) identified 14.10 needs (s.d.=2.34, 95% CI 13.11-14.96) and staff (n=40) identified 13.98 (s.d.=2.97, 95% CI 13.03-14.92). The ratings by carers and staff did not differ significantly, whereas the ratings by users and carers were significantly different, as were ratings by users and staff, (P < 0.01). Table 2 shows the staff ratings for the 25 areas of the CANDID. The mean duration of the interviews at T₁ was 28.25 minutes (s.d.=7.84) for users, 39.56 (s.d.=6.52) for carers and 27.42 (s.d.=5.00) for staff.

Table 2 Assessment of need for the 25 areas of the CANDID

Area of need	No serious need (n (%))	Met need (n (%))	Unmet need (n (%))	Not known (n (%))
Accommodation	0 (0)	36 (90)	4 (10)	0 (0)
Food	1 (2.5)	38 (95)	0 (0)	1 (2.5)
Looking after the home	6 (15)	30 (75)	2 (5)	2 (5)
Self-care	6 (15)	33 (82.5)	0 (0)	1 (2.5)
Daytime activities	3 (7.5)	33 (82.5)	4 (10)	0 (0)
General physical health	22 (55)	18 (45)	0 (0)	0 (0)
Eyesight and hearing	24 (60)	16 (40)	0 (0)	0 (0)
Mobility	35 (87.5)	5 (12.5)	0 (0)	0 (0)
Seizures	32 (80)	7 (17.5)	1 (2.5)	0 (0)
Major mental health problems	21 (52.5)	16 (40)	3 (7.5)	0 (0)
Minor mental health problems	6 (15)	28 (70)	6 (15)	0 (0)
Information	28 (70)	8 (20)	0 (0)	4 (10)
Safety to self	21 (52.5)	17 (42.5)	2 (5)	0 (0)
Exploitation risk	11 (27.5)	27 (67.5)	2 (5)	0 (0)
Safety to others	18 (45)	14 (35)	8 (20)	0 (0)
Inappropriate behaviour	20 (50)	16 (40)	4 (10)	0 (0)
Substance misuse	37 (92.5)	3 (7.5)	0 (0)	0 (0)
Communication	20 (50)	18 (45)	2 (5)	0 (0)
Social relationship	6 (15)	26 (65)	7 (17.5)	0 (0)
Sexual expression	25 (62.5)	10 (25)	4 (10)	1 (2.5)
Caring for someone else	37 (92.5)	1 (2.5)	2 (5)	0 (0)
Basic education	3 (7.5)	27 (67.5)	8 (20)	2 (5)
Transport	6 (15)	28 (70)	4 (10)	2 (5)
Money budgeting	3 (7.5)	18 (45)	18 (45)	1 (2.5)
Welfare benefits	20 (50)	4 (10)	1 (2.5)	15 (37.5)

Validity

Face validity

A number of different perspectives were taken into account during the development process, and comments were incorporated into the final version. Professionals from a variety of disciplines expressed the view that the CANDID is a comprehensive instrument covering a wide range of needs of people with learning disabilities and mental health problems. The CANDID, therefore, has acceptable face validity.

Content validity

All 45 users and carers approached responded to the questionnaire. Following the survey a total score for each need item was calculated and all items were ranked according to this score. The highest scoring items were accommodation and self-care, while the lowest were autistic features and telephone use. No additional items were suggested by more than two respondents.

Consensual validity

Forty-five experts (81.8%) responded to the questionnaire. Regarding the instrument's content, no item was rated as redundant and only ‘communication’ was suggested for inclusion by more than two respondents. Only 5% of respondents rated the instrument's structure as low for ‘helpfulness of anchor points’ and ‘ease of use.’ The draft instrument's language was rated as ‘inappropriate’ by 20% of respondents, and their comments were taken into account in developing the final version. Thus, satisfactory consensus on the content and structure of the instrument was ensured.

Criterion validity

The CANDID summary scores (total number of needs) were compared with the total DAS and GAF scores. In both DAS and GAF, higher scores indicate higher levels of functioning, whereas high CANDID scores indicate high need. The Spearman's ρ correlation coefficients were —33 (P < 0.05) and —47 (P < 0.01) respectively, implying high concurrent validity.

In the individual areas examined (behaviour, communication, mobility, social interaction and self-care), the DAS scores were consistently lower for those assessed by the CANDID as having a need than for those assessed as not having a need, indicating an association between the DAS and CANDID in the expected direction. In the first three areas the differences were statistically significant and the respective mean difference values were 2.95 (P < 0.001, 95% CI 1.63-4.27), 0.79 (P < 0.05, 95% CI 0.60-1.51) and 1.2 (P < 0.001, 95% CI 0.78-1.62). In the remaining two areas, where statistical significance was not reached the DAS items inquired about much narrower areas of functioning than the corresponding CANDID items.

Reliability

Intraclass correlations between summary scores of the two raters (for interrater reliability) and at the two points in time T₁ and T₂ (for test—retest reliability) were calculated using variance components analysis as described above. For interrater reliability the intraclass coefficients were 0.93 for user, 0.90 for carer and 0.97 for staff ratings. For test—retest reliability they were 0.71, 0.69 and 0.86 respectively. On the basis of paired t-tests there was no evidence of relative bias between the two time points or between live and taped interviews. In addition to total number of needs (section 1), the interrater and test—retest reliability of the summary scores for Sections 2, 3 and 4a were calculated. There was a high degree of agreement between raters and across time, and the correlations were generally higher for interrater than for test—retest reliability. The results are shown in Table 3.

Table 3 Test-retest and interrater reliability for the CANDID

CANDID Section	Type of reliability	Intraclass correlation coefficient
		User ratings	Carer ratings	Staff ratings
1 (total number of needs)	Test-retest	0.71	0.69	0.86
	Interrater	0.93	0.90	0.97
2 (help given by relatives/friends)	Test-retest	0.93	0.95	0.96
	Interrater	0.96	0.91	0.96
3a (help given by services)	Test-retest	0.75	0.90	0.88
	Interrater	0.98	0.96	0.92
3b (help needed by services)	Test-retest	0.72	0.87	0.84
	Interrater	0.94	0.93	0.94
4a (right kind of help?)	Test-retest	0.65	0.76	0.64
	Interrater	0.84	0.86	0.88

Interrater and test—retest reliability were also examined for each need area item separately and two measures of agreement were calculated: percentage of complete agreement and κ coefficients. Values of κ in the range 0.81-1.00 indicate ‘almost perfect’ agreement with 0.61-0.80 indicating ‘substantial’, 0.41-0.60 ‘moderate’ and 0.00-0.40 indicating ‘poor’ agreement (Landis & Koch, 1977). Only three κ values were in the ‘poor’ agreement range; all were derived from user ratings and concerned test—retest reliability in the scores of self-care, information and welfare benefits. Values of the κ in some instances were very low despite high complete agreement. Examination of the raw data in such instances showed that this was due to highly skewed distribution of scores. This difficulty with misleading κ values is discussed by Feinstein & Cicchetti (1990).

For interrater reliability the lowest percentages of complete agreement on ratings of presence of need in a defined area were 71.0% for users, 85.1% for carers and 77.5% for staff; only 0.7% of the percentages were below 75%. For test—retest reliability the lowest percentages were 58.3% for users, 66.6% carers and 71.0% for staff; only 4.7% of the percentages were below 75%. Table 4 shows the κ coefficients for each need area item for interrater and test—retest reliability.

Table 4 Identification of need in the 25 areas of the CANDID: κ coefficients for interrater and test-retest reliability

	User		Carer		Staff
	Interrater (n=31)	Test-retest (n=24)	Interrater (n=27)	Test-retest (n=21)	Interrater (n=40)	Test-retest (n=31)
Accommodation	—1	—1	—1	—1	0.84	0.84
Food	0.77	0.78	0.79	1	0.33	—1
Looking after the home	0.74	0.50	1	—1	1	0.63
Self-care	0.87	0.29	1	1	1	0.48
Daytime activities	0.76	—1	0.80	0.78	0.77	0.44
General physical health	0.91	0.88	0.91	0.80	1	0.81
Eyesight and hearing	0.87	0.73	0.93	0.91	0.95	0.92
Mobility	0.96	0.78	0.91	0.77	1	1
Seizures	1	0.88	1	1	1	0.91
Major mental health problems	0.94	0.54	0.86	0.64	0.87	0.61
Minor mental health problems	0.77	0.46	0.75	0.47	0.82	0.70
Information	0.75	0.40	1	0.50	1	0.68
Safety to self	0.70	0.47	0.93	1	0.96	0.88
Exploitation risk	0.89	0.73	0.73	0.46	1	0.71
Safety to others	0.80	0.64	0.86	0.82	0.92	0.75
Inappropriate behaviour	0.96	0.68	0.93	0.48	1	0.62
Substance misuse	0.96	0.46	1	0.50	0.82	1
Communication	0.94	0.69	0.87	0.66	0.95	0.65
Social relations	0.89	0.48	0.85	0.76	0.90	0.93
Sexual expression	1	—1	0.86	0.53	0.81	0.82
Caring for someone else	1	0.66	1	1	1	1
Basic education	0.77	0.77	0.81	0.70	0.60	0.47
Transport	0.70	0.52	0.72	1	0.90	0.86
Money budgeting	0.58	0.62	0.72	0.53	0.83	0.53
Welfare benefits	0.90	0.29	0.78	0.79	0.92	0.84

DISCUSSION

The measurement of need

Two conceptual issues underlie the difficulty in measuring need and are particularly relevant in people with learning disabilities. First, there is no consensus about the definition of need. The following definitions, among others, have been proposed “the requirement of individuals to enable them to achieve acceptable quality of life” (Department of Health Social Services Inspectorate, 1991), and “a problem which can benefit from an existing intervention” (Stevens & Gabbay, 1991). People with learning disabilities often have a complex constellation of difficulties commonly referred to as ‘special needs’, but it has not been established whether either the ‘quality of life’ or ‘ability to benefit’ approach (or indeed any other) contains the necessary and sufficient information for defining need in this population.

Second, there is a lack of consensus about who should assess need. Some argue that need can only be assessed by professionals (Mooney, 1986), whereas others (Bradshaw, 1972) claim that individuals' assessment of their own (‘felt’ and ‘expressed’) needs is valid. The combination of cognitive impairment, mental state abnormalities and behavioural disorders exhibited by adults with learning disabilities and mental health problems may significantly affect their mental capacity. However, it is important to take into account the views of the service users themselves, especially if they differ systematically from those of other assessors (Slade, 1994).

Validity

A balance had to be struck between the utility and the comprehensiveness of the new instrument. The decision to retain or add items was taken on the basis of the balanced views of those who participated in the validity study. Accordingly, one item (communication) was added, whereas four (intimate relationships, autistic features, telephone use and medication) were not retained from the original list of items.

The lack of a ‘gold standard’ instrument necessitated the use of instruments that only indirectly measure level of need. Furthermore, it was only possible to compare five out of the 12 DAS items with the corresponding areas in the CANDID. The remaining seven either did not correspond to any CANDID areas or had their scoring based on different criteria, thus not allowing meaningful comparison.

Reliability

A difficulty associated with testing the interrater reliability of instruments administered via a semi-structured interview is that the second rater may be influenced by the interviewer. The rating of Sections 2-4 is dependent on the rating of the presence of a need in Section 1. Moreover, this process reduces the sample sizes available for analysis and caution therefore is required in interpreting the reliability of Sections 2-4.

Generalisability

The reliability study sample was nonrandom and our research was conducted in only two sites, whereas there are large variations in the philosophy, structure and aims of services providing care for adults with learning disabilities and mental health problems in the UK. Nevertheless, an effort was made to make the sample as representative as possible by including service users from a variety of settings and with a range of levels of learning disability and associated mental or behavioural disorders.

Assessing needs from multiple perspectives is one of the main characteristics of the CANDID. However, the views of the service users are skewed towards the high end of ability, as individuals with severe and profound disability were not able to rate their own needs. Although not investigated in this study, one approach for future work will be to assess the views of an advocate whenever it is not possible to obtain the views of the service user.

Implications for health and social services

A valid and reliable needs-assessment instrument for people with learning disabilities and mental health problems will be a useful clinical and research tool. The increasing costs of health care and lack of consensus about the most effective way of organising and providing health and social care have led government policy to be increasingly informed by evidence-based practice. The CANDID will enable rational use and fair distribution of scarce resources by encouraging needs-led service provision. However, the CANDID was not designed as an outcome measure, so other appropriate instruments should be used when measurement of change over time is required.

The CANDID will facilitate the fulfilment of the local authorities' statutory obligation for needs assessment. It can be used for planning services, both at an individual level (developing individualised care plans) and at a population level (designing a service in a geographical area). The CANDID can, through systematic inquiry, help to identify areas of need that may require further exploration. However, it is a screening instrument rather than a diagnostic one and as such it is not a substitute for health or social care interventions, such as regular health checks.

As with the CAN, the need for separate versions for research and clinical use emerged during the developmental process. The findings reported here were obtained using the research version of the CANDID. Two areas of concern about the draft clinical version have arisen: difficulty with its use in busy routine clinical settings, and the potential loss of useful clinical information caused by the structured nature of responses. The clinical version of the CANDID has adopted a combined approach: it uses the structured format of Section 1 to rate systematically the presence or absence of need, followed by semi-structured sections which allow the recording of relevant clinical information as part of the individual's care plan.

The findings of this study suggest that the CANDID has acceptable validity and reliability when used under the research conditions of this study. More data on its utility and feasibility are required; these characteristics will be established with its application in routine settings in the long term. A pilot study by the core research team aimed at investigating the feasibility of the instrument's use in routine community-based and in-patient settings is currently under way.

Clinical Implications and Limitations

CLINICAL IMPLICATIONS

• ▪ The use of a valid, reliable and brief needs-assessment instrument for adults with learning disabilities and mental health problems encourages comprehensive needs-led individual care planning.

• ▪ The Camberwell Assessment of Need for Adults with Developmental and Intellectual Disabilities (CANDID) allows the systematic collection of needs-based data which, when aggregated, can inform resource allocation and service planning in health and social care for this group.

• ▪ The perspectives of service users and informal carers, as well as professional staff, can be assessed separately and taken into consideration.

LIMITATIONS

• ▪ The sample for the reliability study was relatively small and selected from only two sites.

• ▪ The structure of the interview meant that interrater reliability could not take into account the judgements of the non-interviewing rater if the latter identified the presence of a need when the interviewer did not.

• ▪ Because of the lack of any relevant ‘gold standard’ for measuring needs in this group, concurrent validity was investigated using instruments that only indirectly assess need.