The PAS–ADD Checklist: independent replication of its psychometric properties in a community sample

Abstract

Background

The development of reliable, valid measures of psychopathology in people with intellectual disabilities is an important task. However, independent replication studies are rarely reported.

Aims

To report data on the psychometric properties of the Psychiatric Assessment Schedule for Adults with Developmental Disabilities (PAS–ADD) Checklist.

Method

The PAS–ADD Checklist was completed for 226 adults as part of the assessment process for a specialist mental health service for people with intellectual disabilities.

Results

Internal consistency was acceptable. Factor analysis revealed one main factor that was characterised by items related to mood. The Checklist was sensitive to differences between diagnostic groups and had an overall sensitivity of 66%; its specificity was 70%.

Conclusions

The PAS–ADD Checklist is a quick and easy to use screening tool. Although at present it is the best measure available, it should not be the only method used to identify psychiatric disorders in people with intellectual disabilities.

The lack of psychometrically sound measures of psychopathology in people with intellectual disabilities, identified in the early 1990s (Sturmey et al, 1993), was in part addressed by the Psychiatric Assessment Schedule for Adults with Developmental Disabilities (PAS–ADD) interview (Moss et al, 1993) and more recently the shorter PAS–ADD Checklist questionnaire (Moss et al, 1998). The latter is a screening tool that can be used by untrained people to identify clients with intellectual disabilities at risk of developing a psychiatric disorder. It contains 29 items concerning symptoms of psychiatric disorders, split into five scales (A–E). These scales combine to produce three total scores: 1, affective/neurotic disorder; 2, possible organic disorder; and 3, psychotic disorder. Scores equal to or above specified thresholds indicate that further assessment is necessary. Those who developed the PAS–ADD Checklist found it to be psychometrically sound (Moss et al, 1998). The study reported here provides an independent evaluation of its psychometric properties.

METHOD

Sample

The sample comprised all 226 individuals who were referred over a 3-year period to a specialist mental health service for people with intellectual disabilities. Of these 226 individuals, 140 (62%) were male and 86 (38%) were female. The average age was 34 years (s.d.=13.5). Most (71%) of those referred were White, 19% were African–Caribbean, 6% were Asian and 4% were classed as other ‘non-White’. More than two-thirds (68%) had mild intellectual disability, 20% had moderate intellectual disability and 12% had severe intellectual disability. All lived in the community: 49% lived with their family, 31% lived in supported housing for four or more people which was not health-service funded, 16% lived independently and 4% lived in a health service residence for eight or more people which was funded by the health service. Table 1 lists the psychiatric diagnoses made by the clinician at assessment.

Table 1 Psychiatric diagnoses made by the clinician at assessment (n=226)

Diagnostic category (ICD-10 code)	% (n)
No psychiatric diagnosis	35 (78)
Schizophrenia spectrum disorder	19 (42)
(F20-29)
Depressive disorder (F32-39)	12 (27)
Anxiety (F40-48)	8 (19)
Adjustment reaction (F43)	8 (18)
Dementia (F00-03)	4 (10)
Personality disorder (F60-69)1	8 (18)
Other: delirium (F05) eating disorders	6 (14)
(F50) and hyperkinetic disorders (F90)1

Procedure

Data collection

Each individual attended an assessment interview with a psychiatrist at which information including clinical history and current psychiatric diagnosis was recorded. In addition, a key informant such as a relative or staff member was asked to complete the PAS–ADD Checklist for each individual. The assessing psychiatrist was masked to the PAS–ADD Checklist score at assessment.

Data analysis

Data analysis was conducted using the Statistical Package for the Social Sciences, version 10. Four analyses were conducted. First, in order to look at reliability, item analyses of each of the five scales (A–E) and the three total scores (1–3) were conducted and Cronbach's α was calculated. Alpha values greater than 0.7 are considered acceptable (Nunnally, 1978). Item–total point biserial correlations were also calculated to measure internal consistency. Rogue items, which correlated with a total score less than 0.3, were identified. Second, an exploratory factor analysis of the PAS–ADD Checklist items was conducted, in order to assess if any items in the Checklist were measuring aspects of the same underlying dimensions or factors. A principal components analysis with quartimax rotation was used. The number of factors was determined using a scree plot of the variances before rotation. Third, to assess validity, PAS–ADD Checklist scores were compared with clinical psychiatric diagnoses. To make a valid comparison between each diagnosis, only those diagnoses present in more than ten cases were included. Diagnoses fulfilling this criterion were ‘no psychiatric diagnosis’, schizophrenia spectrum disorder, personality disorder, anxiety disorder, depressive disorder and adjustment reaction. Oneway analysis of variance (ANOVA) with post hoc Scheffé tests were used. Finally, in order to examine the sensitivity and specificity of the Checklist, a summary of the numbers of people who crossed any PAS–ADD Checklist threshold, in relation to the numbers who had a clinical psychiatric diagnosis covered by the Checklist, was calculated.

RESULTS

Reliability

Item analysis

Table 2 summarises the results of the item analysis. Scales A and B and total scores 1 (affective/neurotic disorders) and 2 (possible organic disorders) had alpha values equal to or greater than 0.7. Scales C, D and E and the total score 3 (psychotic disorders) had alpha values equal to 0.6. The median item–total (minus item) point biserial correlations were greater than 0.3. Every scale had a number of rogue items with item–total correlations of less than 0.3. In some cases there were scales, such as scale B, where 4 out of 11 items had item–total correlations of less than 0.3.

Table 2 Values of Cronbach's alpha, item—total point biserial correlations and number of rogue items for the PAS—ADD Checklist scales and scores

	Items n	α	Item—total point serial correlation Median (range)	Rogue items n
Scale
A	7	0.7	0.36 (0.10-0.54)	2
B	11	0.7	0.33 (0.07-0.45)	4
C	4	0.6	0.37 (0.22-0.53)	2
D	2	0.6	0.47 (0.47)	0
E	5	0.6	0.30 (0.24-0.50)	1
Total score
1 Affective/neurotic	22	0.8	0.41 (0.08-0.61)	5
2 Organic	6	0.7	0.52 (0.21-0.56)	2
3 Psychotic	5	0.6	0.30 (0.24-0.50)	1

Factor analysis

The results of the factor analysis are shown in Table 3. The first nine factors had eigen-values greater than 1.0 and these factors accounted for 64% of the variance. An examination of a scree plot suggested a single factor structure, since the first factor accounted for 20% of the variance and the subsequent eight factors accounted for 4–8% of the variance. The first factor was characterised primarily by items related to mood, such as loss of interest and energy, sadness, avoiding conversation, low self-esteem, loss of appetite and confidence, and poor concentration. The second factor was characterised by three items related to sleep disturbance. The third factor was characterised by three items related to psychotic symptoms. It was difficult to characterise subsequent factors clearly.

Table 3 Exploratory factor analysis for the PAS—ADD Checklist

	Factor
Item	1	2	3	4	5	6	7	8	9
Loss of interest	0.80
Loss of energy	0.74
Feeling sad or ‘down'	0.63	0.37
Avoiding conversation	0.65							0.31
Low self-esteem	0.57						0.44		0.53
Loss of appetite	0.59
Loss of confidence	0.55								0.36
Concentration	0.59				0.40	0.40
Sleep broken		0.88
Sleep delayed		0.79
Waking		0.71
Beliefs			0.84
Experience			0.77
Suspicious			0.61
Gesture				0.79	0.80
Language				0.75	0.81
Too happy				0.55
Irritable					0.74
Startled				0.56	0.44				0.32
Restless			0.32		0.64	0.33
Forgets	0.34				0.36	0.39
Self-care	0.41				0.39	0.65	0.42		0.31
General fear			0.64				0.80
Sudden fear					0.82	0.32	0.31	0.62
Incapacitated								0.79
Repeated actions			0.32		0.36			0.57
Suicide							0.79		0.89
Eigenvalue	5.33	2.20	1.83	1.69	1.50	1.44	1.18	1.11	1.02
Variance (%)	19.7	8.1	6.8	6.2	5.6	5.3	4.4	4.1	3.8

Validity

Table 4 shows the PAS–ADD Checklist scores for people who were diagnosed by a clinician as having ‘no diagnosis’, schizophrenia spectrum disorder, personality disorder, anxiety disorder, depressive disorder or adjustment reaction. There was a significant difference between individuals on total score 1 (affective/neurotic disorder), in which people who had depressive disorder scored higher than those with no diagnosis, and all other psychiatric diagnoses. There was a significant difference between individuals on total score 2 (possible organic disorder), in which those with depressive disorder scored higher than those with no diagnosis. There was a significant difference between individuals on total score 3 (psychotic disorder), in which people with schizophrenia spectrum disorder scored higher than those with no psychiatric diagnosis and all other diagnoses.

Table 4 Diagnostic group scores on the three total scores of the PAS—ADD Checklist

Variable	No psychiatric disorder	Schizophrenia spectrum disorder	Personality disorder	Anxiety disorder	Depressive disorder	Adjustment reaction disorder	F	Post hoc Scheffé tests
n	78	42	18	18	28	18
Total score: mean (s.d.) 1 Affective disorders	2.53 (3.47)	4.14 (3.97)	3.39 (4.15)	4.83 (4.00)	9.75 (6.37)	3.78 (4.58)	12.04***	Depression > no pathology, schizophrenia, personality disorder, adjustment reaction, anxiety
2 Organic disorders	1.18 (1.65)	1.50 (1.71)	1.00 (1.50)	1.06 (1.11)	2.54 (2.44)	1.50 (2.09)	2.84*	Depression > no pathology
3 Psychotic disorders	0.60 (1.10)	2.05 (1.99)	0.61 (1.15)	0.44 (0.78)	0.32 (0.72)	0.44 (1.10)	9.71***	Schizophrenia > no pathology, personality disorder, adjustment reaction, anxiety, depression

Sensitivity and specificity

Table 5 shows the numbers of people who crossed any PAS–ADD Checklist threshold in relation to the numbers who had a clinical psychiatric diagnosis covered by the Checklist. The sensitivity of the PAS–ADD Checklist was 66% and the specificity was 70%.

Table 5 Numbers of people who crossed any PAS—ADD Checklist threshold in relation to the numbers who had a clinical psychiatric disorder covered by the Checklist

Presence of disorder	Checklist thresholds crossed
	Yes	No	Total
Psychiatric disorder covered by Checklist	76	40	116
Psychiatric disorder not covered by Checklist/no	33	77	110
psychiatric disorder present
Total	109	117	226

DISCUSSION

Reliability

Internal consistency

The values of Cronbach's alpha for the scales A–E and total scores 1–3 (affective/neurotic, possible organic and psychotic disorders) in this sample were similar to those reported by Moss et al (1998). The majority showed acceptable consistency as they were greater than 0.7; there were, however, three scales and one total score (3, psychotic disorders) which had lower alpha scores (α=0.6). Moss et al (1998) suggest that an alpha score of 0.6 is generally acceptable, although this criterion is not as stringent as the more widely recognised 0.7 threshold (Nunnally, 1978). One of the possible explanations for the lower alpha values of these scales and scores is the fact that they consist of a smaller number of items (Moss et al, 1998). Although it is recognised that such scales can have high alpha values, it may also make the scale more unstable. However, as Moss et al (1998) suggest, a low alpha value does not necessarily mean that the scale will not work well as a screening tool, where the aim is to indicate the possible presence of a psychiatric disorder, not to give a specific diagnosis.

The number of rogue items is perhaps to be expected, as the checklist was not designed to identify specific disorders but rather to indicate the possible presence of a range of psychiatric disorders. There is thus some variation in the items included in each scale or total score to reflect the range of disorders.

Factor structure

Nine factors were initially identified, accounting for 64% of the variance. The first three factors, characterised by mood items, sleep disturbance and psychotic symptoms, are similar to three of the factors identified by the authors of the Checklist, which they characterise as depression, restlessness and psychosis (Moss et al, 1998). The other factors, however, are hard to characterise and account for little of the variance in this study.

Validity

The validity of the PAS–ADD Checklist appears to be good when considering the scores of people who have different psychiatric diagnoses. People who had depressive disorder scored higher on total score 1 (affective/neurotic disorder) than those who did not have this disorder, demonstrating that in terms of affective/neurotic disorders the Checklist performed well and identified the correct people. Individuals with depressive disorder also scored significantly higher than those without this disorder on total score 2 (possible organic disorder), although the significance was relatively low. This is not surprising, because there is some overlap between the scales that contribute to total score 1 and total score 2. Also, no individual in this section of the analysis had an organic disorder, so we would not expect the scores of the people with the disorders that are included to vary significantly on this organic disorder threshold.

People with schizophrenia spectrum disorder scored significantly higher on total score 3 (psychotic disorder) than people with any other diagnosis, confirming that the Checklist performs well on this disorder.

Sensitivity

Any screening tool must be assessed in relation to sensitivity. The main criticism of the PAS–ADD Checklist in this study relates to this measure.

The sensitivity (proportion of people with a psychiatric disorder covered by the Checklist who are correctly classified by the instrument as having a psychiatric disorder) of the PAS–ADD Checklist was 66%. This is lower than the figure of 78% calculated from the findings of the developers of the Checklist (Moss et al, 1998) and is also lower than other screening measures such as the 12-item General Health Questionnaire, which has a sensitivity of 76% (Goldberg et al, 1997). There are several possible explanations for the presence of false negatives. Moss et al (1998) found that the likelihood of crossing the thresholds rose with severity of the illness. Although in our study the severity of clinician diagnosis was not recorded, it might have been the case that some of these people did not have symptoms that were severe enough to be picked up by the Checklist. Of the people not crossing any threshold, 14 had schizophrenia spectrum disorder, which is a chronic disorder. At assessment these people's symptoms might have been absent and therefore not identified by the Checklist, if controlled through medication or if the person's disorder was in remission. Unfortunately, these data were not available, so this can only be proposed as a possible explanation.

The large number of people diagnosed as having an affective disorder but not crossing any of the thresholds (n=25) might be due to the nature of these diagnoses. Although some aspects may be observable, and the Checklist focuses mainly on these elements, scoring highly on the checklist and crossing a threshold is reliant to some extent on the person being able to communicate how he or she is feeling. This may be easier to elicit from people with intellectual disabilities in a clinical assessment rather than by use of a Checklist that is not completed by the patients themselves.

The breakdown of level of intellectual disability in those who had a diagnosis covered by the Checklist but who did not cross the threshold was similar to the breakdown of the total sample.

Although the above explanations may very well be valid, the data are not available to prove them, and the fact remains that the sensitivity of the PAS–ADD Checklist in this study was fairly low. A further consideration raised by this analysis is that 14% of this sample had a psychiatric diagnosis that the PAS–ADD Checklist was not designed to identify and therefore could not be expected to pick up.

If anything, we would expect a screening instrument to be overinclusive rather than underinclusive. In this study 15% of the total sample had no psychiatric disorder, or a psychiatric disorder that was not covered by the Checklist but crossed at least one of its thresholds. This is higher than the 8% of false positives calculated from findings of the Checklist's developers (Moss et al, 1998). For the purpose of screening people for further psychiatric assessment, it is preferable to have false positives rather than false negatives: people with intellectual disability may find going to a psychiatric out-patient clinic very upsetting and a high rate of false positives would be costly. Therefore we would hope for a low false positive rate. The specificity of the PAS–ADD Checklist was 70%, indicating that 70% of people who did not have a psychiatric disorder or had a psychiatric disorder that was not covered by the PAS–ADD Checklist were correctly identified.

We did not explore the sensitivity and false positive rates of the PAS–ADD Checklist with lower threshold scores. However, this may be something to consider in the future.

In summary, the PAS–ADD Checklist had acceptable internal consistency, one main factor characterised by mood items was sensitive to differences between diagnostic groups, and had an overall sensitivity of 66%.

Limitations of the study

There was only a small number of people with an organic disorder in this sample. Consequently, it was difficult to determine how successful the PAS–ADD Checklist was at identifying these disorders. It would also have been useful to have had some measure of the severity of the disorders as clinically diagnosed, as this would have enabled us to comment further on the issue of severity of symptoms affecting the crossing of the threshold scores.

The PAS–ADD Checklist has been revised since our study was completed, and although the items in the two versions differ only slightly, there is some variation in the order the items are presented. Whether this revision will affect the PAS–ADD Checklist's psychometric properties remains to be seen in future research.

Clinical Implications and Limitations

CLINICAL IMPLICATIONS

1. ▪ The Psychiatric Assessment Schedule for Adults with Developmental Disabilities (PAS–ADD) Checklist should not be used as the sole screening method for identifying possible psychiatric diagnoses in people with intellectual disabilities; however, it is the best psychometric measure available.

2. ▪ Appropriateness of referral to mental health services for full assessment, diagnosis and treatment might improve through use of the PAS–ADD Checklist.

3. ▪ The PAS–ADD Checklist appears to have the potential to reduce the level of undetected mental health problems among people with intellectual disabilities.

LIMITATIONS

1. ▪ The number of people with organic disorders was small, making it difficult to determine the success of the PAS–ADD Checklist in identifying them.

2. ▪ There was no measure of the severity of symptoms.

3. ▪ The Checklist has been revised since this study.