Validation of the Social Satisfaction Questionnaire for outcome evaluation in substance use disorders

Abstract

Aims and Method

To develop a scale to measure social satisfaction in people with substance use disorders and to test its psychometric properties. The rationale is that social satisfaction is more universal and relevant to treatment planning than assessing social problems. The new Social Satisfaction Questionnaire (SSQ) was derived from an existing social problems questionnaire and validation was undertaken on two large clinic populations.

Results

An eight-item SSQ was tested and found to have good psychometric properties in terms of test–retest reliability, internal consistency, distribution of responses and concurrent validity.

Clinical Implications

The SSQ is suitable for use as the social domain element of an outcome measures package.

Social functioning has consistently been associated with help-seeking and entry into treatment for alcohol and drug problems (e.g. Moos & Moos, 2004) and is an important domain to measure in the evaluation of treatment outcome; social problems are a key component of current models of the treatment process (Gifford et al, 2006; Orford et al, 2006). Poor levels of social functioning are associated with poorer treatment outcomes and risk of relapse (Moos & Moos, 2006). Substance use disorders span the socio-economic spectrum, and the task of identifying norms to measure social functioning components of treatment outcome is fraught with difficulty.

Standard measurement tools include the European Quality of Life (EuroQoL; Brooks et al, 2003), which is useful for comparisons of health status across populations but is insensitive for specific social outcomes, and the Short Form Health Survey (SF–36; Ware et al, 2000), which is acceptable to service users (Garratt et al, 1993) and suggested as a standard for the National Health Service (NHS) but is too lengthy to be included in routine, substance misuse specific outcome measurement and does not inform treatment goals. Criteria suitable for outcome measurement in routine clinical practice (Raistrick et al, 1994) are universality of scale items, change potential, items written in plain English, self-completion format to avoid therapist bias, satisfactory psychometric properties and brevity of completion. To include these and tap into the subjective experience of help seekers, we decided to measure satisfaction with social circumstances. Thus, the individual's objective social circumstances are less important than how they feel about them both for measurement and as the focus of treatment planning.

The relativity of social functioning was addressed by Corney & Clare (1985) who developed a 33-item Social Problems Questionnaire (SPQ) as a compact, comprehensible, valid and reliable self-report questionnaire for screening individuals in primary care or related settings who are particularly at risk for manifesting social maladjustment and dysfunction (Corney & Clare, 1985: p. 638).

Recognising the difficulties inherent in attempts to measure and compare social circumstances, the SPQ combines items enquiring about objective circumstances with items enquiring about the degree of satisfaction experienced with reference to these. The domains covered are: housing, occupation, finances, social activities and relationships, relatives, marital status, children, other domestic relationships, legal matters, and questions for those living alone. The scale was piloted and tested in primary care samples (including the responses of relatives), in psychiatric and epilepsy out-patient departments and in a social work case-load, and found to be a valid and reliable measure which is readily acceptable for use either by postal or face-to-face contact.

This paper describes the development and validation of the Social Satisfaction Questionnaire (SSQ), which was adapted from the SPQ (Corney & Clare, 1985) as a brief measure of satisfaction and designed for use in routine assessment and treatment outcome evaluation (Tober et al, 2000).

Method

The SSQ was developed in four steps:

1. • step 1 - item reduction from the SPQ to create the eight-item SSQ

2. • step 2 - piloting the SSQ

3. • step 3 - main validation in clinical practice on two large samples

4. • step 4 - test-retest reliability.

Items which did not measure satisfaction directly, those measuring actual circumstances, failing to meet inclusion criteria, or referring to single occupational groups were removed from the SPQ, and some rephrasing of the remaining items was undertaken. This produced an eight-item questionnaire (see data supplement to the online version of this paper). The SSQ was piloted with 20 service users and minor adjustments made to wording and layout.

The psychometric properties of the scale were tested in three separate samples. Samples 1 and 2 were used for testing validity and internal consistency. Sample 1 consisted of 1665 new service users attending NHS services in Leeds and Newcastle. The assessment package comprised self-reported substance use, dependence as measured by the Leeds Dependence Questionnaire (LDQ; Raistrick et al, 1994) and psychopathology as measured by the General Health Questionnaire (GHQ; Goldberg & Williams, 1988). Sample 2 consisted of 6732 new service users attending the Leeds service. For this sample the assessment package had been updated to measure psychopathology using the Clinical Outcomes in Routine Evaluation – Outcome Measure (CORE–OM; Evans et al, 2002). Sample characteristics are presented in Table 1.

Table 1. Characteristics of samples 1 (n=1665) and 2 (n=6732)

	Sample 1		Sample 2
	Newcastle	Leeds
Gender, n (%)
Male	743 (78)	569 (80)	4926 (73)
Female	213 (22)	140 (20)	1806 (27)
Age, years: mean (s.d.)	34.9 (10.5)	33.3 (10.9)	38.4 (9.8)
Main reason for referral, n (%)
Alcohol use	425 (60)	396 (47)	2512 (37.3)
Drug use	283 (40)	453 (53)	4220 (62.7)
LDQ score: mean (s.d.)	19.19 (7.59)	20.37 (7.64)	17.4 (8.4)
GHQ score: mean (s.d.)	23.55 (8.63)	22.47 (8.46)	NA
CORE—OM score: mean (s.d.)	NA	NA	57.1 (21.3)

Sample 3 was used for the purpose of estimating test–retest reliability. The scale was administered as part of a package of three questionnaires to 61 respondents attending an NHS clinic for treatment of their heroin use. The sample consisted of 40 males (65.6%) and 21 females (34.4%), with a mean age of 25.7 years (s.d.=4.9, range 17-37). The mean time between first and second completion of the scale was 5.3 days (s.d.=3.75) with a minimum of 2 days and a maximum of 14 days.

Results

Item analysis

An item analysis of samples 1 and 2 was conducted to examine the homogeneity and internal consistency of the scale. Each of the eight items correlated with total SSQ score (Spearman's rho ranged from 0.57 to 0.66 and 0.60 to 0.68 for samples 1 and 2 respectively) and all were significant at the P < 0.001 level. Cronbach's α was 0.73 and 0.79 respectively and similar values held with each item deleted. In order to examine the factor structure of the scale, responses given by sample 2 were subjected to exploratory principal components analysis, which yielded three factors with an eigen value >1. The first of these had an eigen value of 3.3 accounting for 41% of the variance with item loadings ranging from 0.59 to 0.69.

Factor 2 was specific to accommodation and factor 3 to employment and finance, therefore it was felt that these items did not make sense as stand-alone factors (see Table 2) and so a single factor is a better interpretation of the data.

Table 2. Principal components analysis for the Social Satisfaction Questionnaire (sample 2, n=6732)

	Component
	1	2	3
SSQ1	0.595	0.637	0.220
SSQ2	0.641	0.634	0.082
SSQ3	0.592	-0.259	0.552
SSQ4	0.620	-0.325	0.493
SSQ5	0.662	-0.333	-0.109
SSQ6	0.688	-0.315	-0.250
SSQ7	0.666	0.005	-0.466
SSQ8	0.644	0.005	-0.405

Test–retest reliability

Atest–retest reliability study was conducted with sample 3. The correlation between the total SSQ scores at the two data collection points was 0.85 and the mean correlation between individual item scores was 0.72. Mean scores at the first and second data collection points were compared in order to eliminate the possibility that scores have changed but were still correlated. No significant difference was found in the mean total score for the scale between the first and second scale completion (9.3 and 8.7 respectively) indicating that the responses had remained constant.

Validity

The relationships between the SSQ and other variables were examined, using a correlation technique to demonstrate construct validity. The expectation was that there would be significant but modest correlations given the distinctiveness of the domains of dependence, psychological health and social satisfaction. This was found to be the case. For sample 1 the SSQ was negatively and significantly correlated with measures of psychopathology (r=0.38, P < 0.001) and dependence (r=0.28, P < 0.001).

The above findings were confirmed by a stepwise multiple regression analysis, with SSQ score as the dependent variable and age, gender, GHQ score (psychopathology), LDQ score (dependence) and main reason for referral (alcohol or other drug) as independent variables. Results of this analysis showed that GHQ score and LDQ score predicted SSQ score (GHQ: t=9.01, P < 0.0001; LDQ: t=3.26, P <0.001). Thus, service users with higher psychopathology and dependence were most likely to have a low score for social satisfaction. As expected gender, main reason for referral and treatment site were not predictors of SSQ.

The correlations were reassessed on sample 2 which was large enough to analyse according to the drug groups of opiates, stimulants, cannabis and alcohol. Again the expectation was that there would be low-order but significant correlations between the SSQ and other assessment measures. This was found to be the case. The SSQ was negatively correlated with the LDQ in the range -0.16 to -0.28 and with CORE-OM in the range -0.40 to -0.52 (P=0.001) (Table 3). The scale achieves a close to normal distribution across sample 2, showing a capability to measure the full range of scores.

Table 3. Correlations of the Social Satisfaction Questionnaire score with LDQ and CORE—OM (sample 2, n=6732)

	Opiates	Stimulants	Cannabis	Alcohol	All
LDQ	-0.25	-0.16	-0.28	-0.27	-0.25
CORE—OM	-0.45	-0.40	-0.52	-0.51	-0.45

Discussion

The aim of this study was to develop and validate an instrument to measure a dimension of social functioning that was meaningful in a help-seeking population and suitable for use in routine practice. We have presented evidence that the SSQ has robust internal psychometric properties. Validation of a subjective construct such as satisfaction is problematic, however, we have shown that the SSQ is a unidimensional construct which measures a separate domain to that of dependence and psychological morbidity and is independent of unrelated variables.

The case for a new scale is based on the need for a social domain measure that has clinical utility and acceptability in a busy clinic environment. The scale has face validity in pointing to social domains that are likely to be important to and engage service users in treatment. In sample 2 we combined the SSQ with the LDQ, the CORE-OM and the substance use section of the Maudsley Addiction Profile (MAP; Marsden et al, 1998) to create a comprehensive outcome measurement package which takes approximately 15-20 min to complete. Services less concerned with psychological morbidity can save time by replacing CORE-OM with the MAP measure for this domain. There is a strong argument for using a standard assessment package. For example, the Addiction Severity Index (McLellan et al, 1980) is popular but its reliability and validity have been questioned (Mäkelä, 2004). There may be a stronger case for selecting from a menu of standardised outcome measures, each with strong reliability and validity, which can be packaged to suit local need, and it was with this approach in mind that we developed the SSQ.

The SSQ has been extensively used and found to have clinical utility in two NHS substance misuse services, however, there needs to be further testing of concurrent validity and in different service user groups, beyond problems of substance misuse. Therapists familiar with scale items can incorporate responses into treatment, for example to elicit motivational statements (Ford, 2003) and include service user concerns expressed through the SSQ in care plans. A critical test of the validity and utility of the SSQ will be its ability to predict treatment outcome in combination with other measures in the package and this will be reported in due course.

Declaration of interest

None.