Development, internal consistency and reliability of the Verona Service Satisfaction Scale – European Version

Abstract

Background

Satisfaction with mental health services is an important quality and outcome variable. The Verona Service Satisfaction Scale (VSSS) is a well-established method for measuiring service satisfaction.

Aim

To report the development and reliability study of the European Version of the VSSS (VSSS–EU).

Method

A sample of people with schizophrenia on the case-load of local mental health services in the five European participating countries was assessed. The VSSS–EU was administered at one site in each country at two points in time. Internal consistency and test–retest reliability were assessed and compared between the five sites.

Results

The α coefficient for the VSSS–EU total score in the pooled sample was 0.96 (95% CI 0.94–0.97) and ranged from 0.92 (95% CI 0.60–1.00) to 0.96 (95% CI 0.93–0.98) across the sites. Test–retest reliability for VSSS–EU total score, pooled over sites, was 0.82 (95% CI 0.78–0.85) and ranged from 0.73 (95% CI 0.6–0.86) to 0.93 (95% CI 0.89–0.97) across the sites.

Conclusion

VSSS–EU is a reliable instrument for measuring service satisfaction in people with schizophrenia, for use in comparative cross-national research projects and in routine clinical practice in mental health services across Europe.

Patient satisfaction is an important variable in the evaluation of psychiatric services and it complements the measurements of other outcome variables. It has been suggested (Kalman, 1983) that satisfaction is strictly linked to the effectiveness of the care provided, whereas dissatisfaction is frequently the reason behind patients' discontinuing psychiatric care (Ware et al, 1978; Hansen et al, 1992). Ensuring high levels of patient satisfaction is, therefore, an essential aim for any mental health service, and its measurement constitutes a valid and important aspect of service planning and evaluation (Donabedian, 1966), to the extent that in many countries providers of health care are increasingly required to monitor levels of satisfaction among patients.

Unfortunately, there is a lack of knowledge about the extent to which patients with psychosis are satisfied with services. This seems to be due both to prejudice against such patients, believing them to be incapable of meaningfully judging the care they receive, and to methodological problems of measurement, mainly related to the difficulty of providing instruments which are acceptable to these patients.

Research in the area of satisfaction with psychiatric services has been hampered by the widespread use of many non-standardised methods, so that direct comparison between studies is usually impossible. Most studies have used instruments with few or no data regarding their validity or reliability, and investigators have frequently designed their own instruments for specific studies. As a result, findings are not generalisable (Ruggeri, 1994). In addition, although satisfaction has been demonstrated to be a multi-dimensional concept (Ware et al, 1978), instruments have often been limited to a few broad items which only enquire about one or two dimensions of mental health care. Thus not only may they fail to detect any dissatisfaction - they are inherently unable to detect the reasons for dissatisfaction.

The Verona Service Satisfaction Scale (VSSS) (Ruggeri & Dall'Agnola, 1993) is a questionnaire meant to fill this gap: it is a validated, multi-dimensional scale which measures the satisfaction of patients with mental health services. First, an 82-item version was developed: it consisted of a set of 37 items cross-setting for health services and a set of 45 items specific for mental health services. The former group of items involves aspects meant a priori to be relevant across a broad array of both medical and psychiatric settings, and was derived from the Service Satisfaction Scale (SSS-30) (Greenfield & Attkinson, 1989; Attkinson & Greenfield, 1994; Ruggeri & Greenfield, 1995). The latter group of items involves aspects which are specifically relevant in mental health settings, particularly in community-based services, such as social skills and types of intervention (e.g. admissions, psychotherapy, rehabilitation) and has been developed for the purpose by the authors of the VSSS. VSSS-82, in its versions for patients and relatives, has then been tested for acceptability, content validity, sensitivity and test-retest reliability (Ruggeri & Dall'Agnola, 1993; Ruggeri et al, 1994) in 75 patients and 75 relatives. Finally, a factor analysis was performed (Ruggeri et al, 1996). The combination of results obtained in the validation study and factor analysis gave rise to the Intermediate (VSSS-54) and the Short (VSSS-32) versions, two reliable instruments that can be easily used in everyday clinical settings. So far, the VSSS has been translated into various languages and used in studies performed in many sites in the world (Cozza et al, 1997; Parkman et al, 1997; Leese et al, 1998; Boardman et al, 1999; Clarkson et al, 1999; Henderson et al, 1999; Merinder et al, 1999).

The aim of this paper is to describe the development, translation, cultural validation and reliability of a new European Version of the VSSS (VSSS-EU), for use in multi-site international comparative studies.

METHOD

EPSILON Study

The present study is part of a larger research project, the EPSILON (European Psychiatric Services: Inputs Linked to Outcome Domains and Needs) Study, a comparative cross-national cross-sectional study of the characteristics, needs, quality of life, patterns of care, associated costs and satisfaction levels of people with schizophrenia in five European countries (Becker et al, 1999) which aims to :

1. (a) produce standardised versions of five instruments in key areas of mental health service research in five European languages (Danish, Dutch, English, Italian and Spanish), each of which was converted from the original into the other four languages;

2. (b) obtain and compare data from five regions in different European countries about social and clinical variables, characteristics of mental health care and its costs; and

3. (c) test both instrument and cross-instrument hypotheses.

Five instruments were converted for use from their original language into the other study languages by: (a) accurate translation and back-translation into/from the other languages, (b) checks of cross-cultural applicability using focus groups, and (c) assessment of instrument reliability. Fuller details on the translation and cross-cultural adaptation process are given in this supplement by Knudsen et al (2000). The five instruments were the Camberwell Assessment of Need (CAN) (Phelan et al, 1995), Client Service Receipt Inventory (CSRI) (Beecham & Knapp, 1992), Involvement Evaluation Questionnaire (IEQ) (Schene et al, 1998), Lancashire Quality of Life Profile (LQoLP) (Oliver, 1991; Oliver et al, 1996) and Verona Service Satisfaction Scale (VSSS) (Ruggeri & Dall'Agnola, 1993).

The VSSS reliability results are presented in this paper, while the results for the other scales are given in other papers in this supplement (Chisholm et al, 2000; Gaite et al, 2000; McCrone et al, 2000; van Wijngaarden et al, 2000).

Study sites

Six partners in five European countries (The Netherlands, Denmark, England, Spain and Italy) joined forces. The teams are located in Amsterdam, Copenhagen, London (Centre for Economics in Mental Health, and Section of Community Psychiatry), Santander and Verona. The criteria used to identify study centres and the key summary characteristics of each study site are given in Becker et al (1999).

Case identification

Cases included in the study were adults aged 18-65 inclusive with an ICD-10 diagnosis of any of F20 to F25. Administrative prevalence samples of people with these diagnoses were identified either from Psychiatric Case Registers (in Copenhagen and in Verona) or from the case-loads of local specialist mental health services (in-patients, out-patients and community). Patients needed to have been in contact with mental health services during the 3-month period preceding the start of the study. Thus, an administrative prevalence sample of people with schizophrenia in contact with mental health services was used in each site as the sampling frame. Cases identified were diagnosed using the item group checklist (IGC) of the Schedule for Clinical Assessment in Neuropsychiatry (SCAN) (World Health Organization, 1992). Only patients with an ICD-10 F20 research diagnosis were included in the study.

The exclusion criteria were: current residence in prison, secure residential services or hostels for long-term patients; co-existing learning disability (mental retardation), primary dementia or other severe organic disorder; and extended in-patient treatment episodes longer than one year. The numbers of patients finally included in the study varied from 52 to 107 between the five sites, with a total of 404.

Verona Service Satisfaction Scale - European Version (VSSS-EU)

The VSSS-EU was developed from the Italian VSSS-54 version, which was first translated into Danish, Dutch, English and Spanish by professional translators. The resulting four translations were then backtranslated into Italian. The back-translations were checked by the authors of the VSSS and compared with the original Italian version. Small discrepancies were examined and alterations made to the Italian version in order to preserve the precise meaning of each question, while still producing an understandable and acceptable translation into the various languages. Specific items were changed to adapt them to the context of each country's mental health system. When those changes occurred, the local researchers made a list of modifications. The next step was the focus group process, which discussed the content and the language of the translated instruments. In the light of the comments and recommendations, the instrument was then revised both in its original Italian version and in each of the four translations. On the basis of the focus groups' discussions, minor changes were made to the wording of the items, and a further change was made by deciding to assess separately the skills and behaviour of psychiatrists, psychologists, nurses and social workers, who in the previous version had been assessed in pairs (psychiatrists/psychologists, nurses/social workers) (see Appendix).

Conceptually, the items in the VSSS-EU cover seven dimensions: Overall Satisfaction, Professionals' Skills and Behaviour, Information, Access, Efficacy, Types of Intervention and Relative's Involvement. Items in the first five dimensions cover all areas belonging to Ware's taxonomy of satisfaction (Ware et al, 1983). The last two dimensions, on the other hand, examine domains that have not been assessed systematically in previous studies and have been specifically developed for the VSSS.

Each conceptual dimension of the VSSS-EU consists of a certain number of items that cover various aspects of satisfaction with services (see Appendix for a detailed description of each item) :

1. (a) the Overall Satisfaction dimension consists of three items which cover general aspects of satisfaction with psychiatric services;

2. (b) the Professionals' Skills and Behaviour dimension consists of 24 items which cover various aspects of satisfaction with the professionals' behaviour, such as technical skills, interpersonal skills, cooperation between service providers, respect of patients' rights, etc.; psychiatrists, psychologists, nurses and social workers are assessed in separate items;

3. (c) the Information dimension consists of three items which cover aspects related to satisfaction with information on services, disorders and therapies;

4. (d) the Access dimension consists of two items which cover aspects related to satisfaction with service location, physical layout and costs;

5. (e) the Efficacy dimension consists of eight items which cover aspects related to satisfaction with overall efficacy of the service, and service efficacy on specific aspects such as symptoms, social skills and family relationships;

6. (f) the Types of Intervention dimension consists of 17 items which cover various aspects of satisfaction with care, such as drugs prescription, response to emergency, psychotherapy, rehabilitation, domiciliary care, admissions, housing, recreational activities, work, benefits, etc.

7. (g) the Relative's Involvement dimension consists of six items which cover various aspects of the patient's satisfaction with help given to his/her closest relative, such as listening, understanding, advice, information, help coping with the patient's problems, etc.

The VSSS-EU is designed for self-administration and can be completed without prior training. In cases of cognitive deficit, severe psychopathology or low level of literacy, a research worker may assist the patient and/or the relative by reading through the items with them. Special care must be taken to guarantee confidentiality and anonymity and, in the case of assisted administration, to stress the independence of the research worker from the clinical team. Administering the questionnaire takes 20-30 minutes.

In the VSSS-EU, subjects are asked to express their overall feeling about their experience of the mental health service they have been using in the past year.

For items 1-40, satisfaction ratings are on a 5-point Likert scale (1=terrible, 2=mostly dissatisfactory, 3=mixed, 4=mostly satisfactory, 5=excellent), presented with alternate directionality to reduce stereotypic response. Items 41-54 consist of three questions: first, the subject is asked if he/she has received the specific intervention (Question A: “ Did you receive the intervention x in the last year?”). If the answer is ‘yes’, he/she is asked his/her satisfaction on a 5-point Likert scale (Question B). If the answer is ‘no’, he/she is asked Question C: “Do you think you would have liked to receive intervention x?” (6=no, 7=don't know, 8=yes). These questions permit the estimation of the subject's degree of satisfaction both with the interventions provided and with the professional's decision not to provide an intervention (if that was the case). The latter may be considered a measure of underprovision of care, from the patient's point of view.

Data management

SPSS (Version 6.1 and above) was used for data entry. Data consistency and homogeneity were ensured by the coordinating centre (London) preparing the SPSS templates for use at all participating sites; this ensured consistent data structures across sites. Subsequent cross-instrument merging and analysis was possible as the same patient/carer identifiers were used throughout the study.

Reliability assessment procedures

Reliability testing in the EPSILON project has been conducted on several levels, depending on the nature of the instruments involved and the way they were administered (interviews v. questionnaires). Details on the general methodology of the reliability study are given elsewhere in this supplement (Schene et al, 2000).

As far as the VSSS-EU is concerned, the reliability tests have been performed: (a) on the VSSS-EU total mean score; (b) on the mean scores of each dimension; and (c) item by item, both for the pooled sample and across the five sites. Three kinds of reliability tests have been used: Cronbach's α (Cronbach, 1951), to check the internal consistency of the whole questionnaire and the different dimensions; the intraclass correlation coefficient (ICC) (Bartko & Carpenter, 1976), to evaluate test-retest reliability of the VSSS-EU total mean score and dimension mean scores; and Cohen's weighted κ (Cohen, 1968), to evaluate test-retest reliability of single VSSS-EU items. Additional statistics estimated were the standard errors of measurement, which were obtained from the analysis of variance used to estimate the intraclass correlations.

Statistical analysis was performed using SPSS for Windows, release 7.5 (Norusis, 1997), the Amsterdam α -testing program, ALPHA.EXE based on Feldt et al (1987) and EXCEL for tests of the homogeneity of ICCs.

RESULTS

A total sample of 289 subjects (49 in Amsterdam, 43 in Copenhagen, 81 in London, 50 in Santander and 66 in Verona) completed both test and retest VSSS-EU administration. Retesting was done after a mean of 10.4 days (s.d. 6.03) in the pooled sample, 10.7 days (s.d. 7.45) in Amsterdam, 9.3 days (s.d. 3.57) in Copenhagen, 9.4 days (s.d. 3.93) in London, 6.1 days (s.d. 2.82) in Santander, 15.4 days (s.d. 6.70) in Verona. The shortest and longest test-retest intervals were in Santander and Verona, respectively (Kruskal-Wallis test, P<0.001).

Internal consistency

Table 1 shows the mean scores for each VSSS-EU dimension in the various EPSILON sites and the tests of homogeneity of variance (Levene test) and of means. The latter tests are not discussed in the present study, but will be investigated in a future paper. Assuming that the standard error of measurement is the same in all countries, the reliability coefficient depends on the variance in each sample. Conversely, where standard errors of measurement do differ, sample variables will be affected, since they are composed both of variance in true scores and variation due to measurement error. Therefore, when computing reliability coefficients and testing differences between samples, differences between sites in terms of both sample variance and standard errors of measurement should be considered.

Table 1 The Verona Service Satisfaction Scale - European Version (VSSS-EU) dimensions and total means in the pooled sample and by site

Dimension	Pooled n=399		Amsterdam n=58		Copenhagen n=51		London n=83		Santander n=100		Verona n=107		Test of equality of means (P-value)	Test of equality of s.d. (P-value)
	mean	s.d.	mean	s.d.	mean	s.d.	mean	s.d.	mean	s.d.	mean	s.d.
Overall satisfaction	3.83	0.79	3.90	0.80	4.04	0.79	3.45	0.67	3.79	0.84	4.01	0.72	0.05	0.27
Professionals' skills and behaviour	3.88	0.57	3.97	0.51	4.13	0.56	3.46	0.40	3.94	0.53	4.00	0.60	<0.01	0.03
Information	3.39	0.93	3.66	0.75	3.69	0.86	3.26	0.65	2.93	1.09	3.64	0.88	<0.01	<0.01
Access	3.83	0.73	3.63	0.73	4.19	0.71	3.97	0.67	3.84	0.67	3.64	0.79	<0.01	0.35
Efficacy	3.56	0.74	3.69	0.69	3.80	0.76	3.22	0.55	3.41	0.77	3.81	0.74	<0.01	0.03
Types of intervention	3.64	0.42	3.65	0.47	3.72	0.42	3.68	0.24	3.42	0.41	3.75	0.46	<0.01	<0.01
Relative's involvement	3.39	0.96	3.57	0.92	3.32	1.21	2.91	0.67	3.39	0.96	3.75	0.91	<0.01	<0.01
Total score	3.70	0.50	3.79	0.46	3.89	0.48	3.45	0.34	3.59	0.51	3.86	0.54	<0.01	0.01

The variability of the scores differed across the sites, both in the VSSS-EU total mean score and in all VSSS-EU dimensions, with the exception of Overall Satisfaction and Access; therefore the lack of homogeneity across the sites for the total mean scores and most of the dimension scores may account to some extent for the degree of variability in the reliability coefficients across the sites.

Table 2 shows the α coefficients and the test for the equality of α across the sites. Alpha coefficients indicate the degree to which items exhibit a positive correlation (internal consistency above 0.7 is considered adequate; Bech et al, 1993).

Table 2 Internal consistency of the Verona Service Satisfaction Scale - European Version (VSSS-EU): α coefficients (95% CI) in the pooled sample and by site

		Pooled		Amsterdam		Copenhagen		London		Santander		Verona
	No. of items	n	α	n	α	n	α	n	α	n	α	n	α	Test of equality of α (P-value)
Overall satisfaction	3	384	0.80	57	0.80	45	0.82	83	0.77	97	0.83	102	0.73	0.67
			(0.77-0.83)		(0.71-0.87)		(0.73-0.88)		(0.69-0.83)		(0.78-0.88)		(0.65-0.80)
Professionals' skills and behaviour1	16	275	0.91	38	0.90	24	0.89	75	0.85	72	0.90	66	0.91	0.58
			(0.89-0.92)		(0.89-0.95)		(0.82-0.95)		(0.80-0.89)		(0.86-0.93)		(0.88-0.94)
Information	3	342	0.72	51	0.67	40	0.60	82	0.63	72	0.79	97	0.72	0.39
			(0.68-0.76)		(0.51-0.78)		(0.37-0.75)		(0.49-0.73)		(0.71-0.85)		(0.63-0.79)
Access	2	386	0.06	53	0.16	49	0.96	83	0.08	99	0.77	102	0.29	<0.01
			(-0.11 to 0.20)		(-0.33 to 0.47)		(0.93-0.97)		(-0.33 to 0.36)		(0.68-0.83)		(0.02-0.49)
Efficacy	8	254	0.87	37	0.83	28	0.82	58	0.77	69	0.89	62	0.89	0.11
			(0.84-0.87)		(0.75-0.89)		(0.73-0.90)		(0.68-0.84)		(0.85-0.92)		(0.85-0.92)
Types of intervention	17	115	0.73	19	0.81	8	0.61	52	0.62	4	-2	32	0.77	0.71
			(0.67-0.79)		(0.69-0.90)		(0.17-0.88)		(0.48-0.74)				(0.66-0.86)
Relative's involvement1	5	298	0.89	40	0.85	26	0.93	67	0.81	92	0.91	73	0.88	0.04
			(0.87-0.91)		(0.77-0.91)		(0.88-0.97)		(0.72-0.87)		(0.87-0.94)		(0.82-0.91)
Total score	54	74	0.96	16	0.95	4	—2	33	0.93	2	—2	19	0.96	1.00
			(0.94-0.97)		(0.91-0.98)				(0.90-0.96)				(0.93-0.98)

It should first be noted that the high degree of variability in the number of cases is usually not due to missing values but to the fact that some items (i.e. items on the performances of the different professionals assessed in the VSSS, items on the relatives, items on response of the service to emergencies during the night or at weekends) are not applicable to all patients. For this reason, α values in the VSSS-EU total mean scores have been computed for a small number of cases only. However, they were always over 0.90 and were similar across the sites, with the exception of Relative's Involvement. In the VSSS-EU dimensions, α coefficients ranged from 0.60 (Information dimension, in Copenhagen) to 0.93 (Relative's Involvement dimension, in Copenhagen) and did not differ significantly across the sites. The dimension Access, which consists of just two items (Costs of Service and Physical Layout) measuring different constructs, is a special case. In this dimension, α varied greatly, ranging from 0.08 (London) to 0.96 (Copenhagen), and differed significantly across the sites. Dimensions constituted by a higher number of items are expected to have higher α values. This was true for the dimension Professionals' Skills and Behaviour, but not for the dimension Types of Intervention, which is not expected to have high internal consistency, due to the wide range of different interventions explored by the questionnaire.

On the whole, α values of the pooled sample were good, with the above-mentioned exception of the Access dimension, and ranged from 0.72 (Information) to 0.91 (Professionals' Skills and Behaviour).

Stability

The test-retest reliability was studied by considering the ratings both in each dimension (ICC) and item by item (weighted κ).

Table 3 shows the test-retest reliability for the VSSS-EU total mean score and VSSS-EU dimension scores, both in the pooled sample and at each EPSILON site. ICCs and the standard errors of measurements are reported.

Table 3 Test-retest reliability of Verona Service Satisfaction Scale - European Version (VSSS-EU) dimensions and total means in the pooled sample and by site

Sub-scale	Pooled		Amsterdam		Copenhagen		London		Santander		Verona		Test of equality of ICCs
	n=289		n=49		n=43		n=81		n=50		n=66		(P-value)
	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m
Overall satisfaction	0.66	0.44	0.50	0.51	0.67	0.46	0.76	0.33	0.80	0.37	0.52	0.52	0.01
Professional's skills and behaviour	0.76	0.25	0.66	0.29	0.80	0.24	0.78	0.19	0.86	0.19	0.72	0.31	0.06
Information	0.75	0.14	0.49	0.52	0.75	0.42	0.66	0.37	0.84	0.44	0.76	0.41	<0.01
Access	0.56	0.47	0.56	0.48	0.51	0.49	0.73	0.33	0.43	0.51	0.51	0.55	0.05
Efficacy	0.75	0.34	0.70	0.36	0.76	0.35	0.77	0.26	0.90	0.23	0.59	0.45	<0.01
Types of intervention	0.69	0.22	0.56	0.32	0.64	0.25	0.82	0.09	0.90	0.13	0.66	0.26	<0.01
Relative's involvement	0.78	0.43	0.72	0.47	0.82	0.53	0.70	0.38	0.89	0.32	0.67	0.48	<0.01
Total score	0.82	0.20	0.73	0.24	0.85	0.19	0.82	0.15	0.93	0.13	0.76	0.25	<0.01

The VSSS-EU total mean score in the pooled sample shows a high degreee of reliability (0.82; 95% CI 0.78-0.85), although there are some differences between the sites, with all sites above 0.70 and three sites (Copenhagen, London and Santander) above 0.80. Each of the VSSS-EU dimension mean scores in the pooled sample had a degree of reliability ranging from 0.56 to 0.78. The reliability of the VSSS-EU dimension mean scores across the sites was over 0.50 in all cases, with the exception of the Access dimension in Santander (0.43) and the Information dimension in Amsterdam (0.49). Overall, there was some degree of variability in the ICC coefficients across the sites. This could be due either to differences in measurement errors between different sites, or to lack of homogeneity in the samples. The lower reliabilities tend to be associated with higher standard errors of measurement, and therefore lack of homogeneity in the samples is unlikely to be the explanation for differences in reliability; rather, the performance of the instrument itself differs.

Table 4 shows Cohen's weighted κ for both the pooled sample and the five EPSILON sites, calculated in each VSSS-EU item by taking the disagreements' weight to be equal to the square of the distances. According to Landis & Koch (1977), a κ coefficient of 0.2-0.4 indicates fair agreement, 0.4-0.6 indicates moderate agreement, 0.6-0.8 indicates a substantial agreement and 0.8-1.0 indicates almost perfect agreement, although alternative schemes are possible (see Schene et al, 2000, this supplement).

Table 4 Test-retest reliability of individual items of the Verona Service Satisfaction Scale - European Version (VSSS-EU) for the pooled sample and in each study site. Number (percentage) of items in Cohen's weighted κ bands suggested by Landis & Koch (1977)

Weighted κ	Strength of agreement	Pooled n=289	Amsterdam n=49	Copenhagen n=43	London n=81	Santander n=50	Verona n=66
		n (%)	n (%)	n (%)	n (%)	n (%)	n (%)
0.81-1.0	Almost perfect			2(3)	14 (22)	10 (16)
0.61-0.80	Substantial	31 (49)	8 (13)	23 (36)	36 (57)	33 (52)	8 (12.7)
0.41-0.60	Moderate	32 (51)	37 (59)	25 (40)	11 (17)	16 (25)	26 (41.2)
0.21-0.40	Fair		18 (29)	10 (16)	2 (3)	4 (6)	22 (34.9)
0-0.20	Slight			3 (5)			7 (11.1)

In the pooled sample, all items generated κ coefficients from 0.4 to 0.8, indicating a moderate-substantial agreement. In the various EPSILON sites the percentage of items with κ coefficients exceeding 0.4 (generally accepted as the minimum value) ranged from 54% (Verona) to 97% (London).

A paired sample t-test on the difference between test and retest for the VSSS-EU total mean scores and dimension sub-scores revealed no significant differences, both pooled across sites and at individual sites (with the exception of Access in London and Santander, where the retest values were respectively higher and lower than the time 1 values, P <0.001), thus showing no overall tendency for patients to respond more or less favourably after an interval of time ranging from 1 to 2 weeks.

DISCUSSION

This paper describes the development of the VSSS-EU, a modified version of the VSSS-54, for measuring satisfaction with mental health services in people with schizophrenia and intended to be used in international collaborative studies across Europe. It also describes the assessment of the psychometric properties of the new instrument, with regard to its internal consistency and stability. The investigation of the psychometric properties of the measures of satisfaction is an outstanding issue in research into satisfaction with mental health services. Although there is little doubt that patient satisfaction is an important aspect in the assessment of the quality and outcome of community-based mental health programmes, research in the field suffers from various methodological limitations regarding study design, the construction of the instrument and the lack of attention to its psychometric properties. According to many authors, the lack of attention to methodological aspects and the lack of confidentiality strongly influence the levels of reported satisfaction (Larsen et al, 1979; Weinstein, 1979; Keppler-Seid et al, 1980; Ruggeri, 1994).

In the past few years, the need for a measure which could assess satisfaction with mental health services in a manner which would be correct from a methodological point of view led to the development of the VSSS (Ruggeri & Dall'Agnola, 1993). One of the most important aims during the development of the original version of the VSSS was the avoidance of methodological biases: data previously obtained on content validity, test-retest reliability and factor analysis showed that the original version measures satisfaction in a sensitive, valid and reliable way (Ruggeri & Dall'Agnola, 1993; Ruggeri et al, 1994, 1996). The present paper demonstrates that the newly developed VSSS-EU also has good psychometric properties.

The scale has an excellent overall Cronbach's α (0.96), which confirms its adequacy from the standpoint of internal consistency if it is used as a global satisfaction measure. All dimensions except one (Access) showed good internal consistency, as indicated by coefficients of Cronbach's α ranging from 0.72 for Information to 0.91 for Professionals' Skills and Behaviour. The dimension with the lowest consistency (Access), made up of two items (Cost of Service and Physical Layout), was not expected to show high interrelatedness, because theoretically these two items are quite independent of one another.

The test-retest data provided encouraging results. The stability of the scale was satisfactory both at dimensional level and item by item. Although there was some evidence of differing reliability between sites, which was associated with differing standard errors of measurement rather than differing homogeneity of the samples, all the individual site reliability coefficients for the total score were above 0.7. Interestingly, the lowest test-retest reliability was found in Verona, thus indicating that the translation of the scale did not affect its acceptability to patients. The overall pooled test-retest reliability for the total score was excellent at 0.82 (95% CI 0.78-0.85). Furthermore, there was no evidence of any tendency for the measurements to change systematically over time, either in the pooled sample or across the sites (with the above-mentioned exception of Access).

In conclusion, the analysis of the data presented here demonstrates that the VSSS-EU has good psychometric properties and suggests that it is a reliable instrument for measuring satisfaction with mental health services in people with schizophrenia, for use in comparative cross-national research projects and in routine clinical practice in mental health services across Europe.

APPENDIX

(The number of items in each dimension is shown in parentheses.)

Overall Satisfaction (3)

• 11. Amount of help received

• 20. Kind of services

• 21. Overall satisfaction

Professionals' Skills and Behaviour (16)

• 3a. Professionalism and competence of the psychiatrists

• 3b. Professionalism and competence of the psychologists

• 16a. Thoroughness of psychiatrists

• 16b. Thoroughness of psychologists

• 22a. Professional competence of the nursing staff

• 22b. Professional competence of social workers

• 35a. Thoroughness of nurses

• 35b. Thoroughness of social workers

• 6a. Psychiatrists' manner

• 6b. Psychologists' manner

• 5a. Ability of psychiatrists to listen and understand problems

• 5b. Ability of psychologists to listen and understand problems

• 25a. Nurses' manner

• 25b. Social workers' manner

• 28. Nurses' knowledge of patient's medical history

• 37a. Ability of nurses to listen and to understand problems

• 37b. Ability of social workers to listen and to understand problems

• 2. Behaviour and manners of reception or secretarial staff

• 33. Instruction on what to do between visits

• 18. Cooperation between service providers

• 17. Referring to general practitioner or other specialists

• 40. Opportunity of being followed up by the same professionals

• 10. Confidentiality and respect for patient's rights

• 7. Punctuality of the professionals when patient comes for an appointment

Information (3)

• 12. Explanation of specific procedures and approaches used

• 29. Information on diagnosis and prognosis

• 19. Publicity or information on mental health services which are offered

Access (2)

• 4. The appearance, comfort level and physical layout

• 8. Costs of the service

Efficacy (8)

• 9. Effectiveness of the service in attaining well-being and preventing relapses

• 1. Effectiveness of the service in helping patient deal with problems

• 24. Effectiveness of the service in helping patient improve knowledge and understanding of his/her problems

• 13. Effectiveness of the service in helping patient to relieve symptoms

• 26. Effectiveness of the service in improving the relationship between patient and relative

• 34. Effectiveness of the service in helping patient improve capacity to look after her/himself

• 31. Effectiveness of the service in helping patient establish good relationships outside family environment

• 38. Effectiveness of the service in helping patient improve abilities to work

Types of Intervention (17)

• 14. Response of the service to crisis or urgent needs during office hours

• 15. Response of the service to emergencies during the night, weekends and Bank Holidays

• 39. Help received for unexpected outcomes, discomfort or side-effects of medication

• 41. Medication prescription

• 42. Individual rehabilitation

• 43. Individual psychotherapy

• 44. Compulsory treatment in hospital

• 45. Family therapy

• 46. Living in sheltered accommodation

• 47. Participation in the recreational activities organised by mental health services

• 48. Group psychotherapy

• 49. Sheltered work

• 50. Informal admission to hospital

• 51. Practical help by the sevice at home

• 52. Help in obtaining welfare benefits or exemptions

• 53. Help to find open employment

• 54. Help from the service to join in recreational activities separate from the mental health services

Relative's Involvement (5)

• 30a. Ability of psychiatrists to listen and understand the concerns and the opinions relative may have about patient

• 30b. Ability of psychologists to listen and understand the concerns and the opinions relative may have about patient

• 23. Recommendations about how relative could help

• 32. Information to relative about diagnosis and prognosis

• 36. Effectiveness of the service in helping relative to deal better with patient's problems

• 27. Effectiveness of the service in helping relative improve his/her understanding of patient's problems