Systematic review of effectiveness and satisfaction evaluation in child and adolescent mental health services in Ireland

Abstract

Objectives:

Increasing numbers of youth experience mental illness, and also require and benefit from specialist child and adolescent mental health services (CAMHS). Worldwide, such services are underfunded and under-resourced, and services in Ireland are no different. It is vital that existing services are regularly reviewed for both efficacy and acceptability. Our objective was to review published studies evaluating service user satisfaction with CAMHS in Ireland and CAMHS therapeutic efficacy.

Methods:

MEDLINE, PsycINFO and CINAHL databases were systematically searched. Studies were included if they reported on service user satisfaction or an evaluation of CAMHS in Ireland.

Results:

From an initial 125 articles identified, 15 studies meet the inclusion criteria: four reporting on overall CAMHS satisfaction, three on satisfaction where a specific diagnosis was present, while eight evaluated various interventions offered. Whilst most service users perceived services to be satisfactory, important issues relating to accessibility were present. Evidence of efficacy was present for a small number of interventions, but studies were limited by methodological issues.

Conclusions:

There is a dearth of studies evaluating CAMHS in Ireland. The extant literature suggests a positive experience once accessed, but long waiting times and poor collaboration are seen to limit services users’ experience. More robust methodologically sound studies are urgently required. Given the expected increased demand linked to the current COVID-19 pandemic, coupled with the resultant compromised financial position, it is essential that scant resources are appropriately directed.

Introduction

The majority of mental health problems begin in the first two decades of life, with high rates of persistence into adult life (Kessler et al. 2005). Longitudinal data indicate that these childhood-onset mental health problems do not remit spontaneously (Meltzer et al. 2003), and that there is a greater cost associated with childhood-onset illness than adult-onset (Suhrcke et al. 2008).

Up to 20% of children experience mental health problems, and prevalence rates have been increasing over the years (Bor et al. 2014). It is anticipated that the current COVID-19 pandemic will bring additional demand on services (Gavin et al. 2020). Furthermore, frontline workers are recognised as high risk groups for psycho-social stress and occupational burnout, leading to resourcing difficulties which will further compromise the already stretched services (Murphy, 2020).

In this context, there has been increasing pressure on health services (including child psychiatry services) to provide efficient and timely care, maximising patient flow while maintaining high levels of both satisfaction and effectiveness in achieving clinical outcomes (Fuggle et al. 2016). Satisfaction has been increasingly prioritised as a key means of assessing service performance, with standard validated rating scales developed to measure how satisfied children and parents are with the service (Solberg et al. 2015). To quantify effectiveness, services can examine success in attainment of patient/family goals as well as external measures such as clinician-rated improvement and outcome scales (Fuggle et al. 2016).

Therefore, services are likely to face a tricky balancing act between the potentially competing objectives of maximising efficiency while also maximising both satisfaction and effectiveness. Prioritising efficiency can risk adverse effects on satisfaction and effectiveness, for example, if some patients receive less time and clinical attention as a consequence of striving to maximise patient flow. Perhaps less immediately obvious is that patient satisfaction, family satisfaction and treatment effectiveness are far from being perfectly correlated with each other (Solberg et al. 2015) and thus high performance in one of these areas does not guarantee high performance in the others. Achieving such delicate balances would be a laudable feat even in the most abundantly resourced health service. Unfortunately, specialist child and adolescent mental health services (often referred to as CAMHS) in Ireland have long experienced significant under-resourcing: demand considerably outstrips service availability, with serious pressures on services and their staff as a result (McNicholas, 2018; McNicholas et al. 2020). Since services are subject to such significant constraints, it is important to ensure that the finite and scarce resources available are used optimally, including evaluation of whether the assessment and intervention provided lead to satisfaction and positive effects on clinical outcomes.

This systematic review was conducted to review studies carried out pre-COVID-19 that evaluate satisfaction with, and/or therapeutic effectiveness of, specialist CAMHS in Ireland. The aim was to identify any relevant published peer-reviewed study on this topic regardless of study design, interventions or outcomes used.

Methods

Searches were performed to find relevant literature in relation to children and adolescents in Ireland, attending specialist mental health services in Ireland, and evaluation of satisfaction with and effectiveness of attendance. All types of study design were included. The pre-specified protocol for the search and review process is available from the authors on request.

The databases searched were MEDLINE, PsycINFO and CINAHL (Cumulative Index to Nursing and Allied Health Literature). Searches were conducted in April 2020 based on article titles, abstracts and subjects. The final search was carried out on all databases in April 2020. Supplementary Material Appendix 1 outlines the Medical Subject Heading (MeSH) terms used in the search strategy. In addition to database searches, given that some relevant publications in Ireland have only recently been indexed in MEDLINE, the last 10 years of issues from relevant Irish peer-reviewed journals were manually screened for further articles (Irish Journal of Psychological Medicine, Irish Journal of Medical Science). After identification of relevant articles, the references were hand-searched for additional relevant publications. The co-author performed double-screening of articles identified for inclusion. The procedure outlined by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was followed in the reporting of the results (Moher et al. 2009).

Inclusion and exclusion criteria are shown in Table 1. The services included for the study are the specialist child and adolescent mental health services, often referred to as CAMHS. In Ireland, these are services whose remit is the treatment of moderate to severe mental illness in persons under the age of 18, and clinical leadership is provided by child and adolescent psychiatry doctors. There is often also a multidisciplinary team attached to the service, although this is not present in some settings, for example, liaison child and adolescent psychiatry departments may not have a multidisciplinary team. The MeSH term ‘Child’ includes individuals from age six upwards. While child psychiatry services may treat some children under the age of six, all child psychiatry services in Ireland of which we are aware have a patient caseload with mean age well in excess of six years old. Therefore for simplicity of searching and filtering literature, we decided to include only services where the mean age is at least six as this does not exclude any specialist child mental health services in Ireland to our knowledge.

Table 1. Inclusion and exclusion criteria

Data extraction

On completion of the initial search, the results were screened and duplicates removed. The exclusion criteria were applied to the titles and abstracts of articles, removing any that met these criteria. Remaining articles were reviewed to confirm whether they met the inclusion criteria. Fig 1 shows the flow of articles through this process as per the PRISMA flow diagram.

Fig. 1 PRISMA flow diagram of included articles.

Final data extraction was based on the articles included at the end of this process. For each article, the following data were collated in spreadsheet form: service setting, sample size, participant status (child, parent or staff), study design, measures used, results and any funding or conflict of interest.

Methodological quality

The Mixed Methods Appraisal Tool (MMAT), a validated instrument for systematic reviews, was used to evaluate the methodological quality of studies identified (Hong et al. 2018, 2019).

Results

A total of 15 studies were identified from the search process (Fig. 1). Table 2 provides an overview of the types of study identified, while Table 3 provides details of the individual studies, which are further described in the subsequent paragraphs. The total number of studies identified was limited to 15. There were four studies examining overall service satisfaction, of which two were drawing on the same sample. Three studies examined satisfaction with services for specific disorders (Attention Deficit Hyperactivity Disorder (ADHD)) or eating disorder). Eight examined satisfaction and feasibility of various treatments. All studies involved outpatient CAMHS. No research relating to inpatient or liaison child psychiatry services was identified.

Table 2. Summary of study types and overall finding

Table 3. Included studies and key characteristics

Satisfaction with the service as a whole (N = 4)

Of the 15 articles discovered by the search process, there were 4 retrospective studies which addressed satisfaction with the service as a whole. Three were well-powered and had both child and parent input: two of the three studies (Coyne et al. 2015; McNicholas et al. 2016) used the same multi-site sample while the third had an independent single-site dataset (Church, 2012). The fourth study (McNicholas et al. 2010) involved a sample of parents who attended psychoeducational information evenings, where one element of the feedback related to parents’ satisfaction with the clinical service their child was receiving. One of the four studies (Coyne et al. 2015) was qualitative in design, while the remaining three were quantitative. The mean sample size of the quantitative studies was 208.33 (range 99–426) with an aggregated total of 625 participants across the three studies.

The study by McNicholas and colleagues (2010) included one yes/no question on parents’ overall satisfaction with the specialist mental health service received: this revealed a somewhat lower percentage of satisfaction than the other studies but still a majority reporting satisfaction (60% answered yes).

Church (2012) and McNicholas et al. (2016) both administered the Client Satisfaction Questionnaire (CSQ) to parents and adolescents attending services, although Church reported only some of the results. The CSQ involves a Likert scale with the possible answers for each question ranging from 1 (low satisfaction) to 4 (high satisfaction). Both studies reported high levels of overall satisfaction, parents’ mean satisfaction being slightly higher (mean = 3.6) than among adolescents (mean = 3.2) in the single-site study by Church (2012) with the reverse finding in the multi-site study by McNicholas et al. (2016), where 86% of parents and 92% of adolescents gave a rating above 3. Seventy-three percent of parents and 84% of adolescents felt ‘services had met their needs’ (McNicholas et al. 2016).

Qualitative data from both the Church study (2010) (N = 100, theme from answers to open-ended questions) and the multi-site study by Coyne and colleagues (2015) (N = 27, thematic analysis of interviews and focus groups) converged on similar themes. Staff attitude, positive atmosphere and therapeutic alliance were an important positive noted by parents and adolescents in both studies. However, this was adversely impacted at times by seeing multiple different clinicians, a situation which arose especially in the context of non-consultant doctors regularly changing posts as well as turnover of other clinicians. Parents valued parental support (both from clinicians and in parent support groups) and being given their own time to discuss issues. Some parents reported that they had received and benefitted from such supports. However, there were similar reports of others who had not received such supports and felt dissatisfied as a result. Parents and adolescents placed high value on feeling informed, that their voice was heard and that they were receiving early explanations of the child’s difficulties, progress and the potential care plan. Some expressed satisfaction with this aspect of the service, although in neither study were the frequency of cases given. Positive comments were expressed about CAMHS liaising with the child’s school, and whilst all saw this as relevant, some parents and adolescents felt that there was insufficient communication with schools or that schools themselves were not sufficiently supportive.

Satisfaction with the service in relation to specific disorders (N = 3)

Three of the 15 studies collected retrospective data on satisfaction with the service in relation to a particular disorder. One single-site study examined satisfaction following the development of an ADHD-specific services and elicited perspectives on the experience of being prescribed ADHD medications (Damodaran & Sherlock, 2013) while another examined overall satisfaction with services for ADHD (Carr-Fanning & Mc Guckin, 2018). A multi-site study reported on satisfaction levels from professionals and services users of eating disorder services (McNicholas et al. 2018).

The ADHD studies involved qualitative data collection from 15 young people and 17 of their parents (Carr-Fanning & Mc Guckin, 2018) and questionnaire data on experience of services from 22 young people attending ADHD CAMHS (Damodaran & Sherlock, 2013). In both studies, participants tended to report positive experiences and benefits from medication treatment for ADHD, but frequently felt they were insufficiently informed on treatments offered and did not have sufficient opportunity to participate actively in their care planning. As with the previously discussed studies on overall satisfaction with services, Carr-Fanning and McGuckin found that parents placed a high priority on timely access to assessment and an explanation of their child’s difficulties, expressing a sense of relief when the diagnosis of ADHD was made. Many parents reported dissatisfaction due to difficulties accessing services and lengthy periods from assessment to diagnosis. Parents also emphasised the importance of, and deficiencies in, clinic–school communication and collaboration.

The study examining services for youth with eating disorders (McNicholas et al. 2018) used a mixed-methods methodology drawn from three different populations: (i) qualitative interviews with eight young people, five parents and three treating clinicians, (ii) a questionnaire eliciting health professionals’ views (N = 171) on services for eating disorders and (iii) a non-clinical sample (N = 290) from the school-going population. The first two of these three study cohorts were considered relevant to this review, giving a total of 16 participants in the qualitative arm and 171 clinicians participating in the quantitative arm. Both clinicians and service users were highly critical of available services, particularly in relation to access, perceiving many barriers to delivering and receiving effective care.

One parent reported there is no care, there is no outlet or outreach, citing unacceptable delays and lack of awareness of service provision contributing to deterioration and unacceptable clinical risk. One parent reported their GP recommending emergency department attendance or attendance at private services in lieu of CAMHS. Parents revealed feelings of isolation and helplessness, requesting services supporting the family unit as well as focussed on their child. Clinicians requested clarity regarding care pathways and the adoption of a standardised treatment protocol (McNicholas et al. 2018).

Evaluations of specific interventions (N = 8)

The remaining eight studies examined effectiveness of specific interventions offered by CAMHS with varying endpoints such as change in child anxiety levels, social/emotional behavioural difficulties, mood regulation, patient satisfaction, parental stress and health service utilisation. Given heterogeneity of studies, it was difficult to usefully compare the magnitude of effectiveness across different studies.

Six studies examined efficacy of a particular clinical treatment with one study each reporting on outcomes following child art psychotherapy (Tanil et al. 2018), group cognitive-behavioural therapy for anxiety disorders (O’Brien et al. 2007), dialectical behaviour therapy (Flynn et al. 2019), group music therapy (McDonald et al. 2015), Parents Plus training programme for the parents of children with behavioural and developmental problems (Coughlin et al. 2009) and a family intervention (Wynne et al. 2016). One study examined the introduction of an ADHD-specific clinic (Damodaran et al. 2012), whilst another compared a brief consultation and advisory (BCA) approach with treatment as usual (TAU) for non-complex referrals, using a randomised controlled trial (RCT) design (McGarry et al. 2008).

Three out of the eight studies took place at a single site (O’Brien et al. 2007; Damodaran et al. 2012; McDonald et al. 2015), and five were multi-site. There were three prospective studies with before-and-after measures (McDonald et al. 2015; Wynne et al. 2016; Flynn et al. 2019), two retrospective evaluation studies (Damodaran et al. 2012; Tanil et al. 2018), one non-RCT (Coughlin et al. 2009) and two RCTs (O’Brien et al. 2007; McGarry et al. 2008).

The study types are detailed in Table 3. Six were quantitative and two (McDonald et al. 2015; Tanıl et al. 2018) used mixed-methods. Sample sizes varied across both quantitative (N = 5–199) and qualitative studies (from N = 1–5) and by reporter (clinicians, parents and young people). In one study, the number of parent respondents was unspecified (O’Brien et al. 2007). However, all eight studies suggested positive results from the various interventions offered by CAMHS, with no study highlighting a lack of benefit. The largest and most methodologically robust examined a manualised family intervention with concurrent delivery of a parent and child component, and merging two previously validated interventions – the Parents Plus Adolescent Programme’ (WTOPPAP) and the ‘Working Things Out’ adolescent programme. Outcomes were measured on a number of important child and parent domains, including family functioning, parental stress, along with child behavioural and emotional problems. The authors acknowledge the lack of comparison group as a significant weakness in terms of attributing gains made to the intervention. Outcomes however were very favourable, both in terms of reduced child-rated psychopathology and family functioning and satisfaction.

One study reported on a therapeutic intervention aimed to address the recognised difficulties and dissatisfaction with long waiting lists and risk of drop out (McGarry et al. 2008). In this study, 60 children on CAMHS waiting list were either randomly allocated to BCA care or received TAU. As with the study by Wynne et al. (2016), outcome measures included a broad range of child and family domains taken at baseline and 3 and 6 months post-treatment. Both interventions were found to be effective, but the BCA model was found to be superior to TAU, which persisted 6 months post-baseline, despite no further treatment being offered.

MMAT methodological quality assessment

In order to assess the quality of studies, the MMAT was applied. This is a frequently used tool for literature analysis and systematic reviews where a variety of study designs need to be appraised: it involves a series of yes/no questions that contribute to a final score, with a higher score considered to indicate higher quality (Hong et al. 2018, 2019). In some of the studies identified, only one aspect of the study was relevant to service evaluation, and so in these cases our quality assessment refers only to that particular aspect of the study rather than to the quality of the study as a whole. Eight of the 15 studies (53%) were rated at least 4 out of 5 on MMAT criteria (Supplementary Material, Appendix 2).

Discussion

Child and adolescent specialist mental health services in Ireland are under-resourced with poor or idiosyncratic availability of out-of-hours access, other than via generic emergency departments (McNicholas, 2018). Given these significant resource limitations, it is essential to conduct studies to ensure adequate planning and evaluation of services to allow optimum allocation of scarce resources. However, Health Service Executive and Department of Health planners have limited data on which to build their plans. This review has revealed only a limited number of such evaluations in the published peer-reviewed literature. As described in our results, the most common aim of the published evaluation research was to describe satisfaction and effectiveness in relation to particular interventions provided by the services, while only a few studies examined overall satisfaction with the service or satisfaction with services for specific disorders. There were no studies where the primary aim was evaluation of overall service effectiveness. The quality of the studies was quite varied, and many of those relating to specific disorders or specific interventions had small sample sizes. It was positive that the majority of studies used validated standard questionnaires, such as the Strengths and Difficulties Questionnaire to assess efficacy and the CSQ to assess satisfaction.

Notably, many of the studies on specific interventions evaluated innovative multimodal treatments or programmes and found positive effects of these. However, there was less research on examination of standard practice even though these also require robust evaluation. None of the identified studies specifically examined the cost-effectiveness of services and treatments. Furthermore, apart from the study by McGarry et al. (2008), there was a lack of follow-up evaluation to establish whether benefits were sustained over time. There was also a lack of information on the overall duration of treatment within the service. The studies identified all related to outpatient CAMHS, and no studies were identified presenting evaluations of inpatient or liaison services.

Overall, the systematic review identified a number of common findings. These included overall high levels of satisfaction once services were received, and empirical evidence of benefit from various interventions offered. Positive staff attitudes, therapeutic alliance and continuity of care were found to be reported as important by many respondents. When perceived as experienced or available, respondents typically reported satisfaction with services. For a significant number however, they reported dissatisfaction due to fragmentation and inconsistency of care, and lack parental or family supports, where the needs of both parents and children were considered separately. The importance of clear and prompt diagnoses and treatment plans along with participation in care planning was also seen as crucial to user satisfaction. In many cases, lack of coordination and support between mental health and educational services contributed to parental dissatisfaction. A key concern listed by parents and professionals referring to CAMHS was of difficulty with timely access to assessment, diagnosis and treatment.

Outcome measures were generally not assessed in a blinded fashion, and participants were not blinded. However, blinding of study participants would have been hard to accomplish in all the studies given the nature of the service and treatments provided. The difficulty of implementing blinding or control groups further emphasises the potential benefit of using standardised outcome measures which would facilitate comparisons of effectiveness. Routine use of outcome measures as advocated and practiced by most CAMHS in the UK, to evaluate service effectiveness and acceptability, should become a standard part of Irish CAMHS (Waldron et al. 2018). Only by such consistent review will the strengths and weakness of service delivery be identified allowing the necessary measures and resources to be designated.

Limitations

Our search strategy excluded any services for children with a mean age under 6. However, most specialist mental health services for children in Ireland treat mainly children of primary and secondary school age, and it is unlikely that there were any studies excluded as a result of this element of the search strategy.

The search criteria did not allow examination of youth services where some patients may be under 18, but the average age of service users is over 18. Community youth mental health services such as ‘Jigsaw’ have been developed in recent years for adolescents and young adults with mild to moderate mental health difficulties (O’Reilly et al. 2015), but their remit is different to the specialist services for moderate to severe mental illness, and it would have been difficult to separate relevant elements for adolescents under 18 from young adults over 18 in evaluation.

It is possible that additional material regarding service evaluation is present in ‘grey literature’ not published in academic journals and may have been omitted as a consequence. As far as we are aware, this review is the first exploration of the amount and quality of evidence in the topic area in Ireland. We therefore decided to focus on published materials which have been validated by the peer review process and would be expected to provide a higher quality of evidence. Given that the results indicate that there is a relatively limited amount of peer-reviewed research evidence, future research could build on this review by examining other sources such as health service reports.

Studies which examine satisfaction among children and families currently attending the service have a possible inherent sampling limitation, because those children and families who quickly stopped attending services (or never attended any appointments offered) are less likely to be represented in the sample. As these individuals may have been less satisfied with the service on average, this leads to a possible sampling bias. The design of future studies of satisfaction should take this into account.

Given the diagnostically heterogeneous, differences in interventions, endpoints and sample sizes, it was difficult to compare results across studies’ samples (Table 3). These methodological differences and limitations make interpreting the data difficult. However, confidence in the specialist CAMHS must be increased by the generally positive findings, especially in the two large studies reported.

Conclusions and implications

Despite the importance of scant resources being allocated to tried and tested models to treat childhood-onset mental illness in specialised services, there is currently little evidence in existing literature to critically evaluate this with specific reference to CAMHS in Ireland. No studies were identified that evaluated inpatient and liaison child psychiatry services. Service user surveys where they exist are generally positive, both in terms of services offered and attitudes towards staff, once services are accessed. However, efficacy data are less well studied. Parental and professional concerns regarding perceived lack of information and difficulty accessing services with concerns regarding long waiting lists remain. The need for adequate resourcing and coordination of supports across services and sectors was highlighted. Standardisation of care offered complemented by routine outcome measures should be considered a priority along with individually targeted goal-based outcomes. Although some services are actively collaborating with patients and families and including service users in their management teams, these views and inputs need to be published to inform other services and reassure the public of overall governance and quality assurance.

A greater emphasis on multi-stakeholder feedback and satisfaction measures has been recognised as an important variable to consider in assessing quality of healthcare (Manary et al. 2013) and is consistent with national policy that highlights service user feedback in the development of patient-centred services (Government of Ireland, 2006). This needs to find its way into published literature to help inform and calibrate other services and interventions.

Public discontent with mental health services for children has been widely reported in the general media and at the Seanad Public Consultation Committee hearing. The report that followed (https://data.oireachtas.ie/ie/oireachtas/committee/dail/32/seanad_public_consultation_committee/reports/2017/2017-10-18_report-on-children-s-mental-health-services_en.pdf) suggested that CAMHS was falling short of the targets recommended in A Vision for Change, not unexpected given the reverse trend in overall funding for services, from 13% in 1984 to just over 6% currently. The report perceived this underfunding as ‘a repeated failure by State agencies to build the necessary capacity to provide adequate mental health services to children and adolescents’. The report also suggested a number of practical and short-term solutions to this inadequacy which has a detrimental effect by virtue of long waiting lists. Some of the research identified by this review supports this pessimism with regard to services, but is at risk of overshadowing the many positives reported when the services are accessed, and where treatment offered is found to be satisfactory and effective. It is only by continuous evaluation and input from services users, funders and providers that the necessary gaps can be filled, and innovations as seen in some studies here encouraged. In our efforts to improve services, it is imperative that the ‘baby’ is not thrown out with the bathwater.