Theory of change for complex mental health interventions: 10 lessons from the programme for improving mental healthcare

Theory of change (ToC) has gained prominence in recent years as an alternative way to conceptualise programme design and evaluation in global mental health (De Silva et al. 2014; Asher et al. 2016; Chibanda et al. 2016). This has been fuelled by renewed interest from development funders (Vogel, 2012) and the limitations of conventional research designs to evaluate complex global mental health interventions (Mackenzie, 2008).

ToC is a theory-driven approach for intervention development and evaluation, which makes explicit the causal pathways leading to the outcome of a programme. ToC was initially developed by evaluators working in education and development sectors in the 1990s (Connell & Kubisch, 1998) and has been used for more than 20 years in public health research (Breuer et al. 2016b), mostly in high-income countries. ToC has been used successfully in the USA in interagency planning for youth at risk (Hernandez & Hodges, 2006) and in the evaluation of various government initiatives in the UK (Sullivan et al. 2002; Cole, 2003).

ToC enables planners and evaluators to make explicit a number of aspects of a programme, including the impact and the short-, medium- and long-term outcomes required to achieve the impact. In addition, the ToC outlines necessary interventions, the assumptions inherent in the programme and the context (Vogel, 2012).

We have proposed that ToC could strengthen all four phases of the MRC guidance for the evaluation of complex interventions in both low- and high-resource settings (Craig et al. 2008; De Silva et al. 2014). Specifically, it will assist in the (1) development of the intervention through stakeholder consensus and creating realistic expectations of the impact of interventions; (2) feasibility/piloting stage of the intervention by highlighting knowledge gaps and barriers to implementation; (3) evaluation of the intervention by combining process and outcome evaluation in one framework; and (4) implementation of the intervention by ensuring the interventions and results are relevant to stakeholder's expectations (De Silva et al. 2014).

Since then, we have tested this approach in the Programme for Improving Mental healthcare (PRIME) (Breuer et al. 2014; Breuer et al. 2016a; De Silva et al. 2016), a large multi-country study that aims to provide research evidence for the integration of mental healthcare into primary healthcare in Ethiopia, India, Nepal, South Africa and Uganda (Lund et al. 2012). Here we present 10 lessons we have learnt in PRIME, from developing the ToC through workshops to the evaluation and analysis, which highlight the strengths and limitations of the ToC approach.

ToC is useful when interventions are complex

Complex health interventions contain multiple interacting components, causal strands, feedback loops and emergence of unexpected outcomes (Craig et al. 2008). Interventions can be implemented in multiple locations with different governance structures, increasing the complexity of even simple interventions. ToC focuses on the key, essential outcomes of the programme while allowing flexibility in how this is achieved across different contexts. The complex interventions developed and evaluated by PRIME were district-level mental healthcare plans in five countries (Hanlon et al. 2016). We used the ToC map to summarise a complex system of multiple causal pathways across three levels of the health system (Breuer et al. 2016a). We then mapped intervention packages onto these causal pathways, for example, a detection package at community level, a psychosocial treatment package at facility level or a mental health awareness package at district level. Each of these packages had the same function across countries, but the form differed as they were developed for different countries according to resources, evidence, need, feasibility and acceptability. This allowed comparability across settings but flexibility to ensure the intervention packages were fit for purpose (Hawe et al. 2004).

ToC workshops can help to develop a contextualised mental healthcare plan with stakeholder buy-in

Implementers, researchers, policy makers and service users each have their own implicit understanding of how and why a complex intervention works and what outcome it will achieve (Peters, 2014). Including a variety of stakeholders in ToC workshops allows all stakeholders to co-develop the ToC and make explicit the steps along the causal pathway (Andersen, 2004).

The PRIME ToC development process was extensive and included a cross-country ToC workshop, 2–4 workshops in each of the five PRIME countries and revision of the cross-country ToC (Breuer et al. 2016a). This resulted in one ToC map per country and a ToC across all five countries. The ToCs were influenced by each other and built on prior work of the PRIME consortium including a draft framework that outlined the three levels of the health system at which PRIME planned to intervene (Lund et al. 2012). We found that ToC workshops assisted us in developing a logical ToC map which formed the basis of a contextualised mental health care plan (Breuer et al. 2014). The presence of mental health specialists, researchers, policy makers, district-level health planners and management and service providers during the ToC development ensured that the resulting ToCs and mental healthcare plans were relevant, feasible and that the barriers and facilitating factors supporting this programme were clearly articulated. The presence of stakeholders also ensured their buy-in as they were able to contribute to the conceptualisation of the programme as well as highlight potential problems prior to the development of a detailed implementation plan. Chibanda et al. (2016), who used ToC to develop a counselling intervention for common mental disorders in Zimbabwe, found that early engagement helped to build rapport with stakeholders who provided detailed contextual information. This increased the likelihood that the intervention would be successful.

Although ideally the ToC should be owned by all stakeholders, this is often difficult in practice (Sullivan & Stewart, 2006). Ownership of the PRIME ToCs most closely resembles elite ownership (Breuer et al. 2014). According to (Sullivan & Stewart, 2006), this is ownership by a small group of leaders (including community leaders) who are responsible for implementing the programme. In PRIME, this is due to multiple reasons: (1) the large number of stakeholders involved in the workshops in countries [median 15 (interquartile range 13–22)] making extended consultation difficult; (2) the finalisation of the ToC by PRIME researchers after the workshop; (3) hierarchies within the health service making participation in the workshops uneven (despite our attempts to mitigate this) and (4) the lack of beneficiaries of the programme who attended the ToC workshop (Breuer et al. 2014).

ToC workshops are resource intensive

ToC workshops are resource intensive. These include human resources to plan, facilitate and attend the workshop, and the financial costs of conducting the workshop. To our knowledge, there has been no formal comparison of costs between ToC and other methods to develop and evaluate complex mental health interventions. However, from our experience, there are both higher costs and greater stakeholder input into the development of the ToC compared with other methods of developing a ToC, such as qualitative interviews or document review. Therefore, it is important to determine the level of complexity of the interventions and the extent to which local knowledge and stakeholder buy-in is useful so that a balance between resources and buy-in can be decided on a priori by the research team.

ToCs need champions to drive their development and implementation

ToCs need champions who understand the ToC approach, can drive the ToC development and implementation during the life of the programme (Lee, 2014). This includes organising and facilitating the workshops, finalising the resultant ToC map, getting further stakeholder input where required, finalising the evidence base and indicators for the ToC map and ensuring that the evaluation design of the programme measures the indicators outlined in the ToC. In PRIME, we were the ToC champions who led the work across all five countries and were supported by ToC champions in each country who led and facilitated the country workshops and developed the country ToCs.

Despite having champions, there has been no formal revision of the PRIME ToC in any country yet. There are three likely reasons for this: (1) data collection for the evaluation of the programme has recently been completed and the analysis is underway; (2) the research teams had competing priorities; and (3) no formal ToC revision of the ToC across and within countries was included in the workplan of the programme. In future projects, we would recommend a formal revision process of the ToC at key points in the process, for example, after piloting and after initial implementation of the programme, and after the final summative evaluation. For example, Asher et al. (2015) used the findings of her pilot study to revise her ToC for a community-based rehabilitation intervention for people living with schizophrenia in Ethiopia prior to implementing the intervention in a cluster randomised controlled trial.

The approach to ToC development should remain flexible

If the instructions on how to develop, portray or use a ToC become too prescriptive, ToC runs the risk of becoming yet another monitoring and evaluation tool that is used superficially in order to satisfy the requirements of funding agencies. Prinsen and Nijhof describe how logframes were initially developed often with stakeholders using a problem or objective tree. However, now they are largely templates standardised by funding agencies for completion (Prinsen & Nijhof, 2015). In PRIME, because we developed these ToCs in addition to our formal monitoring and evaluation requirements from our funder, there was flexibility in how the ToCs were developed, which helped to ensure that they represented the causal pathways to change.

ToC can provide a framework for evaluation and assist with identifying indicators for measurement

Once a causal pathway of short-, medium- and long-term outcomes has been developed, indicators are developed for each of these outcomes. This measures the achievement of each step along the causal pathway and distinguishes between implementation failure, where the programme was not implemented as intended, or theory failure, where the programme did produce the expected outcomes (Patton, 2008). In PRIME, we used the ToC to identify common indicators for each ToC outcome across each of the five PRIME countries. This in turn informed the design of evaluation, which allowed the programme to be compared across all PRIME sites (De Silva et al. 2016).

ToC indicators may need to be prioritised to account for time and resource constraints

It is unsurprising that the ToC of complex programmes are complex with many outcomes, causal pathways and feedback loops. The ToC maps can result in a comprehensive evaluation plan, which is made up of multiple study designs collecting various types of data. If resources are limited, it may be necessary to prioritise key indicators within the ToC so that the study designs are focused on collecting data on the most important steps along the causal pathway.

ToC does not prescribe a data collection method

ToC provides a framework to identify the pathways to impact, but it does not prescribe the type of data collection or analysis (Connell & Kubisch, 1998). This allows for the use of a wide range of quantitative and qualitative data collection and analysis techniques such as surveys, in depth interviews, document reviews, cohort studies, nested randomised controlled trials or programme observation (Breuer et al. 2016a). Data for the PRIME ToC indicators were collected using four study designs: repeat community surveys, repeat facility detection surveys, cohort studies and case studies (De Silva et al. 2016).

Combining data to evaluate indicators for short-, medium- and long-term outcomes is possible

In a recent systematic review, we found that many ToC data analysis techniques evaluate each outcome separately and do not combine the evaluation of the short-, medium- and long-term outcomes of the ToC (Breuer et al. 2016b). Methods that show promise for an integrated analysis of indicators in the ToC for complex mental health interventions are comparative case studies (Mookherji & LaFond, 2013), qualitative comparative analysis (Kane et al. 2014) or statistical approaches such as path analysis or structural equation modelling (Adedokun et al. 2011). Comparative case studies allow in-depth comparison of cases but become difficult to compare as cases increase. A conceptual limitation of structural equation modelling and other statistical techniques is the reduction of contextual factors to variables. ToC, as part of the broader school of theory-driven enquiry, is interested in ‘what works for whom in what circumstances’. A reductionist statistical approach may be useful in understanding key causal mechanisms but may obscure some of the complexity related to the context and other factors which co-vary between contexts. In PRIME, we have used a qualitative comparative approach to analyse data from ToC indicators from Nepal and will present this in a subsequent paper. Qualitative comparative approach is a case-orientated approach that uses set theory and Boolean logic to understand patterns across cases (Kane et al. 2014) and holds promise for analysing ToC indicators in mental health services research.

A ToC from a programme can be used as a heuristic device that can be adapted for other similar contexts

Where a ToC has already been developed, it can be used to inform other ToCs for similar programmes (Funnell & Rogers, 2011). Our PRIME ToC had a lot of similarities to ToCs developed in other similar mental health programmes, for example, the Friendship Bench in Zimbabwe and the RISE project in Ethiopia (De Silva et al. 2014; Asher et al. 2015; Breuer et al. 2016a; Chibanda et al. 2016). ToCs developed for one programme could be used as a heuristic device for other programmes. Care should be taken to ensure that the requisite contextual information and stakeholder input is obtained during this process to ensure that the ToC is adequately adapted to the new setting.

Conclusion

In summary, ToC, if applied thoughtfully and consistently, can be of great help in understanding complex interventions and strengthen the approach suggested in the MRC guidance for complex health interventions (Craig et al. 2008). Where resources are available to conduct ToC workshops, it offers a flexible approach to develop a complex intervention. It provides a comprehensive way to identify indicators to measure the short-, medium- and long-term outcomes on the pathway to impact. Indicators can then be prioritised if required and evaluation designs developed accordingly. Various data analysis approaches such as qualitative comparative analysis show promise to evaluate indicators from the ToC.