Impact of patient involvement in mental health research: longitudinal study

Abstract

Background

It is deemed good practice to involve patients routinely in research but no study has investigated the practical benefits, particularly to successful recruitment.

Aims

To identify whether patient involvement is associated with study success.

Method

All studies listed on the Mental Health Research Network (MHRN) portfolio database (n = 374) were interrogated using logistic regression, ANOVA and Pearson's correlation to identify associations with study characteristics, funding bodies and recruitment success.

Results

Patient involvement increased over time although in some areas of research it was limited. Some funders, especially the National Institute for Health Research (NIHR), had more associated patient involvement than others. Studies that involved patients to a greater extent were more likely to have achieved recruitment targets (χ² = 4.58, P<0.05), defined as reaching at least 90% of the target.

Conclusions

This is the first time associations with study success have been identified for patient involvement. Researchers might now consider ways to involve patients more comprehensively as this is associated with study success. Further research is needed to explore this finding.

The National Institute for Health Research (NIHR) coordinates and funds research throughout the National Health Service (NHS) in England. Topic-specific clinical research networks form part of the NIHR, and these each focus on a specific health condition. One of these is the Mental Health Research Network (MHRN). The topic-specific networks provide the infrastructure necessary to conduct research in the NHS by offering supporting services to facilitate research and also assist with recruitment. Callaghan et al provide a detailed account of the support the MHRN can provide, including some examples. ¹

One of MHRN's key interests is to increase the involvement of patients and their families or carers in research, beyond taking part as participants. This is to ensure that the research questions asked are valuable to patients and are asked in a way that makes sense to them. Patients' input into the design and implementation of research programmes should improve both the quality and feasibility of the research studies in mental health, but most importantly may enhance the translational value of research into clinical practice. Patient and public involvement in research is recommended by an increasing number of research funders, and patients have been involved in setting research priorities, ^2,3 contributing to the vexed question of which outcomes to choose for clinical trials, ⁴ providing alternative methodology, ^5,6 generating outcomes, ⁷ and showing that patient-researchers can elicit different types of information to non-patients. ⁸ Patient involvement can also indicate how research designs might have an impact on the results. ⁹

Recent studies highlighted the personal benefits to those patients involved in research, which promotes social inclusion and provides a sense of well-being at the individual level. ^10-12 However, no study has explored whether patient involvement in research is associated with key factors for study success such as study feasibility (and therefore recruitment success), or whether there are barriers - or facilitators - to involvement such as the complexity of the study. This gap may exist because of the lack of data available to explore such associations. The MHRN portfolio of studies since 2004 contains valuable information on recruitment success, study complexity measured by such aspects as the length and frequency of follow-up, randomisation procedures, as well as patient involvement. We now therefore have the means to assess patient involvement in mental health research and to interrogate potentially meaningful associations for studies in the UK NHS. The following specific questions were examined: (a) has patient involvement in studies increased over time; (b) are there any factors that are associated with increased or decreased involvement; and (c) is patient involvement associated with increased or decreased recruitment success?

Method

Sample of studies

The studies considered were all non-commercial studies registered on the MHRN portfolio database accessed on 12 October 2011 that were ongoing (n = 206), completed (n = 135), in set up (n = 28) or suspended (n = 5), making a total of 374 studies. The database is the most comprehensive database of mental health research in England and contains all studies supported by the MHRN since its inception. The database contains information at an individual study level on the current or future plans for patient or carer involvement. Supported studies are funded by the NIHR or NIHR partners (organisations awarding grants relevant to the NHS in open national competition with external peer review).

Measures

For each study the following variables were recorded.

Level of patient involvement

The principal investigator selects the highest level of involvement relevant to their study using the following categories (from lowest to highest levels of involvement): (a) patient consultation; (b) researcher-initiated collaboration; (c) jointly-initiated collaboration; (d) patient-initiated collaboration; and (e) patient-controlled studies. For the analysis, the latter three categories were merged to ensure adequate numbers. A recent audit of a random sample of supported studies found that in 85% of the studies audited, investigators were involving patients as they had set out to, which lends validity to the reported extent of patient involvement. ¹³

Study complexity

This MHRN scale was derived through a Delphi exercise with experienced researchers, MHRN clinical leads and MHRN managers that produced a set of items that might predict recruitment difficulties. The total score was valid in predicting recruitment problems in MHRN portfolio studies in 2008. Items include the frequency of assessments, inclusion of participants without capacity to consent, number of sites required and number of face-to-face follow-up interviews. Complexity scores range from 1 to 17, with higher scores indicating higher complexity.

Successful recruitment

For this study, the pragmatic decision was made to define successful recruitment a priori as a binary variable with success specified as reaching >90% of the proposed recruitment target. This is because this level of recruitment allows the investigators some power in assessing their primary outcome, whereas investigating whether recruitment increases from 30 to 40% of target is not of much benefit scientifically as it still does not allow the main hypothesis to be tested. Our 90% level is also the cut-point used nationally to identify studies that are considered as not recruiting successfully (see NIHR ¹⁴ ). Only closed studies were considered for analyses involving recruitment (n = 124; 11 further studies were excluded from this analysis as they fell into clinical study groups (see below) that did not have adequate numbers for analysis).

Primary clinical study groups

This classification was derived by clustering studies into the categories suggested in the Strategic Analysis of UK Mental Health Research Funding. ¹⁵ The final categories were produced by grouping similar clinical study groups. This resulted in seven mental health categories that had sufficient numbers of studies for between-group analyses. These were: (a) psychotic disorders; (b) mood disorders; (c) developmental disorders; (d) personality disorders; (e) other common mental disorders; (f) social interventions; and (g) services research. A total of 21 studies were excluded from all analyses involving clinical study groups as they fell into groups that lacked adequate numbers for reliable analysis (‘dementias’ (n = 5) and ‘physical disabilities’ (n = 16)).

Funding body

Funding bodies were grouped into five categories: (a) NIHR; (b) Medical Research Council (MRC); (c) government (non-NIHR); (d) charities and non-profit organisations; and (e) international.

Study type

Three separate variables were accessible: (a) observational v. interventional v. both; (b) follow-up v. non-follow-up; and (c) randomised v. non-randomised.

Study entry order

Studies were ordered on the portfolio by the date they were identified by the MHRN.

Statistical analysis

Change in patient involvement over time was assessed by correlating study entry order with level of patient involvement using Pearson's product moment. Predictors of levels of patient involvement were explored using multinomial logistic regression. The independent variables were funder, clinical study group, complexity, randomisation (yes/no), follow-up (yes/no) and study type (observational/interventional/both). Potential factors were entered into the model using a backward elimination approach with retention criterion for the variable to be included in the model set at 0.05.

Predictors of successful recruitment were explored using binary logistic regression again with a backward conditional method to identify variables that predicted whether a study hit the recruitment target (>90%). The independent variables were patient involvement, funder, clinical study group, complexity, randomisation (yes/no), follow-up (yes/no), and study type (observational/interventional/both). Logistic regression was chosen as the analytic approach as it has several advantages over multiple linear regression that are important for this analysis; namely that the independent variables do not contain interval-level data, nor are they normally distributed. All data were analysed using IBM SPSS version 20.0 for Windows.

Results

Mental Health Research Network portfolio of studies

Of the 374 studies included in the portfolio, 207 (55%) were observational, 154 (41.5%) were interventional and the remainder both. A total of 213 studies (57%) included a follow-up; and 144 (39%) studies were randomised. Table 1 provides data on the number and percentage of studies in each funder category and clinical study group.

Are there predictors of level of patient involvement?

Patient involvement was modestly but significantly correlated with study entry order, r = 0.12, P<0.05 showing that involvement has increased over time.

Multinomial logistic regression supported a model including the variables clinical study group, funder and study type (observational v. interventional v. both). The model was significantly better at predicting level of patient involvement than the intercept-only model, χ²(24) = 80.25, P<0.001 (pseudo R ²: Cox and Snell: 0.203, Nagelkerke: 0.233) and resulted in 56.1% correct classification of cases. Funder type was the most strongly associated factor (χ²(8) = 41.61, P<0.001), followed by clinical study group (χ²(12) = 27.10, P<0.01) and study type (observational v. interventional v. both) was close to significance (χ²(4) = 8.73, P = 0.068).

Table 2 shows the model results for the comparison of researcher-initiated collaboration v. consultation only. This shows that when comparing these two levels of involvement, it is only being funded by the NIHR that is significantly associated with the higher level of involvement.

Table 3 shows the model results for the comparison of the highest level of involvement v. consultation only. Clinical study groups for developmental disorders, personality disorders and social interventions were all more likely to fall into the lowest category of patient involvement relative to the category denoting highest patient involvement. Table 3 also shows that NIHR-funded studies were more likely to be in the category of highest patient involvement, relative to the lowest category of patient involvement.

Table 1 Details on the make-up of the Mental Health Research Network database

Category	n (%)
Clinical study group
Psychotic disorders	104 (28)
Mood disorders	59 (16)
Other common mental disorders	34 (9)
Developmental disorders	25 (7)
Personality disorders	37 (10)
Social interventions	33 (9)
Services research	61 (16)
Excluded	21 (5)
Funder
National Institute for Health Research	227 (61)
Medical Research Council	60 (16)
Government	24 (6)
Charities/not for profit	25 (7)
International	38 (10)

Figure 1 depicts the proportion of studies that involved patients at each of the three levels of involvement as a function of funding body. The proportion of studies at each level of involvement is depicted on the vertical axis. The horizontal line indicates the mean proportion of studies that involved patients at the highest level when collapsing the funder categories. This shows that the funders with the largest proportion of studies that included the highest level of patient involvement are charity/non-profit funders (28%) and the NIHR (23%). The NIHR also had the smallest proportion of studies in the consultation-only category suggesting the emphasis on patient involvement is more pervasive than for any other funder. The funder least likely to involve patients at the highest level was the MRC, with just 10% of MRC-funded studies involving patients at the top level.

Is patient involvement associated with successful recruitment?

Mean recruitment across all closed studies was 85% (s.e. = 1.85) and 80 studies (59%) reported more than 90% recruitment. Study type (follow up v. non-follow-up), complexity score and level of patient involvement all predicted whether a study achieved its recruitment target and the overall model was significant (χ²(4) = 12.58, P<0.05; pseudo R ²: Cox and Snell: 0.096, Nagelkerke: 0.131). The results are shown in Table 4.

Table 2 Logistic regression results showing associations with patient involvement: consultation only v. researcher-initiated collaboration

Predictor	β (s.e.)	Wald's χ2 (d.f.)	P	Odds ratio a
Clinical study group
Psychotic disorders	–0.33 (0.43)	0.60 (1)	0.44	0.72
Mood disorders	0.26 (0.50)	0.28 (1)	0.60	1.30
Other common mental disorders	–0.66 (0.56)	1.41 (1)	0.24	0.52
Developmental disorders	–0.59 (0.57)	0.05 (1)	0.31	0.56
Personality disorders	–0.28 (0.51)	0.30 (1)	0.59	0.76
Social interventions	–0.01 (0.52)	0.000 (1)	0.99	0.99
Funder
National Institute for Health Research	1.53 (0.44)	12.30 (1)	0.001 ***	4.59
Medical Research Council	0.10 (0.52)	0.36 (1)	0.85	1.10
Government	–0.40 (0.64)	0.004 (1)	0.95	0.96
Charities/not for profit	0.10 (0.71)	0.02 (1)	0.89	1.11

a. Odds ratios are for consultation-only relative to researcher-initiated collaboration.

^*** P<0.001.

Non-follow-up studies were less likely to hit the recruitment target and as complexity increases the likelihood of achieving the recruitment target falls. The analysis also demonstrates that studies with higher levels of patient involvement were more likely to achieve recruitment success. This is true in spite of the fact that recruitment rates have not increased over time, therefore this finding is unlikely to be due to a general secular increase in recruitment success.

Discussion

We investigated patient involvement using a comprehensive database and we have revealed that: (a) patient involvement has increased over time; (b) (i) studies regarding developmental disorders, personality disorders and social interventions were associated with lower levels of involvement, and (ii) NIHR- and charitably-funded studies were associated with greater levels of patient involvement; and (c) lesser study complexity, and increased patient involvement were associated with recruitment success. Follow-up studies were also more likely to achieve their recruitment target.

Study context, strengths and limitations

We explored whether patient involvement was associated with recruitment success after the effects of other variables had been taken into account and found that there was an association over and above the contribution of other factors. To put this result into some context, one strength of this study is that we had the most detailed data available on patient involvement that fulfils the categories of involvement thought to be important by INVOLVE - the UK organisation with a remit for patient and public involvement in research. We know of no other databases across the UK or the world that have baseline and longitudinal data of this calibre. Of course, we would have liked to investigate even more variables that would allow more specific recommendations. This also meant that we could not account for the potential contributions of factors like quantity of funding or design factors such as basic science v. epidemiological research, which also might limit the level of patient participation. The design with the highest internal validity when investigating patient involvement would be a randomised controlled design. However, this would involve denying involvement to principal investigators who request it and masking in such a trial might be problematic and so negate many of the advantages of the randomised controlled trial design. Even with these caveats our results show an association with patient involvement. We have used data from the MHRN database which an audit has found to be reliable. ¹³ In order to more fully understand the mechanism of why involvement and any of the other factors are associated with study success, the MHRN would need to alter their portfolio forms to include greater detail.

Fig. 1 Proportion of studies in each funding category.

NIHR, National Institute for Health Research; MRC, Medical Research Council.

Possible mechanisms and implications

As expected the design of studies is associated with achieving recruitment targets. The main variables identified here were complexity and presence of follow-up. Complexity is associated with participant recruitment in the expected direction with more complex studies having more difficulty in recruitment. However the finding regarding follow-up studies is in the opposite direction to that expected; we presumed that more follow-up sessions would reduce the likelihood of recruiting to target, perhaps by way of reduction in willingness of patients to take part. It is possible that the association results from researchers adopting different procedures for recruitment including more involvement of care coordinators and more contact with potential participants. It is recommended by patient bodies that participants should be kept up to date with the progress of studies, for example with newsletters; these activities may also promote recruitment as the study is disseminated via informal patient involvement. This might be tested by asking potential participants how they heard about the study and what attracted them to take part.

Table 3 Logistic regression results showing associations with patient involvement: consultation only v. highest level of involvement

Predictor	β (s.e.)	Wald's χ2 (d.f.)	P	Odds ratio a
Clinical study group
Psychotic disorders	–0.38 (0.47)	0.66 (1)	0.42	0.68
Mood disorders	–0.78 (0.60)	1.69 (1)	0.19	0.46
Other common mental disorders	–1.25 (0.67)	3.46 (1)	0.06	0.29
Developmental disorders	–2.84 (1.13)	6.31 (1)	0.012 *	0.06
Personality disorders	–1.79 (0.74)	5.82 (1)	0.016 *	0.17
Social interventions	–1.94 (0.86)	5.14 (1)	0.023 *	0.14
Funder
National Institute for Health Research	1.49 (0.61)	5.97 (1)	0.015 *	4.45
Medical Research Council	–0.33 (0.77)	0.18 (1)	0.67	0.72
Government	0.03 (0.88)	0.001 (1)	0.97	1.03
Charities/not for profit	1.11 (0.87)	1.64 (1)	0.20	3.03

a. Odds ratios are for consultation-only relative to the highest level of involvement.

^* P<0.05.

Table 4 Logistic regression results showing predictors of successful recruitment

Predictor	β (s.e.)	Wald's χ2 (d.f.)	P	Odds ratio a
Non-follow-up	–1.48 (0.61)	5.94 (1)	0.015 *	0.23
Complexity	–0.19 (0.08)	6.55 (1)	0.010 *	0.83
Involvement
Researcher initiated	0.49 (0.41)	1.41 (1)	0.236	1.63
Jointly initiated or higher	1.42 (0.66)	4.58 (1)	0.032 *	4.12

a. Odds ratios represent the ratio of the odds of not achieving recruitment target v. achieving recruitment target.

^* P<0.05.

After these complexity and follow-up factors have been taken into account study success was associated with higher levels of patient involvement. This is the first report of such an association, perhaps because the data on levels of involvement and study design for multiple studies have never been available before. The data reported are associations and do not equate to causality, as other factors not investigated might also contribute. However, there does seem to be a trend towards a dose-response relationship, as studies which included researcher-initiated collaboration were 1.63 times more likely to recruit to target than studies that only consulted patients, whereas studies that included the highest level of involvement were 4.12 times more likely to recruit to target.

There are a number of possible explanations as to why patient involvement was associated with successful recruitment: (a) the language used in materials such as information sheets is more appealing or easier to understand for patients because of vetting by other patients; (b) patients contribute insight into the realities of living with a mental health problem and therefore understand which designs will be the least burdensome; and/or (c) patients are more willing to participate in research that they know has involved other patients, as the principle of patient involvement is in itself appealing.

Barriers and facilitators to involvement

Patient involvement has increased over time but this has not been across all areas of clinical study. Some clinical study groups have not routinely involved patients at the higher levels of involvement. This is despite the encouraging finding that high levels of patient involvement in research are currently taking place in some difficult contexts. It is plausible that there are inherent difficulties when attempting to involve patients in some areas. For example patient involvement in personality disorders research may be more difficult because of the severe and enduring social functioning problems and slow remission that characterise the disorder. ¹⁶ Equally developmental disorders research may struggle to involve patients at a high level as these studies typically involve participants with learning impairments. Further organisations such as the Young Person's Advisory Group developed by the NIHR Medicines for Children Network may aid such involvement. ¹⁷

National Institute for Health Research-funded studies were more likely to have higher levels of patient involvement. This reflects the different expectations of funding bodies as there is a high level of expectation in the NIHR, with specific questions and explanations of patient and public involvement in their applications. Given the finding of an association with study success, other funders may get more value for money if they increased their expectations to match those of the NIHR. However, it is also important that patient involvement is not considered a panacea to overcome recruitment difficulties, as certain protocols are likely to continue to struggle with recruitment because of issues such as equipoise, which clearly cannot be overcome by patient involvement alone. Equally funders must be wary of tokenistic involvement that may result from an emphasis on involvement in their application form, especially if researchers believe that their funding depends on it. This potential for tokenism was also identified in a report on research ethics applications by INVOLVE, the UK national advisory group supporting greater public involvement in healthcare. This study discovered that although 62% of applications indicated a plan for involving patients, 69% of these plans were not confirmed by information in the free-text boxes. ¹⁸ This, of course, was not true of the MHRN data we have interrogated. Free text boxes were included in the adoption forms, so investigators had to justify why the involvement was felt to be of a particular nature, and if this was not supported then more information was requested. Trivedi & Wykes also point to barriers that need to be overcome in order to prevent tokenism in a clear set of principles for involvement. ¹⁹

A recent complementary report conducted for the MHRN investigated patient involvement by interviewing researchers and patients about their experiences. ¹³ Although there were clear tangible benefits, the report also described a few problems because of a lack of understanding about how to involve patients effectively in research, resulting in a poor experience for both parties. We suggest that further support is necessary and should be considered by funders to ensure involvement is meaningful and likely to be cost-effective. We note that some NIHR programmes now have a finance section specifically dedicated to patient and public involvement but other support may also be necessary, for example training and mentoring of researchers and patients.

Research will vary in the extent that it can benefit from patient involvement; however, the analysis here revealed that studies funded by one UK funder, the MRC, had poor levels of patient involvement. There is less emphasis on involvement both in the MRC application and review process. The MHRN's Feasibility and Support to Timely recruitment for Research (FAST-R) service provides quick, easy access to people with experience of mental health problems who are able to offer their expert advice on potential recruitment barriers and the viability of information sheets and consent forms. Following discussions with the MRC the FAST-R service will be publicised on their website to encourage researchers to incorporate more patient involvement into their research designs.

Future directions

We have shown increases in patient involvement over time that probably reflect both the emphasis of funders and NIHR MHRN support for such activities. Patient involvement described at the start of the study is associated with study success even after other likely predictive factors such as study design, complexity and clinical group under investigation have been taken into account. As this is the first study to carry out such an investigation there are a number of additional unanswered questions about patient involvement that need consideration. These are discussed above. Without a database detailing both the levels of involvement and information regarding study design and recruitment success, these questions cannot be answered. Currently, the most detailed evidence is only available via the MHRN database. The next set of studies need to identify the specific mechanism by which patient involvement seemingly improves recruitment. This would require both the level of involvement together with more details on the progress of involvement in the study. Without these data all we can conclude is that patient involvement is associated with, but not necessarily a cause of, study success.

Funding

The source of funding was the National Institute for Health Research Mental Health Research Network. T.W. would like to acknowledge the support of the National Institute for Health Research (NIHR) Mental Health Biomedical Research Centre at South London and Maudsley NHS Foundation Trust, and the NIHR for her Senior Investigator Award.