Online screening and feedback to increase help-seeking for mental health problems: population-based randomised controlled trial

Abstract

Background

Community-based screening for mental health problems may increase service use through feedback to individuals about their severity of symptoms and provision of contacts for appropriate services.

Aims

The effect of symptom feedback on service use was assessed. Secondary outcomes included symptom change and study attrition.

Method

Using online recruitment, 2773 participants completed a comprehensive survey including screening for depression (n=1366) or social anxiety (n=1407). Across these two versions, approximately half (n=1342) of the participants were then randomly allocated to receive tailored feedback. Participants were reassessed after 3 months (Australian New Zealand Clinical Trials Registry ANZCTR12614000324617).

Results

A negative effect of providing social anxiety feedback to individuals was observed, with significant reductions in professional service use. Greater attrition and lower intentions to seek help were also observed after feedback.

Conclusions

Online mental health screening with feedback is not effective for promoting professional service use. Alternative models of online screening require further investigation.

Screening for mental health problems in clinical settings has been purported to increase recognition and lead to better treatment outcomes. ¹ However, evidence from systematic reviews and meta-analyses suggests that screening alone has little impact on the detection and management of depression by clinicians. ^2–5 The US Preventive Services Task Force now recommends routine screening for depression only if there are systems in place to deliver adequate treatment and follow-up. ⁶ Furthermore, the UK National Institute for Health and Care Excellence (NICE) guidelines no longer recommend screening in primary care. ⁷ Nevertheless, research has tended to focus on the use of screening tools in clinical settings including primary care, rather than in population settings. Screening in the population may empower the individual to seek appropriate care by providing them with tailored feedback about their symptoms and providing recommendations for appropriate services. ^8,9 The rise of internet technology has enabled population screening with feedback to be rapidly disseminated. ^10,11 Uncontrolled studies have suggested that providing feedback from community-based screening may be effective for encouraging service use ¹² and encouraging retention in research studies. ¹¹ A quasi-experimental study has suggested that depression screening may reduce suicide in Japanese older adults. ¹³ However, there have been no randomised controlled trials (RCTs) of online screening, to evaluate how the use of online screening and feedback platforms might impact outcomes for individuals at risk of mental health problems. In addition, there have been very few studies of community-based screening programmes for anxiety disorders. ^10,14

The current study describes the outcomes of an RCT that aimed to evaluate whether screening with tailored feedback – including listings and linkage to appropriate clinical resources – would increase help seeking from professional sources. Use of professional services, rather than informal sources of help, was chosen as the primary outcome because health professionals are more likely to provide evidence-based treatments and more accurate assessment and information than other sources. ¹⁵ A number of secondary outcomes were also investigated, specifically whether screening would increase intentions to seek help, decrease symptom levels for the target disorder, increase quality of life, decrease disability or decrease attrition from the study. Screening for two of the most common mental health problems, depression and social anxiety, was examined in two independent samples recruited simultaneously. Participants in each sample were screened online and randomly allocated to receive: (a) tailored feedback on their symptoms with appropriate resources, or (b) no feedback on their symptoms. On the basis of previous uncontrolled screening trials, ^9,12 it was hypothesised that participants in the feedback conditions for each of the disorders would have significantly increased rates of service use after 3 months, compared with those in the control (no feedback) conditions.

Method

Participants and procedure

Participants were recruited from the online social media website Facebook. The target population of Facebook users aged ≥18 years was 8.8 million, representing approximately 45% of the total Australian population aged ≥18. From August to December 2014, a series of advertisements were placed on Facebook targeting Australian adults with the wording: ‘Assessing Mental Health Survey: Participate in a study examining your mental health by completing a 40 minute survey now’. These advertisements linked individuals to one of two versions of the survey. The surveys were administered online using LimeSurvey, with data stored on a secure server at the Australian National University (ANU), Canberra. The study had ethics approval from the ANU Human Research Ethics Committee (protocol #2013/509). The trial was registered with the Australian New Zealand Clinical Trials Registry (ANZCTR12614000324617).

Two versions of the survey were administered, with each version providing feedback on symptoms of a different mental disorder: depression or social anxiety. During the recruitment period, 27 158 people clicked the advertisement and 12 240 ‘liked’ the study's page. A total of 6292 people consented to participate in the survey, with 3323 (52.8% of consenters) completing the survey. Of these, 2773 (83.4% of completers) consented to participate in the follow-up assessment by providing an email address at the end of the survey, with 966 (34.8% response rate) commencing the follow-up and 895 (32.3%) completing the follow-up assessment. This sample size provided 95% power to detect a 20% increase in service use (from 58.0% to 69.6%) at follow-up. A CONSORT diagram of participant flow through the study is presented in Fig. 1.

Fig. 1 CONSORT diagram showing flow of participants in the trial

Participants were provided with comprehensive survey information before giving consent to participate. The information sheet outlined what was involved in the survey, including the potential risks of participation, and provided contact information for psychological and crisis services across Australia. The survey took approximately 40–60 min to complete. Towards the end of the survey, participants were asked whether they would be willing to complete a brief survey after 3 months, by providing their email address. Those who provided an email address were randomly allocated (simple randomisation by concealed computer assignment in 1:1 ratio) to receive tailored feedback about their mental health (intervention group for depression or social anxiety) or receive no feedback (control group). Participants completed a brief survey (approximately 15 min) 3 months after the initial survey, with two email reminders given 1 week apart when the follow-up survey was due.

Intervention conditions

Participants in the feedback intervention condition for depression or social anxiety were informed that their symptoms indicated ‘low risk’, ‘at risk’ or ‘high risk’. Category membership was determined based on scores on either the Patient Health Questionnaire (PHQ-9) ¹⁶ or the Social Phobia Screener (SOPHS) ^17,18 for depression and social anxiety respectively. Low-risk participants were classified as those scoring <10 on the PHQ-9 or <6 on the SOPHS. At-risk participants scored 10–19 on the PHQ-9 or 6–11 on the SOPHS, whereas high-risk participants scored >19 on the PHQ-9 or >11 on the SOPHS. These cut points were determined based on previous validation studies of the screening instruments. ^18,19

Feedback was provided using a traffic-light image as illustrated in Fig. DS1, towards the end of the survey approximately 10–20 min after completion of the screening measure. The text of the low-risk feedback was presented below the image in the following format: ‘What does it mean? Your [depression/social anxiety] score was in the low-risk category. This suggests that you are unlikely to be experiencing [depression/social anxiety]. If you would like more information on [depression/social anxiety], a number of websites provide information about the treatment and management of [depression/social anxiety]’. Links to a number of websites providing evidence-based information were then provided. In the at-risk condition, a similar format was used but with additional information: ‘Your [depression/social anxiety] score was in the at risk category. This suggests that you may be at risk of experiencing [depression/social anxiety]. You may benefit from seeking help from one of the resources listed below’. Brief psychoeducation regarding specific evidence-based treatment options and treatment sources for the disorder was then provided, and a list of evidence-based effective online therapy programmes was provided. The at-risk and high-risk groups also received the same informational resources as the low-risk groups. The high-risk groups for depression and social anxiety received similar feedback to the at-risk group, with wording slightly altered to indicate increased risk: ‘Your [depression/social anxiety] score was in the high risk category. This suggests that you are likely to be experiencing problems with [depression/social anxiety]’. In addition, professional help seeking from a general practitioner (GP) or mental health professional was encouraged: ‘Many people find that seeking help from a GP or mental health professional is helpful for reducing the symptoms of depression. Take a look at the resources below to find an appropriate service for you’.

Control conditions

Participants in the control conditions for depression and social anxiety did not receive any feedback about their symptom levels. However, to meet ethical and duty of care requirements, all participants received generic, untailored advice at the conclusion of the survey that they should contact a GP, crisis telephone line, online support/information service or crisis service if they were concerned about their mental health.

Measures

The primary outcome was self-reported professional service use at the 3-month follow-up assessment. The Actual Help Seeking Questionnaire (AHSQ) ²⁰ was administered to all participants, enquiring: ‘Have you sought help for a mental health problem from any of the following sources in the past 3 months?’, followed by 10 response choices (mental health professional, doctor/GP, intimate partner, friend, parent, other relative, telephone helpline, minister/religious leader, other, nobody). Participants who checked ‘Mental health professional (e.g. psychologist, social worker, counsellor)’ or ‘Doctor/GP’ were classified as using professional services.

A number of secondary outcomes were also investigated. Symptom severity for the disorder of focus (depression or social anxiety) was assessed using the PHQ-9 (9 items) or SOPHS (5 items) respectively. Symptom scores on these scales can range from 0 to 27 and 0 to 20 respectively, with higher scores indicating greater symptom severity. These scales have previously been shown to be accurate in screening for risk of disorder ^18,19 and had high internal consistency in the current sample (Cronbach's α=0.93 and 0.96 for the PHQ-9 and SOPHS respectively). Intentions to seek help from a medical professional (mental health professional or doctor/GP) for a mental health problem were assessed using two items from the General Help Seeking Questionnaire (GHSQ), ²¹ with total scale scores ranging from 1 to 14 and higher scores indicating greater intentions to seek help. Health-related quality of life was assessed using the 12-item Assessment of Quality of Life (AQoL-4D) instrument. ²² The scale covers four dimensions, independent living, relationships, senses and mental health, and had fair internal consistency in the current sample (Cronbach α=0.78). Utility scores were calculated as prescribed by the scale authors, ranging from 0 to 1, with higher utility scores indicating greater quality of life. Mental health-related disability was assessed based on self-reported days out of role (i.e. number of days for which the individual was completely unable to work or carry out normal activities) in the past month due to mental health problems. All outcomes were assessed at the baseline and follow-up assessments. Independent predictors were assessed based on self-report questions at baseline, including age, gender, education, employment, area of residence and language spoken at home.

Analysis

Sample characteristics were compared across conditions (feedback v. no feedback) and versions of the intervention (depression v. social anxiety), based on χ ²-tests for categorical variables and F-tests from one-way ANOVA for continuous variables. Binary logistic regression analysis was used to compare participants who completed the follow-up assessment with those who did not to identify correlates of attrition.

All analyses were undertaken on an intent-to-treat (ITT) basis. Mixed model repeated measures (MMRM) analyses ²³ were used to include all available data from participants who consented to follow-up (n=2773). This approach yields unbiased estimates of intervention effects under the assumption that data were missing at random. An unstructured matrix was assumed and degrees of freedom were estimated using Satterthwaite's correction. Analyses were conducted using the combined sample from both versions of the intervention (depression and social anxiety), repeated separately within the two versions to test for disorder-specific effects and repeated separately for the three levels of feedback (low risk, at risk, high risk). The analysis of the primary outcome, professional service use, was based on a binary outcome, necessitating the use of a mixed effects logit analysis that accounted for initial service use and incorporated all available data. This analysis was conducted in StataIC v10 (StataCorp, College Station, Texas, USA) using the xtlogit command. All remaining analyses were conducted by using SPSS v20 (IBM Corp, Chicago, Illinois, USA).

Results

Sample characteristics at baseline by version (depression and social anxiety) and intervention condition (feedback v. no feedback) are shown in Table DS1. There were no significant differences between the intervention and control conditions on the basis of any demographic or clinical indicators, with the exception of gender and screening status. Males were more highly represented in the depression version ( ${\chi }_1^2$ =12.55, P=0.006), and participants in the depression version were more likely to screen as at risk or high risk than those in the social anxiety version ( ${\chi }_1^2$ =14.68, P=0.023). However within the two versions, there were no significant differences between the intervention and control groups. The majority of participants were middle-aged, female and well educated. The sample tended to have elevated depression and social anxiety symptoms, with mean scores close to clinical cut points. Quality-of-life scores were lower than population norms, ²⁴ and participants had high rates of service use and a mean of 3.4 days out of role in the past month due to mental health problems.

Attrition effects

Attrition was examined using a binary logistic regression, to examine whether receiving feedback or other participant characteristics were associated with completion of the follow-up assessment (Table 1). The depression version had higher completion rates than the social anxiety version. Receiving feedback was associated with significantly less completion of follow-up, with approximately 31% higher odds of completion among those who did not receive feedback overall (27% for the depression version and 36% for the social anxiety version). However, the level of feedback (reflecting symptom severity) was not significantly associated with attrition. There were also significantly higher levels of completion among participants who were older, were employed, had higher quality of life or had greater intentions to seek help for a mental health problem.

Table 1 Binary logistic regression model examining factors associated with the completion of the follow-up assessment at 3 months

	Estimate	s.e.	Odds ratio	P
Survey version				<0.001
Social anxiety (reference)	0.00		1.00
Depression	1.18	0.09	3.24	<0.001
Intervention condition				0.003
No feedback control (reference)	0.00		1.00
Feedback intervention	−0.26	0.09	0.77	0.003
Feedback level				0.576
Low risk (reference)	0.00		1.00
At risk	0.01	0.11	1.01	0.953
High risk	0.16	0.17	1.18	0.332
Age group				0.023
18-25 (reference)	0.00		1.00
26-35	0.40	0.21	1.49	0.054
36-45	0.53	0.19	1.70	0.004
46-55	0.43	0.18	1.54	0.015
56-65	0.62	0.18	1.85	0.000
>65	0.44	0.20	1.56	0.029
Gender				0.668
Female (reference)	0.00		1.00
Male	−0.05	0.11	0.96	0.668
Employment				0.014
Employed (reference)	0.00		1.00
Not in employment	−0.24	0.10	0.79	0.014
Language spoken				0.210
English only (reference)	0.00		1.00
Another language	−0.23	0.19	0.79	0.210
Years of education	0.03	0.02	1.03	0.072
Area of residence				0.217
Metropolitan (reference)	0.00		1.00
Regional	−0.13	0.10	0.88	0.183
Rural	−0.20	0.13	0.82	0.127
AQoL utility score	0.48	0.22	1.62	0.026
Days out of role	0.00	0.01	1.00	0.549
Professional help-seeking intentions	0.06	0.03	1.06	0.026
Constant	−2.43	0.35	0.09	<0.001

AQoL, Assessment of Quality of Life.

Bold values indicate P<0.05.

Intervention effects

To examine whether feedback modified outcomes, the interaction between time and condition from linear and binary mixed effects models was tested. These are presented in Table 2 for the total sample, for each version of the intervention, and based on symptom feedback level. There were no overall effects of feedback on help-seeking behaviour or any secondary outcomes. However, in the social anxiety version only, there were small significant effects of symptom feedback on professional service use (between group effect size: Cohen's d=−0.17) ²⁵ and help-seeking intentions (Cohen's d=−0.19), both favouring the control condition (no feedback). The significant interaction effect for the social anxiety intervention on use of professional health services is illustrated in Fig. 2, with service use increasing more for participants in the control group, most prominently among high-risk participants (although no significant subgroup effect was found for high-risk participants). In a sensitivity analysis, we included variables associated with attrition in the mixed models, and found no new significant intervention effects for any outcome, whereas the effects of social anxiety feedback on help-seeking behaviours (Z=−2.03, P=0.042) and help-seeking intentions (F=6.1, P=0.014) remained significant.

Fig. 2 Effect of feedback on service use in the social anxiety intervention

Table 2 Interaction effects between time (pre/post) and intervention condition (feedback/control) from linear and binary mixed effects models for the total sample and subgroups ^a

Sample	F/Z b	d.f.	P
Total sample
Professional service use	−0.6		0.547
AQoL utility score	3.0	1, 959.8	0.082
Days out of role	0.3	1, 1080.7	0.571
Help-seeking intentions	1.5	1, 1107.2	0.227
Depression score (PHQ-9)	0.1	1, 915.8	0.724
Social anxiety score (SOPHS)	0.2	1, 922.7	0.684
Version 1 (depression)
Professional service use	1.1	0.287
AQoL utility score	1.5	1, 657.7	0.224
Days out of role	0.1	1, 726.0	0.823
Help-seeking intentions	0.3	1, 770.9	0.591
Depression score (PHQ-9)	0.6	1, 649.8	0.431
Among low risk: PHQ-9 score	0.1	1, 378.8	0.713
Among at risk: PHQ-9 score	1.8	1, 169.1	0.187
Among high risk: PHQ-9 score	0.1	1, 69.4	0.810
Version 2 (social anxiety)
Professional service use	−2.1		0.038
AQoL utility score	1.6	1, 300.1	0.205
Days out of role	0.1	1, 342.8	0.720
Help-seeking intentions	6.0	1, 326.7	0.015
Social anxiety score (SOPHS)	0.7	1, 297.9	0.389
Among low risk: SOPHS score	0.5	1, 185.3	0.491
Among at risk: SOPHS score	0.4	1, 61.5	0.544
Among high risk: SOPHS score	4.0	1, 38.0	0.051
Low risk participants*
Professional service use	−1.5		0.134
AQoL utility score	0.1	1, 588.9	0.727
Days out of role	0.5	1, 672.9	0.460
Help-seeking intentions	4.4	1, 681.6	0.036
At risk participants*
Professional service use	1.1		0.276
AQoL utility score	2.9	1, 253.7	0.092
Days out of role	0.5	1, 261.8	0.462
Help-seeking intentions	0.7	1, 279.0	0.399
High risk participants*
Professional service use	−1.5		0.123
AQoL utility score	2.7	1, 113.7	0.102
Days out of role	0.5	1, 130.1	0.498
Help-seeking intentions	0.8	1, 127.3	0.380

AQoL, assessment of quality of life; PHQ, Patient Health Questionnaire; SOPHS, Social Phobia Screener.

a models are adjusted for main effects of time and condition, except for models marked *, which are adjusted for version, time and condition and all 2/3-way interactions between these variables.

b F-tests are based on time × condition interaction terms from linear mixed models; z-tests are based on time × condition interaction terms from binary mixed models. Italic values indicate Z-test. bold values indicate P<0.05.

To further explore the significant findings, service use at baseline and follow-up across versions was tabulated among completers only, as shown in Table DS2, with participants classified as ongoing service users (using professional services at both time points), service use exiters (using services only at baseline), new service users (using services only at follow-up) or non-service users (at neither time point). There were no differences overall in terms of participants entering or exiting treatment within versions, suggesting the effects were general rather than specific to participants already in treatment. Nevertheless, this three-way breakdown of completing participants (condition × disorder focus × risk status) had limited power to find effects.

Discussion

The current trial assessed whether online screening with feedback increased professional service use in a large community-based sample. Results indicated very little benefit of providing tailored feedback based on online screening to promote formal help seeking. If anything, there appeared to be a small negative effect of providing feedback to individuals, with reductions in professional service use among those given feedback about symptoms of social anxiety. Greater attrition and lower intentions to seek help were also observed in this group, suggesting that feedback for social anxiety may actually be detrimental to both help-seeking outcomes and research engagement.

There are a number of potential explanations for these findings, which were contrary to our hypotheses. Participants given feedback that they were at risk of social anxiety were provided with links to online evidence-based programmes for reducing social anxiety symptoms. It is possible that these participants used these programmes and found them beneficial, resulting in less need for traditional face-to-face services. However, there was no significant change in symptoms reported, suggesting that this explanation may not fully account for the study's observations. Unfortunately, use of internet programmes for the treatment of social anxiety symptoms was not assessed in the trial, due to the complexity in differentiating online content that is evidence-based from content that is not based on self-report. Specifically, the public may have difficulty distinguishing different forms of online support and their quality ²⁶ and difficulty in self-reporting which programmes they used, further complicated by a need to account for levels of engagement. Another explanation for the findings may be that social anxiety by its nature may involve avoidance of face-to-face services, ²⁷ with treatment delay common among individuals with social anxiety symptoms. ²⁸ Feedback regarding symptom levels may have inadvertently exacerbated avoidance behaviours. Another explanation may be that the control group may have been prompted to seek help based on their impressions of their responses to the mental health scales, accompanied by the provision of contact information for mental health services at the end of the survey. Providing such information without directive feedback may be less confronting, particularly for individuals with social anxiety.

The finding that participants who received feedback had higher attrition was also unexpected. The mechanism underlying this finding is unclear. It may be that those who did not receive feedback anticipated that participation might lead to additional insight into their mental health, whereas those who received feedback were more satisfied with their participation at the end of the baseline survey when they received feedback. Further examination of this outcome is warranted. Other factors significantly associated with attrition were younger age, poorer quality of life, unemployment, low help-seeking intentions and receiving the social anxiety survey version. Greater adherence among older and employed participants has been observed previously. ^11,29 The effect of quality of life suggests that poor health may be a barrier to research participation, whereas the effect of help-seeking intentions may reflect a propensity towards agreeableness being associated with both help-seeking intentions and survey completion. The version effect (depression v. social anxiety) may be an artefact of the recruitment method; although recruitment for both versions of the survey occurred simultaneously, there were times when recruitment for one version may have been dominant (due to Facebook algorithms and fluctuations in public interest). This may have led to differences in the samples recruited for the two versions of the survey.

This study was the first RCT to test online mental health screening and feedback, and the first to trial the effects of social anxiety feedback. The study benefitted from recruitment of a large community-based sample. However, limitations of the findings should be noted. First, possible factors (e.g. online service use, survey satisfaction) that may impact help-seeking behaviours and engagement with the study were not collected. Additionally, the measures used were based on self-report, which may be prone to response biases and not adequately capture professional service use. Nevertheless, the use of an online survey with limited identifiable information reduced the risk of participants giving socially desirable responses. Second, the service use measure broadly queried use of services for mental health problems, rather than for the specific disorder of interest. Therefore, changes in service use may have reflected mental health problems unrelated to the focus of the intervention.

Third, the sample may have been prone to self-selection biases, with underrepresentation of males and overrepresentation of individuals with mental health problems. Therefore, the findings may not generalise to other forms of online screening, although it might be anticipated that people experiencing mental health symptoms are more likely to self-screen. ¹¹ Fourth, the follow-up period of 3 months was chosen to strike a balance between attrition (longer study periods may result in greater drop-out) ³⁰ and sufficient time to observe changes in service use. Nevertheless, the period may not have been sufficient for many of the participants to demonstrate a change in help-seeking behaviours, given the evidence of long-term treatment delay particularly in social anxiety. ²⁸ In addition, despite the use of robust statistical methods that account for differential attrition, completion rates for the follow-up assessment were suboptimal, although similar to other fully online studies. ³¹ The possibility that attrition was positively associated with help-seeking behaviour remains; future investigation into reasons for attrition may be warranted. Finally, despite the large sample, effect sizes were small, suggesting a number of other factors that are important to service use outcomes were not included in the analyses.

In conclusion, the findings suggest that providing tailored feedback based on online screening may be ineffective for promoting professional service use or for mental health outcomes. These findings echo cautions that there is little evidence to support screening in primary care settings. ³ Effective screening may require embedding screening tools within a mental health service, rather than simply using feedback to encourage service use. However, given the present findings that feedback may be detrimental to service use outcomes and research engagement, clinicians and researchers should be cautious about using screening feedback to support engagement from patients or participants. Further investigation is warranted into other uses of screening, with or without feedback, such as using screening to tailor services and identify specific targets for intervention.