Evidence-based mental health and e-learning: a guide for clinicians

Summary

The internet provides access to what is often a bewildering array of medical knowledge on mental health, some (but not all) of which is evidence based. As well as information for clinicians, there has been a dramatic increase in the variety and quality of information available for patients and carers. In this article we discuss the advantages and limitations of the types of information available, with suggested sites and strategies for assessing their relative merits.

Use of the internet has expanded exponentially over the past decade, increasing worldwide by 806% between 2000 and 2015. The biggest percentage increase over that time has occurred in Africa, but the greatest absolute numbers using the internet are in Asia (48% of total users). By geographic region, North America has the greatest penetrance, with 88% of the population using the internet (Internet World Stats 2015). In the UK in 2014, 83% of homes had internet access, 50% of adults used internet banking and 53% accessed social media sites (Cotton 2014); 93% now have a mobile phone (Ofcom 2015).

The internet is a powerful tool that allows access to multiple types of learning, of which some, but by no means all, use evidence-based information. The term e-learning refers to the use of internet technologies to deliver a broad array of different teaching tools that enhance knowledge and performance; it is also called web-based learning, online learning, distributed learning, computer-assisted instruction and internet-based learning (Ruiz 2006). Many sites are part of blended learning, where e-learning is combined with a more traditional face-to-face instructor-led training, for example using an online tutorial to supplement a lecture (see Table 1 for definitions and examples).

TABLE 1 Useful terms in e-learning

Using the internet to access mental health information

The vast number of information sites on the internet can be overwhelming for both patients and clinicians alike. For example, a simple search via Google (accessed April 2015) using the search terms ‘evidence-based mental health’ yielded more than 60 000 000 results. A specific clinical question such as ‘evidence for exercise in depression’ yielded more than 89 000 000 results. The records retrieved range from reviews of the evidence and primary research articles in peer-reviewed journals to newspaper reports and advertisements for treatment centres. So, the internet user is swamped with an array of sites of variable quality that are neither necessarily pertinent to the original question nor ranked in order of reliability.

Paradoxically, this array of sites and information means that, although clinicians have a large number of questions regarding patient care, they pursue answers to these questions via the internet in only half of cases (Ely 2005). The most common reason for clinicians not researching an answer online is the perception that they will not be able to find it among the multitude of possible information sites (Ely 2005). This is confirmed, as in a quarter of cases when clinicians do look for an answer they cannot find it (Ely 2007).

In addition, the immediate access to knowledge gives the often erroneous impression that the information found is the most up-to-date available. For example, a study in which emergency doctors used internet search engines to answer clinical questions showed that they found the correct answers in only 59% of cases. However, their confidence in the accuracy of their answers was far higher: they believed that 92% of their answers were reliable enough to use in patient care (Krause 2011). A key challenge, therefore, is not establishing whether information is available via the internet, but managing that information in a way that allows access to high-quality, evidence-based data.

Summary websites

Giving clinicians access to user-friendly summaries improves clinical knowledge and reduces error (Hoogendam 2008). The summary websites listed in Tables 2 and 3 are suggested as a starting point for exploration of sites of this type. The lists are not intended to be exhaustive, as websites are rapidly superseded by new ones, and current sites change in content, methodological rigour and the efficiency with which they keep up to date with the evidence base (Banzi 2011). However, despite these reservations, it is helpful for clinicians to have suggestions of websites, and we have divided the listed sites loosely into those directed primarily at clinicians (Table 2) and those directed primarily at the public (Table 3). Inevitably there will be overlap, with many sites providing information for both. Given the changing nature and quality of sites, we would encourage all clinicians to practise ‘evidence-based-ly’ (Cipriani 2014), by applying the tools of evidence-based medicine to assess the quality of sites before using them or recommending them to patients and carers. We return to the problem of quality assessment towards the end of our article, in the section ‘Potential disadvantages of e-learning’.

TABLE 2 A selection of websites that include information on evidence-based mental health aimed at clinicians^a

TABLE 3 Some websites aimed at informing the public about evidence-based mental health^a

Web-based resources can be divided by the format they use to deliver knowledge or by the primary group for which the information has been prepared.

Different e-learning formats

Summary sites

Summary sites (webpages providing a summary of knowledge) are the most static form of internet-based learning about the current state of evidence in mental health. Nevertheless, they are the most easily and frequently accessed first port of call, particularly for clinicians (but also patients) to update themselves quickly. Systematic reviews in the Cochrane Library, National Institute for Health and Care Excellence (NICE) guidance and evidence-based search engines such as the TRIP database (see Table 2 for website addresses) provide a quick and easy reference guide. The advantage of these sites is that the evidence presented has been carefully appraised, synthesised and summarised by qualified experts who follow rigorous methodological guidance and make stringent efforts to avoid bias (Higgins 2011). Some also provide plain language summaries for the public.

Although a quick and (mainly) reliable source, these static websites do not necessarily create a dynamic learning environment for engaging an audience with matters of mental health. The evidence is presented in a concise and technical way that can make for dense reading. Even if regularly updated (at varying time periods depending on the site), there is little real-time interaction or opportunity for dynamic feedback.

Interactive summary sites

In general, learning is greatly enhanced by interaction. For example, a systematic review of continuing education meetings showed that interactive workshops can result in moderately large changes in professional practice, whereas didactic sessions alone are unlikely to change it (Forsetlund 2009). There are a number of information resources that offer interactive features. Examples include the journal Evidence-Based Mental Health, which has started a programme of tweet chats and Google hangouts, the network Students 4 Best Evidence, which regularly posts blogs and social media activities, and the Mental Elf, via comments, social media, note-taking, webinars and gamification. The advantages are that the interaction involved makes learning an engaging activity that can be individualised to each learner's needs. However, as with all interactive modes of learning, particularly in real time, there is limited opportunity for editing of information: the learner's experience relies heavily on the expertise and skills of those who engage in the process.

New ways to access mental health information

As internet access has dramatically increased, so has the role of internet users. Ofcom (2015) reports that 61% of the UK adult population use their mobile phone to access the internet, so the digital divide of access is fast closing. However, a new divide is opening up between those who use the web as consumers (i.e. finding information, using email, online shopping) and those who are contributors (i.e. bloggers, people actively commenting and engaging in public debate, for example via social media).

In the past 5 years we have seen a dramatic increase in the use of blogs and social media by clinicians, researchers and the general public. This democratisation of research has expanded the discussion about new evidence beyond the traditional intellectual and literal paywalls of publishers and academic journals. Access to full-text research remains a barrier to many, but science bloggers now provide an alternative way of keeping up to date with the latest reliable evidence.

Twitter

Nowhere online is this debate more vibrant and immediate than on Twitter, which has become the primary social network for keeping up to date with new mental health research. Twitter subscribers create their own personalised feeds of information by following organisations and individuals who share their areas of interest. Examples include real-time tweet chats (e.g. #WeNurses, #EBMHchat), and live tweeting from conferences (e.g. the Royal College of Psychiatrists’ International Congress #RCPsychIC) and organisations with automated feeds of publications (e.g. @OxPsychiatry). Discussions often arise around specific themes (e.g. using a Twitter hashtag to follow a debate at a conference or via an online journal club) and are usefully archived and organised for posterity.

Webinars

An alternative is the use of webinars, which involve videoconferencing via routes such as Skype, Google Hangouts and WebEx. They are hosted by a range of organisations, some of which are open access (such as the Google Hangouts organised by Evidence-Based Mental Health (http://blogs.bmj.com/ebmh) or the public campfires organised by National Elf Service), whereas others require membership or subscription (such as the BMJ Masterclasses webinars at http://masterclasses.bmj.com/webinars).

Gamification

Game-based learning is an accessible and engaging method of learning. An example is the Mental Elf (www.nationalelfservice.net/mental-health), where members’ website activity is automatically compiled using gamification techniques, so users can learn about new research while progressing through an engaging ‘game’. Educational apps are already a popular way of providing learning to the public (see ‘Monitoring systems and apps’ below) and can also be useful for professionals. For example, the Resuscitation Council (UK) in combination with the production company UNIT9 has developed Lifesaver (www.life-saver.org.uk), accessible via the website or a downloadable app, for professionals and the public to learn cardiopulmonary resuscitation (CPR). Lifesaver is a live-action movie showing real people in real places played like a game with different options for what you might choose to do next. The user learns by doing, as the video shows the positive and negative consequences of each decision made.

Information directed primarily at practitioners/clinicians

Websites providing clinical information

Table 2 lists a selection of websites directed primarily at clinicians. However, the accessibility of the internet means that information is no longer categorised purely as suitable for one group or another (unless the website has a restricted access to a certain group) and many of these sites are used by the public and incorporate lay information and summaries.

Websites aimed specifically at delivering CPD and learning packages

A large number of websites and webpages deliver training and learning packages. Many have been developed to help clinicians meet their continuing professional development (CPD) requirements – either those set by their employer or regulatory organisation, or those they have set themselves for their own development. In terms of delivery of learning, these packages vary enormously. Some use methods similar to a traditional lecture, with a static set of information slides and an assessment at the end, perhaps in the form of multiple choice questions. The other end of the spectrum is represented by multimedia online CPD that may incorporate video and/or audio tools and can be enhanced by reflective learning and the facility for the user to personalise and adapt outputs for their own local environment. For example, the GMC's Learning Disabilities website (www.gmc-uk.org/learningdisabilities) helps health professionals to learn about good and bad practice through interactive video tutorials and reflective learning.

The route of access to learning also varies. Some online CPD can be accessed free, but some is restricted to certain groups (e.g. all doctors registered with the General Medical Council (GMC) can access CPD via www.doctors.net, all members and associate members of the Royal College of Psychiatrists can access Trainees Online (TrON) at tron.rcpsych.ac.uk). Others require an extra subscription (e.g. the Royal College of Psychiatrists’ CPD Online modules at www.psychiatrycpd.co.uk). Other learning is restricted by making it available only to members of a certain National Health Service (NHS) trust, for example by publishing it only on that trust's intranet, and these pages may or may not link to a stand-alone external internet site.

In addition, such web-based teaching is often combined with face-to-face teaching, for example completing online resuscitation learning before a hands-on practical assessment (so-called blended learning).

Online courses aimed at learning the principles of evidence-based medicine

The advantages of e-learning, including its flexibility and accessibility, have also been applied to the teaching of evidence-based medicine (EBM). Traditionally, EBM has been taught through standalone courses, conferences, workshops, journal clubs or educational meetings, but these have been criticised as being insufficiently integrated into daily clinical practice and clinical postgraduate training. A systematic review showed that standalone education improved basic knowledge of EBM, but that the evidence for an improvement in practice was lacking (Coomarasamy 2004). Kulier and colleagues have developed a clinically integrated e-learning course for teaching the basic principles of EBM to postgraduates and compared its effectiveness with that of a traditional lecture-based course of equivalent content (Kulier 2008, 2009; Hadley 2010). They ran the course with different groups of postgraduate trainees in a range of European countries, including the UK, and concluded that it was at least as effective as traditional teaching and was less costly, easier to update and well accepted.

Resources available to, or directed at, patients, carers and the general public

Information websites

Individuals are increasingly making use of online resources to support their own health, either to supplement their knowledge about treatment or instead of seeking help. In the UK, 43% of internet users report having used the web to access health information in 2014, compared with 18% in 2007 (Cotton 2014).

Research studies focusing on mental health information sites for the public have often highlighted poor usability and quality (Reavley 2011). The readability of information provided (the complexity of its vocabulary and syntax) is often set at a high level of difficulty, preventing a significant proportion of patients from understanding it (Gralton 2010), and its quality can be variable (Ferreira-Lay 2008). However, these sites have great potential in providing information to patients about their diagnosis and treatment options and, in response to the demand for more information, an increasing number of higher-quality sites have been developed in recent years. A selection of these is given in Table 3. Some of the listed websites have been approved by The Information Standard (www.england.nhs.uk/tis), a certification scheme for health and social care information established by the Department of Health to help people decide which information is reliable. Organisations achieving Information Standard certification have undergone rigorous assessment to check that the methodology they have used to produce their information is clear, accurate, balanced, evidence based and up to date (note that any user-generated information, such as comments on a page or on services, blogs and forums, is excluded from the scope of certification). It is important to note that Information Standard certification does not guarantee that every piece of information produced by a certified organisation is of high quality; instead it indicates that the methodology used to produce the information is reliable. We would encourage clinicians to assess the quality of any sites themselves before recommending them to patients and carers.

Websites for lay people can also have a significant effect in informing others. The interactive nature of some websites (e.g. via feedback) means that these sites can help to inform professionals and policy makers about the views of patients. For example, healthtalk.org provides information about health issues by sharing people's real-life experiences. It is aimed at the general public, but the patients’ experiences have been used in training health professionals (www.healthtalk.org/learning-and-teaching/health-care-students). A Day In The Life (https://dayinthelifemh.org.uk) is a year-long project collecting the everyday experiences of people with mental health problems on four calendar dates (‘snapshots’) between 2014 and 2015.

Another element has been the development of online communities such as the Elefriends forum run by Mind (https://elefriends.org.uk) and the Big White Wall (www.bigwhitewall.com), which allow members to talk online to other members and gain support and advice.

Web-based treatment programs

Web-based programs can be used to deliver treatments such as cognitive–behavioural therapy to patients. A full discussion is beyond the scope of this article, but this is an area that has expanded exponentially in recent years. However, despite effective interventions, the transition of computerised interventions into care is slow, perhaps because in general patients do not perceive them to be as acceptable as face-to-face treatment (Musiat 2014).

Monitoring systems and apps

The combination of smartphone technology, email and the internet has allowed the development of remote monitoring systems that supply day-today feedback from patients to supplement the clinical consultation and provide active evidence for clinicians and patients to decide together on treatment plans. An example is the True Colours self-management system (https://oxfordhealth.truecolours.nhs.uk/www), developed at the University of Oxford to monitor prospectively those with bipolar disorder. Regular prompts sent via email or text message ask patients to rate their mood (both depressive and manic symptoms) and answer any individualised questions that they or their clinicians wish to be included. Patients can also add events such as medication changes, psychological treatments or life events. Results are summarised in a graph that shows the progression of symptoms and can be reviewed by both the patient and clinician to inform treatment decisions (Bopp 2010).

Mobile phone apps† such as MyJourney (aimed at young people: www.sabp.nhs.uk/eiip/app) and Moodometer (www.2gether.nhs.uk/moodometer-app) allow the public to monitor their own mood and gain a picture of the factors that affect it, as well as accessing advice. ClinTouch (clintouch.com) is an example of an app that allows users to monitor symptoms of psychosis.

Other apps are also available. For example, Doc Ready (www.docready.org) is a digital tool that helps young people to prepare and make the most of mental health-related GP visits. MOMO (www.mindofmyown.org.uk) is an app to help young people express their views more clearly, get involved in meetings and make better decisions with their social care team.

Apps vary in quality, just as websites do, and it can be difficult to assess their relative quality beyond star ratings on retailers’ websites and the subjective feedback of other users. There has been increased interest in developing instruments specifically to classify and rate the quality of apps, and use of scales such as the Mobile App Rating Scale (MARS) can provide an indicator of overall quality as assessed using the subscales of engagement, functionality, aesthetics and information quality (Stoyanov 2015). Websites such as the NHS Choices Health Apps Library (apps.nhs.uk) review apps for quality.

What are the advantages of e-learning?

Accessibility

A key advantage of e-learning is that it can be accessed at any time and is immediately available to anyone with a connection to the internet. Geography is no barrier and, in theory, there is no physical limit to the numbers attending a course or accessing information (Cook 2007). This immediate access to educational materials via e-learning is crucial, as learning is often an unplanned experience, prompted by clinical dilemmas in diagnosis or treatment (Ward 2001).

Transparency and democracy

E-learning can be accessed anonymously, allowing the learning to be more transparent and democratic than in traditional communications. It can inform not only participants, but the wider community and policy makers. For example, in the recent Ebola outbreak, media coverage in the USA and the UK was criticised for being factually misleading and overly focused on the small number of cases outside West Africa. A Twitter chat hosted by The Lancet showed that the use of social media can help to balance the trend for biased reporting: ‘there is no hierarchy on Twitter – anyone can join in, express an opinion, or ask a question. All voices are equal’ (Lancet 2014). In only 1 hour, there were more than 300 000 impressions (the total number of times tweets were viewed), and new ideas emerged from both the experts and the audience. The separation between scientists, health professionals and policy makers was reduced, creating a new diverse community where everyone had a voice and the opportunity to contribute and rate the experience on different parameters.

Immediacy

Online learning is instantly accessible, and the best types of online learning are updated frequently. Updating electronic content is easier than updating printed material (Chu 1998): e-learning technologies allow educators to revise their content simply and quickly.

Feedback is a key element of learning (Forsetlund 2009). This can be synchronous in delivery, where instructors and learners communicate in real time (e.g. internet chat forums and instant messaging) or asynchronous, where learners pace themselves and feedback is delivered via email, online bulletin boards, newsgroups or other technologies (Ruiz 2006).

Individualising learning

On more sophisticated sites, users can individualise their learning to suit their particular needs. Learners can control the content, sequence, pace, time and media used in their learning (Chodorow 1996). At a basic level, users might select only the modules of a course that are relevant to them. With more advanced sites, users may interact with tutors to create their own learning programme. This individualised approach is usually not possible in group face-to-face teaching, where the needs of the group take precedence. However, it is important, because more active engagement and personalisation of the learning experience motivates learners to become more engaged with the content (Clark 2002).

Standards and accountability

E-learning standardises course content and delivery, unlike, for instance, a lecture given to separate sections of the same course. Automated tracking and reporting of learners’ activities reduces the administrative burden. Outcomes assessment can be incorporated, to determine whether learning has occurred. Sites often incorporate automated log-keeping and assessments providing proof of learning for CPD logs. These can be personalised, with individual learning aims created by the user and reflections on the learning achieved.

Use of innovative methods

Innovative methods include the use of virtual patients, multimedia and interactivity. For example, the GMC's Learning Disabilities website helps doctors improve the care they offer people with intellectual disabilities (www.gmc-uk.org/learningdisabilities). The website uses a video of a play about a young woman with Down syndrome who faces problems accessing healthcare. The play highlights bad practice and the potential pitfalls in treating people with intellectual disabilities. The website also contains best-practice examples from the GMC guidance, alongside video interviews with experts in the field (patients and professionals).

Potential disadvantages of e-learning

Variable quality

The quality and regulation of the different sites varies significantly. Accessibility is an advantage, but it can also be a disadvantage when inaccurate or misleading information is rapidly disseminated before it can be corrected (Cook 2007). Users are often left to assess the quality of websites themselves. A number of assessment tools have been developed and, although they vary, most cover one or more of the following: accuracy, completeness, readability, accountability, and design and technical criteria (Reavley 2011). One of the most commonly used is DISCERN (Box 1), a 16-item checklist (Charnock 1999) designed to be used by consumers without content expertise to assess the quality of healthcare information. DISCERN primarily assesses the reliability of information, and acts as a good checklist for this, but it pays less attention to accessibility and usability, which are also essential elements for healthcare information. Although applicable to the internet, it was not specifically designed for internet-based information. Online tools are also available, such as the LIDA instrument, which uses a series of questions to assess the accessibility, usability and reliability of information presented on healthcare websites via a set of free tools to help website developers (www.minervation.com/lida-tool). Codes of conduct have also been developed, including the eHealth Code of Ethics (www.ihealthcoalition.org/ehealth-code-of-ethics) and the Health on the Net Foundation (www.hon.ch).

BOX 1 Summary of the DISCERN checklist of questions to assess the quality and reliability of health information

Section 1: Reliability

• Are the aims clear?

• Does it achieve its aims?

• Is it relevant?

• Is it clear what sources of information were used?

• Is it clear when the information used was produced?

• Is it balanced and unbiased?

• Does it provide details of sources of support and further information?

• Does it refer to areas of uncertainty?

Section 2: Quality of information

• Does it describe how each treatment works?

• Does it describe the benefits of treatment?

• Does it describe the risks of treatment?

• Does it describe what would happen if no treatment is used?

• Does it describe how treatment choices affect quality of life?

• Is it clear that there may be more than one possible treatment choice?

• Does it provide support or shared decision-making?

Section 3: Overall rating

(After Charnock 1999)

It is also important to remember that the use of evidence-based terminology (such as ‘systematic review’, ‘randomised controlled trial’) is not a guarantee of quality. Critical appraisal tools such as the Critical Appraisal Skills Programme (CASP) checklists (www.casp-uk.net) can help readers decide how much weight to give to a particular study or summary.

Usability

Although there is potential for innovative and exciting learning, many websites are poorly designed, with a static interface and little interaction. Setting up high-quality web-based learning can be costly of both money (e.g. funding multimedia additions, actors simulating patients) and time (e.g. maintaining online discussion forums). In addition, the quality may vary. Live discussions such as Twitter chats will only be as good as the quality of the participants and the nominated ‘expert’. Finally, technological problems can mean the teaching is not available, either unintentionally, or because many NHS trusts do not allow access to certain sites or technology.

Conclusions

The internet has great potential as an immediately accessible source of high-quality evidence-based information on mental health, but there are a number of potential pitfalls. To avoid an overwhelming array of sites, most users turn to summary sites for information or interaction. These sites are only as reliable as the methods used to synthesise the information, and so it is the responsibility of the clinician to assess each site (and to continue to do so as more information is added) in order to evaluate the quality and ‘evidence-based’ credentials it claims. Notwithstanding all the potential pitfalls, the advantages of embracing the new digital age in mental health are worth actively pursuing: creative and interactive use of the internet allows the dissemination and discussion of high-quality information in an immediate, instantly accessible and democratic format.

MCQs

Select the single best option for each question stem

1. 1 Which of the following statements is true?

   1. a giving clinicians access to user-friendly summary sites improves clinical knowledge and reduces error

   2. b all ‘evidence-based’ summary sites keep up to date with current knowledge

   3. c DISCERN is a checklist designed only for experienced clinicians to assess the quality of healthcare information

   4. d Information Standard guarantees that all the information on a certified site is reliable

   5. e DISCERN focuses primarily on the usability and accessibility of healthcare websites, rather than the reliability of information.

2. 2 Which of the following statements is true?

   1. a traditional teaching of evidence-based medicine (EBM) is clearly superior to e-learning

   2. b stand-alone education in EBM improves basic knowledge and substantially improves practice

   3. c MARS is an example of a mobile app rating scale that can be used to assess the overall quality of an app

   4. d the presence of evidence-based terminology such as ‘systematic review’ and ‘randomised controlled trial’ proves that an e-learning site is reliable

   5. e immediate access to knowledge via the internet gives the correct impression that the information found is the most up to date available.

3. 3 As regards clinicians using the internet to search for answers to clinical questions:

   1. a it is estimated that they use the internet to search for answers in 80% of cases

   2. b it is estimated that when they use the internet they find an answer in only a quarter of cases

   3. c one study reported that the accuracy of answers was 80%

   4. d one study reported that their confidence in their answers was 50%

   5. e the most common reason clinicians give for not using the internet was the perception that they would not be able to find the answer.

4. 4 As regards internet terminology:

   1. a Web 1.0 are more dynamic user applications than Web 2.0

   2. b blogs, wikis and social network sites are examples of Web 1.0

   3. c RSS is a type of web feed

   4. d MOOCs are online courses, usually charging a fee and restricted in entry requirements

   5. e wikis are static websites written by one expert in the area of interest.

5. 5 As regards e-learning:

   1. a it can also be referred to as distributed learning

   2. b searching the internet produces results that are ranked in terms of reliability

   3. c communicating online (e.g. via a Twitter chat) prioritises the voices of those with recognised expertise over those of the public

   4. d a disadvantage is that accurate information can be disseminated quickly to large numbers of people

   5. e an advantage is that inaccurate information can be disseminated quickly to large numbers of people.

MCQ answers

1

a

2

c

3

e

4

c

5

A