Evaluating PsychEd: a mental health and well-being educational programme for secondary schools

Abstract

Background

Young people's mental health and well-being is an important concern in the UK. Provision of education and support to schools has been highlighted as an area for improvement; however, evidence-based programmes are scarce and costly.

Aims

To provide an acceptable education programme to improve pupils' confidence and knowledge of mental health and well-being. It covered the mental and emotional well-being outcomes set by the Scottish Government in their schools' curriculum.

Method

Lessons were designed by A.P. and delivered by volunteer doctors and medical students, and supervised by a psychiatrist. Outcomes were measured using questionnaires before and after lessons with optional comments.

Results

PsychEd was piloted in 2016 in six schools to pupils between the ages of 11 and 18. There was a statistically significant improvement in pupil confidence and knowledge after the lessons (P < 0.001). Of the pupils 84% felt that having lessons on mental health was useful. Qualitative feedback was also collected and coded into positive, constructive and negative comments. In total, 72% of pupil comments were positive.

Conclusions

PsychEd showed promising results. Future areas of development include reaching a greater number of local authority schools, longer training for volunteers and provision of teaching materials to teachers for their own use.

Declaration of interest

None.

The mental health and well-being of young people in the UK is a growing concern with 50% of mental health problems stablished by the time young people turn 14 and 75% by the age of 25.¹ Almost 12% of 11–16 year olds have a diagnosed mental disorder with over half a million (approximately 5%) of under-18 year olds referred to specialist psychiatric care in NHS England in 2016–17.^2,3 English children's satisfaction with their life ranked below Poland and Nepal in a recent British Medical Association survey with suggested potential causes for this such as: social media, austerity measures, loss of community and increased stress in schools.^4,5 Third sector surveys have found that 66% of teachers think they lack adequate training in mental health, 91% report mental health problems are worsening in their students and 84% of schools feel ‘unequipped to cope with [pupil] mental health problems’.^6,7 Stormont et al in 2011 found that teachers are also often unaware of the local resources available to them, which makes choosing evidence-based interventions difficult.⁸ The Scottish Government and the UK Government have both released strategies calling for reforms in mental health education and teacher training.^9,10 The UK Government have also pledged £300 m to appoint a mental health lead in every school with increased training for teachers.¹¹

In line with these strategies A.P. developed PsychEd in 2016. PsychEd is a pilot educational programme aimed at secondary school pupils, designed to cover the Scottish Curriculum for Excellence's mental and emotional well-being outcomes. These outcomes focus on identifying and understanding emotions, developing positive coping strategies and knowing where to seek help for oneself and others.¹² The aims of the PsychEd programme were therefore as follows: (a) create lessons that are positively received by pupils and teachers; and (b) increase pupil confidence and knowledge by giving them the information and tools necessary to look after their own mental health and well-being and support others.

Method

Participants

All 47 secondary schools in Greater Edinburgh were invited to take part in the programme, both local authority (managed by City of Edinburgh Council) and independent. Special schools were not approached. School head teachers were contacted by letter. Seven schools expressed interest and six took part. These were Boroughmuir High School, George Heriot's School, Merchiston Castle School, St. George's School, Clifton Hall School and Balerno High School. All schools consented to the collecting of individual, anonymised feedback from pupils and teachers. Students were offered the chance to give feedback but this was optional. According to the policy activities that constitute research at the University of Edinburgh, this work met criteria for operational improvement activities exempt from ethics review.

The majority of the schools' leads felt that the programme would be best delivered to 13- to 16-year-olds in their third and fourth years of school. The reasoning for this was that in Scotland these two years are preparation for pupils' first set of national exams and the schools felt that the programme would have a positive effect on exam stress and pupil well-being. One school requested lessons for 12- to 13-year-olds (second year of school), one asked for an additional special session for 17- to 18-year-olds (last year of school) and one school requested lessons for all their first to fourth year pupils (11- to 16-year-olds).

Development of teaching materials

In accordance with the aims mentioned above, the teaching materials were designed to meet the Scottish Curriculum for Excellence outcomes for mental, emotional and physical well-being.¹² Example materials were created by the lead author. These consisted of PowerPoint presentations, designed to last 20 min, and interactive material. The presentations introduced the subject, focusing on basic scientific background and facts. Activities were designed around three areas: understanding concepts, self-help with a focus on mindfulness and cognitive behavioural models and ways to help others. All presentations concluded with a summary of what to do in an emergency situation and a list of helpful resources. In total 17 senior teachers with a background in pupil guidance reviewed the materials, chose components they felt to be helpful and clarified which age groups they wanted to target.

Schools were offered materials that covered self-harm, substance misuse (with a focus on drug-induced psychosis), eating disorders, anxiety, depression, schizophrenia, personality disorders and bipolar affective disorder. The main areas requested were anxiety and general mental well-being. One school requested a special lesson on sleep and one wanted two sessions on psychosis for their school leavers. Materials were adapted after discussions with teachers and tailored to meet individual school requests.

Programme delivery

Delivery of the lessons at schools was accomplished by a trained group of volunteers, both doctors and medical students. These were recruited from NHS Lothian and the University of Edinburgh over a period of 2 months. All volunteers had clearance for working with adults and children under the Scottish Protecting Vulnerable Groups scheme.

Volunteers were recruited in a variety of ways. Student medical societies were contacted and asked to advertise volunteering to all their members. In addition, an online advertisement was placed on the medical students' online portal. Doctors were approached via internal email and a presentation at their mandatory work induction. In total 42 students and doctors volunteered and attended training sessions.

All school sessions had one doctor as the lead tutor. Lessons were delivered to a class of on average 20–30 pupils and were led by volunteers with a teacher present throughout. School sessions lasted between 40 and 50 min in line with the periods of the schools visited.

Evaluation of the programme

An evaluation tool was created with the advice of a professional evaluator, responsible for evaluating the initial success of Computing at School, a UK wide initiative currently at 1767 schools that promotes the teaching of computer science in collaboration with British Computer Society, the chartered institute for information technology.

Feedback gathered was both quantitative, to provide specific information relative to our objectives, and qualitative in order to gain an overarching view of the programme and how it was received. An initial questionnaire was developed for pupils. This was designed to be short to encourage compliance in completion and focused on knowledge, confidence and ability to help and support peers. There were three questions in total, which were asked before the lesson was delivered and then repeated afterwards. Two of these used a Likert five-point scale and one had a yes/no/maybe option. Space for free-text feedback was also provided.

Consideration was given to the fact that pupils could disclose an issue that required further input, such as an illness or child protection issue. Pupil feedback was optional and all pupils were encouraged to talk to their teacher if they had concerns. Teachers were responsible for collecting the feedback forms. After the session all teachers were invited to provide anonymous qualitative feedback.

Analysis

Analysis was performed by A.P. The Wilcoxon signed-rank test was used to look for overall differences before and after the programme in terms of pupils' confidence, knowledge and attitude to the lessons. Spearman's rank correlation was also used to explore relationships between age groups, pupils' confidence and knowledge level.

Qualitative feedback was evaluated by looking for patterns and themes. Words were coded into positive, negative and ambivalent feedback to allow comparison. Then the words used in each category were analysed separately. Word clouds were also used to identify common words and ideas. Feedback from teachers and pupils was also compared with identify commonalities and discrepancies. Any anomalies in the feedback were considered in light of the year group, the school, the lesson, the tutor and, in the case of the pupil feedback, the quantitative scoring linked to it.

Results

The pilot programme was delivered in six schools over a period of 1 year, with 32 lessons and 1052 pupils. In total 379 questionnaires were returned and 6 were discarded because of incomplete answers. This resulted in 373 complete questionnaires with 127 pieces of written feedback from pupils and 31 from teachers. The results below are separated into quantitative and qualitative sections.

Quantitative analysis

Participant characteristics

Of the respondents, 70% were from independent schools. Apart from Year 5, who received no classes, all other secondary school years are represented ranging from Year 1 to Year 6. In total, 53.6% of all pupils received lessons focused on mental well-being with the two other major subjects covered being anxiety and psychosis. Year 6 had the largest number of pupils who returned feedback (30.8%). Years 1–4 had a feedback-return rate of 14–21%.

Descriptive statistics

Descriptive statistics in Table 1 show that the mean score improved for both knowledge and confidence among pupils after lessons. The majority of the pupils also felt that it was important to have lessons on mental health and were glad that they had had the lesson afterwards.

Table 1 Comparison of before and after questionnaire scores^a

IQR, interquartile range; W, Wilcoxon rank-signed test.

a. Questions 1 and 3 have scales of 1–5; question 2, 1–3. Significant values are in bold.

Table 1 shows a significant difference in knowledge and pupil confidence before and after the lessons. Pupils' feelings regarding the lesson do not change, with the majority of students glad that they had received the lesson. This was replicated individually in the two most common lesson groups – mental health and well-being (confidence: W = 313.5; knowledge: W = 471. P < 0.001) and psychosis (confidence: W = 147; knowledge: W = 75. P < 0.001) – which accounted for 81% of the entire sample.

Significant associations are found between pupils' self-reporting of knowledge and confidence before the lesson (ρ = 0.32–0.45; P < 0.001) in all groups. This relationship is preserved after the lesson (ρ = 0.35–0.51; P < 0.001). Pupils' belief in the importance of mental health teaching is neither correlated to their self-reported knowledge nor their confidence. No statistically significant correlation was found between pupils' confidence, knowledge or opinion of mental health lessons and their school year.

Confidence

Pupil confidence significantly improved after receiving the lesson. Figure 1 compares the overall scores before and after. Overall the greatest differences were seen in Years 4 and 6. Year 3 had the highest initial self-reported confidence with the smallest overall increase whereas Year 2 felt their confidence was lowest before the lesson. Years 2 and 3 were the only school years where nobody initially rated their confidence below a 2. Of Year 4 pupils, 10.9% initially rated their confidence at a 1, in comparison with 1.9% of Year 1 pupils and 4.3% of Year 6 pupils.

Fig. 1 Pupil confidence before and after lesson.

After the lesson the highest reported confidence was in Year 3 whereas the lowest remained in Year 2. Year 4 was the only year where there was a rating of 1 in confidence post-lesson. The greatest number of responses with a rating of 5 post-questionnaire was returned by Year 3, where the proportion doubled to 52%; however, the largest overall increase in ratings of 5 was in Year 2.

Knowledge

Figure 2 compares pupils' self-reported knowledge before and after the lesson. Overall the greatest differences were seen in Years 1 and 4. The smallest increase was in Year 6. Year 2 was the only school year where nobody initially rated their knowledge below a 2, andh only Year 3 had any responses of 5 prior to the lesson. In total, 17.65% of Year 1 pupils initially rated their knowledge at a 1. For Year 4 pupils, in comparison, this was 12.63% of pupils rated their knowledge at a 1; the figures for Year 6 and Year 3 pupils were 3.48% and 5.33%, respectively.

Fig. 2 Pupil knowledge before and after lesson.

After the lesson the highest reported knowledge was in Year 2 with the lowest in Year 1 (despite them showing the greatest improvement). Years 1 and 4 each returned a rating of 1 after the lesson. The rest of the years had minimum scores of a rating of 2 with only Years 1–3 having any responses of a rating of 5. The greatest number of responses of a rating of 5 post-questionnaire was again returned by Year 3, with 4%; overall the most common response across all the years after the lesson was a rating of 4. Across all year groups initial scores for knowledge were lower than confidence, however, they showed greater improvement post-lesson.

Utility

Pupils did not show any significant difference in their answers before and after the session (P = 0.3–0.5). There was no correlation between their beliefs about lessons and their knowledge or confidence. Overall 84% pupils felt that having lessons on mental health was useful with 80% glad that they had had the session. The concordance before and after was 84% for ‘yes’, 20% for ‘no’ and 20% for ‘not sure’. The biggest change in answer before and after was seen from ‘yes’ to ‘not sure’ with 15.6% of the 320 initial ‘yes’ respondents changing. One out of the initial five ‘no’ answerers repeated their answer at the end of the session, whereas three respondents changed their after response to ‘yes’.

Qualitative evaluation

In total there were 127 pieces of qualitative feedback gathered from pupils and 31 from teachers. These were coded for initial comparison. All feedback was split into positive, constructive and negative. There was one piece of feedback that did not fit into one of these categories: ‘ponies don't ply’. This was treated as anomalous and excluded. The majority of comments from both teachers and pupils were positive.

Positive feedback

Of the pupil feedback, 72% was considered to be positive. These comments were analysed to look for commonalities in word use and themes. These are presented in a word cloud (Fig. 3). The five most commonly used words (with the frequency of each in parentheses were): good (24 references), helpful (22), lot (13), interesting (12) and learn (12).

Fig. 3 Word cloud representing positive pupil feedback.

Positive teacher feedback represented 57% of their comments. Lesson (13), good (5), pupils (4), feedback (3) and love (3) were their five most commonly used words (with the frequency of each in parentheses). The main positive theme was that pupils had enjoyed learning something new, which they thought had been helpful. According to teacher feedback, certain parts of the lessons were particularly interesting, such as the interactive components. The pupil themes differed slightly from the teacher feedback that focused on good delivery of lessons and the relevance of the subject material. This was particularly positive if they felt it tied in with other lessons being taught in school.

Constructive feedback

The second most common category for feedback was constructive, making up 18% and 29% of feedback from pupils and teachers, respectively. Constructive feedback from pupils was almost entirely focused on additional information they would have liked to have had included. This ranged from longer sessions to separate topics. All the responses from pupils requested something different and there was no commonality. The second theme was a criticism of interactive work – either the mindfulness session or psychosis session. Four respondents felt that the instructions could have been clearer with one also pointing out that there was not enough time to engage in mindfulness properly. Finally, in 3 of the 23 responses was the idea that the lessons, although helpful, had sometimes touched on subjects that made the pupils feel slightly upset or anxious. The feedback from pupils was spread across the different lessons with no particular focus.

Teacher feedback mainly focused on lesson delivery. This was accounted for by two lessons delivered on anxiety to Year 3 pupils at two different schools. Four of the six respondents felt that parts of the delivery could have been improved, such as: classrooms allocated for the lesson, tutor confidence and the layout of the session. The other two respondents felt that it would be better to have first-hand accounts from the tutors of mental health problems they themselves had experienced as this would make the lessons more relatable for their pupils.

Negative feedback

The negative feedback from teachers and pupils accounted for the smallest part of the comments received. Pupils' feedback centred on the lesson length, expressing that they struggled to concentrate or absorb all the information. The rest focused on one particular lesson, where they felt the volunteer looked anxious. This meant that they lost confidence in what he was saying.

Teachers gave three pieces of negative feedback. All three centred on lesson delivery and two of these were from one lesson where they felt that the tutor lacked confidence and was not well prepared. This was the same lesson as the pupils had commented on negatively above. The final comment concerned a misunderstanding in communication. When the pupils' qualitative feedback was linked to the quantitative feedback, 4 out of the 13 comments still gave the lesson a ‘yes’ for utility after it finished. One pupil responded ‘no’ and the rest were ‘not sure’. All of the pupils rated their confidence and knowledge as improved despite this.

Discussion

This study was designed to evaluate the success of the PsychEd programme by assessing attitudes to mental health lessons and to see if pupil education about mental health and well-being increases their knowledge and confidence in helping themselves and others.

The findings support the construct validity of the questionnaire used and the reliability of the programme in delivering lessons in mental health. Congruence was found across all lessons and age groups with statistically significant increases in pupil confidence and knowledge, independent of the lesson delivered or the tutor giving it. This provides strong preliminary evidence for using PsychEd lessons on a wider scale.

The third dimension measured was pupils' perceptions of the utility of mental health lessons. This was where the biggest differences in single answers were found. Although these were not classed as significant, 16% of pupils who had initially stated that it was good to receive such lessons were unsure after the lesson whether it had been useful. This tallied with some of the qualitative feedback where some students requested more time for lessons or the inclusion of different topics. It highlighted the importance of asking pupils what topics they want to cover.

As was mentioned in the introduction there has been much publicity on mental health provision in schools. Lessons and pupil engagement are only one part of that and it is difficult to provide a ‘one size fits all’ approach. These findings support the idea that mental health education can be beneficial throughout school, rather than as a one-off approach during a specific year group. The lessons were adapted to be more complex as the pupil ages increased to allow for students' understanding of science and peer-to-peer relationships. Our findings supported including lessons from as early as Year 1 at secondary school; however, there is a need for further research into adapting approaches for younger children.

Similarly, research has shown that integrating mental health and well-being into the whole school ethos is beneficial in the long term.¹³ PsychEd was limited in provision by what specific schools requested and returned on average only once per year to give classes again. Schools' general ethos varied depending on funding and availability of guidance staff. Despite this, our findings show that a small intervention can be positive and suggests that many pupils find these sessions engaging.

Strengths and limitations

The highly statistically significant improvements in knowledge and confidence are a clear strength of this study. These were seen independent of age or lesson covered and were also backed up by qualitative feedback received from teachers and pupils. There are several key elements to the PsychEd programme that may have helped to facilitate these results.

First, the school's ability to choose which lesson(s) they felt would be most beneficial to their students was a clear advantage; both pupils and teachers reported favourably on the material covered. Given the diversity of pupils and schools, this was a particularly important facet of the programme and is more suitable than a ‘one size fits all’ approach. Our findings showed that there was significant improvement regardless of lesson chosen. Other research has also supported this idea; MindMatters, a programme piloted in 24 Australian schools that provided educational resources, also found positive results when they modified their programme according to school needs.¹⁴

Second, the programme was offered to all students, rather than singling out a small section already identified as ‘at risk’. It focused on positive aspects of mental well-being and the importance of these in everyday life, rather than prevention. Previous studies have evaluated programmes in children with difficult behaviour or disadvantaged backgrounds as preventative measures from future illness, substance misuse or antisocial behaviour.¹⁵ Although these are useful they ignore the greater societal benefits of mental health promotion, such as: reduction in stigma, contribution to the community and ability to manage life stressors.¹⁶

The present study also has limitations. First, the study sample consisted of a majority of independent schools, as they expressed interest. This could have increased the potential for selection bias as schools with positive attitudes to mental health might have taken more interest. These schools are already well funded with a variety of programmes at their disposal whereas other local authority schools in less well-off economic areas might benefit more from the lessons.

Second, 69.2% of lessons were delivered by different tutors with different backgrounds and knowledge levels (the exception was psychosis, which was taught solely by A.P.). This led to one example of a lesson receiving more negative feedback from teachers and pupils because of the lack of confidence and perceived anxiety of the tutor. One possible adjustment to this would be the inclusion of tutors who were willing to discuss their personal experience of mental health first hand. This was requested by some teachers in the qualitative feedback. Schachter et al found that introducing teenagers to people with mental health issues and opening frank discussion reduced stigma overall.¹⁷ This might have led to a more positive experience for pupils with this particular tutor. Overall, however, it is unlikely this significantly skewed results; both lessons on mental health and well-being (12 tutors) and psychosis (1 tutor) showed improvement.

Third, the questionnaire used only focused on pupils' attitudes to the lessons immediately before and after their delivery. Ideally a follow-up questionnaire would also be added to see whether pupils' confidence and knowledge of mental health is sustained after several months. This might provide support for more regular lessons being interwoven into the school schedule. Finally, the evaluation of the PsychEd programme was entirely subjective. Although pupil and teacher perceptions are crucial to evaluating the utility of a programme it is important to note that they may have been influenced by other external factors such as school ethos, timing of lesson and importance placed on such lessons by regular teaching staff. Previous research has noted that any perceived stigma of teachers and schools to mental health significantly influences teenagers in their attitudes and awareness of mental health issues.¹⁸ It is unlikely that this would have been an issue in these lessons, as they were requested by the teachers who were present; however, it could affect lessons delivered as part of a wider programme in schools where less importance might be placed on mental health and well-being.

Future areas of development

PsychEd was an initial pilot project and it is important to consider how it could be sustained and expanded in the future. Since 2016 the Scottish Government have released their Mental Health Strategy 2017–2027, which places importance on the reform of personal and social education.¹⁹ This requires increased teacher confidence and evidence-based teaching materials for schools. Our results support the utility of our teaching materials and tutor training. Future developments could include creation of an online portal that would allow dissemination of materials and training videos. This would enable a second, larger-scale study covering a broader range of schools and geographical areas. It would also serve as a useful precursor to anyone considering formal qualifications in mental health.