Mixed-methods evaluation of a leadership development programme for higher specialist trainees

Abstract

Aims and method

Newly appointed consultant psychiatrists have reported that management and leadership is an area for which they are unprepared. Our aim was to evaluate the impact of a leadership development programme based on the principles of ‘action learning’ for higher trainees. A questionnaire survey was sent to 54 trainees and consultants who had attended the programme. Qualitative interviews were conducted with 15 participants using the concept of maximum variance sampling.

Results

The relevance of the topics covered, the opportunity for free discussion and increased understanding of National Health Service policy were rated highest. At the end of a thematic analysis, themes were organised into four major categories: (a) lack of engagement with management during training; (b) the lasting impact of the leadership development programme; (c) understanding the larger organisational context; and (d) transition to consultant psychiatrist.

Clinical implications

The findings suggest that programmes such as this can be successful in raising awareness and increasing engagement in medical leadership and management.

Modern healthcare is increasingly complex and management and leadership are now a key part of a doctor's professional work regardless of specialty and setting. It is a requirement as laid out in Good Medical Practice ¹ and Management for Doctors. ² Newly appointed consultant psychiatrists have consistently reported that this is an area of their work for which they feel most unprepared. ³ A report by The Commission for Health Improvement ⁴ identified the difficulties in securing doctors’ interest in management, particularly for those working in mental health, but reported that services do better when doctors get involved.

A review of the literature suggests that trainees are still experiencing difficulties acquiring management skills despite some innovative initiatives. ^5,6 For example, Hewson et al ⁷ and Spurgeon & Vassilas ⁸ introduced management courses. Feedback suggested that these initiatives were well received, but the samples in both studies were relatively small, made up of newly appointed consultants only ⁷ and as the evaluations took place straight after training the authors were unable to examine any lasting impact.

The High Quality Care for All: NHS Next Stage Review report ⁹ pledged to incorporate leadership development and training into both undergraduate and postgraduate curricula. The Medical Leadership Competency Framework ¹⁰ and associated Medical Leadership Curriculum ¹¹ set out a minimum set of management and leadership skills that all doctors now have to acquire to be able to fulfil their practitioner roles effectively. Along with Mental Health: New Ways of Working for Everyone ¹² and the reforms laid out in the Health and Social Care Bill 2011, ¹³ it is likely that the role of the consultant psychiatrist will change again and that the years ahead will be very different. For this reason it is imperative that our future psychiatrists are equipped with the necessary skills to function efficiently and effectively within an ever increasing complex healthcare system. ¹⁴ The aim of this evaluation was to examine the impact of the leadership development programme using a combination of research methods, including a questionnaire survey and qualitative in-depth interviews.

Action learning

Reg Revans ¹⁶ coined the term action learning to heal the split between thinking and doing to help us to understand how human beings can work together more effectively. In the late 1920s as a doctoral student of astrophysics and then a fellow at Cambridge University, he worked alongside eight Nobel Prize winners. These experiences fuelled strong beliefs that underpin the key principles of action learning. These include: (a) learning starts from not knowing; (b) people who take responsibility in a situation have the best chance of taking actions that will make a difference; (c) learning involves both what is taught and read and a questioning insight; and (d) learning should be greater than the rate of change. ¹⁵

Method

Objectives of the management leadership programme

The Mersey Care National Health Service (NHS) Trust specialist registrar leadership development programme was initiated in 2006 in an attempt to raise awareness and increase engagement in medical leadership and management. The programme was run by two of the authors (D.F. and M.C.) so did not involve any additional costs to the Trust. Based on the principles of ‘action learning’ ¹⁵ the programme consisted of bi-monthly meetings for trainees who were in their final 2 years of training in psychiatry, as part of the Mersey Deanery training scheme. Each meeting was facilitated by two of the authors (D.F. and M.C.), and included a guest speaker who would describe their area of work, and then a more structured action learning set would take place to take forward ideas. An action learning approach was encouraged in this way. The guest speakers varied considerably and include a senior manager at a major utility company, various Mersey Care NHS Trust managers (deputy director of finance, director of service user and carer involvement, deputy director of human resources chief executive and clinical director), the medical director for hospices and a journalist. Trainees were also able to lead the discussion as guest speakers on a few occasions. They selected topics connected to leadership and undertook some research before leading the next meeting.

Questionnaire

Based on the findings of Hewson et al ⁷ and Spurgeon & Vassilas ⁸ a questionnaire was drawn up to examine the impact of the leadership development programme. The questionnaire consisted of 15 items broken down into two sections. Section one consisted of ten items on a five-point Likert scale that rated the style, relevance and amount of personal involvement in the programme. Section two consisted of five items on a seven-point Likert scale that covered five aspects of management appreciation.

Qualitative interview

A semi-structured interview schedule was drawn up on the basis of the results from the questionnaire. This would allow additional questions to be asked on the basis of responses given. The schedule consisted of 19 items covering the following domains: (a) perspectives on the leadership development programme; (b) perspectives on management and leadership; and (c) perspectives on the transition to consultant psychiatrist. Interviews were conducted face to face and audio-recorded. The transcripts were subject to a thematic analysis. ¹⁷

Recruitment

One of the authors, C.H., sent the questionnaire electronically and by post to 54 trainees and consultants who had attended the programme. Thirteen had left the Trust and could not be traced. For the qualitative interviews we aimed to recruit a minimum of 15 participants from the 54 who attended the programme using the concept of maximum difference sampling. ¹⁸ This involves deciding beforehand what elements of a population to include in a sample. One of the main advantages of using this sampling technique is that it seeks to uncover a full range of views rather than focus on just typical or average cases. It also ensures that most or all of the perceptions that might be important are uncovered. ¹⁹ For this study the sample would include at least one consultant who had completed the programme: (a) 36 months previously; (b) 24 months previously; (c) 12 months previously; (d) a currently acting consultant; (e) as well as a specialist trainee year 6 (ST6) and year 5 (ST5). Most qualitative studies aim for a minimum of 15 participants. ¹⁹ For the purpose of this study a larger sample would have been too labour intensive and time consuming. A letter of invitation was sent electronically and by post by C.H. to all the trainees who had been sent the survey earlier.

Procedure

As this was an evaluation of an educational initiative formal National Research Ethics Service ethical approval was not required. To improve the response rate to the questionnaire participants were informed that answers would be aggregated and were advised to post back the questionnaire anonymously. The only information requested was on grade. C.H. conducted and transcribed the qualitative interviews. Participants were reassured that the final transcripts would contain nothing that might identify them personally before being viewed by other members of the research team.

Analysis

Data from the questionnaires were entered into predictive analytics software version 18 (PASWS v18 on Windows) and subject to summary statistics and standard hypothesis tests. Data from the qualitative interviews were subject to an inductive thematic analysis. ¹⁷ C.H. and P.K. performed and led the analysis. Through its theoretical freedom, thematic analysis provides a flexible and useful research tool, which can potentially provide a rich and detailed, yet complex account of data. A theme was identified if it captured something important in relation to the interview questions. ¹⁸ To improve the trustworthiness of the analysis the first five transcripts were coded by C.H. and sorted into initial themes. The transcripts were then passed on to the research team for feedback. This involved researchers from different disciplinary backgrounds including two consultant psychiatrists (D.F. and O.W.), a clinical psychologist (P.K.), a speech therapist (M.C.) and a social worker (C.H.). This prevented the personal or disciplinary biases of a single researcher from dominating the analysis. ²⁰ Modifications were made to the interview schedule on the basis of feedback received. ²⁰ The next five transcripts along with initial categories and confirmatory or disconfimatory evidence for initial themes were circulated to the team before embarking on the final five interviews. At the end of the analysis the themes were organised into four major categories through consensus between all researchers. ²¹

Results

Questionnaire

The overall response rate to the questionnaire was 21 responses from a possible 41 respondents (51%); 67% were higher specialist trainees and 33% were consultant psychiatrists. All questions had a 100% response rate.

In section one of the questionnaire, the relevance of the topics covered and the opportunity for free discussion were rated highest, with a mean and median score of four (Table 1). This was followed by the appropriateness of the written material, the opportunity to discuss local services, understanding the management roles of a consultant and an increased understanding of the wider organisational context. The trainees rated questions five and seven higher than the consultants and this difference reached statistical significance (Mann-Whitney U-test P<0.5).

TABLE 1 Section one of the course evaluation ^a

Item	Mean score	Median score
1. How relevant did you find the topics covered in the action learning sets (seminars)	4.0	4
2. How appropriate was the written material provided	3.5	4
3. How useful was it to have had the opportunity to discuss examples of local services	3.8	4
4. How valuable did you find the opportunities for free discussion	4.0	4
5. How engaged were you in the planning and organising of this approach	3.0	3
6. How helpful was the course in terms of enhancing your understanding of the wider organisational       context of mental health services	3.8	4
7. How useful was the action learning approach in developing your understanding of the possible       management roles of a consultant psychiatrist	3.6	4
8. To what degree did the action learning approach create opportunities for you to reflect on your       thinking	3.2	3
9. How useful was the action learning approach in terms of influencing the things that you do now	2.9	3
10. To what extent has action learning changed the way you now work with groups	2.7	3

^a. Score on a 5-point Likert scale (1, not at all; 5, extremely).

In section two of the questionnaire, increased understanding of the NHS policy and awareness of factors affecting the Trust was rated highest (Table 2). The trainees rated question 13 and 15 higher than the consultants and this difference reached statistical significance (U-test P<0.5).

TABLE 2 Section two of the course evaluation ^a

Item	Mean score	Median score
11. Increased understanding of the NHS policy and political context	5.3	6.0
12. Awareness of factors affecting the NHS Trust	5.0	5.0
13. Better grasp of the commissioning process	4.4	5.0
14. Better appreciation of budget management	4.6	5.0
15. Awareness of own strengths and weaknesses	4.4	5.0

NHS, National Health Service.

^a. Score on a 7-point Likert scale (1, strongly disagree; 7, strongly agree).

Qualitative interviews

A total sample of n = 15 consisting of higher trainees and consultants were recruited for the in-depth interviews (Table 3). At the end of the thematic analysis the themes were organised into four major categories.

TABLE 3 Maximum variance sample

Grade	Number recruited
Consultant in post 36 months	3
Consultant in post 24 months	2
Consultant in post 12 months	1
Acting consultant	2
Higher specialist trainee (ST6)	4
Higher specialist trainee (ST5)	3

Theme one: a lack of engagement with management during training

The early years of training were described as being dominated by the need to pass exams and build clinical skills.

‘Until you pass your exams management feels a million miles away.’ (trainee)

The trainees felt that the programme should have been introduced earlier, whereas the consultants felt it would have been more beneficial during their first year in post. As one consultant commented:

‘Until you actually take a post and you sit down there, you don't really know what's going to come. I wish I could do the action learning again now. The sessions would have a real practical application now.’ (consultant)

The need for more hands-on practical management experience was a constant theme for both the trainees and the consultants. Opportunities for this seemed to depend on individual placements and supervisors.

‘If the person mentoring you is quite influential you will get a project where you will be able to develop and flourish in many ways.’ (acting consultant)

Shadowing a manager was viewed as useful but only to a point. Those who had been given the opportunity to locum before taking a consultant post said that it had really helped and suggested building this into the final stages of training, along with formal supervision on management.

‘Because as an SpR [specialist registrar] you know all about the clinical issues, staffing issues. But once you take a locum post, then you know about all the pressures, about management issues. You have to balance everything.’ (consultant)

A number of trainees felt that the Medical Leadership Competency Framework ¹⁰ and Medical Leadership Curriculum ¹¹ too prescriptive and based too much on competencies. The reduction in special interest sessions from four a week to two in which to achieve a number of competencies was viewed as difficult.

‘We need all these things in our portfolio, but it's not explicit how we should go about it. There is a sense that people are just ticking off activities. The role of supervision on placements should be about learning something you can apply on a wider basis.’ (trainee)

Theme two: the lasting impact of the leadership development programme

There were many examples of action learning being incorporated into everyday practice. Respondents talked of sharing their difficulties and learning from and with other colleagues.

‘Much less of a traditional model here where the consultant has all the answers. We share problems and solutions. Shared responsibility of the decision-making. It consolidates team support.’ (consultant)

But this was not always possible because of the expectations placed on consultants from junior staff.

‘When I go to the ward, that sense of hierarchy is much more obvious. People can be quite deferential to you which I find uncomfortable. If it's the wrong decision you make, you can be left feeling quite exposed. If you have debate and other people have an opinion it feels a lot safer.’ (consultant)

Action learning was viewed positively by the majority of participants. The sessions allowed them to practice the skills they would need in the future.

‘It was a good exercise [action learning sets]. They are the type of skills you need as a manager. Being diplomatic and putting things across in a way that helps you find solutions, instead of just complaining all the time.’ (trainee)

The programme had inspired many trainees to pursue higher degrees in health and social care management.

‘I was inspired to take this MBA [Master of Business Administration]. My special interest has been management. The MBA supplements the introduction provided by the action learning.’ (trainee)

However, being a future leader of an NHS organisation was not for everyone. Although it was theoretically important to know about finances, commissioning, etc., it was felt that some people were perhaps not seen as natural leaders.

‘I'm never going to be clinical director material. I still get frustrated with juggling targets and clinical need. I prefer to be. I'm okay with the operational type level of management. I could never be involved at a more strategic level.’ (consultant)

Theme three: understanding the larger organisational context

All the respondents felt the programme had given them a good grounding in the structure of the NHS, management and finances. A number of sessions involving people in higher management from within the local trust were particularly valued. This made the people in higher management more accessible to the trainees and gave them personalities. It also challenged many assumptions that many trainees held.

‘I'd previously been much more quick to jump to conclusions about higher management without really having all the facts. I was finding out that managers were a lot more clued in to what was going on. Quelled some false assumptions.’ (consultant)

Respondents were also very impressed with a number of external non-clinical managers who presented on the programme.

‘The saddest things for doctors, they just talk too much to other doctors. What really helped was people from all different walks of life. The issues to do with managing were common to all professionals, not just psychiatry.’ (trainee)

Theme four: making the transition to consultant psychiatrist

Becoming a consultant generated widespread anxiety, but this tended to reduce over time. The first 12 months were described by all as the most difficult.

‘That was a big shock. People's expectations and the power you have. People look to you for everything. But we can all do it and that's how we survive. After 3 years you just know where you stand.’ (consultant)

Being aware that there is a limitation on how much you can prepare for being a consultant and being open to continual learning on the job seemed to create a degree of resilience.

‘There's a limitation on how much you can prepare for it. You're never fully prepared for the challenges you will face. You need different skills at different stages. Being a manager and leader takes many years to learn.’ (consultant)

One of the most difficult aspects of the transition for all the consultants was going from being supervised to supervising junior doctors. Most felt totally unprepared for this and they found it one of their greatest challenges.

‘You're not responsible for a trainee's behaviour. They are. But that's been difficult for me. Feeling like I have to apologise for things. Like it's a reflection on me. It's an on-going steep learning curve.’ (consultant)

Discussion

The leadership development programme was well received by the trainees and consultants, with the questionnaire scoring highly across most sections. This finding was validated through the qualitative interviews. For a number of respondents the principles of action learning had a lasting impact. There were numerous examples in the interviews of trainees and consultants using the principles in everyday practice and in special projects. For a number of respondents the action learning had whetted their appetite and generated a genuine interest in management and leadership as a future career. This is important because engagement in medical management and leadership has been shown to be associated with better outcomes. ²² Engaging trainees in action learning appears, therefore, to be an approach that should be developed to help lead to higher quality care.

Compared with the findings of Hewson et al ⁷ and Spurgeon & Vassilas, ⁸ the mean and median scores in this study were slightly lower. However, this study was made up of a larger sample size and included both trainees and consultants who had been in post ranging from 12 to 36 months. The trainees rated questions 5, 7, 13 and 15 higher than the consultants, and this difference reached statistical significance. This could suggest a number of things: (a) the trainees value the programme more but this tends to reduce once they move into the consultant role; (b) the programme has changed in some way and newer cohorts are rating it higher; or (c) the trainees themselves have changed in some way.

The findings from the qualitative interviews support the view that the trainees have changed and that the non-clinical management role of the consultant is becoming more widely embraced. But many were still finding it difficult to gain hands on management experience. Having management incorporated into formal supervision might be one solution, but the trainees themselves may also have to be more proactive in creating opportunities to take on more managerial responsibilities. For those with a genuine interest in medical management as a future career, having more dedicated time to enhance skills might be another option.

This study is one of only a small number to systematically examine the extent to which initiatives such as this are preparing trainees for future consultant posts. The findings from this evaluation suggest that leadership development programmes can be successful in raising awareness and increasing engagement in medical leadership and management.