Instrument for the evaluation of higher surgical training experience in the operating theatre

Abstract

Objective

The operating theatre, as the primary learning environment for surgeons, needs to be conducive to achieve successful training. A validated 27-item questionnaire aimed at evaluating the training experience of higher surgical trainees in the operating theatre was developed.

Methods

The initial questionnaire was developed using a literature review and a focus group. Items were validated with content validity index (CVI) and Cronbach's alpha.

Results

The initial version of 33 questions was modified in the focus group into a 29-item 4-point Likert scale questionnaire covering 3 areas. Of these 29 items, 27 reached the threshold CVI of 0.87, and they achieved a Cronbach's alpha of 0.89 from 17 responses.

Conclusion

The quantitative validations in the instrument are comparable to other existing medical education evaluation tools. Aspects of non-technical skills and human factors were featured heavily and perceived to be important for learning in the operating theatre.

Introduction

Surgical training in the UK consists of a well-defined curriculum set by the Intercollegiate Surgical Curriculum Project.¹ In order to meet the outcome of this curriculum and progress through different stages in their training, surgical trainees need to demonstrate competency in specialty-based knowledge, clinical judgement and operative skills at each benchmark review. Given the time constraints of the European Working Time Directive and general service provision within the National Health Service, educational opportunities are limited, and these opportunities need to be utilised strategically.² The operating theatre, being the primary learning environment for trainees to acquire procedural skills, needs to be optimised to ensure effective training can take place.

The operating theatre as a learning environment is well studied in current literature.^3,4 Common themes of such a conducive learning environment include relationship-based mentoring, time availability and structured teaching.⁵ Trainees’ positive perception of the quality of the learning environment is imperative for successful training.^3,6 As such, studies evaluating the learning environment in the form of trainee-led feedback questionnaires have been performed. Notable examples include the Postgraduate Hospital Educational Environment Measure⁷ and the Anaesthetic Trainee Theatre Educational Environment Measure.⁸ An effective feedback instrument will allow trainers to identify the strengths and weaknesses in the learning environment, guide future changes, and ensure successful training.

Regarding feedback instruments for the operating theatre learning environment, the Surgical Learning Educational Environment Measure⁹ was developed by Cassar almost two decades ago and is orientated towards basic surgical trainees.⁹ In this study, we built on this questionnaire; specifically, developed an updated feedback instrument that reflects the evolvement of surgical training in the past 20 years, and tailored this instrument towards higher surgical trainees. This instrument was then validated in a three-stage validation process and is currently utilised in our local specialty training programme.

Methods

Initial development of items

We used the Surgical Learning Educational Environment Measure⁹ as a starting point in the development of our instrument. This questionnaire, developed by Cassar, was principally aimed at basic surgical trainees; we therefore removed the items that are less relevant to higher surgical trainees. Further changes were made after reviewing the Anaesthetic Trainee Theatre Educational Environment Measure,⁸ the Postgraduate Hospital Educational Environment Measure⁷ and the Dundee Ready Educational Environment Measure.¹⁰ The initial items for inclusion in our own questionnaire were then put through a validation process, initially with a focus group.

Focus group

This first version of the instrument was taken to a focus group of locally recruited higher surgical trainees. The focus group was facilitated by the authors. The trainees’ views on the instrument's items were explored. Discussions were focused on a review of the proposed item set, whether any further relevant items should be added and whether the wording of any items needed to be changed. The end product was then taken to the second stage of the validation process.

Content validity and internal consistency

The second version of the instrument was sent out to the local higher surgical trainees and consultant trainers of various surgical subspecialties. These trainees and trainers were asked to review the instrument and determine how relevant each individual item is when assessing the operating theatre learning environment. Each item was scored on a Likert scale of 1 (not relevant, can be excluded from the instrument) to 4 (highly relevant, must be included in the instrument). Two content validity index (CVI) values for each item were independently derived from the scores in the trainee and trainer groups. Seven higher surgical trainees and seven consultant trainers were involved in this process; the acceptable content validity index of a panel of this size is 0.83.¹¹ Only items that reached a cut-off content validity index in the trainee group were included in the third version of the instrument. Content validity indices in the trainer group were used to compare and contrast with the scores in the trainee group.

The third version of the instrument was sent out to the local higher surgical trainees. They were asked to complete the questionnaire regarding their training experience at their current rotation. Each item was scored on a Likert scale of 1 (strongly disagree) to 4 (strongly agree). The scores were reversed in negatively worded items. Cronbach's α was derived from these responses for the overall instrument as well as the three subdomains that the instrument aims to measure: trainer support and supervision, operating opportunities, and operating theatre atmosphere.

Ethical considerations

This study was registered with our hospital clinical governance department and compliant with our institutional ethical guidelines.

Results

Focus group

The first version of the instrument consisted of 33 questions covering 3 subdomains: trainer support and supervision, operating opportunities, and operating theatre atmosphere. Eight higher surgical trainees in otolaryngology were present in the focus group, and the discussion lasted for 27 minutes. Their views of the attributes of a conducive learning environment in the operating theatre were explored. There was consensus that the three subdomains are appropriate qualitative measures and important factors in forming the learning environment in the operating theatre. Five items, thought to be redundant or non-specific, were subsequently removed or integrated into existing items (Table 1). The item ‘I can comfortably express my preference if music is played in theatre’ was added. At the end of the focus group, the instrument consisted of 29 items, and this version was taken to the second stage of the validation process.

Table 1. Items removed in focus group

Content validity index

From the responses collected from higher surgical trainees, 20 items (69.0 per cent) had a content validity index of 1, 7 items (24.1 per cent) had a content validity index of 0.86, 1 item (3.4 per cent) had a content validity index of 0.71 and 1 item (3.4 per cent) had a content validity index of 0.43. The critical content validity index cut-off for a panel of seven is 0.83.¹¹ Item ‘The theatre staff are friendly’, with a content validity index of 0.71, and item 29, ‘I can comfortably express my preference if music is played in theatre’, with a content validity index of 0.43, were removed. At the end of this validation stage, 27 items that had reached the cut-off content validity index from trainee responses were included. From the responses collected from consultant trainers, 27 items reached the cut-off content validity index of 0.83. Item ‘I get bleeped during operations’ had a content validity index of 0.71, and item ‘I can comfortably express my preference if music is played in theatre’ had a content validity index of 0.43.

Cronbach's α

The 27-item questionnaire received 17 responses from the local higher surgical trainees. Cronbach's α of the overall questionnaire was 0.89. Of the three subscales in the instrument, Cronbach's α was 0.81 for trainer support and supervision, 0.77 for operating opportunities, and 0.67 for operating theatre atmosphere.

Scores of final 27 items

The mean score of all the respondents was 62.5 (range, 51–75; standard deviation = 8.60). With strongly disagree converted to 0, strongly agree converted to 3 and reversed scoring on negatively worded items, the minimal and maximal scores obtainable in the instrument with four-point Likert scales were 0 and 81. This derives a mean percentage score of 77.2 per cent in the overall instrument from the respondents. The percentage scores of the three subscales were 84.0 per cent for trainer support and supervision, 69.4 per cent for operating opportunities and 80.7 per cent for operating theatre atmosphere. Item ‘I feel that I have autonomy in theatre’ had the highest percentage score of 90.2 per cent, and item ‘There is too much time pressure on elective lists for me to get the most out of theatre teaching’ had the lowest percentage score of 56.9 per cent.

The final instrument at the end of the validation process consisted of 27 items (Table 2).

Table 2. Instrument for evaluating higher surgical training experience in operating theatre

Discussion

The operating theatre as a learning environment is unique in that it is a high-risk environment and shaped heavily by human factors. Modern surgical training curriculums often incorporate skills laboratory and simulations to navigate this. However, non-technical skills such as team work, leadership and situation awareness are difficult to address in a simulated environment.¹² Compounded by the lack of suitable analogous surgical models for the simulation of procedures, the operating theatre remains the predominant learning environment for trainees to acquire both technical and non-technical skills, despite recent technological advances. Therefore, it is imperative that the operating theatre learning environment is evaluated regularly to address its deficiencies, as perceived by trainees themselves, in order to enhance learner performance.

In our study, items were taken from the widely used Surgical Theatre Educational Environment Measure and discussed in a focus group of higher surgical trainees. Cassar's instrument has been adapted and validated in various studies aimed at different populations.^13–15 In these studies, the subscale structure of the instrument was largely preserved in the factor analyses; hence, it is not surprising that the participants in our focus group agreed on the three existing subscales as being the main constructs in the operating theatre learning environment. Similarly, as in the aforementioned studies, population- and discipline-specific factors precipitated the collapse of items in our version of the instrument, which was aimed at higher surgical trainees. ‘I am too busy doing other work to go to theatre’ was removed, as trainees felt the item was not applicable to them because higher surgical trainees will have protected operating theatre sessions incorporated into their job plan. ‘I am so stressed in theatre that I do not learn as much as I could’ was removed, as trainees felt that this statement did not identify any elements in the learning environment as the source of stress, and could be interpreted as trainees not coping with the programme.

The content validity index and Cronbach's α both indicate that our instrument has high content validity and internal consistency. Two items were excluded from the instrument because they did not reach a critical content validity index after being reviewed by the panel of higher surgical trainees. It is surprising that item ‘The theatre staff are friendly’ did not reach the critical content validity index, as operating theatre staff personalities have been found to be a consistent theme in influencing the operating theatre atmosphere and, hence, the learning environment.^8,14 This could be because the item was perceived to be redundant, as there are already other items addressing operating theatre nursing staff and anaesthetists in the instrument. The friendliness of operating theatre staff was also accessed by item ‘I feel part of a team in theatre’; therefore, the removed item was not assessing a unique element and, hence, would not reduce the overall scope of the instrument. Item ‘I can comfortably express my preference if music is played in theatre’ had the lowest content validity index of 0.43 in both the higher surgical trainee panel and the consultant trainer panel. Interestingly, this was an extra item added upon discussion with the focus group in the initial stage of the study. Participants felt that music improves the calmness and atmosphere of the operating theatre. Current literature supports this view, but has suggested that music can also be distracting and cause impaired communications.¹⁶ Studies have also shown that surgeons are the most empowered group amongst operating theatre staff when it comes to choosing music.¹⁷ However, the importance of this was not recognised by either panel in our study.

• • The operating theatre is the primary learning environment for trainees to acquire procedural skills and needs to be optimised to ensure effective training

• • A 27-item questionnaire was developed to evaluate the training experience of higher surgical trainees in the operating theatre

• • The content validity index and Cronbach's α indicated that the instrument has high content validity and internal consistency compared to similar evaluation questionnaires

• • Three themes are perceived to be important for higher surgical training: trainer support and supervision, operating opportunities, and theatre atmosphere

• • The instrument is applicable across surgical subspecialties and can be easily adapted for other training programmes

Cronbach's α of the final 27-item instrument was 0.89. This is comparable to similar established instruments, such as the Surgical Theatre Educational Environment Measure (Cronbach's α = 0.877)⁹ and the Postgraduate Hospital Educational Environment Measure (Cronbach's α = 0.899).¹⁸ When looking at individual Cronbach's α values in the three subscales, these values are also in the same range as studies that broke down the Postgraduate Hospital Educational Environment Measure into individual subscales.¹⁹

Although our study aimed to develop and validate a feedback instrument, the initial responses from higher surgical trainees were also analysed in order to provide a current view of the operating theatre learning environment. The mean score of our study population (77.2 per cent) was similar to the mean score of the Surgical Theatre Educational Environment Measure (74.4 per cent).⁹ Similarly, operating opportunities had the lowest subscale score compared with the other subscales. This may reflect that providing opportunities for trainees to operate is the most challenging aspect to optimise in the learning environment, as this is dependent on multiple factors, such as variety of cases and time pressure.

A conducive learning environment has been shown to be associated with improved learner performances and can help develop trainees’ surgical confidence.²⁰ By regularly evaluating how trainees perceive their training experiences in the operating theatre, programmes can identify areas that need to be improved in order to ensure successful training. The limitation of our instrument lies within the validation process, as only local higher surgical trainees were involved, and this could incur a selection bias. Factors and items perceived to be important could be population-specific, making the instrument only applicable to surgical training programmes with similar curriculum structure and work culture. This is particularly relevant in the UK with the introduction of the revised curriculum in August 2021, as surgical training across subspecialties has been largely unified with Generic Professional Capabilities and Capabilities in Practice.²¹ Nonetheless, the transparent development and validation process depicted in our study can be easily adapted by other institutions or specialty training programmes, thereby creating a feedback instrument that is targeted at clinical learners at different levels and the wider clinical learning environment.