Hostname: page-component-78c5997874-g7gxr Total loading time: 0 Render date: 2024-11-02T18:59:55.018Z Has data issue: false hasContentIssue false

Instrument for the evaluation of higher surgical training experience in the operating theatre

Published online by Cambridge University Press:  07 July 2022

K L Lau*
Affiliation:
Department of Otolaryngology – Head and Neck Surgery, Sunderland Royal Hospital, Sunderland, UK
R Scurrah
Affiliation:
Department of Otolaryngology – Head and Neck Surgery, Sunderland Royal Hospital, Sunderland, UK
H Cocks
Affiliation:
Department of Otolaryngology – Head and Neck Surgery, Sunderland Royal Hospital, Sunderland, UK
*
Author for correspondence: Dr Kin Lun Lau, Department of Otolaryngology – Head and Neck Surgery, Sunderland Royal Hospital, Kayll Road, Sunderland SR4 7TP, UK E-mail: [email protected]
Rights & Permissions [Opens in a new window]

Abstract

Objective

The operating theatre, as the primary learning environment for surgeons, needs to be conducive to achieve successful training. A validated 27-item questionnaire aimed at evaluating the training experience of higher surgical trainees in the operating theatre was developed.

Methods

The initial questionnaire was developed using a literature review and a focus group. Items were validated with content validity index (CVI) and Cronbach's alpha.

Results

The initial version of 33 questions was modified in the focus group into a 29-item 4-point Likert scale questionnaire covering 3 areas. Of these 29 items, 27 reached the threshold CVI of 0.87, and they achieved a Cronbach's alpha of 0.89 from 17 responses.

Conclusion

The quantitative validations in the instrument are comparable to other existing medical education evaluation tools. Aspects of non-technical skills and human factors were featured heavily and perceived to be important for learning in the operating theatre.

Type
Main Article
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press on behalf of J.L.O. (1984) LIMITED

Introduction

Surgical training in the UK consists of a well-defined curriculum set by the Intercollegiate Surgical Curriculum Project.Reference McKee1 In order to meet the outcome of this curriculum and progress through different stages in their training, surgical trainees need to demonstrate competency in specialty-based knowledge, clinical judgement and operative skills at each benchmark review. Given the time constraints of the European Working Time Directive and general service provision within the National Health Service, educational opportunities are limited, and these opportunities need to be utilised strategically.Reference Parsons, Blencowe, Hollowood and Grant2 The operating theatre, being the primary learning environment for trainees to acquire procedural skills, needs to be optimised to ensure effective training can take place.

The operating theatre as a learning environment is well studied in current literature.Reference Lyon3,Reference Croghan, Phillips and Howson4 Common themes of such a conducive learning environment include relationship-based mentoring, time availability and structured teaching.Reference Kieu, Stroud, Huang, Smith, Spychal and Hunter-Smith5 Trainees’ positive perception of the quality of the learning environment is imperative for successful training.Reference Lyon3,Reference Nordquist, Hall, Caverzagie, Snell, Chan and Thoma6 As such, studies evaluating the learning environment in the form of trainee-led feedback questionnaires have been performed. Notable examples include the Postgraduate Hospital Educational Environment MeasureReference Chan, Sum, Lim, Chew, Samarasekera and Sim7 and the Anaesthetic Trainee Theatre Educational Environment Measure.Reference Holt and Roff8 An effective feedback instrument will allow trainers to identify the strengths and weaknesses in the learning environment, guide future changes, and ensure successful training.

Regarding feedback instruments for the operating theatre learning environment, the Surgical Learning Educational Environment MeasureReference Cassar9 was developed by Cassar almost two decades ago and is orientated towards basic surgical trainees.Reference Cassar9 In this study, we built on this questionnaire; specifically, developed an updated feedback instrument that reflects the evolvement of surgical training in the past 20 years, and tailored this instrument towards higher surgical trainees. This instrument was then validated in a three-stage validation process and is currently utilised in our local specialty training programme.

Methods

Initial development of items

We used the Surgical Learning Educational Environment MeasureReference Cassar9 as a starting point in the development of our instrument. This questionnaire, developed by Cassar, was principally aimed at basic surgical trainees; we therefore removed the items that are less relevant to higher surgical trainees. Further changes were made after reviewing the Anaesthetic Trainee Theatre Educational Environment Measure,Reference Holt and Roff8 the Postgraduate Hospital Educational Environment MeasureReference Chan, Sum, Lim, Chew, Samarasekera and Sim7 and the Dundee Ready Educational Environment Measure.Reference Roff10 The initial items for inclusion in our own questionnaire were then put through a validation process, initially with a focus group.

Focus group

This first version of the instrument was taken to a focus group of locally recruited higher surgical trainees. The focus group was facilitated by the authors. The trainees’ views on the instrument's items were explored. Discussions were focused on a review of the proposed item set, whether any further relevant items should be added and whether the wording of any items needed to be changed. The end product was then taken to the second stage of the validation process.

Content validity and internal consistency

The second version of the instrument was sent out to the local higher surgical trainees and consultant trainers of various surgical subspecialties. These trainees and trainers were asked to review the instrument and determine how relevant each individual item is when assessing the operating theatre learning environment. Each item was scored on a Likert scale of 1 (not relevant, can be excluded from the instrument) to 4 (highly relevant, must be included in the instrument). Two content validity index (CVI) values for each item were independently derived from the scores in the trainee and trainer groups. Seven higher surgical trainees and seven consultant trainers were involved in this process; the acceptable content validity index of a panel of this size is 0.83.Reference Polit, Beck and Owen11 Only items that reached a cut-off content validity index in the trainee group were included in the third version of the instrument. Content validity indices in the trainer group were used to compare and contrast with the scores in the trainee group.

The third version of the instrument was sent out to the local higher surgical trainees. They were asked to complete the questionnaire regarding their training experience at their current rotation. Each item was scored on a Likert scale of 1 (strongly disagree) to 4 (strongly agree). The scores were reversed in negatively worded items. Cronbach's α was derived from these responses for the overall instrument as well as the three subdomains that the instrument aims to measure: trainer support and supervision, operating opportunities, and operating theatre atmosphere.

Ethical considerations

This study was registered with our hospital clinical governance department and compliant with our institutional ethical guidelines.

Results

Focus group

The first version of the instrument consisted of 33 questions covering 3 subdomains: trainer support and supervision, operating opportunities, and operating theatre atmosphere. Eight higher surgical trainees in otolaryngology were present in the focus group, and the discussion lasted for 27 minutes. Their views of the attributes of a conducive learning environment in the operating theatre were explored. There was consensus that the three subdomains are appropriate qualitative measures and important factors in forming the learning environment in the operating theatre. Five items, thought to be redundant or non-specific, were subsequently removed or integrated into existing items (Table 1). The item ‘I can comfortably express my preference if music is played in theatre’ was added. At the end of the focus group, the instrument consisted of 29 items, and this version was taken to the second stage of the validation process.

Table 1. Items removed in focus group

Content validity index

From the responses collected from higher surgical trainees, 20 items (69.0 per cent) had a content validity index of 1, 7 items (24.1 per cent) had a content validity index of 0.86, 1 item (3.4 per cent) had a content validity index of 0.71 and 1 item (3.4 per cent) had a content validity index of 0.43. The critical content validity index cut-off for a panel of seven is 0.83.Reference Polit, Beck and Owen11 Item ‘The theatre staff are friendly’, with a content validity index of 0.71, and item 29, ‘I can comfortably express my preference if music is played in theatre’, with a content validity index of 0.43, were removed. At the end of this validation stage, 27 items that had reached the cut-off content validity index from trainee responses were included. From the responses collected from consultant trainers, 27 items reached the cut-off content validity index of 0.83. Item ‘I get bleeped during operations’ had a content validity index of 0.71, and item ‘I can comfortably express my preference if music is played in theatre’ had a content validity index of 0.43.

Cronbach's α

The 27-item questionnaire received 17 responses from the local higher surgical trainees. Cronbach's α of the overall questionnaire was 0.89. Of the three subscales in the instrument, Cronbach's α was 0.81 for trainer support and supervision, 0.77 for operating opportunities, and 0.67 for operating theatre atmosphere.

Scores of final 27 items

The mean score of all the respondents was 62.5 (range, 51–75; standard deviation = 8.60). With strongly disagree converted to 0, strongly agree converted to 3 and reversed scoring on negatively worded items, the minimal and maximal scores obtainable in the instrument with four-point Likert scales were 0 and 81. This derives a mean percentage score of 77.2 per cent in the overall instrument from the respondents. The percentage scores of the three subscales were 84.0 per cent for trainer support and supervision, 69.4 per cent for operating opportunities and 80.7 per cent for operating theatre atmosphere. Item ‘I feel that I have autonomy in theatre’ had the highest percentage score of 90.2 per cent, and item ‘There is too much time pressure on elective lists for me to get the most out of theatre teaching’ had the lowest percentage score of 56.9 per cent.

The final instrument at the end of the validation process consisted of 27 items (Table 2).

Table 2. Instrument for evaluating higher surgical training experience in operating theatre

Discussion

The operating theatre as a learning environment is unique in that it is a high-risk environment and shaped heavily by human factors. Modern surgical training curriculums often incorporate skills laboratory and simulations to navigate this. However, non-technical skills such as team work, leadership and situation awareness are difficult to address in a simulated environment.Reference Moorthy, Munz, Adams, Pandey and Darzi12 Compounded by the lack of suitable analogous surgical models for the simulation of procedures, the operating theatre remains the predominant learning environment for trainees to acquire both technical and non-technical skills, despite recent technological advances. Therefore, it is imperative that the operating theatre learning environment is evaluated regularly to address its deficiencies, as perceived by trainees themselves, in order to enhance learner performance.

In our study, items were taken from the widely used Surgical Theatre Educational Environment Measure and discussed in a focus group of higher surgical trainees. Cassar's instrument has been adapted and validated in various studies aimed at different populations.Reference Mahoney, Crowe and Harris13Reference Clapham, Wall and Batchelor15 In these studies, the subscale structure of the instrument was largely preserved in the factor analyses; hence, it is not surprising that the participants in our focus group agreed on the three existing subscales as being the main constructs in the operating theatre learning environment. Similarly, as in the aforementioned studies, population- and discipline-specific factors precipitated the collapse of items in our version of the instrument, which was aimed at higher surgical trainees. ‘I am too busy doing other work to go to theatre’ was removed, as trainees felt the item was not applicable to them because higher surgical trainees will have protected operating theatre sessions incorporated into their job plan. ‘I am so stressed in theatre that I do not learn as much as I could’ was removed, as trainees felt that this statement did not identify any elements in the learning environment as the source of stress, and could be interpreted as trainees not coping with the programme.

The content validity index and Cronbach's α both indicate that our instrument has high content validity and internal consistency. Two items were excluded from the instrument because they did not reach a critical content validity index after being reviewed by the panel of higher surgical trainees. It is surprising that item ‘The theatre staff are friendly’ did not reach the critical content validity index, as operating theatre staff personalities have been found to be a consistent theme in influencing the operating theatre atmosphere and, hence, the learning environment.Reference Holt and Roff8,Reference Nagraj, Wall and Jones14 This could be because the item was perceived to be redundant, as there are already other items addressing operating theatre nursing staff and anaesthetists in the instrument. The friendliness of operating theatre staff was also accessed by item ‘I feel part of a team in theatre’; therefore, the removed item was not assessing a unique element and, hence, would not reduce the overall scope of the instrument. Item ‘I can comfortably express my preference if music is played in theatre’ had the lowest content validity index of 0.43 in both the higher surgical trainee panel and the consultant trainer panel. Interestingly, this was an extra item added upon discussion with the focus group in the initial stage of the study. Participants felt that music improves the calmness and atmosphere of the operating theatre. Current literature supports this view, but has suggested that music can also be distracting and cause impaired communications.Reference Moris and Linos16 Studies have also shown that surgeons are the most empowered group amongst operating theatre staff when it comes to choosing music.Reference Narayanan and Gray17 However, the importance of this was not recognised by either panel in our study.

  • The operating theatre is the primary learning environment for trainees to acquire procedural skills and needs to be optimised to ensure effective training

  • A 27-item questionnaire was developed to evaluate the training experience of higher surgical trainees in the operating theatre

  • The content validity index and Cronbach's α indicated that the instrument has high content validity and internal consistency compared to similar evaluation questionnaires

  • Three themes are perceived to be important for higher surgical training: trainer support and supervision, operating opportunities, and theatre atmosphere

  • The instrument is applicable across surgical subspecialties and can be easily adapted for other training programmes

Cronbach's α of the final 27-item instrument was 0.89. This is comparable to similar established instruments, such as the Surgical Theatre Educational Environment Measure (Cronbach's α = 0.877)Reference Cassar9 and the Postgraduate Hospital Educational Environment Measure (Cronbach's α = 0.899).Reference Vieira18 When looking at individual Cronbach's α values in the three subscales, these values are also in the same range as studies that broke down the Postgraduate Hospital Educational Environment Measure into individual subscales.Reference Koutsogiannou, Dimoliatis, Mavridis, Bellos, Karathanos and Jelastopulu19

Although our study aimed to develop and validate a feedback instrument, the initial responses from higher surgical trainees were also analysed in order to provide a current view of the operating theatre learning environment. The mean score of our study population (77.2 per cent) was similar to the mean score of the Surgical Theatre Educational Environment Measure (74.4 per cent).Reference Cassar9 Similarly, operating opportunities had the lowest subscale score compared with the other subscales. This may reflect that providing opportunities for trainees to operate is the most challenging aspect to optimise in the learning environment, as this is dependent on multiple factors, such as variety of cases and time pressure.

A conducive learning environment has been shown to be associated with improved learner performances and can help develop trainees’ surgical confidence.Reference Lees, Zheng, Daniels and White20 By regularly evaluating how trainees perceive their training experiences in the operating theatre, programmes can identify areas that need to be improved in order to ensure successful training. The limitation of our instrument lies within the validation process, as only local higher surgical trainees were involved, and this could incur a selection bias. Factors and items perceived to be important could be population-specific, making the instrument only applicable to surgical training programmes with similar curriculum structure and work culture. This is particularly relevant in the UK with the introduction of the revised curriculum in August 2021, as surgical training across subspecialties has been largely unified with Generic Professional Capabilities and Capabilities in Practice.Reference Lund21 Nonetheless, the transparent development and validation process depicted in our study can be easily adapted by other institutions or specialty training programmes, thereby creating a feedback instrument that is targeted at clinical learners at different levels and the wider clinical learning environment.

Data availability statement

The data that support the findings of this study are available upon reasonable request.

Competing interests

None declared

Footnotes

Dr K L Lau takes responsibility for the integrity of the content of the paper

Presented at the North of England Otolaryngology Society Spring Meeting, 21 March 2022, Crewe, UK.

References

McKee, RF. The Intercollegiate Surgical Curriculum Programme (ISCP). Surgery 2008;26:411–16Google Scholar
Parsons, BA, Blencowe, NS, Hollowood, AD, Grant, JR. Surgical training: the impact of changes in curriculum and experience. J Surg Educ 2011;68:4451CrossRefGoogle ScholarPubMed
Lyon, P. A model of teaching and learning in the operating theatre. Med Educ 2004;38:1278–87CrossRefGoogle Scholar
Croghan, SM, Phillips, C, Howson, W. The operating theatre as a classroom: a literature review of medical student learning in the theatre environment. Int J Med Educ 2019;10:7587CrossRefGoogle ScholarPubMed
Kieu, V, Stroud, L, Huang, P, Smith, M, Spychal, R, Hunter-Smith, D et al. The operating theatre as classroom: a qualitative study of learning and teaching surgical competencies. Educ Health 2015;28:22–8CrossRefGoogle ScholarPubMed
Nordquist, J, Hall, J, Caverzagie, K, Snell, L, Chan, M-K, Thoma, B et al. The clinical learning environment. Med Teach 2019;41:366–72CrossRefGoogle ScholarPubMed
Chan, CYW, Sum, MY, Lim, WS, Chew, NWM, Samarasekera, DD, Sim, K. Adoption and correlates of Postgraduate Hospital Educational Environment Measure (PHEEM) in the evaluation of learning environments–a systematic review. Med Teach 2016;38:1248–55CrossRefGoogle ScholarPubMed
Holt, MC, Roff, S. Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM). Med Teach 2004;26:553–8CrossRefGoogle ScholarPubMed
Cassar, K. Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Med Teach 2004;26:260–4CrossRefGoogle ScholarPubMed
Roff, S. The Dundee Ready Educational Environment Measure (DREEM)—a generic instrument for measuring students’ perceptions of undergraduate health professions curricula. Med Teach 2005;27:322–5CrossRefGoogle ScholarPubMed
Polit, DF, Beck, CT, Owen, SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007;30:459–67CrossRefGoogle ScholarPubMed
Moorthy, K, Munz, Y, Adams, S, Pandey, V, Darzi, A. A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg 2005;242:631–9CrossRefGoogle Scholar
Mahoney, A, Crowe, PJ, Harris, P. Exploring Australasian surgical trainees’ satisfaction with operating theatre learning using the ‘surgical theatre educational environment measure’. ANZ J Surg 2010;80:884–9CrossRefGoogle ScholarPubMed
Nagraj, S, Wall, D, Jones, E. The development and validation of the mini-surgical theatre educational environment measure. Med Teach 2007;29:e192–7CrossRefGoogle ScholarPubMed
Clapham, M, Wall, D, Batchelor, A. Educational environment in intensive care medicine—use of Postgraduate Hospital Educational Environment Measure (PHEEM). Med Teach 2007;29:e184–91CrossRefGoogle ScholarPubMed
Moris, DN, Linos, D. Music meets surgery: two sides to the art of “healing”. Surg Endosc 2013;27:719–23CrossRefGoogle Scholar
Narayanan, A, Gray, AR. First, do no harmony: an examination of attitudes to music played in operating theatres. Age 2018;35:3544Google Scholar
Vieira, JE. The Postgraduate Hospital Educational Environment Measure (PHEEM) questionnaire identifies quality of instruction as a key factor predicting academic achievement. Clinics (Sao Paulo) 2008;63:741–6CrossRefGoogle ScholarPubMed
Koutsogiannou, P, Dimoliatis, ID, Mavridis, D, Bellos, S, Karathanos, V, Jelastopulu, E. Validation of the Postgraduate Hospital Educational Environment Measure (PHEEM) in a sample of 731 Greek residents. BMC Res Notes 2015;8:734Google Scholar
Lees, MC, Zheng, B, Daniels, LM, White, JS. Factors affecting the development of confidence among surgical trainees. J Surg Educ 2019;76:674–83CrossRefGoogle ScholarPubMed
Lund, J. The new general surgical curriculum and ISCP. Surgery (Oxf) 2020;38:601–6CrossRefGoogle ScholarPubMed
Figure 0

Table 1. Items removed in focus group

Figure 1

Table 2. Instrument for evaluating higher surgical training experience in operating theatre