Hostname: page-component-586b7cd67f-r5fsc Total loading time: 0 Render date: 2024-11-27T19:10:16.508Z Has data issue: false hasContentIssue false
  • English
  • Français

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents

Published online by Cambridge University Press:  11 May 2015

Andrew Koch Hall ,
William Pickett  and
Jeffrey Damon Dagnone
Andrew Koch Hall*
Affiliation:
Department of Emergency Medicine, Queen's University, Kingston, ON
William Pickett
Affiliation:
Department of Emergency Medicine, Queen's University, Kingston, ON
Jeffrey Damon Dagnone
Affiliation:
Department of Emergency Medicine, Queen's University, Kingston, ON
*
Department of Emergency Medicine, Kingston General Hospital, Empire 3, 76 Stuart Street, Kingston, ON K7L 2V7

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.
Objective:

We sought to develop and validate a three-station simulation-based Objective Structured Clinical Examination (OSCE) tool to assess emergency medicine resident competency in resuscitation scenarios.

Methods:

An expert panel of emergency physicians developed three scenarios for use with high-fidelity mannequins. For each scenario, a corresponding assessment tool was developed with an essential actions (EA) checklist and a global assessment score (GAS). The scenarios were (1) unstable ventricular tachycardia, (2) respiratory failure, and (3) ST elevation myocardial infarction. Emergency medicine residents were videotaped completing the OSCE, and three clinician experts independently evaluated the videotapes using the assessment tool.

Results:

Twenty-one residents completed the OSCE (nine residents in the College of Family Physicians of Canada– Emergency Medicine [CCFP-EM] program, six junior residents in the Fellow of the Royal College of Physicians of Canada–Emergency Medicine [FRCP-EM] program, six senior residents in the FRCP-EM). Interrater reliability for the EA scores was good but varied between scenarios (Spearman rho 5 [1] 0.68, [2] 0.81, [3] 0.41). Interrater reliability for the GAS was also good, with less variability (rho 5 [1] 0.64, [2] 0.56, [3] 0.62). When comparing GAS scores, senior FRCP residents outperformed CCFP-EM residents in all scenarios and junior residents in two of three scenarios (p , 0.001 to 0.01). Based on EA scores, senior FRCP residents outperformed CCFP-EM residents, but junior residents outperformed senior FRCP residents in scenario 1 and CCFPEM residents in all scenarios (p 5 0.006 to 0.04).

Conclusions:

This study outlines the creation of a high-fidelity simulation assessment tool for trainees in emergency medicine. A single-point GAS demonstrated stronger relational validity and more consistent reliability in comparison with an EA checklist. This preliminary work will provide a foundation for ongoing future development of simulationbased assessment tools.

Keywords

assessmentemergency medicinemedical educationresidentresuscitationsimulation
Type
Original Research • Recherche originale
Information
Canadian Journal of Emergency Medicine , Volume 14 , Issue 3 , March 2012 , pp. 139 - 146
Copyright
Copyright © Canadian Association of Emergency Physicians 2012

References

REFERENCES

1.McLaughlin, S, Fitch, MT, Goyal, DG, et al. Simulation in graduate medical education 2008: a review for emergency medicine. Acad Emerg Med 2008;15:1117–29, doi:10.1111/j.1553-2712.2008.00188.x.CrossRefGoogle ScholarPubMed
2.Miller, GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(9 Suppl):S63–7.CrossRefGoogle ScholarPubMed
3.Epstein, RM. Assessment in medical education. N Engl J Med 2007;356:387–96, doi:10.1056/NEJMe078002.CrossRefGoogle ScholarPubMed
4.Sherbino, J, Bandiera, G, Frank, J. Assessing competence in emergency medicine trainees; an overview of effective methodologies. Can J Emerg Med 2008;10:365.CrossRefGoogle Scholar
5.Lammers, RL, Davenport, M, Korley, F, et al. Teaching and assessing procedural skills using simulation: metrics and methodology. Acad Emerg Med 2008;15:1079–87, doi:10.1111/j.1553-2712.2008.00233.x.CrossRefGoogle ScholarPubMed
6.Boulet, JR. Summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008;15:1017–24, doi:10.1111/j.1553-2712.2008.00228.x.CrossRefGoogle Scholar
7.Spillane, L, Hayden, E, Fernandez, R, et al. The assessment of individual cognitive expertise and clinical competency: a research agenda. Acad Emerg Med 2008;15:1071–8,doi:10.1111/j.1553-2712.2008.00271.x.CrossRefGoogle ScholarPubMed
8.Okuda, Y, Bryson, EO, DeMaria, S Jr, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med 2009;76:330–43, doi:10.1002/msj.20127.CrossRefGoogle ScholarPubMed
9.Fritz, PZ, Gray, T, Flanagan, B. Review of mannequin-based high-fidelity simulation in emergency medicine. Emerg Med Australas 2008;20:1–9, doi:10.1111/j.1742-6723.2007.CrossRefGoogle ScholarPubMed
10.Okuda, Y, Bond, W, Bonfante, G, et al. National growth in simulation training within emergency medicine residency programs, 2003–2008. Acad Emerg Med 2008;15:1113–6, doi:10.1111/j.1553-2712.2008.00195.x.CrossRefGoogle ScholarPubMed
11.Binstadt, ES, Walls, RM, White, BA, et al. A comprehensive medical simulation education curriculum for emergency medicine residents. Ann Emerg Med 2007;49:495, doi:10.1016/j.annemergmed.2006.08.023.CrossRefGoogle ScholarPubMed
12.Philibert, I. Acreditation Council for Graduate Medical Education. ACGME Bulletin. December 2005. Available at: http://www.acgme.org (accessed April 2010).Google Scholar
13.Adler, MD, Trainor, JL, Siddall, VJ, et al. Development and evaluation of high-fidelity simulation case scenarios for pediatric resident education. Ambul Pediatr 2007;7:182–6, doi:10.1016/j.ambp.2006.12.005.CrossRefGoogle ScholarPubMed
14.Brett-Fleegler, MB, Vinci, RJ, Weiner, DL, et al. A simulatorbased tool that assesses pediatric resident resuscitation competency. Pediatrics 2008;121:e597“603, doi:10.1542/peds.2005-1259.CrossRefGoogle ScholarPubMed
15.Morgan, PJ, Cleave-Hogg, DM, Guest, CB, et al. Validity and reliability of undergraduate performance assessments in an anesthesia simulator. Can J Anaesth 2001;48:225–33, doi:10.1007/BF03019750.CrossRefGoogle Scholar
16.Murray, DJ, Boulet, JR, Avidan, M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology 2007;107:705–13, doi:10.1097/01.anes.0000286926.01083.9d.CrossRefGoogle Scholar
17.Weller, JM, Bloch, M, Young, S, et al. Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists. Br J Anaesth 2003;90:43–7, doi:10.1093/bja/90.1.43.CrossRefGoogle ScholarPubMed
18.Berkenstadt, H, Ziv, A, Gafni, N, et al. Incorporating simulation-based objective structured clinical examination into the Israeli National Board Examination in Anesthesiology. Anesth Analg 2006;102:853–8, doi:10.1213/01.ane.0000194934.34552.ab.CrossRefGoogle ScholarPubMed
19.Girzadas, DV Jr, Clay, L, Caris, J, et al. High fidelity simulation can discriminate between novice and experienced residents when assessing competency in patient care. Med Teach 2007;29:452–6, doi:10.1080/01421590701513698.CrossRefGoogle ScholarPubMed
20.Dagnone, JD. Assessment tool for a standardized ventricular fibrillation cardiac arrest. Can J Emerg Med 2009;11:262.Google Scholar
21.Andreatta, PB, Gruppen, LD. Conceptualising and classifying validity evidence for simulation. Med Educ 2009;43:102835, doi:10.1111/j.1365-2923.2009.03454.x.CrossRefGoogle ScholarPubMed
22.Walls, RM. Airway. In: Marks, JA, editor. Rosen’s emergency medicine. Philadelphia: Mosby Elsevier; 2006. p. 2–26.Google Scholar
23.Morgan, PJ, Cleave-Hogg, D, Guest, CB. A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator. Acad Med 2001;76:1053–5, doi:10.1097/00001888-200110000-00016.CrossRefGoogle ScholarPubMed
24.Regehr, G, MacRae, H, Reznick, RK, et al. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 1998;73:993–7, doi:10.1097/00001888-199809000-00020.CrossRefGoogle Scholar