INTRODUCTION
Deliberate practice is widely acknowledged as important for the development of expertiseReference Ericsson 1 and requires that learners engage in effortful activities to incrementally close gaps between current and optimal skill performance while supported by feedback from expert mentorsReference Ericsson 2 , Reference Ericsson, Krampe and Tesch-Romer 3 , suggesting experience alone is insufficient in developing expertise. In education settings these supportive features are readily available; however, after entry to practice, these features may become inconsistent or limitedReference Pusic, Kessler and Szyld 4 - Reference Pusic, Boutis and Hatala 6 .
Paramedics practice in settings where important elements in the development of expertise may be limited. For instance, there is a high degree of independence, access to external feedback is limited, expert mentors and senior clinicians are unavailable and opportunities to engage in deliberate practice are infrequent. Paramedics are left to self-assess, a process which may be inherently limitedReference Eva and Regehr 7 , reflecting on their actions with the risk of mistakenly re-enforcing poor technical or cognitive strategies in the absence of clear errors. This creates situations where practice variations may emerge that satisfy rather than optimize performance.
We chose to examine this unique phenomenon—practicing in settings where features supportive of the development of expertise are limited—using pre-hospital airway management as an example. Out-of-hospital airway management by paramedics is associated with low success ratesReference Diggs, Yusuf and De Leo 8 - Reference Hubble, Brown and Wilfong 10 , high rates of complicationsReference Wang, Lave and Sirio 11 - Reference Wang, Kupas and Paris 15 and, in some cases, poor patient outcomesReference Wang, Sweeney and O’Connor 16 , Reference von Goedecke, Herff and Paal 17 . Some have suggested that entry-to-practice training (e.g., manikin v. cadaver v. live patient)Reference Walker, Jensen and Leroux 18 - Reference Westgard, Peterson and Salzman 21 and/or skill degradationReference Wang, Balasubramani and Cook 22 - Reference Barnes 25 might explain these findings. However, to our knowledge, this issue of practicing in settings where features that support the development of expertise are unavailable has not been considered within the context of this problem.
Our primary research question was therefore: Among a group of paramedics with similar entry-to-practice training, what happens to clinical performance in settings where access to features that support the development of expertise are inconsistent or limited? As a secondary research question, we also asked: What is the relationship between experience and clinical performance among our sample? Based on our conceptual framework, we predicted that the paramedics’ clinical performances would vary, and we further hypothesized that experience would not be a predictor of clinical acumen.
METHODS
Overview
We used a convergent mixed methods study design, involving both quantitative and qualitative data collection and analysis.Reference Creswell 26 We invited paramedics with varying degrees of experience, both in years of service and place of employment, to complete two full clinical cases (involving airway management) in simulation.
This study was conducted at the Centennial College Inter-professional Simulation Center, located in Toronto, Ontario, Canada and was approved by the Centennial College Research Ethics Board (REB #184).
Participant pool and context of practice
In Ontario, Canada, two levels of paramedics provide pre-hospital emergency care: primary and advanced care paramedics. Primary Care Paramedics (PCPs) complete a two-year community college diploma program and provide basic life support (e.g., cardiopulmonary resuscitation (CPR), intravenous access, “symptom relief” medications). Advanced Care Paramedics (ACPs) complete a third year of community college training and provide advanced life support, including oral and naso-tracheal intubation, supraglottic airway insertion and surgical cricothryroidotomy. In Ontario, ACP training programs may vary in structure but are consistent in accreditation requirements 27 with respect to content. Specific to airway management, ACP students learn in progressively complex environments including simulation-based, clinical (e.g., operating room) and field settings, with standardized minimum competency thresholds for each setting. Upon completion of their training, ACP students are required to complete a provincial written examination for certification and undergo additional knowledge and performance-based testing through their respective Regional Base Hospital (regulatory bodies responsible for paramedic practice and education; n=7) for authorization to practice. Once practicing, ACPs complete 24-hours of classroom and simulation-based mandatory continuing education per year. In Ontario, all paramedics practice under a set of provincially standardized advanced and basic life support patient care standards thus ensuring a consistent scope of practice. 28 , 29
Participant recruitment and selection
We used convenience sampling to recruit our participants.Reference Marshall 30 , Reference Coyne 31 Eligibility requirements included: (a) practicing ACPs; and (b) trained at an accredited college in Ontario. We excluded paramedics who have cross-trained in another health profession and also paramedics at the critical care level where context of training and care is different with respect to airway management. We distributed invitations to participate in the study via workplace email. All participants completed a questionnaire that included basic demographic information (age, sex, years of experience), annual exposure to airway management (intubation, supraglottic airway insertion, cricothyroidotomy) and whether the participant had completed any specialized airway management courses. Our intent was to recruit a sample of participants who, as much as reasonably possible, were similar except for years of clinical experience.
Simulation cases
We created two scenarios for this study, designed for their ability to approximate both a very complicated (Scenario 1) and routine (Scenario 2) airway management case. The order of case presentation was randomized. Both scenarios involved adult patients (using Laerdal SimMan®) requiring advanced airway management (e.g., supraglottic airway insertion or endotracheal intubation), however in Scenario 1, advanced airway management was scripted to be unsuccessful (i.e., a “can’t intubate”, “can’t ventilate” crisis). Both cases were pilot tested by experienced paramedics with teaching responsibilities at the advanced care level to ensure content validity. See Appendix 1 for additional case details.
Procedures
We asked the participants to work through the cases as they would in real clinical practice. Each case included two standardized actors that played the role of PCPs to assist the participants as needed. Both cases were scripted to continue until effective ventilation was achieved (a surgical airway for Scenario 1 or successful advanced airway placement for Scenario 2) or 15 minutes had elapsed, whichever occurred first. All performances were recorded using four strategically positioned cameras.
Outcome measures and data analysis
Based on our conceptual framework, we decided a priori to explore three outcome variables: (1) variation in the decision paths (i.e., sequence of steps); (2) schemas (i.e., cognitive frameworks that help organize and process information) that inform how the cases are managed; and (3) the relationship between experience (in years) and clinical performance (see Figure 1).
Figure 1 Study Process Overview.
Decision paths
We defined decision paths as the sequence of steps taken in managing the case and the alignment (or lack thereof) with a criterion standard, in this case, an airway management algorithm published by Wang et al.Reference Wang, Kupas and Greenwood 32 (see Figure 2). To our knowledge, this is the only published algorithm specifically intended to guide out-of-hospital airway management by paramedics. This algorithm is not taught during ACP training and is not used in clinical practice in our context and was intended to be a ‘neutral’ reference point from which to assess between-participant variation in management decisions. We modified the algorithm slightly by substituting the term “advanced airway” for “endotracheal intubation” since supraglottic airway devices are considered equivalent in our setting.
Figure 2 Algorithm for Prehospital Airway Management by Wang et al, 2005.
Two investigators (JM, SD) used video review to code and sequence the participants’ management decisions based on observable behaviors. This required both researchers to agree on each coded behavior with a third researcher (WT) providing a final decision if consensus could not be reached. We then mapped a series of “ideal” pathways through the scenarios using the algorithm and then qualitatively analyzed the coded sequences of participant actions by comparing them with the ideal pathways and with each other for concordance and emerging commonalities, using ordered letters to indicate participant paths. At times, participants performed actions that were not part of the algorithm; these were coded as “X” interventions (Table 1).
Table 1 Legend for coded behaviors
Schema-based behaviors
To explore this cognitive component of practice, each video was reviewed for observable behaviors that could suggest schemas that the participants held and applied when managing the scenarios. Two investigators (JM, SD) independently recorded hand written memos while observing the scenarios, intending to capture the granularity of the participants’ clinical performance. These notes were analyzed using qualitative descriptive analysisReference Sandelowski 33 , Reference Sandelowski 34 , a process that involves organizing observations of events into themes, with JM and SD meeting regularly during the coding process and resolving discrepancies through consensus.
Experience as a Predictor
Finally, our conceptual framework also suggests that experience alone would not predict expertise. To explore this relationship, we compared experience practicing at the advanced care level with three markers of clinical performance: (a) a previously validated paramedic-specific global rating scale (GRS)Reference Tavares, Leblanc and Mausz 35 , Reference Tavares, Boet and Theriault 36 , (b) a task-specific checklist and (c) time to achieve effective ventilation (defined above). The GRS evaluates clinical performance across seven dimensions (situation awareness, history gathering, patient assessment, decision making, resource utilization, communication and procedural skills) considered representative of paramedic practice using a 7-point adjectival scale (i.e., 1=“unsafe”; 7=“exceptional”)Reference Tavares, Boet and Theriault 36 (see Appendix 2). The task-specific checklist was constructed using accepted definitions for procedural errors in airway management drawn from the literature (Table 2). Finally, time from patient contact until effective ventilation was measured in minutes and seconds.
Table 2 Procedural error definitions
Two investigators (JM, SD) with experience using the GRS evaluated each performance using both the global rating scale and the checklist. For the first three participants, the investigators evaluated the performances jointly to achieve consensus on performance expectations and scoring; all subsequent evaluations took place independently. Inter-rater reliability was calculated using the inter-class coefficient for both the GRS scores (by dimension) and the incidence of errors (using the checklist). The relationship between participant experience (in years) and clinical performance was assessed using correlation analyses between (a) each dimension on the GRS; (b) the number of errors committed by each participant; and (c) and the time to achieve ventilation. Assuming a conservative correlation coefficient of 0.45, we required approximately 27 participants to obtain 95% confidence intervals of 0.1. We hypothesized that experience would fail to correlate with any clinical performance variable.
RESULTS
Thirty ACPs from six paramedic services in southern Ontario, representing a mix of urban, suburban and rural contexts with large variation in call volumes completed a total of 59 simulations. One participant declined to complete the second scenario for personal reasons. Twenty-five (83%) participants were male; the participants had an average of 12.0 (SD=6.6) years of experience as paramedics and an average of 7.2 (SD 5.8) years experience practicing at the advanced care level. Full demographic characteristics are presented in Table 3.
Table 3 Participant demographics
SGA = Supraglottic Airway.
Scenario content validity
To check the content validity of our scenarios, we conducted a number of analyses to assess the difficulty of the scenarios and whether the scenarios were perceived to be realistic by the participants. We report these results in Appendix 3; in brief, Scenario 1 was more challenging than scenario 2 (as expected) and the scenarios were felt to convey a high degree of physical, conceptual and emotional realism.
Decision paths
The participants’ coded behaviors are presented in Tables 4 (scenario 1) and 5 (scenario 2); we achieved complete agreement on the coded behaviors between the researchers with no irreconcilable discrepancies. Only one participant (during Scenario 2) followed one of the “ideal” pathways exactly. The participants varied extensively in the sequencing of steps, interventions chosen and points at which the scenarios were (or were not) successfully resolved. In contrast to what should have been a linear and algorithmic problem to solve, we observed that no two participants approached the clinical problem in the same way.
Table 4 Coded actions for Scenario 1
Table 5 Coded action for Scenario 2
Schema-based behaviors
Our analysis revealed a number of potential schemas (underlying cognitive processes that appeared to be driving behavior) that were applied by participants. We broadly categorized these as relating to situation awareness, decision making and procedural sills.
Related to situation awareness, we observed that participants had a tendency to focus in on specific tasks, often at the expense of an awareness of the underlying primary problem—a failed airway resulting in critical hypoxemia. For example, we observed that participants would attend to the patient’s deteriorating vital signs (e.g., bradycardia and hypotension), not realizing that these were symptoms of the airway problem and as a result, would fail to establish a patent airway. This also resulted in patient deterioration going undetected.
Regarding decision-making, we also observed schemas related to decision or care thresholds and strategies. For example, we observed variability related to when advanced airway use was considered or attempted, this was especially true of the threshold at which the participants would consider a surgical airway—a decision point usually determined a priori during training. Also, for many we observed a reversal of airway management strategies with participants often attempting more advanced procedures (e.g., intubation) prior to preparatory or less invasive techniques (e.g., use of adjuncts or airway positioning), despite standards of practice recommending the opposite.
Finally, we observed problematic schemas designed to mitigate challenges with procedural skills. For instance, we observed participants failing to follow best practice guidelines for intubating technique (poor patient positioning, levering the laryngoscope on the patient’s teeth, inappropriate equipment selection, etc.). An illustrative example involved a number of participants who would knowingly leave an esophageal endotracheal tube in place and then attempt to insert a second endotracheal tube (sometimes with a laryngoscope, sometimes without), in the (mistaken) belief it would enter the trachea.
In summary, participants seem to adopt individualized schemas over time that appear to develop through their clinical practice that in many cases may only satisfy (and in some ways threaten) rather than optimize patient care and safety.
Experience as a predictor
Table 6 lists the average GRS scores, number of errors committed per participant (based on a checklist) and average time to achieve ventilation for both scenarios. We achieved excellent inter-observer agreement in the global rating scale evaluation for the domains considered particularly important for the study (ICC=0.89–0.96 for situation awareness, decision making and procedural skills) and for the number of errors observed per participant (ICC=0.89–0.93). We found no correlation between years of experience at the advanced care level and overall GRS scores (Scenario 1: r=0.13, p=0.47; Scenario 2: r= −0.10, p=0.58), or the average number of errors committed (Scenario 1: r=0.10, p=0.57; Scenario 2: r=0.25, p=0.17) or finally, the time to achieve effective ventilation (Scenario 1: r=0.53, p=0.78; Scenario 2: r=0.27, p=0.15). We repeated the correlation analyses using Spearman’s correlation and obtained similar results.
Table 6 Clinical performance analysis results
ICC = inter-class correlation, SD = standard deviation. SA = situation awareness; HG = history gathering; PA = patient assessment; DM = decision making; RU = resource utilization; CM = communication; PS = procedural skill; OV = overall score.
* All GRS scores evaluated using 7-point adjectival scale; 1=“unsafe”, 7=“exceptional”.
DISCUSSION
Paramedicine allows for practice and the accumulation of experience, both of which are necessary for professional development, but lacks other essential features (i.e., access to feedback, deliberate practice, and proximal supervision) for the development of expertise. Our goal in this study was to explore how clinical experience in scenarios where these features are limited or inconsistent affects performance. Our results suggest that even when faced with a highly algorithmic problem, variations or reformulations of practice can emerge. While previous work has identified variation between cliniciansReference Mercuri, Natarajan and Norman 37 , Reference Mercuri and Gafni 38 , particularly with respect to guideline adherenceReference Mercuri, Sherbino and Sedran 39 , to our knowledge the concept of reformulations of practice has not been described. A search of the education literature revealed no previous use of the term reformulations of practice. We propose defining reformulations of practice as behaviors, schemas, and decision paths that evolve over time through experience but unpredictably and often detrimentally. While some variation between providers would be expected to occur naturally, in an algorithmic domain of practice like airway management, the amount of variation we observed was significant. Further, experience failed to mitigate the effects, collectively suggesting that inherent profession-level structures may be placing a ceiling on the development of paramedic expertise. Our study has important implications for the profession regarding the maintenance of competence.
While there is still debate within the education community regarding the optimal timing and amount of feedbackReference Walsh, Ling and Wang 40 - Reference Cheng, Eppich and Grant 43 , the concept that feedback enhances learning has been well establishedReference Lefroy, Watling and Teunissen 44 . Feedback can be intrinsic (i.e., immediate, haptic) or external (i.e., provided by an expert mentor or coach) and is intended to identify deficiencies in performance and prescribe corrective guidance.Reference Sigrist, Rauter and Riener 41 , Reference Ramani and Krackov 45 , Reference Wulf, Shea and Lewthwaite 46 That we observed such heterogeneity in performance among a largely homogenous group of clinicians suggests that feedback is still occurring but that it is intrinsically—and internally—derived. This was evidenced in a highly algorithmic domain, where presumably reformulations of practice are less likely. This raises concerns for less technical or more cognitively dependent skills that are susceptible to the same limitations. The contextual issues in paramedicine may contribute challenges in maintaining competence, especially with complex but infrequently used skills. This requires careful consideration to the structuring of practice and learning opportunities available to paramedics.
Deliberate practice has widely been acknowledged as a necessary condition for learning and expertise.Reference van de Wiel, Van den Bossche and Janssen 47 , Reference McGaghie, Issenberg and Cohen 48 It has also been suggested that achieving a level of proficiency beyond what is initially required for competence (i.e., overlearning) may help to slow the degradation of skills.Reference Pusic, Kessler and Szyld 4 , Reference Pusic, Pecaric and Boutis 5 , Reference Pusic, Brydges and Kessler 49 Another option includes promoting and supporting a culture of ongoing, individualized simulation-based learning. This provides a useful platform to maintain low frequency, high complexity skills by allowing for deliberate practice in a realistic environment while neutralizing concerns over patient safety.Reference Cook, Hatala and Brydges 50 - Reference McGaghie, Issenberg and Petrusa 53 Finally, another strategy may involve recording clinical encounters and using the footage to facilitate educational debriefings after high acuity cases. This strategy has been used successfully in neonatalReference Carbine, Finer and Knodel 54 and trauma resuscitationsReference Scherer, Chang and Meredith 55 , Reference Oakley, Stocker and Staubli 56 ; however, some logistical issues would need to be resolved for use in paramedicine. Implementing systems to support the ongoing competence of paramedics and other health care providers is a worthy goal for educators and administrators alike.
For clinicians and the research community, the results of our study might shed new light on paramedic airway management. Lack of experience, limited initial education, and low frequency of use have been previously suggested as problematic.Reference Wang, Seitz and Hostler 20 - Reference Wang, Kupas and Hostler 23 We suggest that even if those issues are resolved, until features associated with the development of expertise are addressed, problems may continue to persist.
LIMITATIONS
We acknowledge some limitations in our study. First, we assume that our participants were homogenous as a result of similar entry-to-practice training, however, without the benefit of a more structured longitudinal study, the possibility of confounding exists. Second, simulation can only ever serve as a surrogate for reality; it is possible that the behavior of the participants was influenced by the simulated nature of the study. Third, our sample size was limited. This raises the possibility of a type 2 error; however, this applies only to a secondary question conducted primarily as a test of our conceptual framework. Finally, the schemas governing performance were inferred—other schemas may be responsible for the behavior we observed.
CONCLUSION
Among a group of paramedics with similar entry-to-practice and ongoing training, we observed significant variation between providers and a number of potentially problematic reformulations of practice. We believe these variations may emerge as a function of limited opportunities for feedback and deliberate practice and suggest that administrators and educators consider the implications of this study in supporting the ongoing competence of clinicians performing complex skills.
Acknowledgements: JM and WT were responsible for the conception, design and implementation of the study, JM, SD, and MMcC were primarily responsible for analyzing the data, CL, AW, and EF provided important logistical support during data collection. All authors contributed to the development of the manuscript and approve its content. The authors wish to thank Peel Regional Paramedic Services, York Region Paramedic Services and Centennial College for important material contributions and also wish to express their gratitude to the dozens of paramedic student volunteers who assisted with the implementation of this study.
Competing Interests: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper. This study was funded by the Ontario Association of Paramedic Chiefs.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/cem.2016.371
