The writer’s guide to education scholarship in emergency medicine: Education innovations (part 3)

Abstract

Objective

The scholarly dissemination of innovative medical education practices helps broaden the reach of this type of work, allowing scholarship to have an impact beyond a single institution. There is little guidance in the literature for those seeking to publish program evaluation studies and innovation papers. This study aims to derive a set of evidence-based features of high-quality reports on innovations in emergency medicine (EM) education.

Methods

We conducted a scoping review and thematic analysis to determine quality markers for medical education innovation reports, with a focus on EM. A search of MEDLINE, EMBASE, ERIC, and Google Scholar was augmented by a hand search of relevant publication guidelines, guidelines for authors, and website submission portals from medical education and EM journals. Study investigators reviewed the selected articles, and a thematic analysis was conducted.

Results

Our search strategy identified 14 relevant articles from which 34 quality markers were extracted. These markers were grouped into seven important themes: goals and need for innovation, preparation, innovation development, innovation implementation, evaluation of innovation, evidence of reflective practice, and reporting and dissemination. In addition, multiple outlets for the publication of EM education innovations were identified and compiled.

Conclusion

The publication and dissemination of innovations are critical for the EM education community and the training of health professionals. We anticipate that our list of innovation report quality markers will be used by EM education innovators to support the dissemination of novel educational practices.

INTRODUCTION

Innovation in medical education is becoming an increasingly common form of scholarship. ^{1 , 2} Emergency medicine (EM) has contributed immensely to this field, and our specialty is poised to continue to leading in the future. ³ In the face of increased public accountability, reduced working hours, competency-based medical education (CBME), ⁴ and new educational delivery models, modern medical education is in a state of transition. ⁵ Simultaneously, the number of outlets for the publication of innovative work is increasing. Although innovations were historically shoe-horned into standard research report formats in the past, many medical education and EM journals now have specific sections or categories for innovation reports. ^{6 , 7} One prominent example is the Brief Educational Report section in the CJEM. ⁸ Despite this increase in outlets for dissemination of innovative work, there is no clear consensus statement or set of guidelines to inform educators hoping to publish papers about their medical education innovations.

To innovate is defined in the Oxford Dictionaries as: “[To] make changes in something established, especially by introducing new methods, ideas, or products.” ⁹ Innovations in medical education are often developed to improve a program or to solve a problem. Examples include curricular improvements, development of assessment tools, or creation of faculty development programs. ¹⁰ Barrows ¹¹ points out that many publications of educational innovations have historically taken the form of descriptions of the innovations with an evaluation of learner or teaching satisfaction or an assessment of learner knowledge acquisition immediately post-implementation. He argues that these restricted evaluations are of “limited usefulness and are subject to bias and error, and they provide no information as to whether the information or skills acquired will be recalled and applied.” ¹¹ To be publishable, manuscripts must deal with topics that matter to the teachers, administrators, and scholars in the medical education community. ¹² If so, this begs the question: What are the key features of a well-written educational innovation report?

The realm of scholarship, publication, and career advancement has been referred to as an ‘alien culture’ to junior clinician educators, with many finding scholarly writing and dissemination strategies foreign and unfamiliar. ¹² Many innovations in medical education remain as improvements at their site of development only, and they are never successfully disseminated. ^{11 , 13} For our specialty to benefit from innovative work, innovators must describe and disseminate their work so that others can learn from, adopt, or build on these new techniques. ¹¹ Published innovations move the work beyond local change, to benefit many more trainees, physicians, educators, administrators, and the public. ¹⁰ These new and improved ideas are key drivers of progress in medical education. ¹³ Because scholarly publication is often a key performance marker for academic health professionals, publishing is additionally advantageous for career development. ¹⁰

It was the goal of this study to derive a set of evidence-based features of high-quality innovation reports in EM to assist clinician educators hoping to publish innovation studies and help current educators assess innovation papers for quality.

METHODS

Database search

A scoping review, as described by Arksey and O’Malley, ¹⁴ was undertaken to collate existing literature on how to publish an innovation report in medical education. The MEDLINE, EMBASE, and ERIC databases were searched along with Google Scholar on January 21, 2016, using “and/or” combinations of variations of the following keywords: “innovation,” “advancement,” “medical education,” “publication,” “guide,” “manual,” “instruction,” “improvement,” “dissemination,” and “how-to.” A librarian was consulted to assist with the terms and databases selected. We chose these databases because they captured the bulk of published material relating to medical education and medical education innovation. All searches were limited to English-language papers but were not limited in publication date. Titles were reviewed initially by two investigators independently (AKH, CH). Articles were included for subsequent review at each stage if the reviewing investigator felt the title (or abstract) indicated that the content of the article would in any way help answer the question: What are the key features of a high-quality medical education innovation report? Innovation reports themselves were excluded. A calibration step was undertaken after each of 10 titles to discuss rationale for inclusion. At 50 titles, agreement was perfect (κ=1). We hypothesize that perfect agreement was a result of broad inclusion criteria and substantial discussion between investigators prior to initiation of the title review. Subsequent titles and abstracts were reviewed by one of the two investigators.

Hand search

Anticipating the paucity of journal articles that we would identify in our database search, we also performed a hand search, looking for other guidelines or instructions for the publication of education innovations. The websites, publication guidelines, guidelines for authors, and submission portals of PubMed-indexed journals that focused on EM and Medical Education were searched for potentially relevant information along with a selection of pediatric EM and simulation-focused journals chosen by the study investigators by consensus based on perceived relevance. The titles of all MEDLINE-indexed journals in EM and medical education were reviewed by one investigator (CH). Several were excluded from the hand search based on an alternative focus (e.g., critical care), journal inactivity, or duplication. A complete list of journals searched is included as a supplemental appendix. Additionally, the reference lists of the included full-text articles from the previously mentioned database searches and hand search were reviewed for articles warranting inclusion per the criteria mentioned previously.

Analysis

Two investigators (AKH, CH) independently reviewed the selected full-text articles, including text, diagrams, figures, and tables to extract quality indicators. A master list of quality indicators was compiled, and a thematic analysis was performed by a single investigator (AKH) to group and reduce this list. This investigator is a board-certified EM physician who has completed a Masters of Medical Education degree and has experience in the successful publication of education innovations and peer-review for medical education and EM journals. The thematic analysis followed the analysis technique (i.e., charting and thematic analysis) described by Arksey and O’Malley. ¹⁴ To enhance the rigour of this analysis, three other investigators (CH, TMC, FB) reviewed and edited the groupings and list to generate a final list by consensus.

Demographics of publication venues

We sought to establish a list of venues for the publication of education innovations, which may be usable by authors hoping to publish education innovations in EM. Concurrent with the process of literature review and hand searching, a list of venues was compiled by the reviewing authors (AKH, CH). Medical education journals were included if they published a section or type of publication specifically relating to innovations. EM journals were included if they contained either a Medical Education section, or a specific section or publication category for educational innovations.

RESULTS

Database search

The MEDLINE search yielded 419 citations. We reviewed the titles of each citation for relevance or duplication, and eliminated 389 citations. These citations underwent an abstract review, which then excluded an additional 26 citations, leaving 4 citations. The search of EMBASE yielded 584 citations. We reviewed the titles of each citation for relevance or duplication, eliminating 558 citations. Abstract review excluded an additional 20 citations, leaving 6 citations. The search of the ERIC database yielded 283 citations. We reviewed the titles of each citation for relevance or duplication, which eliminated 270 citations. All remaining excluded citations were eliminated by abstract review. The Google Scholar search yielded 2,130,000 results. As presented in the search strategy by Chan et al., ¹⁵ the abstracts for the top 500 search results were reviewed. Eight abstracts were deemed relevant. These citations underwent an abstract review, which excluded an additional six citations, leaving two citations. As outlined in Figure 1, unique articles of 1,786 were identified via database searches in total, and 10 were retained for full text review. The hand search yielded an additional 27 articles, and reference searching yielded 1 article. In total, we performed a full-text review of 38 articles and excluded 24 records based on irrelevance as determined by a consensus of the reviewing investigators (AKH, CH).

Figure 1 Flow diagram for literature review.

Thematic analysis

Table 1 outlines the complete list of themes and quality indicators for an innovation report derived from the thematic analysis of quality indicators found in the literature. The sources of the items are indicated.

Table 1 Features of a high-quality innovation report

Demographics of publication venues

As summarized in Table 2, we identified multiple potential venues for the publication of education innovations. Whereas innovations are published often as original research articles in EM and medical education journals, some journals have a dedicated section or category dedicated to innovations, or indicate in their instructions for authors that educational innovations are encouraged. Blanchard et al. ¹⁰ provided an excellent summary of venues in medical education journals. Within EM-specific journals, there are a few noteworthy examples. For example, CJEM has a section titled Brief Educational Report, ⁸ which publishes articles on innovative educational advances in EM. Additionally, a new sister journal of Academic Emergency Medicine titled, Academic Emergency Medicine Education and Training, ²⁵ calls for authors to submit pieces on innovations in a number of areas (e.g., curriculum development, instructional strategies, validity evidence development, assessment, quality improvement, and faculty development).

Table 2 Innovation report publication venues

^* Extracted from Blanchard et al. ^{7 , 10}

^† Self-reported impact factor as of October 2016.

^‡ Not considering submissions as of October 2016.

DISCUSSION

Whereas evaluating and disseminating work about innovations is not new, the concept of the innovation report is relatively novel. As such, there is limited guidance in the literature about markers of quality in innovation reports. Without consensus-based standards, each journal has provided a unique set of guidelines for this type of publication. We performed a scoping review to develop a list of recommendations, advice, and guidelines for the publication of education innovations. We have extracted key items from these resources to establish a set of quality markers for innovation reports.

The themes established in our analysis parallel Kern’s six-step curricular development framework: 1) problem identification, 2) needs assessment of targeted learners, 3) goals and objectives, 4) educational strategies, 5) implementation, and 6) evaluation and feedback. ²⁶ Key features such as problem identification, needs assessment, and implementation are quite similar between Kern’s framework and our list. Our list expands on these concepts with an emphasis on transparency and reproducibility in the development and implementation process, the imperative of building on existing literature and theory, and providing a more detailed process of evaluation, including reflective practice. The additional theme of reporting and dissemination highlights that this list is not just about key features of an innovation, but rather a high-quality innovation report.

A review of the quality markers suggests that an ideal innovation report is written to share ideas that address a pressing need or existing problem that is of broad interest. Educational innovation and the scholarship around it create a community of scholars where the interaction works to stimulate ideas and move the field forward. Dissemination serves to benefit individuals in the form of career development and the community of educators by driving progress in medical education. For publication in academic journals, innovations need to be novel and transferrable to other contexts. It is important for such reports to paint a clear picture of the context of the problem and the solution. Ideally, innovations are grounded in education theory/frameworks and evaluated in some capacity beyond satisfaction and knowledge acquisition. For those looking for guidance in evaluating their innovations, there are several potential frameworks that may be used for innovation or program evaluation, including the Kirkpatrick Model, ²⁷ McGaghie 3Ts translational framework, ²⁸ and Logic Model. ²⁹ Finally, a thoughtful analysis of the impact or potential impact of the innovation and a description of how it fits in the “big picture” should be included. Overall, Kanter ¹³ describes the ideal innovation report as one that provides insight that fosters a cycle of progressive thinking and leads to further innovation and development.

We hope that the list generated by our scoping review will be of specific value to junior EM educators. The heterogeneity of innovation reports, variable requirements of journals, and the relative novelty of this category of manuscript have all contributed to an absence of, or confusion relating to, guidance for publishing. The value of this list is in its clarity and generalizability, being applicable to innovation reports across multiple fields and specific journals. With this list, derived from the best available literature, junior educators can perform strong innovation research and produce subsequent manuscripts. In the future, this list may be refined, built-upon, and consolidated via a consensus process with education-research methodologists, knowledge translation specialists, and journal editors to create a set of formal guidelines similar to PRISMA ³⁰ and CONSORT ³¹ publications.

LIMITATIONS

There are multiple important limitations in our work. Firstly, there is the difficulty defining the medical education innovations type of publication. Innovations are sometimes published as original research using traditional quantitative or qualitative methodology. As such, there is overlap between what may now be termed an innovation report and other publication types. For the purposes of our study, we focused on brief or summarized, educational innovation reports, as exemplified by the Brief Educational Report section in CJEM. ⁸ It is important to note that this list of quality markers is an aggregate list of markers derived from multiple papers, editorials, and guidelines for authors, and does not necessarily correspond with the contributor guidelines for the Brief Educational Report.

There are also limitations of our literature review. We chose a scoping review as a way to capture key concepts related to quality in innovation reports. We felt that a systematic review would be inappropriate, as such reviews aim to answer narrow and specific questions. ¹⁴ We recognized that our question would be difficult to answer given a perceived paucity of existing literature and restriction to English-language papers, so we added a more detailed hand search of journals and electronic guidelines to find additional relevant contributions. Interestingly, our hand search yielded the same number of articles as the traditional database search. This could be an indicator of an overly specific or inadequate database search. Most of the documents found in the hand search were “guidelines for authors” or “aims and scope” submission information from Web-based documents, which are not typically found via systematic database searches of the peer-reviewed literature.

An additional limitation of the literature review was our use of only one of two investigators to review most of the titles and abstracts. This decision was informed by an initial perfect agreement in the review of the first 50 titles. Including additional reviewers might have increased the number of articles included; however, the effect of this is hopefully minimized by the very broad inclusion criteria (single question), with a goal of inclusion rather than exclusion. It can also be noted that over half of the articles were excluded after a full-text review, giving additional evidence that the search was reasonably inclusive. Despite this, there is the potential for introduction of bias without a double-review of all titles.

CONCLUSIONS

The publication and dissemination of innovations are critical for the collective benefit of the community of medical educators and trainees. Our study has attempted to summarize the existing material providing guidance for the publication of medical education innovations and present a list of quality markers for innovation reports. We anticipate that this list could be used by medical education innovators as a set of features to guide the development and publication of medical education innovations in EM. The next steps include consolidating and prioritizing our list of findings to create formal reporting guidelines for innovation reports in medical education.