A writer’s guide to education scholarship: Quantitative methodologies for medical education research (part 1)

Abstract

Background

Quantitative research is one of the many research methods used to help educators advance their understanding of questions in medical education. However, little research has been done on how to succeed in publishing in this area.

Objective

We conducted a scoping review to identify key recommendations and reporting guidelines for quantitative educational research and scholarship.

Methods

Medline, ERIC, and Google Scholar were searched for English-language articles published between 2006 and January 2016 using the search terms, “research design,” “quantitative,” “quantitative methods,” and “medical education.” A hand search was completed for additional references during the full-text review. Titles/abstracts were reviewed by two authors (BT, PC) and included if they focused on quantitative research in medical education and outlined reporting guidelines, or provided recommendations on conducting quantitative research. One hundred articles were reviewed in parallel with the first 30 used for calibration and the subsequent 70 to calculate Cohen’s kappa coefficient. Two reviewers (BT, PC) conducted a full text review and extracted recommendations and reporting guidelines. A simple thematic analysis summarized the extracted recommendations.

Results

Sixty-one articles were reviewed in full, and 157 recommendations were extracted. The thematic analysis identified 86 items, 14 categories, and 3 themes. Fourteen quality evaluation tools and reporting guidelines were found.

Discussion

This paper provides guidance for junior researchers in the form of key quality markers and reporting guidelines. We hope that quantitative researchers in medical education will be informed by the results and that further work will be done to refine the list of recommendations.

Résumé

Introduction

La recherche quantitative est l’une des nombreuses méthodes de recherche utilisées pour aider les éducateurs à mieux comprendre les questions en matière d’enseignement de la médecine. Toutefois, peu de recherche a été faite sur la manière de s’y prendre pour réussir à publier des articles dans le domaine. Aussi avons-nous mené un examen de la portée afin de relever les principales recommandations et les grandes lignes directrices sur l’établissement de rapports en ce qui concerne la recherche quantitative en enseignement et la production de travaux scientifiques.

Méthodes

Les auteurs ont compulsé les bases de données Medline, ERIC et Google Scholar à la recherche d’articles publiés en anglais, entre 2006 et janvier 2016 à l’aide des termes research design, quantitative, quantitative methods et medical education. Une recherche manuelle de références complémentaires a aussi été effectuée durant l’examen des textes en version intégrale. Les titres et les résumés analytiques ont fait l’objet d’un examen par deux auteurs (BT, PC), et les articles ont été retenus s’ils portaient principalement sur la recherche quantitative en enseignement de la médecine et s’ils présentaient des lignes directrices sur l’établissement de rapports ou s’ils faisaient état de recommandations sur la manière d’effectuer de la recherche quantitative. Cent articles ont été examinés en parallèle, dont les 30 premiers aux fins d’étalonnage, et les 70 autres, aux fins du calcul de la valeur kappa de Cohen. Deux examinateurs (BT, PC) ont procédé à l’examen des textes en version intégrale, puis en ont dégagé les recommandations et les lignes directrices sur l’établissement de rapports. C’est à l’aide d’une simple analyse thématique que les recommandations ont ensuite été résumées.

Résultats

Les auteurs ont procédé à l’examen de 61 articles en version intégrale, et dégagé 157 recommandations. L’analyse thématique a permis de diviser la matière en 3 thèmes, en 14 catégories et en 86 éléments. Il s’est dégagé finalement de la recherche 14 outils d’évaluation de la qualité et lignes directrices sur l’établissement de rapports.

Discussion

L’article offre aux jeunes chercheurs un guide présentant les principaux marqueurs de qualité et les grandes lignes directrices sur l’établissement de rapports. Nous espérons que les résultats obtenus sauront guider les futurs spécialistes de la recherche quantitative en enseignement de la médecine et que d’autres travaux de recherche seront entrepris afin d’étoffer la liste des recommandations.

INTRODUCTION

In 2013 the Canadian Association of Emergency Physicians (CAEP) conducted an inaugural academic symposium to develop a list of recommendations to promote educational scholarship among emergency physicians in Canada. ^{1 - 3} The resulting manuscripts emphasized the importance of educational scholarship among emergency physicians in Canada and outlined the systemic and institutional supports necessary to develop clinician educators. Subsequently, this was identified as an area of leadership for emergency physicians. ^{4 , 5} Despite this success, it is challenging to begin conducting research in medical education.

Most residents and physicians in emergency medicine have familiarity with the critical appraisal of quantitative diagnostic and therapeutic trials. However, translating this familiarity into research success remains challenging. ⁶ This is especially true in medical education, where biomedical methodologies are often proposed to study questions that are more appropriately investigated using techniques developed in the social sciences. ⁷ The perception that medical education research is “easy” relative to clinical research also persists, ⁷ despite the availability of numerous masters and PhD level programs focused on preparing trainees to conduct rigorous medical education research.

Herein, as part of a series of writer’s guides aimed at residents and junior scholars, we describe a scoping review ⁸ of the literature that outlines a list of recommendations for conducting and publishing quantitative educational research. In addition, we present a list of guidelines that establish the reporting and writing standards relevant to this field. We hope that this writer’s guide will aid junior investigators as they conduct quantitative research in medical education.

METHODS

Search methodology

The authors conducted a focused literature review of three databases. An initial search using the MEDLINE database was restricted to English-language articles published from 2006 through January 2016 using “and/or” combinations of the following search terms: “research design,” “quantitative,” “quantitative methods,” and “medical education.” A second search using ERIC database was performed of English language articles published from 2006 through January 2016 using “and/or” combinations of the following search terms: “research design,” “quantitative,” “quantitative methods,” and “medical education.” A final search was conducted using Google Scholar, which was limited to English-language papers from 2006-2016 using the keywords “medical education” and “quantitative research.” These articles were amalgamated in Mendeley, and duplicates were removed.

Inclusion criteria

Articles focused on quantitative research in medical education that outlined reporting guidelines, discussed the evaluation of quality in quantitative research, or provided advice on conducting quantitative research. Articles that focused on other forms of education scholarship, mixed methodologies, or did not address any of the study questions were excluded.

Article review

The results of the literature search were uploaded into Mendeley for analysis. Two authors (BT, PC) developed and performed the literature search and conducted the title and abstract review. The first 30 articles were reviewed in parallel to calibrate the reviewers. The subsequent 70 articles were reviewed separately, but in parallel. A kappa value was calculated to quantify inter-rater reliability. The remaining articles were then divided between the two authors (BT, PC) and reviewed by a single reviewer. A hand search of the reference lists of the included articles was conducted to identify those that were not found in the literature search.

A full text review of the included articles was conducted by two authors (BT, PC) with disagreements resolved through a discussion resulting in consensus. During the full text review, any quality evaluation criteria or reporting guidelines mentioned in the papers were abstracted along with recommendations for the conduct of quantitative medical education research.

Analysis

During the full text review, a list of quality markers was developed by two reviewers (BT, PC) by extracting recommendations mentioned in text, figures, charts, diagrams, pictures, or tables. These recommendations were annotated and organized by article from which they were extracted. Two authors (BT, PC) performed a simple thematic analysis refining and aggregating the recommendations into a final common list. Items deemed alike were amalgamated into a single descriptive statement (Table 1). Simultaneously, during the full text review, reporting guidelines and quality assessment tools that were relevant to quantitative research and were mentioned in the text, figures, charts, diagrams, pictures, or tables were extracted and compiled.

Table 1 Results of the qualitative analysis of literature-based recommendations for the performance of quantitative medical education research

RESULTS

Scoping review

The results of the literature review are outlined in Figure 1. The database search identified 356 articles (270 from Medline and 88 from ERIC with 2 duplicates), of which 46 were included on title/abstract review. The Google Scholar search conducted on January 18, 2016 returned 895,000 results. Of the 250 which were reviewed, 7 were included on title/abstract review. An additional 8 articles were included from a review of the references of the included studies.

Figure 1 Flow diagram for systematic review.

The title/abstract review of articles 31 to 100 resulted in concordance on 97.1% (68/70) of the items with a kappa of 0.935 (95% CI=0.847-1.0). The full text review excluded an additional 32 articles; 157 recommendations were extracted from the remaining 29 articles.

Thematic analysis

The simple thematic analysis of the recommendations for quantitative medical education research resulted in 3 themes (study design, writing, and submission), 14 categories, and 86 items (see Table 1).

Quality evaluation tools and reporting guidelines

Table 2 outlines the fourteen quality evaluation tools and reporting guidelines that were found during the full text review, which were applicable to quantitative medical education research.

Table 2 Summary of reporting guidelines and quality assessment tools relevant to quantitative medical education research

*AEM Score = Academic Emergency Medicine Score, MERSQI = Medical Education Research Study Quality Index.

DISCUSSION

Our scoping review ⁸ identified 86 recommendations for conducting and 14 tools/guidelines for reporting high-quality medical education research using quantitative methods. The recommendations found in the literature were congruent; however, their usefulness will be limited by the high number of items that were found. Although the qualitative analysis decreased the gross number of recommendations substantively, 86 may still be too many to consider. Regardless, we believe that reporting the complete list will be helpful to both inform future research and allow junior researchers to identify those items most pertinent to them. Moving forward, we will aim to triage the recommendations to determine which are the most important so that those can be focused on. Multiple methods of achieving consensus among diverse populations have been proposed and could be applied to identify the most important recommendations. ²³

A significant number of quality evaluation criteria and reporting guidelines were found that were applicable to quantitative medical education research. Whereas some of these tools are quite broad, ¹³ covering quantitative research in general, others are extremely specific, outlining specifically how to report quantitative medical education research using a specific methodology. ¹⁶ Because these guidelines outline the criteria for identifying and producing high-quality research, they should serve as a starting point for junior medical education scholars who are designing quantitative studies. We anticipate that these scholars will know what type of quantitative research that they are conducting and quickly identify the most relevant tools for their work using Table 2.

Although this work is likely to provide a helpful starting point for junior researchers, the complexity of medical education research is such that no advice or sets of guidelines can replace formal training in research methodologies nor advice from experts in the field. ^{6 , 24} Indeed, as the depth and scope of quantitative research in medical education continue to grow, increasing amounts of expertise and/or collaboration with skilled partners will be needed to produce high-quality, publication-worthy scholarship. ²⁴

LIMITATIONS

Although our literature review returned a reasonable number of articles for review, the restrictiveness of the search could be criticized. We believe that the databases and search terms that were selected were appropriate for the topic of interest and, given that medical education research has advanced substantially in the past decade, restricting the review to articles published within the last 10 years allowed more modern and applicable lists of recommendations and reporting guidelines to be identified. However, in limiting the search to the English language, we likely missed literature that could have added to our findings.

CONCLUSION

In this paper, we attempted to provide guidance for junior researchers and authors in the form of key recommendations and reporting guidelines. We achieved this through a scoping review of the literature and thematic analysis, with a focus on quantitative educational research and scholarship. We believe that these results will assist junior researchers who heed the recommendations and guide them to key tools for their type of research.