Hostname: page-component-78c5997874-s2hrs Total loading time: 0 Render date: 2024-11-15T21:14:48.357Z Has data issue: false hasContentIssue false
  • English
  • Français

Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

Part of: Special Collection - Methods

Published online by Cambridge University Press:  18 December 2020

Margaret Sampson Open the ORCID record for Margaret Sampson [Opens in a new window] ,
Nassr Nama Open the ORCID record for Nassr Nama [Opens in a new window] ,
Katharine O'Hearn Open the ORCID record for Katharine O'Hearn [Opens in a new window] ,
Kimmo Murto Open the ORCID record for Kimmo Murto [Opens in a new window] ,
Ahmed Nasr Open the ORCID record for Ahmed Nasr [Opens in a new window] ,
Sherri L. Katz Open the ORCID record for Sherri L. Katz [Opens in a new window] ,
Gail Macartney Open the ORCID record for Gail Macartney [Opens in a new window] ,
Franco Momoli Open the ORCID record for Franco Momoli [Opens in a new window]  and
J. Dayre McNally Open the ORCID record for J. Dayre McNally [Opens in a new window]
Margaret Sampson*
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada
Nassr Nama
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada Department of Pediatrics, BC Children's Hospital, Vancouver, British Columbia, Canada
Katharine O'Hearn
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada
Kimmo Murto
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
Ahmed Nasr
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
Sherri L. Katz
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
Gail Macartney
Affiliation:
Faculty of Nursing, University of Prince Edward Island, Charlottetown, Prince Edward Island, Canada
Franco Momoli
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
J. Dayre McNally
Affiliation:
Children's Hospital of Eastern Ontario, Ottawa, Ontario, Canada Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
*
Author for correspondence: Margaret Sampson, E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Introduction

Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies.

Methods

Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions.

Results

Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases.

Discussion

Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective.

Conclusions

The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

Keywords

PubMedRelevance rankingSystematic reviews as topic
Type
Method
Information
International Journal of Technology Assessment in Health Care , Volume 37 , Issue 1 , 2021 , e7
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Fiorini, N, Canese, K, Starchenko, G, Kireev, E, Kim, W, Miller, V et al. Best match: New relevance search for PubMed. PLoS Biol. 2018;16:e2005343.CrossRefGoogle ScholarPubMed
Mortensen, ML, Adam, GP, Trikalinos, TA, Kraska, T, Wallace, BC. An exploration of crowdsourcing citation screening for systematic reviews. Res Synth Methods. 2017;8:366–86.CrossRefGoogle ScholarPubMed
Bannach-Brown, A, Przybyła, P, Thomas, J, Rice, ASC, Ananiadou, S, Liao, J et al. Machine learning algorithms for systematic review: Reducing workload in a preclinical review of animal studies and reducing human screening error. Syst Rev. 2019;8. doi:10.1186/s13643-019-0942-7CrossRefGoogle Scholar
Nama, N, Sampson, M, Barrowman, N, Sandarage, R, Menon, K, Macartney, G et al. Crowdsourcing the citation screening process for systematic reviews: Validation study. J Med Internet Res. 2019;21:e12953.CrossRefGoogle ScholarPubMed
Nama, N, Barrowman, N, O'Hearn, K, Sampson, M, Zemek, R, McNally, JD. Quality control for crowdsourcing citation screening: The importance of assessment number and qualification set size. J Clin Epidemiol. 2020;122:160–2.CrossRefGoogle ScholarPubMed
Olorisade, BK, Brereton, P, Andras, P. The use of bibliography enriched features for automatic citation screening. J Biomed Inform. 2019;94:103202.CrossRefGoogle ScholarPubMed
Sampson, M, Tetzlaff, J, Urquhart, C. Precision of healthcare systematic review searches in a cross-sectional sample. Res Synth Methods. 2011;2:119–25.CrossRefGoogle Scholar
Sampson, M, Barrowman, NJ, Moher, D, Clifford, TJ, Platt, RW, Morrison, A et al. Can electronic search engines optimize screening of search results in systematic reviews: An empirical study. BMC Med Res Methodol. 2006;6:7.CrossRefGoogle ScholarPubMed
Sampson, M, de Bruijn, B, Urquhart, C, Shojania, K. Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews. J Clin Epidemiol. 2016;78:108–15.CrossRefGoogle ScholarPubMed
Bramer, WM, De Jonge, GB, Rethlefsen, ML, Mast, F, Kleijnen, J. A systematic approach to searching: An efficient and complete method to develop literature searches. J Med Libr Assoc. 2018;106:531–41.CrossRefGoogle ScholarPubMed
Clark, J, Carter, M, Honeyman, D, Cleo, G, Auld, Y, Booth, D et al. The polyglot search translator (PST): Evaluation of a tool for improving searching in systematic reviews: A randomised cross-over trial. In: The 25th Cochrane Colloquium; 2018 Sep 16–18; Cochrane: Edinburgh, UK.Google Scholar
Ashkanase, J, Nama, N, Sandarage, RV, Penslar, J, Gupta, R, Ly, S et al. Identification and evaluation of controlled trials in Pediatric Cardiology: Crowdsourced scoping review and creation of accessible searchable database. Can J Cardiol. 2020;36 (11):1795–1804. doi: 10.1016/j.cjca.2020.01.028.CrossRefGoogle ScholarPubMed
Nama, N, Menon, K, Iliriani, K, Pojsupap, S, Sampson, M, O'Hearn, K et al. A systematic review of pediatric clinical trials of high dose vitamin D. PeerJ. 2016;4:e1701. doi: 10.7717/peerj.1701.CrossRefGoogle ScholarPubMed
Kantor, N, Wayne, C, Nasr, A. Symptom development in originally asymptomatic CPAM diagnosed prenatally: A systematic review. Pediatr Surg Int. 2018;34:613–20.CrossRefGoogle ScholarPubMed
Blinder, H, Momoli, F, Bokhaut, J, Bacal, V, Goldberg, R, Radhakrishnan, D et al. Predictors of adherence to positive airway pressure therapy in children: A systematic review and meta-analysis. Sleep Med. 2020;69:19–33.CrossRefGoogle ScholarPubMed
Clark, J, Glasziou, P, Del Mar, C, Bannach-brown, A, Scott, AM. How to complete a full systematic review in 2 weeks: processes, facilitators and barriers. J Clin Epidemiol. 2020. doi:10.1016/j.jclinepi.2020.01.008CrossRefGoogle Scholar
Higgins, JP, Thomas, J, editors. Searching for and selecting studies. Cochrane handbook for systematic reviews of interventions. Version 6 2019 [cited 2020 Feb 25]. Available from: https://training.cochrane.org/handbook/current/chapter-04#section-4-6..Google Scholar
Miwa, M, Thomas, J, O'Mara-Eves, A, Ananiadou, S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242–53.CrossRefGoogle ScholarPubMed
Khabsa, M, Elmagarmid, A, Ilyas, I, Hammady, H, Ouzzani, M. Learning to identify relevant studies for systematic reviews using random forest and external information. Mach Learn. 2016;102:465–82.CrossRefGoogle Scholar
Edinger, T, Cohen, AM. A large-scale analysis of the reasons given for excluding articles that are retrieved by literature search during systematic review. AMIA Annu Symp Proc. 2013;2013:379–87.Google ScholarPubMed
Supplementary material: File

Sampson et al. supplementary material

Sampson et al. supplementary material 1

Download Sampson et al. supplementary material(File)
File 16.3 KB
Supplementary material: File

Sampson et al. supplementary material

Sampson et al. supplementary material 2

Download Sampson et al. supplementary material(File)
File 31.8 KB