Hostname: page-component-78c5997874-s2hrs Total loading time: 0 Render date: 2024-11-16T11:15:38.789Z Has data issue: false hasContentIssue false
  • English
  • Français

Consensus Among Experts and Research Synthesis: A Comparison of Methods

Published online by Cambridge University Press:  10 March 2009

Paul M. Wortman ,
Joshua M. Smyth ,
John C. Langenbrunner  and
William H. Yeaton
Paul M. Wortman
Affiliation:
State University of New York at Stony Brook
Joshua M. Smyth
Affiliation:
State University of New York at Stony Brook
John C. Langenbrunner
Affiliation:
The World Bank
William H. Yeaton
Affiliation:
University of Michigan
Get access Rights & Permissions [Opens in a new window]

Abstract

A comparison of two assessment methods, consensus among experts and research synthesis of the scientific literature, was performed using a surgical procedure, carotid endarterectomy (CE), as an example. These two methods have been widely advocated as being scientifically valid. While the comparison revealed a number of areas of general agreement, important differences between the two methods emerged. For example, 30-day mortality for asymptomatic patients was considered an effective outcome (ranked first) by the synthesis, but only “quivocal” (ranked third) of six major indicators reported by the consensus method. The synthesis results are also consistent with other literature reviews as well as with recent large-scale randomized trial results. A number of factors that could account for differences between the two methods were examined. Overall, use of consensus panels may be appropriate early in the development of an intervention where the evidence is sparse, while quantitative research synthesis is preferable when a number of high-quality studies have been performed.

Type
General Essays
Information
International Journal of Technology Assessment in Health Care , Volume 14 , Issue 1 , Winter 1998 , pp. 109 - 122
Copyright
Copyright © Cambridge University Press 1998

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

1.American Heart Association. Ad Hoc Committee on Carotid Surgery Standards of the Stroke Council, Special report. Circulation, 1989, 79, 472–74.Google Scholar
2.Antman, E. M., Lau, J., Kupelnick, B., et al. A comparison of results from meta-analyses of randomized control trials and recommendations of clinical experts: Treatments for myocardial infarction. Journal American Medical Association, 1992, 268, 240–48.CrossRefGoogle ScholarPubMed
3.Callow, A. D., Caplan, L. R., Correll, J. W., et al. Carotid endarterectomy: What is its current status? American Journal of Medicine 1988, 85, 835–38.CrossRefGoogle ScholarPubMed
4.Campbell, D. T. (1986). Relabeling internal and external validity for applied social scientists. In Trochim, W. (ed.), Advances in quasi-experimental designs for research. Chicago: Rand-McNally, 1986.Google Scholar
5.Campbell, D. T., & Fiske, D. W.Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin 1959, 56, 81–105.CrossRefGoogle ScholarPubMed
6.Chalmers, T. C., Levin, H., Sacks, H. S., et al. Meta-analysis of clinical trials as a scientific discipline, I: Control of bias and comparison with large co-operative trials. Statistics in Medicine, 1987, 6, 315–25.CrossRefGoogle ScholarPubMed
7.Cook, T. D., & Campbell, D. T.Quasi-experimentation: Design & analysis issues for field settings, Boston: Houghton Mifflin, 1979.Google Scholar
8.Cook, T. D., & Leviton, L. L.Reviewing the literature: A comparison of traditional methods with meta-analysis. Journal of Personality 1980, 48, 449–72.CrossRefGoogle Scholar
9.Cooper, H., & Hedges, L. V., (eds.). The handbook of research synthesis. New York: Russell Sage, 1994.Google Scholar
10.Droitcour, J., Silberman, G., & Chelimsky, E.Cross-design synthesis: A new form of meta-analysis for combining results from randomized clinical trials and medical practice data bases. International Journal of Technology Assessment in Health Care, 1993, 9, 440–49.CrossRefGoogle Scholar
11.Elstein, A. S.Clinical judgment: Psychological research and medical practice. Science 1976, 194, 696–700.CrossRefGoogle ScholarPubMed
12.Executive Committee for the Asymptomatic Carotid Atherosclerosis Study. Endarterectomy for asymptomatic carotid artery stenosis. Journal of the American Medical Association, 1995, 273, 1421–28.CrossRefGoogle Scholar
13.Fields, W. S., Malenikov, V., Meyer, J. S., et al. Joint study of extracranial occlusion. Journal of the American Medical Association, 1970, 210, 1993–2003.Google Scholar
14.Fink, A., Kosecoff, J., Chassin, M., & Brook, R.Consensus methods: Characteristics and guidelines for use. American Journal of Public Health, 1984, 74, 979–83.CrossRefGoogle ScholarPubMed
15.Fisher, E. S., Malenka, D. J., Solomon, N. A., et al. Risk of carotid endarterectomy in the elderly. American Journal of Public Health 1989, 12, 1617–20.CrossRefGoogle Scholar
16.Fredrickson, D. S.Seeking technical consensus on medical interventions. Clinical Research, 1978, 26, 116–17.Google Scholar
17.Glass, G. V.Primary, secondary, and meta-analysis of research. Education Researcher, 1976, 5, 3–8.CrossRefGoogle Scholar
18.Hartmann, D. P. (1977). Considerations in the choice of interobserver reliability estimates. Journal of Applied Behavior, 1976, 11, 103–16.Google Scholar
19.Health Care Financing Administration. Geographic variations in commonly performed procedures under medicare. Baltimore: U.S. Department of Health and Human Services, 1990.Google Scholar
20.Light, R. J.Six evaluation issues that synthesis can resolve better than single studies. New Directions for Program Evaluation, 1984, 24, 57–73.CrossRefGoogle Scholar
21.Lomas, J., Anderson, G., Enkin, M., et al. The role of evidence in the consensus process: Results from a Canadian consensus exercise. Journal of the American Medical Association 1988, 259, 3001–05.CrossRefGoogle ScholarPubMed
22.Me Auliffe, W. E.Measuring the quality of medical care: Process versus outcome. Milbank Memorial Fund Quarterly/Health and Society 1988, 57, 118–52.CrossRefGoogle Scholar
23.Meehl, P. E.Clinical versus statistical prediction: A theoretical analysis and review of evidence. Minneapolis: University of Minnesota Press, 1954.CrossRefGoogle Scholar
24.Merrick, N. J., Fink, A., Brook, R. H., et al. Indications for selected medical and surgical procedures—A literature review and ratings for appropriateness: Carotid endarterectomy. Santa Monica, CA: RAND Corporation, 1986, Publication R-3204/6CWF/HF/PMT/RWJ.Google Scholar
25.Merrick, N. J., Fink, A., Park, R. E., et al. Derivation of clinical indications for carotid endarterectomy by an expert panel. American Journal of Public Health 1987, 77, 187–90.CrossRefGoogle ScholarPubMed
26.Mulley, A. G., & Eagle, K. A.What is inappropriate care? [editorial]. Journal of the American Medical Association, 1988, 260, 540–541.CrossRefGoogle ScholarPubMed
27.North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. New England Journal of Medicine, 1991, 325, 445–53.CrossRefGoogle Scholar
28.Ozminkowski, R. J., Wortman, P. M., & Roloff, D. W.Inborn/outborn status and neonatal survival: A meta-analysis of non-randomized studies. Statistics in Medicine, 1988, 7, 1207–21.CrossRefGoogle ScholarPubMed
29.Park, R. E., Fink, A., Brook, R. H., et al. Physician ratings of appropriate indications for three procedures: Theoretical indications vs. indications used in practice. American Journal of Public Health 1989, 79, 445–47.CrossRefGoogle ScholarPubMed
30.Park, R. E., Fink, A., Brook, R. H., et al. Physician ratings of appropriate indications for six medical and surgical procedures. American Journal of Public Health, 1986, 76, 766–72.CrossRefGoogle ScholarPubMed
31.Perry, S., & Kalberer, J. T.The NIH consensus development program and the assessment of health-care technologies: The first two years. New England Journal of Medicine 1988, 303,169–72.CrossRefGoogle Scholar
32.Schwartz, S., & Griffin, T.Medical thinking: The psychology of medical judgment and decision making. New York: Springer-Verlag, 1986.CrossRefGoogle Scholar
33.Shaw, D. A., Venables, G. S., Cartlidge, N. E. D., et al. Carotid endarterectomy in patients with transient cerebral ischaemia. Journal of Neurologic Sciences, 1984, 64, 45–53.CrossRefGoogle ScholarPubMed
34.Sze, P. C., Reitman, D., Pincus, M. M., et al. Antiplatelet agents in the secondary prevention of stroke: Meta-analysis of the randomized control trials. Stroke 1988, 436–42.CrossRefGoogle Scholar
35.Winslow, C. M., Solomon, D. H., Chassin, M. R., et al. The appropriateness of carotid endarterectomy. New England Journal of Medicine 1988, 318, 721–27.CrossRefGoogle ScholarPubMed
36.Wortman, P. M., Vinokur, A., & Sechrest, L.Do consensus conferences work? A process evaluation of the NIH Consensus Development Program. Journal of Health Politics, Policy and Law 1988, 13, 469–97.CrossRefGoogle Scholar
37.Wortman, P. M., & Yeaton, W. H.Using research synthesis in medical technology assessment. International Journal of Technology Assessment in Health Care 1987, 3, 509–22.CrossRefGoogle ScholarPubMed
38.Yeaton, W. H., Langenbrunner, J. C., Smyth, J. M., & Wortman, P. M.Exploratory research synthesis: Methodological considerations for addressing limitations in data quality. Evaluation and the Health Professions 1995, 18, 283–303.CrossRefGoogle ScholarPubMed
39.Yeaton, W. H., & Wortman, P. M.On the reliability of meta-analytic reviews: The role of intercoder agreement. Evaluation Review 1993, 17, 292–309.CrossRefGoogle Scholar
40.Zurbruegg, H. R., Seiler, R. W., Grolimund, P., & Mattle, H.Morbidity and mortality of carotid endarterectomy: A literature review of the results reported in the last 10 years. Acta Neurochirurgica 1987, 84, 3–12.CrossRefGoogle ScholarPubMed