Hostname: page-component-cd9895bd7-7cvxr Total loading time: 0 Render date: 2024-12-18T22:00:41.916Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing the accuracy of forecasting: Applying standard diagnostic assessment tools to a health technology early warning system

Published online by Cambridge University Press:  01 August 2004

Sue Simpson ,
Chris Hyde ,
Alison Cook ,
Claire Packer  and
Andrew Stevens
Sue Simpson
Affiliation:
University of Birmingham
Chris Hyde
Affiliation:
University of Birmingham
Alison Cook
Affiliation:
University of Birmingham
Claire Packer
Affiliation:
University of Birmingham
Andrew Stevens
Affiliation:
University of Birmingham
Get access Rights & Permissions [Opens in a new window]

Abstract

Objectives: Early warning systems are an integral part of many health technology assessment programs. Despite this finding, to date, there have been no quantitative evaluations of the accuracy of predictions made by these systems. We report a study evaluating the accuracy of predictions made by the main United Kingdom early warning system.

Methods: As prediction of impact is analogous to diagnosis, a method normally applied to determine the accuracy of diagnostic tests was used. The sensitivity, specificity, and predictive values of the National Horizon Scanning Centre's prediction methods were estimated with reference to an (imperfect) gold standard, that is, expert opinion of impact 3 to 5 years after prediction.

Results: The sensitivity of predictions was 71 percent (95 percent confidence interval [CI], 0.36–0.92), and the specificity was 73 percent (95 percent CI, 0.64–0.8). The negative predictive value was 98 percent (95 percent CI, 0.92–0.99), and the positive predictive value was 14 percent (95 percent CI, 0.06–0.3).

Conclusions: Forecasting is difficult, but the results suggest that this early warning system's predictions have an acceptable level of accuracy. However, there are caveats. The first is that early warning systems may themselves reduce the impact of a technology, as helping to control adoption and diffusion is their main purpose. The second is that the use of an imperfect gold standard may bias the results. As early warning systems are viewed as an increasingly important component of health technology assessment and decision making, their outcomes must be evaluated. The method used here should be investigated further and the accuracy of other early warning systems explored.

Keywords

Healthcare technologyForecastingSensitivity and specificityReference standardsExpert opinion
Type
RESEARCH REPORTS
Information
International Journal of Technology Assessment in Health Care , Volume 20 , Issue 3 , August 2004 , pp. 381 - 384
Copyright
© 2004 Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Fuchs V, Sox H Jr. 2001 Physicians views of the relative importance of thirty medical innovations. Health Aff. 20: 30– 42.Google Scholar
Hailey D, Topfer L-A, Wills F. 2001 Providing information on emerging health technologies to provincial decision makers: A pilot project. Health Policy. 58: 15– 26.Google Scholar
Hui S, Zhou X. 1998 Evaluation of diagnostic tests without gold standards. Stat Methods Med Res. 7: 354– 370.Google Scholar
Knottnerus J, Van Weel C, Muris J. 2002 Evidence base of clinical diagnosis. Evaluation of diagnostic procedures. BMJ. 324: 477– 480.Google Scholar
Milne R, Clegg A, Stevens A. 2003 HTA responses and the classic HTA report. J Public Health Med. 25: 102– 106.Google Scholar
Robert G, Stevens A, Gabbay J. 1999 ‘Early Warning systems’ for identifying new healthcare technologies. Health Technol Assess. 3: 1– 108.Google Scholar
Simpson S, Carlsson P, Douw K, et al. 2002 A comparative analysis of early warning systems demonstrates differences in methods and structure: A survey of EuroScan member agencies. In: The Challenge of Collaboration Proceedings of the 18th Annual Meeting of the International Society of Technology Assessment in Health Care, Jena: Urban & Fischer Verlag 189.
Spiby J. 1987 Future and emerging medical technology. A study to investigate advance in medical technology in the United Kingdom. Submission to membership of the Faculty of Community Medicine. London: Faculty of Public Health Medicine, Royal College of Physicians
Yerushalmy J. 1947 Statistical problems in assessing methods of medical diagnosis with special reference to x-ray techniques. Public Health Rep. 62: 1432– 1439.Google Scholar