Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-dk4vv Total loading time: 0 Render date: 2024-12-27T18:07:13.116Z Has data issue: false hasContentIssue false

Part V - Computational Perception

Published online by Cambridge University Press:  20 December 2018

Edited by
Ehsan Samei  and
Elizabeth A. Krupinski
Ehsan Samei
Affiliation:
Duke University Medical Center, Durham
Elizabeth A. Krupinski
Affiliation:
Emory University, Atlanta
Get access

Summary

A summary is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.
Image of the first page of this content. For PDF version, please use the ‘Save PDF’ preceeding this image.'
Type
Chapter
Information
The Handbook of Medical Image Perception and Techniques , pp. 357 - 414
Publisher: Cambridge University Press
Print publication year: 2018

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

References

American College of Radiology. (2003). ACR BI-RADS: MRI. In: ACR BI-RADS: Breast Imaging Reporting and Data System: Breast Imaging Atlas. Reston, VA: ACR.Google Scholar
Barrett, H.H., Kupinski, M.A., Clarkson, E. (2005). Probabilistic foundations of the MRMC method. Proc SPIE, 5749, 21–31.CrossRefGoogle Scholar
Beiden, S.V., Wagner, R.F., Campbell, G. (2000). Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol, 7, 341–349.Google Scholar
Bird, R.E., Wallace, T.W., Yankaskas, B.C. (1992). Analysis of cancers missed at screening mammography. Radiology, 184, 613–617.Google Scholar
Birdwell, R.L., Bandodkar, P., Ikeda, D.M. (2005). Computer-aided detection with screening mammography in a university hospital setting. Radiology, 236, 451–457.Google Scholar
Bornefalk, H., Hermansson, A.B. (2005). On the comparison of FROC curves in mammography CAD systems. Med Phys, 32, 412–417.Google Scholar
Bunch, P.C., Hamilton, J.F., Sanderson, G.K., Simmons, A.H. (1977). A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE, 127, 124–135.CrossRefGoogle Scholar
Chakraborty, D.P. (2000). The FROC, AFROC and DROC variants of the ROC analysis. In: Beutel, J., Kundel, H., Van Metter, R. (eds.) Handbook of Medical Imaging, Volume 1. Physics and Psychophysics. Bellingham, WA: SPIE, pp. 771–798.Google Scholar
Chan, H.P., Sahiner, B., Helvie, M.A., et al. (1999). Improvement of radiologists’ characterization of mammographic masses by using computer-aided diagnosis: an ROC study. Radiology, 212, 817–827.Google Scholar
Chen, W., Giger, M.L., Bick, U. (2006a). A fuzzy c-means (FCM) based approach for computerized segmentation of breast lesions in dynamic contrast-enhanced MR images. Acad Radiol, 16, 63–72.Google Scholar
Chen, W., Giger, M.L., Bick, U., Newstead, G. (2006b). Automatic identification and classification of characteristic kinetic curves of breast lesions on DCE-MRI. Med Phys, 33, 2878–2887.Google Scholar
Clarkson, E., Kupinski, M.A., Barrett, H.H. (2006). A probabilistic model for the MRMC method. Part I. Theoretical development. Acad Radiol, 13, 1410–1421.Google Scholar
Cupples, T., Cunningham, J.E., Reynolds, J.C. (2005). Impact of computer-aided detection in a regional screening mammography program. AJR, 185, 944–950.Google Scholar
Dean, J.C., Iivento, C.C. (2006). Improved cancer detection using computer-aided detection with diagnostic and screening mammography: prospective study of 104 cancers. AJR, 187, 20–28.Google Scholar
Dorfman, D.D., Berbaum, K.S., Metz, C.E. (1992). Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol, 27, 723–731.CrossRefGoogle Scholar
Feig, S.A., Sickes, E.A., Evans, W.P., Linver, M.N. (2004). Re: Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 1260–1261.Google Scholar
Fenton, J.J., Taplin, S.H., Carney, P.A., et al. (2007). Influence of computer-aided detection on performance of screening mammography. N Engl J Med, 356, 1399–1409.Google Scholar
Flehinger, B.J., Kimmel, M., Melamed, M.R. (1992). The effect of surgical treatment on survival from early lung cancer. Implications for screening. Chest, 101, 1013–1018.CrossRefGoogle ScholarPubMed
Freedman, M., Lo, S., Lure, F., et al. (2001). Computer-aided detection of lung cancer on chest radiographs: algorithm performance vs. radiologists’ performance by size of cancer. Proc SPIE, 4319, 150–159.Google Scholar
Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection. prospective study of 12,860 patients in a community breast center. Radiology, 222, 781–786.Google Scholar
Gallas, B.D. (2006). One-shot estimate of MRMC variance: AUC. Acad Radiol, 13, 353–362.Google Scholar
Gallas, B.D., Brown, D.G. (2008). Reader studies for validation of CAD systems. Neural Networks, 21, 387–397.Google Scholar
Gallas, B.D., Pennello, G.A., Myers, K.J. (2007). Multi-reader multi-case variance analysis for binary data. J Opt Soc Am A, 24(12), B70–B80.Google Scholar
Giger, M.L., Huo, Z., Kupinski, M.A., Vyborny, C.J. (2000). Computer-aided diagnosis in mammography. In: Sonka, M., Fitzpatrick, M.J. (eds.) Handbook of Medical Imaging, Volume 2. Medical Imaging Processing and Analysis. Bellingham, WA: SPIE, pp. 915–1004.Google Scholar
Giger, M.L., Huo, Z., Vyborny, C.J., et al. (2003). Results of an observer study with an intelligent mammographic workstation for CAD. In: Peitgen, H.-O. (ed.) Digital Mammography, IWDM 2002. Berlin: Springer, pp. 297–303.Google Scholar
Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. Am J Roentgenol, 190(4), 854–859.Google Scholar
Gur, D., Sumkin, J.H., Rockette, H.E., et al. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.Google Scholar
Gur, D., Bandos, A.I., Cohen, C.S., et al. (2008). The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology, 249, 47–53.Google Scholar
Hara, A.K., Johnson, C.D., Reed, J.E., et al. (1997). Detection of colorectal polyps with CT colography: initial assessment of sensitivity and specificity. Radiology, 205, 59–65.Google Scholar
Helvie, M., Hadjiiski, L., Makariou, E., et al. (2004). Sensitivity of noncommercial computer-aided detection system for mammographic breast cancer detection: pilot clinical trial. Radiology, 231, 208–214.Google Scholar
Henschke, C.I., Naidich, D.P., Yankelevitz, D.F., et al. (2001). Early lung cancer action project: initial findings on repeat screenings. Cancer, 92, 153–159.3.0.CO;2-S>CrossRefGoogle ScholarPubMed
Horsch, K., Giger, M.L., Vyborny, C.J., et al. (2006). Multi-modality computer-aided diagnosis for the classification of breast lesions: observer study results on an independent clinical dataset. Radiology, 240, 357–368.Google Scholar
Huo, Z., Giger, M.L., Vyborny, C.J., et al. (1995). Analysis of spiculation in the computerized classification of mammographic masses. Med Phys, 22, 1569–1579.Google Scholar
Huo, Z., Giger, M.L., Vyborny, C.J., et al. (2002). Effectiveness of CAD in the diagnosis of breast cancer: an observer study on an independent database of mammograms. Radiology, 224, 560–568.Google Scholar
Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750.Google Scholar
Jiang, Y., Nishikawa, R.M., Schmidt, R.A., et al. (1999). Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol, 6, 22.Google Scholar
Jiang, Y., Nishikawa, R.M., Schmidt, R.A., et al. (2001). Potential of computer-aided diagnosis to reduce variability in radiologists’ interpretations of mammograms depicting microcalcifications. Radiology, 220, 787–794.Google Scholar
Khoo, L.A.L., Taylor, P., Given-Wilson, R.M. (2005). Computer detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.Google Scholar
Krupinski, E.A., Kundel, H.L., Judy, P.F., et al. (1998). Key issues for image perception research. Radiology, 209, 611–612.Google Scholar
Kuhl, C.K., Mielcareck, P., Klaschik, S., et al. (1999). Dynamic breast MR imaging: are signal intensity time course data useful for differential diagnosis of enhancing lesions? Radiology, 211, 101–110.Google Scholar
Kundel, H. (1975). Peripheral vision, structured noise and film reader error. Radiology, 114, 269–273.CrossRefGoogle ScholarPubMed
Kupinski, M.A., Giger, M.L. (1998). Automated seeded lesion segmentation on digital mammograms. IEEE Trans Med Imag, 17, 510–517.Google Scholar
Kupinski, M., Clarkson, E., Barrett, H. (2006). A probabilistic model for the MRMC method, part 2: validation and applications. Acad Radiol, 13, 1422–1430.CrossRefGoogle ScholarPubMed
McFarland, E.G., Brink, J.A., Pilgram, T.K., et al. (2001). Spiral CT colonography: reader agreement and diagnostic performance with two- and three-dimensional image-display techniques. Radiology, 218, 375–383.Google Scholar
Metz, C.E. (1978). Basic principles of ROC analysis. Semin Nucl Med, 8, 283–298.Google Scholar
Metz, C.E. (1986). ROC methodology in radiologic imaging. Invest Radiol, 21, 720–733.Google Scholar
Metz, C.E. (2000). Fundamental ROC analysis. In: Beutel, J., Kundel, H., Van Metter, R. (eds.) Handbook of Medical Imaging, Volume 1. Physics and Psychophysics. Bellingham, WA: SPIE, pp. 751–769.Google Scholar
Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar
Muramatsu, C., Li, Q., Suzuki, K., et al. (2005). Investigation of psychophysical measures for evaluation of similar images for mammographic masses: preliminary results. Med Phys, 32, 2295–2304.Google Scholar
Nappi, J., Yoshida, H. (2003). Feature-guided analysis for reduction of false positives in CAD of polyps for computed tomographic colonography. Med Phys, 30, 1592–1601.Google Scholar
Nishikawa, R.M., Giger, M.L., Vyborny, C.J., et al. (2001). Prospective computer analysis of cancers missed on screening mammography. In: Digital Mammography 2000, Proceedings of the 5th International Workshop on Digital Mammography. Madison, WI: Medical Physics, pp. 493–498.Google Scholar
Petrick, N., Haider, M., Summers, R.M., et al. (2008). CT colonography with computer-aided detection as a second reader: an observer performance study. Radiology, 246(1), 148–156.Google Scholar
Renfrew, D.L., Franken, E.A., Jr., Berbaum, K.S., Weigelt, F.H., AbuYousef, M.M. (1992). Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology, 183, 145–150.Google Scholar
Royster, A.P., Fenlon, H.M., Clarke, P.D., Nunes, D.P., Ferrucci, J.T. (1997). CT colonoscopy of colorectal neoplasms: two-dimensional and three-dimensional virtual-reality techniques with colonoscopic correlation. Am J Roentgenol, 169, 1237–1242.Google Scholar
Sahiner, B., Chan, H.-P., Petrick, N., Wagner, R.F., Hadjiiski, L. (2000). Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys, 27, 1509–1522.Google Scholar
Samuelson, F.W., Petrick, N. (2006). Comparing image detection algorithms using resampling. Proceedings of the 2006 IEEE International Symposium on Biomedical Imaging, pp. 1312–1315.Google Scholar
Shah, P.K., Austin, J.H., White, C.S., et al. (2003). Missed non-small cell lung cancer: radiographic findings of potentially resectable lesions evident only in retrospect. Radiology, 226, 235–241.Google Scholar
Sone, S., Li, F., Yang, Z.G., et al. (2001). Results of three-year mass screening programme for lung cancer using mobile low-dose spiral computed tomography scanner. Br J Cancer, 84, 25–32.Google Scholar
Sonka, M., Fitzpatrick, M.J. (eds.) (2000). Handbook of Medical Imaging, Volume 2. Medical Imaging Processing and Analysis. Bellingham, WA: SPIE.Google Scholar
Summers, R.M., Johnson, C.D., Pusanik, L.M., et al. (2001). Automated polyp detection at CT colonography: feasibility assessment in a human population. Radiology, 219, 51–59.Google Scholar
Summers, R.M., Yao, J., Pickhardt, P.J., et al. (2005). Computed tomographic virtual colonoscopy computer-aided polyp detection in a screening population. Gastroenterology, 129, 1832–1844.Google Scholar
Suzuki, K., Yoshida, H., Nappi, J., Dachman, A.H. (2006). Massive-training artificial neural network (MTANN) for reduction of false positives in computer-aided detection of polyps: suppression of rectal tubes. Med Phys, 33, 3814–3824.Google Scholar
Swensen, S.J., Jett, J.R., Hartman, T.E., et al. (2003). Lung cancer screening with CT: Mayo Clinic experience. Radiology, 226, 756–761.Google Scholar
Taylor, S.A., Charman, S.C., Lefere, P., et al. (2007). CT colonography: investigation of the optimum reader paradigm by using computer-aided detection software. Radiology, 246(2), 463–471.Google Scholar
Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar
Yoshida, H., Dachman, A. (2004). Computer-aided diagnosis for CT colonography. Semin Ultrasound CT MR, 25, 419–431.Google Scholar
Yoshida, H., Masutani, Y., MacEneaney, P., Rubin, D.T., Dachman, A.H. (2002). Computerized detection of colonic polyps at CT colonography on the basis of volumetric features: pilot study. Radiology, 222: 327–336.Google Scholar
Yousef, W.A., Wagner, R.F., Loew, M.H. (2005). Estimating the uncertainty in the estimated mean area under the ROC curve of a classifier. Patt Recog Lett, 26, 2600–2610.Google Scholar
Yousef, W.A., Wagner, R.F., Loew, M.H. (2006). Assessing classifiers from two independent data sets using ROC analysis: a nonpara-metric approach. IEEE Trans Patt Anal Mach Intell, 28, 1809–1817.Google Scholar
Yuan, Y., Giger, M.L., Li, H., Suzuki, K., Sennett, C. (2007). A dual-stage method for lesion segmentation on digital mammograms. Med Phys, 34, 4180–4193.Google Scholar
Zheng, B., Ganott, M.A., Britton, C.A., et al. (2001). Soft-copy mammographic reading with different computer-assisted detection cuing environments: preliminary findings. Radiology, 221, 633–640.Google Scholar
Zheng, B., Swensson, R.G., Golla, S., et al. (2004). Detection and classification performance levels of mammographic masses under different computer-aided detection cueing environments. Acad Radiol, 11, 396–406.Google Scholar
Zheng, B., Mello-Thoms, C.C., Wang, X.-H., et al. (2007). Interactive computer-aided diagnosis of breast masses: computerized selection of visually similar image sets from a reference library. Acad Radiol, 14, 917–927.Google Scholar

References

Alberdi, E., Povyakalo, A.A., Strigini, L., Ayton, P., Hartswood, M., Procter, R., Slack, R. (2005). Use of computer-aided detection (CAD) tools in screening mammography: a multidisciplinary investigation. Br J Radiol, 78 Spec No 1, S31–S40.Google Scholar
American College of Radiology (ACR) (2003). The Breast Imaging Reporting and Data System Atlas. Reston, VA: American College of Radiology.Google Scholar
Ballard-Barbash, R., Taplin, S.H., Yankaskas, B.C., Ernster, V.L., Rosenberg, R.D., Carney, P.A., Barlow, W.E., et al. (1997). Breast Cancer Surveillance Consortium: a national mammography screening and outcomes database. AJR Am J Roentgenol, 169, 1001–1008.Google Scholar
Barlow, W.E., Chi, C., Carney, P.A., Taplin, S.H., D’Orsi, C., Cutter, G., Hendrick, R.E., et al. (2004). Accuracy of screening mammography interpretation by characteristics of radiologists. J Natl Cancer Inst, 96, 1840–1850.Google Scholar
Beam, C.A., Layde, P.M., Sullivan, D.C. (1996). Variability in the interpretation of screening mammograms by US radiologists. Findings from a national sample. Arch Intern Med, 156, 209–213.Google Scholar
Beiden, S.V., Wagner, R.F., Doi, K., Nishikawa, R.M., Freedman, M., Lo, S.C., Xu, X.W. (2002). Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol, 9, 1036–1043.Google Scholar
Birdwell, R.L., Ikeda, D.M., O’Shaughnessy, K.F., Sickles, E.A. (2001). Mammographic characteristics of 115 missed cancers later detected with screening mammography and the potential utility of computer-aided detection. Radiology, 219, 192–202.Google Scholar
Birdwell, R.L., Bandodkar, P., Ikeda, D.M. (2005). Computer-aided detection with screening mammography in a university hospital setting. Radiology, 236, 451–457.Google Scholar
Brem, R.F., Baum, J., Lechner, M., Kaplan, S., Souders, S., Naul, L.G., Hoffmeister, J. (2003). Improvement in sensitivity of screening mammography with computer-aided detection: a multiinstitutional trial. AJR Am J Roentgenol, 181, 687–693.Google Scholar
Chan, H.P., Doi, K., Vyborny, C.J., Schmidt, R.A., Metz, C.A., Lam, K.L., Ogura, T., et al. (1990). Improvement in radiologists’ detection of clustered microcalcifications on mammograms. The potential of computer-aided diagnosis. Invest Radiol, 25, 1102–1110.Google Scholar
Ciatto, S., Ambrogetti, D., Collini, G., Cruciani, A., Ercolini, E., Risso, G., Rosselli Del Turco, M. (2006). Computer-aided detection (CAD) of cancers detected on double reading by one reader only. Breast, 15, 528–532.Google Scholar
Cupples, T.E., Cunningham, J.E., Reynolds, J.C. (2005). Impact of computer-aided detection in a regional screening mammography program. AJR Am J Roentgenol, 185, 944–950.Google Scholar
Dean, J.C., Ilvento, C.C. (2006). Improved cancer detection using computer-aided detection with diagnostic and screening mammography: prospective study of 104 cancers. AJR Am J Roentgenol, 187, 20–28.Google Scholar
Destounis, S.V., DiNitto, P., Logan-Young, W., Bonaccio, E., Zuley, M.L., Willison, K.M. (2004). Can computer-aided detection with double reading of screening mammograms help decrease the false-negative rate? Initial experience. Radiology, 232, 578–584.Google Scholar
Elmore, J.G., Wells, C.K., Howard, D.H. (1994). Variability in radiologists’ interpretations of mammograms. N Engl J Med, 331, 1493–1499.Google Scholar
Feig, S.A., Sickles, E.A., Evans, W.P., Linver, M.N. (2004). Re: Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 1260–1261; author reply 1261.Google Scholar
Fenton, J.J., Taplin, S.H., Carney, P.A., et al. (2007). Influence of computer-aided detection on performance of screening mammography. N Engl J Med, 356, 1399–1409.Google Scholar
Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology, 220, 781–786.Google Scholar
Gilbert, F.J., Astley, S.M., McGee, M.A., Gillan, M.G., Boggis, C.R., Griffiths, P.M., Duffy, S.W. (2006). Single reading with computer-aided detection and double reading of screening mammograms in the United Kingdom National Breast Screening Program. Radiology, 241, 47–53.Google Scholar
Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. AJR Am J Roentgenol, 190, 854–859.Google Scholar
Gur, D., Sumkin, J.H., Rockette, H.E., et al. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.Google Scholar
Gur, D., Bandos, A.I., Fuhrman, C.R., Klym, A.H., King, J.L., Rockette, H.E. (2007). The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol, 14, 49–53.Google Scholar
Helvie, M.A., Hadjiiski, L., Makariou, E., et al. (2004). Sensitivity of noncommercial computer-aided detection system for mammographic breast cancer detection: pilot clinical trial. Radiology, 231, 208–214.Google Scholar
Ikeda, D.M., Birdwell, R.L., O’Shaughnessy, K.F., Sickles, E.A., Brenner, R.J. (2004). Computer-aided detection output on 172 subtle findings on normal mammograms previously obtained in women with breast cancer detected at follow-up screening mammography. Radiology, 230, 811–819.Google Scholar
Jemal, A., Siegel, R., Ward, E., Hao, Y., Xu, J., Murray, T., Thun, M.J. (2008). Cancer statistics, 2008. CA Cancer J Clin, 58, 71–96.CrossRefGoogle Scholar
Jiang, Y., Miglioretti, D.L., Metz, C.E., Schmidt, R.A. (2007). Breast cancer detection rate: designing imaging trials to demonstrate improvements. Radiology, 243, 360–367.Google Scholar
Karssemeijer, N., Otten, J.D., Verbeek, A.L.M., Groenewoud, J.H., de Koning, H.J., Hendriks, J.H.C.L., Holland, R. (2003). Computer-aided detection versus independent double reading of masses on mammograms. Radiology, 227, 192–200.Google Scholar
Kegelmeyer, W.P., Jr., Pruneda, J.M., Bourland, P.D., Hillis, A., Riggs, M.W., Nipper, M.L. (1994). Computer-aided mammographic screening for spiculated lesions. Radiology, 191, 331–337.Google Scholar
Khoo, L.A., Taylor, P., Given-Wilson, R.M. (2005). Computer-aided detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.CrossRefGoogle ScholarPubMed
Ko, J.M., Nicholas, M.J., Mendel, J.B., Slanetz, P.J. (2006). Prospective assessment of computer-aided detection in interpretation of screening mammography. AJR Am J Roentgenol, 187, 1483–1491.Google Scholar
Kobayashi, T., Xu, X.W., MacMahon, H., Metz, C.E., Doi, K. (1996). Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology, 199, 843–848.Google Scholar
Krupinski, E.A., Jiang, Y. (2008). Anniversary paper: evaluation of medical imaging systems. Med Phys, 35, 645–659.Google Scholar
Marx, C., Malich, A., Facius, M., Grebenstein, U., Sauner, D., Pfleiderer, S.O., Kaiser, W.A. (2004). Are unnecessary follow-up procedures induced by computer-aided diagnosis (CAD) in mammography? Comparison of mammographic diagnosis with and without use of CAD. Eur J Radiol, 51, 66–72.Google Scholar
Metz, C.E. (1978). Basic principles of ROC analysis. Semin Nucl Med, 8, 283–298.Google Scholar
Metz, C.E. (1989). Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol, 24, 234–245.Google Scholar
Moberg, K., Bjurstam, N., Wilczek, B., Rostgard, L., Egge, E., Muren, C. (2001). Computed assisted detection of interval breast cancers. Eur J Radiol, 39, 104–110.Google Scholar
Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar
Nishikawa, R.M. (2006). Modeling the effect of computer-aided detection on the sensitivity of screening mammography. In: Astley, S.M., Brady, M., Rose, C., Zwiggelaar, R. (eds.) Digital Mammography. London: Springer, pp. 46–53.CrossRefGoogle Scholar
Pisano, E.D., Gatsonis, C., Hendrick, E., et al. (2005). Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med, 353, 1773–1783.Google Scholar
Schmidt, R.A., Newstead, G.M., Linver, M.N., Eklund, G.W., Metz, C.E., Winkler, M.N., Nishikawa, R.M. (1998). Mammographic screening sensitivity of general radiologists. In: Karssemeijer, N., Thijssen, M., Hendriks, J., van Erning, L. (eds.) Digital Mammography. Dordrecht: Kluwer Academic, pp. 383–388.Google Scholar
Skaane, P., Kshirsagar, A., Stapleton, S., Young, K., Castellino, R.A. (2007). Effect of computer-aided detection on independent double reading of paired screen-film and full-field digital screening mammograms. AJR Am J Roentgenol, 188, 377–384.Google Scholar
Taplin, S.H., Rutter, C.M., Lehman, C.D. (2006). Testing the effect of computer-assisted detection on interpretive performance in screening mammography. AJR Am J Roentgenol, 187, 1475–1482.CrossRefGoogle ScholarPubMed
Taylor, P., Given-Wilson, R.M. (2005). Evaluation of computer-aided detection (CAD) devices. Br J Radiol, 78 Spec No 1, S26– S30.Google Scholar
te Brake, G.M., Karssemeijer, N., Hendriks, J.H. (1998). Automated detection of breast carcinomas not detected in a screening program. Radiology, 207, 465–471.Google Scholar
Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar
Warren Burhenne, L.J., Wood, S.A., D’Orsi, C.J., Feig, S.A., Kopans, D.B., O’Shaughnessy, K.F., Sickles, E.A., et al. (2000). Potential contribution of computer-aided detection to the sensitivity of screening mammography. Radiology, 215, 554–562.Google Scholar
Zheng, B., Shah, R., Wallace, L., Hakim, C., Ganott, M.A., Gur, D. (2002). Computer-aided detection in mammography: an assessment of performance on current and prior images. Acad Radiol, 9, 1245–1250.CrossRefGoogle ScholarPubMed

References

Abbey, C.K., Wu, Y., Burnside, E.S., Wunderlich, A., Samuelson, F.W., Boone, J.M. (2016). A utility/cost analysis of breast cancer risk prediction algorithms. Proc SPIE Med Imag, 9887, 97871J.Google Scholar
Adrion, W.R., Branstad, M.A., Cherniavsky, J.C. (1982). Validation, verification, and testing of computer software. ACM Comp Surv, 14, 159–192.Google Scholar
Aerts, H.J.W.L. (2016). The potential of radiomic-based phenotyping in precision medicine: a review. JAMA Oncol, 2, 1636–1642.Google Scholar
Agresti, A., Coull, B.A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. Am Statist, 52, 119–126.Google Scholar
Altman, D.G., Bland, J.M. (1999). Statistics notes – treatment allocation in controlled trials: why randomise? Br Med J, 318, 1209–1209.Google Scholar
Aoki, T., Oda, N., Yamashita, Y., Yamamoto, K., Korogi, Y. (2011). Usefulness of computerized method for lung nodule detection in digital chest radiographs using temporal subtraction images. Acad Radiol, 18, 1000–1005.Google Scholar
Armato, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P. et al. (2011). The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys, 38, 915–931.Google Scholar
Bandos, A.I., Rockette, H.E., Song, T., Gur, D. (2009). Area under the free-response ROC curve (FROC) and a related summary index. Biometrics, 65, 247–256.Google Scholar
Belle, A., Thiagarajan, R., Soroushmehr, S.M.R., Navidi, F., Beard, D.A., Najarian, K. (2015). Big data analytics in healthcare. BioMed Res Intl, 2015, 1–16.Google Scholar
Bergstra, J., Yamins, D., Cox, D. (2013). Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, 28, 115–123.Google Scholar
Bogoni, L., Ko, J.P., Alpert, J., Anand, V., Fantauzzi, J., Florin, C.H., Koo, C.W., Mason, D., Rom, W., Shiau, M., Salganicoff, M., Naidich, D.P. (2012). Impact of a computer-aided detection (CAD) system integrated into a picture archiving and communication system (PACS) on reader sensitivity and efficiency for the detection of lung nodules in thoracic CT exams. J Digit Imag, 25, 771–781.Google Scholar
Boone, D., Mallett, S., McQuillan, J., Taylor, S.A., Altman, D.G., Halligan, S. (2015). Assessment of the incremental benefit of computer-aided detection (CAD) for interpretation of CT colonography by experienced and inexperienced readers. PLos One, 10.Google Scholar
Bornefalk, H. (2005). Estimation and comparison of CAD system performance in clinical settings. Acad Radiol, 12, 687–94.Google Scholar
Brown, L.D., Cai, T.T., DasGupta, A. (2001a). Interval estimation for a binomial proportion. Stat Sci, 16, 101–117.Google Scholar
Brown, L.D., Cai, T.T., DasGupta, A. (2001b). Interval estimation for a binomial proportion-comment-rejoiner. Stat Sci, 16, 117–133.Google Scholar
Brown, M.S., Goldin, J.G., Rogers, S., Kim, H.J., Suh, R.D., McNitt-Gray, M.F., Shah, S.K., Truong, D., Brown, K., Sayre, J.W., Gjertson, D.W., Batra, P., Aberle, D.R. (2005). Computer-aided lung nodule detection in CT: results of large-scale observer test. Acad Radiol, 12, 681–686.Google Scholar
Bunch, P.C., Hamilton, J.F., Sanderson, G.K., Simmons, A.H. (1978). A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng, 4, 166–171.Google Scholar
Byron, S.A., Van Keuren-Jensen, K.R., Engelthaler, D.M., Carpten, J.D., Craig, D.W. (2016). Translating RNA sequencing into clinical diagnostics: opportunities and challenges. Nat Rev Genet, 17, 257–271.Google Scholar
Chakraborty, D.P. (1989). Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 16, 561–568.Google Scholar
Chakraborty, D.P. (2006a). Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol, 13, 1187–1193.Google Scholar
Chakraborty, D.P. (2006b). A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol, 51, 3449–3462.Google Scholar
Chakraborty, D.P. (2008). Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol, 15, 1554–1566.Google Scholar
Chakraborty, D.P. (2017) FROC methodology website. Available at: www.devchakraborty.com/index.php (accessed November 10, 2017).Google Scholar
Chakraborty, D.P., Berbaum, K.S. (2004). Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys, 31, 2313–2330.Google Scholar
Chakraborty, D.P., Winter, L.H.L. (1990). Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology, 174, 873–881.Google Scholar
Chan, H.P., Wei, D., Helvie, M.A., Sahiner, B., Adler, D.D., Goodsitt, M.M., Petrick, N. (1995). Computer-aided classification of mammographic masses and normal tissue: linear discriminant analysis in texture feature space. Phys Med Biol, 40, 857.Google Scholar
Chan, H.P., Wei, J., Zhang, Y.H., Helvie, M.A., Moore, R.H., Sahiner, B., Hadjiiski, L., Kopans, D.B. (2008). Computer-aided detection of masses in digital tomosynthesis mammography: comparison of three approaches. Med Phys, 35, 4087–4095.Google Scholar
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P. (2002). SMOTE: synthetic minority over-sampling technique. J Artif Intellig Res, 16, 321–357.CrossRefGoogle Scholar
Choudhury, K.R., Paik, D.S., Yi, C.A., Napel, S., Roos, J., Rubin, G.D. (2010). Assessing operating characteristics of CAD algorithms in the absence of a gold standard. Med Phys, 37, 1788–1795.Google Scholar
Dachman, A.H., Obuchowski, N.A., Hoffmeister, J.W., Hinshaw, J.L., Frew, M.I., Winter, T.C., Van Uitert, R.L., Periaswamy, S., Summers, R.M., Hillman, B.J. (2010). Effect of computer-aided detection for CT colonography in a multireader, multicase trial. Radiology, 256, 827–835.Google Scholar
De Boo, D.W., Uffmann, M., Weber, M., Bipat, S., Boorsma, E.F., Scheerder, M.J., Freling, N.J., Schaefer-Prokop, C.M. (2011). Computer-aided detection of small pulmonary nodules in chest radiographs: an observer study. Acad Radiol, 18, 1507–1514.Google Scholar
de Groot, J.A.H., Janssen, K.J.M., Zwinderman, A.H., Moons, K.G.M., Reitsma, J.B. (2008). Multiple imputation to correct for partial verification bias revisited. Stat Med, 27, 5880–5889.Google Scholar
de Hoop, B., De Boo, D.W., Gietema, H.A., van Hoorn, F., Mearadji, B., Schijf, L., van Ginneken, B., Prokop, M., Schaefer-Prokop, C. (2010). Computer-aided detection of lung cancer on chest radiographs: effect on observer performance. Radiology, 257, 532–540.Google Scholar
Dorfman, D.D., Alf, E. (1969). Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals – rating-method data. J Math Psych, 6, 487–496.Google Scholar
Dorfman, D.D., Berbaum, K.S., Metz, C.E., Lenth, R.V., Hanley, J.A., Dagga, H.A. (1997). Proper receiver operating characteristic analysis: the bigamma model. Acad Radiol, 4, 138–149.Google Scholar
Dwork, C. (2011). The promise of differential privacy a tutorial on algorithmic techniques. In: Ostrovsky, R. (ed.) 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, pp. 1–2.Google Scholar
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., Roth, A. (2015). The reusable holdout: preserving validity in adaptive data analysis. Science, 349, 636–638.Google Scholar
Edwards, D.C., Kupinski, M.A., Metz, C.E., Nishikawa, R.M. (2002). Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys, 29, 2861–2870.Google Scholar
Edwards, D.C., Metz, C.E., Kupinski, M.A. (2004). Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans Med Imag, 23, 891–895.Google Scholar
Efron, B. (1983). Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Statis Assn, 78, 316–331.Google Scholar
Efron, B., Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Stat Sci, 1, 54–75.Google Scholar
Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115.Google Scholar
Evans, K.K., Birdwell, R.L., Wolfe, J.M. (2013). If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screening. PLos One, 8.Google Scholar
Evans, B.J., Burke, W., Jarvik, G.P. (2015). The FDA and genomic tests – getting regulation right. N Engl J Med, 372, 2258–2264.Google Scholar
FDA (2012a). Guidance for industry and FDA staff: clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device data – premarket approval (PMA) and premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187315.pdf (accessed November 21, 2017).Google Scholar
FDA (2012b). Guidance for industry and FDA staff: computer-assisted detection devices applied to radiology images and radiology device data – premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187294.pdf (accessed November 21, 2017).Google Scholar
FDA (2013). Design considerations for pivotal clinical investigations for medical devices – guidance for industry, clinical investigators, institutional review boards and Food and Drug Administration staff. Available at: www.fda.gov/RegulatoryInformation/Guidances/ucm373750.htm (accessed November 21, 2017).Google Scholar
FDA (2017a). QuantX DEN170022 reclassification order. Available at: www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?ID=DEN170022 (accessed November 21, 2017).Google Scholar
FDA (2017b). iMRMC software. Available at: https://github.com/DIDSR/iMRMC (accessed November 10, 2017).Google Scholar
Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Trans Patt Anal Mach Intell, 32, 1627–1645.Google Scholar
Fenton, J.J., Abraham, L., Taplin, S.H., Geller, B.M., Carney, P.A., D’Orsi, C. et al. (2011). Effectiveness of computer-aided detection in community mammography practice. J Natl Cancer Inst, 103, 1152–1161.Google Scholar
Fisichella, V.A., Jaderling, F., Horvath, S., Stotzer, P.O., Kilander, A., Bath, M., Hellstrom, M. (2009). Computer-aided detection (CAD) as a second reader using perspective filet view at CT colonography: effect on performance of inexperienced readers. Clin Radiol, 64, 972–982.Google Scholar
Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology, 220, 781–786.Google Scholar
Gallas, B.D., Brown, D.G. (2008). Reader studies for validation of CAD systems. Neural Networks, 21, 387–397.Google Scholar
Gallas, B.D., Chan, H.P., D’Orsi, C.J., Dodd, L.E., Giger, M.L., Gur, D., Krupinski, E.A., Metz, C.E., Myers, K. J., Obuchowski, N.A., Sahiner, B., Toledano, A.Y., Zuley, M.L. (2012). Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 19, 463–477.Google Scholar
Gallas, B.D., Pisano, E., Cole, E., Myers, K. (2017). Impact of different study populations on reader behavior and performance metrics: initial results. Proc SPIE Med Imag, 10136.Google Scholar
Gehan, E.A., Freireich, E.J. (1974). Non-randomized controls in cancer clinical trials. N Engl J Med, 290, 198–203.Google Scholar
Gilbert, F.J., Astley, S.M., Gillan, M.G.C., Agbaje, O.F., Wallis, M.G., James, J., Boggis, C.R.M., Duffy, S.W. (2008). Single reading with computer-aided detection for screening mammography. N Engl J Med, 359, 1675–1684.Google Scholar
Godoy, M.C.B., Kim, T.J., White, C.S., Bogoni, L., de Groot, P., Florin, C., Obuchowski, N., Babb, J.S., Salganicoff, M., Naidich, D.P., Anand, V., Park, S., Vlahos, I., Ko, J.P. (2013). Benefit of computer-aided detection analysis for the detection of subsolid and solid lung nodules on thin- and thick-section CT. Am J Roentgenol, 200, 74–83.Google Scholar
Gomez, S.S., Tabanera, M.T., Bolivar, A.V., Miranda, M.S., Mazo, A.B., Diaz, M.R., Miravete, P.M., Asturiano, E.L., Cacho, P.M., Macias, T.D. (2011). Impact of a CAD system in a screen-film mammography screening program: a prospective study. Eur J Radiol, 80, E317–E321.Google Scholar
Goo, J.M., Kim, H.Y., Lee, J. W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T., Goo, J.M., Kim, H.Y., Lee, J.W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T. (2008). Is the computer-aided detection scheme for lung nodule also useful in detecting lung cancer? J Comp Assist Tomogr, 32, 570–575.Google Scholar
Gossmann, A., Sahiner, B., Pezeshk, A. (2018). Test data reuse for evaluation of adaptive machine learning algorithms: over-fitting to a fixed “test” dataset and a potential solution. Proc SPIE Med Imag, 10577Google Scholar
Greenspan, H., van Ginneken, B., Summers, R.M. (2016). Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imag, 35, 1153–1159.Google Scholar
Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. Am J Roentgenol, 190, 854–859.Google Scholar
Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P.C., Mega, J.L., Webster, R. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316, 2402–2410.Google Scholar
Gur, D., Rockette, H.E., Armfield, D.R., Blachar, A., Bogan, J.K., Brancatell, G., Britton, C.A., Brown, M.L., Davis, P.L., Ferris, J.V., Fuhrman, C.R., Golla, S.K., Katyal, S., Lacomis, J.M., McCook, B.M., Thaete, F.L., Warfel, T.E. (2003). Prevalence effect in a laboratory environment. Radiology, 228, 10–14.Google Scholar
Gur, D., Sumkin, J.H., Rockette, H.E., Ganott, M.A., Hakim, C., Hardesty, L.A., Poller, W.R., Shah, R., Wallace, L. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.Google Scholar
Gur, D., Bandos, A.I., Fuhrman, C.R., Klym, A.H., King, J.L., Rockette, H.E. (2007). The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol, 14, 49–53.Google Scholar
Hadjiiski, L., Sahiner, B., Helvie, M.A., Chan, H.-P., Roubidoux, M.A., Paramagul, C., Blane, C., Petrick, N., Bailey, J., Klein, K. (2006). Breast masses: computer-aided diagnosis with serial mammograms. Radiology, 240, 343–356.Google Scholar
Hand, D.J., Till, R.J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learn, 45, 171–186.Google Scholar
Hanley, J.A. (1988). The robustness of the “binormal assumptions” used in fitting (ROC) curves. Med Decis Making, 8, 197–203.Google Scholar
Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K., Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K. (2008). Evaluation of computer-aided diagnosis (CAD) software for the detection of lung nodules on multidetector row computed tomography (MDCT): JAFROC study for the improvement in radiologists’ diagnostic accuracy. Acad Radiol, 15, 1505–1512.Google Scholar
Horsch, K., Giger, M.L., Metz, C.E. (2008). Potential effect of different radiologist reporting methods on studies showing benefit of CAD. Acad Radiol, 15, 139–152.Google Scholar
Huo, Z.M., Giger, M.L., Vyborny, C.J., Wolverton, D.E., Schmidt, R.A., Doi, K. (1998). Automated computerized classification of malignant and benign masses on digitized mammograms. Acad Radiol, 5, 155–168.Google Scholar
Hupse, R., Samulski, M., Lobbes, M., den Heeten, A., Imhof-Tas, M.W., Beijerinck, D., Pijnappel, R., Boetes, C., Karssemeijer, N. (2013). Standalone computer-aided detection compared to radiologists’ performance for the detection of mammographic masses. Eur Radiol, 23, 93–100.Google Scholar
ICRU. (2008). Receiver Operating Characteristic Analysis in Medical Imaging. Bethesda, MD: International Commission of Radiation Units and Measurements.Google Scholar
IMDRF SaMD Working Group, Software as a Medical device (SaMD). (2017) Clinical evaluation. Available at: www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-clinical-evaluation.pdf (accessed November 21, 2017).Google Scholar
Jesneck, J.L., Lo, J.Y., Baker, J.A. (2007). Breast mass lesions: computer-aided diagnosis models with mammographic and sonographic descriptors. Radiology, 244, 390–398.Google Scholar
Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750.Google Scholar
Jiang, Y.L., Metz, C.E., Nishikawa, R.M., Schmidt, R.A. (2006). Comparison of independent double readings and computer-aided diagnosis (CAD) for the diagnosis of breast calcifications. Acad Radiol, 13, 84–94.Google Scholar
Kantarjian, H., Yu, P. (2015). Artificial intelligence, big data, and cancer. JAMA Oncol, 1, 573–574.Google Scholar
Karssemeijer, N., Otten, J.D.M., Verbeek, A.L.M., Groenewoud, J.H., de Koning, H.J., Hendriks, J.H.C.L., Holland, R. (2003). Computer-aided detection versus independent double reading of masses on mammograms. Radiology, 227, 192–200.Google Scholar
Kasai, S., Li, F., Shiraishi, J., Doi, K. (2008). Usefulness of computer-aided diagnosis schemes for vertebral fractures and lung nodules on chest radiographs. Am J Roentgenol, 191, 260–265.Google Scholar
Khoo, L.A.L., Taylor, P., Given-Wilson, R.M. (2005). Computer-aided detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.Google Scholar
Kligerman, S., Cai, L., White, C.S. (2013). The effect of computer-aided detection on radiologist performance in the detection of lung cancers previously missed on a chest radiograph. J Thorac Imag, 28, 244–252.Google Scholar
Kooi, T., Litjens, G., van Ginneken, B., Gubern-Merida, A., Sancheza, C.I., Mann, R., den Heeten, A., Karssemeijer, N. (2017). Large scale deep learning for computer aided detection of mammographic lesions. Med Image Analysis, 35, 303–312.Google Scholar
Kosinski, A.S., Barnhart, H.X. (2003). A global sensitivity analysis of performance of a medical diagnostic test when verification bias is present. Stat Med, 22, 2711–2721.Google Scholar
Kotsiantis, S.B., Zaharakis, I.D., Pintelas, P.E. (2006). Machine learning: a review of classification and combining techniques. Artif Intell Rev, 26, 159–190.Google Scholar
Krizhevsky, A., Sutskever, I., Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. In: Pereira, F.C., Burges, J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, Vol. 25. Red Hook, NY: Curran Associates, pp. 1097–1105.Google Scholar
Kumar, V., Gu, Y., Basu, S., Berglund, A., Eschrich, S.A., Schabath, M.B., Forster, K., Aerts, H.J.W.L., Dekker, A., Fenstermacher, D., Goldgof, D.B., Hall, L.O., Lambin, P., Balagurunathan, Y., Gatenby, R.A., Gillies, R.J. (2012). Radiomics: the process and the challenges. Magn Reson Imag, 30, 1234–1248.Google Scholar
Lambin, P., Rios-Velazquez, E., Leijenaar, R., Carvalho, S., van Stiphout, R.G.P.M., Granton, P., Zegers, C.M.L., Gillies, R., Boellard, R., Dekker, A., Aerts, H.J.W.L. (2012). Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer, 48, 441–446.Google Scholar
LeCun, Y., Bengio, Y., Hinton, G. (2015). Deep learning. Nature, 521, 436–444.Google Scholar
Li, F., Arimura, H., Suzuki, K., Shiraishi, J., Li, Q., Abe, H., Engelmann, R., Sone, S., MacMahon, H., Doi, K. (2005). Computer-aided detection of peripheral lung cancers missed at CT: ROC analyses without and with localization. Radiology, 237, 684–690.Google Scholar
Li, Q., Gavrielides, M.A., Sahiner, B., Myers, K.J., Zeng, R., Petrick, N. (2015). Statistical analysis of lung nodule volume measurements with CT in a large-scale phantom study. Med Phys, 42, 3932–3947.Google Scholar
Metz, C. E., Pan, X. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psych, 43, 1–33.Google Scholar
Metz, C.E., Wang, P.L., Kronman, H.B. (1984). A new approach for testing the significance for differences between ROC curves measured from correlated data. In: Deconinck, F. (ed.) Information Processing in Medical Imaging. The Hague, The Netherlands: Martinus Nijhoff, pp. 432–445.Google Scholar
Miller, D.P., O’Shaughnessy, K.F., Wood, S.A., Castellino, R.A. (2004). Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions. Proc SPIE Med Imag, 5372, 173–184.Google Scholar
Moin, P., Deshpande, R., Sayre, J., Messer, E., Gupte, S., Romsdahl, H., Hasegawa, A., Liu, B.J. (2011). An observer study for a computer-aided reading protocol (CARP) in the screening environment for digital mammography. Acad Radiol, 18, 1420–1429.Google Scholar
Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar
Mossman, D. (1999). Three-way ROCs. Med Decis Making, 19, 78–89.Google Scholar
Naaktgeboren, C.A., de Groot, J.A.H., Rutjes, A.W.S., Bossuyt, P.M.M., Reitsma, J.B., Moons, K.G.M. (2016). Anticipating missing reference standard data when planning diagnostic accuracy studies. Br Med J, 352.Google Scholar
Nakas, C.T., Yiannoutsos, C.T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Stat Med, 23, 3437–3449.Google Scholar
Neuhaus, J.M., Kalbfleisch, J.D. (1998). Between- and within-cluster covariate effects in the analysis of clustered data. Biometrics, 54, 638–645.Google Scholar
Nietert, P.J., Ravenel, J.G., Taylor, K.K., Silvestri, G.A. (2011). Influence of nodule detection software on radiologists’ confidence in identifying pulmonary nodules with computed tomography. J Thorac Imag, 26, 48–53.Google Scholar
Nishikawa, R.M., Pesce, L.L. (2009). Computer-aided detection evaluation methods are not created equal. Radiology, 251, 634–636.Google Scholar
Obuchowski, N.A. (2005). ROC analysis [see comment]. Am J Roentgenol, 184, 364–72.Google Scholar
Obuchowski, N.A., Gallas, B.D., Hillis, S.L. (2012). Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol, 19, 1508–1517.Google Scholar
O’Keefe, R.M., O’Leary, D.E. (1993). Expert system verification and validation: a survey and tutorial. Artif Intell Rev, 7, 3–42.Google Scholar
Petrick, N., Haider, M., Summers, R.M., Yeshwant, S.C., Brown, L., Iuliano, E.M., Louie, A., Choi, J.R., Pickhardt, P.J. (2008). CT colonography with computer-aided detection as a second reader: observer performance study. Radiology, 246, 148–156.Google Scholar
Petrick, N., Sahiner, B., Armato, S.G., III, Bert, A., Correale, L., Delsanto, S., Freedman, M.T., Fryd, D., Gur, D., Hadjiiski, L., Huo, Z., Jiang, Y., Morra, L., Paquerault, S., Raykar, V., Salganicoff, M., Samuelson, F., Summers, R.M., Tourassi, G., Yoshida, H., Zheng, B., Zhou, C., Chan, H.-P. (2013). Evaluation of computer-aided detection and diagnosis systems. Med Phys, 40, 087001-1, 087001-17.Google Scholar
Pezeshk, A., Sahiner, B., Zeng, R.P., Wunderlich, A., Chen, W.J., Petrick, N. (2015). Seamless insertion of pulmonary nodules in chest CT images. IEEE Trans Biomed Eng, 62, 2812–2827.Google Scholar
Pezeshk, A., Petrick, N., Chen, W.J., Sahiner, B. (2017). Seamless lesion insertion for data augmentation in CAD training. IEEE Trans Med Imag, 36, 1005–1015.Google Scholar
Pisano, E.D., Gatsonis, C., Hendrick, E., Yaffe, M. (2005). Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med, 353, 1773–1783.Google Scholar
Popescu, L.M. (2011). Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve. Med Phys, 38, 5690–5702.Google Scholar
Rao, J.N.K., Scott, A.J. (1992). A simple method for the analysis of clustered binary data. Biometrics, 48, 577–585.Google Scholar
Regge, D., Della Monica, P., Galatola, G., Laudi, C., Zambon, A., Correale, L., Asnaghi, R., Barbaro, B., Borghi, C., Campanella, D., Cassinis, M.C., Ferrari, R., Ferraris, A., Golfieri, R., Hassan, C., Iafrate, F., Iussich, G., Laghi, A., Massara, R., Neri, E., Sali, L., Venturini, S., Gandini, G. (2013). Efficacy of computer-aided detection as a second reader for 6–9-mm lesions at CT colonography: multicenter prospective trial. Radiology, 266, 168–176.Google Scholar
Sahiner, B., Chan, H.-P., Petrick, N., Wagner, R.F., Hadjiiski, L.M. (2000). Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys, 27, 1509–1522.Google Scholar
Sahiner, B., Chan, H.-P., Roubidoux, M.A., Helvie, M.A., Hadjiiski, L.M., Ramachandran, A., Paramagul, C., LeCarpentier, G.L., Nees, A., Blane, C. (2004). Computerized characterization of breast masses on three-dimensional ultrasound volumes. Med Phys, 31, 744–754.Google Scholar
Sahiner, B., Chan, H.P., Roubidoux, M.A., Hadjiiski, L.M., Helvie, M.A., Paramagul, C., Bailey, J., Nees, A., Blane, C. (2007). Computer-aided diagnosis of malignant and benign breast masses in 3D ultrasound volumes: effect on radiologists’ accuracy. Radiology, 242, 716–724.Google Scholar
Sahiner, B., Chan, H.P., Hadjiiski, L.M. (2008). Performance analysis of three-class classifiers: properties of a 3-D ROC surface and the normalized volume under the surface for the ideal observer. IEEE Trans Med Imag, 27, 215–227.Google Scholar
Sahiner, B., Chan, H.-P., Hadjiiski, L.M., Cascade, P.N., Kazerooni, E.A., Chughtai, A.R., Poopat, C., Song, T., Frank, L., Stojanovska, J. (2009). Effect of CAD on radiologists’ detection of lung nodules on thoracic CT scans: analysis of an observer performance study by nodule size. Acad Radiol, 16, 1518–1530.Google Scholar
Samuelson, F.W., Abbey, C.K. (2017). The reproducibility of changes in diagnostic figures of merit across laboratory and clinical imaging reader studies. Acad Radiol, 24, 1436–1446.Google Scholar
Samuelson, F.W., Petrick, N. (2006). Comparing image detection algorithms using resampling. 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 1312–1315.Google Scholar
Schalekamp, S., van Ginneken, B., Heggelman, B.G.F., Imhof-Tas, M., Somers, I., Brink, M., Spee, M., Schaefer-Prokop, C.M., Karssemeijer, N. (2014). New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs. Br J Radiol, 87, 1036.Google Scholar
Scurfield, B.K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. J Math Psych, 40, 253–269.Google Scholar
Shimauchi, A., Giger, M.L., Bhooshan, N., Lan, L., Pesce, L.L., Lee, J.K., Abe, H., Newstead, G.M. (2011). Evaluation of clinical breast MR imaging performed with prototype computer-aided diagnosis breast MR imaging workstation: reader study. Radiology, 258, 696–704.Google Scholar
Stages of Breast Cancer. (2017). Available from: www.breastcancer.org/symptoms/diagnosis/staging (accessed November 10, 2017).Google Scholar
Starr, S.J., Metz, C.E., Lusted, L.B., Goodenough, D.J. (1975). Visual detection and localization of radiographic images. Radiology, 116, 533–538.Google Scholar
Stepan-Buksakowska, I.L., Accurso, J.M., Diehn, F.E., Huston, J., Kaufmann, T.J., Luetmer, P.H., Wood, C.P., Yang, X., Blezek, D.J., Carter, R., Hagen, C., Horinek, D., Hejcl, A., Rocek, M., Erickson, B.J. (2014). Computer-aided diagnosis improves detection of small intracranial aneurysms on MRA in a clinical setting. Am J Neuroradiol, 35, 1897–1902.Google Scholar
Swensson, R.G. (1996). Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys, 23, 1709–1725.Google Scholar
Thrall, J.H. (2016). Trends and developments shaping the future of diagnostic medical imaging: 2015 annual oration in diagnostic radiology. Radiology, 279, 660–666.Google Scholar
Uchiyama, Y., Asano, T., Kato, H., Hara, T., Kanematsu, M., Hoshi, H., Iwama, T., Fujita, H. (2012). Computer-aided diagnosis for detection of lacunar infarcts on MR images: ROC analysis of radiologists’ performance. J Digit Imag, 25, 497–503.Google Scholar
University of Chicago. (2017). Metz ROC software. Available at: http://metz-roc.uchicago.edu/MetzROC (accessed November 10, 2017).Google Scholar
University of Iowa. (2017). ROC software. Available at: http://perception.radiology.uiowa.edu/ (accessed November 10, 2017).Google Scholar
van Beek, E.J.R., Mullan, B., Thompson, B. (2008). Evaluation of a real-time interactive pulmonary nodule analysis system on chest digital radiographic images: a prospective study. Acad Radiol, 15, 571–575.Google Scholar
Wagner, R.F., Beam, C.A., Beiden, S.V. (2004). Reader variability in mammography and its implications for expected utility over the population of readers and cases. Med Decis Making, 24, 561–572.Google Scholar
Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar
Way, T., Chan, H.P., Hadjiiski, L., Sahiner, B., Chughtai, A., Song, T.K., Poopat, C., Stojanovska, J., Frank, L., Attili, A., Bogot, N., Cascade, P.N., Kazerooni, E.A. (2010). Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists’ performance. Acad Radiol, 17, 323–332.Google Scholar
Wei, J., Chan, H.P., Sahiner, B., Zhou, C., Hadjiiski, L.M., Roubidoux, M.A., Helvie, M.A. (2009). Computer-aided detection of breast masses on mammograms: dual system approach with two-view analysis. Med Phys, 36, 4451–4460.Google Scholar
Xin, H., Metz, C.E., Tsui, B.M.W., Links, J.M., Frey, E.C. (2006). Three-class ROC analysis – a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imag, 25, 571–581.Google Scholar
Yanagawa, M., Honda, O., Yoshida, S., Ono, Y., Inoue, A., Daimon, T., Sumikawa, H., Mihara, N., Johkoh, T., Tomiyama, N., Nakamura, H. (2009). Commercially available computer-aided detection system for pulmonary nodules on thin-section images using 64 detectors-row CT: preliminary study of 48 cases. Acad Radiol, 16, 924–933.Google Scholar
Yoon, H.J., Zheng, B., Sahiner, B., Chakraborty, D.P. (2007). Evaluating computer-aided detection algorithms. Med Phys, 34, 2024–2038.Google Scholar
Zeiler, M.D., Fergus, R. (2014). Visualizing and understanding convolutional networks. In: Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I. Cham: Springer International Publishing, pp. 818–833.Google Scholar
Zweig, M.H., Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem, 39, 561–577.Google Scholar

References

Abramson, R.G., Su, P.F., Shyr, Y. (2012). Quantitative metrics in clinical radiology reporting: a snapshot perspective from a single mixed academic-community practice. Magn Reson Imag, 30(9), 1357–1366.Google Scholar
Birkelo, C.C., Chamberlain, W.E., Phelps, P.S., et al. (1947). Tuberculosis case finding: a comparison of the effectiveness of various roentgenographic and photofluorographic methods. JAMA, 133, 359–366.Google Scholar
Jaffe, T.A., Wickersham, N.W., Sullivan, D.C. (2010). Quantitative imaging in oncology patients: Part 2, oncologists’ opinions and expectations at major U.S. cancer centers. AJR Am J Roentgenol, 195(1), W19-W30.Google Scholar
Joint Committee for Guides in Metrology Working Group 1. (2008). Evaluation of measurement data – Supplement 1 to the Guide to the expression of uncertainty in measurement. Available at: www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf (accessed December 21, 2017).Google Scholar
Kessler, L.G., Barnhart, H.X., Buckler, A.J., et al. (2015). The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions. Stat Methods Med Res, 24(1), 9–26.Google Scholar
Macari, M., Megibow, A.J. (2011). Focal cystic pancreatic lesions: variability in radiologists’ recommendations for follow-up imaging. Radiology, 259(1), 20–23.Google Scholar
Quantitative Imaging Biomarkers Alliance. Available online at: http://rsna.org/qiba (accessed December 21, 2017).Google Scholar
Raunig, D.L., McShane, L.M., Pennello, G., et al. (2015). Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment. Stat Methods Med Res, 24(1), 27–67.Google Scholar
Sullivan, D.C., Schwartz, L.H., Zhao, B. (2013). The imaging viewpoint: how imaging affects determination of progression-free survival. Clin Cancer Res, 19(10), 2621–2628.Google Scholar
Sullivan, D.C., Obuchowski, N.A., Kessler, L.G., Raunig, D.L., Gatsonis, C., Huang, E.P. et al. (2015). RSNA-QIBA metrology working group. Metrology standards for quantitative imaging biomarkers. Radiology, 277(3), 813–825.Google Scholar
Zhao, B., Tan, Y., Bell, D.J., Marley, S.E., Guo, P., Mann, H. et al. (2013). Exploring intra- and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals. Eur J Radiol, 82(6), 959–968.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×