Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-jkksz Total loading time: 0 Render date: 2024-12-27T18:12:27.090Z Has data issue: false hasContentIssue false

26 - Evaluation of CAD and Radiomic Tools

from Part V - Computational Perception

Published online by Cambridge University Press:  20 December 2018

By
Berkman Sahiner  and
Nicholas Petrick
Edited by
Ehsan Samei  and
Elizabeth A. Krupinski
Ehsan Samei
Affiliation:
Duke University Medical Center, Durham
Elizabeth A. Krupinski
Affiliation:
Emory University, Atlanta
Get access

Summary

A summary is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.
Image of the first page of this content. For PDF version, please use the ‘Save PDF’ preceeding this image.'
Type
Chapter
Information
The Handbook of Medical Image Perception and Techniques , pp. 389 - 406
Publisher: Cambridge University Press
Print publication year: 2018

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abbey, C.K., Wu, Y., Burnside, E.S., Wunderlich, A., Samuelson, F.W., Boone, J.M. (2016). A utility/cost analysis of breast cancer risk prediction algorithms. Proc SPIE Med Imag, 9887, 97871J.Google Scholar
Adrion, W.R., Branstad, M.A., Cherniavsky, J.C. (1982). Validation, verification, and testing of computer software. ACM Comp Surv, 14, 159–192.Google Scholar
Aerts, H.J.W.L. (2016). The potential of radiomic-based phenotyping in precision medicine: a review. JAMA Oncol, 2, 1636–1642.Google Scholar
Agresti, A., Coull, B.A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. Am Statist, 52, 119–126.Google Scholar
Altman, D.G., Bland, J.M. (1999). Statistics notes – treatment allocation in controlled trials: why randomise? Br Med J, 318, 1209–1209.Google Scholar
Aoki, T., Oda, N., Yamashita, Y., Yamamoto, K., Korogi, Y. (2011). Usefulness of computerized method for lung nodule detection in digital chest radiographs using temporal subtraction images. Acad Radiol, 18, 1000–1005.CrossRefGoogle ScholarPubMed
Armato, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P. et al. (2011). The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys, 38, 915–931.CrossRefGoogle ScholarPubMed
Bandos, A.I., Rockette, H.E., Song, T., Gur, D. (2009). Area under the free-response ROC curve (FROC) and a related summary index. Biometrics, 65, 247–256.Google Scholar
Belle, A., Thiagarajan, R., Soroushmehr, S.M.R., Navidi, F., Beard, D.A., Najarian, K. (2015). Big data analytics in healthcare. BioMed Res Intl, 2015, 1–16.Google Scholar
Bergstra, J., Yamins, D., Cox, D. (2013). Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, 28, 115–123.Google Scholar
Bogoni, L., Ko, J.P., Alpert, J., Anand, V., Fantauzzi, J., Florin, C.H., Koo, C.W., Mason, D., Rom, W., Shiau, M., Salganicoff, M., Naidich, D.P. (2012). Impact of a computer-aided detection (CAD) system integrated into a picture archiving and communication system (PACS) on reader sensitivity and efficiency for the detection of lung nodules in thoracic CT exams. J Digit Imag, 25, 771–781.Google Scholar
Boone, D., Mallett, S., McQuillan, J., Taylor, S.A., Altman, D.G., Halligan, S. (2015). Assessment of the incremental benefit of computer-aided detection (CAD) for interpretation of CT colonography by experienced and inexperienced readers. PLos One, 10.Google Scholar
Bornefalk, H. (2005). Estimation and comparison of CAD system performance in clinical settings. Acad Radiol, 12, 687–94.CrossRefGoogle ScholarPubMed
Brown, L.D., Cai, T.T., DasGupta, A. (2001a). Interval estimation for a binomial proportion. Stat Sci, 16, 101–117.Google Scholar
Brown, L.D., Cai, T.T., DasGupta, A. (2001b). Interval estimation for a binomial proportion-comment-rejoiner. Stat Sci, 16, 117–133.Google Scholar
Brown, M.S., Goldin, J.G., Rogers, S., Kim, H.J., Suh, R.D., McNitt-Gray, M.F., Shah, S.K., Truong, D., Brown, K., Sayre, J.W., Gjertson, D.W., Batra, P., Aberle, D.R. (2005). Computer-aided lung nodule detection in CT: results of large-scale observer test. Acad Radiol, 12, 681–686.Google ScholarPubMed
Bunch, P.C., Hamilton, J.F., Sanderson, G.K., Simmons, A.H. (1978). A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng, 4, 166–171.Google Scholar
Byron, S.A., Van Keuren-Jensen, K.R., Engelthaler, D.M., Carpten, J.D., Craig, D.W. (2016). Translating RNA sequencing into clinical diagnostics: opportunities and challenges. Nat Rev Genet, 17, 257–271.Google Scholar
Chakraborty, D.P. (1989). Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 16, 561–568.Google Scholar
Chakraborty, D.P. (2006a). Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol, 13, 1187–1193.Google Scholar
Chakraborty, D.P. (2006b). A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol, 51, 3449–3462.CrossRefGoogle Scholar
Chakraborty, D.P. (2008). Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol, 15, 1554–1566.CrossRefGoogle ScholarPubMed
Chakraborty, D.P. (2017) FROC methodology website. Available at: www.devchakraborty.com/index.php (accessed November 10, 2017).Google Scholar
Chakraborty, D.P., Berbaum, K.S. (2004). Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys, 31, 2313–2330.Google Scholar
Chakraborty, D.P., Winter, L.H.L. (1990). Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology, 174, 873–881.CrossRefGoogle Scholar
Chan, H.P., Wei, D., Helvie, M.A., Sahiner, B., Adler, D.D., Goodsitt, M.M., Petrick, N. (1995). Computer-aided classification of mammographic masses and normal tissue: linear discriminant analysis in texture feature space. Phys Med Biol, 40, 857.Google Scholar
Chan, H.P., Wei, J., Zhang, Y.H., Helvie, M.A., Moore, R.H., Sahiner, B., Hadjiiski, L., Kopans, D.B. (2008). Computer-aided detection of masses in digital tomosynthesis mammography: comparison of three approaches. Med Phys, 35, 4087–4095.Google Scholar
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P. (2002). SMOTE: synthetic minority over-sampling technique. J Artif Intellig Res, 16, 321–357.Google Scholar
Choudhury, K.R., Paik, D.S., Yi, C.A., Napel, S., Roos, J., Rubin, G.D. (2010). Assessing operating characteristics of CAD algorithms in the absence of a gold standard. Med Phys, 37, 1788–1795.Google Scholar
Dachman, A.H., Obuchowski, N.A., Hoffmeister, J.W., Hinshaw, J.L., Frew, M.I., Winter, T.C., Van Uitert, R.L., Periaswamy, S., Summers, R.M., Hillman, B.J. (2010). Effect of computer-aided detection for CT colonography in a multireader, multicase trial. Radiology, 256, 827–835.Google Scholar
De Boo, D.W., Uffmann, M., Weber, M., Bipat, S., Boorsma, E.F., Scheerder, M.J., Freling, N.J., Schaefer-Prokop, C.M. (2011). Computer-aided detection of small pulmonary nodules in chest radiographs: an observer study. Acad Radiol, 18, 1507–1514.Google Scholar
de Groot, J.A.H., Janssen, K.J.M., Zwinderman, A.H., Moons, K.G.M., Reitsma, J.B. (2008). Multiple imputation to correct for partial verification bias revisited. Stat Med, 27, 5880–5889.CrossRefGoogle ScholarPubMed
de Hoop, B., De Boo, D.W., Gietema, H.A., van Hoorn, F., Mearadji, B., Schijf, L., van Ginneken, B., Prokop, M., Schaefer-Prokop, C. (2010). Computer-aided detection of lung cancer on chest radiographs: effect on observer performance. Radiology, 257, 532–540.CrossRefGoogle ScholarPubMed
Dorfman, D.D., Alf, E. (1969). Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals – rating-method data. J Math Psych, 6, 487–496.Google Scholar
Dorfman, D.D., Berbaum, K.S., Metz, C.E., Lenth, R.V., Hanley, J.A., Dagga, H.A. (1997). Proper receiver operating characteristic analysis: the bigamma model. Acad Radiol, 4, 138–149.Google Scholar
Dwork, C. (2011). The promise of differential privacy a tutorial on algorithmic techniques. In: Ostrovsky, R. (ed.) 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, pp. 1–2.Google Scholar
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., Roth, A. (2015). The reusable holdout: preserving validity in adaptive data analysis. Science, 349, 636–638.Google Scholar
Edwards, D.C., Kupinski, M.A., Metz, C.E., Nishikawa, R.M. (2002). Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys, 29, 2861–2870.Google Scholar
Edwards, D.C., Metz, C.E., Kupinski, M.A. (2004). Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans Med Imag, 23, 891–895.CrossRefGoogle ScholarPubMed
Efron, B. (1983). Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Statis Assn, 78, 316–331.Google Scholar
Efron, B., Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Stat Sci, 1, 54–75.Google Scholar
Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115.Google Scholar
Evans, K.K., Birdwell, R.L., Wolfe, J.M. (2013). If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screening. PLos One, 8.Google Scholar
Evans, B.J., Burke, W., Jarvik, G.P. (2015). The FDA and genomic tests – getting regulation right. N Engl J Med, 372, 2258–2264.Google Scholar
FDA (2012a). Guidance for industry and FDA staff: clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device data – premarket approval (PMA) and premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187315.pdf (accessed November 21, 2017).Google Scholar
FDA (2012b). Guidance for industry and FDA staff: computer-assisted detection devices applied to radiology images and radiology device data – premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187294.pdf (accessed November 21, 2017).Google Scholar
FDA (2013). Design considerations for pivotal clinical investigations for medical devices – guidance for industry, clinical investigators, institutional review boards and Food and Drug Administration staff. Available at: www.fda.gov/RegulatoryInformation/Guidances/ucm373750.htm (accessed November 21, 2017).Google Scholar
FDA (2017a). QuantX DEN170022 reclassification order. Available at: www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?ID=DEN170022 (accessed November 21, 2017).Google Scholar
FDA (2017b). iMRMC software. Available at: https://github.com/DIDSR/iMRMC (accessed November 10, 2017).Google Scholar
Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Trans Patt Anal Mach Intell, 32, 1627–1645.Google Scholar
Fenton, J.J., Abraham, L., Taplin, S.H., Geller, B.M., Carney, P.A., D’Orsi, C. et al. (2011). Effectiveness of computer-aided detection in community mammography practice. J Natl Cancer Inst, 103, 1152–1161.Google Scholar
Fisichella, V.A., Jaderling, F., Horvath, S., Stotzer, P.O., Kilander, A., Bath, M., Hellstrom, M. (2009). Computer-aided detection (CAD) as a second reader using perspective filet view at CT colonography: effect on performance of inexperienced readers. Clin Radiol, 64, 972–982.CrossRefGoogle ScholarPubMed
Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology, 220, 781–786.CrossRefGoogle Scholar
Gallas, B.D., Brown, D.G. (2008). Reader studies for validation of CAD systems. Neural Networks, 21, 387–397.Google Scholar
Gallas, B.D., Chan, H.P., D’Orsi, C.J., Dodd, L.E., Giger, M.L., Gur, D., Krupinski, E.A., Metz, C.E., Myers, K. J., Obuchowski, N.A., Sahiner, B., Toledano, A.Y., Zuley, M.L. (2012). Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 19, 463–477.Google Scholar
Gallas, B.D., Pisano, E., Cole, E., Myers, K. (2017). Impact of different study populations on reader behavior and performance metrics: initial results. Proc SPIE Med Imag, 10136.Google Scholar
Gehan, E.A., Freireich, E.J. (1974). Non-randomized controls in cancer clinical trials. N Engl J Med, 290, 198–203.Google Scholar
Gilbert, F.J., Astley, S.M., Gillan, M.G.C., Agbaje, O.F., Wallis, M.G., James, J., Boggis, C.R.M., Duffy, S.W. (2008). Single reading with computer-aided detection for screening mammography. N Engl J Med, 359, 1675–1684.Google Scholar
Godoy, M.C.B., Kim, T.J., White, C.S., Bogoni, L., de Groot, P., Florin, C., Obuchowski, N., Babb, J.S., Salganicoff, M., Naidich, D.P., Anand, V., Park, S., Vlahos, I., Ko, J.P. (2013). Benefit of computer-aided detection analysis for the detection of subsolid and solid lung nodules on thin- and thick-section CT. Am J Roentgenol, 200, 74–83.Google Scholar
Gomez, S.S., Tabanera, M.T., Bolivar, A.V., Miranda, M.S., Mazo, A.B., Diaz, M.R., Miravete, P.M., Asturiano, E.L., Cacho, P.M., Macias, T.D. (2011). Impact of a CAD system in a screen-film mammography screening program: a prospective study. Eur J Radiol, 80, E317–E321.Google Scholar
Goo, J.M., Kim, H.Y., Lee, J. W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T., Goo, J.M., Kim, H.Y., Lee, J.W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T. (2008). Is the computer-aided detection scheme for lung nodule also useful in detecting lung cancer? J Comp Assist Tomogr, 32, 570–575.Google Scholar
Gossmann, A., Sahiner, B., Pezeshk, A. (2018). Test data reuse for evaluation of adaptive machine learning algorithms: over-fitting to a fixed “test” dataset and a potential solution. Proc SPIE Med Imag, 10577Google Scholar
Greenspan, H., van Ginneken, B., Summers, R.M. (2016). Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imag, 35, 1153–1159.Google Scholar
Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. Am J Roentgenol, 190, 854–859.Google Scholar
Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P.C., Mega, J.L., Webster, R. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316, 2402–2410.Google Scholar
Gur, D., Rockette, H.E., Armfield, D.R., Blachar, A., Bogan, J.K., Brancatell, G., Britton, C.A., Brown, M.L., Davis, P.L., Ferris, J.V., Fuhrman, C.R., Golla, S.K., Katyal, S., Lacomis, J.M., McCook, B.M., Thaete, F.L., Warfel, T.E. (2003). Prevalence effect in a laboratory environment. Radiology, 228, 10–14.Google Scholar
Gur, D., Sumkin, J.H., Rockette, H.E., Ganott, M.A., Hakim, C., Hardesty, L.A., Poller, W.R., Shah, R., Wallace, L. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.CrossRefGoogle ScholarPubMed
Gur, D., Bandos, A.I., Fuhrman, C.R., Klym, A.H., King, J.L., Rockette, H.E. (2007). The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol, 14, 49–53.Google Scholar
Hadjiiski, L., Sahiner, B., Helvie, M.A., Chan, H.-P., Roubidoux, M.A., Paramagul, C., Blane, C., Petrick, N., Bailey, J., Klein, K. (2006). Breast masses: computer-aided diagnosis with serial mammograms. Radiology, 240, 343–356.Google Scholar
Hand, D.J., Till, R.J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learn, 45, 171–186.Google Scholar
Hanley, J.A. (1988). The robustness of the “binormal assumptions” used in fitting (ROC) curves. Med Decis Making, 8, 197–203.Google Scholar
Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K., Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K. (2008). Evaluation of computer-aided diagnosis (CAD) software for the detection of lung nodules on multidetector row computed tomography (MDCT): JAFROC study for the improvement in radiologists’ diagnostic accuracy. Acad Radiol, 15, 1505–1512.Google Scholar
Horsch, K., Giger, M.L., Metz, C.E. (2008). Potential effect of different radiologist reporting methods on studies showing benefit of CAD. Acad Radiol, 15, 139–152.Google Scholar
Huo, Z.M., Giger, M.L., Vyborny, C.J., Wolverton, D.E., Schmidt, R.A., Doi, K. (1998). Automated computerized classification of malignant and benign masses on digitized mammograms. Acad Radiol, 5, 155–168.Google Scholar
Hupse, R., Samulski, M., Lobbes, M., den Heeten, A., Imhof-Tas, M.W., Beijerinck, D., Pijnappel, R., Boetes, C., Karssemeijer, N. (2013). Standalone computer-aided detection compared to radiologists’ performance for the detection of mammographic masses. Eur Radiol, 23, 93–100.CrossRefGoogle ScholarPubMed
ICRU. (2008). Receiver Operating Characteristic Analysis in Medical Imaging. Bethesda, MD: International Commission of Radiation Units and Measurements.Google Scholar
IMDRF SaMD Working Group, Software as a Medical device (SaMD). (2017) Clinical evaluation. Available at: www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-clinical-evaluation.pdf (accessed November 21, 2017).Google Scholar
Jesneck, J.L., Lo, J.Y., Baker, J.A. (2007). Breast mass lesions: computer-aided diagnosis models with mammographic and sonographic descriptors. Radiology, 244, 390–398.Google Scholar
Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750.Google Scholar
Jiang, Y.L., Metz, C.E., Nishikawa, R.M., Schmidt, R.A. (2006). Comparison of independent double readings and computer-aided diagnosis (CAD) for the diagnosis of breast calcifications. Acad Radiol, 13, 84–94.Google Scholar
Kantarjian, H., Yu, P. (2015). Artificial intelligence, big data, and cancer. JAMA Oncol, 1, 573–574.CrossRefGoogle ScholarPubMed
Karssemeijer, N., Otten, J.D.M., Verbeek, A.L.M., Groenewoud, J.H., de Koning, H.J., Hendriks, J.H.C.L., Holland, R. (2003). Computer-aided detection versus independent double reading of masses on mammograms. Radiology, 227, 192–200.Google Scholar
Kasai, S., Li, F., Shiraishi, J., Doi, K. (2008). Usefulness of computer-aided diagnosis schemes for vertebral fractures and lung nodules on chest radiographs. Am J Roentgenol, 191, 260–265.Google Scholar
Khoo, L.A.L., Taylor, P., Given-Wilson, R.M. (2005). Computer-aided detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.Google Scholar
Kligerman, S., Cai, L., White, C.S. (2013). The effect of computer-aided detection on radiologist performance in the detection of lung cancers previously missed on a chest radiograph. J Thorac Imag, 28, 244–252.Google Scholar
Kooi, T., Litjens, G., van Ginneken, B., Gubern-Merida, A., Sancheza, C.I., Mann, R., den Heeten, A., Karssemeijer, N. (2017). Large scale deep learning for computer aided detection of mammographic lesions. Med Image Analysis, 35, 303–312.Google Scholar
Kosinski, A.S., Barnhart, H.X. (2003). A global sensitivity analysis of performance of a medical diagnostic test when verification bias is present. Stat Med, 22, 2711–2721.Google Scholar
Kotsiantis, S.B., Zaharakis, I.D., Pintelas, P.E. (2006). Machine learning: a review of classification and combining techniques. Artif Intell Rev, 26, 159–190.Google Scholar
Krizhevsky, A., Sutskever, I., Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. In: Pereira, F.C., Burges, J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, Vol. 25. Red Hook, NY: Curran Associates, pp. 1097–1105.Google Scholar
Kumar, V., Gu, Y., Basu, S., Berglund, A., Eschrich, S.A., Schabath, M.B., Forster, K., Aerts, H.J.W.L., Dekker, A., Fenstermacher, D., Goldgof, D.B., Hall, L.O., Lambin, P., Balagurunathan, Y., Gatenby, R.A., Gillies, R.J. (2012). Radiomics: the process and the challenges. Magn Reson Imag, 30, 1234–1248.CrossRefGoogle ScholarPubMed
Lambin, P., Rios-Velazquez, E., Leijenaar, R., Carvalho, S., van Stiphout, R.G.P.M., Granton, P., Zegers, C.M.L., Gillies, R., Boellard, R., Dekker, A., Aerts, H.J.W.L. (2012). Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer, 48, 441–446.Google Scholar
LeCun, Y., Bengio, Y., Hinton, G. (2015). Deep learning. Nature, 521, 436–444.Google Scholar
Li, F., Arimura, H., Suzuki, K., Shiraishi, J., Li, Q., Abe, H., Engelmann, R., Sone, S., MacMahon, H., Doi, K. (2005). Computer-aided detection of peripheral lung cancers missed at CT: ROC analyses without and with localization. Radiology, 237, 684–690.CrossRefGoogle ScholarPubMed
Li, Q., Gavrielides, M.A., Sahiner, B., Myers, K.J., Zeng, R., Petrick, N. (2015). Statistical analysis of lung nodule volume measurements with CT in a large-scale phantom study. Med Phys, 42, 3932–3947.Google Scholar
Metz, C. E., Pan, X. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psych, 43, 1–33.Google Scholar
Metz, C.E., Wang, P.L., Kronman, H.B. (1984). A new approach for testing the significance for differences between ROC curves measured from correlated data. In: Deconinck, F. (ed.) Information Processing in Medical Imaging. The Hague, The Netherlands: Martinus Nijhoff, pp. 432–445.Google Scholar
Miller, D.P., O’Shaughnessy, K.F., Wood, S.A., Castellino, R.A. (2004). Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions. Proc SPIE Med Imag, 5372, 173–184.Google Scholar
Moin, P., Deshpande, R., Sayre, J., Messer, E., Gupte, S., Romsdahl, H., Hasegawa, A., Liu, B.J. (2011). An observer study for a computer-aided reading protocol (CARP) in the screening environment for digital mammography. Acad Radiol, 18, 1420–1429.Google Scholar
Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar
Mossman, D. (1999). Three-way ROCs. Med Decis Making, 19, 78–89.Google Scholar
Naaktgeboren, C.A., de Groot, J.A.H., Rutjes, A.W.S., Bossuyt, P.M.M., Reitsma, J.B., Moons, K.G.M. (2016). Anticipating missing reference standard data when planning diagnostic accuracy studies. Br Med J, 352.Google Scholar
Nakas, C.T., Yiannoutsos, C.T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Stat Med, 23, 3437–3449.Google Scholar
Neuhaus, J.M., Kalbfleisch, J.D. (1998). Between- and within-cluster covariate effects in the analysis of clustered data. Biometrics, 54, 638–645.Google Scholar
Nietert, P.J., Ravenel, J.G., Taylor, K.K., Silvestri, G.A. (2011). Influence of nodule detection software on radiologists’ confidence in identifying pulmonary nodules with computed tomography. J Thorac Imag, 26, 48–53.Google Scholar
Nishikawa, R.M., Pesce, L.L. (2009). Computer-aided detection evaluation methods are not created equal. Radiology, 251, 634–636.Google Scholar
Obuchowski, N.A. (2005). ROC analysis [see comment]. Am J Roentgenol, 184, 364–72.Google Scholar
Obuchowski, N.A., Gallas, B.D., Hillis, S.L. (2012). Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol, 19, 1508–1517.Google Scholar
O’Keefe, R.M., O’Leary, D.E. (1993). Expert system verification and validation: a survey and tutorial. Artif Intell Rev, 7, 3–42.Google Scholar
Petrick, N., Haider, M., Summers, R.M., Yeshwant, S.C., Brown, L., Iuliano, E.M., Louie, A., Choi, J.R., Pickhardt, P.J. (2008). CT colonography with computer-aided detection as a second reader: observer performance study. Radiology, 246, 148–156.Google Scholar
Petrick, N., Sahiner, B., Armato, S.G., III, Bert, A., Correale, L., Delsanto, S., Freedman, M.T., Fryd, D., Gur, D., Hadjiiski, L., Huo, Z., Jiang, Y., Morra, L., Paquerault, S., Raykar, V., Salganicoff, M., Samuelson, F., Summers, R.M., Tourassi, G., Yoshida, H., Zheng, B., Zhou, C., Chan, H.-P. (2013). Evaluation of computer-aided detection and diagnosis systems. Med Phys, 40, 087001-1, 087001-17.Google Scholar
Pezeshk, A., Sahiner, B., Zeng, R.P., Wunderlich, A., Chen, W.J., Petrick, N. (2015). Seamless insertion of pulmonary nodules in chest CT images. IEEE Trans Biomed Eng, 62, 2812–2827.Google Scholar
Pezeshk, A., Petrick, N., Chen, W.J., Sahiner, B. (2017). Seamless lesion insertion for data augmentation in CAD training. IEEE Trans Med Imag, 36, 1005–1015.Google Scholar
Pisano, E.D., Gatsonis, C., Hendrick, E., Yaffe, M. (2005). Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med, 353, 1773–1783.Google Scholar
Popescu, L.M. (2011). Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve. Med Phys, 38, 5690–5702.Google Scholar
Rao, J.N.K., Scott, A.J. (1992). A simple method for the analysis of clustered binary data. Biometrics, 48, 577–585.Google Scholar
Regge, D., Della Monica, P., Galatola, G., Laudi, C., Zambon, A., Correale, L., Asnaghi, R., Barbaro, B., Borghi, C., Campanella, D., Cassinis, M.C., Ferrari, R., Ferraris, A., Golfieri, R., Hassan, C., Iafrate, F., Iussich, G., Laghi, A., Massara, R., Neri, E., Sali, L., Venturini, S., Gandini, G. (2013). Efficacy of computer-aided detection as a second reader for 6–9-mm lesions at CT colonography: multicenter prospective trial. Radiology, 266, 168–176.Google Scholar
Sahiner, B., Chan, H.-P., Petrick, N., Wagner, R.F., Hadjiiski, L.M. (2000). Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys, 27, 1509–1522.Google Scholar
Sahiner, B., Chan, H.-P., Roubidoux, M.A., Helvie, M.A., Hadjiiski, L.M., Ramachandran, A., Paramagul, C., LeCarpentier, G.L., Nees, A., Blane, C. (2004). Computerized characterization of breast masses on three-dimensional ultrasound volumes. Med Phys, 31, 744–754.Google Scholar
Sahiner, B., Chan, H.P., Roubidoux, M.A., Hadjiiski, L.M., Helvie, M.A., Paramagul, C., Bailey, J., Nees, A., Blane, C. (2007). Computer-aided diagnosis of malignant and benign breast masses in 3D ultrasound volumes: effect on radiologists’ accuracy. Radiology, 242, 716–724.Google Scholar
Sahiner, B., Chan, H.P., Hadjiiski, L.M. (2008). Performance analysis of three-class classifiers: properties of a 3-D ROC surface and the normalized volume under the surface for the ideal observer. IEEE Trans Med Imag, 27, 215–227.Google Scholar
Sahiner, B., Chan, H.-P., Hadjiiski, L.M., Cascade, P.N., Kazerooni, E.A., Chughtai, A.R., Poopat, C., Song, T., Frank, L., Stojanovska, J. (2009). Effect of CAD on radiologists’ detection of lung nodules on thoracic CT scans: analysis of an observer performance study by nodule size. Acad Radiol, 16, 1518–1530.Google Scholar
Samuelson, F.W., Abbey, C.K. (2017). The reproducibility of changes in diagnostic figures of merit across laboratory and clinical imaging reader studies. Acad Radiol, 24, 1436–1446.Google Scholar
Samuelson, F.W., Petrick, N. (2006). Comparing image detection algorithms using resampling. 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 1312–1315.Google Scholar
Schalekamp, S., van Ginneken, B., Heggelman, B.G.F., Imhof-Tas, M., Somers, I., Brink, M., Spee, M., Schaefer-Prokop, C.M., Karssemeijer, N. (2014). New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs. Br J Radiol, 87, 1036.Google Scholar
Scurfield, B.K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. J Math Psych, 40, 253–269.Google Scholar
Shimauchi, A., Giger, M.L., Bhooshan, N., Lan, L., Pesce, L.L., Lee, J.K., Abe, H., Newstead, G.M. (2011). Evaluation of clinical breast MR imaging performed with prototype computer-aided diagnosis breast MR imaging workstation: reader study. Radiology, 258, 696–704.Google Scholar
Stages of Breast Cancer. (2017). Available from: www.breastcancer.org/symptoms/diagnosis/staging (accessed November 10, 2017).Google Scholar
Starr, S.J., Metz, C.E., Lusted, L.B., Goodenough, D.J. (1975). Visual detection and localization of radiographic images. Radiology, 116, 533–538.Google Scholar
Stepan-Buksakowska, I.L., Accurso, J.M., Diehn, F.E., Huston, J., Kaufmann, T.J., Luetmer, P.H., Wood, C.P., Yang, X., Blezek, D.J., Carter, R., Hagen, C., Horinek, D., Hejcl, A., Rocek, M., Erickson, B.J. (2014). Computer-aided diagnosis improves detection of small intracranial aneurysms on MRA in a clinical setting. Am J Neuroradiol, 35, 1897–1902.Google Scholar
Swensson, R.G. (1996). Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys, 23, 1709–1725.Google Scholar
Thrall, J.H. (2016). Trends and developments shaping the future of diagnostic medical imaging: 2015 annual oration in diagnostic radiology. Radiology, 279, 660–666.Google Scholar
Uchiyama, Y., Asano, T., Kato, H., Hara, T., Kanematsu, M., Hoshi, H., Iwama, T., Fujita, H. (2012). Computer-aided diagnosis for detection of lacunar infarcts on MR images: ROC analysis of radiologists’ performance. J Digit Imag, 25, 497–503.Google Scholar
University of Chicago. (2017). Metz ROC software. Available at: http://metz-roc.uchicago.edu/MetzROC (accessed November 10, 2017).Google Scholar
University of Iowa. (2017). ROC software. Available at: http://perception.radiology.uiowa.edu/ (accessed November 10, 2017).Google Scholar
van Beek, E.J.R., Mullan, B., Thompson, B. (2008). Evaluation of a real-time interactive pulmonary nodule analysis system on chest digital radiographic images: a prospective study. Acad Radiol, 15, 571–575.Google Scholar
Wagner, R.F., Beam, C.A., Beiden, S.V. (2004). Reader variability in mammography and its implications for expected utility over the population of readers and cases. Med Decis Making, 24, 561–572.Google Scholar
Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar
Way, T., Chan, H.P., Hadjiiski, L., Sahiner, B., Chughtai, A., Song, T.K., Poopat, C., Stojanovska, J., Frank, L., Attili, A., Bogot, N., Cascade, P.N., Kazerooni, E.A. (2010). Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists’ performance. Acad Radiol, 17, 323–332.Google Scholar
Wei, J., Chan, H.P., Sahiner, B., Zhou, C., Hadjiiski, L.M., Roubidoux, M.A., Helvie, M.A. (2009). Computer-aided detection of breast masses on mammograms: dual system approach with two-view analysis. Med Phys, 36, 4451–4460.Google Scholar
Xin, H., Metz, C.E., Tsui, B.M.W., Links, J.M., Frey, E.C. (2006). Three-class ROC analysis – a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imag, 25, 571–581.Google Scholar
Yanagawa, M., Honda, O., Yoshida, S., Ono, Y., Inoue, A., Daimon, T., Sumikawa, H., Mihara, N., Johkoh, T., Tomiyama, N., Nakamura, H. (2009). Commercially available computer-aided detection system for pulmonary nodules on thin-section images using 64 detectors-row CT: preliminary study of 48 cases. Acad Radiol, 16, 924–933.Google Scholar
Yoon, H.J., Zheng, B., Sahiner, B., Chakraborty, D.P. (2007). Evaluating computer-aided detection algorithms. Med Phys, 34, 2024–2038.Google Scholar
Zeiler, M.D., Fergus, R. (2014). Visualizing and understanding convolutional networks. In: Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I. Cham: Springer International Publishing, pp. 818–833.Google Scholar
Zweig, M.H., Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem, 39, 561–577.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×