Receiver Operating Characteristic (ROC) Analysis

Article Sidebar

PDF HTML
Published: Jul 14, 2017
DOI: https://doi.org/10.14786/flr.v5i2.250
Keywords:
Receiver Operating Characteristic analysis, ROC, observer performance, visual expertise

Main Article Content

Elizabeth Anne Krupinski
Emory University

Abstract

Visual expertise covers a broad range of types of studies and methodologies. Many studies incorporate some measure(s) of observer performance or how well participants perform on a given task. Receiver Operating Characteristic (ROC) analysis is a method commonly used in signal detection tasks (i.e., those in which the observer must decide whether or not a target is present or absent; or must classify a given target as belonging to one category or another), especially those in the medical imaging literature. This frontline paper will review some of the core theoretical underpinnings of ROC analysis, provide an overview of how to conduct an ROC study, and discuss some of the key variants of ROC analysis and their applications.

Article Details

How to Cite
Krupinski, E. A. (2017). Receiver Operating Characteristic (ROC) Analysis. Frontline Learning Research, 5(3), 31–42. https://doi.org/10.14786/flr.v5i2.250
Issue
Vol. 5 No. 3 (2017): Frontline Learning Research - Special issue "Visual Expertise"
Section
Articles

FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.

Author Biography

Elizabeth Anne Krupinski, Emory University

Dr. Krupinski is a Professor and Vice Chair for Research at Emory University in the Departments of Radiology & Imaging Sciences, Psychology and Medical Informatics. She received her BA from Cornell, MA from Montclair State and PhD from Temple, all in Experimental Psychology. Her interests are in medical image perception, observer performance, medical decision making, and human factors. She is Associate Director of Evaluation for the Arizona Telemedicine Program. She has published extensively in these areas, and has presented at conferences nationally and internationally. She is Past Chair of the SPIE Medical Imaging Conference, Past President of the American Telemedicine Association, President of the Medical Image Perception Society, and Past Chair of the Society for Imaging Informatics in Medicine. She serves on a number of editorial boards for both radiology and telemedicine journals and is the Co-Editor of the Journal of Telemedicine & Telecare. She serves regularly as a grant reviewer for the NIH, DoD, TATRC and other federal, state and international funding agencies and has served as a member of a number of FDA review panels.

References

Institute of Medicine. (1999). To Err is Human: Building a Safer Health Care System. Washington, DC: National Academy Press.

Schulman, K.A., Kim, J.J. (2000). Medical errors: how the US government is addressing the problem. Curr Control Trials Cardiovasc Med, 1(1), 35-37. DOI: 10.1186/cvm-1-1-035

Wald, A. (1950). Statistical Decision Functions. New York, NY: Wiley, Inc.

Peterson, W.W., Birdsall, T.L., Fox, W.C. (1954). The theory of signal detectability. IRE Prof Gp In Theory Trans PGIT, 4, (4), 171-212. DOI: 10.1109/TIT.1954.1057460

Tanner, W.P., Swets, J.A. (1954). A decision-making theory of visual detection. Psych Rev, 61, (6), 401-409. PMID: 13215690

Green, D.M., Swets, J.A. (1974). Signal Detection Theory and Psychophysics. Huntington, NY: Krieger Publishers.

Egan, J.P. (1975). Signal detection theory and ROC analysis. New York, NY: Academic Press.

Lusted, L.B. (1960). Logical analysis in roentgen diagnosis. Radiol, 74,178-193. DOI: http://dx.doi.org/10.1148/74.2.178

Lusted, L.B. (1968). Introduction to Medical Decision Making. Springfield, IL: Charles C. Thomas Publishers.

Lusted, L.B. (1969). Perception of the Roentgen image: Applications of signal detection theory. Rad Clin N Am, 7, 435-459.

Lusted, L.B. (1971). Signal detectability and medical decision making. Science, 171, (3977), 1217-1219. DOI: 10.1126/science.171.3977.1217

McNeil, B.J., Adelstein, S.J. (1976). Determining the value of diagnostic and screening tests. J Nuc Med, 17, (6), 439-448. PMID:1262961

McNeil, B.J., Hanley, J.A. (1984). Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Dec Making, 4, (2), 137-150. DOI:10.1177/0272989X8400400203

McNeil, B.J., Keeler, E., Adelstein, S.J. (1975). Primer on certain elements of medical decision making. NE J Med, 293, (5), 211-215. DOI: 10.1056/NEJM197507312930501

Swets, J.A., Pickett, R.M. (1982). Evaluation of Diagnostic Systems. Methods from signal detection theory. New York, NY: Academic Press.

Birkelo, C.C., Chamberlain, W.E., Phelps, P.S. (1947). Tuberculosis case finding. A comparison of the effectiveness of various roentgenographic and photofluorographic methods. JAMA, 133, (6), 359-366. PMID: 20281873

Garland, L.H. (1949). On the scientific evaluation of diagnostic procedures. Radiol, 52, (3), 309-328. DOI: http://dx.doi.org/10.1148/52.3.309

Edwards, D.C. (2013). Validation of Monte Carlo estimates of three-class ideal observer operating points for normal data. Acad Radiol, 20 (7), 908-914. DOI: 10.1016/j.acra.2013.04.002

Nakas, C.T. (2014). Developments in ROC surface analysis and assessment of diagnostic markers in three-class classification problems. REVSTAT – Stat J, 12 (1), 43-65.

Petrick, N., Gallas, B.D., Samuelson, F.W., Wagner, R.F., Myers, K.J. (2005). Influence of panel size and expert skill on truth panel performance when combining expert ratings. Proc SPIE Med Imag, 5749, 596286. DOI: 10.1117/12.596286

Kundel, H.L., Polansky, M. (1997). Mixture distribution and receiver operating characteristic analysis of bedside chest imaging with screen-film and computed radiography. Acad Radiol, 4 (1), 1-7. PMID:

Zou, K.H., Hall, W.J., Shapiro, D.E. (1997). Smooth non-parametric receiver operating characteristic (ROC)

curves for continuous diagnostic tests. Stats Med, 16 (19), 2143-2156. PMID: 9330425

Hanley, J.A., McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiol, 143 (1), 29-36. DOI: http://dx.doi.org/10.1148/radiology.143.1.7063747

Hajian-Tilaki, K.O., Hanley, J.A., Joseph, L., Collet, J.P. (1997). A comparison of parametric and

nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making, 17 (1), 94-102. DOI:10.1177/0272989X9701700111

Dorfmam, D.D., Alf, E. (1968). Maximum likelihood estimation of parameters of signal detection theory: a direct solution. Psychometrika, 33 (1), 117-124. DOI: 10.1007/BF02289677

Dorfman, D.D., Alf, E. (1969). Maximum likelihood estimation of parameters of signal detection theory and

determination of confidence intervals – rating method data. J Math Psychol, 6 (3), 487-496. DOI: http://dx.doi.org/10.1016/0022-2496(69)90019-4

Hanley, J.A. (1988). The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis

Making, 8 (3), 197-203. DOI: 10.1177/0272989X8800800308

Metz, C.E., Herman, B.A., Shen, J.H. (1998). Maximum-likelihood estimation of ROC curves from

continuously-distributed data. Stats Med, 17 (9), 1033-1053. PMID: 9612889

Faraggi, D., Reiser, B. (2002). Estimation of the area under the ROC curve. Stats Med, 21 (20), 3093-3106.

DOI: 10.1002/sim.1228

McClish, D.K. (1989). Analyzing a portion of the ROC curve. Med Decis Making, 9 (3), 190-195. DOI:

1177/0272989X8900900307

Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for

highly sensitive diagnostic tests. Radiol, 201 (3), 745-750. DOI: 10.1148/radiology.201.3.8939225

Swets, J.A. (1979). ROC analysis applied to the evaluation of medical imaging techniques. Radiol, 14 (2),

-121. PMID: 478799

Swets, J.A., Dawes, R.M., Monahan, J. (2000). Psychological science can improve diagnostic decisions.

Psych Sci Public Interest, 1 (1), 1-26. DOI: 10.1111/1529-1006.001

Dorfman, D.D., Berbaum, K.S., Metz. C.E., Lenth, R.V., Hanley, J.A., Dagga, H.A. (1997). Proper receiver

operating characteristic analysis: the bigamma model. Acad Radiol, 4 (2), 138-149. PMID: 9061087

Metz, C.E., Pan, X. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J

Math Psych, 43 (1), 1-33. DOI: 10.1006/jmps.1998.1218

Hanley, J.A., McNeil, B.J. (1983). A method for comparing the areas under receiver operating characteristic

curves derived from the same cases. Radiol, 148 (3), 839-843. DOI: 10.1148/radiology.148.3.6878708

Delong, E.R., Delong, D.M., Clarke-Pearson, D.L. (19880. Comparing the areas under two or more

correlated receiver operating characteristics curves: a non-parametric approach. Biometrics, 44 (3), 837-845. PMID: 3203132

Metz, C.E., Kronman, H.B. (1980). Statistical significance tests for binormal ROC curves. J Math Psych, 22

(3), 218-243. DOI: http://dx.doi.org/10.1016/0022-2496(80)90020-6

Dorfman, D.D., Berbaum, K.S., Metz, C.E. (1992). Receiver operating characteristic rating analysis:

generalization to the population of readers and patients with the jackknife method. Invest Radiol, 27(9), 723-731. PMID: 1399456

Obuchowski, N.A. (1997). Testing for equivalence of diagnostic tests. Am J Roentgen, 168 (1), 13-17. DOI:

2214/ajr.168.1.8976911

Starr, S.J., Metz, C.E., Lusted, L.B., Goodenough, D.J. (1975). Visual detection and localization of

radiographic images. Radiol, 116 (3), 533-538. DOI:10.1148/116.3.533

Swensson, R.G. (1996). Unified measurement of observer performance in detecting and localizing target

objects on images. Med Phys, 23 (10), 1709-1725. DOI:10.1118/1.597758

Bunch, P.C., Hamilton, J.F., Sanderson, G.K. Simmons, A.H. (1978). A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photogr Eng, 4,166–171.

Chakraborty, D.P., Berbaum, K.S. (2004). Observer studies involving detection and localization: modeling,

analysis, and validation. Med Phys, 31 (8), 2313-2330. DOI: 10.1118/1.1769352

Chakraborty, D.P. (2005). Recent advances in observer performance methodology: jackknife free-response ROC (JAFROC). Rad Protect Dosim, 114 (1), 26-31. DOI: 10.1093/rpd/nch512

Chakraborty, D.P. (2006). Analysis of location specific observer performance data: validated extensions of

the jackknife free-response (JAFROC) method. Acad Radiol, 13 (10), 1187-1193. DOI: 10.1016/j.acra.2006.06.016

Chakraborty, D.P., Winter, L.H.L. (1990). Free-response methodology: alternate analysis and the new

observer-performance experiment. Radiol, 174 (3), 873-881. DOI: 10.1148/radiology.174.3.2305073

Zhou, X.H., Obuchowski, N.A., McClish, D.K. (2002). Statistical Methods in Diagnostic Medicine. New York, NY: Wiley.

Obuchowski, N.A. (1994). Computing sample size for receiver operating characteristic studies. Invest

Radiol, 29 (2), 238-243. DOI:10.2214/ajr.175.3.1750603

Obuchowski, N.A. (2000). Sample size tables for receiver operating characteristic studies. Am J Roentgen,

(3), 603-608. DOI:10.2214/ajr.175.3.1750603

Obuchowski, N.A. (2004). How many observers care needed in clinical studies of medical imaging? Am J Roentgen, 182 (4), 867-869. DOI: 10.2214/ajr.182.4.1820867

University of Iowa Medical Image Perception ROC Software. http://perception.radiology.uiowa.edu/Software/ReceiverOperatingCharacteristicROC/tabid/120/Default.aspx Last accessed April 13, 2016.

University of Chicago ROC Software. http://metz-roc.uchicago.edu/ Last accessed April 13, 2016.

Dev Chakraborty’s FROC Web Site. http://perception.radiology.uiowa.edu/Software/ReceiverOperatingCharacteristicROC/tabid/120/Default.aspx Last accessed April 13, 2016.

MedCalc Statistical Software. https://www.medcalc.org/manual/roc-curves.php Last accessed April 13, 2016.

Analyse-It. http://analyse-it.com/docs/220/method_evaluation/roc_curve_plot.htm Last accessed April 13, 2016.

NCSS Statistical Software. http://www.ncss.com/software/ncss/procedures/ Last accessed April 13, 2016.

SPSS Statistics. http://www-03.ibm.com/software/products/en/spss-statistics Last accessed April 13, 2016.

STATA Data Analysis and Statistical Software. http://www.stata.com/features/overview/receiver-operating-characteristic/ Last accessed April 13, 2016.