Receiver Operating Characteristic (ROC) Analysis
Main Article Content
Abstract
Visual expertise covers a broad range of types of studies and methodologies. Many studies incorporate some measure(s) of observer performance or how well participants perform on a given task. Receiver Operating Characteristic (ROC) analysis is a method commonly used in signal detection tasks (i.e., those in which the observer must decide whether or not a target is present or absent; or must classify a given target as belonging to one category or another), especially those in the medical imaging literature. This frontline paper will review some of the core theoretical underpinnings of ROC analysis, provide an overview of how to conduct an ROC study, and discuss some of the key variants of ROC analysis and their applications.
Article Details
How to Cite
Krupinski, E. A. (2017). Receiver Operating Characteristic (ROC) Analysis. Frontline Learning Research, 5(3), 31–42. https://doi.org/10.14786/flr.v5i2.250
Section
Articles
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
Institute of Medicine. (1999). To Err is Human: Building a Safer Health Care System. Washington, DC: National Academy Press.
Schulman, K.A., Kim, J.J. (2000). Medical errors: how the US government is addressing the problem. Curr Control Trials Cardiovasc Med, 1(1), 35-37. DOI: 10.1186/cvm-1-1-035
Wald, A. (1950). Statistical Decision Functions. New York, NY: Wiley, Inc.
Peterson, W.W., Birdsall, T.L., Fox, W.C. (1954). The theory of signal detectability. IRE Prof Gp In Theory Trans PGIT, 4, (4), 171-212. DOI: 10.1109/TIT.1954.1057460
Tanner, W.P., Swets, J.A. (1954). A decision-making theory of visual detection. Psych Rev, 61, (6), 401-409. PMID: 13215690
Green, D.M., Swets, J.A. (1974). Signal Detection Theory and Psychophysics. Huntington, NY: Krieger Publishers.
Egan, J.P. (1975). Signal detection theory and ROC analysis. New York, NY: Academic Press.
Lusted, L.B. (1960). Logical analysis in roentgen diagnosis. Radiol, 74,178-193. DOI: http://dx.doi.org/10.1148/74.2.178
Lusted, L.B. (1968). Introduction to Medical Decision Making. Springfield, IL: Charles C. Thomas Publishers.
Lusted, L.B. (1969). Perception of the Roentgen image: Applications of signal detection theory. Rad Clin N Am, 7, 435-459.
Lusted, L.B. (1971). Signal detectability and medical decision making. Science, 171, (3977), 1217-1219. DOI: 10.1126/science.171.3977.1217
McNeil, B.J., Adelstein, S.J. (1976). Determining the value of diagnostic and screening tests. J Nuc Med, 17, (6), 439-448. PMID:1262961
McNeil, B.J., Hanley, J.A. (1984). Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Dec Making, 4, (2), 137-150. DOI:10.1177/0272989X8400400203
McNeil, B.J., Keeler, E., Adelstein, S.J. (1975). Primer on certain elements of medical decision making. NE J Med, 293, (5), 211-215. DOI: 10.1056/NEJM197507312930501
Swets, J.A., Pickett, R.M. (1982). Evaluation of Diagnostic Systems. Methods from signal detection theory. New York, NY: Academic Press.
Birkelo, C.C., Chamberlain, W.E., Phelps, P.S. (1947). Tuberculosis case finding. A comparison of the effectiveness of various roentgenographic and photofluorographic methods. JAMA, 133, (6), 359-366. PMID: 20281873
Garland, L.H. (1949). On the scientific evaluation of diagnostic procedures. Radiol, 52, (3), 309-328. DOI: http://dx.doi.org/10.1148/52.3.309
Edwards, D.C. (2013). Validation of Monte Carlo estimates of three-class ideal observer operating points for normal data. Acad Radiol, 20 (7), 908-914. DOI: 10.1016/j.acra.2013.04.002
Nakas, C.T. (2014). Developments in ROC surface analysis and assessment of diagnostic markers in three-class classification problems. REVSTAT – Stat J, 12 (1), 43-65.
Petrick, N., Gallas, B.D., Samuelson, F.W., Wagner, R.F., Myers, K.J. (2005). Influence of panel size and expert skill on truth panel performance when combining expert ratings. Proc SPIE Med Imag, 5749, 596286. DOI: 10.1117/12.596286
Kundel, H.L., Polansky, M. (1997). Mixture distribution and receiver operating characteristic analysis of bedside chest imaging with screen-film and computed radiography. Acad Radiol, 4 (1), 1-7. PMID:
904086.
Zou, K.H., Hall, W.J., Shapiro, D.E. (1997). Smooth non-parametric receiver operating characteristic (ROC)
curves for continuous diagnostic tests. Stats Med, 16 (19), 2143-2156. PMID: 9330425
Hanley, J.A., McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiol, 143 (1), 29-36. DOI: http://dx.doi.org/10.1148/radiology.143.1.7063747
Hajian-Tilaki, K.O., Hanley, J.A., Joseph, L., Collet, J.P. (1997). A comparison of parametric and
nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making, 17 (1), 94-102. DOI:10.1177/0272989X9701700111
Dorfmam, D.D., Alf, E. (1968). Maximum likelihood estimation of parameters of signal detection theory: a direct solution. Psychometrika, 33 (1), 117-124. DOI: 10.1007/BF02289677
Dorfman, D.D., Alf, E. (1969). Maximum likelihood estimation of parameters of signal detection theory and
determination of confidence intervals – rating method data. J Math Psychol, 6 (3), 487-496. DOI: http://dx.doi.org/10.1016/0022-2496(69)90019-4
Hanley, J.A. (1988). The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis
Making, 8 (3), 197-203. DOI: 10.1177/0272989X8800800308
Metz, C.E., Herman, B.A., Shen, J.H. (1998). Maximum-likelihood estimation of ROC curves from
continuously-distributed data. Stats Med, 17 (9), 1033-1053. PMID: 9612889
Faraggi, D., Reiser, B. (2002). Estimation of the area under the ROC curve. Stats Med, 21 (20), 3093-3106.
DOI: 10.1002/sim.1228
McClish, D.K. (1989). Analyzing a portion of the ROC curve. Med Decis Making, 9 (3), 190-195. DOI:
10.1177/0272989X8900900307
Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for
highly sensitive diagnostic tests. Radiol, 201 (3), 745-750. DOI: 10.1148/radiology.201.3.8939225
Swets, J.A. (1979). ROC analysis applied to the evaluation of medical imaging techniques. Radiol, 14 (2),
109-121. PMID: 478799
Swets, J.A., Dawes, R.M., Monahan, J. (2000). Psychological science can improve diagnostic decisions.
Psych Sci Public Interest, 1 (1), 1-26. DOI: 10.1111/1529-1006.001
Dorfman, D.D., Berbaum, K.S., Metz. C.E., Lenth, R.V., Hanley, J.A., Dagga, H.A. (1997). Proper receiver
operating characteristic analysis: the bigamma model. Acad Radiol, 4 (2), 138-149. PMID: 9061087
Metz, C.E., Pan, X. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J
Math Psych, 43 (1), 1-33. DOI: 10.1006/jmps.1998.1218
Hanley, J.A., McNeil, B.J. (1983). A method for comparing the areas under receiver operating characteristic
curves derived from the same cases. Radiol, 148 (3), 839-843. DOI: 10.1148/radiology.148.3.6878708
Delong, E.R., Delong, D.M., Clarke-Pearson, D.L. (19880. Comparing the areas under two or more
correlated receiver operating characteristics curves: a non-parametric approach. Biometrics, 44 (3), 837-845. PMID: 3203132
Metz, C.E., Kronman, H.B. (1980). Statistical significance tests for binormal ROC curves. J Math Psych, 22
(3), 218-243. DOI: http://dx.doi.org/10.1016/0022-2496(80)90020-6
Dorfman, D.D., Berbaum, K.S., Metz, C.E. (1992). Receiver operating characteristic rating analysis:
generalization to the population of readers and patients with the jackknife method. Invest Radiol, 27(9), 723-731. PMID: 1399456
Obuchowski, N.A. (1997). Testing for equivalence of diagnostic tests. Am J Roentgen, 168 (1), 13-17. DOI:
10.2214/ajr.168.1.8976911
Starr, S.J., Metz, C.E., Lusted, L.B., Goodenough, D.J. (1975). Visual detection and localization of
radiographic images. Radiol, 116 (3), 533-538. DOI:10.1148/116.3.533
Swensson, R.G. (1996). Unified measurement of observer performance in detecting and localizing target
objects on images. Med Phys, 23 (10), 1709-1725. DOI:10.1118/1.597758
Bunch, P.C., Hamilton, J.F., Sanderson, G.K. Simmons, A.H. (1978). A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photogr Eng, 4,166–171.
Chakraborty, D.P., Berbaum, K.S. (2004). Observer studies involving detection and localization: modeling,
analysis, and validation. Med Phys, 31 (8), 2313-2330. DOI: 10.1118/1.1769352
Chakraborty, D.P. (2005). Recent advances in observer performance methodology: jackknife free-response ROC (JAFROC). Rad Protect Dosim, 114 (1), 26-31. DOI: 10.1093/rpd/nch512
Chakraborty, D.P. (2006). Analysis of location specific observer performance data: validated extensions of
the jackknife free-response (JAFROC) method. Acad Radiol, 13 (10), 1187-1193. DOI: 10.1016/j.acra.2006.06.016
Chakraborty, D.P., Winter, L.H.L. (1990). Free-response methodology: alternate analysis and the new
observer-performance experiment. Radiol, 174 (3), 873-881. DOI: 10.1148/radiology.174.3.2305073
Zhou, X.H., Obuchowski, N.A., McClish, D.K. (2002). Statistical Methods in Diagnostic Medicine. New York, NY: Wiley.
Obuchowski, N.A. (1994). Computing sample size for receiver operating characteristic studies. Invest
Radiol, 29 (2), 238-243. DOI:10.2214/ajr.175.3.1750603
Obuchowski, N.A. (2000). Sample size tables for receiver operating characteristic studies. Am J Roentgen,
175 (3), 603-608. DOI:10.2214/ajr.175.3.1750603
Obuchowski, N.A. (2004). How many observers care needed in clinical studies of medical imaging? Am J Roentgen, 182 (4), 867-869. DOI: 10.2214/ajr.182.4.1820867
University of Iowa Medical Image Perception ROC Software. http://perception.radiology.uiowa.edu/Software/ReceiverOperatingCharacteristicROC/tabid/120/Default.aspx Last accessed April 13, 2016.
University of Chicago ROC Software. http://metz-roc.uchicago.edu/ Last accessed April 13, 2016.
Dev Chakraborty’s FROC Web Site. http://perception.radiology.uiowa.edu/Software/ReceiverOperatingCharacteristicROC/tabid/120/Default.aspx Last accessed April 13, 2016.
MedCalc Statistical Software. https://www.medcalc.org/manual/roc-curves.php Last accessed April 13, 2016.
Analyse-It. http://analyse-it.com/docs/220/method_evaluation/roc_curve_plot.htm Last accessed April 13, 2016.
NCSS Statistical Software. http://www.ncss.com/software/ncss/procedures/ Last accessed April 13, 2016.
SPSS Statistics. http://www-03.ibm.com/software/products/en/spss-statistics Last accessed April 13, 2016.
STATA Data Analysis and Statistical Software. http://www.stata.com/features/overview/receiver-operating-characteristic/ Last accessed April 13, 2016.
Schulman, K.A., Kim, J.J. (2000). Medical errors: how the US government is addressing the problem. Curr Control Trials Cardiovasc Med, 1(1), 35-37. DOI: 10.1186/cvm-1-1-035
Wald, A. (1950). Statistical Decision Functions. New York, NY: Wiley, Inc.
Peterson, W.W., Birdsall, T.L., Fox, W.C. (1954). The theory of signal detectability. IRE Prof Gp In Theory Trans PGIT, 4, (4), 171-212. DOI: 10.1109/TIT.1954.1057460
Tanner, W.P., Swets, J.A. (1954). A decision-making theory of visual detection. Psych Rev, 61, (6), 401-409. PMID: 13215690
Green, D.M., Swets, J.A. (1974). Signal Detection Theory and Psychophysics. Huntington, NY: Krieger Publishers.
Egan, J.P. (1975). Signal detection theory and ROC analysis. New York, NY: Academic Press.
Lusted, L.B. (1960). Logical analysis in roentgen diagnosis. Radiol, 74,178-193. DOI: http://dx.doi.org/10.1148/74.2.178
Lusted, L.B. (1968). Introduction to Medical Decision Making. Springfield, IL: Charles C. Thomas Publishers.
Lusted, L.B. (1969). Perception of the Roentgen image: Applications of signal detection theory. Rad Clin N Am, 7, 435-459.
Lusted, L.B. (1971). Signal detectability and medical decision making. Science, 171, (3977), 1217-1219. DOI: 10.1126/science.171.3977.1217
McNeil, B.J., Adelstein, S.J. (1976). Determining the value of diagnostic and screening tests. J Nuc Med, 17, (6), 439-448. PMID:1262961
McNeil, B.J., Hanley, J.A. (1984). Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Dec Making, 4, (2), 137-150. DOI:10.1177/0272989X8400400203
McNeil, B.J., Keeler, E., Adelstein, S.J. (1975). Primer on certain elements of medical decision making. NE J Med, 293, (5), 211-215. DOI: 10.1056/NEJM197507312930501
Swets, J.A., Pickett, R.M. (1982). Evaluation of Diagnostic Systems. Methods from signal detection theory. New York, NY: Academic Press.
Birkelo, C.C., Chamberlain, W.E., Phelps, P.S. (1947). Tuberculosis case finding. A comparison of the effectiveness of various roentgenographic and photofluorographic methods. JAMA, 133, (6), 359-366. PMID: 20281873
Garland, L.H. (1949). On the scientific evaluation of diagnostic procedures. Radiol, 52, (3), 309-328. DOI: http://dx.doi.org/10.1148/52.3.309
Edwards, D.C. (2013). Validation of Monte Carlo estimates of three-class ideal observer operating points for normal data. Acad Radiol, 20 (7), 908-914. DOI: 10.1016/j.acra.2013.04.002
Nakas, C.T. (2014). Developments in ROC surface analysis and assessment of diagnostic markers in three-class classification problems. REVSTAT – Stat J, 12 (1), 43-65.
Petrick, N., Gallas, B.D., Samuelson, F.W., Wagner, R.F., Myers, K.J. (2005). Influence of panel size and expert skill on truth panel performance when combining expert ratings. Proc SPIE Med Imag, 5749, 596286. DOI: 10.1117/12.596286
Kundel, H.L., Polansky, M. (1997). Mixture distribution and receiver operating characteristic analysis of bedside chest imaging with screen-film and computed radiography. Acad Radiol, 4 (1), 1-7. PMID:
904086.
Zou, K.H., Hall, W.J., Shapiro, D.E. (1997). Smooth non-parametric receiver operating characteristic (ROC)
curves for continuous diagnostic tests. Stats Med, 16 (19), 2143-2156. PMID: 9330425
Hanley, J.A., McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiol, 143 (1), 29-36. DOI: http://dx.doi.org/10.1148/radiology.143.1.7063747
Hajian-Tilaki, K.O., Hanley, J.A., Joseph, L., Collet, J.P. (1997). A comparison of parametric and
nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making, 17 (1), 94-102. DOI:10.1177/0272989X9701700111
Dorfmam, D.D., Alf, E. (1968). Maximum likelihood estimation of parameters of signal detection theory: a direct solution. Psychometrika, 33 (1), 117-124. DOI: 10.1007/BF02289677
Dorfman, D.D., Alf, E. (1969). Maximum likelihood estimation of parameters of signal detection theory and
determination of confidence intervals – rating method data. J Math Psychol, 6 (3), 487-496. DOI: http://dx.doi.org/10.1016/0022-2496(69)90019-4
Hanley, J.A. (1988). The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis
Making, 8 (3), 197-203. DOI: 10.1177/0272989X8800800308
Metz, C.E., Herman, B.A., Shen, J.H. (1998). Maximum-likelihood estimation of ROC curves from
continuously-distributed data. Stats Med, 17 (9), 1033-1053. PMID: 9612889
Faraggi, D., Reiser, B. (2002). Estimation of the area under the ROC curve. Stats Med, 21 (20), 3093-3106.
DOI: 10.1002/sim.1228
McClish, D.K. (1989). Analyzing a portion of the ROC curve. Med Decis Making, 9 (3), 190-195. DOI:
10.1177/0272989X8900900307
Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for
highly sensitive diagnostic tests. Radiol, 201 (3), 745-750. DOI: 10.1148/radiology.201.3.8939225
Swets, J.A. (1979). ROC analysis applied to the evaluation of medical imaging techniques. Radiol, 14 (2),
109-121. PMID: 478799
Swets, J.A., Dawes, R.M., Monahan, J. (2000). Psychological science can improve diagnostic decisions.
Psych Sci Public Interest, 1 (1), 1-26. DOI: 10.1111/1529-1006.001
Dorfman, D.D., Berbaum, K.S., Metz. C.E., Lenth, R.V., Hanley, J.A., Dagga, H.A. (1997). Proper receiver
operating characteristic analysis: the bigamma model. Acad Radiol, 4 (2), 138-149. PMID: 9061087
Metz, C.E., Pan, X. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J
Math Psych, 43 (1), 1-33. DOI: 10.1006/jmps.1998.1218
Hanley, J.A., McNeil, B.J. (1983). A method for comparing the areas under receiver operating characteristic
curves derived from the same cases. Radiol, 148 (3), 839-843. DOI: 10.1148/radiology.148.3.6878708
Delong, E.R., Delong, D.M., Clarke-Pearson, D.L. (19880. Comparing the areas under two or more
correlated receiver operating characteristics curves: a non-parametric approach. Biometrics, 44 (3), 837-845. PMID: 3203132
Metz, C.E., Kronman, H.B. (1980). Statistical significance tests for binormal ROC curves. J Math Psych, 22
(3), 218-243. DOI: http://dx.doi.org/10.1016/0022-2496(80)90020-6
Dorfman, D.D., Berbaum, K.S., Metz, C.E. (1992). Receiver operating characteristic rating analysis:
generalization to the population of readers and patients with the jackknife method. Invest Radiol, 27(9), 723-731. PMID: 1399456
Obuchowski, N.A. (1997). Testing for equivalence of diagnostic tests. Am J Roentgen, 168 (1), 13-17. DOI:
10.2214/ajr.168.1.8976911
Starr, S.J., Metz, C.E., Lusted, L.B., Goodenough, D.J. (1975). Visual detection and localization of
radiographic images. Radiol, 116 (3), 533-538. DOI:10.1148/116.3.533
Swensson, R.G. (1996). Unified measurement of observer performance in detecting and localizing target
objects on images. Med Phys, 23 (10), 1709-1725. DOI:10.1118/1.597758
Bunch, P.C., Hamilton, J.F., Sanderson, G.K. Simmons, A.H. (1978). A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photogr Eng, 4,166–171.
Chakraborty, D.P., Berbaum, K.S. (2004). Observer studies involving detection and localization: modeling,
analysis, and validation. Med Phys, 31 (8), 2313-2330. DOI: 10.1118/1.1769352
Chakraborty, D.P. (2005). Recent advances in observer performance methodology: jackknife free-response ROC (JAFROC). Rad Protect Dosim, 114 (1), 26-31. DOI: 10.1093/rpd/nch512
Chakraborty, D.P. (2006). Analysis of location specific observer performance data: validated extensions of
the jackknife free-response (JAFROC) method. Acad Radiol, 13 (10), 1187-1193. DOI: 10.1016/j.acra.2006.06.016
Chakraborty, D.P., Winter, L.H.L. (1990). Free-response methodology: alternate analysis and the new
observer-performance experiment. Radiol, 174 (3), 873-881. DOI: 10.1148/radiology.174.3.2305073
Zhou, X.H., Obuchowski, N.A., McClish, D.K. (2002). Statistical Methods in Diagnostic Medicine. New York, NY: Wiley.
Obuchowski, N.A. (1994). Computing sample size for receiver operating characteristic studies. Invest
Radiol, 29 (2), 238-243. DOI:10.2214/ajr.175.3.1750603
Obuchowski, N.A. (2000). Sample size tables for receiver operating characteristic studies. Am J Roentgen,
175 (3), 603-608. DOI:10.2214/ajr.175.3.1750603
Obuchowski, N.A. (2004). How many observers care needed in clinical studies of medical imaging? Am J Roentgen, 182 (4), 867-869. DOI: 10.2214/ajr.182.4.1820867
University of Iowa Medical Image Perception ROC Software. http://perception.radiology.uiowa.edu/Software/ReceiverOperatingCharacteristicROC/tabid/120/Default.aspx Last accessed April 13, 2016.
University of Chicago ROC Software. http://metz-roc.uchicago.edu/ Last accessed April 13, 2016.
Dev Chakraborty’s FROC Web Site. http://perception.radiology.uiowa.edu/Software/ReceiverOperatingCharacteristicROC/tabid/120/Default.aspx Last accessed April 13, 2016.
MedCalc Statistical Software. https://www.medcalc.org/manual/roc-curves.php Last accessed April 13, 2016.
Analyse-It. http://analyse-it.com/docs/220/method_evaluation/roc_curve_plot.htm Last accessed April 13, 2016.
NCSS Statistical Software. http://www.ncss.com/software/ncss/procedures/ Last accessed April 13, 2016.
SPSS Statistics. http://www-03.ibm.com/software/products/en/spss-statistics Last accessed April 13, 2016.
STATA Data Analysis and Statistical Software. http://www.stata.com/features/overview/receiver-operating-characteristic/ Last accessed April 13, 2016.