The problem of Area Under the Curve

    Research output: Contribution to conferencePaper

    18 Citations (Scopus)

    Abstract

    Receiver Operating Characterics curves have recently received considerable attention as mechanisms for comparing different algorithms or experimental results. However the common approach of comparing Area Under the Curve has come in for some criticism with alternatives such as Area Under Kappa and H-measure being proposed as alternative measure. However, these measures have their own idiosyncracies and neglect certain advantages of RoC analysis that do not carry over to the proposed approach. In addition they suffer from the general desire for a one fits all mentality as opposed to a pareto approach. That is we want a single number to optimize, rather than a suite of numbers to trade off. The starting point for all of this research is the inadequacy and bias of traditional measures such as Accuracy, Recall and Precision, and F-measure - these should never be used singly or as a group, and ROC analysis is a very good alternative to them if used correctly, but treating it as a graph rather than trying to boil it down to a single number. Other measures that seek to remove the bias in traditional measures include Krippendorf's Alpha, Scott's Pi, Powers' Informedness and Markedness, as well as a great many variant Kappa statistics. The original Kappa statistics were intrinsically dichotomous but the family has been well generalized to allow for multiple classes and multiple sources of labels. We discuss the proper and improper use of ROC curves, the issues with AUC and Kappa, and make a recommendation as to the approach to take in comparing experimental algorithms or other kinds of tests.

    Original languageEnglish
    Pages567-573
    Number of pages7
    DOIs
    Publication statusPublished - 2012
    EventICIST 2012 -
    Duration: 23 Mar 2012 → …

    Conference

    ConferenceICIST 2012
    Period23/03/12 → …

    Fingerprint Dive into the research topics of 'The problem of Area Under the Curve'. Together they form a unique fingerprint.

  • Cite this