Receiver operating characteristic curve

1 · What it is

An ROC curve is a graph of how a binary classifier behaves across all of its possible thresholds. A binary classifier sorts things into two groups, such as spam and not spam. It gives each item a score, and a threshold decides which group the item lands in. The vertical axis is the true positive rate, another name for recall: the share of real positives the model catches. The horizontal axis is the false positive rate: the share of real negatives it wrongly flags. The name, receiver operating characteristic, is left over from Second World War radar, where it helped tell a real signal from noise.

Take the made-up example in the figure: five real positives and five real negatives, each with a score. A threshold above every score predicts nothing positive, so both rates are 0. Lower it to 0.9 and one positive clears it, giving a true positive rate of 1 in 5, or 0.2, with no false alarms. At 0.5, four positives and two negatives clear it, so the point is 0.4 across and 0.8 up. At 0.3 it is 0.6 across and 1.0 up. Lowering the threshold only lets more examples through, so the points move up and to the right, joined by straight lines.

The area under that curve, AUC, has a neat meaning. It is the chance that a randomly chosen positive gets a higher score than a randomly chosen negative. Here, 20 of the 25 positive and negative pairs are ranked the right way round, so the full curve, using every threshold, has an AUC of 20 ÷ 25 = 0.80. The three thresholds in the figure give 0.78, a little less, because a coarser curve cuts a corner. A model that guesses at random follows the diagonal and scores 0.5; a perfect model scores 1.0. In the Python library scikit-learn, roc_curve returns the points and roc_auc_score returns the area.