Machine learning algorithms: why the cup occasionally appears half-empty

The burgeoning interest in the use of machine learning (ML) algorithms for prediction has led to vigorous debate on whether, despite all the hype, they genuinely outperform more traditional regression based approaches to prediction [1,2,3]. The paper “Predicting non-responders to lifestyle intervention in prediabetes: a machine learning approach” by Foppiani et al. [4] in this month’s edition of EJCN highlights both the advantages that ML algorithms can offer, and the common trap of judging performance using overall classification accuracy. For this study, the authors goal was to determine whether a ML approach improves the ability to identify individuals with pre-diabetes who did or did not respond to a 12-month lifestyle intervention. Adequate response was considered a normalisation of blood glucose (<100 mg/dL) and the intervention consisted of a hypocaloric omnivorous Mediterranean style diet plus following the WHO physical activity guidelines. Since predicting patients that will respond to treatment is the basis of precision-medicine [5], the study essentially evaluates whether ML algorithms can support a precision-medicine approach to blood glucose control by better selecting likely responders to a lifestyle intervention. After training 11 different supervised learning algorithms and a standard logistic regression (LR) model, only a Random Forest (RF) achieved a higher correct classification fraction (CCF) (68.5%), than a completely naive model that simply classified all subjects according to the majority class (68.0%) which were the non-responders. Thus, only an additional 4 from 734 subjects (0.5%) were correctly classified by employing a RF algorithm. At face value, this appears very scant reward for the considerable time, effort and cost for deploying an algorithm that requires collating data on 59 variables taken from blood and urine examinations, abdominal ultrasounds, vital signs, indirect calorimetry, bioimpedance analysis, anthropometry, demographic data and medical history. However, classification accuracy on its own offers only a limited story when a more subtle and nuanced evaluation of performance is essential...