The Necessity of Leave One Subject Out (LOSO) Cross Validation for EEG Disease Diagnosis

Sajeev Kunjan, T. S. Grummett, K. J. Pope, D. M.W. Powers, S. P. Fitzgibbon, T. Bastiampillai, M. Battersby, T. W. Lewis

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

High variability between individual subjects and recording sessions is a known fact about scalp recorded EEG signal. While some do, the majority of the EEG based machine learning studies do not attempt to assess performance of algorithms across recording sessions or across subjects, instead studies use the whole data-set available for training and testing, using an established k-fold cross validation technique and thus missing performance in a real-life setting on an unseen subject. This study primarily aimed to show how important is to have a leave-one-subject-out (LOSO) evaluation done for any scalp recorded EEG based machine learning. This study also demonstrates effectiveness of a Multilayer Perceptron (MLP) in getting good LOSO accuracy from balanced, clean EEG data, without any pre-processing in comparison with traditional machine learning algorithms. The study used data from participants diagnosed with schizophrenia, as well as a group of participants with no known neurological disorder. Classification was done using traditional methods and MLP to classify the participants as belonging to disease or control subjects. Results shows that 85% accuracy on unseen subject was achievable from a clean data-set. MLP is seen to be effective in finding features by which schizophrenia could be detected from clean EEG data. LOSO evaluation done with this proven MLP configuration using carefully and intentionally corrupted data clearly indicate that for disease diagnosis, the k-fold classification result is misleading. Therefore, evaluation of any scalp recorded EEG based disease classification method must use a LOSO style cross-validation.

Original languageEnglish
Title of host publicationBrain Informatics
Subtitle of host publication14th International Conference, BI 2021 Virtual Event, September 17–19, 2021 Proceedings
EditorsMufti Mahmud, M Shamim Kaiser, Stefano Vassanelli, Qionghai Dai, Ning Zhong
Place of PublicationSwitzerland
PublisherSpringer Nature
Pages558-567
Number of pages10
ISBN (Electronic)9783030869939
ISBN (Print)9783030869922
DOIs
Publication statusPublished - 2021
Event14th International Conference on Brain Informatics, BI 2021 - Virtual, Online
Duration: 17 Sep 202119 Sep 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12960 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference14th International Conference on Brain Informatics, BI 2021
CityVirtual, Online
Period17/09/2119/09/21

Bibliographical note

Funding Information:
This work was supported by the National Health and Medical Research Council, the Flinders Medical Centre Foundation, the Clinician?s Special Purpose Fund of the Flinders Medical Centre, and an equipment grant from the Wellcome Trust, London, U.K.. Acknowledgements. We thank Prof Michael Baigent and Dr Randall Long (Department of Psychiatry and Flinders University and Medical Centre), Dr Cate Houen (Central Adelaide Local Health Network, SA Psychiatry Training Unit), Dr Emma Whitham and Prof John Willoughby (Department of Neurology, Flinders University and Medical), for their contributions in collecting and classifying the clinical material and providing the clinical background.

Publisher Copyright:
© 2021, Springer Nature Switzerland AG.

Keywords

  • Classification
  • Cross-subject classification
  • EEG
  • Leave one subject out
  • LOSO
  • Machine learning
  • MLP

Fingerprint

Dive into the research topics of 'The Necessity of Leave One Subject Out (LOSO) Cross Validation for EEG Disease Diagnosis'. Together they form a unique fingerprint.

Cite this