On classification in the case of a medical data set with a complicated distribution

Martti Juhola, Henry Joutsijoki, Heikki Aalto, Timo P. Hirvonen

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Abstract In one of our earlier studies we noticed how straightforward cleaning of our medical data set impaired its classification results considerably with some machine learning methods, but not all of them, unexpectedly and against intuition compared to the original situation without any data cleaning. After a more precise exploration of the data, we found that the reason was the complicated variable distribution of the data although there were only two classes in it. In addition to a straightforward data cleaning method, we used an efficient way called neighbourhood cleaning that solved the problem and improved our classification accuracies 5–10%, at their best, up to 95% of all test cases. This shows how important it is first very carefully to study distributions of data sets to be classified and use different cleaning techniques in order to obtain best classification results.
Original languageEnglish
JournalApplied Computing and Informatics
Volume10
Issue number1
Pages (from-to)52 - 67
Number of pages16
ISSN2210-8327
DOIs
Publication statusPublished - 2014
MoE publication typeA1 Journal article-refereed

Fields of Science

  • Data mining, Data cleaning, Complicated data distributions, Classification
  • 113 Computer and information sciences

Cite this