Feature Selection in Supervised Learning Problems using Data Mining Approach

Abstract

When analyzing high dimensional massive data, it is desirable to identify a few important features that affect a certain outcome of interest using feature selection methods. Feature selection methods play more important role when the number of features far exceeds the number of observations , which makes the traditional statistical methods infeasible for data analysis. This study presents quantitative and qualitative analysis results of applying feature selection methods in two case studies, multivariate calibration for determining soil carbonate content and cancer prediction using gene expression data, as a regression and a classification problems, with comparison of their performance on each case study. Feature selection methods compared include Least Angle Regression algorithm (LARS), Least Absolute Shrinkage and Selection operator (Lasso), Genetic Algorithm (GA), and classical methods such as forward and stepwise selection. Selected subsets by each method are used for the input of Support Vector Machines (SVM) for supervised modeling. Root Mean Square Error Prediction (RMSEP), Mean Squared Error (MSE) and Prediction Error (PE) and Area Under the Curve are quantitative criteria used to compare the methods. Due to a small number of samples, bootstrapping also applied when building models in order to obtain more reliable results. The results of case study 1 show high ability of LARS and Lasso in extracting effective features on carbonate content determination, as well as choosing the most true features (wavenumbers), while the least prediction errors were (RMSEP, MSE, and PE) obtained by applying subset selected by LARS. On the other hand, in case study 2 also LARS and Lasso show high accuracy in predicting prostate cancer.