Ensemble Approaches for Classification

Abstract

We present a classification method based on ensembles of multinomial logistic regression models, with each model fitted from a different set of predictors determined by a random partition of the feature space.? Our method allows an ensemble of multinomial logistic regression models to be used for high-dimensional data sets without a variable pre-selection.? A goal of the proposed method is to enable class prediction which is impossible by a single logistic regression model for a high-dimensional data due to the restriction that the sample size needs to be larger than the number of predictors.? The performance of the proposed model is compared to a single multinomial logit model.? We also present a new weighted voting ensemble method called WAVE.? A weight vector of classifiers that assigns higher weights to classifiers performing better on hard-to-classify instances is proposed.? To assess the degree of difficulty in classification, a weight vector of instances is also considered.? The two weight vectors influence each other and converge to a performance matrix of classifiers.? The final prediction of the ensemble of classifiers is obtained from this process.? The performance of the proposed method will be evaluated using real and simulated data sets. ?