An ordered search with a large margin classifier for feature selection

In this paper we propose a novel feature selection method for binary classification problems based on an ordered search process to explore the space of feature subsets. The method, called Admissible Ordered Search (AOS), uses a monotone evaluation function based on margin values calculated by a large margin classifier with an arbitrary Lp norm. This includes large margin classifiers with the L∞ norm, which minimize the L1 norm and are very useful in feature selection, since they produce sparse solutions. An important contribution of this paper is the development of the projected margin concept. This value is computed as the maximal margin vector projected into a lower-dimensional subspace and it is used as an upper bound for the hypothesis value. This enables great economy in runtime and consequently efficiency in the search process as a whole. AOS was tested on several problems and its results were compared to other feature selection methods. The experiments demonstrate the competitive performance of the proposed method in terms of generalization power and computational efficiency. •We propose an ordered search algorithm for feature selection problems.•A monotone evaluation function based on margin values is proposed.•The projected margin concept is introduced as an upper bound for margin values.•The strategy can be used with a L1 large margin classifier for sparseness.