Evaluation of feature selection methods based on artificial neural network weights

•A new methodology to evaluate the stability of WBFS is proposed.•A voting approach was used to compare different importance rankings.•Experiments on simulated and real-world datasets.•Existing WBFS methods are not stable on real-world datasets. Weight-based feature selection (WBFS) are methods used to measure the contribution of input to output in a trained artificial neural network (ANN). Furthermore, algorithms such as Garson’s rely upon a single best neural network model or the mean importance value of several ANNs. However, different initialization weights lead to different importance values, as reported in other studies. These differences are misleading since each rank could result in different scores, altering the position of a variable in a given rank. Therefore, we propose a new methodology to assess the stability of a WBFS method. In essence, the idea is to use a voting approach to evaluate the importance of rankings. The results showed that Garson’s, Olden’s and Yoon’s algorithms are more stable methods when applied to artificial datasets. Nevertheless, its stability is considerably reduced when applied to real-world datasets. Hence, we concluded that future work should take into consideration the aforementioned instability of existing WBFS methods as applied to complex real-world data.