On the comparison of classifiers' performance in emotion classification: Critiques and suggestions

In literature there is a huge body of references available which compare various classifiers in a particular application. However, the reliability of such a comparison is only valid if the model parameters, performance criteria and training environment are chosen in a fair framework, as successful application of a classifier is dependent on the those parameters. In this study we attempt to answer the questions below in a emotion detection framework, using classifiers such as KNN, SVM, RBF and MLP: Is the success of a classifier enough to make the claim that a classifier is "the best one" in a particular classification task? How is it possible to carry out a fair comparison between classifiers?