Towards automatically filtering fake news in Portuguese

•An unprecedented fake news collection in the Portuguese language is presented.•Important open questions related to detecting fake news are raised and properly answered.•A comprehensive performance evaluation of established classification methods and features are presented.•Results with bag-of-words outperformed the results with the state-of-the art Word2Vec and FastText techniques.•The combination of linguistic-based features and bag-of-words-based features is recommended. In the last years, the popularity of smartphones and social networks has been contributing to the spread of fake news. Through these electronic media, this type of news can deceive thousands of people in a short time and cause great harm to individuals, companies, or society. Fake news has the potential to change a political scenario, to contribute to the spread of diseases, and even to cause deaths. Despite the efforts of several studies on fake news detection, most of them only cover English language news. There is a lack of labeled datasets of fake news in other languages and, moreover, important questions still remain open. For example, there is no consensus on what are the best classification strategies and sets of features to be used for automatic fake news detection. To answer this and other important open questions, we present a new public and real dataset of labeled true and fake news in Portuguese, and we perform a comprehensive analysis of machine learning methods for fake news detection. The experiments were performed using different sets of features and employing different types of classification methods. A careful analysis of the results provided sufficient evidence to respond appropriately to the open questions. The various evaluated scenarios and the drawn conclusions from the results shed light on the potentiality of the methods and on the challenges that fake news detection presents.