Multi-lingual text classification models to detect hate and offensive text

Hate speech is discourse that exploits stereotypes to create an antisocial worldview of hatred. Hatred is spread in today’s society through the usage of social media such as blogs and newsgroups. In this article, an overview of our method for detecting hate speech in many languages is discussed with English, Hindi and Punjabi being the primary focus of the article. The fundamental challenge for machine learning systems is the lack of readily available data of a high quality that has been tagged. To counter the issue data totaling around 510,000 lines was scraped from a variety of social networking platforms and labeled. Around 11% of the data was considered hateful and 89% was considered neutral. The data was manually labeled after being partially classified automatically using a word list that had been specified. The primary problem is the wide variety of verbal expressions of hatred, not all of which can be adequately represented by a dictionary. In order to train a number of separate models for text classification, a number of various algorithms, such as Random Forest, Linear SVC, Logistic Regression, and MultinomialNB, were tested.