IRlab@IITV at SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media Using SVM

This paper describes the IRlab@IIT-BHU system for the OffensEval 2020. We take the SVM with TF-IDF features to identify and categorize hate speech and offensive language in social media for two languages. In subtask A, we used a linear SVM classifier to detect abusive content in tweets, achieving a macro F_1 score of 0.779 and 0.718 for Arabic and Greek, respectively.


Introduction
Nowadays, the internet is accessed by half of the world population 1 . People from different cultures and educational backgrounds interact on social media. They express their opinions, dissent and even hatred against any individual, or a group using some hate and offensive language. But use of hate and offensive language not only lowers the morale of individuals, but may cause mental agony and trauma. So automatic identification of offensive languages is need of the hour. Almost all social media have policies against use of abusive language, but identification is a challenge. It is not possible to monitor manually or using a static set of rules. Hate speech and offensive language belongs to natural language and therefore NLP techniques can be tried to search for offensive content from textual data. The enthusiasm to explore these sub-fields is to possibly limit the offensive language and hate speech on user-generated content, particularly on social media. Two popular social media platforms for researchers to study are Twitter and Facebook, a social network website where people "tweet" and "message" short posts.
We mainly focus on detecting whether a tweet or message contains offensive content or not (Subtask A). In this paper, a machine learning technique is used to identify the offensive language. Using the Arabic dataset annotated by Mubarak et al.,(2020) and Greek dataset annotated by Pitenis et al.,(2020), we train our Support Vector Machine (SVM) classifier using term frequency-inverse document frequency(TF-IDF) as features. The main aim of this paper is to establish a baseline for this two languages (Arabic and Greek) with traditional machine learning approach.
The rest of this paper is organized as follows. In Section 2, we briefly outline some previous work on offensive language identification. The dataset description, features extraction, and model description are presented in Section 3. Results are presented in section 4, followed by conclusion and future work in Section 5.

SemEval Sub-Task
According to  the aim of SemEval2020 is to build a system able to detect offensive content. For the first time in SemEval2019, three subtasks were proposed by Zampieri et al.,(2019). We only participate in Sub-task A for Arabic and Greek languages. Some tweet examples from training dataset are listed in Table 2 and Table 3 respectively. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 1 https://news.itu.int/itu-statistics-leaving-no-one-offline/ • Sub-task A : Offensive language Identification Sub-task A is coarse-grained binary classification system which classify tweets into two different classes -Non Offensive (NOT) -This tweet does not contain any offense speech.
• Sub-task B : Categorization of Offensive post Sub-task B is only for English dataset. It is a binary classification task. The posts labeled as OFF in Sub-task A, are further categorised into the following two: -Targeted Insult (TIN): -These posts contain an insult or threat to an individual or a group.
-Untargeted (UNT): -These posts contain swear words, expression of frustration but not targeted to any person or group.
• Sub-task C : Offense target identification Sub-task C is a multi-class classification task. After classifying a post as TIN in Sub-task B, it is further classified into following three types: Most of the previous works are based on English, but some work also reported in other languages like Arabic by Mubarak et al.,(2017), German by Ross et al.,(2017), Slovene by Fiser et al.,(2017)) and Chinese by Su et al.,(2017). Mubarak et al.,(2017) created a set of obscene words as a seeding list called SeedWords (SW) and computed Log Odds Ratio (LOR) for each word unigram and bigram. Like SemEval, GermEval was organized by Wiegand et al., (2018). Not only monolingual, but some attempted a more challenging case of code mixed language like Hinglish by Mathur et al.,(2018). HatEval 5 also organised to detect hate speech in Twitter. In this competition, they fixed two specific different targets, immigrants and women in both languages, Spanish and English. HASOC 6 is a shared task on Hate Speech and Offensive Content Identification in Indo-European Languages(English, Hindi, and German).

Data and Methodology
The subtasks in the shared task are different from each other. In Sub-task A, the aim is to identify offensive tweets. Sub-task A requires sensitivity to meticulous changes in the meaning of the word, while Sub-task B is more pronounced. However, both suffer from data sparsity. The earlier work on hate speech and offensive content identification depended on feature engineering. The handmade feature extraction requires knowledge about language, which is a difficult task for a person who is new to the language. Our classification method depends upon TF-IDF features. Figure 1 shows the methodology of the paper.

Dataset
We have used a training dataset which contains 8000 tweets for Arabic language (2020) and 8743 tweets for Greek language (2020) provided by SemEval 2020 Task

Data Preprocessing
We perform the following pre-processing operations on the dataset.
• All Twitter data are cleaned using tweet pre-processing library 7 .
• All stopwords are excluded using NLTK 8 library and we apply tokenization.

Feature Selection
Feature extraction plays an essential role in any classification task. The TF-IDF is used to extract the features from the pre-processed data. It is a weighting scheme based on the count of terms that are present in every post with the terms present in the entire corpus. We have chosen the feature-weight because it extracts the most descriptive terms from the post.

Model
Sub-task A requires the classifier to identify offensive language or not. As training dataset is imbalanced, we decided to use linear SVM classifier for binary classification. As mentioned by Fountana et al.(2018) and Kumar et al.(2018b) SVM classifier performs well when features are well selected. As noted by Dinakar et al.,(2011), SVM are a class of powerful methods for classification tasks, involving the construction of hyper-planes that at the largest distance to the nearest training points. Several papers cite support-vector machines as the state of the art methods for textual classification. For SVM classifier we used Scikit-learn implementation and data provided by shared task.

Results
We trained a linear SVM model on the training set and tested them on the validation set, using Scikit-learn 9 . The experimental results were based on tf-idf features. The result of the classifier on validation data is shown in Table 4. The highest F 1 scores are reported by team hwijeen(0.852), ALAMIHamza(0.9017) for Greek and Arabic language respectively.

Conclusion and future work
In this study, we report our systems for OffensEval Sub-task A. We trained a linear SVM classifier with a TF-IDF feature model to detect offensive tweets and messages. The evaluation result indicates that our system is capable of detecting offensive language and provides a good potential of identifying targets. However, there is room for improvement. In the future, to capture the subtle meaning and to overcome data sparsity, we plan to investigate a combination of selected surface features and pre-trained word embedding.