Leveraging Topic Relatedness for Argument Persuasion

Argumentation exposes individuals to conﬂict-ing viewpoints and can help them make more informed decisions based on the pros and cons of a particular issue. While recent studies of argumentation in Natural Language Processing have mainly focused on understanding the effect of various factors of persuasion (i.e. the source, audience, and language style), the impact of exploiting the relationships among controversial topics when predicting argument per-suasiveness remains under-explored. In this paper, we model the relatedness among controversial topics utilizing an embedding-based method based on individuals’ stances on the topics. We then leverage these topic embedding features and incorporate topic semantics features extracted from the arguments along with the previously studied factors of persuasion. We show that incorporating both types of topic relatedness features explicitly leads to signiﬁcant improvement in predicting persua-siveness and also helps enhance generalization to rare topics, in a few-shot setting.


Introduction
Emergence of social media and online argumentative forums provide users with a platform to gain information, express, and form opinions on a diverse set of controversial topics (i.e. issues). The increasing importance of these online platforms has motivated NLP researchers to use these platforms as one of the main domains to study the important factors of persuasion. In particular, prior work has shown that characteristics of the speaker (i.e. source), prior beliefs of the audience (Lukin et al., 2017;Durmus and Cardie, 2018), and language style (Feng and Hirst, 2011;Tan et al., 2016) are important factors in determining persuasiveness of the arguments in online argumentation platforms. Although there has been evidence in previous studies of Social Sciences that people's perceptions on a particular controversial topic may be related to their perceptions on other controversial topics (Judd and Krosnick, 1989;Sapra, 2012), the impact of exploiting this relationship among controversial topics are under-explored in NLP studies of persuasion. In this paper, we explicitly study the effect of incorporating topic relatedness among controversial topics in predicting argument persuasiveness.
To study the impact of involving topic relatedness in argument persuasion, we define two types of features: (1) topic embedding features and (2) topic semantics features. Prior work has shown that topic is an important factor (Das et al., 2016) to determine whether an emotional vs. a logical argument will be received positively by the audience. We hypothesize that encoding underlying relationship among topics with topic embedding features will be helpful in predicting persuasion since similar strategies may be effective for related topics. We further define topic semantics features to encode how focused vs. divergent each of the arguments made by the debaters is given the discussion topic, similar to Zhang et al. (2016).
We first develop an embedding-based technique inspired by (Barkan and Koenigstein, 2016) to determine the relationship among controversial topics. This methodology leverages users' stances on the topics to determine the relationship among them. We then incorporate the topic embedding features and topic semantics features, along with the previously studied factors of persuasion. We find that incorporating the topic relatedness features help improve state-of-the-art results in persuasion prediction. Moreover, we conduct experiments in a few-shot setting and show that these features help models achieve significantly better generalization performance for the rare topics.

Dataset
We use DDO  for our study. DDO includes 67,315 debates from 23 different categories, 36,294 users with their background information (e.g. political ideology, and religious ideology), and 198,759 votes from the users when they are as readers of these debates. For each debate, two debaters with different viewpoints express their opinions on a controversial topic in rounds. After the debate, voters evaluate the debaters with respect to various criteria and they share whether any of the debaters changed their stance on the topic. Users also have an opportunity to share their demographic and ideological information such as gender, ethnicity, income level, education level, political ideology, and religious ideology. They also share their stance on a pre-defined list of controversial topics (i.e. BIG IS-SUES, such as Abortion, Gay Marriage, and Global Warming etc.) 1 that we use to determine semantic relatedness among these controversial topics.

Topic Related Embeddings
Topic Embeddings. To capture the underlying relatedness between the debate topics, we learn the embedding for each controversial topics with a method inspired by Barkan and Koenigstein (2016). We hypothesize that the users' opinions (i.e. whether they are SUPPORTING or OPPOSING) on similar topics are related. We treat a set of controversial issues with the same stance from a user as a set of words appearing in the same context and use adapted Skip-gram algorithm proposed by (Mikolov et al., 2013). The embedding vectors are optimized by predicting the topic similarity that is defined as the probability that a pair of topics appearing in the same group with respect to users' opinions. We then can use these vectors to compare the similarity of each pair of the big issues. Table 1 shows the most similar controversial issues for each of the given issues, where the similarity is calculated by the cosine similarity between the embedding vectors. We observe that some of these associations can be more related to relatively intuitive topic similarity between these issues (i.e., Capitalism and Flax Tax, and Environmental Protection and Global Warming Exists). However, in some cases, the similarity between controversial issues may have more complicated motivations such as users' underlying ideologies (i.e., as in the case of Environmental Protection and Abortion which can be justified by the study conducted by Sapra (2012)). Therefore, this method may help identify relationships among controversial issues that are not as intuitive to come up with. Topic-Centric Attribute Embeddings. There is evidence showing there is a strong association between the users' demographics and their stances towards controversial topics (Sapra, 2012;Tedin et al., 1977). Although prior studies of persuasion has studied the effect of users' attributes on persuasiveness (Lukin et al., 2017;Cardie, 2018, 2019), they did not explicitly model the relationship between the users' attributes and their stance towards the controversial topics. To explicitly model this relationship, we create embeddings for users' attributes in a similar way as the issue embeddings introduced in Section 3.1. We optimize the embeddings with the probability of a certain attribute appearing with a given issue-stance pair for each particular user. Table 2 shows top similar issue-stance combinations for users with given political and religious ideology categories. Similarly, we calculate similarity by the cosine similarity between the embedding vectors. This approach reveals certain associations between users attributes (e.g. Political Ideology) and their stances (e.g. PRO vs. CON  example, we find that users with Conservative vs. Liberal political ideologies has different views on Global Warming (i.e. CON vs. PRO respectively) issue looking at the similarity of corresponding embeddings.

Predicting Persuasiveness
We aim to predict which debater, either the PRO or CON side, expresses more persuasive arguments in the debate (i.e., received more votes for the "Made more convincing arguments" criterion.). Debate Topic Representation. We collect debates related to list of issues in BIG ISSUES by using the words of these issues as keywords (e.g. debate topic "Abortion should be illegal if it pregnancy does not endanger the mother's life and she is adult." is related to BIG ISSUE "ABORTION"). The dataset includes 2,893 debates and 10,441 votes. We represent the debate topic with the topic embedding of the corresponding BIG ISSUE as introduced in Section 3.1. We further encode the text of the debate topic with a fine-tuned BERT (Devlin et al., 2019) taking average embedding of all the tokens to get the representation of the topic semantics. We concatenate these two types of embeddings to get the final representation of the debate topic.
Representing User Information. Previous work shows that both characteristics of the debaters and the audience and the linguistic features of the debate arguments are important factors in persuasion studies (Lukin et al., 2017;Cardie, 2018, 2019;Longpre et al., 2019). Similar to prior work, to encode the background information, we first represent the user background with one-hot representation (ONE-HOT) to capture the users' selections on the categories (e.g., gender, political ideology, religious ideology, and etc.) or the opinion similarity with the voters. However, this representation can be very sparse and not relevant to the topic information. Therefore, we also experiment with the topic-centric embedding-based method (ATT-EMB) proposed in Section 3.1. We compute the background similarity as the cosine similarity of the representation vectors for the users (i.e., debaters and voters).
Linguistic Features. Consistent with the prior work (Durmus and Cardie, 2018), to encode the arguments in the debates, we extract linguistic features including the information about the style (i.e., length, links), sentiment polarity, subjectivity (Wilson et al., 2005), and argument lexicon features (Somasundaran and Wiebe, 2010)) etc. Similar to the topic semantics introduced in Section 3.2, we also represent the semantics of the arguments with the same fine-tuned BERT.
Proposed Model. We employ a model that contains separate encoders to represent the debater characteristics, arguments, and topic-related features, as shown in Figure 1. The model encodes the debater's background information and opinions towards the BIG ISSUES, and combines the linguistic features extracted from arguments to represent the users. For the text in the debate, the model consists a siamese network structure (Semantic Block + FFNN block) (Reimers and Gurevych, 2019) to encode the relation between arguments and debate topics. Then, the model extracts the issue representation from the pretrained issue embedding introduced in Section 3.1. Finally, the user representation, together with the representation for the argument semantics and topic embedding, is passed through a multi-layer feed-forward neural network to predict the voter's perception on the persuasiveness.  Table 3: Macro F1 scores. Ours denotes the model explained in Figure 1. We split the collected debates to train (70%), validation (15%), and test (15%) sets.

Baselines
(1) Majority Baseline: We assign the label that appears the most in training set to be the prediction for all test instances.
(2) SVM: We concatenate debate topic representation, user and linguistic features and classify with SVM with RBF kernel.    Table 3 demonstrates the macro F1 scores for the baselines and the ablations for our model. We observe that our model outperforms the featurebased baselines significantly. For both the SVM model and the deep models, we observe a large performance drop when we use one-hot embedding representation features instead of topic-centric user embeddings. This shows that encoding the relationship between the user background and their opinions on the topics explicitly improves the prediction performance significantly. The experiments on Bi-LSTM and SBERT show that although large-scale pretrained language representation model helps achieve better performance than the baseline, encoding the semantics with deep neural network encoders alone is not as effective as our proposed method. Comparing to the baselines, our proposed method that utilizes the information from different components (i.e. users, language and topic) is more effective. Ablation study shows that components that encode the topic semantics (i.e. ATT-EMB and TOPIC) play an important role to achieve the best performance.
Few-shot Setting. We study whether the topic embeddings also enhance the generalizability across different issues. We split the debates in the test set into frequent and rare categories looking at how often debates with the same topic appear in the training set (more than 200 vs. less than 20). Table 4 shows the results comparing to the baselines for corresponding 324 frequent and 131 rare debates. We see that the gap between the prediction performance is significantly more when we remove the attribute embedding and the topic embeddings, which indicates that the topic-related embeddings benefit the knowledge transferring among debates with different topics.

Related Work
Persuasion Studies. Understanding the characteristics of persuasive language has been an im-portant area of study in the Sociology, Psychology (Kelman, 1961;Burgoon et al., 1975;Chaiken, 1987;Tykocinskl et al., 1994;Chambliss andGarner, 1996) andNLP communities (Hasan andNg, 2014;Habernal and Gurevych, 2016a,b;Fang et al., 2016;Al-Khatib et al., 2017;Wang et al., 2019). The emergence of social media and argumentative forums has further attracted researchers to study the dynamics of persuasion on these platforms, including Twitter (Tan et al., 2014), ChangeMyView community on Reddit (Tan et al., 2016;Hidey et al., 2017) and DDO . In this work, we use DDO since it includes a wide-range of user information including users' opinions on various controversial topics as well as well-structured debates with audience votes.
Topic Aware Argument Mining. Farra et al. (2015) studied the effect of topic relevancy or consistency on essay scoring. Bosc et al. (2016) proposes a dataset of Social Media data with coarse topic labels extracted from the hashtags (e.g., #Ap-pleWatch). Zeng et al. (2020) designed a model to encode the latent topics of argumentative conversations. Unlike the previous work, our work studies the effect of argument topic for structured debates explicitly for predicting persuasion.

Conclusion
In this paper, we study the impact of topicrelatedness in debate persuasion and find that involving the semantics and features of the debate topics will achieve the best performing model. Moreover, we find that using pretrained embeddings that jointly encode the issues related to the topics and people's characteristics will largely benefit the training process and generalizability. Finally, we find that focusing on the debate topic in making arguments can be an effective strategy in online debates.