SWEAT: Scoring Polarization of Topics across Different Corpora

Understanding differences of viewpoints across corpora is a fundamental task for computational social sciences. In this paper, we propose the Sliced Word Embedding Association Test (SWEAT), a novel statistical measure to compute the relative polarization of a topical wordset across two distributional representations. To this end, SWEAT uses two additional wordsets, deemed to have opposite valence, to represent two different poles. We validate our approach and illustrate a case study to show the usefulness of the introduced measure.


Introduction
In this short paper, we introduce a method to score polarization of different corpora with respect to a given topic. The method is intended to support studies where two different corpora are compared (e.g., news sources inspired by different political positions or social communities characterized by different viewpoints) to investigate whether they convey implicit attitudes towards a given topic. This corpus-wise comparison -its main peculiarity with respect to the large body of work proposed to study and correct bias in NLP models (we refer to (Garrido-Muñoz et al., 2021) for a detailed survey on bias in NLP models) -is based on a new measure that we introduce, the Sliced Word Embedding Association Test (SWEAT).
SWEAT is an extension of the Word Embedding Association Test (WEAT) proposed by Caliskan et al. (2017), which measures the comparative polarization for a pair of topical wordsets (e.g., insects and flowers) against a pair of attribute wordsets (e.g., pleasant and unpleasant) in a single-corpus distributional model (e.g. 1950 American newspaper articles). In this context, with polarization we refer to the phenomenon for which two communities have opposite attitudes against some topic.
With SWEAT we extend this approach by measuring the relative polarization for a single topical wordset -the topic, using a pair of stable attribute wordsets deemed to have opposite valence -the two poles, in a pair of aligned distributional models representing the semantics of two different corpora. We explain the rationale behind SWEAT with an example. Suppose that we want to investigate whether two different Italian news sources, e.g., La Repubblica (known to be closer to center-left political positions) and Il Giornale (known to be closer to center-right political positions) hold different and opposite viewpoints about a topic, e.g., "Berlusconi" (a reference center-right Italian politician in the recent past). We can collect a news corpus from La Repubblica and one from Il Giornale to train two different distributional models in such a way that they are aligned (Hamilton et al., 2016b;Carlo et al., 2019;Cassani et al., 2021).
We expect that some words have stable meanings while other change across corpora reflecting the different viewpoints. We can then select a set of words describing the "Berlusconi" topic, whose representations are expected to differ across corpora, and two wordsets having respectively positive and negative valence (the two poles), whose representations are expected to be stable across corpora. The main idea behind SWEAT is the following: if the two corpora hold polarized views about the topic, the "Berlusconi" wordset will be associated more strongly with the positive rather than with the negative pole in one corpus (Il Giornale), while the opposite association will hold in the other corpora (La Repubblica). SWEAT measures this difference and reports effect size and significance.
Contributions. We introduce the SWEAT, a novel statistical measure to study relative polarization in distributional representations. We additionally introduce a lexicon selection pipeline and an easy-to-use code to create visualizations. We believe our measure can be useful for different use cases in the computational social science field. We share a repository with an easy to use implementation of our measure. 1 2 Background: WEAT Caliskan et al. (2017) introduce the Word Embedding Association Test (WEAT) to test whether distributional representations exhibited the same implicit biases detected in social sciences studies through behaviorally-measured word associations (Greenwald et al., 1998).
The WEAT compares the relative associations of a pair of target concepts X and Y (e.g., Science and Arts) to a pair of attribute concepts A and B (e.g., Male and Female) in a distributional vector space E; X, Y , A, and B are all sets that contain representative words for the concept. The statistical measure is based on the following formula: The value s(w, A, B) is instead computed as: Where the effect size is defined as: Significance is computed through a permutation test (Dwass, 1957) over the possible partition of equal size for the union of target-wordsets The p-value is computed as the rate of scores, from all possible permutations, that are higher than the tested one: Depending on the sign of the score the association could be either X ∼ A, Y ∼ B for positive scores and X ∼ B, Y ∼ A for negative ones. where the ∼ indicates semantic association. 1 https://github.com/vinid/SWEAT

SWEAT Pipeline
Our SWEAT pipeline is composed of two main components: a statistical test to evaluate relative polarization and a lexicon refinement pipeline -based on aligned word embeddings -to help researchers select the pole wordsets.

Measuring Polarization with SWEAT
The SWEAT measure operates following the same structure as the WEAT with one key difference: given two corpora SWEAT uses two corpusspecific distributional representations E 1 and E 2 instead of one. Thus, the relative mean associations depend explicitly on the corpus-specific embedding functions that map words to the respective embedding spaces.
We define the SWEAT score S(W, E 1 , E 2 , A, B) as follows: where W is the topical-wordset and A and B are the pole-wordsets. Similarly as in WEAT, the score sign indicates the detected associations: a positive sign indicates that the representations in E 1 of the target wordset are relatively more associated with the pole-wordset A than the representations of the same words in E 2 , while a negative sign indicates the opposite. The effect size and significance level follow the same structure as the WEAT and are omitted for brevity.
Observe that the wordsets A and B are not forced to represent opposite concepts (e.g., positive and negative), but, if they are, SWEAT provides a measure for scoring polarization.
To preserve comparability between the embedding spaces, it is important to align them, a task that is common in temporal word embeddings (Hamilton et al., 2016b;Carlo et al., 2019) and in crosslingual word embeddings (Ruder et al., 2019). The alignment step ensures the comparability of the slices when computing the similarities and supports lexicon refinement as described below.

Lexicon Refinement
The definition of the pole-wordsets (A and B) can be supported by the use of lexica. However, words in a lexicon might have different meanings depending on the context in a representation E k ; Thus, we first apply a refinement method based on the aligned word embedding spaces. We leverage aligned word embeddings to filter out unstable words from the lexicon. We keep a word in the lexicon if moving its word vector from E 1 to E 2 brings us to the same word and vice-versa. This process is done to ensure that the selected pole words do not change the representation between corpora.
Moreover, low-frequency words tend to produce lower-quality representations. We apply the Zipf (Van Heuven et al., 2014) relative frequency measure to exclude such low-frequency words from the pole-wordsets; following the recommendations in (Van Heuven et al., 2014), we select words with Zipf score > 5.

Visualization
SWEAT comes with easy to create visualizations. For example Figure 1 shows four cumulative visualizations illustrating four explored relations. For each model, the two color-coded areas indicate the sums, over all the topical-wordset elements x ∈ X, of the mean associations to the two the pole-wordsets A and B, one for each color.
The black cumulate dot over each bar-chart indicates the sum of its two color-coded parts.
These visualizations should not be used to directly compare different SWEAT scores because the horizontal scale adapts to the effects' magnitude; effect sizes can be used instead.

Interpretation of SWEAT
To conclude, we can summarize how to interpret SWEAT scores and plots as follows. SWEAT measures a difference in the association between corpus-specific representations of a topical-wordset and two attribute wordsets. When the effect size is high in absolute value, the difference is also high. When attribute wordsets represents two poles, a high difference in association reveals different attitudes towards the two pole-wordsets. When the cumulative associations between the corpus-specific representations of the topic and either pole-wordset have different signs, SWEAT reveals a polarization effect in addition to a plain difference.

Validation on the Reddit Corpus
We validated the SWEAT measures using subcorpora from the Reddit Corpus. 2 The corpus, previously used by Hamilton et al. (2016a), contains all the posts on the social media platform Reddit for the year 2014.
We selected for this experiment the English boards AnarchoCapitalism-Frugal and Baby-Bumps-Childfree. The first pair is centered around topics of finance and capitalism, with AnarchoCapitalism (AC) expected to have positive polarization, due to its pro-capitalism beliefs, and Frugal (FR) negative.
The second pair is focused on parenting (i.e. having children), with BabyBumps (BB) expressing positive opinions about this matter and Child-Free (CF) being negatively polarized. We do pairwise comparisons using the SWEAT measure. The embeddings are generated and aligned using TWEC (Carlo et al., 2019;Cassani et al., 2021).
Lexicon and Wordset Selection. For the polewordsets, we used the Inquirer Sentiment Lexicon (Stone et al., 1966) that was selected for its large size and generality. We used the pipeline described in the above section as a filter.
The topical wordsets for each pair, AC-FR and BB-CF, were selected in a semiautomatic fashion (see Appendix) in order to manually remove ambiguous terms which could bias the final result (e.g., the term fine is polysemous denoting both a financial penalty and good, and thus had to be removed from both pole and topical wordsets to prevent undue associations). The two derived topic wordset for capitalism and parenting are presented below: capitalism: market, value, economic, capital, price, wealth, profit, companies, interest, cost, competition, trade parenting: baby, birth, child, daughter, family, father, kid, mother, pregnant, son, parent, children Table 1 shows the results for each subreddit pair. Statistically significant relative polarizations are detected on the appropriate topics only (α = 0.01): polarization in the parenting topic is observed when considering the ChildFree-vs-BabyBumps subreddits and not when considering the AnarchoCapitalis-vs-Frugal subreddits, and vice-versa when considering the financial topic. Effect sizes are also coherent with the experiment setup, presenting very large (Cohen, 1988) magnitudes for the two relations of interest. Figure 1 presents the four cumulative visualizations illustrating the four explored relations. The Figure shows two centered horizontal stacked barcharts, one for each aligned model E 1 , E 2 . The two significant polarizations can be seen on the main diagonal: in the top left for the capitalism topic the FR corpus can be seen being strongly negative while the AC one has an overall positive but more nuanced position; in the bottom right, the polarization difference over the parenting topic is much more pronounced, with CF carrying a strong negative polarization in stark contrast with the positive one expressed by BB, thus confirming our hypotheses of polarization of the subreddits.

Case Study on Italian Newspapers
We provide an analysis on Silvio Berlusconi that focuses on the homonymous politician who, among other things, is the indirect owner of the Italian Il Giornale (GIO) newspaper and is, as an exponent of the center-right coalition, seldom praised by more left-leaning newspaper such as La Repubblica (REP). The dataset (see Appendix) contains a collection of roughly 40 thousand articles in Italian from March 2018 to June 2019 for REP and GIO.
As a lexicon we used Sentix (Basile and Nissim, 2013). Only the elements meeting all the following empirical criteria were considered in the final lexicon: POS tagging as either adjective or verb; intensity equal or greater to 0.75; either positive or negative score equal or greater to 0.75; polarity equal to ±1 and that are single-word lemma. We then applied the lexicon filtering pipeline. We conducted the analysis on the topic Silvio Berlusconi manually selecting the following topical-wordset: Berlusconi: cavaliere, berlusconi, arcore, mediaset, fininvest, silvio, rete4, fi, pdl, iene, vespa, tg5 The polarization detected by the SWEAT analysis confirms a significantly more positive polarization in the GIO corpus and a more negative one in REP (SWEAT = −0.795, d = −1.3276, p-value = 0.0038). Figure 2 shows the results of this association and confirms our general intuition.

Limitations
The SWEAT is able to capture relative semantic polarizations of topic wordsets. However, the measure is not suitable for fully unsupervised datadriven inference on the polarization of a corpora collection, similarly to the WEAT measure. The measure results are dependent on the choice of topic wordsets, and it is possible to find that some wordset is significantly polarized even though its elements are not semantically related to each other. For this reason, it is advised that researchers exercise caution in selecting which words to include in the topic wordset, and which to exclude.

Related Work
There has been great attention to the problem of polarization and bias in word embeddings in the latest years. We refer the reader to (Garrido-Muñoz et al., 2021;Blodgett et al., 2020) for relevant works. SWEAT and the applications we have shown have also a strong connection with the body of work related to the detection of hyper-partisan media (Kiesel et al., 2019;Bestgen, 2019), where extreme left-wing or right-wing positions of the news have to be detected.
We want to mention that the introduction of the WEAT measure has have been of great influence on the community. Indeed recently, Zhou et al. (2019) extended WEAT to have specific support for gendered languages (where different terms can have male and female counterparts). Chaloner and Maldonado (2019) use the WEAT measure to show how different domains contain different biases. Instead, Lauscher and Glavaš (2019) propose XWEAT as a cross-lingual version of the WEAT measure.

Conclusions
We have described a new measure to compute relative polarization on corpus-specific distributional representations. Measuring implicit attitudes is a crucial endeavor in social sciences, and in particular social psychology (De Houwer et al., 2009). Recent developments have shown that such attitudes are not only observable in human behavior, but can also be captured via text analyses (Caliskan et al., 2017;Bhatia, 2017). SWEAT brings a further advancement in this research line by allowing to validate hypotheses on the polarization of content from different text sources.

Ethical Considerations
SWEAT can be used to capture relative polarization and we are aware that the measures can also be used to test the significance of ethical polarization in different contexts. As we remark in Section 6 the measure we implemented comes with some limitations that have to be considered during experiments.
Topical Candidate Selection An initial set of candidate words was identified by collecting the top-100 most frequent 6 words from the shared vocabulary of all four corpora after removing stopwords and language operators (i.e. elements with little semantic connotation like auxiliary verbs or adverbs).
The topical wordset was then compared with the polarization wordsets for that pair, removing ambiguous terms, and lastly a manual separation into topics was performed to identify the true wordset of interest and removing other subtopics (for example, the candiate wordset for parenting also included terms related hospitals and medicine or the capitalism one had terms related to cryptocurrencies).

A.2 Italian NewsPapers
Data from Italian Newspapers have been kindly provided by SpazioDati. 7 We removed punctuation from the text and made it lowercase.
We used the following empirical criteria to select a subset of the Sentix lexicon: POS tagging as either adjective or verb; intensity equal or greater to 0.75; either positive or negative score equal or greater to 0.75; polarity equal to ±1 and where single-word lemma. This is the list of words we extracted: positive: meglio, bello, migliore, considerato, giusto, felice, importante, grande, semplice, maggiore negative: morto, peggio, difficile, vecchio, impossibile, pericoloso, male, purtroppo

B Replication: Experiment Parameters
To generate the aligned word embedding representations, we train TWEC (Carlo et al., 2019) using a dimension of the embeddings equal to 100 and a window size of 5. Iterations are set to 5 for both the static and the dynamic iterations.

C Details: Visualization
To aid the exploration and interpretation of SWEAT results the framework implements two main visualizations: an aggregate plot for the two models of the cumulative associations to the two pole-wordsets ( Figure 3); a detailed view of the association distribution over the two pole-wordsets for each element of the topical-wordset (Figure 4). 6 ordered by average Zipf measure 7 https://spaziodati.eu/en/ The first visualization (Figure 3) shows two centered horizontal stacked bar-charts, one for each aligned model E 1 , E 2 . For each model the two color-coded areas β A , β B indicate the sums, over all the topical-wordset elements w ∈ W , of the mean associations to the two the pole-wordsets A and B, one for each color.
The black cumulate dot over each bar-chart indicates the sum of its two color-coded parts, i.e. the w s(w, E k , A, B) relative to the model E k associated to that bar-chart. The final score is not directly encoded in the visualization, but is given by the difference between the position of the first and second cumulate dots.
The first visualization can also be broken down in its main parts, for which we provide a second visualization ( Figure 4) also divides the two aligned models E 1 , E 2 into distinct views, this time two side by side canvasses of pairs of color-coded boxplots 8 . In each canvass a pairs of boxplots represent, for an element of the topical-wordset x i ∈ X, the two distributions δ x i A , δ x i B of its associations to the pole-wordsets A and B, using the same colorcoding as the first visualization; across the two canvasses pairs are horizontally aligned, indicating the association distributions of the same topical word in the two models.
Additionally for each pair of boxplots a colorcoded arrow connects the two means to help illustrate the relative mean association s(x i , E k , A, B) for that word: the arrow always connects A → B and takes the color of the dominant pole, i.e. the one with the stronger mean association which in turn contributes in the cumulative sum for the final score.

D Details: Computing Infrastructure
We ran the experiments on a common laptop. All the computations are run on CPU, the model name is: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz. The running time for the SWEAT measure is bound by the number of words in the wordset and in the lexicon; however the computation of the SWEAT described in this paper take few minutes.