“Killing Me” Is Not a Spoiler: Spoiler Detection Model using Graph Neural Networks with Dependency Relation-Aware Attention Mechanism

Several machine learning-based spoiler detection models have been proposed recently to protect users from spoilers on review websites. Although dependency relations between context words are important for detecting spoilers, current attention-based spoiler detection models are insufficient for utilizing dependency relations. To address this problem, we propose a new spoiler detection model called SDGNN that is based on syntax-aware graph neural networks. In the experiments on two real-world benchmark datasets, we show that our SDGNN outperforms the existing spoiler detection models.


Introduction
Spoilers on review websites, which reveal critical details of the original works, can ruin an appreciation for the works. Review websites, such as Rotten Tomato, IMDb, and Metacritic, provide self-reporting systems that tag spoiler information to warn users of spoilers. However, since selfreporting systems depend solely on the active participation of users, they cannot handle the fastgrowing volume of newly generated reviews. During the past decade, several machine learning-based spoiler detection (SD) models have been proposed to solve the inefficiency of self-reporting systems. Guo and Ramakrishnan (2010) proposed an automatic SD model that measures the similarity between reviews and synopses of movies. Support vector machine (SVM)-based SD models using handcrafted features have been proposed (Boyd-Graber et al., 2013;Jeon et al., 2016). Recently, attention-based SD models that utilize metadata of  Figure 1: Attention-based models focused on the word "killing" because the word is frequently used in spoiler sentences, which results in incorrect predictions. review documents achieve state-of-the-art performance on the SD task (Chang et al., 2018;Wan et al., 2019).
However, the attention-based SD models have a lack of using dependency relations between context words. Dependency relations are useful for capturing the semantics of given sentences and detecting spoilers. As shown in Figure 1, although the phrase "killing me" is not a spoiler because the phrase is a metaphor, the attention-based SD models often focus on the word "killing" and classify sentences that contain the phrase "killing me" as spoilers. By providing the information that the word "me" is used as the direct object of the verb "killing," SD models can understand that the phrase is a metaphor.
In this paper, we propose SDGNN, which is a new Spoiler Detection model based on syntaxaware Graph Neural Networks (GNNs) (Marcheggiani and Titov, 2017) for leveraging dependency relations between context words in sentences to fully capture the semantics. With the success of GNNs We also propose a dependency relationaware attention mechanism, which is a modification of the gating mechanism used by syntaxaware GNNs, to be suitable for the spoiler detection task. In SD, considering the relative importance of dependency relations. However, existing syntaxaware GNN-based models compute the importance of each dependency relation individually in sentences without considering the context of the given sentence. Our proposed dependency relation-aware attention mechanism considers the relative importance of dependency relations. Also, we adopt a previously proposed genre-aware pooling method (Chang et al., 2018) to utilize the genre of works efficiently. In the experiments, we demonstrate the effectiveness of SDGNN on two real-world benchmark datasets in both quantitative and qualitative ways.

Our Approach
SDGNN classifies whether a given sentence x = (w 1 , w 2 , · · · , w n ) is a spoiler sentence. SDGNN consists of three stages: contextualized word representation, dependency relation-aware attention mechanism, and genre-aware pooling.
Contextualized Word Representation Each word w in the given sentence x is represented with the pretrained word embedding vector (Pennington et al., 2014). We then utilize bi-directional LSTMs (Hochreiter and Schmidhuber, 1997) to encode contextualized word representations h (0) ∈ R d .
Dependency Relation-Aware Attention Mechanism While the gating mechanism in syntaxaware GNNs (Marcheggiani and Titov, 2017;Nguyen and Grishman, 2018) computes the scalar weight of each dependency relation, it does not consider the relative importance of dependency relations, which varies depending on the context of the given sentence. We present a dependency relation-aware attention mechanism that considers the relative importance of dependency relations in the given sentence. The relation-aware attention weights are computed as follows: whereâ k L(u,v) is a scalar attention weight of the dependency relation label L(u, v) of the edge between word nodes u, v. g is the non-linear function and exp(·) is an exponential function.
Finally, we aggregate the latent feature for each node u as follows: where W (k) ∈ R d×d and b (k) ∈ R d are the weight matrix and bias term, respectively, for the k-th layer. f is a non-linear function. h (0) is the outputs of the bi-directional LSTMs in the previous stage. There are two main differences between our proposed dependency-aware attention mechanism and the gating mechanism used by syntax-aware GNNS. First, the dependency-aware attention mechanism employs the softmax function to capture the relative importance of dependency relations, while the gating mechanism computes the scalar weights by the inner-product of latent features of words and dependency relations. Second, the gating mechanism utilizes only three dependency relations (forward, backward, and self ) because of the over-parameterization issue. On the other hand, our proposed dependency relation-aware attention mechanism utilizes all the 82 types of dependency relations without suffering from the overparameterization issue since the weight matrix in Equation 3 does not depend on the number of relations. The number of trainable parameters of SDGNN is proportionate to d 2 while that of syntaxaware GNNs is proportionate to |L| · d 2 , where |L| is the number of relations.
Genre-Aware Pooling Genre information is useful for detecting spoilers. To leverage genre information, we employ a genre-aware pooling method following Chang et al. (2018). The genre-aware pooling computes the attention weights between the latent features of words and a genre feature captured from genre information of works. We then obtain a latent feature vector x for the given sentence x.
Optimization We compute the spoiler probabilityŷ of the given sentence x with the following the linear transformation: where w and b are trainable parameters, and σ is a sigmoid function. We use the weighted binary cross entropy (Wan et al., 2019) as the loss function.
where y id the ground truth of spoiler information and D indicates the dataset. η is a hyperpameter used to balance the number of spoiler and nonspoiler labels in the training data. All the trainable parameters of SDGNN are updated by minimizing the loss function with gradient descent.   and ReLU for f . We set k = 2 for SyntacticGCN, C-GCN, and SDGNN. We use η = 0.05 for the Goodreads dataset, which is unbalanced.

Results
The experimental results are summarized in Table 2. Evaluation results show that our proposed SDGNN outperforms all the baseline models including attention-based models. This result demonstrates that our proposed dependency relation-aware attention mechanism contributes to improving SD performance.

Analysis of Relative Importance
To further demonstrate the usefulness of the relative importance of dependency relations, we conducted quantitative and qualitative analysis.
Quantitative We compared SDGNN with the more syntax-aware GNN-based models, Syntac-ticGCN (Marcheggiani and Titov, 2017) and C-GCN (Zhang et al., 2018). We trained and evaluated the models on the Goodreads dataset. We   Figure 2: Partial graphs of dependency parse trees with dependency relation-aware attention weights.
utilize contextualized word representations and the genre-aware pooling method to SyntacticGCN and C-GCN. The evaluation results are summarized in Table 3. Our proposed SDGNN outperformed Syn-tacticGCN and C-GCN. This result demonstrates that our proposed attention mechanism is effective by considering the relative importance of dependency relations. Although SDGNN significantly reduced the number of parameters, SDGNN achieved better results compared to SyntacticGCN and C-GCN.
Qualitative In Figure 2, the attention weights of the adverbial modifier (advmod) linked to the words "eventually" and "viciously" are high, which indicates that adverbial modifiers frequently can be important hints for detecting spoilers. In the right partial graph, the attention weight of the (dobj) is relatively higher than that in the left partial graph. Since the word "mother" is not typically used as the object of the word "kill" in the original works, the phrase "kill mother" is a critical hint in detecting spoilers, and SDGNN effectively captures the phrase.

Case Study
We sample several sentences from the test set of the Goodreads dataset to explore how the models detect spoilers. Table 4 shows the visualization of attention scores in the pooling layer obtained by DNSD and SDGNN, respectively. The first sentence contains the verb "killing," but it is not a spoiler sentence because the phrase "killing me" is a metaphor. In this case, DNSD failed to correctly classify the sentence since DNSD cannot fully capture the semantics of the sentence. On the other hand, SDGNN focused on not only the word "killing" but also on the word "me" and classified the sentence correctly since SDGNN employs the dependency relation (dobj) between the word "me" and the word "killing". The second sentence is a non-spoiler because it is obvious that villains are vicious in most original works. DNSD classified the sentence as a spoiler because the model solely focused on individual words such as "villains", "vicious", and "insane", rather than the understanding of the overall semantics of the sentence. On the other hand, SDGNN classified the sentence correctly as the word "are" is used to describe characters in many cases, and SDGNN understands the semantics of the sentence.

Discussion
Dependency Parsing on User-Generated Texts The spoiler datasets are user-generated texts, which are intrinsically noisy. To examine the influence of noises on dependency parsing results and the performance of SDGNN, we sampled 100 sentences from Goodreads. We manually classified whether the sentences are noise or not, and 28 of 100 sentences were classified as noisy sentences. Dependency parsing results on well-structured sentences seem good, but dependency parsing results on noisy sentences are poor. However, there is no significant gap in performance. SDGNN achieved 85.7% accuracy on noisy sentences and 87.5% accuracy on well-structured sentences. Since our proposed dependency relation-aware attention mechanism of SDGNN filters noisy information, SDGNN could detect spoilers even on noisy sentences.

Subjectivity in Judging Spoilers
Since judging a sentence as a spoiler is a subjective task, label inconsistency occurs in spoiler datasets crawled from self-reporting systems. Guo and Ramakrishnan (2010) found that 23% of the labels of their manually labeled data is different from the original labels of IMDb reviews. One of the ways to mitigate label inconsistency is to solidify the definition of a spoiler. Although the TV Tropes site defines spoilers, efforts should be made for a more rigorous and linguistic definition in future studies. Another possible way is to employ reviewers' information in detecting spoilers. Reviewer biases of SpoilerNet can alleviate label inconsistency between users.

Conclusion
In this paper, we proposed a novel spoiler detection model called SDGNN which is based on syntaxaware GNNs that utilize dependency relations between context words. We also proposed a dependency relation-aware attention mechanism for considering the relative importance of dependency relations. In the experiments, our proposed SDGNN model achieved the state-of-the-art performance on two spoiler datasets. Our experimental results demonstrate the effectiveness of dependency relations in the spoiler detection task and our dependency relation-aware attention mechanism.