Aspect-Category-Opinion-Sentiment Quadruple Extraction with Implicit Aspects and Opinions

Product reviews contain a large number of implicit aspects and implicit opinions. However, most of the existing studies in aspect-based sentiment analysis ignored this problem. In this work, we introduce a new task, named Aspect-Category-Opinion-Sentiment (ACOS) Quadruple Extraction, with the goal to extract all aspect-category-opinion-sentiment quadruples in a review sentence and provide full support for aspect-based sentiment analysis with implicit aspects and opinions. We furthermore construct two new datasets, Restaurant-ACOS and Laptop-ACOS, for this new task, both of which contain the annotations of not only aspect-category-opinion-sentiment quadruples but also implicit aspects and opinions. The former is an extension of the SemEval Restaurant dataset; the latter is a newly collected and annotated Laptop dataset, twice the size of the SemEval Laptop dataset. We finally benchmark the task with four baseline systems. Experiments demonstrate the feasibility of the new task and its effectiveness in extracting and describing implicit aspects and implicit opinions. The two datasets and source code of four systems are publicly released at https://github.com/NUSTM/ACOS.


Introduction
As a fine-grained sentiment analysis task, aspectbased sentiment analysis (ABSA) has received continuous attention. Its core task is to extract the opinion target described by an entity and its aspect (collectively referred to as aspect) from product reviews, and identify the sentiment toward the aspect (Liu, 2012). The standard aspect-based sentiment analysis task includes two basic subtasks: aspect extraction and aspect-based sentiment classification. By integrating the two subtasks, one can   identify an aspect-sentiment pair (g, s), where g is an aspect term, and s is the sentiment polarity toward the aspect. (Hu and Liu, 2004;Qiu et al., 2011) pointed out that the correlation between the aspect term and the opinion term is helpful for better ABSA. The following studies in this direction includes aspect-opinion co-extraction (Wang et al., 2016aYu et al., 2018;Li et al., 2018;Dai and Song, 2019), aspect-opinion pair extraction (Chen et al., 2020a;, and aspect-opinion-sentiment triple extraction (Peng et al., 2020;Wu et al., 2020;Mao et al., 2021), etc. However, most of the existing studies only considered the extraction of explicit aspects and opinions, while ignored the implicit ones. In fact, product reviews contain a large amount of implicit aspects and opinions. Table 1 summarizes the percentage of implicit aspects and opinions in the SemEval Restaurant dataset and our new Laptop dataset. It can be seen that nearly 44% of the review sentences contain implicit aspects or implicit opinions in the Laptop domain, and the percentage of sentences containing both implicit aspects and implicit opinions also exceeds 8%. Similar percentages can be observed in the Restaurant domain. Although some studies have attempted to solve the implicit aspect problem (Liu et al., 2005;Poria et al., 2014;Chen and Chen, 2016;Wan et al., 2020) or the implicit opinion problem (Lazhar and Guiyassa, 2016) from respective perspectives, there is still a lack of a unified framework that fully discusses and solves the implicit aspect and implicit opinion problems.
In this work, we introduce a new task named Aspect-Category-Opinion-Sentiment (ACOS) Quadruple Extraction, with the goal to extract all aspect-category-opinion-sentiment quadruples in a review sentence, and provide full support for aspect-level sentiment analysis with implicit aspects and opinions. As shown in Figure 1, in the review sentence "Looks nice and the surface is smooth, but certain apps take seconds to respond", surface is an aspect, Design is its category, smooth is the opinion toward this aspect, and Positive is the corresponding sentiment. The four elements are combined into an explicit quadruple surface-Design-smooth-Positive. In addition to that, there are two other quadruples that need to be extracted: Null-Design-nice-Positive which contains an implicit aspect, and apps-Software-Null-Negative which contains an implicit opinion.
The new ACOS Quadruple Extraction task has the following two challenges: • In term of dataset, so far there was no available dataset that is fully annotated with aspectcategory-opinion-sentiment quadruples including all implicit aspects and opinions; • In terms of modeling complexity, the task includes two extraction problems (aspect extraction, opinion extraction) and two classification problems (category classification, sentiment classification). It is challenging to effectively model the four subtasks together to construct quadruples containing implicit aspects and implicit opinions.
To address these issues, we further construct two new datasets, Restaurant-ACOS and Laptop-ACOS, for the new task. The former is an extension of the existing SemEval Restaurant dataset, based on which we add the annotation of implicit aspects, implicit opinions, and the quadruples. The latter is a brand new one collected from the Amazon Laptop domain. It has twice size of the SemEval Loptop dataset, and is annotated with quadruples containing all explicit/implicit aspects and opinions. We finally benchmark the task by establishing four baseline systems, Double-Propagation-ACOS, JET-ACOS, TAS-BERT-ACOS and Extract-Classify-ACOS, by adapting the representative approaches in aspect-opinion pair extraction, aspect-category-opinion triple extraction or aspect-opinion-sentiment triple extraction to ACOS Quadruple Extraction. The experiments on the two ACOS datasets demonstrate the feasibility of the new ACOS Quadruple Extraction task and its effectiveness in extracting and describing implicit aspects and implicit opinions.
The contributions of this work can be summarized as follows: • We introduce a new task named Aspect-Category-Opinion-Sentiment Quadruple Extraction, to address the implicit aspects/opinions issues in ABSA; • We construct two new datasets for the task, with ACOS quadruple annotations including implicit aspects/opinions; • We benchmark the task with four baseline systems. The experiments demonstrate the new task's advantage in addressing the implicit aspect/opinion issues.

Task
We first define the four elements of the ACOS Quadruple Extraction task based on (Liu, 2012). (Peng et al., 2020;Mao et al., 2021) provided good summaries of recent tasks and terminology in ABSA. For simplicity, in this paper we use aspect, category, opinion and sentiment to denote aspect term, aspect category, opinion term and sentiment polarity, respectively. They are defined as follows: • Aspect denotes an entity and its aspect indicating the opinion target, which is normally a word or phrase in the text; • Category represents a unique predefined category for the aspect in a particular domain; • Opinion refers the subjective statement on an aspect, which is normally a subjective word or phrase in the text; • Sentiment is the predefined semantic orientation (e.g., Positive, Negative, or Neutral) toward the aspect.
Aspect-Category-Opinion-Sentiment (ACOS) Quadruple Extraction is then defined as a task to extract a set of aspect-category-opinion-sentiment quadruples described in a review sentence containing n words r=[w 1 , . . . , w n ]: where a i -c j -o k -s l denotes an aspect-categoryopinion-sentiment quadruple, a i is the extracted aspect, c j ∈ C is its category, o k is the extracted opinion, and s l ∈ {Positive, Neutral, Negative} is its corresponding sentiment. 1 Note that a review sentence usually contains multiple aspects and opinions. The ACOS Quadruple Extraction task does not only identify four elements, but also combine them into a set of valid quadruples, meanwhile considering implicit aspects/opinions. As the implicit aspect/opinion is not explicitly expressed as a word or phrase, in case of implicit aspect we set a as NULL and use category c to describe the opinion target, and in case of implicit opinion we set o as NULL and use sentiment s to describe the semantic orientation.

Datasets
We construct two new datasets, Restaurant-ACOS and Laptop-ACOS, for the ACOS Quadruple Extraction task.

Source
The Restaurant-ACOS dataset is constructed based on the SemEval 2016 Restaurant dataset (Pontiki et al., 2016) and its expansion datasets (Fan et al., 2019;. Laptop-ACOS is a brand new Laptop dataset collected from the Amazon platform at the years of 2017 and 2018 (covering ten types of laptops under six brands such as ASUS, acer, Samsung, Lenovo, MBP, MSI and so on). It contains 4,076 review sentences, much larger than the SemEval Laptop datasets.

Annotation
The SemEval 2016 Restaurant dataset (Pontiki et al., 2016) was annotated with explicit and implicit aspects, categories, and sentiment. (Fan et al., 2019; further added the opinion annotations. We integrate their annotations to construct aspect-category-opinion-sentiment quadruples and further annotate the implicit opinions. For Laptop-ACOS, we annotate the four elements and their corresponding quadruples all by ourselves. We employ the aspect categories defined in the SemEval 2016 Laptop dataset. Two PhD students familiar with aspect-based sentiment analysis are selected as annotators for independent annotation with the annotation tool introduced by (Yang et al., 2017a). The strict quadruple matching F 1 score between two annotators is 75.86%, which indicates a substantial agreement between two annotators (Kim and Klinger, 2018). In case of disagreement, a third expert will be asked to make the final decision.

Statistics and Analysis
The basic statistics of the two datasets are reported in Table 2. The Restaurant-ACOS dataset contains 2286 sentences with 3658 quadruples, and the Laptop-ACOS dataset contains 4076 sentences with 5758 quadruples. As we have mentioned, a large percentage of the quadruples contain implicit aspects or implicit opinions. By comparing two datasets, it can be observed that Laptop-ACOS has higher percentage of implicit opinions than Restaurant-ACOS.
In Table 3, we further compare our two ACOS datasets with the existing representative datasets   For Restaurant-ACOS, we integrate the above annotations to construct ACOS quadruples. But it should be noted that we keep the sentences with implicit aspects in Restaurant-2016, and further annotate the implicit opinions. As a result, the size (including sentences, AO pairs and AOS triples) of Restaurant-ACOS is about 1.6 times that of Restaurant-2016-AO and Restaurant-2016-AOS.
The new Laptop-ACOS has 4076 review sentences. The numbers of annotations for aspect, category, opinion and sentiment are 4958, 4992, 5378 and 4958, respectively. By combining these elements, we construct 5035 AS pairs, 5726 AO pairs, 5731 AOS triples, 5227 ACS triples and 5758 ACOS quadruples, nearly twice the size of Restaurant-ACOS. 2 2 It is worth noting that the Restaurant-ACOS and Laptop-

Methods
We benchmark the ACOS Quadruple Extraction task with four baseline systems, namely, Double-Propagation-ACOS, JET-ACOS, TAS-BERT-ACOS and Extract-Classify-ACOS, by adapting the representative approaches in aspectopinion pair extraction, aspect-category-opinion triple extraction or aspect-opinion-sentiment triple extraction to ACOS Quadruple Extraction.

Double-Propagation-ACOS
Since Double Propagation (DP) is one of the representative rule-based methods for aspect-opinionsentiment triple extraction (Qiu et al., 2011), we propose to adapt it to our ACOS quadruple extraction task by first extracting all the aspect-opinionsentiment triples, followed by assigning the aspect category for each extracted triple. We name the adapted approach as Double-Propagation-ACOS.
Specifically, we first follow the DP algorithm to extract the aspect-opinion-sentiment triples, where we utilize the syntactic relations between aspects and opinions to iteratively extract them in each review, and rely on the sentiment lexicon to assign sentiments (i.e., Positive, Negative, and Neutral) to aspects and opinions in a bootstrapping manner. Second, to identify the aspect category of each extracted triple, we use the following strategy: if the aspect in the triple is in the training set, we take its most co-occurred aspect category as the final aspect category; otherwise, we adopt the aspect ACOS datasets are available for all subtasks in ABSA, including aspect-based sentiment classification, aspect-sentiment pair extraction, aspect-opinion pair extraction, aspect-opinionsentiment triple extraction, aspect-category-sentiment triple extraction, etc. category of the nearest aspect in the input review as the final aspect category.
Based on the two steps mentioned above, we can extract the ACOS quadruples in each review sentence.

JET-ACOS
As one of the state-of-the-art approaches for aspectopinion-sentiment triple extraction, JET  introduced an end-to-end framework to this task, by combining the identification of aspects, their corresponding opinions, and their sentiment polarities with a position-aware tagging scheme 3 . Similar to Double-Propagation-ACOS, we adapt JET to our task by first extracting the triple with JET, followed by predicting the aspect category for each extracted triple.
Specifically, we first obtain the candidate aspectopinion-sentiment triples based on JET, and then design a BERT-based model to get the aspect category of the extracted triples. Given the review sentence r, we first feed it to BERT to get the context-aware token representation H as follows: where h r = [h 1 , . . . , h n ] is the output representation for r. Next, given an extracted triple a-o-s, we can obtain the representation of the aspect and the opinion as u a = avg(h a ) and u o = avg(h o ), where avg(h a ) and avg(h o ) are the average vectors of words in the aspect h a and the opinion h o , respectively. We then concatenate u a and u o , and feed it to a fully-connected layer with the Sigmoid function for each category c: Given a-o-s and c, y c = 1 indicates a valid quadruple, and y c = 0 indicates an invalid quadruple.
In the training stage, we adopt the standard binary cross-entropy loss for optimization. In the inference stage, we combine the extracted aspectopinion-sentiment triples from JET and our predicted aspect categories to get all the quadruples from each review sentence.

TAS-BERT-ACOS
TAS-BERT (Wan et al., 2020) is one of the state-ofthe-art method for aspect-category-sentiment triple extraction, which integrates aspect category-based sentiment classification and aspect extraction in a unified framework by attaching the aspect category and the sentiment polarity to the review sentence and using it as the input of BERT. To adapt TAS-BERT to our ACOS extraction task, we propose to adopt the input transformation strategy in TAS-BERT to perform category-sentiment conditional aspect-opinion co-extraction, following by filtering out the invalid aspect-opinion pairs to form the final quadruples.
Specifically, given a review sentence r, an aspect category c ∈ C, and a sentiment s ∈ S, the input is constructed as follows: We then feed x to BERT to get the context-aware token representation H: where h r = [h 1 , . . . , h n ] is the output representation for r, h cs is the output representation for the concatenation of c and s, and h [CLS] is used for category-sentiment verification. We then perform aspect-opinion co-extraction over H by modeling it as a single sequence labeling task. Specifically, we employ a modified Begin-Inside-Outside (BIO) tagging scheme, which consists of five tags: {BA, IA, BO, IO, O}, indicating the beginning and inside of the aspect, the beginning and inside of the opinion, and others. We feed h r to a CRF layer to extract the aspects and opinions in r with respect to the input category c and sentiment s as follows: Y ao = [y ao 1 , . . . , y ao n ] = CRF(h 1 , . . . , h n ); (6) Next, we perform Cartesian Product on the extracted aspects and opinions to obtain a set of candidate aspect-category-opinion-sentiment quadruples:  On the basis of S ACOS , we average the vectors of tokens in the aspect and opinion, and then feed their concatenation [u a ; u o ] to a quadruple filter: (8) where y acos = 1 indicates a valid quadruple, and y acos = 0 indicates an invalid quadruple.

Extract-Classify-ACOS
Finally, we propose Extract-Classify-ACOS by adapting one of the representative aspect-opinion co-extraction system  to our ACOS quadruple extraction task. Specifically, the first step performs aspect-opinion co-extraction, and the second step predicts category-sentiment given the extracted aspect-opinion pairs. As shown in Figure 2, we first insert two [CLS] tokens at the beginning and the end of the review sentence r, and then feed the transformed input to BERT to obtain the context-aware token representations H as follows: Similar to the method in TAS-BERT-ACOS, the explicit aspect-opinion co-extraction is based on a CRF layer with the modified BIO tagging scheme.  We further apply two binary classification tasks on the [CLS] tokens to predict whether there is implicit aspect or implicit opinion. Thus, we can obtain the potential aspect set S A , opinion set S O , and perform Cartesian Product on S A and S O to obtain a set of candidate aspect-opinion pairs: Next, we model the category-sentiment classification as a multiple multi-class classification problem. Specifically, for each category c, we concatenate the average vectors of each aspect-opinion pair a-o, and feed them to a fully-connected layer with Softmax function as follows: (11) where s aoc ∈ {Positive, Negative, Neutral, Invalid} denotes its sentiment given current a-o and c, or indicates an invalid quadruple.

Experiments
We evaluate the performance of four baselines systems on two ACOS quadruple datasets.

Experimental Settings and Evaluation Metrics
In Extract-Classify-ACOS, we adopt BERT base (Devlin et al., 2018) as the basic encoder, which consists of 12 stacked Transformer blocks. During training, we use the AdamW optimizer of BERT with weight decay fix. The maximum length of the review sentence is set to 128, covering all sentences in two datasets. We set the batch size and learning rates in aspect opinion co-extraction and categorysentiment classification as [32, 2e-5] and [16, 3e-5], respectively. The dropout rate is set as 0.1. The batch size and learning rate in the category classification of JET-ACOS and the aspect-opinion pair filtering in TAS-BERT-ACOS are all set as [8, 5e-5], other settings of these two modules are the same as Extract-Classify-ACOS. We divide the original dataset into a training set, a validation set and a testing set according to Table 4.   In evaluation, a quadruple is viewed as correct if and only if the four elements as well as their combination are exactly the same as those in the gold quadruple. On this basis, we calculate the Precision and Recall, and use F 1 score as the final evaluation metric for AOCS Quadruple Extraction. Table 5 reports the ACOS quadruple extraction performance of four different systems on the two datasets. It can be seen that Double-Propagation-ACOS gets the lowest performance. It is reasonable that only using rules is somehow difficult to identify multiple implicit elements and their complex combinations in reviews.

Main Results
JET-ACOS and TAS-BERT-ACOS achieve comparable F 1 performance: the former is better on Restaurant-ACOS dataset and the latter is better on Laptop-ACOS.
Extract-Classify-ACOS achieves the best performance among four baseline systems. It outperforms JET-ACOS by 5.60 percentage points on Restaurant-ACOS and outperforms TAS-BERT-ACOS by 8.49 percentage points on Laptop-ACOS, respectively. The main advantage is that Extract-Classify-ACOS can achieve robustly higher recall score. In comparison, JET-ACOS has higher or comparable precision score but its recall is much lower.
It is also worth noting that the F 1 score of Extract-Classify-ACOS on both datasets are not high (0.4461 and 0.3580). It is reasonable because the evaluation metric is based on exact matching and the ACOS Quadruple Extraction is a more complicated task than the traditional ABSA tasks.

Effectiveness of Modeling of Implicit Aspects/Opinions
As we have mentioned, a large percentage of review sentences contain implicit aspects/opinions. Therefore, efficient modeling of implicit aspects/opinions is of great importance. To investigate the ability of different systems in addressing the implicit aspects/opinion problem, in Table 6  We further compare the performance on differ-

Case study
In Table 7, we further conduct case study by comparing the predictions of some representative approaches on five ABSA tasks including Aspect-Sentiment (AS) Pair extraction, Aspect-Opinion (AO) Pair extraction, Aspect-Category-Sentiment (ACS) Triple extraction, Aspect-Opinion-Sentiment (AOS) Triple extraction, and ACOS extraction. We choose four different sentences according to whether the aspect/opinion is explicit or implicit, and observe the predictions obtained by different approaches. It can be observed that: 1) RACL (Chen and Qian, 2020) accurately extracts the AS pairs in case of EA & EO, but it does not support implicit aspects and it fails to make predictions in case of EA & IO on our testing sentence; 2) SDRN (Chen et al., 2020a) is only capable of aspect-opinion pair extraction in case of EA & EO; 3) JET  can only extract aspectopinion-sentiment triples in case of EA & EO; 4) Although TAS-BERT (Wan et al., 2020) supports aspect-category-sentiment triple extraction for either implicit aspect or implicit opinion, it fails to give accurate predictions in case of IA & EO and IA & IO on our testing sentences; 5) As for the three ACOS baseline systems, JET-ACOS is only capable of ACOS quadruple extraction in case of EA & EO, and has a false prediction. TAS-BERT-ACOS and Extract-Classify-ACOS support ACOS quadru-ple extraction in case of both implicit aspects and implicit opinions. TAS-BERT-ACOS performs better than JET-ACOS but still fails in the case of IA & EO. Extract-Classify-ACOS performs generally the best and produces more accurate predictions in all cases.
In recent years, on the basis of traditional ABSC, a series of expansion tasks have appeared in this field. We divide these work into the following four categories: Aspect-Sentiment Pair Extraction. It also can be viewed as joint aspect extraction and ABSC. (Mitchell et al., 2013) first explored the opendomain aspect-sentiment extraction task by designing a variety of conditional random fieldbased models based on traditional discrete features. With the recent trend of deep learning, researchers have proposed various neural pipeline approaches Hu et al., 2019) or joint learning approaches for this task He et al., 2019;Chen and Qian, 2020).
Aspect-Opinion Pair Extraction. (Hu and Liu, 2004) first addressed the task in a pipeline manner. (Chen et al., 2020a) proposed to extract aspectopinion pairs with a double-channel recurrent network while taking the correlation between aspects and opinions into consideration.  designed a span-based multi-task learning framework to extract aspect-opinion pairs jointly. The work on aspect-opinion co-extraction (Wang et al., 2016aYu et al., 2018) can be viewed as the first stage of aspect-opinion pair extraction.

Aspect-Opinion-Sentiment Triple Extraction.
Considering the relation between aspect and opinion, (Hu and Liu, 2004) designed a feature-based opinion summary system, which identifies explicit aspect, opinion and sentiment, and integrates them into review opinion summaries. (Qiu et al., 2011) further proposed a Double Propagation method to utilize the syntactic relations between aspects and opinions to iteratively extract the aspect-opinionsentiment triples. More recently, (Peng et al., 2020) proposed a two-stage framework to first extract aspect-sentiment pairs and opinions separately, followed by matching them to obtain aspect-opinionsentiment triples.  further proposed an end-to-end position-aware tagging scheme to model the relations among aspect, opinion and sentiment. (Wu et al., 2020) proposed a Grid Tagging Scheme to address this problem. (Mao et al., 2021;Chen et al., 2021) transformed the triple extraction task into multi-turn machine reading comprehension task and achieved state-of-the-art performances.
Aspect-Category-Sentiment Triple Extraction. Previous two categories only focus on explicit aspect-based sentiment analysis, while ignoring the implicit aspects. To address this issue, (Liu et al., 2005) designed rule-based method to find the corresponding implicit aspects through the opinion existing in the review sentence. With the recent advances of pre-trained models, (Wan et al., 2020) proposed a BERT-based architecture to address this task in an end-to-end fashion.
Since the problem of implicit aspect and implicit opinion has not been systematically addressed in previous studies, in this work we introduce a new task for Aspect-Category-Opinion-Sentiment (ACOS) Quadruple Extraction with implicit aspects and opinions, construct two ACOS Quadruple datasets, and benchmark the task with four baseline systems.

Conclusions and Future Work
In this paper, we introduce a new task, Aspect-Category-Opinion-Sentiment (ACOS) Quadruple Extraction, aiming to systematically address the implicit aspect/opinion problem. We construct two new datasets for this task, with ACOS annotations including implicit aspects and implicit opinions. We finally benchmark the task with four baseline systems. Experiments demonstrate the advantages of the new task in aspect-based sentiment analysis with implicit aspects/opinions. The focus of this paper is the introduction of the new task and datasets. The proposed four baseline systems are relatively simple and leave much room for further improvements. We welcome future work proposing stronger models on this task. We also welcome the usage of our datasets on the other ABSA tasks.